Unity embeddedbrowser browser plug-in event communication

Embedded Browser The browser plug-in is based on Chrome Embedded browser in the kernel , Embedded in many development languages and frameworks , Provide convenient web browsing support for developers .
Embedded Browser The browser works by running the browser process in the background , Analyze the website through the communication of the background browser , synthesis Texture Map frame , Pass to Unity, meanwhile Unity You can call the interface , Interact with browser processes .

Embedded This integration method is different from the mobile terminal WebView Integration mode （WebView Is to open up a piece of screen space for WebView Use , The caller and WebView There is not much correlation between ）, Provide more free integration channels , Web pages can be displayed in 3D space and solid surface , More flexible use .

Simulate mouse input

Usually , Whether it's Screen Canvas Or three-dimensional Mesh Superficial Embedded page , The plug-in has provided a seamless input method , Provided internally MapPointerToBrowser It has helped users convert 3D coordinates to Browser Plane coordinates .
If we want to customize the input , Such as remote control , Send control coordinates and mouse button communication through the remote host , Drive non user hardware input , The usual approach is to introduce Windows DLL, The requirements can be achieved by simulating the mouse

  #region DLLs
  [DllImport("user32.dll")]
  private static extern int SetCursorPos(int x, int y); // Set cursor position 
  [DllImport("user32.dll")]
  private static extern bool GetCursorPos(ref int x, ref int y); // Get cursor position 
  [DllImport("user32.dll")]
  static extern void mouse_event(MouseEventFlag flags, int dx, int dy, uint data, System.UIntPtr extraInfo); //  Mouse events 
                                                                                                      //  Method parameter description 
                                                                                                      // VOID mouse_event(
                                                                                                      // DWORD dwFlags, // motion and click options
                                                                                                      // DWORD dx, // horizontal position or change
                                                                                                      // DWORD dy, // vertical position or change
                                                                                                      // DWORD dwData, // wheel movement
                                                                                                      // ULONG_PTR dwExtraInfo // application-defined information
                                                                                                      // );

  public enum MouseEventFlag : uint
  {
    
    Move = 0x0001,
    LeftDown = 0x0002,
    LeftUp = 0x0004,
    RightDown = 0x0008,
    RightUp = 0x0010,
    MiddleDown = 0x0020,
    MiddleUp = 0x0040,
    XDown = 0x0080,
    XUp = 0x0100,
    Wheel = 0x0800,
    VirtualDesk = 0x4000,
    Absolute = 0x8000
  }
  #endregion

Calling code

  IEnumerator MouseClick(int x, int y)
  {
    
    SetCursorPos(x, y);
    mouse_event(MouseEventFlag.LeftDown, 0, 0, 0, System.UIntPtr.Zero);
    yield return new WaitForSeconds(0.1f);
    mouse_event(MouseEventFlag.LeftUp, 0, 0, 0, System.UIntPtr.Zero);
  }

When debugging and using this method , Because the window resolution is not 1：1 Full screen causes coordinate mapping error 、 The window is not in the front desk 、 Interfere with normal mouse operation and so on , Use is very unfriendly .

call Embedded Interface direct input

View the source code BrowserInput.cs, You can see that the plug-in handles the input event code

	private void HandleMouseInput() {
    
		var handler = browser.UIHandler;
		var mousePos = handler.MousePosition;
		var currentButtons = handler.MouseButtons;
		var mouseScroll = handler.MouseScroll;
		if (mousePos != prevPos) {
    
			BrowserNative.zfb_mouseMove(browser.browserId, mousePos.x, 1 - mousePos.y);
		}
		// ...
}

Core code

BrowserNative.zfb_mouseMove(browser.browserId, mousePos.x, 1 - mousePos.y);

EmbeddedBrowser.Browser Is virtual to Unity The use of Browser object , The actual browser object is BrowserNative A series of definitions DLL Interface .
browserId Is an array index , Tag a list of all actual browser instances
EmbeddedBrowser.Browser Add... When creating allBrowsers list

      unsafeBrowserId = newId;
      allBrowsers[unsafeBrowserId] = this;

Take a ,BrowserNative Contains a series of browser interfaces , Directly calling these interfaces can extend the functions defined by the plug-in .
Code excerpt

	/** * Reports the mouse's current location. * x and y are in the range [0,1]. (0, 0) is top-left, (1, 1) is bottom-right */
	public delegate void Calltype_zfb_mouseMove(int id, float x, float y);
	public static Calltype_zfb_mouseMove zfb_mouseMove;


	public delegate void Calltype_zfb_mouseButton(int id, MouseButton button, bool down, int clickCount);
	public static Calltype_zfb_mouseButton zfb_mouseButton;


	/** Reports a mouse scroll. One "tick" of a scroll wheel is generally around 120 units. */
	public delegate void Calltype_zfb_mouseScroll(int id, int deltaX, int deltaY);
	public static Calltype_zfb_mouseScroll zfb_mouseScroll;


	/** * Report a key down/up event. Repeated "virtual" keystrokes are simulated by repeating the down event without * an interveneing up event. */
	public delegate void Calltype_zfb_keyEvent(int id, bool down, int windowsKeyCode);
	public static Calltype_zfb_keyEvent zfb_keyEvent;


	/** * Report a typed character. This typically interleaves with calls to zfb_keyEvent */
	public delegate void Calltype_zfb_characterEvent(int id, int character, int windowsKeyCode);
	public static Calltype_zfb_characterEvent zfb_characterEvent;


	/** Register a function to call when console.log etc. is called in the browser. */
	public delegate void Calltype_zfb_registerConsoleCallback(int id, ConsoleFunc callback);
	public static Calltype_zfb_registerConsoleCallback zfb_registerConsoleCallback;


	public delegate void Calltype_zfb_evalJS(int id, string script, string scriptURL);
	public static Calltype_zfb_evalJS zfb_evalJS;

call BrowserNative Interface , The only mark you need is browserId, This is in EmbeddedBrowser.Browser Is a protected variable

    /** Handle to the native browser. */
    [NonSerialized]
    internal protected int browserId;

We add a public attribute to it

    /** Handle to the native browser. */
    [NonSerialized]
    internal protected int browserId;
    public int BrowserId {
     get {
     return browserId; } } //  Increase public attribute exposure browserId

after , We can call it directly BrowserNative Interface simulates mouse input

  public void UpdateMouseEvent(EventType eventType, ViewRect rect)
  {
    
    float x = rect.x / rect.width;
    float y = rect.y / rect.height;
    Vector2 pos = MapPointerToBrowser(x, y, viewRect);
    if (eventType == EventType.MouseDown)
    {
    
      ZenFulcrum.EmbeddedBrowser.BrowserNative.zfb_mouseMove(browser.BrowserId, pos.x, pos.y); //  Be careful ：x,y Coordinate for 0-1 Range 
      ZenFulcrum.EmbeddedBrowser.BrowserNative.zfb_mouseButton(browser.BrowserId, ZenFulcrum.EmbeddedBrowser.BrowserNative.MouseButton.MBT_LEFT, true, 1);
    }
    else if (eventType == EventType.MouseUp)
    {
    
      ZenFulcrum.EmbeddedBrowser.BrowserNative.zfb_mouseButton(browser.BrowserId, ZenFulcrum.EmbeddedBrowser.BrowserNative.MouseButton.MBT_LEFT, false, 1);
    }
    else if (eventType == EventType.MouseMove)
    {
    
      ZenFulcrum.EmbeddedBrowser.BrowserNative.zfb_mouseMove(browser.BrowserId, pos.x, pos.y); //  Be careful ：x,y Coordinate for 0-1 Range 
    }
  }