Menu GUI options for Leap Motion and Unity3D

I have been playing around with the Leap Motion and Unity3D mainly, and received some positive encouragement from early tutorials I published in this area around getting the Leap to work with Unity standard as well as the Leap enablement of the BootCamp tutorial and the Leap enablement of the Car tutorial.
For the past few months, I have been working on my first commercial application (Paper Scissors Rock) specifically targeted at the Leap Motion controller.
As part of that effort, I have experimented with various types of interaction with Unity. From a gameplay POV, it all depends on your game, and the key thing (for me anyway) is to use the uniqueness of the Leap, either as far as using the 3rd dimension (z axis) or the gestures support.
From a “Menu” support, we enter the realm of 2D, where the interaction (often) is more akin to a touch screen.
Imagine a simple 2D game / screen and you want to push some buttons on the screen. Experiments included:
Experiment 1: Initially, I used the concept of the Z Axis to “touch” button. So you move you hand / finger around and change the mouse cursor as a result, and then to “click”, you move your finger/hand close to the screen.
Pros: very easy to implement. Simply monitor the z axis and if it crosses a “threshold”, you implement the “click” behaviour.
Cons: Usability is not great. I tested this and while I could use it, every other person that tried to use it was confused (because of the lack of tactile feedback, it is tough to know when you are “clicking”.
Mitigation: I initially implemented a “cursor glow” which basically overlays a glow on the cursor when it is in “clicking” range, and also had to implement a “lock-out” function to ensure only 1 click get registered.. But frankly, the usability was just not there.
Experiment 2: Gestures. You can use gestures (either the built in ones or your own) to signify a click.
Pros: Again, easy to implement, you just watch for a gesture and implement the “click” functionality
Cons: 1st, I had problem with the “gesture” impacting the cursor position (as you hand / finger) tends to change ion a gesture. I settled (as an example) for shooting wit played fingers etc. (see the Bootcamp tutorial) which made that quite usable. Same as above, you would need to implement a lock-out (lucky for shooting, it had its own timeout between bullets)
Experiment 3: “Hover Clicks”. This is where you hover over a button, receive feedback (usually by a clockwise animation) and then “click”. This has been discussed very extensively at the Leap Motion forums and theLIFT even provided some free UI elements to help (see here)
Pros: Very usable, I found that the testers loved this interaction
Cons: Can be hard to retrofit existing code / games.
With that last point in mind, I ended up creating a very simple hover click concept, without having to retrofit all my UI elements. To do that, I basically did the following.
I had an existing game with menus that were implemented in Unity3D via using a Raycast.

feedbackhover

Retrofitting existing code in Unity 3D

To “retrofit” hover buttons, I had to essentially add 3 key things:

  1. First I had to add a test raycast to check whether I am on top of something “click-able”
  2. Then I implemented a generic function which keeps track of how long I am hovering on top of a clickable and sets the “hoverClick” variable accordingly
  3. Lastly I implemented a visual feedback in “onGUI.”

Step 1

I modified the Update function of my main script to include the test raycast to check whether I am on top of a “clickable”
Check out the final results if you are interested: http://games4leap.com/rock-paper-scissors/
And some sample code below

// do a test ray to see if we are on top of a clickable
Ray testRay = Camera.mainCamera.ScreenPointToRay(CurrMousePos);
if (Physics.Raycast(testRay, out hit))
{
if (hit.collider.gameObject.GetComponent())
	{
		// The below is specific to my game.
// But basically need to get a unique string for what you are hovering on top of
		buttonHover = hit.collider.gameObject.GetComponent();
		string clickablebutton = buttonHover.id + buttonHover.who + buttonHover.idLevel.ToString();
		// MouseOverClickable keeps track of whether we are on top of the same button
		// as well as starting the timer to see if a hoverclick is initiated.
		MouseOverClickable(clickablebutton);
	}
	else
	{
		// we are not on top off a button, so still call MouseOverClickable, but with an ever changing ID
		MouseOverClickable("");
		clickableElapsedTime = 0F;
	}
}
else
{
	MouseOverClickable("");
	clickableElapsedTime = 0F;
}

 

This is the function that basically keeps track of whether we are hovering on top of

We basically keep track of how long we are hovering the same button and if we are hovering greater than 1 second, then we set a variable (hoverClick to true),.  In the update function of the code, we check for hoverClick and implement that functionality

public  void MouseOverClickable(string ClickableID)
{
	if (ClickableID == "")
	{
		lastClickableID = ClickableID;
		hoverClick = false;
		clickableElapsedTime = 0F;
	}
	else if (ClickableID == lastClickableID)
	{
		clickableElapsedTime = clickableElapsedTime + Time.deltaTime;
		if (clickableElapsedTime > 1)
		{
			// elapsed time is greater than 1 second, so trigger a hoverClick
			hoverClick = true;
			clickableElapsedTime = 0F;
		}
	}
	else
	{
		lastClickableID = ClickableID;
		hoverClick = false;
		clickableElapsedTime = 0F;
	}
}

The code in OnGUI that actually draws the timer functionality

void OnGUI () 
{
	// so we only need to show the "timer glow" if we have the conditions of:
	if (clickableElapsedTime > 0) 
	{
		// Work out which if 12 bitmaps to use (showing progression clockwise)
		int glowTimerIndex = Mathf.CeilToInt(11 * clickableElapsedTime);
		if (glowTimerIndex > 11)
		{
			glowTimerIndex = 11;
		}
		Texture2D cursorGlowImg = CursorGlowArray[glowTimerIndex];
		graphicRect = new Rect(x - (cursorGlowImg.width/2), y - (cursorGlowImg.height/2) , cursorGlowImg.width, cursorGlowImg.height);;
		GUI.DrawTexture(graphicRect, cursorGlowImg);
	}
}

6 thoughts on “Menu GUI options for Leap Motion and Unity3D

  1. Pingback: Hover-Click in Leap Motion

  2. Your tutorials have been amazing! Are you going to share your source for this like you have been for the previous posts? I would like to see how you are handling the primary/secondary interactions.

    • Hi, Evan, thanks for your feedback.
      The solution I created was to “retro-fit” some existing code with “hover-Click” fucntionality. Unfortunately, the code I retrofitted is a full commercial game and I am not intending on publishing that. So, the obvious answer is that I need to do another specific tutorial on retorfitting an open source game out there with Hover-Click…
      Any suggestions on which game to use? if not, my plan would be take something simple like thd C# Game Eamples on the Asset store and retrofit those to work with the Leap… Timeline may be a few weeks though, as I need to concentrate on updating my Leap Motion Controller Airspace submission for Rock Paper Scissors before public Beta 🙂

  3. Hmm, how about something simple. Like just click a button. Basically it would be incredibly useful to track the primary and secondary pointable coordinates for mouse like interaction.

    I can get these coordinates out of the sandbox demo, but I really like the slimmed down method youve got going on in your demos. Plus im kind of new to all this and unraveling that mess is pretty challenging.

  4. Hey Pierre,

    You’ve got some really great stuff showing off the Leap Motion! I was hoping that you could show me how to turn the Leap Motion’s movement into the mouse movement. I’m trying to do something similar to what you have in “Experiment 1” where the user will be able to use the Leap Motion to interact with a 2D GUI and use the ScreenTap gesture to click on an object that they’re hovering over.

    Any suggestions?

Leave a Reply

Your email address will not be published. Required fields are marked *