Now that we’ve introduced the Leap Motion controller, it’s time to do something interesting with it. [If you want to cut to the chase, scroll down to the bottom of the post for a video of the results.]
The team at Leap Motion has provided a very decent set of language options via the controller’s SDK, including Java, JavaScript, Python, C++ and C#. In order to prototype a quick (and perhaps just a little dirty) integration with AutoCAD, I’ve gone ahead and used C#. I’m currently using v0.7.1 of the SDK, having started with v0.6.6 and migrated my code to the much-more-stable-on-my-particular-system v0.7.0.
I’m using Leap Motion with AutoCAD 2013 inside a Windows 7 VM within Parallels Desktop 8 on my MacBook Pro running OS X Mountain Lion. In general the controller is very responsive, but I’ve found that enabling “low resource” mode to reduce the power/USB bandwidth makes it more reliable, otherwise the USB device disconnects and reconnects frequently. I know that this configuration is probably not optimal for testing out the device’s capabilities, but on the other hand I’ve found the controller to still be very capable and responsive when used inside this virtualized environment.
The code I’ve put together works pretty well for this particular system configuration, but do be aware it hasn’t been generalised to work on different machines: when I did try it on my native Windows 8 box, for instance, I found the throughput higher and the code needed tweaking.
There are also quirks when working with different sizes of model. If using the code to navigation a larger model, for instance, the view changes have a tendency to get backed up: my code should probably check the timestamps of frames received from the Leap Motion controller and discard those that are no longer relevant. But that’s all for a real-world implementation, which this is very much not.
A quick word on the gestures themselves: in the spirit of KISS, I decided to stick to palm-level gestures (i.e. I didn’t jump through hoops to detect individual fingers’ relative positions). As long as the device detects more than a couple of fingers, the code will attempt to interpret gestures: I’ve left the case for a single or couple of fingers for a future post in this series. :-)
Here are the gestures to navigate a model in 2D or 3D:
- To pan, the user should hold their palm flat above the device and move it laterally (left, right, forwards, backwards or a combination of those directions).
- To zoom in or out, the user should raise their palm upwards or downwards.
- To orbit left or right, the user should tilt their palm left or right (via a simple wrist rotation).
- To orbit up or down, the user should tilt their palm forwards or backwards (by raising or lowering their wrist).
I’ve avoided free orbit as that will very quickly get you into trouble. Constraining orbit to a left-right “turntable” rotation with the option to orbit up/down also makes usage much more straightforward, in my opinion.
Here’s the C# code that implements our simple navigation gestures inside AutoCAD:
using System;
using System.Threading;
using Autodesk.AutoCAD.ApplicationServices;
using Autodesk.AutoCAD.ApplicationServices.Core;
using Autodesk.AutoCAD.DatabaseServices;
using Autodesk.AutoCAD.EditorInput;
using Autodesk.AutoCAD.Geometry;
using Autodesk.AutoCAD.Runtime;
using Leap;
namespace LeapMotionIntegration
{
public class Camera
{
// Members
private Document _doc = null;
private ViewTableRecord _vtr = null;
private ViewTableRecord _initial = null;
public Camera(Document doc)
{
_doc = doc;
_initial = doc.Editor.GetCurrentView();
_vtr = (ViewTableRecord)_initial.Clone();
}
// Reset to the initial view
public void Reset()
{
_doc.Editor.SetCurrentView(_initial);
if (_vtr != null)
{
_vtr.Dispose();
_vtr = (ViewTableRecord)_initial.Clone();
}
_doc.Editor.Regen();
}
// Zoom in or out
public void Zoom(double factor)
{
// Adjust the ViewTableRecord
_vtr.Height *= factor;
_vtr.Width *= factor;
// Set it as the current view
_doc.Editor.SetCurrentView(_vtr);
// Zoom requires a regen for the gizmos to update
_doc.Editor.Regen();
}
// Pan in the specified direction
public void Pan(double leftRight, double upDown)
{
// Adjust the ViewTableRecord
_vtr.CenterPoint =
_vtr.CenterPoint +
new Vector2d(
leftRight * _vtr.Width,
upDown * _vtr.Height
);
// Set it as the current view
_doc.Editor.SetCurrentView(_vtr);
}
// Orbit by angle around axis
public void Orbit(Vector3d axis, double angle)
{
// Adjust the ViewTableRecord
_vtr.ViewDirection =
_vtr.ViewDirection.TransformBy(
Matrix3d.Rotation(angle, axis, Point3d.Origin)
);
// Set it as the current view
_doc.Editor.SetCurrentView(_vtr);
}
public void OrbitVertical(double angle)
{
// Adjust the ViewTableRecord
_vtr.ViewDirection =
_vtr.ViewDirection.TransformBy(
Matrix3d.Rotation(
angle,
_vtr.ViewDirection.GetPerpendicularVector(),
Point3d.Origin
)
);
// Set it as the current view
_doc.Editor.SetCurrentView(_vtr);
}
}
public class NavigationListener : Listener
{
private Editor _ed;
private Camera _cam;
private SynchronizationContext _ctxt;
public NavigationListener(
Editor ed, Camera cam, SynchronizationContext ctxt
)
{
_ed = ed;
_cam = cam;
_ctxt = ctxt;
}
private void WriteLine(String line)
{
if (_ed != null)
{
_ctxt.Post(a => { _ed.WriteMessage(line + "\n"); }, null);
}
}
private void CallOnCam(SendOrPostCallback cb)
{
if (_cam == null)
WriteLine("Cam is null.");
else
_ctxt.Post(cb, null);
}
private void Reset()
{
CallOnCam(a => { _cam.Reset(); } );
}
private void Zoom(double factor)
{
CallOnCam(a => { _cam.Zoom(factor); });
}
private void Pan(double leftRight, double upDown)
{
CallOnCam(a => { _cam.Pan(leftRight, upDown); });
}
private void Orbit(Vector3d axis, double angle)
{
CallOnCam(a => { _cam.Orbit(axis, angle); });
}
private void OrbitVertical(double angle)
{
CallOnCam(a => { _cam.OrbitVertical(angle); });
}
public override void OnInit(Controller controller)
{
//WriteLine("\nInitialized");
}
public override void OnConnect(Controller controller)
{
//WriteLine("Connected");
}
public override void OnDisconnect(Controller controller)
{
WriteLine("Disconnected");
}
public override void OnFrame(Controller controller)
{
// Get the most recent frame
var frame = controller.Frame();
var hands = frame.Hands;
var numHands = hands.Count;
// Only proceed if we have at least one hand
if (numHands >= 1)
{
// Get the first hand and its velocity to check for
// zoom or pan
var hand = hands[0];
var handVel = hand.PalmVelocity;
if (handVel == null)
handVel = new Vector(0, 0, 0);
// Check if the hand has any fingers
var fingers = hand.Fingers;
// Only proceed if we see at least two fingers detected
if (fingers.Count > 2)
{
// Set a flag to see whether we detect zoom or pan
bool zoomOrPan = false;
// Create an AutoCAD vector from the hand's velocity
var handVec =
new Vector3d(handVel.x, handVel.y, handVel.z);
// Get the largest element and its [absolute] value
var largestDim = handVec.LargestElement;
var largestVal = handVec[largestDim];
var largestAbs = Math.Abs(largestVal);
// Depending on the largest value we know to zoom or pan
switch (largestDim)
{
case 1:
if (largestAbs > 250)
{
if (frame.Id % 2 == 0)
{
WriteLine(
"Zoom " + (largestVal < 0 ? "in" : "out")
);
Zoom(largestVal > 0 ? 1.1 : 0.9);
}
zoomOrPan = true;
}
break;
default:
if (largestAbs > 100)
{
var x = 0.02 * -handVel.x / largestAbs;
var y = 0.02 * handVel.z / largestAbs;
WriteLine(
"Pan " +
(largestDim == 0 ?
(largestVal < 0 ? "left" : "right") :
(largestVal < 0 ? "up" : "down")
)
);
Pan(x, y);
zoomOrPan = true;
}
break;
}
// If not zoom or pan, we check for orbit
if (!zoomOrPan && largestAbs < 100)
{
// To determine whether we have to orbit, get the normal
// to the hand's palm
var handNorm = hand.PalmNormal;
if (Math.Abs(handNorm.Roll) > 0.4)
{
// Orbit left or right when there is "roll"
WriteLine(
"Orbit " + (handNorm.Roll < 0 ? "right" : "left")
);
var zAxis =
_ed.CurrentUserCoordinateSystem.CoordinateSystem3d.
Zaxis;
Orbit(zAxis, 0.5 * handNorm.Roll * Math.PI / 12);
}
else if (handNorm.Pitch < -1.5 || handNorm.Pitch > -1.0)
{
// Orbit up or down when there is "pitch"
var pitch =
handNorm.Pitch < -1.5 ?
handNorm.Pitch + 1.5 :
Math.Abs(handNorm.Pitch + 1.0);
WriteLine(
"Orbit " + (pitch < 0 ? "up" : "down")
);
OrbitVertical(0.5 * pitch * Math.PI / 12);
}
}
// Process Windows messages at the end of each frame
System.Windows.Forms.Application.DoEvents();
}
}
}
}
public class NavigationCommands
{
[CommandMethod("LEAP")]
public void LeapMotionNavigation()
{
var doc =
Application.DocumentManager.MdiActiveDocument;
var db = doc.Database;
var ed = doc.Editor;
// Create a camera for our current document to adjust the view
var cam = new Camera(doc);
// Creating a blank form makes sure the SyncContext is
// set properly for this thread
using (var f1 = new Form1())
{
var ctxt = SynchronizationContext.Current;
try
{
if (ctxt == null)
{
ed.WriteMessage(
"\nCurrent sync context is null."
);
return;
}
// Create our navigation listener to receive events
var listener = new NavigationListener(ed, cam, ctxt);
using (listener)
{
if (listener == null)
{
ed.WriteMessage("\nCould not create listener.");
return;
}
// Use the listener to create the Leap Motion
// controller
using (var controller = new Controller(listener))
{
if (controller == null)
{
ed.WriteMessage("\nCould not create controller.");
return;
}
// Loop until cancelled
bool done = false;
do
{
var pr =
ed.GetString(
"\nPress enter to reset view or esc to quit"
);
if (pr.Status == PromptStatus.OK)
{
cam.Reset();
}
else if (pr.Status == PromptStatus.Cancel)
{
done = true;
}
}
while (!done);
}
}
}
catch (System.Exception ex)
{
ed.WriteMessage("\nException: {0}", ex.Message);
}
}
}
}
}
A few quick comments on the implementation…
As the Leap Motion “listener” actually executes on a separate thread, we need to do some work to make sure anything we do inside AutoCAD executes on the UI thread. We use a SynchronizationContext for this – we use it to post a particular callback to execute on the UI thread using our CallOnCam() helper – but to make sure the SynchronizationContext is set properly on the main thread we need to create a dummy Form (the implementation of which is not included here – the basic Wizard-generated WinForm is quite adequate) inside our code.
Otherwise we enter the navigation mode via a command (LEAP), rather than just being able to use it passively (i.e. outside a command). We will see a more passive implementation, later in the week, but one that relies on Windows Messages rather than direct AutoCAD API calls.
Here’s a quick video of this in action. It’s actually a longer video with some geometry creation at the beginning, so if you want a preview of what’s coming in the next post, feel free to rewind. ;-)
