After a completely ridiculous wait of close to 4 months, I finally received my Kinect a few weeks ago. Apart from it being the fastest selling consumer electronics device in history, the delay was also due to the fact I was holding out for the very popular Xbox 360 250Gb Slim bundle (and also because the vendor I chose fumbled the order during the final few weeks, which just added insult to injury). I’d done my homework before receiving the Xbox, and realised that the bundled Kinect would not come with the external power supply needed to connect it to older Xbox systems and – more importantly – to a standard PC. So I went ahead and procured one of those – from ricardo.ch, the Swiss equivalent of ebay.com – so I’d be ready to go when it arrived.
I’ve now spent a few hours playing around with the Kinect – both with the Xbox and my PC – to get a feel for its capabilities. I haven’t been disappointed: the device is way cool, and is going to change fundamentally the way we think about HCI.
I won’t get into how badly I play Kinect Adventures or Sports, but I did want to discuss my initial attempts at integrating the Kinect with AutoCAD.
Overall there are two things I want to try with the Kinect:
- Some simple model capture using the Kinect’s RGB-D camera
- Perhaps using transient graphics or a jig to generate a dynamic preview of the input
- Finally generate a point cloud with full colour information
- Some simple gesture-based control of AutoCAD
The first seemed a fun approach to test out the Kinect interface, although ultimately the currently-available precision (the resolution has apparently been limited to 640 x 480 to keep the processing snappy) would probably not prove usable for any real-world model capture.
For the second I can see a few ways to utilise the Kinect, whether to create models or navigate through them. There are lots of gnarly issues to address if doing this properly, so I only see myself attempting to tackle a very small (and hopefully clearly defined) subset of the overall problem.
In terms of the actual implementation, it seems there are a few avenues open (and I may be missing some, as I – like most people – am new to all this):
- Use the Code Laboratories NUI Platform Driver/SDK
- NUI stands for Natural User Interface
- Use OpenKinect / libfreenect
- Use Boris Scheiman’s nKinect abstraction layer
- This currently works on top of option 1, but will apparently be ported to option 2, in due course
- Wait for the official SDK from Microsoft Research
As I want to integrate with AutoCAD, my first choice is to use a stable Windows driver & SDK combination. At the time of writing, this currently means option 1 (or 3), although I’m hoping there’ll be some kind of release announced for option 4 at MIX11 over the next few days.
After I briefly discussed nKinect in this previous post, Boris Scheiman has kindly been in touch to discuss the features I’d need from nKinect. Thanks to Boris, the latest version of the toolkit includes point cloud generation and I’ll certainly be taking a look at that, before long.
My first attempts have focused on getting option 1 up and running. After installing the driver and SDK, it was pretty straightforward to get the CLNUIDeviceTest sample application working with the Kinect attached to my PC.
What has proven a lot less obvious is to get the same code working inside AutoCAD: after making the initial call to CLNUIDevice.GetDeviceCount(), which tells you how many Kinect devices you have attached, the call to CLNUIDevice.GetDeviceSerial(0) (which should return the first – and only, in my case – Kinect attached) causes AutoCAD to crash uncontrollably. This method should return a string, so my initial reaction – after thinking through the fact it couldn’t be a DLL dependency issue, as the first API call succeeded – was to look at the string marshalling used for the data. I fooled around with that for a while, but didn’t manage to find a solution (which leads me to believe the problem is elsewhere).
In the end I went back to working on a standalone EXE which at least gets the data from the Kinect in a format that’s importable into AutoCAD. I created a very simple WPF application, with the main form containing a single button called “CaptureButton” and this C# code behind it:
using System.Runtime.InteropServices;
using System.Windows.Media;
using System.Windows;
using System.IO;
using System;
namespace KinectIntegration
{
public partial class MainWindow : Window
{
// We're dealing with 640 x 480 input
const int width = 640;
const int height = 480;
public MainWindow()
{
InitializeComponent();
}
private static int GetDepth(int d)
{
// Extract the bits we need from a depth value
return d & 0x7FF;
}
private static void GetDepths(IntPtr source, int[] dest)
{
if (source != IntPtr.Zero)
{
for (int i = 0; i < dest.Length; i++)
{
dest[i] = GetDepth(Marshal.ReadInt16(source, i * 2));
}
}
}
private static Color GetColor(int c)
{
// Extract the ARGB values from a 32-bit integer
byte b = (byte)(c & 0xFF);
byte g = (byte)((c >> 8) & 0xFF);
byte r = (byte)((c >> 16) & 0xFF);
byte a = (byte)((c >> 24) & 0xFF);
// Return the corresponding color
return Color.FromArgb(a, r, g, b);
}
private static void GetColors(IntPtr source, Color[] dest)
{
if (source != IntPtr.Zero)
{
for (int i = 0; i < dest.Length; i++)
{
dest[i] = GetColor(Marshal.ReadInt32(source, i * 4));
}
}
}
private void ExportPointCloud(int[] depths, Color[] cols)
{
// Hardcode the creation of a text file to C:\
using (StreamWriter sw = new StreamWriter("c:\\pc.txt"))
{
// For each pixel, write a line to the text file:
// X, Y, Z, R, G, B
for (int i = 0; i < height; i++)
{
for (int j = 0; j < width; j++)
{
Color col = cols[(i * width) + j];
sw.WriteLine(
"{0}, {1}, {2}, {3}, {4}, {5}",
j, height - i, -depths[(i * width) + j],
col.R, col.G, col.B
);
}
}
}
}
private void CaptureButton_Click(
object sender, RoutedEventArgs e
)
{
try
{
// Check how many devices we have
int devNum = CLNUIDevice.GetDeviceCount();
if (devNum < 1)
{
// We need at least one connected
return;
}
// For now work with the first device found
string devSerial = CLNUIDevice.GetDeviceSerial(0);
// Start the motor and the camera
IntPtr motor = CLNUIDevice.CreateMotor(devSerial);
IntPtr camera = CLNUIDevice.CreateCamera(devSerial);
// Reset the motor position
CLNUIDevice.SetMotorPosition(motor, 0);
// Create our color and depth images
NUIImage colorImage = new NUIImage(width, height);
NUIImage depthImage = new NUIImage(width, height);
// If the camera started...
if (CLNUIDevice.StartCamera(camera))
{
// ... capture color and depth images
CLNUIDevice.GetCameraColorFrameRGB32(
camera, colorImage.ImageData, 500
);
CLNUIDevice.GetCameraDepthFrameRAW(
camera, depthImage.ImageData, 500
);
// Extract the depth information
int[] depths = new int[width * height];
GetDepths(depthImage.ImageData, depths);
// And the color information
Color[] cols = new Color[width * height];
GetColors(colorImage.ImageData, cols);
// Create a point cloud file for processing
// by txt2las.exe
ExportPointCloud(depths, cols);
// Stop the camera
CLNUIDevice.StopCamera(camera);
}
// And reset/destroy everything
CLNUIDevice.SetMotorLED(motor, 0);
CLNUIDevice.DestroyMotor(motor);
CLNUIDevice.DestroyCamera(camera);
// Closes the main dialog
this.Close();
}
catch { }
}
}
}
Once the “capture” is complete, we have a point cloud file in “C:\pc.txt” with a line of X, Y, Z, R, G, B values for each point.
It’s now a simple matter to use our old friend txt2las.exe (which we’ve used extensively when bringing point cloud data into AutoCAD) to create a LAS, using the string “txt2las –parse xyzRGB –i pc.txt –o pc.las”:
And from there POINTCLOUDINDEX / POINTCLOUDATTACH will bring in the point cloud into AutoCAD:
Rotating the view in 3D allows us to see that is at least has some rudimentary depth information (aside from the IR-based depth camera losing accuracy at distance, we passed the depth in verbatim rather than normalising it in some way with the 640 x 480 indeces we used for the X & Y coordinates in the point cloud):
The data is coming in reasonably well, at this stage, so we have a foundation upon which to build a more dynamic solution. I’ll probably spend a little time, this week, seeing if it’s possible to get the same thing working directly inside AutoCAD (while crossing my fingers for the release of the official Windows drivers).
Update
I fixed a minor glitch in the code (negating the Z coordinate to make it depth instead of height) and retook the screenshots (this time at night rather during the day, as that’s when I’m updating the post :-).
