Fun with Kinect and libfreenect

Hacking Kinect for holiday fun!

This Christmas my awesome brother gave me a Kinect, hitting the rare spot of something I really want but can't justify getting for myself. If you've been living under an internet rock, the Kinect is a 3-D camera marketed as an xbox 360 accessory that has been embraced and extended to do all sorts of really cool stuff. I'll take this as an opportunity to brush up on C, learn some openGL, and try to hack together some cool piece of interactive art before NYE.

Day 1

Installed libfreenect and drove myself a little bit crazy trying to get the python wrapper (and consequently numpy) to work on OSX, with its non-standard python binaries. Figuring a C-refresher would be worth it, I compiled the C-examples and modified glcplview to:

calibrate the RGB camera feed with the depth feed by changing the x and y offsets
trim the foreground / background image by filtering for high and low distances

Easy stuff, but I'm fairly new to 3-d graphics -- hopefully I can do more interesting things tomorrow.

Compile line:
gcc -lfreenect -lfreenect_sync -framework GLUT -framework OpenGL -o glpclview glpclview.c

Gotcha:
The header files are installed in /usr/local/include/libfreenect, but are sought in /usr/local/include/. Copy them to /usr/local/include or use -I/usr/local/include/libfreenect for maximum win

diff --git a/glpclview.c b/glpclview.c index 50d1b6a..a2ca941 100644 --- a/glpclview.c +++ b/glpclview.c @@ -48,6 +48,8 @@ int mx=-1,my=-1; // Prevous mouse coordinates int rotangles[2] = {0}; // Panning angles float zoom = 1; // zoom factor int color = 1; // Use the RGB texture or just draw it as color +float x_offset=0, y_offset=0, z_offset=0; +int wall_const_front = 0, wall_const_back=0; // Do the projection from u,v,depth to X,Y,Z directly in an opengl matrix // These numbers come from a combination of the ros kinect_node wiki, and @@ -81,6 +83,7 @@ void LoadRGBMatrix() -6.98445586e+00, 3.31139785e+00, 0.00000000e+00, 1.09167360e-02 }; glMultMatrixf(mat); + glTranslatef(x_offset,y_offset,z_offset); } void mouseMoved(int x, int y) @@ -116,6 +119,7 @@ void DrawGLScene() short *depth = 0; char *rgb = 0; uint32_t ts; + if (freenect_sync_get_depth((void**)&depth, &ts, 0, FREENECT_DEPTH_11BIT) < 0) no_kinect_quit(); if (freenect_sync_get_video((void**)&rgb, &ts, 0, FREENECT_VIDEO_RGB) < 0) @@ -126,6 +130,8 @@ void DrawGLScene() int i,j; for (i = 0; i < 480; i++) { for (j = 0; j < 640; j++) { + if( depth[i*640+j] < 700 + wall_const_front ) depth[i*640+j] = 0; + if( depth[i*640+j] > 2048 - 980 - wall_const_back ) depth[i*640+j] = 0; xyz[i][j][0] = j; xyz[i][j][1] = i; xyz[i][j][2] = depth[i*640+j]; @@ -185,6 +191,28 @@ void keyPressed(unsigned char key, int x, int y) zoom /= 1.1f; if (key == 'c') color = !color; + if (key == 'x') + x_offset -= 0.01f; + if (key == 'X') + x_offset += 0.01f; + if (key == 'y') + y_offset -= 0.01f; + if (key == 'Y') + y_offset += 0.01f; + if (key == '[') + wall_const_back += 5; + if (key == ']') + wall_const_back -= 5; + if (key == '{') + wall_const_front += 5; + if (key == '}') + wall_const_front -= 5; + + printf("wall_const_back=%d wall_const_front=%d x_offset=%f y_offset=%f\n",\ + wall_const_back,\ + wall_const_front,\ + x_offset,\ + y_offset); } void ReSizeGLScene(int Width, int Height)

Day 2

I ported over the coloration method of glview.c and swapped the color map with the regular RGB camera input. This highlighted the need for calibration of the unit, something Nicolas Burrus has been working on. Here's a quick-and-dirty video of what I've been able to do thus far with the provided drivers. I'd love to probe further along with some fun visualizations (and will, in due time), but I have started and semi-abandoned several projects already. For now, I want to wrap up some loose ends with YouVJ before getting too deep into kinect, especially since real-work (the stuff that pays) is getting intense as well.

Day 3

Found the super cool Open Point Library and attempted to install it on my OSX box. Sadly, I was greeted with the error:

philippp@philippp-macbookpro ~/dev/PCL-trunk/build $ sudo ln -s /opt/X11/lib/libXt.dylib /opt/local/lib/libXt.dylib philippp@philippp-macbookpro ~/dev/PCL-trunk/build $ sudo ln -s /opt/X11/lib/libSM.dylib /opt/local/lib/libSM.dylib philippp@philippp-macbookpro ~/dev/PCL-trunk/build $ sudo ln -s /opt/X11/lib/libICE.dylib /opt/local/lib/libICE.dylib philippp@philippp-macbookpro ~/dev/PCL-trunk/build $ sudo ln -s /opt/X11/lib/libX11.dylib /opt/local/lib/libX11.dylib philippp@philippp-macbookpro ~/dev/PCL-trunk/build $ sudo ln -s /opt/X11/lib/libXext.dylib /opt/local/lib/libXext.dylib philippp@philippp-macbookpro ~/dev/PCL-trunk/build $ sudo ln -s /opt/X11/lib/libGL.dylib /opt/local/lib/libGL.dylib philippp@philippp-macbookpro ~/dev/PCL-trunk/build $ make