OpenNI2 driver

[hanyazou]

I've roughly ported the OpenNI2FreenectDriver for libfreenect2. While this port is little bit messy, it works with PCL's pcl_openni2_viewer and OpenNI2's NiViewer on my Mac OSX.
Please look at https://github.com/hanyazou/libfreenect2/tree/master/OpenNI2-Freenect2Driver. I'm glad you give me some ideas of how I merge this work with the project.

[floe]

Thanks for your work! Just for my understanding, does that mean you can now use the NiTE skeleton tracking module with the Kinect v2?

[hanyazou]

I'd like to try on it next weekend. But the primesense web site was closed and I could not find NiTE middleware. Do you have correct download link?

[JasCodes]

try here
https://github.com/PrimeSense

[gaborpapp]

openni2: https://github.com/occipital/openni2
nite2 archive: https://bitbucket.org/kaorun55/openni-2.2/src/2f54272802bfd24ca32f03327fbabaf85ac4a5c4/NITE%202.2%20%CE%B1/?at=master

[hanyazou]

I've tried the sample executable in the NiTE binary, NiTE-MacOSX-x64-2.2.tar.zip. I did some debugging and added hacking code to my driver because the NiTE required some compatibilities with old Kinect drivers to work correctly. The default depth resolution must be 640x480 and device name must be "Kinect" exactly. ("Kinect 2" does not work for the NiTE.)

Now, that looks working.

[HenningJ]

Nice work. I modified your code a little to make it compile on Windows 7 with Visual Studio 2008 (ripped out some c++11 stuff and replaced sys/time.h with boost/chrono), now it works great for me :-)

[HenningJ]

After working with this some more, I noticed a couple of problems:

1. Big memory leak. It only ran for a minute or so, before it crashed with a bad_allocation exception. I'm using 32bit instead of 64bit, so my memory is full rather quickly, and you might not have noticed that at all.
   I guess you have to delete the frame pointer in the libfreenect2::FrameListener::onNewFrame() method. At least that's what protonect does, once there is a new frame.
2. When closing the OpenNI device, you invalidate an iterator before using it in the following line.

[HenningJ]

Also, this seems to add a lot of overhead to the libfreenect2 processing. With protonect, I get around 30 fps with the OpenGL depth processor, with this I barely get 10 fps. Maybe too much work is being done in the onNewFrame() callback? Maybe you can just store the pointer to the new frame and do the actual creation of a new OpenNI frame in a different thread?

[hanyazou]

@HenningJ I've fixed onNewFrame() in 4e8eb54894. Please look at this and let me know if you still have the problem. I will investigate the closing problem and the overhead next weekend. Thank you so much for your testing and comments.

[HenningJ]

You're right, no more memory leak. 👍

It would also be nice to turn off the registration (for speed), and a way to get the original resolution (HD) RGB image.

[skarard]

Hello,

I have attempted to follow what you are doing.

OpenNi2 SimpleViewer complained that the depth and color streams were different sizes. So I was able to change the DepthDriver.cpp to resolve that issue.

You mentioned that NiTE2 looks for "Kinect" and a depth resolution of 640x480.

I then change DepthDriver.cpp resolution settings back to 640x480 I received Illegal instruction (core dumped) from NiTE2 UserViewer. If the depth resolution is not 640x480 NiTE2 UserViewer seg faults instead.

[HenningJ]

@hanyazou I played around some more and did the following changes:

• Instead of directly processing the frames in the onNewFrame() callback, I'm using the SyncMultiFrameListener like in protonect
• the OpenNI frames are created in a separate thread
• the libreenect2::Registration is applied to the whole image instead of each pixel individually

Before these changes, I had framerates like the following:

[OpenCLDepthPacketProcessor] avg. time: 10.8811ms -> ~91.9023Hz
[TurboJpegRgbPacketProcessor] avg. time: 153.137ms -> ~6.53011Hz
 26017347 VERBOSE    [FPS] Depth: 29.36 Color: 7.39                  <- inside OpenNI

Now I got this:

[OpenCLDepthPacketProcessor] avg. time: 10.7307ms -> ~93.1904Hz
[TurboJpegRgbPacketProcessor] avg. time: 22.0074ms -> ~45.4393Hz
 22014990 VERBOSE    [FPS] Depth: 29.32 Color: 29.57

Much nicer. Now OpenNI isn't slowing down libfreenect2 anymore.
I can do a pull request to your repository tomorrow, if you like.

[hanyazou]

@HenningJ I like! Please send your improvement to me.

I must merge from upstream too. And I have some bugs. It might take a little time to do them.