OpenGLDepthPacketProcessor not working on Intel GPUs (Mac/OS X)

[dominiklessel]

Hello there!

I successfully compiled libfreenect2 on my MacBook Pro Retina (2013; QuadCore i7; Intel Iris GPU), but the IR/Depth Steam is just black. My first thought on this is a incompatible GPU … Does anybody have the same problem?

Some debugging code I added in SyncMultiFrameListener::onNewFrame(Frame::Type type, Frame *frame) (frame_listener_impl.cpp:69) outputted the following information:

if ( type == 4 ) {
  printf( "Got depth frame!\n" );
  printf( "width: %i\n", int(frame->width) );
  printf( "height: %i\n", int(frame->height) );
  printf( "bytes per pixel: %i\n", int(frame->bytes_per_pixel) );
  double sum;
  for (int i = 0; i < frame->width*frame->height*frame->bytes_per_pixel; ++i)
    sum += frame->data[i];
  printf( "sum: %f\n", sum );
}

Got depth frame!
width: 512
height: 424
bytes per pixel: 4
sum: 0

After skimming through the code I ended up looking at the different DepthPacketProcessors and swapped OpenGLDepthPacketProcessor with CpuDepthPacketProcessor (libfreenect.cpp:76) and commented out depth_packet_processor_(0) (libfreenect.cpp:277) to make Protonect compile again.

Now Protonect is outputting IR and Depth information, but turns my MacBook into a pocket heater:

[floe]

So far the OpenGL depth processor has only been tested on Nvidia GPUs, it's probably a rather trivial issue that prevents Intel GPUs from working.

[dominiklessel]

Okay, thanks!

That's something I can't fix on my own ... But I can test everything if someone provides a fix.

[RyanGordon]

So I've started to look into this (Trying to get opengl processor working on my mac too). Even with the debug shader on, I get a blank screen. I used the XCode OpenGL Profiler tools to give me any errors that might be happening and got this:

Edit: I did some looking around and it seems like this is pretty standard error in GLEW and isn't correlated to our problem/doesn't affect rendering

glGetString(GL_EXTENSIONS); returns: 0x00000000
Error: GL_INVALID_ENUM
Context: 0x7fa6ab870a00
Virtual Screen: 0/2
kCGLCPCurrentRendererID: 16925952 (0x01024500)
GL_RENDERER: Intel HD Graphics 5000 OpenGL Engine
GL_VENDOR: Intel Inc.
GL_VERSION: 4.1 INTEL-8.28.30
kCGLCPGPUFragmentProcessing: GL_TRUE
kCGLCPGPUVertexProcessing: GL_TRUE
Function call stack:
0: 0x1054463cb in glewContextInit in <libGLEWmx.1.10.0.dylib>
1: 0x1044ee19a in libfreenect2::OpenGLContext::OpenGLContext(GLFWwindow_) at opengl.cpp: 47
2: 0x1044f99a6 in libfreenect2::OpenGLDepthPacketProcessor::OpenGLDepthPacketProcessor(void_) at opengl_depth_packet_processor.cpp: 712
3: 0x1045019ed in libfreenect2::Freenect2DeviceImpl::Freenect2DeviceImpl(libfreenect2::Freenect2Impl_, libusb_device_, libusb_device_handle*, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) at libfreenect2.cpp: 347
4: 0x104503d97 in libfreenect2::Freenect2::openDevice(int) at libfreenect2.cpp: 669
5: 0x1044de29b in main at Protonect.cpp: 61
6: 0x7fff8b95c5fd in start in <libdyld.dylib>
7: 0x00000001 in

Not exactly sure if this is an issue, it seems like they just may not be initialized properly?

[billinghamj]

Same problem here

[christiankerl]

can you check whether you can get a simple glfw/opengl example working with this GPU and the setup logic in https://github.com/OpenKinect/libfreenect2/blob/master/examples/protonect/src/opengl_depth_packet_processor.cpp#L665 (all the glfw calls there and the libfreenect2::OpenGLContext class)

thanks!

[RyanGordon]

I've only been able to debug a little bit on my side (hopefully I have more time soon) but what I've been able to do is edit examples/protonect/src/shader/debug.fs with:

-  Color = texelFetch(Data, uv);
+  Color = vec4(0.0, 0.0, 1.0, 1.0);

and then I get two of the windows to show up blue (with debug on) which I think means that all of the setup logic is fine. So it definitely seems like something is going on within one of the fragment shaders itself

[christiankerl]

nice.

ok, can you try to write the blue color to the Debug variable in stage1.fs and see if it shows up as well. afterwards try to write the contents of the ZTable texture to Debug, i.e.,

Debug = vec4(vec3(texelFetch(ZTable, uv).x), 1);

This should show some grey and black stripes. If this doesn't work there seems to be a problem with rectangle textures. Adding

#extension GL_ARB_texture_rectangle: enable

after #version 330 might help.

[RyanGordon]

I set the Debug in stage1 to be blue and that worked, and then I set it to Debug = vec4(vec3(texelFetch(ZTable, uv).x), 1.0); in stage1.fs and it showed up only as white.

I added "#extension GL_ARB_texture_rectangle: enable" right after "#version 330" to every shader file and restarted the protonect and it still shows up as white

[RyanGordon]

An interesting observation I had is that if I change "#extension GL_ARB_texture_rectangle: enable" to anything other than "enable" my blue texture and the white that shows up using your suggested debug statement stops working and goes black. If I leave out "#extension GL_ARB_texture_rectangle: enable" all together the blue and white texture work which seems to indicate that in fact that extension is enabled by default already

[christiankerl]

ok, interesting. so the textures might not be the problem. the all white color might come from the fact that the values in ZTable are large. Try dividing by 5000 or 10000. Otherwise you can also write some custom data to the ZTable texture in OpenGLDepthPacketProcessor::loadZTableFromFile().

[christiankerl]

maybe the downloading is a problem. can you write some custom data to the Infrared output in stage1.fs and see if it ends up in the IR frame?

[RyanGordon]

Yeah, I thought downloading might have been a problem. If i comment out the infrared line in stage1.fs and replace it with "Infrared = 100.0;" then I see a gray screen in the Debug window (like this: https://cloudup.com/ch2FEp1BCiZ) but still a black screen in the IR frame.

Separately, if I set set Debug to: "Debug = vec4(vec3(texelFetch(ZTable, uv).x / 10000), 1.0);" Then I also get a gray screen on the Debug window with a circularish pattern (like this: https://cloudup.com/iuO3D-1Vxfi)

[christiankerl]

the ztable looks good, when describing how the output should look like I mixed it up with the p0table.
maybe you can also try to display the 3 different p0tables in stage1.fs

hm, in Protonect.cpp you can check the values of the IR frame like this:

cv::Mat ir_(ir->height, ir->width, CV_32FC1, ir->data);

double min, max;
cv::minMaxIdx(ir_, &min, &max);
std::cout << "ir min: " << min << " max: " << max << std::endl;