Question regarding frame latency / comparison to official SDK

[julienaubert]

Does anyone have any information on how libfreenect2 compares to the official SDK? Is the performance, e.g. frame latency for RGB/depth similar or do they differ? Is there room for improvement in libfreenect2? and if so where is the bottleneck?

[floe]

I think the performance is quite similar when you use the OpenGL or OpenCL packet processor. CPU fallback is significantly worse (OTOH, the official SDK doesn't have a CPU fallback at all). About the latency, I unfortunately have no real idea, I don't think anybody has yet performed any measurements.

[bellekci]

I have also noticed that much more CPU resources are used. Considerıng that depth calculation and filtering is done using GPU, why is the CPU fallback that worse? Maybe, we can take a look into it.

[kohrt]

Much more CPU resources used, compared to what? The official SDK, or the between GPU and CPU-fallback?
The GPU is designed to run the same code on different data in parallel on many cores, that why it is so fast. The CPU implementation is single threaded but it could be parallelized to use all cores, that would speed things up, but although increases CPU load a lot, while still not being able to get 30 Hz.
Maybe in combination with AVX instructions to parallelize the single operations even more could increase the processed frames per second even more. But most PCs have a graphic card which is much better in doing this kind of stuff, so I am not sure if it is worth the effort.
Regarding the latency, the depth processing takes ~2 ms with OpenCL and ~4 ms with OpenGL on a Nvidia 970. The JPEG decoding takes much longer. Maybe this could be improved by using some GPU based video acceleration like libva.

[bellekci]

I meant compared to the official SDK. CPU with libfreenect2 seems to be more busy compared to the offical SDK. I did not do a very fair comparison as I was comparing libfreenect2 in linux with official sdk in Windows but this is what I have experienced in identical machines.
Did @floe not mean that? Unlike official SDK, there is a significant CPU fallback in libfreenect2. At least, this is what I have understood.

[kohrt]

The high CPU load comes probably from the JPEG decoding.
No, he meant that libfreenect2 has a CPU fallback option to compute everything on the CPU instead of using the GPU. The official SDK won't work without a GPU.

[xlz]

@wiedemeyer I created a test program with libva. On i7-4770R/Iris Pro 5200, libva JPEG decoding takes 5.6ms/173Hz, and TurboJPEG 14.1ms/70Hz.

Will create patch for libfreenect2 soon.

[xlz]

Re OpenGL: my previous testing on a Windows machine with CUDA showed OpenGL performance would degrade to below 60Hz after a while, probably because of CPU load by JPEG decoding. CUDA performance remained stable.

[kohrt]

@xlz That sounds great. Is the CPU load reduced a lot, when using libva? If I remember correctly libva should also work with GPUs from other vendors, right?

[xlz]

@wiedemeyer Yes, CPU load is half of that with TurboJPEG.

I'm not sure of AMD's stuff. Nvidia's VAPAU and AMD's UVC compete with Intel's VA-API. If there is support for VA-API on non-Intel GPUs, the support would still be limited.

But at least VDPAU is not supported on Jetson TK1. I plan to create a JPEG decoder with CUDA NPP, which will be compatible with CUDA GPUs. Not sure about AMD GPUs.

[boulabiar]

Is it possible to deactivate jpeg decoding at all ?
Depending on the context, some one would need only 3D point cloud, or maybe
IR but not the RGB input.

How to select only 3D point clound and deactivate all other processing ?

Thanks !

On Fri, Apr 17, 2015 at 10:43 AM, Thiemo Wiedemeyer <
notifications@github.com> wrote:

@xlz https://github.com/xlz That sounds great. Is the CPU load reduced
a lot, when using libva? If I remember correctly libva should also works
with GPUs from other vendors, right?

—
Reply to this email directly or view it on GitHub
#192 (comment)
.

[bellekci]

I do not need RGB either. I have only synced depth data to the listener and I thought jpeg decoding would be skipped but I haven't checked if it was the case.
If we only sync depth to the listener and skip color listener, would there be still a reason to have CPU load?

[kohrt]

On their project page (http://www.freedesktop.org/wiki/Software/vaapi/) they mentioned drivers from different vendors that implement VA-API.

[xlz]

Seems like the VDPAU backend for VA-API package is called vdpau-va-driver in Ubuntu/Debian. I don't have any Nvidia machines to test the backend abstraction though.

Ideally one can simply write another VDPAU JPEG decoder without the abstraction layer. It should not be hard. Although on Windows and Mac it looks like much harder because the APIs are quite unclear.

[zardchim]

@xlz

Any news for libva :) ?