OpenCVKinect 2.0 - Acquiring Kinect depth stream in OpenCV

It has been almost two years since I first wrote the code for OpenCVKinect. It has been really good to know that it has been used by a number of other students/developers at GitHub for collecting and analysing Kinect depth streams in OpenCV. I have had some feedback about a possible bug and some students have asked how they can visualize the depth maps in a better way. So today, after a long time, I am releasing the first official update to this project.

Reading a Kinect Depth Image in OpenCV

While reading a captured Kinect Depth Image, it is important to read it in a way that retains the original data. Using a regular cv::imread function call can significantly modify the data stored in a Kinect Depth Image. This is because of the fact that a regular cv::imread function call uses default method, which assumes:

1. The input image is color (three channels: RGB)
2. The depth (number of bits per pixel) of input image is UCHAR (CV_8U) or 8bits/pixel

This is, obviously, not true for a Kinect Depth Image, for this image is a special type of grayscale image. The Kinect Depth Image contains only one channel (like any other grayscale image), however the depth of this image is actually UINT16 or unsigned int (CV_16UC) instead of UCHAR. This difference in depth is because of the fact that a Kinect Depth Image contains more values and hence it requires more bits per pixel to store this information (i.e. 16bits/pixel). Now that we know what makes it different, lets see how it can be read inside OpenCV code.

To read a Depth Image use cv::imread function with CV_LOAD_IMAGE_UNCHANGED flag. This will not change the data, reading it in its original state.
e.g. cv::imread("DepthInput.png", CV_LOAD_IMAGE_UNCHANGED);

I will be updating this post later to include details on how to capture and store a data stream from Kinect using OpenCV.

OpenCVKinect: Acquiring Kinect Data Streams in OpenCV

Click here to go to code download step directly
_______________________________________________________________________________________________

Edit (26/05/2016) : I have updated the OpenCVKinect to fix some bugs and make use of different visualization for depth maps. Details of this newer version can be seen here. All other details mentioned in this blog post still apply to the updated version.
_______________________________________________________________________________________________

Holiday season is here, and I have finally found sometime to write something for this blog. Sometime back I wrote a guide for compiling OpenCV with OpenNI support. Since that post a lot has been changed in the newer OpenNI 2.x SDK, there has been a lot of improvements for using Microsoft Kinect Sensor. One of the major change is the Object Oriented Implementation as opposed to pure C implementation in previous OpenNI 1.x versions. I have had many requests asking me to write a tutorial for compiling OpenCV with the new OpenNI 2.x support. Although this is not possible with the current SDK and OpenCV, this blog post introduces an implementation for acquiring Kinect Data Streams in OpenCV.

After a long time, I recently started working with Kinect Sensor again, and therefore I wanted to access the data streams in a more efficient manner. While it looked straightforward to use the built in function calls in OpenNI 2.x, I was more comfortable with using OpenCV format instead. Therefore I wanted to hide all that detail of OpenNI Objects and their methods in a simple and convenient way. To achieve this, I wrote my own implementation and this blog post officially releases this code as open source, for anyone to use/modify/update to. Right now I have some basic methods, but in future I am thinking of adding simple image processing/data recording methods too (to make it easier for beginners).

How to train your dragon?

toothless

No! this post is not about the dragon from the animated movie (although it's one of my favourite). However this picture explains almost everything there is about the topic. For those of you who have seen this movie, computer vision machines can be thought to be like the dragon which can not fly. You have to train it about every single incident and how it should react to each one of them in order to fly.

Yup! that's right, this post is about training. Training a computer!

Just a tip of the Iceberg..

I have been working for a while on my dissertation. The main aim of this project is to build a gesture recognition system which is both reliable and robust; to the extent that it can replace the already existing human computer interaction (HCI) techniques. Despite the idea being clear, I have been asked a lot of questions about the project.

Right now I am still in the final stages to finish the approach I have planned to use. However to answer some of these questions, I have uploaded a video which shows an implementation of similar gesture recognition technique I did in the start of this project.

Here is a short demo video:

This was not very reliable or robust technique. However I learned a lot of vital information while implementing this. The idea behind this is still at the base of a very complex dynamic hand gesture recognition system I am currently working on.

Will be uploading more information soon..

Compiling OpenCV with OpenNI for Microsoft Kinect Sensor

I have implemented code for using OpenCV 2.4.x with latest OpenNI 2.x SDK. Detailed tutorial and code on this newer method can be found here

After installing OpenNI, as I am more familiar with OpenCV, therefore I wanted to work with the data from Kinect sensor using OpenCV. I searched online and a lot of people had been able to compile and run OpenCV with OpenNI. However I could not find any guide on how to do this. I managed to compile OpenCV successfully with OpenNI after some research, and now I am writing this tutorial for those of you who are in the same situation as me.

If you already know how to compile OpenCV with Visual C++ then this tutorial will be straight forward for you. However for completeness sake I will write everything in detail, so even if you are new to OpenCV you wont have any problems compiling from the source code.

If you want to use OpenCV with Microsoft Kinect using OpenNI SDK then, obviously, the first pre requisite is that you should have already installed and set up Kinect sensor using OpenNI and NITE libaries. If you have not already done that then please head over to OpenNI installation Tutorial and first install everything needed to set up. If you have successfully set up everything correctly then precompiled samples inside OpenNI and NITE folders should be working with your Kinect Sensor., which means you are good to go with this tutorial.

Installation of OpenNI with NITE middleware for Microsoft Kinect

Update (31/03/2013): Since writing this blog post, there have been a number of updates to the OpenNI SDK. Now the framework is totally changed, and the links to the OpenNI and NITE binaries used in this tutorial have been moved to another location. You will need to download the specific version of binaries from the new links, rest of the installation process is still the same.

Okay, so this post is again a guide for installation of another library. This time it is OpenNI library for Microsoft Kinect. I am currently working on a computer vision project which utilizes the power of Microsoft Kinect's depth sensor. While researching for my project, I found out that OpenNI library is better than Kinect SDK in many ways. It is opensource, cross-platform, has skeletal as well as hand tracking (which I needed for my project), gesture recognition and ability to use OpenCV were many reasons for choosing OpenNI over Kinect SDK.

OpenNI contains the SDK to access RGB and Depth data from depth sensors containing primesense hardware which includes Microsoft Kinect. However to use skeletal tracking, hand tracking, gesture recognition or any other natural interaction functions in an application a natural interaction library called NITE is required. To use these libraries with Microsoft Kinect, drivers for using Kinect sensor with OpenNI and NITE are required. To install these libraries and drivers is the easiest part, however to make the sensor working with these libraries is really difficult. I followed a number of guides online, however I could not find any guide which could completly resolve all the problems which come after installation and make the sample programs work. I am writing this guide to make sure that most people get the sensor working with these open source libraries.