Wednesday, 4 July 2012

Compiling OpenCV with OpenNI for Microsoft Kinect Sensor

I have implemented code for using OpenCV 2.4.x with latest OpenNI 2.x SDK. Detailed tutorial and code on this newer method can be found here

After installing OpenNI, as I am more familiar with OpenCV, therefore I wanted to work with the data from Kinect sensor using OpenCV. I searched online and a lot of people had been able to compile and run OpenCV with OpenNI. However I could not find any guide on how to do this. I managed to compile OpenCV successfully with OpenNI after some research, and now I am writing this tutorial for those of you who are in the same situation as me.

If you already know how to compile OpenCV with Visual C++ then this tutorial will be straight forward for you. However for completeness sake I will write everything in detail, so even if you are new to OpenCV you wont have any problems compiling from the source code.

If you want to use OpenCV with Microsoft Kinect using OpenNI SDK then, obviously, the first pre requisite is that you should have already installed and set up Kinect sensor using OpenNI and NITE libaries. If you have not already done that then please head over to OpenNI installation Tutorial and first install everything needed to set up. If you have successfully set up everything correctly then precompiled samples inside OpenNI and NITE folders should be working with your Kinect Sensor., which means you are good to go with this tutorial.


Building Binaries
To compile OpenCV-2.4.1 with OpenNI support, first download OpenCV-2.4.1 source code and extract it to a folder. For simplicity I use the folder C:\OpenCVFiles\ for all OpenCV installations.

Next step is to use CMAKE to prepare code to be compiled into binaries and libraries. Install latest version of CMAKE with default settings. Once installed open CMAKE from desktop shortcut.
Now browse to the source folder and also enter the build folder as below. I use \install folder for the build but you can use any other location, just make sure that you remember where you placed your build files as these will be needed later to set up your project.


Click configure

Select Visual Studio 10 with default native compilers

Click configure

Check option WITH_OPENNI and click configure again

Make sure that the folders to OpenNI and Kinect drivers are correct and click configure again. If everything is fine then there will be no entries marked in red. To make sure that the build is configured properly with OpenNI, see the OPENNI field in the output window, it should say YES and should be configured with SensorKinect drivers.


Now finally click Generate and go to install folder to find the file named OpenCV.sln. Open this project file and in the solution explorer towards the left hand side, select ALL_BUILD and build solution. This will take a while.


Once the solution is build successfully for ALL_BUILD option, from solution explorer select INSTALL and again build solution. This will compile the binaries inside the folder
C:\OpenCVFiles\OpenCV-2.4.1\Install\Install


That's it for building the binaries. Next step is to configure your pc to use these binaries in a project.

Setting up pc to use the compiled binaries
To change the settings of your pc to use these binaries with your projects, you have to add the include, lib and bin folders inside the folder
C:\OpenCVFiles\OpenCV-2.4.1\Install\Install
to the environment variables. To do this go to Computer->Properties->Advanced System Settings->Environment Variables and under System Variables select variable named Path and click edit



Now if you have installed to the same location as I have done then add the following lines at the end of this variable, making sure that each entry is separated by a ;

If you have compiled in a different location then changes the following lines accordingly
C:\OpenCVFiles\OpenCV-2.4.1\Install\Install\bin;C:\OpenCVFiles\OpenCV-2.4.1\Install\Install\lib;C:\OpenCVFiles\OpenCV-2.4.1\Install\Install\include;


Click ok and apply the settings.

Setting up your first project
To set up a new project open Visual Studio and create a new win32 console project. Go to properties of this project and follow the following steps

Go to Linker->input->additional dependencies and add the following lines

C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_calib3d241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_contrib241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_core241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_features2d241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_flann241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_gpu241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_highgui241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_imgproc241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_legacy241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_ml241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_nonfree241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_objdetect241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_photo241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_stitching241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_ts241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_video241d.lib
C:\OpenCVFiles\OpenCV-2.4.1\Install\lib\opencv_videostab241d.lib
Go to VC++ Directories and add following lines to Include Directories:
C:\OpenCVFiles\OpenCV-2.4.1\Install\Install\include\opencv2
C:\OpenCVFiles\OpenCV-2.4.1\Install\Install\include\opencv
C:\OpenCVFiles\OpenCV-2.4.1\Install\Install\include
and the following lines to Library Directories:
C:\OpenCVFiles\OpenCV-2.4.1\Install\install\lib
This guide uses the debug libraries, but if you need libraries for a release version then just erase the d at the end of each lib file.

To run a sample code, download the openni_capture.cpp file navigate to samples\cpp\openni_capture.cpp inside opencv folder and add it to your project. Build and Compile the code and if everything is working fine then you will be able to capture and display data from Kinect sensor directly using OpenCV.

Thats all! Please leave a comment if this tutorial helped you, or if you might want to add something to it, or if you are having problem at any of the above steps.

References:
http://docs.opencv.org/doc/user_guide/ug_highgui.html
http://seevisionc.blogspot.co.uk/2012/06/installation-of-openni-with-nite.html

38 comments:

  1. Great job man, you've saved me. I've been looking for this info for a month. Just a little detail, your opennni_capture.cpp file link is broken. That's a shame becuse I now have to look over for examples to start programming >_< jejeje. Anyway, Thanks!

    ReplyDelete
    Replies
    1. you are welcome. The online link is not working, however you can find this example inside opencv folder in samples\cpp\openni_capture.cpp

      Delete
  2. great job mate, but when i tried to run i 'd problem "Can not open a capture object."
    I think that there is a problem when the program tries to grab data from the kinect sensor... any idea!
    this happens with my own program, but with openni_capture project sample all things are aright,
    thanks in advance..

    ReplyDelete
    Replies
    1. I might be wrong but, have you added the openni libraries to your project? That might be the problem, but I'm not sure.

      Delete
    2. If you have compiled OpenCV successfully with OpenNI support then you dont need to add OpenNI libraries to your project.

      And if the example is working fine, then there might be logical mistake in your code. What I did at this stage was to try to understand the example and then write a simple code to display the captured depth data.

      Delete
  3. Hi Mr. Muhammad Asad,
    Thanks for the turoial, i'm successfully to do that.
    Those run slowly, but if i switch off the disparity map it will run fast (but still slow if compare with pure openni), but it's ok for me.

    I wanna ask
    first, is opencv kinect could be do skeleton tracking?,any clue for the code?
    and secondly, how can i accessing the depth map?

    ReplyDelete
  4. Hi Sir, thank you for your post. Yes; you've saved me too.

    I'll ask you about openni samples. Is there another examples different from openni_capture.cpp. I could not find in internet. Could you give links.

    Thank you again Sir.

    ReplyDelete
    Replies
    1. inside OpenNI and NITE installation folders you can find a samples folder which contains some sample code

      Delete
  5. Hi Mr. Muhammad Asad,
    Thanks for the turoial, its really helpfuel.
    I just have some questions, I added all the dlls from the OPenNI and OPenCV folders to the debug file which has the.exe application of my project. I added the openni_capture.cpp into my project. But when I run no output appears, so I don't know what the output should look like to know if I am on the right track or not.

    Thank You

    ReplyDelete
    Replies
    1. It should look like a grayscale image showing depth.. i will try to include an example image soon so anyone following this instruction can have an idea about it

      Delete
  6. Hi, mr.asad..
    I have problem, can you help me please?!

    I was doing setting openCV and openNI with your tutorial.
    but my project have problem,
    with output :

    1>------ Build started: Project: tracking1, Configuration: Debug Win32 ------
    1>Build started 31/12/2012 16:50:47.
    1>InitializeBuildStatus:
    1> Touching "Debug\tracking1.unsuccessfulbuild".
    1>ClCompile:
    1> All outputs are up-to-date.
    1> tracking1.cpp
    1>c:\opencvfiles\opencv-2.4.1\install\install\include\opencv2\flann\logger.h(66): warning C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
    1> c:\program files\microsoft visual studio 10.0\vc\include\stdio.h(234) : see declaration of 'fopen'
    1>c:\opencvfiles\opencv-2.4.1\install\install\include\opencv2\legacy\blobtrack.hpp(187): warning C4996: 'strcpy': This function or variable may be unsafe. Consider using strcpy_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
    1> c:\program files\microsoft visual studio 10.0\vc\include\string.h(105) : see declaration of 'strcpy'
    1>c:\opencvfiles\opencv-2.4.1\install\install\include\opencv2\legacy\blobtrack.hpp(256): warning C4996: 'strcat': This function or variable may be unsafe. Consider using strcat_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.
    1> c:\program files\microsoft visual studio 10.0\vc\include\string.h(110) : see declaration of 'strcat'
    1>c:\users\putra\documents\visual studio 2010\projects\tracking1\tracking1\tracking1.cpp(37): warning C4996: 'xn::Context::InitFromXmlFile': Use other overload!
    1> c:\program files\openni\include\xncppwrapper.h(5576) : see declaration of 'xn::Context::InitFromXmlFile'
    1>c:\users\putra\documents\visual studio 2010\projects\tracking1\tracking1\tracking1.cpp(90): warning C4996: 'kbhit': The POSIX name for this item is deprecated. Instead, use the ISO C++ conformant name: _kbhit. See online help for details.
    1> c:\program files\microsoft visual studio 10.0\vc\include\conio.h(130) : see declaration of 'kbhit'
    1>c:\users\putra\documents\visual studio 2010\projects\tracking1\tracking1\tracking1.cpp(121): warning C4244: '=' : conversion from 'double' to 'int', possible loss of data
    1>c:\users\putra\documents\visual studio 2010\projects\tracking1\tracking1\tracking1.cpp(122): warning C4244: '=' : conversion from 'double' to 'int', possible loss of data
    1>c:\users\putra\documents\visual studio 2010\projects\tracking1\tracking1\tracking1.cpp(140): warning C4996: 'xn::Context::Shutdown': You may use Release() instead, or count on dtor
    1> c:\program files\openni\include\xncppwrapper.h(5706) : see declaration of 'xn::Context::Shutdown'
    1>c:\users\putra\documents\visual studio 2010\projects\tracking1\tracking1\tracking1.cpp(53): warning C4101: 'hUserCallbacks' : unreferenced local variable
    1>ManifestResourceCompile:
    1> All outputs are up-to-date.
    1>LINK : fatal error LNK1104: cannot open file 'C:\Program Files\OpenNI\Lib.obj'
    1>
    1>Build FAILED.
    1>
    1>Time Elapsed 00:00:06.16
    ========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========


    what do you think about that ?!
    thank you.

    ReplyDelete
    Replies
    1. please include and make sure correct path to OpenNI lib folder is included.

      Delete
  7. Hello,

    I have done configuration exactly like you mentioned.
    open cv and open ni with prime sense... I have generated files using cmake and then compiled it in visual studio.

    but when I run the openni_capture.cpp,

    I always get "Can not open a capture object" ... please let me know if any one has this problem and solution for it... thanks.

    ReplyDelete
    Replies
    1. There is a problem with your kinect drivers. Please make sure you uninstall Kinect SDK and its drivers before following this guide. Also make sure to install latest working primesense drivers for kinect sensor. Inside both OpenNI and Nite folder is a samples bin folder. Try executing the .exe binaries to check if you have installed everything correctly.

      Delete
    2. I followed this tutorial and had the same problem. I could run all of the nite and openni examples but got a "Can not open a capture object" when i tried to compile the c++ code.

      I was using OpenCV 2.4.3 and everything was installed in C:\OpenCV243

      I followed everything with cmake in the tutorial so after I generated it my install solution was C:\OpenCV243\install

      I then deleted all the folders in C:\OpenCV243 except for the install folder and I was able to connect to my Kinect

      Delete
    3. The problem is not for Kinect SDK. I also faced same problem. For some other purpose I have used openCV path in my System varibles. So, I removed that entry from System variables and kept those 3 entry for ../install/lib;../install/bin;../install/include. Then it was working. But it is working slow compared to pure OpenNI.

      Thanks.

      Delete
  8. Hi Muhammad,

    Thanks for your good tutorial. Have you tried this on Windows 64bit? openNI2? Should I uninstall Microsoft Kinect SDK beforehand? I set the parameters but CMake can't find them anyways! I'm trying to run "linemod" sample in openCV which works with OpenNI.

    Thank you,
    Abbas

    ReplyDelete
  9. Hi,

    Thanks for your tutorial! It was incredibly helpful. I'm trying to use the release version of OpenCV instead of the debug version, but it seems like only the debug libraries got built and are in the /lib/ folder. I tried building the OpenCV.sln as Release, but that failed. Any ideas?

    Thanks,
    Andrew

    ReplyDelete
    Replies
    1. Look for a CMAKE_BUILD_TYPE flag and set its value to RELEASE
      Unfortunately I cannot find this flag in my CMAKE (it used to be there sometime back)
      Rest of the process should be same,

      After you have compiled everything you will be able to see lib and bin files without 'd' in the end. 'd' means debug
      For example: instead of 'opencv_calib3d243d.lib' you will see files with name 'opencv_calib3d243.lib'

      Delete
  10. Hello Sir,
    I am working on ubuntu 12.04 lts. I am having a problem with kinect_maps.cpp sample thats available on the net. When I try to compile it, it shows the error "Can not open a capture object." . I find this strange cause I followed the instructions from "http://docs.opencv.org/doc/user_guide/ug_highgui.html" and yet it does not work.

    ReplyDelete
    Replies
    1. Unfortunately there is not a lot which I can help you with the information in your comment. I have not compiled these libraries on Linux therefore I have relatively less idea about that.

      However it looks like the libraries are either not compiled properly or they are not linked properly.
      Has anything worked on these libraries since you compiled?

      The problem could also be in the drivers (try google search on these)

      Delete
  11. Hello Muhammad Asad,

    Thanks for your tutorial. Due to some reason now I want to switch to OpenNI 2.2. And I also want to use OpenCV 2.4.6 and Microsoft SDK 1.7. I have tried your tutorial to compile OpenNI 2.2 with OpenCV but it is not working. As OpenNI 2.x doesn't require SensorKinect, Cmake is unable to find OpenNI.

    Please help.

    ReplyDelete
    Replies
    1. I got it. It actually doesn't need to be compiled together like previous versions of OpenNI 1.x. So,

      i.Configure MVS 10 for OpenNI 2.2 and then
      ii. Configure MVS 10 for OpenCV 2.4.6

      Thanks.

      Delete
    2. I also want to use OpenNI 2.2 and OpenCV 2.4.6 and Microsoft SDK 1.7. What do you mean by your two steps. How to configure MVS 10 for OpenNI 2.2? For configuring MVS 10 for OpenCV 2.4.6, just as usual by using CMake with OpenNI? Thanks!

      Delete
    3. I think by configuring he means adding paths for include and lib for each sdk.

      Delete
  12. Hello,
    I try to configure with Cmake (my environnement : Windows64bits, OpenCv 2.4.5 Openni2.0) but Cmake don"t find the variable of Openni :

    WARNING, OpenNI library directory (set by OPENNI_LIB_DIR variable) is not found or does not have OpenNI libraries.
    WARNING, OpenNI include directory (set by OPENNI_INCLUDE_DIR variable) is not found or does not have OpenNI include files.
    WARNING, PrimeSensor Module binaries directory (set by OPENNI_PRIME_SENSOR_MODULE_BIN_DIR variable) is not found or does not have PrimeSensor Module binaries.

    So the flags : OPenni in Cmake is : No and PrimeSensor is : No ...

    Can you tell where is the probleme ?
    My variable environement are :
    1) OPENNI_INCLUDE_DIR(C:\Program Files\OpenNI2\Include\)
    2) OPENNI_LIB_DIR(C:\Program Files\OpenNI2\Lib\)

    I don't understand the origin of the problem, can you help me ,?
    Best regards

    ReplyDelete
    Replies
    1. hi there,
      You dont need to compile openni2 with opencv... this guide is for old version of openni.
      Please check the comment above which highlights the process for openni2.

      I will try to write a new tutorial on using OpenNI2, KINECT SDK and OPENCV simultaneously in a project.

      Delete
    2. Okay Thanks you for your reply, currently I use the Openni's function to get the data and copy it on Opencv's buffers to do processing and display it, So i use the two librairies separately.

      Delete
    3. Hey Muhammad,
      Dit you write tuto for OpenNi 2.0 seeting with opencv ?

      Delete
    4. Hi Amine,

      Many thanks for your comment. No I have not done that yet, been busy with work. I will try to do it over this weekend.

      Asad

      Delete
    5. From what I gathered from the web, there is no direct support of OpenNI 2.2 in OpenCV 2.4.6. for now (but I could be simply wrong). The best solution I found so far is to use a wrapper around OpenNI to convert data streams to your needs. For me it means converting Kinect images to Mat format. One simple wrapper with example can be found here. I just eliminated Point Cloud components from that.

      Delete
    6. I have implemented something similar for OpenNI 2.2. This can be found here

      Delete
  13. "This guide uses the debug libraries, but if you need libraries for a release version then just erase the d at the end of each lib file."

    im new here, and i don't understand what "release version" mean

    ReplyDelete
    Replies
    1. first link on this explains the difference:

      http://lmgtfy.com/?q=difference+between+release+and+debug

      Delete
  14. Hi! where i can get sample code of openni_capture.cpp? the link doesn't work! And do you have any link to tutorial or documentation of work by openCV with openNI? appreciate!

    ReplyDelete
    Replies
    1. https://github.com/Itseez/opencv/blob/master/samples/cpp/openni_capture.cpp

      Delete
    2. first link on this gives you tutorials:

      http://lmgtfy.com/?q=opencv+openni

      Delete