In image processing, one of the most successful object detectors devised is the Viola and Jones detector, proposed in their seminal CVPR paper in 2001. A popular implementation used by image processing researchers and implementers is provided by the OpenCV library. In this post, I’ll show you how run the OpenCV object detector in MATLAB for Windows. You should have some familiarity with OpenCV and with the Viola and Jones detector to work through this tutorial.

### Steps in the object detector

MATLAB is able to call functions in shared libraries. This means that, using the compiled OpenCV DLLs, we are able to directly call various OpenCV functions from within MATLAB. The flow of our MATLAB program, including the required OpenCV external function calls (based on this example), will go something like this:

1. `cvLoadHaarClassifierCascade:` Load object detector cascade
2. `cvCreateMemStorage:` Allocate memory for detector
3. `cvLoadImage:` Load image from disk
4. `cvHaarDetectObjects:` Perform object detection
5. For each detected object:
1. `cvGetSeqElem:` Get next detected object of type `cvRect`
2. Display this detection result in MATLAB
6. `cvReleaseImage:` Unload the image from memory
7. `cvReleaseMemStorage:` De-allocate memory for detector
8. `cvReleaseHaarClassifierCascade:` Unload the cascade from memory

The first step is to load the OpenCV shared libraries using MATLAB’s `loadlibrary()` function. To use the functions listed in the object detector steps above, we need to load the OpenCV libraries `cxcore100.dll`, `cv100.dll` and `highgui100.dll`. Assuming that OpenCV has been installed to `"C:\Program Files\OpenCV"`, the libraries are loaded like this:

```opencvPath = 'C:\Program Files\OpenCV';
includePath = fullfile(opencvPath, 'cxcore\include');

fullfile(opencvPath, 'bin\cxcore100.dll'), ...
fullfile(opencvPath, 'cxcore\include\cxcore.h'), ...
'alias', 'cxcore100', 'includepath', includePath);
fullfile(opencvPath, 'bin\cv100.dll'), ...
fullfile(opencvPath, 'cv\include\cv.h'), ...
'alias', 'cv100', 'includepath', includePath);
fullfile(opencvPath, 'bin\highgui100.dll'), ...
fullfile(opencvPath, 'otherlibs\highgui\highgui.h'), ...
'alias', 'highgui100', 'includepath', includePath);```

You will get some warnings; these can be ignored for our purposes. You can display the list of functions that a particular shared library exports with the `libfunctions()` command in MATLAB For example, to list the functions exported by the `highgui` library:

```>> libfunctions('highgui100')

Functions in library highgui100:

cvConvertImage             cvQueryFrame
cvCreateCameraCapture      cvReleaseCapture
cvCreateFileCapture        cvReleaseVideoWriter
cvCreateTrackbar           cvResizeWindow
cvCreateVideoWriter        cvRetrieveFrame
cvDestroyAllWindows        cvSaveImage
cvDestroyWindow            cvSetCaptureProperty
cvGetCaptureProperty       cvSetMouseCallback
cvGetTrackbarPos           cvSetPostprocessFuncWin32
cvGetWindowHandle          cvSetPreprocessFuncWin32
cvGetWindowName            cvSetTrackbarPos
cvGrabFrame                cvShowImage
cvMoveWindow
cvNamedWindow```

The first step in our object detector is to load a detector cascade. We are going to load one of the frontal face detector cascades that is provided with a normal OpenCV installation:

```classifierFilename = 'C:/Program Files/OpenCV/data/haarcascades/haarcascade_frontalface_alt.xml';
libstruct('CvSize',struct('width',int16(100),'height',int16(100))));```

The function `calllib()` returns a `libpointer` structure containing two fairly self-explanatory fields, `DataType` and `Value`. To display the return value from `cvLoadHaarClassifierCascade()`, we can run:

```>> cvCascade.Value

ans =

flags: 1.1125e+009
count: 22
orig_window_size: [1x1 struct]
real_window_size: [1x1 struct]
scale: 0
stage_classifier: [1x1 struct]

The above output shows that MATLAB has successfully loaded the cascade file and returned a pointer to an OpenCV `CvHaarClassifierCascade` object.

### Prototype M-files

We could now continue implementing all of our OpenCV function calls from the object detector steps like this, however we will run into a problem when `cvGetSeqElem` is called. To see why, try this:

`libfunctions('cxcore100', '-full')`

The `-full` option lists the signatures for each imported function. The signature for the function `cvGetSeqElem()` is listed as:

`[cstring, CvSeqPtr] cvGetSeqElem(CvSeqPtr, int32)`

This shows that the return value for the imported `cvGetSeqElem()` function will be a pointer to a character (`cstring`). This is based on the function declaration in the `cxcore.h` header file:

`CVAPI(char*)  cvGetSeqElem( const CvSeq* seq, int index );`

However, in step 5.1 of our object detector steps, we require a `CvRect` object. Normally in C++ you would simply cast the character pointer return value to a `CvRect` object, but MATLAB does not support casting of return values from `calllib()`, so there is no way we can cast this to a `CvRect`.

The solution is what is referred to as a prototype M-file. By constructing a prototype M-file, we can define our own signatures for the imported functions rather than using the declarations from the C++ header file.

Let’s generate the prototype M-file now:

```loadlibrary(...
fullfile(opencvPath, 'bin\cxcore100.dll'), ...
fullfile(opencvPath, 'cxcore\include\cxcore.h'), ...
'mfilename', 'proto_cxcore');```

This will automatically generate a prototype M-file named `proto_cxcore.m` based on the C++ header file. Open this file up and find the function signature for `cvGetSeqElem` and replace it with the following:

```% char * cvGetSeqElem ( const CvSeq * seq , int index );

fcns.name{fcnNum}='cvGetSeqElem'; fcns.calltype{fcnNum}='cdecl'; fcns.LHS{fcnNum}='CvRectPtr'; fcns.RHS{fcnNum}={'CvSeqPtr', 'int32'};fcnNum=fcnNum+1;```

This changes the return type for `cvGetSeqElem()` from a `char` pointer to a `CvRect` pointer.

We can now load the library using the new prototype:

```loadlibrary(...
fullfile(opencvPath, 'bin\cxcore100.dll'), ...
@proto_cxcore);```

### An example face detector

We now have all the pieces ready to write a complete object detector. The code listing below implements the object detector steps listed above to perform face detection on an image. Additionally, the image is displayed in MATLAB and a box is drawn around any detected faces.

```opencvPath = 'C:\Program Files\OpenCV';
includePath = fullfile(opencvPath, 'cxcore\include');
inputImage = 'lenna.jpg';

fullfile(opencvPath, 'bin\cxcore100.dll'), @proto_cxcore);
fullfile(opencvPath, 'bin\cv100.dll'), ...
fullfile(opencvPath, 'cv\include\cv.h'), ...
'alias', 'cv100', 'includepath', includePath);
fullfile(opencvPath, 'bin\highgui100.dll'), ...
fullfile(opencvPath, 'otherlibs\highgui\highgui.h'), ...
'alias', 'highgui100', 'includepath', includePath);

libstruct('CvSize',struct('width',int16(100),'height',int16(100))));

%% Create memory storage

cvStorage = calllib('cxcore100', 'cvCreateMemStorage', 0);

cvImage = calllib('highgui100', ...
if ~cvImage.Value.nSize
end

%% Perform object detection

cvSeq = calllib('cv100', ...
'cvHaarDetectObjects', cvImage, cvCascade, cvStorage, 1.1, 2, 0, ...
libstruct('CvSize',struct('width',int16(40),'height',int16(40))));

%% Loop through the detections and display bounding boxes

for n = 1:cvSeq.Value.total
cvRect = calllib('cxcore100', ...
'cvGetSeqElem', cvSeq, int16(n));
rectangle('Position', ...
[cvRect.Value.x cvRect.Value.y ...
cvRect.Value.width cvRect.Value.height], ...
'EdgeColor', 'r', 'LineWidth', 3);
end

%% Release resources

calllib('cxcore100', 'cvReleaseImage', cvImage);
calllib('cxcore100', 'cvReleaseMemStorage', cvStorage);

As an example, the following is the output after running the detector above on a greyscale version of the Lenna test image:

Note: If you get a segmentation fault attempting to run the code above, try evaluating the cells one-by-one (e.g. by pressing Ctrl-Enter) – it seems to fix the problem.

Hi,

Very good and useful article !
Thanks!

Do you think that it’s the best way to use OpenCV, IPP, … from Matlab ?

I thought about new .NET (C++.NET) Matlab interface from Matlab 2009.

@Vladimir – Thanks, I’ve never tried using the .NET interface (I’m still using MATLAB R2006a). If you were to take that approach, I suppose you would want to first create a .NET wrapper around the OpenCV libraries. Take a look at OpenCVDotNet – I don’t think the project is still active, but might give you some ideas anyway.

Hi,
I run the code in MATLAB (R2009a), however, I found the image could not be loaded. More specifically, the following line returns a null pointer:

cvImage = calllib(‘highgui100′, …

All the codes before this line look fine. Could you enlighten me on the possible cause?

@dong – cvLoadImage returns a null if the image fails to load. There can be lots of reasons – the filename may be incorrect, the image may be invalid, etc. I’d suggest you should try getting a simple C++ version of your code working first, and then port it over to MATLAB.

The idea of adding to Matlab some of the functionalities of OpenCV is awesome, but I can’t find the libraries in my OpenCV folder I should share in Matlab. I’m using OpenCV 2.2 for Windows and Matlab 7.11, and I am afraid you are using another version and maybe for Linux. Which version are you using? Should I use Windows or Linux?

From other Matlab/Flash project, I got cxcore100.dll and cv100.dll, but I don’t have the header files of these dlls, so I can’t shared the libraries in Matlab. Can I follow your guide and succeed if you send to me these files?

Thanks and keep with this outstanding work.

• Thanks – I used Windows but I used an older version of OpenCV than you (I originally wrote this code in 2008). I have just downloaded version 2.2 for Windows and all of the DLL files seem to have been reorganised and renamed. You could try downloading a pre-2.0 version of OpenCV (older versions are available on Sourceforge, I think I used 1.1pre1), or you could try to work out what the relevant libraries and functions for face detection are in the latest version. I’d be interested to hear if you get it working with version 2.2!

Why must it have an OpenCV dependency? Why not write the whole code in Matlab? Is there a catch?

• You certainly could implement it from scratch in pure MATLAB, but the OpenCV library is well written, well documented and well maintained.

Hi:

A very informative article. Thanks. Is it possible to convert cvImage to a MATLAB image to display inside MATLAB with imshow()? On the other hand, how to convert a MATLAB image to cvImage to be process by any OpenCV function through Calllib()? How to pass CV_RGB(r, g, b) (that is a macor/in-line function) as an argument in Calllib()?

Thanks,

Chien

• Take a look at the documentation for the IplImage structure. The raw image data is stored in the `imageData` variable and with a bit of fiddling it should be possible to convert this raw data into a MATLAB matrix. Be warned though – JPEGs come in a lot of different varieties (colour depth, number of channels, interlaced vs progressive, etc), so you might find that any code you write to process the raw images may break when you try it out with JPEGs from different sources. The OpenCV FAQ contains some code showing how to decode raw data which might be of help.

As to your question about C/C++ macros, these cannot be used in MATLAB unfortunately.

fullfile(opencvPath, ‘bin\cxcore100.dll’), …
fullfile(opencvPath, ‘cxcore\include\cxcore.h’), …
‘mfilename’, ‘proto_cxcore’);
This code was run, but proto_cxcore.m was not generated. this code is not error! Can you explain it? thanks!

• I’m not too sure. Have you checked that paths actually exist? Are you able to generate prototype files for any other of the OpenCV libraries?

no..i am not able to generate any prototype file.

8. i got this mistake :

??? Attempt to reference field of non-structure array.

Error in ==> Untitled at 53
rectangle(‘Position’,[cvRect.Value.x cvRect.Value.y
cvRect.Value.width cvRect.Value.height],…
can you explain ?

• Can you print out the value of cvRect from the MATLAB console?

• nope ! ( i dont know why ?

9. I liked the article
but when I tried to apply the code
I downloaded the opencv binary file and installed and built it using VS2010 as the instructions
but I can’t these three files
cxcore100.dll, cv100.dll and highgui100.dll

• Is there anything in your build directory after you build?

• my build directory already contains folders with libraries before I build using VS2010
and they are

common
gpu
include
python
x64
x86

but they don’t include those libraries

• That looks like you are using a newer version of OpenCV (since your build directory contains “python” and the Python bindings only came in version 2, AFAIK). I originally wrote this code using OpenCV version 1.1pre1. However in the latest version of OpenCV, all of the libraries seem to have been reorganised and renamed. To get this working you could either download version 1.1pre1, or try to figure out what all of the libraries and functions have been renamed to in the latest version of OpenCV.

• thank you for this notification
I will try to install the old version

sorry, Who can help me ?.I have a problem.
fullfile(opencvPath, ‘bin\cxcore100.dll’), …
fullfile(opencvPath, ‘cxcore\include\cxcore.h’), …
‘mfilename’, ‘proto_cxcore’);
Warning: The library class ‘cxcore100′ already exists. Use a classname alias.

i can not create proto_cxcore.m .==>I can not run code in Face tracking without a Kalman filter(http://blog.cordiner.net/2011/05/03/object-tracking-using-a-kalman-filter-matlab/)

• This message means that you have already loaded the library. If you want to load it again, you should unload it first using the `unloadlibrary` function.

11. I installed version 1
and found the correct libraries
but when I execute this line

fullfile(opencvPath, ‘bin\cxcore100.dll’), …
fullfile(opencvPath, ‘cxcore\include\cxcore.h’), …
‘alias’, ‘cxcore100′, ‘includepath’, includePath);

I get the following error

Warning: Warnings messages were produced while parsing. Check the functions you
intend to use for correctness. Warning text can be viewed using:
The actual error is at the end of this output.
*********

Failed to parse type ‘union Cv32suf { int i ; unsigned u ; float f ; } Cv32suf’ original input ‘union Cv32suf { int i ; unsigned u ; float f ; } Cv32suf’
Found on line 639 of input from line 164 of file C:\Program Files\OpenCV\cxcore\include\cxtypes.h

Failed to parse type ‘union Cv64suf { int64 i ; uint64 u ; double f ; } Cv64suf’ original input ‘union Cv64suf { int64 i ; uint64 u ; double f ; } Cv64suf’
Found on line 647 of input from line 172 of file C:\Program Files\OpenCV\cxcore\include\cxtypes.h

Found on line 853 of input from line 393 of file C:\Program Files\OpenCV\cxcore\include\cxtypes.h

Found on line 853 of input from line 393 of file C:\Program Files\OpenCV\cxcore\include\cxtypes.h

Eval of const expression 32 ][ 2 failed with error Unmatched right square bracket at (eval 6) line 1, at end of line
syntax error at (eval 6) line 1, near "32 ]”
Missing right curly or square bracket at (eval 6) line 2, at end of line
.
Found on line 1261 of input from line 811 of file C:\Program Files\OpenCV\cxcore\include\cxtypes.h

Found on line 1261 of input from line 811 of file C:\Program Files\OpenCV\cxcore\include\cxtypes.h

Found on line 1697 of input from line 1292 of file C:\Program Files\OpenCV\cxcore\include\cxtypes.h

Found on line 2298 of input from line 133 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2305 of input from line 140 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2305 of input from line 140 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2308 of input from line 143 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2312 of input from line 147 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2357 of input from line 192 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2357 of input from line 192 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2362 of input from line 197 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2362 of input from line 197 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2369 of input from line 204 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2369 of input from line 204 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2380 of input from line 215 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2380 of input from line 215 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2390 of input from line 227 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2390 of input from line 227 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2399 of input from line 236 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2402 of input from line 239 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2406 of input from line 243 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2406 of input from line 243 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2415 of input from line 252 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2415 of input from line 252 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2466 of input from line 303 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2477 of input from line 315 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2543 of input from line 387 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2543 of input from line 387 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2559 of input from line 412 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2559 of input from line 412 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2808 of input from line 680 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2808 of input from line 680 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2834 of input from line 707 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2834 of input from line 707 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 2839 of input from line 712 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 3699 of input from line 1619 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 3705 of input from line 1625 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 3705 of input from line 1625 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 3710 of input from line 1630 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 3710 of input from line 1630 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 3758 of input from line 1678 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 3770 of input from line 1690 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 3778 of input from line 1698 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 3782 of input from line 1702 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Found on line 3785 of input from line 1705 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h
*********
??? Error using ==> loadlibrary at 480
There was an error running the loader mfile. Use the mfilename option
to produce a file that you can debug and fix if needed. Please report
this error to the MathWorks so we can improve this function.

Caused by:
Error using ==> feval
Error: File: cxcore100_proto.m Line: 579 Column: 9
The input character is not valid in MATLAB statements or expressions.

• I tried also the version 1.1 pre 1 but I get the same error

• Your error message is pointing to: cxcore100_proto.m Line: 579 Column: 9. What’s on this line?

• I can’t find this file
as it is saved in Temp folder and when I get there it is erased

i also got the same problem here i dont know what to do

12. I have just started using opencv,completely new to this language.It seems the opencv which i installed (OpenCV-2.3.1-win-superpack) neither has a bin folder nor the i.e bin\cxcore100.dll . Moreover,there are several versions of cxcore.h which are present in C:\Program Files\OpenCV\opencv\include and C:\Program Files\OpenCV\opencv\build .

• The article was written using an older version of OpenCV, and the latest version does not seem to be backwards compatible. Read this comment for further information.

Hey alister, I am using OpenCV2.1. Can you please point out the location your ‘proto_cxcore’ is made? Everything runs fine till that point and no error is given. Get the feeling that it is created somewhere.

Hey it’s me again. To anybody having the same problem as me, I finally figured out what the problem was. You just have to make sure to unload cxcore100 library (cxcore210d in my case) before running the proto_cxcore line.

15. opencvPath = ‘C:\opencv';
includePath = ‘C:\opencv\build\include';

fullfile(‘C:\opencv\build\x86\vc9\bin’, ‘opencv_core242.dll’),…
fullfile(opencvPath, ‘build\include\opencv\cxcore.h’), …
‘alias’, ‘cxcore’, ‘includepath’, includePath);

hello, I am using opencv 242 and matlab 2011b. When i try to load the library, it loads. But when I try to call (libfunctions) it I get an error. It does not find it!

Hi

Warning: Warnings messages were produced while parsing. Check the functions you intend to use for
correctness. Warning text can be viewed using:
Warning: Function D:\tracking\load.m has the same name as a MATLAB builtin. We suggest you rename the
function to avoid a potential name conflict.
In onCleanup>onCleanup.delete at 61
C:\OpenCV\bin\cxcore100.dll is not a valid Win32 application.