![]() |
University of Murcia, Spain ![]() |
Block programmingStructure of Computer Vision algorithmsComputer Vision applications usually perform intense processing on "heavy" data flows, which can be often logically divided into different stages depending on the nature of the involved algorithms. The following image, for example, shows the conceptual structure for one of such algorithms, the Canny edge detector:
![]() The process of aplying the algorithm on an input image is divided in several processing stages: gaussian filtering, horizontal gradient, vertical gradient, hysteresis thresholding, etc... In QVision, each of these stages can be easily designed and developed separatedly as a different and completely independent task, which is coded and encapsulated independently on a data processing block. When all of these blocks are coded and tested, they can be linked together to create the final application. This approach has some interesting programming advantages:
QVision block programming architectureEach one of these data processing blocks is modelled in the QVision by a software object. These block objects -so called worker objects, in QVision terminology- must be created in the main function (or the main thread of the application), and are always subclasses of the generic QVWorker class included in the QVision. Later, and to create the application structure, the programmer must establish the needed links between them.The following picture illustrates a conceptual diagram for the structure of a typical QVision application, showing the different elements composing it and the relationships between them:
![]() There are three main kind of processing blocks in the QVision:
Three different kind of dependencies can be stablished between the different blocks to build the applications architecture. The first two, synchronous data connection and asynchronous data connection establish a directed data dependency between two workers. They imply that periodically -and with different synchronization policies, depending on the kind of link; more on this later- the source block of the dependency link will send a value to the destination block of the dependency. On the other hand, the third kind of link, the so called event connection, establishes a directed control dependency between two workers. When one of these links is created, it allows the source block of the dependency to eventually send control events to the destination processing blocks of the link. Those events can be captured in the destination blocks to modify their behaviour accordingly to the received event -much like in the well known signal-slot mechanism of the Qt Library-. Data sharing and event management between blocksDynamic properties are the tool used in the QVision architecture to perform data sharing between the different processing blocks in an application using synchronous or asynchronous data connections. A dynamic property is similar to an object property specified in a class (namely stored data values), with the exception that they can be added and deleted from the container object at execution time.Every worker in the QVision is a dynamic property container object. The QVWorker class inherits from a parent class named QVPropertyContainer the functionality to create new dynamic properties inside the worker objects, and store and retrieve their values. More importantly, dynamic properties also allow for the desired capability of object introspection -which permits to an object to know in execution time which properties it contains, and what are their types-. This makes dynamic property containers suitable to be used to perform data sharing between processing blocks, decoupling the design and development of the workers from the real data connections, which are instead established in a latter stage of the development of the final application structure. This way, workers are programmed in a truly independent way of their specific current usage, ready to be reused at any time in a completely different context. This data introspection feature was also used to implement an automatic and fully functional graphical user interface for each application, which literally "discovers" the set of input parameters of each worker in the application, and then offers an adequate set of widgets to manipulate them at execution time. To allow this user interface to inspect parameters, the developer of the algorithm must only use the dynamic property system to establish the input and output data of each worker, for the graphical user interface to be able to discover them, read their values, and modify accordingly depending on the interaction of the final user with the dedicated widget. Event connections are managed using signals and slots, an event notification mecanism already provided by the Qt. The class QVWorker offers functionality for event definition, linking and processing in the QVision. Specific usage of these mechanisms will be reviewed in the example at the end of this section. Multithreaded approachMoreover, thanks to this modular design of applications in terms of workers, the execution of applications based on the QVision is highly parallelizable. For example, remember the task division schema for the Canny algorithm previously seen at the beginning of this section. Each stage is translated to an independent QVision processing block, and then connected to adjacent tasks using the adequate data links. Then, by simply mapping the execution of each worker to a different host thread, when the application runs in a multicore architecture, we can obtain a significant parallel performance gain.By doing so, we are basically dividing the processing of the different (multipath) pipeline stages between different processing units -much like in a hardware pipeline-, thus increasing the data processing thoughput of the algorithm. Of course, to take advantage of this implicit parallelism, the application must iterate on a stream of input images (otherwise, there will not be any chance for real computation overlapping between subsequent stages of the pipeline). But this is usually the case in any computer vision application, which works on sequences of frames sequentially extracted from a video file or a video camera. To make this approach work, the data sharing between processing blocks must be thread safe. This is again ensured using the dynamic property system to temporarily store the input and output data values in the processing block objects, because read and write operations for these properties have been internally programmed in a thread safe manner. Thus, all the involved synchronization is conveniently hidden to the final programmer, which really has not to worry at all about it: everything works behind the scenes, providing this multipath pipeline-style parallelism "for free". In essence, QVision applications can improve their performance when running on multicore/multiCPU systems if the overall processing block structure is well designed and ballanced, and this without requiring too much parallel programming expertise from the developer. Synchronization and thread-safe data sharing occurs automatically, and without requiring from the programmer explicit use of thread locks, semaphores, or any other typical low level tools of classical multithread programming. Synchronous vs. asynchronous data linksData links can be synchronous or asynchronous. Looking again at our Canny example, each pipeline stage or independent task must process its linked input image(s) or data only when the previous stages in the pipeline have conveniently computed and feeded them to it as input . The synchronous data links are used to specify that kind of temporal dependence, ensuring that the n iteration of a given worker does not start until the corresponding n iterations of its synchronously linked input workers have been correctly completed.Asynchronous links, on the other hand, can connect different logical parts of a QVision application, whenever a periodic update or data sharing is needed between two different pipeline structures, but without the temporal constraints of the synchronous case. For example, the developer can create real-time applications with one or several (perhaps heavy) background processes, which won't block the execution of the main process, but will periodically share some data or information needed by the latter to perform its computations. Another example could be a computer vision application with a set of decoupled algorithms, which produce heterogeneous pieces of high level information that must be subsequently used to update a common physical model of the world. Very probably, the different involved (subsets of) workers might have very different processing speeds; but the asynchronous nature of the communication betweeen them all will avoid the slowest algorithms to block while waiting for the slowest. Still, a common model comprising information from all of them may be maintained and adequately updated at a different pace. Creating block structured applications: a first exampleWe will review in this section the creation of a moderately complex application using the block oriented programming design of the QVision -the application is in fact very simple, but we think it illustrates well the more important aspects of QVision block oriented programming-. In this specific example, the application will perform a simple moving edge detection on the input video sequence, based on the Canny edge detector.
First step: performing Canny edge detectionOur first approximation will apply the Canny edge detector on the frames of an input video sequence, and will simply display the detected edges on the image on an image canvas.This is the corresponding Qt's project file:
include(/usr/local/QVision.0.1.1/qvproject.pri) TARGET = movingEdgeDetector SOURCES = movingEdgeDetector.cpp The project contains a source file called movingEdgeDetector.cpp. The source code is the following:
#include <QVApplication> #include <QVMPlayerCamera> #include <QVDefaultGUI> #include <QVImageCanvas> #include <QVCannyEdgeDetector> int main(int argc, char *argv[]) { QVApplication app(argc, argv, "Example program for QVision library. Obtains Canny borders from input video frames." ); QVDefaultGUI interface; QVMPlayerCamera camera("Video"); QVCannyEdgeDetector cannyWorker("Canny worker"); camera.link(&cannyWorker, "Input image"); QVImageCanvas cannyDisplayer("Canny"); cannyDisplayer.linkProperty(cannyWorker,"Input image"); cannyDisplayer.linkProperty(cannyWorker,"Output contours"); return app.exec(); } The application structure contains three main blocks. The most important one, the QVCannyEdgeDetector object, is the only data processing block of this first example application, and internally contains the code needed to detect edges on input images using the Canny edge detector algorithm. The second one is an input block, a QVMPlayerCamera object, which provides a constant flow of sequenced frames to the Canny edge detector block from an arbitrary video source. Finally, we also create a third output block, which is a QVImageCanvas object, which will display the original input image for the Canny edge detector algorithm, and the detected edges:
![]() The QVision contains a library of ready to use data processing block classes, which can be used by the developer to create new worker objects and use them to construct new applications. Though still modest, this library keeps constantly growing with workers added by the QVision mantainers and, hopefully in the future, by users/researchers who program their own algorithms and methods. The QVCannyEdgeDetector is currently included amongst them. The Input image and Output contours of the cannyWorker object are two dynamic properties contained in that object. The first property Input image must contain the input image whenever the processing object starts the processing of a new frame. The Output contours property will contain the resulting edges obtained by applying the Canny algorithm on the input image. Of course, each of these properties have their adequately predefined type, but, for now, we will not worry about these details (later in this example we will get back to this issue). By linking the Input image property with the QVMPlayerCamera object camera, we specify that whenever the camera reads a new image frame from its configured video source, it will be stored in the Input image property, and the Canny object will process it. The QVImageCanvas object can display images and the resulting edges of the Canny algorithm. By linking both properties, Input image and Output contours, with the cannyDisplayer object of type QVImageCanvas, we stablish that whenever the cannyWorker processing block obtains a new list of edges from an image, the cannyDisplayer object will automatically update the displayed image and edges. Once compiled using the qmake and make tools, we can execute the application using the following line:
./movingEdgeDetector --URL=/home/user/videos/penguin.avi The application will open the image canvas, and display the edges on each input image on the video sequence:
![]() Second step: add a movement detectorNow we will add two workers and connect them adequately in the application, to create a basic movement detector. In order to do it, we first include these headers in the application:
#include <QVImageRetarderWorker> #include <qvippworkers.h> They contain definitions for the worker classes QVImageRetarderWorker and QVIPPAbsDiffWorker, which we will combine to create the movement detector. These workers will obtain a time diference of the input video sequence. We add the following code to the main function:
int main(int argc, char *argv[]) { [...] QVImageRetarderWorker<uChar,1> retarderWorker("Image retarder worker"); camera.link(&retarderWorker, "Input image"); QVIPPAbsDiffWorker<uChar,1> absDiffWorker("Absolute difference worker"); camera.link(&absDiffWorker, "src1"); retarderWorker.linkProperty("Output image", &absDiffWorker, "src2", QVWorker::SynchronousLink); QVImageCanvas movementDisplayer("Movement detector"); movementDisplayer.linkProperty(absDiffWorker,"dest"); return app.exec(); } This updates the block structure of the application: ![]() The application will now show the following output: ![]() Thrid step: adding an edge movement detectorWorker creationSuppose we want to detect which of the edges obtained with the Canny worker are moving. In order to do it, we will adopt a very simple approach, in order to keep the example simple, though illustrative enough: we will create a fourth worker which will mix the information coming from the QVIPPAbsDiffWorker and the QVCannyEdgeDetector. In this case, we create this new worker from scratch (not using a pre-existing worker from the library). Basically, we will make the new worker to produce an image displaying the edges detected by the QVCannyEdgeDetector which fall inside an area where some movement was detected by the movement detector.We proceed by creating a new class, which will be used to create the worker object. This class must inherit from the QVWorker base class, and should include a constructor method, and implement the virtual method iterate(), from the QVWorker class:
#include <QVPolyline> class MovingEdgesDetector: public QVWorker { public: MovingEdgesDetector(QString name): QVWorker(name) { addProperty< QList<QVPolyline> >("Image borders", inputFlag); addProperty< QVImage<uChar,1> >("Movement image", inputFlag); addProperty< QVImage<uChar,1> >("Moving borders image", outputFlag); } void iterate() { // Read parameters const QList< QVPolyline > imageBorders = getPropertyValue< QList<QVPolyline> >("Image borders"); const QVImage<uChar,1> movementImage = getPropertyValue< QVImage<uChar,1> >("Movement image"); // Create and initialize to zero the output image, with the same dimensions as 'movementImage' QVImage<uChar,1 > movingBordersImage(movementImage.getCols(), movementImage.getRows()); Set(0, movingBordersImage); // Traverse edge point lists, checking if their location at the movement image // has a greyscale value greater than 10. If so, set that location at image // 'movingBordersImage' to 255. foreach(QVPolyline edge, imageBorders) foreach(QPoint edgePoint, edge) if (movementImage(edgePoint) > 10) movingBordersImage(edgePoint) = 255; // Store resulting image setPropertyValue< QVImage<uChar,1> >("Moving borders image", movingBordersImage); timeFlag("Publish resulting images"); } }; Observe that the constructor of the worker object creates the input and output dynamic properties of the worker, using the templated method addProperty. These properties are required for the data processing block to communicate with the rest of blocks of the application: input properties will receive input values from other blocks in the application, while the worker itself will store its resulting output values to share them with the rest of the blocks at the end of each iteration. The iterate method, reimplemented from base QVWorker class, defines the code which will obtain the output data by processing each new set of input values. Usually, first instructions in this method read the input properties values. In this case, input properties are Image borders, which will store a list of polylines representing the edges obtained with the Canny edge detector, and the Movement image, coming from the movement detector of the application. The function creates a new image to draw the points contained in the edges detected with Canny which fall inside a region of the movement image with a gray-scale value greater than 10. This condition might indicate that the corresponding pixel value varied on a small period of time, and thus it corresponds to a changing area of the image. Once the movingBordersImage image has been obtained, it is stored in an output property named Moving borders image, and the iterate method finishes. Finally, and once the new worker interface and internal behaviour has been completely defined, we will create an instace of movement detector object in the main function, and connect it with with synchronous links to the different elements of the application, so it can correctly work. The following lines of code before the call to the exec() method of the application instance will do the work:
int main(int argc, char *argv[]) { [...] MovingEdgesDetector movingEdgesDetector("Moving edges detector"); cannyWorker.linkProperty("Output contours", &movingEdgesDetector, "Image borders", QVWorker::SynchronousLink); absDiffWorker.linkProperty("dest", &movingEdgesDetector, "Movement image", QVWorker::SynchronousLink); QVImageCanvas edgeMovementDisplayer("Movement detector"); edgeMovementDisplayer.linkProperty(movingEdgesDetector,"Moving borders image"); return app.exec(); } This is the final block diagram of our sample application:
![]() Compiling and executing the application with a video file showing a moving object (in this case a person) against a fixed background shows how the application segments the edges corresponding to the person who is moving in the video sequence:
![]()
Asynchronous data connectionsSo far we have only worked with synchronous data linking between workers. These links were stablished using the method linkProperty from the QVPropertyHolder class. Up to now, the last parameter of these calls has been always the same value: QVWorker::SynchronousLink. That made the data link to be synchronous.Using the value QVWorker::AsynchronousLink in that parameter would make the corresponding link asynchronous. For example, suppose we wanted to add a face detector based on Canny edges to the application developed in the previous section. We can assume that the processing of that detector -i.e., the sequence of instructions executed in its iterate method- would be slow. Therefore, should we connect the corresponding worker to the main processing chain of the application using a synchronous link, it would dramatically slow down the frame rate of the original moving edge detector. The solution is to link the new block to the overall block diagram of the application using an asynchronous link. For example, if we had developed a worker class CannyFaceDetector which obtains the edges from a Canny edge detector and performs a (maybe slow) face detection procedure with them, the following code illustrates how we could add it to our application:
int main(int argc, char *argv[]) { [...] CannyFaceDetector cannyFaceDetector("Canny edges based face detector"); cannyWorker.linkProperty("Output contours", &cannyFaceDetector, "Input contours", QVWorker::AsynchronousLink); // Add any output block connected to the 'cannyFaceDetector', storing or displaying // its results. return app.exec(); } The final application block diagram for this last example would be the following:
![]() Event connections
|