=================== Tutorial: |project| =================== The aim of this tutorial is to give you an overview of how computer vision tasks can be performed in a distributed fashion using the middleware :ref:`RSB ` and assorted libraries. For this purpose we will explain example programs in C++ and |python|, which receive input images via :ref:`RSB `, do some processing on them using the well-known `OpenCV library `_, and publish their processing results again via :ref:`RSB `. The example projects can be downloaded and are based on the project structure explained in the :ref:`Build System Essentials Tutorial`. In case the explanations about the build system used in this tutorial are too sparse, please refer to the respective section in that tutorial. .. note:: While this tutorial is generally usable for everyone, it was specifically created for the use in a lecture. Some explanations are specific for this lecture and marked in special boxes. .. note:: For questions and feedback regarding :ref:`RSB `, you can write a mail to the mailing-list rsb@lists.cor-lab.uni-bielefeld.de. You can subscribe to the mailing-list `here `_. There is also an IRC channel for :ref:`RSB ` support: irc://irc.freenode.net/rsb. Bug reports are always welcome on `our bug-tracker `_. Preconditions ============= In order to complete the tutorial you need a working installation of the following software packages on a Linux computer: * :ref:`RSB ` version 0.10, language implementation in the programming language you would like to work with * `rsb-tools-cl `_, version 0.10 * `rsb-gstreamer `_, version 0.10 * :ref:`RST and rst-converters ` in version 0.10 * `OpenCV `_, at least version 2.4.8 (should be installed via system packages) * `CMake`_ (should be installed via system packages) For the installation of the RSx software, please refer to :ref:`the installation instructions `. Moreover, you need a webcam for generating live images. You should be familiar with the programming language you are going to use in this tutorial and basic knowledge about `build tools and compiler invocations `_ in this language should exist. .. container:: isy The required software is already installed on the lab computers and you will stream webcam images from your own camera on your computer. In order to use the ISY installation of these dependencies (``PATH``, ...), please execute the following command for each terminal (or by adding it to your login profile) that you want to use in this tutorial: .. code-block:: sh module load isyenv This command will also define the ``$prefix`` variable, which is used constantly throughout the tutorial. It points to the installation prefix for this semester, which resides in a subdirectory of ``/vol/isy``. Tutorial ======== The following outline will give you a rough idea of the steps performed in this tutorial. .. sectnum :: .. contents:: :local: .. note:: In this online documentation we will only list relevant source code lines. If it is unclear to you how they fit into the whole program, please refer to the .. container:: lang-multi .. container:: lang-cpp :download:`C++ solution project `. .. container:: lang-python :download:`Python solution project `. .. warning:: This is not the project you should start the tutorial with! Setting up RSB Connection and Webcam Streaming ---------------------------------------------- For this tutorial we will use a simple :term:`socket` based :term:`transport` of :ref:`RSB `. As :ref:`RSB ` supports this functionality out of the box, there is no specific configuration required. .. note:: If you want to ensure that everything is correctly set up, you can specifically enable the socket :term:`transport` in the configuration of :ref:`RSB ` and disable other transports. For this purpose, create or change :file:`~/.config/rsb.conf` with/to these contents: .. code-block:: ini [transport.inprocess] enabled = 0 [transport.socket] enabled = 1 [transport.spread] enabled = 0 These settings should be the defaults, however. Setting up a RSB logger ....................... In order to see the :term:`events ` which are exchanged via :ref:`RSB `, we will firstly set up a logger which plots the different :term:`scopes ` and their activity against time. In order to do this, we simply call the `rsb-loggercl tool `_ with the correct ``--style`` argument on the socket transport root. .. code-block:: sh $prefix/bin/rsb-loggercl0.10 --style=timeline/scope socket:/ .. note:: When using socket based transport, the first launched :ref:`RSB ` process will bind to the socket and works as a server process. Therefore, after launching the logger, we should keep it open during the entire tutorial to avoid socket blocking/binding issues. This will not be an issue when using other transports (e.g. spread based). Using a Webcam .............. We will use the `GStreamer multimedia framework `_ to capture data from the webcam and send it over :ref:`RSB ` using the GStreamer plugins. .. container:: isy Each of you should be supplied with a webcam in order to set up your own streaming pipeline. For this purpose, ensure that you have installed the :ref:`RSB ` plugins as described and launch: .. code-block:: sh gst-launch-0.10 gconfvideosrc ! video/x-raw-yuv, width=320, height=240, framerate=30/1 ! ffmpegcolorspace ! video/x-raw-rgb, blue_mask=16711680, green_mask=65280, red_mask=255 ! queue ! rsbvideosink "scope=/video" Afterwards, images will be streamed on the :ref:`RSB ` :term:`scope` ``/video``. Leave the streaming running during the whole work on the tutorial. Using the pre-recorded bag file ............................... If you do not have a webcam, you can also use a `pre-recorded video snippet `_ as an alternative. It was recorded using the `bag-recordcl `_ tool (part of the `rsbag-tools-cl `_) and optionally replaces the use of a webcam. .. container:: isy For your convenience we put this file already in the isy volume so that you do not need to download the file. .. code-block:: sh $prefix/share/data/short-video.tide You can simply re-play the file with the `bag-playcl `_ command. .. code-block:: sh $prefix/bin/bag-playcl0.10 $prefix/share/data/short-video.tide socket: .. note:: The replay stops after a successful run. For this tutorial you can simply loop the execution of this command in order to get a continuous video stream. .. code-block:: sh while true; do $prefix/bin/bag-playcl0.10 $prefix/share/data/short-video.tide socket: ; sleep 1; done; Getting the Scratch Project --------------------------- We have created a version of the example code without the instructions discussed in this tutorial to speed up the initial setup phase. .. container:: lang-multi .. container:: lang-cpp Please, download the :download:`scratch project ` and extract it on your computer. .. container:: isy The project template builds a single binary named ``rsbcv``. In order to prevent name clashes, you should rename the built executable to a unique name for your group. Therefore, modify the following line in the :file:`CMakeListst.txt` in the project root folder. .. literalinclude:: example-projects/cpp-final/CMakeLists.txt :language: cmake :start-after: mark-start::binary-name :end-before: mark-end::binary-name :linenos: In order to verify that the required libraries are properly installed on your computer and to get you started with `CMake`_, we will try to configure and compile this test project: #. Inside the root folder of the downloaded project, create a folder called ``build``: .. code-block:: sh cd path/to/cpp mkdir build cd build #. Call `CMake`_ to configure the project. This means `CMake`_ will search for all dependencies of our test project and create a Makefile. .. code-block:: sh cmake -DCMAKE_BUILD_TYPE=debug -DCMAKE_PREFIX_PATH=${prefix} .. ``${prefix}`` indicates for each :term:`upstream` project, the path where it was installed to. .. container:: isy ``${prefix}`` is set to ``/vol/isy/current/releases/trusty`` by the module ``isyenv``. .. note:: In case that multiple required libraries are installed into different prefixes, then you need to indicate for each of the required libraries where they can be found. For the above call this would look like: .. code-block:: sh cmake -DCMAKE_BUILD_TYPE=debug -DRSC_DIR=${prefix}/share/rsc0.10/ -DRSB_DIR=${prefix}/share/rsb0.10/ -DRST_DIR=${prefix}/share/rst0.10/ -Drst-converters_DIR=${prefix}/share/rst-converters0.10 .. `CMake`_ should exit successfully with a log comparable to this one:: -- The C compiler identification is GNU -- The CXX compiler identification is GNU -- Check for working C compiler: /usr/bin/gcc -- Check for working C compiler: /usr/bin/gcc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- protoc does not support matlab -- Found PROTOBUF: /usr/lib/libprotobuf.so -- Performing Test CHECK_CXX_FLAG_pipe -- Performing Test CHECK_CXX_FLAG_pipe - Success -- Performing Test CHECK_CXX_FLAG_Wall -- Performing Test CHECK_CXX_FLAG_Wall - Success -- Performing Test CHECK_CXX_FLAG_Wextra -- Performing Test CHECK_CXX_FLAG_Wextra - Success -- Performing Test CHECK_CXX_FLAG_DIAGNOSTICS -- Performing Test CHECK_CXX_FLAG_DIAGNOSTICS - Success -- Found Subversion: /usr/bin/svn (found version "1.6.17") -- Configuring done -- Generating done -- Build files have been written to: /homes/jwienke/workspace/distributed-computer-vision-with-rsb/build/cpp #. Afterwards, you can try to compile the project using :program:`make`. If this succeeded, you can finally launch the example binary for the first time by executing:: src/rsbcv The program will exit immediately as you will have to fill it with life during the remainder of this tutorial. You do not need to modify the build system during the course of this tutorial. Please remember to check the :ref:`Build System Essentials ` for details on that, especially when you start developing your own code! Inside the project you will find a single source file :download:`example-projects/cpp/src/rsbcv/binary.cpp`. We have already provided a method to parse two :term:`scopes ` from the command line: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::commandline-arguments :end-before: mark-end::commandline-arguments :linenos: .. container:: lang-python Please, download the :download:`scratch project ` and extract it on your computer. .. container:: isy The project template contains a script with a single name ``rsbcv-harris``. In order to prevent name clashes, you should rename the script that is going to be built to a unique name for your group. Therefore, modify the left side of the following line in the :file:`setup.py` in the project root folder. .. literalinclude:: example-projects/python-final/setup.py :language: python :start-after: mark-start::binary-name :end-before: mark-end::binary-name :linenos: In order to verify that the required libraries are properly installed on your computer and to get you started with setuptools for |python| we will try to compile and install the project without any modifications: #. Choose a temporary prefix where you are going to install the project for testing, e.g. ``prefix=/tmp/testinstall``. Execute the following code to set up a |python| site directory in this prefix and make it available in the shell: .. code-block:: sh export prefix=/tmp/testinstall mkdir -p $prefix/lib/python2.7/site-packages export PYTHONPATH=$prefix/lib/python2.7/site-packages:$PYTHONPATH .. note:: In case you are working with a different |python| version, adapt the aforementioned path to that version. #. Change the working directory to the root folder of the downloaded project and execute: .. code-block:: sh python2 setup.py install --prefix=$prefix This will install the project in the selected prefix. #. Launch the installed program for testing purposes: .. code-block:: sh $prefix/bin/rsbcv-harris The program will exit immediately as you will have to fill it with life during the remainder of this tutorial. .. container:: isy You will need to use your custom program name instead of the generic one given here for launching. You do not need to modify the build system during the course of this tutorial. Please remember to check the :ref:`Build System Essentials ` for details on that, especially when you start developing your own code! Inside the project you will find a single source file :download:`example-projects/python/rsbcv/__init__.py`. We have already provided a method to parse two :term:`scopes ` from the command line: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::commandline-arguments :end-before: mark-end::commandline-arguments :linenos: .. note:: It might happen that you are unable to kill the running |python| program using :kbd:`Control + C`. This can happen if a dependency used by :ref:`RSB ` blocks this signal. Instead, you can try :kbd:`Control + 4`. However, if none of these work, you can still kill it using :kbd:`killall -15 rsbcv-harris*`. This makes it possible for the user of the resulting executable to specify the :term:`scope` from which images will be received and the :term:`scope` on which the processed results will be published. Receiving Images ---------------- To receive :term:`events ` using :ref:`RSB ` you need to instantiate a :term:`listener`. A :term:`listener` is an object which receives all incoming :term:`events ` on a specified :term:`scope` and passes them to registered :term:`handlers `, which are callbacks provided by the user of :ref:`RSB `. .. container:: lang-multi .. container:: lang-cpp We will now create a :term:`listener` to receive images from the streamed webcam in OpenCV's image :term:`data type` ``IplImage``. All of this will happen in the :cpp:func:`main` function after the command line parsing (except including headers). #. Include the required headers for :ref:`RSB `: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::rsb-headers :end-before: mark-end::rsb-headers :linenos: #. Get the :cpp:class:`rsb:rsb::Factory` instance. .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::factory :end-before: mark-end::factory :linenos: The :cpp:class:`rsb:rsb::Factory` is responsible for creating :term:`RSB participants ` like :term:`listeners `. #. Create a :cpp:class:`rsb:rsb::Listener` on the input :term:`scope` specified via the command line option: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::listener :end-before: mark-end::listener :linenos: #. The :term:`listener` will receive :term:`events ` now but it still needs to know what to do with these :term:`events `. In our case we would like to store them in a queue for later processing. Therefore, we first instantiate such a queue. As the queue will contain OpenCV ``IplImage`` objects, we first need to include the OpenCV headers: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::opencv-headers :end-before: mark-end::opencv-headers :linenos: We also need a queue implementation which is synchronized so that multiple threads can operate on it in parallel. For this purpose we will use the implementation from `RSC `_ and include the respective header: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::queue-header :end-before: mark-end::queue-header :linenos: Afterwards, create an instance of this queue class for ``IplImage`` instances: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::create-queue :end-before: mark-end::create-queue :linenos: As this queue needs to get passed around later, we have to create it on the heap using the ``new`` operator. To avoid manual memory management and possible memory leaks, we maintain the instance of the queue inside a `boost::shared_ptr `_. The :cpp:class:`rsc::threading::SynchronizedQueue` itself also uses templates to be usable for varying data types. In this case it will maintain :cpp:class:`boost::shared_ptr`\ s of ``IplImage``. The reason for this is that every received :ref:`RSB ` :term:`event` contains its :term:`payload ` inside these shared pointers and hence we will resemble this structure. Finally, the ``1`` in the constructor call of the queue instance limits the queue size to 1. Assuming that our own program is too slow to process all incoming images, this will ensure we do not queue up more and more images. Instead, old images will be deleted and only the most recent one will remain in the queue. #. Now that the queue is available, we need to instruct the :term:`listener` to store the images contained in the received :term:`events ` into this queue. For this purpose we install a specialized handler called :cpp:class:`rsb:rsb::QueuePushHandler` that comes with :ref:`RSB ` which pushes the data of each :term:`event` into a queue. First, include the respective header: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::queuepushhandler :end-before: mark-end::queuepushhandler :linenos: And afterwards install the handler in the :cpp:class:`rsb:rsb::Listener` instance: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::handler :end-before: mark-end::handler :linenos: #. We have instructed :ref:`RSB ` to fill the queue with the most recent images. Now we can start a main working loop which reads from the queue and in the first iteration displays the received images: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::working-loop :end-before: mark-end::working-loop :append: } :linenos: The :cpp:member:`rsc:rsc::threading::SynchronizedQueue::pop` method of the queue will block if our processing is faster than the streaming from the webcam and the queue was hence empty. At this point, we convert the legacy data type ``IplImage`` into the ``cv::Mat`` data type which is used in current OpenCV versions. Images of this type can be displayed using ``cv::imshow`` which, in our example, creates a new window with the name ``input`` (if it does not already exist) and displays the received image. However, we need to explicitly trigger repainting of the window by calling ``cv::waitKey``. You can try to compile the program now again by calling :program:`make` and execute it as described before. However, you will not yet see the webcam images. Instead, warnings like this one will be printed:: 1349972760057 rsb.spread.ReceiverTask [WARN]: ReceiverTask::notifyHandler catched std exception: No converter for wire-schema or data-type `.rst.vision.Image'. Available converters: {.*: *rsb::converter::ByteArrayConverter[wireType = std::string, wireSchema = .*, dataType = bytearray] at 0x20c2460 .rsb.protocol.EventId: *EventIdConverter[wireType = std::string, wireSchema = dummy, dataType = rsb::EventId] at 0x20c23d0 bool: *rsb::converter::BoolConverter[wireType = std::string, wireSchema = bool, dataType = bool] at 0x20c1850 uint64: *rsb::converter::Uint64Converter[wireType = std::string, wireSchema = uint64, dataType = unsigned long] at 0x20c2bf0 utf-8-string: *rsb::converter::StringConverter[wireType = std::string, wireSchema = utf-8-string, dataType = std::string] at 0x20c2750 void: *rsb::converter::VoidConverter[wireType = std::string, wireSchema = void, dataType = void] at 0x20c2cb0} :ref:`RSB `, so far, does not know how to decode the received data (which are in a binary format for network transmission) and transform them into ``IplImages``. We need to install a so called :term:`converter` for this purpose. In this case the rst-converters project provides :cpp:class:`rst:rst::converters::opencv::IplImageConverter`, which does exactly the required thing. To install this :term:`converter`, first include the respective header file: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::image-converter-header :end-before: mark-end::image-converter-header :linenos: Moreover, we need to include a header from :ref:`RSB ` to register this converter in :ref:`RSB `'s repository of known :term:`converters `: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::repository-header :end-before: mark-end::repository-header :linenos: Finally, we can install the ``IplImageConverter``. This needs to be done *before* creating the :term:`listener`: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::converter-registration :end-before: mark-end::converter-registration :linenos: After recompiling the program you should finally see the unprocessed images from the webcam. .. container:: lang-python We will now create a :term:`listener` to receive images from the streamed webcam in OpenCV's image :term:`data type` ``cv.iplimage``. All of this will happen in the :py:func:`detectHarris` function after the command line parsing. #. Import the required package for :ref:`RSB `: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::import-rsb :end-before: mark-end::import-rsb :linenos: #. Create a :py:class:`rsb:rsb.Listener` on the input :term:`scope` specified via the command line option: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::listener :end-before: mark-end::listener :linenos: #. The :term:`listener` will receive :term:`events ` now but it still needs to know what to do with these :term:`events `. In our case we would like to store them in a queue for later processing. Therefore, we first instantiate such a queue. As the queue will contain OpenCV images, which are represented as :py:mod:`numpy` arrays in recent versions of OpenCV, we first need to import the :py:mod:`opencv` module and the :py:mod:`numpy` module in the global namespace: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::opencv :end-before: mark-end::opencv :linenos: Moreover, we need to import a queue class: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::import-queue :end-before: mark-end::import-queue :linenos: Afterwards, create an instance of this queue for the images: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::queue :end-before: mark-end::queue :linenos: As we only want to process the most recent image, the queue is limited to size one. However, it is important to note that the |python| queue implementation does not throw away old images in case your code is too slow to cope with the amount of incoming images. We will handle this case in the next step. #. Now that the queue is available, we need to instruct the :term:`listener` to store the images contained in the received :term:`events ` into this queue. For this purpose we need to install a :term:`handler`, which is a :py:func:`callable` accepting one argument, the :term:`event`: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::handler :end-before: mark-end::handler :emphasize-lines: 6 :linenos: The function first tries to erase the only item that can potentially be in the queue if our own processing code was too slow to grab it from the queue in the time since the last insert. This operation throws an exception if no element was in the queue (which would be ideal because no image was missed) and we can safely ignore this exception. Afterwards, the new image is inserted into the queue. As the new API version of OpenCV from the :py:mod:`cv2` module preferably works with :py:mod:`numpy` arrays as the data type for images, but we will receive backwards-compatible :py:class:`cv.iplimage` instances, we need to convert the received data to this representation, which is done in line 6 before filling the queue. Finally, the function is added as a :term:`handler` to the :term:`listener` created earlier. #. We have instructed :ref:`RSB ` to fill the queue with the most recent images. Now we can start a main working loop which reads from the queue and in the first iteration displays the received images: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::working-loop :end-before: mark-end::working-loop :linenos: The :py:meth:`Queue.Queue.get` method of the queue will block if our processing is faster than the streaming from the webcam and the queue was hence empty. :py:func:`cv2.imshow` creates a new window with the name ``input`` if it is not existing yet and displays the received image. However, we need to explicitly trigger repainting of the window by calling :py:func:`cv2.waitKey`. .. note:: We use the variable :py:mod:`doRun` as a condition for the loop in order to be able to abort the program at any point in time in a clean way. For that we will write a simple signal handler which will catch :kbd:`Control + C` and exit the main loop. .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::exit-strategy :end-before: mark-end::exit-strategy :linenos: If you don not want to do this, you can simply use a :py:mod:`while True:` loop. You can try to install the program now again as described before and execute it. However, you will not yet see the webcam images. Instead, warnings like this one will be printed:: Exception in thread Thread-6: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 504, in run self.__target(*self.__args, **self.__kwargs) File "/home/languitar/.local/lib/python2.7/site-packages/rsb_python-0.8.0-py2.7.egg/rsb/transport/rsbspread/__init__.py", line 216, in __call__ raise e KeyError: '.rst.vision.Image' :ref:`RSB `, so far, does not know how to decode the received data (which are in a binary format for network transmission) and transform them into ``cv.Iplimage``\ s. We need to install a so called :term:`converter` for this purpose. In this case the rst-converters project provides :py:class:`rst:rstconverters.opencv.IplimageConverter`, which does exactly the required thing. To install this :term:`converter`, first import it: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::image-converter :end-before: mark-end::image-converter :linenos: Moreover, we need to include a function from :ref:`RSB ` to register this converter in :ref:`RSB `'s repository of known :term:`converters `: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::repository-function :end-before: mark-end::repository-function :linenos: Finally, we can install the ``IplimageConverter``. This needs to be done *before* creating the :term:`listener`: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::converter-registration :end-before: mark-end::converter-registration :linenos: After reinstalling the program you should finally see the unprocessed images from the webcam. Processing the Received Image ----------------------------- We will now apply a Harris Corner detector which is included in OpenCV and produces a set of corner points when given an input image. These sets of points will be sent via :ref:`RSB ` (e.g. for further processing). .. container:: lang-multi .. container:: lang-cpp #. Detect Harris corners: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::headers-cornerharris :end-before: mark-end::headers-cornerharris :linenos: This header file provides the interface to the Harris corner detector. .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::cornerharris :end-before: mark-end::cornerharris :linenos: We convert the image to grayscale using the ``cvtColor`` function and also create an empty Matrix using ``cv::Mat::zeros`` into which the detector can store its detection result mask. The ``cv::cornerHarris`` function performs the actual detection and stores the discovered corner points into ``dst``. The additional arguments for the ``cv::cornerHarris`` can be used to influence the detection of corners in the image. For now we simply choose the arguments ``2, 3, 0.04``, as they will produce good results for this tutorial. #. Provide debug output for the detected features: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::display-keypoints :end-before: mark-end::display-keypoints :linenos: A good way to visualize the detected points is overlaying them over the input image. This can be achieved by calling the ``circle`` function, which draws a circle for a point into a copy of an existing image. As before, using ``cv::imshow`` and ``cv::waitkey``, this new image can be displayed. In principle one of the calls to ``cv::waitKey`` could be removed to avoid an unnecessary amount of repainting. You should now be able to recompile the program. .. container:: lang-python #. Detect Harris corners: In order to detect Harris corners we have call the according function on our input image: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::cornerharris :end-before: mark-end::cornerharris :linenos: The :py:meth:`cv2.cornerHarris` method performs the detection. We have to convert our color image to grayscale before using this method as the |python| bindings do not accept color images. This is done by using :py:meth:`cv2.cvtColor`. The arguments for the :py:meth:`cv2.cornerHarris` can be used to influence the detection of corners in the image. For now we simply choose the arguments :py:mod:`2, 3, 0.04`, as they will produce good results for this tutorial. Finally, we create a mask which contains all corner points. #. Provide debug output for the detected features: A good way to visualize the detected points is overlaying them over the input image: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::draw-keypoints :end-before: mark-end::draw-keypoints :linenos: Now that we have this function we can display the resulting image: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::display-keypoints :end-before: mark-end::display-keypoints :linenos: With the first line of this code fragment we provide a copy of our original image on which the keypoints are drawn in the next line. As before, using ``cv2::imshow`` and ``cv2::waitkey``, this new image can be displayed. In principle one of the calls to ``cv2::waitKey`` could be removed now to avoid an unnecessary amount of repainting. You should now be able to reinstall the program. After launching you should see two debug windows: One with the original image and a second window with the same image and the detected keypoints. Publishing the Processed Results via RSB ---------------------------------------- Now we will make the detected keypoint locations available via :ref:`RSB `. In order to publish information to the network you will use an :term:`informer` instance. Data that is sent over the network needs to be serialized into a binary representation that is understandable by all interested system components. That means components need to agree on a specific binary format. To ensure this fact, we maintain a library of data types called :ref:`RST `. This library contains data type definitions for many different robotics and intelligent systems purposes specified using `Google Protocol Buffers `_. Protocol buffers allows us to define data types in an abstract interface definition language that can then be translated into concrete classes for most major programming languages. These classes act as data holders and include serialization functionality. For this tutorial we will use one of the data types from RST, namely ``rst.geometry.PointCloud2DInt``, to encapsulate and serialize the detected Harris corners. As this is a protocol buffers based data type, we will also learn how to enable RSB to send such data types by registering a ``ProtocolBufferConverter`` inside RSB. .. container:: lang-multi .. container:: lang-cpp #. Create an :term:`informer`. For this purpose first include the respective header file: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::informer-header :end-before: mark-end::informer-header :linenos: And afterwards instantiate a new :term:`informer` instance using the :ref:`RSB ` factory. This code **must** be outside of the main processing loop to avoid creating the instance over and over again, which will waste performance. .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::informer-instance :end-before: mark-end::informer-instance :linenos: The :cpp:class:`rsb:rsb::Informer` class requires a template argument with the :term:`data type` it will send. Additionally, it is also maintained by a shared pointer. In this case, a typedef exists inside :cpp:class:`rsb:rsb::Informer`, which is the :cpp:member:`rsb:rsb::Informer::Ptr` member. The :term:`informer` is created on the output :term:`scope` specified through the command line option. #. Publish the resulting image at the end of the main loop: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::publish :end-before: mark-end::publish :linenos: This populates the ``PointCloud2DInt`` data structure with the detected points and then publishes it using the :term:`informer`. However, the publishing can only work if :ref:`RSB ` knows how to send the ``PointCloud2DInt`` :term:`data type` over the network. #. Register the PointCloud data type The ``PointCloud2DInt`` :term:`data type` is made available and prepared for network transmission by including the following headers: .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::rst-headers :end-before: mark-end::rst-headers :linenos: After that, :ref:`RSB ` has to be informed about the new :term:`data type` and its associated :term:`converter` by registering an instance of the ``ProtocolBufferConverter`` for our point cloud data type. .. literalinclude:: example-projects/cpp-final/src/rsbcv/binary.cpp :language: cpp :start-after: mark-start::rst-converter-registration :end-before: mark-end::rst-converter-registration :linenos: .. container:: lang-python #. Create an :term:`informer`. In order to send data, we instantiate a new :term:`informer` for the data type ``PointCloud2DInt``, which we will use to send the computed keypoints. This code should be outside of the main processing loop to avoid creating the instance over and over again. .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::informer :end-before: mark-end::informer :linenos: To create a :py:class:`rsb.Informer` instance we need to specify the :term:`scope` on which to send the results and the data type which shall be sent. In this case, the :term:`informer` is created on the output :term:`scope` specified through the command line option, which will be passed into the factory function. #. Publish the resulting image at the end of the main loop: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::publish :end-before: mark-end::publish :linenos: This populates the ``PointCloud2DInt`` data structure with the detected points and then publishes it using the :term:`informer`. However, the publishing can only work if :ref:`RSB ` knows how to send the ``PointCloud2DInt`` :term:`data type` over the network. #. Register the PointCloud data type The ``PointCloud2DInt`` :term:`data type` is made available and prepared for network transmission by first importing the required data type and a :term:`converter` for it: .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::rst-imports :end-before: mark-end::rst-imports :linenos: After that, :ref:`RSB ` has to be informed about the new :term:`data type` and its associated :term:`converter`. .. literalinclude:: example-projects/python-final/rsbcv/__init__.py :language: python :start-after: mark-start::rst-registration :end-before: mark-end::rst-registration :linenos: After reinstalling and launching your program the results will be published on the specified output :term:`scope`. This fact will not produce any immediately noticeable effect. Therefore, the final section explains how to inspect the published results. .. container:: isy Be sure to select a unique output :term:`scope` in the ISY Lab using the command line arguments. Otherwise your output will be mixed with the one of other groups. Verifying the Output -------------------- To verify that the output is actually generated and is correct we will inspect the data sent on the network using the :ref:`RSB logger `: .. code-block:: sh $prefix/bin/rsb-loggercl0.10 -I $prefix/share/rst0.10/proto/stable -I $prefix/share/rst0.10/proto/sandbox -l $prefix/share/rst0.10/proto/sandbox/rst/geometry/PointCloud2DInt.proto socket:/YOUR/OUTPUT/SCOPE Where ``YOUR/OUTPUT/SCOPE`` has to be replaced with the name of the scope on which you publish the results. You should see a continuous stream of :term:`events ` appearing on the selected output :term:`scope`. .. container:: isy The :ref:`logger` binary (and other tools) are installed in the prefix at ``/vol/isy/current/releases/trusty/bin``. .. note:: More information regarding the logger can be found in the :ref:`online documentation `. Indices and Tables ================== * :ref:`genindex` * :ref:`search`