Example: Recording Audio/Video data from the Nao robot

This (very specific) example use case shows how to record audio and video data from the internal sensors of the NAO robot built by Aldebaran Robotics and gives some pointers on what you can then do with this data.

This section just lists some common example commands and tries to give some pointers to more documentation. You should definitely check out the other sections of this tutorial!

Warning

This is a very specific use case that will probably only apply to a small group of RSBag users. If you don’t have access to a NAO robot, this is not a good example for you!

Installing and configuring required software

First of all you should make sure that you have current versions of both NAOqi (1.12) and RSB (at the moment 0.6 seems to be the best choice and we expect this version through the rest of this tutorial, but this might get updated to 0.7 or 0.8 soon) installed on your robot.

  • If you are a member of a research project in collaboration with Bielefeld University (e.g. HUMAVIPS or SoziRob), the easiest way to get RSB on your robot is probably to use the pre-built installation images provided on the CI server. For NAOqi 1.12 and RSB 0.6, you can download these images here.
  • Choose the architecture corresponding to your Nao robot (geode for the initial, old Nao heads and atom for the newer current Nao heads and the special stereo head), download the image and copy it to /home/nao on the robot.
  • SSH to the robot as the normal nao user, extract the .tar.gz in the home directory /home/nao as that user.

Make sure that you have current versions of RSB (at the moment 0.6 is the recommended version used in this tutorial), RST and the RSBag tools installed on your host PC.

  • Installation instructions for RSB can be found in the RSB manual.
  • For the RSBag tools, there is a version available that already includes built-in support for the RST data types. Get it here from the CI server.

You should also have a working RSB transport configuration. For more details on how to configure this, see the RSB manual. If you are unsure, for RSB 0.6. the recommended configuration would be to run one (!) Spread daemon on your host computer. The default Spread settings should be fine for this, so you can just start the Spread daemon like this:

$ spread -n localhost

Configure RSB on your robot (and all computers that want to get data via RSB from your robot) with the config file .config/rsb.conf like this

[transport.spread]
enabled = 1
host = <Computer IP running Spread>

[transport.inprocess]
enabled = 0

Note

Make sure you are connected to your robot via a fast physical connection (e.g. direct cable link or GBit switch, not via WiFi), otherwise you might not be able to capture all the data in realtime.

Preparing your robot and the environment

To enable streaming video and audio data from the robot, you first need to make sure that the NAOqi modules for sending images and audio buffers can use RSB. After stopping NAOqi on the robot (nao stop), you should set up the environment like this:

$ source rsb-interface/setup.sh

You also need to make sure that NAOqi knows that it should load the RSB A/V modules by including their full paths in the [user] section of /home/nao/naoqi/preferences/autoload.ini like this:

/home/nao/rsb-interface/lib/naoqi/librsbvideosender.so
/home/nao/rsb-interface/lib/naoqi/librsbaudiosender.so

To make starting the streaming easier, it is also recommended to add the ActionLauncher to the same file in the [python] section like this:

/home/nao/naoqi/bin/ActionLauncher

After all this, you have to restart NAOqi on the robot with this exact command line:

/usr/bin/naoqi-bin -v

Capturing audio and video data from the robot

Before you can capture audio or video data from the robot, you have to tell the modules on the robot to start sending data via RSB. If you have installed and enabled the ActionLauncher as detailed in the last section, the easiest way to do this is by tapping the rear tactile sensor on the head of the robot four times in a row. If it worked, the robot will respond by saying “Started RSB streaming” and by starting to send data over RSB. You could also automate this step with e.g. a Python script by calling the startStreaming() method of the RSBVideoSender and RSBAudioSender NAOqi modules.

If you want to record the audio data from NAO into a file, you can do so with the bag-record program like this:

$ bag-record -o naoaudio.tide spread://localhost:4803/nao/audio

This will write all the raw audio buffers recorded from the robots microphones into a file called naoaudio.tide in the current directory which you can later process and convert using other tools like bag-cat.

For video data, the corresponding command would be:

$ bag-record -o naovideo.tide spread://localhost:4803/nao/vision

If you wanted to capture all the RSB communication, including all audio and video data sent from the robot, you can also call bag-record like this:

$ bag-record -o everything.tide spread://localhost:4803/

For more details you can also take a look at the Recording section of this tutorial or the RSBag manual.

Examining what was recorded

There are several ways you can get an overview over the data captured in a .tide file.

You can examine the file using the bag-info program to see how many event s of what type are in the channel s collected in that log file, e.g.:

$ bag-info file.tide
File "file.tide"
Events  : 19,718
Start   : 2012-01-11T13:04:11.104469+01:00
End     : 2012-01-11T13:21:41.378410+01:00
Duration: 1050.273941
Channel "/nao/vision/top/:.rst.vision.Image"
  Type    : (RSB-EVENT .rst.vision.Image)
  Format  :
  Events  : 19,718
  Start   : 2012-01-11T13:04:11.104469+01:00
  End     : 2012-01-11T13:21:41.378410+01:00
  Duration: 1050.273941
  Rate    : 18.77414951495974

You can also use the very versatile bag-cat tool to e.g. take a look at the timestamps of all the individual event s (e.g. image frames or audio buffers) captured in a file:

bag-cat --style 'programmable/template :template "${create-unix-nsec}\\n"' file.tide)
1314539467177910000
1314539623750522000
...

See the --help-for=all option of bag-cat for more things you can include in templates like this.

Replaying captured data

To replay the captured data, e.g. to test or benchmark your modules while you are developing them, you can use the bag-play program like this:

$ bag-play naovideo.tide 'spread://localhost:4803'

This will play back the recorded video frames over RSB at a speed as close to the originally recorded timing as possible. You can receive them in your modules on the normal scope s (e.g. /nao/vision/0 for the first camera) that the live data from the robot also uses.

For more details on the options of the replay process (e.g. if you want to play back the data at a different speed than that with which it was originally recorded), see the Replaying Log Files section of this tutorial or the RSBag manual.

Converting captured audio and video data to different formats

To convert recorded audio data to a wave file, you can use the bag-cat tool together with a conversion script like this:

$ bag-cat --style 'programmable/script :script #p"export-audio-as-wav.lisp"' \
      naoaudio.tide                                                          \
      > output.wav

Note

The file export-audio-as-wav.lisp is a conversion script that you can download e.g. at this job on the CI server.

To convert the individual video frames recorded from NAO into timestamped PNG images, you can use a similar conversion routine:

$ bag-cat --style 'programmable/script :script #p"export-video-as-pngs.lisp"' naovideo.tide

This will create one PNG file per frame in the current folder.

Note

The same note as above applies on where to get the export-video-as-pngs.lisp script.

Warning

Converting a long recording into individual PNG images will create a very large amount of files!

Converting both audio and video to a common video container file is a bit more complicated because of the synchronization and conversion to a fixed frame rate that needs to be involved. One solution that exists is to use the rsb-gstreamer plugins for the GStreamer audio and video processing framework to set up a pipeline that will take care of the synchronization and allows to write a synchronized stream into a container format, which is playable by standard video players (VLC etc.).

To use this solution, first set up the video writing pipeline like this:

$ record_mp4.sh output.mp4 /nao/vision/0 /nao/audio/all

Note

You can also use record_ogv.sh (which will use the .ogv container format instead of the more common MP4 standard). Both scripts come with the source of the rsb-gstreamer project.

Then, in a second console, start playing back a file containing both recorded audio and video data like this:

$ bag-play -r 'fixed-rate :rate 30' everything.tide 'spread://localhost:4803'

You then need to wait until the second command finishes going through the whole recording (which will take quite some time), after which you can stop the first script by pressing Ctrl+C once. This should have written a synchronized video and audio file called output.mp4 in the current directory.

For more conversion options, take a look at other sections of this tutorial (e.g. Encoding into Sound Files, Encoding Frames into Individual Image Files and Encoding Audio and Video) and the RSBag manual.

Warning

Exporting video files this way assumes that the sound chunks received from NAO have no gaps. At the moment, this seems to be a serious issue with the current NAOqi release. A second way to export videos without sound has been implemented because of this. Have a look here: https://code.cor-lab.org/projects/rsb-video-writer/wiki