Writing Converters

Basic Concept

Warning

This example has been written with simplicity in mind and should not be used as a blueprint for production code. In particular:

  • It does not handle machine Endianess properly
  • It does not handle machine word sizes properly
  • It leaks memory
  • It does not validate any data

Note

In many practical cases, some of these problems (and much of the manual work) can be avoided by using an IDL-specification in combination with code generation and generic converters as described in the google protocol buffers documentation.

In RSB, converters are used to serialize and deserialize programming-language objects for transportation (e.g. over a network connection). RSB comes with converters for the fundamental types listed here. However, in some use-cases it is necessary to use additional converters for domain-specific data types and/or serialization mechanisms.

This example demonstrates how to add such converters to RSB using the running example of a converter for a fictional SimpleImage data type.

In order to implement a new converter, the following information is required:

  • To/from which wire type will the converter serialize/deserialize? In our example, the wire type is an array of bytes (or more formally an array of octets) which is represented in C++ using std::string.

  • Which data type or (data types) will be handled by the converter? The struct @SimpleImage@ in our example (please note that the data type is identified using a string for comparison, not the class itself).

  • What is the wire schema of the converter? In our example, we use the following ad-hoc wire schema:

    Name

    simple-image

    Binary layout

    One integer encoding the image width, one integer encoding the image height, width x height bytes for the image data.

TODO

converter_tutorial::SimpleImage Domain Class

The domain data type:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#pragma once

namespace converter_tutorial {

struct SimpleImage {
    int width;
    int height;
    unsigned char* data;
};

}

converter_tutorial::SimpleImageConverter Class

For the actual converter implementation, four things are needed:

  1. The C++ representation of the wire type has to be passed to the rsb::converter::Converter interface as a template parameter.
  2. The wire schema and data type name have to be passed to the rsb::converter::Converter constructor.
  3. The rsb::converter::Converter::serialize method has to be implemented.
  4. The rsb::converter::Converter::deserialize method has to be implemented.

A naive and incomplete implementation can be found in the following listings:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
#pragma once

#include <rsb/converter/Converter.h>

namespace converter_tutorial {

/**
 * A simple converter for the SimpleImage struct. For educational use only.
 */
class SimpleImageConverter: public rsb::converter::Converter<std::string> {
public:
    SimpleImageConverter();

    std::string serialize(const rsb::AnnotatedData& data,
            std::string& wire);

    rsb::AnnotatedData deserialize(const std::string& wireSchema,
            const std::string& wire);
};

}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
#include "SimpleImageConverter.h"

#include "SimpleImage.h"

using namespace std;

using namespace boost;

using namespace rsb;
using namespace rsb::converter;

namespace converter_tutorial {

// We have to pass two arguments to the base-class constructor:
// 1. The data-type
// 2. The wire-schema
//
// Note: this could also be written as
// Converter<string>("simple-image", RSB_TYPE_TAG(SimpleImage))
// to infer the "string" name of the data-type using RTTI.
SimpleImageConverter::SimpleImageConverter() :
    Converter<string> ("converter_tutorial::SimpleImage", "simple-image", true) {
}

string SimpleImageConverter::serialize(const AnnotatedData& data, string& wire) {
    // Ensure that DATA actually holds a datum of the data-type we
    // expect.
    assert(data.first == getDataType()); // this->getDataType() == "converter_tutorial::SimpleImage"

    // Force conversion to the expected data-type.
    //
    // NOTE: a dynamic_pointer_cast cannot be used from void*
    boost::shared_ptr<const SimpleImage> image =
            static_pointer_cast<const SimpleImage> (data.second);

    // Store the content of IMAGE in WIRE according to the selected
    // binary layout.
    //
    // NOTE: do not use this kind of "serialization" for any real code.
    int numPixels = image->width * image->height;
    wire.resize(4 + 4 + numPixels);
    copy((char*) &image->width, ((char*) &image->width) + 4, wire.begin());
    copy((char*) &image->height, ((char*) &image->height) + 4, wire.begin() + 4);
    copy((char*) image->data, ((char*) image->data) + numPixels,
            wire.begin() + 8);

    // Return the wire-schema of the serialized representation in
    // WIRE.
    return getWireSchema(); // this->getWireSchema() == "simple-image"
}

AnnotatedData SimpleImageConverter::deserialize(const string& wireSchema,
        const string& wire) {
    // Ensure that WIRE uses the expected wire-schema.
    assert(wireSchema == getWireSchema()); // this->getWireSchema() == "simple-image"

    // Allocate a new SimpleImage object and set its data members from
    // the content of WIRE.
    //
    // NOTE: do not use this kind of "deserialization" for any real
    // code.
    SimpleImage* image = new SimpleImage();
    image->width = *((int*) &*wire.begin());
    image->height = *((int*) &*(wire.begin() + 4));
    image->data = new unsigned char[image->width * image->height];
    copy(wire.begin() + 8, wire.begin() + 8 + image->width * image->height,
            image->data);

    // Return (a shared_ptr to) the constructed object along with its
    // data-type.
    return make_pair(getDataType(), boost::shared_ptr<SimpleImage> (image));
}

}

Using the Converter

A simple program that demonstrates the use of our SimpleImageConverter can be found in

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#include <boost/shared_ptr.hpp>

#include <rsb/Factory.h>
#include <rsb/converter/Repository.h>

#include "SimpleImage.h"
#include "SimpleImageConverter.h"

using namespace boost;

using namespace rsb;
using namespace rsb::converter;

using namespace converter_tutorial;

int main() {
    // Register our converter within the collection of converters for
    // the string wire-type (which is used for arrays of octets in
    // C++).
    //
    // Try senderNoConverter.cpp to see what happens, if the converter
    // is not registered.
    shared_ptr<SimpleImageConverter> converter(new SimpleImageConverter());
    converterRepository<std::string>()->registerConverter(converter);

    // Create an Informer object that is parametrized with the
    // data-type SimpleImage.
    Informer<SimpleImage>::Ptr informer =
            getFactory().createInformer<SimpleImage> (
                    Scope("/tutorial/converter"));

    // Construct and send a SimpleImage object.
    shared_ptr<SimpleImage> image(new SimpleImage());
    image->width = 10;
    image->height = 10;
    image->data = new unsigned char[100];
    informer->publish(image);

    return EXIT_SUCCESS;
}

A similar program in which the registration of the converter is missing can be found in

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#include <boost/shared_ptr.hpp>

#include <rsb/Factory.h>

#include "SimpleImage.h"

using namespace boost;

using namespace rsb;

using namespace converter_tutorial;

// This program demonstrates the effect of using a data-type for which
// no converter is available (or at least not registered).
int main() {
    Informer<SimpleImage>::Ptr informer =
            getFactory().createInformer<SimpleImage> (
                    Scope("/tutorial/converter"));

    shared_ptr<SimpleImage> image(new SimpleImage());
    image->width = 10;
    image->height = 10;
    image->data = new unsigned char[100];
    informer->publish(image);

    return EXIT_SUCCESS;
}

This second program serves the purpose of familiarizing you with the “missing-converter” error message, that you will encounter sooner or later ;)

TODO
TODO

Using Protocol Buffer Types in Converters for Custom Domain Types

Imagine you have a custom domain type (in this example called FooBar) in your software, which you want to use with RSB. For being able to directly send and receive this type, you need a new converter, which handles this type. Moreover, the generated data to be sent over the wire should be compatible with a data type defined using google protocol buffers (e.g. from the RST library). For this example, we assume the protocol buffers type is called rst.foo.Bar. To implement a converter matching these assumptions, adapt the following explanations according to your actual data types:

TODO

Create a header file called FooBarConverter.h and fill it with the following contents:

#include <rsb/converter/Converter.h>
#include <rsb/converter/ProtocolBufferConverter.h>

#include <rst/foo/Bar.pb.h>

class FooBarConverter: public rsb::converter::Converter<std::string> {
public:

    FooBarConverter();
    virtual ~FooBarConverter();

    std::string getWireSchema() const;

    std::string serialize(const rsb::AnnotatedData &data, std::string &wire);
    rsb::AnnotatedData deserialize(const std::string &wireType,
            const std::string &wire);

private:

    rsb::converter::ProtocolBufferConverter<rst::foo::Bar> converter;

};

Afterwards, create an implementation file called FooBarConverter.cpp along the following lines:

#include "FooBarConverter.h"

#include <rsc/runtime/TypeStringTools.h>

using namespace std;

FooBarConverter::FooBarConverter() :
        rsb::converter::Converter<string>("unused", RSB_TYPE_TAG(FooBar)) {
}

FooBarConverter::~FooBarConverter() {
}

string FooBarConverter::getWireSchema() const {
    return converter.getWireSchema();
}

string FooBarConverter::serialize(const rsb::AnnotatedData &data,
        string &wire) {

    assert(data.first == this->getDataType());

    boost::shared_ptr<FooBar> source = boost::static_pointer_cast<FooBar>(data.second);
    boost::shared_ptr<rst::foo::Bar> dest(new rst::foo::Bar());

    // TODO 1: extract data from domain type and fill it into the protobuf message

    return converter.serialize(
            rsb::AnnotatedData(rsc::runtime::typeName<rst::foo::Bar>(), dest),
            wire);

}

rsb::AnnotatedData FooBarConverter::deserialize(
        const std::string &wireType, const std::string &wire) {

    boost::shared_ptr<rst::foo::Bar> source =
            boost::static_pointer_cast<rst::foo::Bar>(
                    converter.deserialize(wireType, wire).second);

    boost::shared_ptr<FooBar> dest(new FooBar);

    // TODO 2: Extract data from the protobuf message and fill it into your domain type

    return rsb::AnnotatedData(getDataType(), dest);

}

Your custom conversion logic of how to convert between the Protocol Buffer messages and your custom data types needs to be implemented at the two places indicated with TODO comments. The google protocol buffers documentation explains how to access the values from in the protocol buffer message.

Note

Please use a reasonable namespace for you actual converter.

TODO
TODO