Redis Augmentor - datacratic/rtbkit GitHub Wiki

The following page introduces RTBkit' Redis Augmentor, its architecture and how to use it.

As described in this page, an RTBKIT::Augmentor:

• Receives a series of bid requests and will add information to them in the form of tags or data. This information can be either per account or global to all accounts.
• Subscribes to the Agent Configuration Service to know the augmentation for each client(bidding agent)

The complete source code for our RedisAugmentor, is available in the rtbkit repository under the following 3 files:

• rtbkit/plugins/augmentor/redis_augmentor.h
• rtbkit/plugins/augmentor/redis_augmentor.cc
• rtbkit/plugins/testing/redis_augmentor_test.cc

This page describes how to write an Augmentor as an instance of the SyncAugmentor. We shall also assume that [accounts](https://github.com/rtbkit/rtbkit/blob/master/rtbkit/common/account_key. h) follow the <campaign>.<strategy> pattern. For more information on accounts, see the banker's documentation.

The general structure of keys which our RedisAugmentor loooks for, is

 prefix:path:value

where prefix is the arbitrary string "RTBkit:aug", and value the value of the attribute of a bid request Json object, as described by a path, using the following syntax:

 - "." => root node
 - ".[n]" => elements at index 'n' of root node (an array value)
 - ".name" => member named 'name' of root node (an object value)
 - ".name1.name2.name3"
 - ".[0][1][2].name1[3]"
 - ".%" => member name is provided as parameter
 - ".[%]" => index is provied as parameter

In our case, the root node needs not to be provided in the key, as it obviously is the bid request in question. For a given agent, the keys are fetched from its associated AugmentationConfig configuration object, as path into the bid request passed along with the AugmentationRequest.

(from rtbkit/plugins/testing/redis_augmentor_test.cc)

The following bid request:

 {"id":"85885bb0-b91b-11e2-c4cf-7fba90171555","timestamp":1368153863.008756,"isTest":false,"url":"http://myonlinearcade.com/","ipAddress":"166.13.20.21","userAgent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101       Firefox/20.0","language":"fr","protocolVersion":"0.3","exchange":"appnexus","provider":"appnexus","winSurcharges":{"surcharge":{"USD/1M":50}},"winSurchageMicros":{"surcharge":{"USD/1M":50}},"location":{"countryCode":"CA",          "regionCode":"QC","cityName":"Laval","postalCode":"0","dma":0,"timezoneOffsetMinutes":-1},"segments":{"appnexus":["memberId1357"],"browser":["Mozilla Firefox"],"device_type":["Computer"],"os":["Microsoft Windows 7"]},"userIds":    {"an":"5273283952213481305","xchg":"5273283952213481305"},"imp":[{"id":"156331815539876686","banner":{"w":728,"h":90},"formats":["728x90"]}],"spots":[{"id":"156331815539876686","banner":{"w":728,"h":90},"formats":["728x90"]}]};

the following commands issued on our Redis instance:

 set  "RTBkit:aug:winSurcharges.surcharge.USD/1M:50"  123.45
 set  "RTBkit:aug:id:85885bb0-b91b-11e2-c4cf-7fba90171555" 9876
 set  "RTBkit:aug:url:http://myonlinearcade.com/" "JSCRIPT"

and two agents configured as following:

    TestAgent agent1(proxies, "bob-the-agent");
    agent1.init();
    agent1.start();
    agent1.configure();
    {
        AugmentationConfig aug_conf;
        aug_conf.name = "redis";
        aug_conf.required = true;
        auto& v =  aug_conf.config;
        Json::Value av(Json::arrayValue);
        av.append("winSurcharges.surcharge.USD/1M");
        av.append("id");
        av.append("exchange");
        av.append("foo.bar"); // not found
        v["aug-list"] = av;
        v["aug-prefix"] = "RTBkit:aug";
        agent1.config.addAugmentation(aug_conf);
    }
    agent1.doConfig (agent1.config);

and:

    TestAgent agent2(proxies, "alice-the-agent");
    agent2.config.account = {"aliceCampaign", "aliceStrategy"};
    agent2.init();
    agent2.start();
    agent2.doConfig(agent2.config);
    {
        AugmentationConfig aug_conf;
        aug_conf.name = "redis";
        aug_conf.required = true;
        auto& v =  aug_conf.config;
        Json::Value av(Json::arrayValue);
        av.append("id");
        av.append("url"); // not found
        v["aug-list"] = av;
        v["aug-prefix"] = "RTBkit:aug";
        agent2.config.addAugmentation(aug_conf);
    }
    agent2.doConfig (agent2.config);

then our RedisAugmentor should produce the following AugmentationList:

[{"account":["aliceCampaign","aliceStrategy"],"augmentation":{"data":{"RTBkit:aug:id:85885bb0-b91b-11e2-c4cf-7fba90171555":"9876","RTBkit:aug:url:http://myonlinearcade.com/":"JSCRIPT"}}},{"account":["testCampaign","testStrategy"], "augmentation":{"data":{"RTBkit:aug:id:85885bb0-b91b-11e2-c4cf-7fba90171555":"9876","RTBkit:aug:winSurcharges.surcharge.USD/1M:50":"123"}}}]

RTBkit's Redis Augmentor is a AsyncAugmentor, which overrides its

void
RedisAugmentor::
onRequest(const AugmentationRequest & request, SendResponseCB sendResponse) ;/* override */;

is composed of a:

RTBKIT::AgentConfigurationListener agent_config_;

and aggregates or is composed of a

std::shared_ptr<Redis::AsyncConnection> redis_ ;

Given our RTBKIT::AugmentationList, a SendResponseCB is defined as following:

typedef std::function<void (const AugmentationList &)> SendResponseCB;

Redis::AsyncConnection is a thin layer around hiredis, which as far as we are concerned, provides an asynchronous, object oriented API around a physical connection to a Redis instance, with convenient timeout managements, as well as commands pipelining (which are both used in our case). Redis::AsyncConnection has its own events loop, on which it services augmentation callbacks.

It is also important to realise that the inherited AsyncAugmentor, is composed of (amongst other things) a thread pool, which it uses in order to service inbound augmentation requests. The size of this servicing thread poll is a parameter of our base Redis::AsyncConnection; and also that the life cycle of a bid request consists in a sequence of the following events:

1. The exchange sends a bid request to the router where it is parsed by the exchange connector
2. The router relays the bid request to the Augmenter
3. The augmenter returns augmentations("+") to the router
4. The router sends augmented bid request("BR+") to the bidding agent

In the case of our RedisAugmentor, the sequence 2-3 above, can be further magnified as following:

1. The router relays the bid request to the RedisAugmentor
2. For each agent associated with this bid request, the RedisAugmentor tries to fetch an <augmentations>.<redis>.<config>.<aug-list> JSON::ArrayValue from each associated AgentConfigEntry, eventuelly upserting an ordered map jobs indexed by Redis keys, mapping to a set of AccountKey (<campaign>.<strategy>).
3. Out of the keys indexing jobs, the RedisAugmentor then builds a batch of Redis commands, and submits it to its Redis::AsyncConnection member, along with a redis callback defined as a C++11 lambda capturing our job map by value, as well as the SendResponseCB passed as a parameter to the call. Note that these pipelined redis commands are submited with a 4 milliseconds timeout.
4. Upon being called back on the redis thread, our lambda submitted above then constructs an AugmentationList according to both job and the Redis::Results passed as a callback argument by our Redis::AsyncConnection and calls SendResponseCB to pass it back to the calling AugmentorLoop run by our router.
5. The RedisAugmentor returns augmentations("+") to the router

Voilà.