How to write a bidding agent - datacratic/rtbkit GitHub Wiki

In this document we'll write a simple fixed price bidding agent. This will be made up of three major components:

  • The agent's configuration which controls the type of bid requests that the router will forward to the agent.
  • The bidding logic which decides how much we want to bid on each spot.
  • The pacer which controls the rate at which we spend our budget.

The annotated source code for the full bidding agent that we will present in this example is available in rtbkit's repository.

Agent Configuration

In this section, we'll create an AgentConfig object which will hold basic information about our bidding agent as well as setup various filters which will ensure that the routers will only forward bid requests that we're interesting in.

Let's get started:

AgentConfig config;
void setConfig()
{
    config = AgentConfig();

Pretty straightforward but we had to start somewhere.

    config.account = {"a", "very", "long", "account", "name", "thingy"};

Here we set up the account using the new and fancy C++11 initialization list. Accounts are used to keep track of the budget associated with each campaign and they're organized in a hierarchy. How this hierarchy is setup is entirely up to the writer of the bidding agents. A typical setup would be <campaign>.<strategy> but, as you can see in the example, you can use whatever scheme best fits your setup.

A more lengthy description of accounts can be found in the banker's documentation.

    config.creatives.push_back(Creative(160, 600, "LeaderBoard"));
    config.creatives.push_back(Creative(300, 250, "BigBox"));
    config.creatives.push_back(Creative(728, 90,  "LeaderBoard"));

This bit sets up the format of the creatives that our agent is interested in bidding on. Any bid requests that don't contain at least one spot that fits one of these impression formats will be filtered out by the router and will never make it to our bidding agent. Less work for us, woo!

In addition to this mandatory filter, there are several other optional filters that can be specified in the agent's configuration:

  • hosts and urls
  • languages and locations
  • segments
  • exchanges
  • position relative to the page's fold
  • hour of the week
  • user partitions
  • augmentation tags (see below)
    AugmentationConfig augConfig("frequency-cap-ex");
    augConfig.required = true;
    augConfig.config = Json::Value(42);
    augConfig.filters.include.push_back("pass-frequency-cap-ex");
    config.addAugmentation(augConfig);

Next, we tell the router that we'd be interested in having our bid requests augmented by our frequency cap example and set the frequency cap ceiling to 42. Since we don't want to see bid requests that haven't been augmented or that have hit our frequency cap ceiling, we also tell the router to filter out any bid request that hasn't been augmented by the frequency cap augmenter and that don't have the augmentation tag set by the frequency cap example.

In this case, we use the augmenters output purely for filtering but it's still possible to retrieve information injected by an augmenter or while we're making our bid decision.

    doConfig(config.toJson());
}

Finally, we tell the world about our new shiny configuration using BiddingAgent's doConfig function. When called, doConfig will publish the agent's configuration to the AgentConfigurationService which will then repeat it to every AgentConfigurationListener which includes the router, the post-auction loop and any augmenters that have per agent configuration.

Note that an agent can change its configuration at any time by calling doConfig. This is useful to update certain filters like bidProbability or maxInFlight which regulates the volume of the bid request stream that makes it to the agent.

Bidding Logic

Now that we've told the router what kind of bid requests our agent is interested in, we can start bidding. Note that since all the nitty gritty filtering, budgeting and real time constraints are handled by the router, the agent can concentrate solely on its bidding logic. This makes our life that much more easier.

void bid(
    double timestamp,
    const Id & id,
    std::shared_ptr<RTBKIT::BidRequest> bidRequest,
    Bids bids,
    double timeLeftMs,
    const Json::Value & augmentations,
    WinCostModel const & wcm)
{

For our example, we'll shove the entire logic of the bidding agent within a bid function that will be called by the BiddingAgent class every time it receives a bid request from a router.

    for (Bid & bid : bids) {

The bids object we get in the BiddingAgent callback is used to make our bid. It contains a Bid objects for each of the available spot that match our agent's configuration and that we can bid on. Spots from the bid request that our agent can't bid on will not appear in this array.

        int availableCreative = bid.availableCreatives.front();
        (void) config.creatives[availableCreative];
        (void) bidRequest->spots[bid.spotIndex];

Each Bid object contains:

  • A set of creative indexes which indicates the creatives of our configuration that we can use to bid on the given spot.
  • A spot index which we can use to query the bid request for more details about the spot we're trying to bid on.

Additionally, we can peruse the BidRequest object and the augmentations json blob to gain additional data to inform our bid decision. For our example, we don't need fancy logic so we'll just pick the first available creative and continue on our merry way.

        bid.bid(availableCreative, USD_CPM(2));
    }

Here we register our bid with the Bid object by passing along the creative we wish to display and the bid price. To specify monetary values, we use the Amount class which is useful to prevent bankruptcy due to bad currency conversions or a mis-scaled value. For example, if we wanted extra precisions to calculate the bid price, we could use the MicroUSD_CPM struct instead of the USD_CPM struct and RTBKit will make sure that the value is properly rescaled when passed to the router.

    Json::Value metadata = 42;

If required, we can also attach some metadata to our bids which will be passed back to our bidding agent in the bid results and the post auction events.

    doBid(id, bids, metadata, wcm);
}

Moment of truth: time to place our bid with the router by calling BiddingAgent's doBid function. Once the bid is placed, we'll receive a reply from the router for each impressions that we bid on which will trigger one of the following BiddingAgent callbacks:

  • onWin: we won the auction for the impression.
  • onLoss: we lost the auction for the impression.
  • onNoBudget: our account doesn't have the required budget to place the bid.
  • onTooLate: the auction was sent back to the exchange before we could place our bid.
  • onDroppedBid: the router didn't receive a bid response for a given bid request.
  • onInvalidBid: our bid was incorectly setup.

If we won the auction, the post-auction loop may trigger these additional callbacks:

  • onImpression: our creative was shown to the user.
  • onClick: the user clicked on the creative.
  • onVisit: the user did an action after having clicked on the creative.

Note that the wcm parameter can be used to compute the real cost of the impression and augment the data attached to the WinCostModel object if necessary. More details are available here How to write a win cost model.

Pacer

We have one final missing piece for our bidding agent: the periodic allocation of budget also known as pacing. Pacing is a good idea because it allows us to distribute the budget for a campaign over the entire duration of the campaign instead of spending it all in the middle night. Budgets are managed by the master banker which uses the accounts of the various agents to transfer budgets from parent accounts to child accounts. Note that the router is in charge of ensuring that we never go over the allocated budget.

The following pace function will be called periodically (we'll see how a little later) to allocate a small portion of our budget which the agent can use to bid.

SlaveBudgetController budgetController;
bool accountSetup = false;
void pace()
{
    if (!accountSetup) {
        accountSetup = true;
        budgetController.addAccountSync(config.account);
    }

So the first step is the get the account information for our bidding agent from the master banker. We do this by initiallizing a SlaveBudgetController object which will act as a proxy to the master banker. This controller periodically communicates with the master banker to sync its budgets.

     budgetController.topupTransferSync(config.account, USD(1));
}

All that remains is to transfer an amount from the our parent's account into our agent's account and we're done. Simple as pie!

Putting It All Together

Almost done; all that's left is a little bit of glue code.

struct FixedPriceBiddingAgent : public BiddingAgent {

Let's create our agent's class which will house the setConfig, bid and pace function that we developed earlier. This class should either compose or derive the BiddingAgent class which handles the boiler-plate router protocol details.

    void init()
    {
        strictMode(false);
        onBidRequest = bind(
                &FixedPriceBiddingAgent::bid, this, _1, _2, _3, _4, _5, _6);

In the init function we initialize the various components that make up our service. We start by setting up the bidding agent's callbacks. The strictMode function call supresses errors when BiddingAgent receives a message for which there's no callback defined. Disabling these checks is useful when writting simple example agents or tests.

        budgetController.init(getServices()->config);
        budgetController.start();

Next we setup our proxy to the master banker used by our pacer.

        addPeriodic("FixedPriceBiddingAgent::pace", 10.0,
                [&] (uint64_t) { this->pace(); });

Here we exploit the fact that BiddingAgent is a MessageLoop to add a periodic event source which will call our pacer every 10 seconds.

        BiddingAgent::init();
    }

Finally, we make sure that BiddingAgent is also initialized.

    void start()
    {
        BiddingAgent::start();
        setConfig();
    }
} // struct FixedPricedBiddingAgent

In the start function we start the various message loops that compose our service. While we're at it, we also take the opportunity to send our configuration to the agent configuration service so that we can start receiving bid requests.

Note that I'm skipping the shutdown function because, as we'll see below, our service is not meant to be shutdown cleanly. In fact, the shutdown functions in the various RTBKit services are only ever usefull when writting tests and none of the production runners call shutdown. We prefer to close services by simply issuing a signal to the process which implies that all our services are crash resistant (or will be eventually) including the bidding agents.

int main(int argc, char** argv)
{

Home stretch!

    ServiceProxies serviceProxies;

RTBKit requires a Zookeeper instance to do service discovery and a Carbon instance to dump runtime metrics. These are accessed through a ServiceProxies object which acts as a proxy to these services. While this object can be constructed manually, it's easier to just setup a bootstrap.json file and let the object initalize itself. Note that the various services in ServiceProxies have default localized versions which can be used to write tests.

    FixedPriceBiddingAgent agent(serviceProxies, "fixed-price-agent-ex");
    agent.init();
    agent.start();
    while (true) this_thread::sleep_for(chrono::seconds(10));
}

We finish things up by instantiating our agent using the service proxies object we just created and a unique service name which will be used for service discovery through Zookeeper. All that's left is to initialize the agent, start it and put the main thread to sleep while the background message loop does its thing.

And that's it. We now have a fully functional fixed price bidding agent.

⚠️ **GitHub.com Fallback** ⚠️