How to write a bidding agent - datacratic/rtbkit GitHub Wiki
In this document we'll write a simple fixed price bidding agent. This will be made up of three major components:
- The agent's configuration which controls the type of bid requests that the router will forward to the agent.
- The bidding logic which decides how much we want to bid on each spot.
- The pacer which controls the rate at which we spend our budget.
The annotated source code for the full bidding agent that we will present in this example is available in rtbkit's repository.
In this section, we'll create an AgentConfig
object which will hold basic information about our bidding agent as well as setup various filters which will ensure that the routers will only forward bid requests that we're interesting in.
Let's get started:
AgentConfig config;
void setConfig()
{
config = AgentConfig();
Pretty straightforward but we had to start somewhere.
config.account = {"a", "very", "long", "account", "name", "thingy"};
Here we set up the account using the new and fancy C++11 initialization list. Accounts are used to keep track of the budget associated with each campaign and they're organized in a hierarchy. How this hierarchy is setup is entirely up to the writer of the bidding agents. A typical setup would be <campaign>.<strategy>
but, as you can see in the example, you can use whatever scheme best fits your setup.
A more lengthy description of accounts can be found in the banker's documentation.
config.creatives.push_back(Creative(160, 600, "LeaderBoard"));
config.creatives.push_back(Creative(300, 250, "BigBox"));
config.creatives.push_back(Creative(728, 90, "LeaderBoard"));
This bit sets up the format of the creatives that our agent is interested in bidding on. Any bid requests that don't contain at least one spot that fits one of these impression formats will be filtered out by the router and will never make it to our bidding agent. Less work for us, woo!
In addition to this mandatory filter, there are several other optional filters that can be specified in the agent's configuration:
- hosts and urls
- languages and locations
- segments
- exchanges
- position relative to the page's fold
- hour of the week
- user partitions
- augmentation tags (see below)
AugmentationConfig augConfig("frequency-cap-ex");
augConfig.required = true;
augConfig.config = Json::Value(42);
augConfig.filters.include.push_back("pass-frequency-cap-ex");
config.addAugmentation(augConfig);
Next, we tell the router that we'd be interested in having our bid requests augmented by our frequency cap example and set the frequency cap ceiling to 42
. Since we don't want to see bid requests that haven't been augmented or that have hit our frequency cap ceiling, we also tell the router to filter out any bid request that hasn't been augmented by the frequency cap augmenter and that don't have the augmentation tag set by the frequency cap example.
In this case, we use the augmenters output purely for filtering but it's still possible to retrieve information injected by an augmenter or while we're making our bid decision.
doConfig(config.toJson());
}
Finally, we tell the world about our new shiny configuration using BiddingAgent
's doConfig
function. When called, doConfig
will publish the agent's configuration to the AgentConfigurationService
which will then repeat it to every AgentConfigurationListener
which includes the router, the post-auction loop and any augmenters that have per agent configuration.
Note that an agent can change its configuration at any time by calling doConfig
. This is useful to update certain filters like bidProbability
or maxInFlight
which regulates the volume of the bid request stream that makes it to the agent.
Now that we've told the router what kind of bid requests our agent is interested in, we can start bidding. Note that since all the nitty gritty filtering, budgeting and real time constraints are handled by the router, the agent can concentrate solely on its bidding logic. This makes our life that much more easier.
void bid(
double timestamp,
const Id & id,
std::shared_ptr<RTBKIT::BidRequest> bidRequest,
Bids bids,
double timeLeftMs,
const Json::Value & augmentations,
WinCostModel const & wcm)
{
For our example, we'll shove the entire logic of the bidding agent within a bid
function that will be called by the BiddingAgent
class every time it receives a bid request from a router.
for (Bid & bid : bids) {
The bids
object we get in the BiddingAgent
callback is used to make our bid. It contains a Bid
objects for each of the available spot that match our agent's configuration and that we can bid on. Spots from the bid request that our agent can't bid on will not appear in this array.
int availableCreative = bid.availableCreatives.front();
(void) config.creatives[availableCreative];
(void) bidRequest->spots[bid.spotIndex];
Each Bid
object contains:
- A set of creative indexes which indicates the creatives of our configuration that we can use to bid on the given spot.
- A spot index which we can use to query the bid request for more details about the spot we're trying to bid on.
Additionally, we can peruse the BidRequest
object and the augmentations json blob to gain additional data to inform our bid decision. For our example, we don't need fancy logic so we'll just pick the first available creative and continue on our merry way.
bid.bid(availableCreative, USD_CPM(2));
}
Here we register our bid with the Bid
object by passing along the creative we wish to display and the bid price. To specify monetary values, we use the Amount
class which is useful to prevent bankruptcy due to bad currency conversions or a mis-scaled value. For example, if we wanted extra precisions to calculate the bid price, we could use the MicroUSD_CPM
struct instead of the USD_CPM
struct and RTBKit will make sure that the value is properly rescaled when passed to the router.
Json::Value metadata = 42;
If required, we can also attach some metadata to our bids which will be passed back to our bidding agent in the bid results and the post auction events.
doBid(id, bids, metadata, wcm);
}
Moment of truth: time to place our bid with the router by calling BiddingAgent
's doBid
function. Once the bid is placed, we'll receive a reply from the router for each impressions that we bid on which will trigger one of the following BiddingAgent
callbacks:
-
onWin
: we won the auction for the impression. -
onLoss
: we lost the auction for the impression. -
onNoBudget
: our account doesn't have the required budget to place the bid. -
onTooLate
: the auction was sent back to the exchange before we could place our bid. -
onDroppedBid
: the router didn't receive a bid response for a given bid request. -
onInvalidBid
: our bid was incorectly setup.
If we won the auction, the post-auction loop may trigger these additional callbacks:
-
onImpression
: our creative was shown to the user. -
onClick
: the user clicked on the creative. -
onVisit
: the user did an action after having clicked on the creative.
Note that the wcm
parameter can be used to compute the real cost of the impression and augment the data attached to the WinCostModel
object if necessary. More details are available here How to write a win cost model.
We have one final missing piece for our bidding agent: the periodic allocation of budget also known as pacing. Pacing is a good idea because it allows us to distribute the budget for a campaign over the entire duration of the campaign instead of spending it all in the middle night. Budgets are managed by the master banker which uses the accounts of the various agents to transfer budgets from parent accounts to child accounts. Note that the router is in charge of ensuring that we never go over the allocated budget.
The following pace
function will be called periodically (we'll see how a little later) to allocate a small portion of our budget which the agent can use to bid.
SlaveBudgetController budgetController;
bool accountSetup = false;
void pace()
{
if (!accountSetup) {
accountSetup = true;
budgetController.addAccountSync(config.account);
}
So the first step is the get the account information for our bidding agent from the master banker. We do this by initiallizing a SlaveBudgetController
object which will act as a proxy to the master banker. This controller periodically communicates with the master banker to sync its budgets.
budgetController.topupTransferSync(config.account, USD(1));
}
All that remains is to transfer an amount from the our parent's account into our agent's account and we're done. Simple as pie!
Almost done; all that's left is a little bit of glue code.
struct FixedPriceBiddingAgent : public BiddingAgent {
Let's create our agent's class which will house the setConfig
, bid
and pace
function that we developed earlier. This class should either compose or derive the BiddingAgent
class which handles the boiler-plate router protocol details.
void init()
{
strictMode(false);
onBidRequest = bind(
&FixedPriceBiddingAgent::bid, this, _1, _2, _3, _4, _5, _6);
In the init
function we initialize the various components that make up our service. We start by setting up the bidding agent's callbacks. The strictMode
function call supresses errors when BiddingAgent
receives a message for which there's no callback defined. Disabling these checks is useful when writting simple example agents or tests.
budgetController.init(getServices()->config);
budgetController.start();
Next we setup our proxy to the master banker used by our pacer.
addPeriodic("FixedPriceBiddingAgent::pace", 10.0,
[&] (uint64_t) { this->pace(); });
Here we exploit the fact that BiddingAgent
is a MessageLoop
to add a periodic event source which will call our pacer every 10 seconds.
BiddingAgent::init();
}
Finally, we make sure that BiddingAgent
is also initialized.
void start()
{
BiddingAgent::start();
setConfig();
}
} // struct FixedPricedBiddingAgent
In the start
function we start the various message loops that compose our service. While we're at it, we also take the opportunity to send our configuration to the agent configuration service so that we can start receiving bid requests.
Note that I'm skipping the shutdown
function because, as we'll see below, our service is not meant to be shutdown cleanly. In fact, the shutdown
functions in the various RTBKit services are only ever usefull when writting tests and none of the production runners call shutdown
. We prefer to close services by simply issuing a signal to the process which implies that all our services are crash resistant (or will be eventually) including the bidding agents.
int main(int argc, char** argv)
{
Home stretch!
ServiceProxies serviceProxies;
RTBKit requires a Zookeeper instance to do service discovery and a Carbon instance to dump runtime metrics. These are accessed through a ServiceProxies
object which acts as a proxy to these services. While this object can be constructed manually, it's easier to just setup a bootstrap.json
file and let the object initalize itself. Note that the various services in ServiceProxies
have default localized versions which can be used to write tests.
FixedPriceBiddingAgent agent(serviceProxies, "fixed-price-agent-ex");
agent.init();
agent.start();
while (true) this_thread::sleep_for(chrono::seconds(10));
}
We finish things up by instantiating our agent using the service proxies object we just created and a unique service name which will be used for service discovery through Zookeeper. All that's left is to initialize the agent, start it and put the main thread to sleep while the background message loop does its thing.
And that's it. We now have a fully functional fixed price bidding agent.