Fetch Once, Replay Often: Teaching a Waterfall Auction to Speak Bidding

Our largest legacy demand partner rebuilt its infrastructure around real-time bidding and stopped accepting duplicate requests for the same impression opportunity. Bids from a non-compliant caller would be penalized or rejected.

That is a reasonable thing for a buyer to ask for. It was also, for us, structurally impossible. The legacy waterfall did not merely happen to send duplicate requests — duplicate requests were the mechanism. Complying meant rewriting how the thing worked, on a deadline set outside the company, and the demand channel was what was at risk if we missed it. This was never an efficiency project.

What made it interesting is that the waterfall was already on its way out. Mobile advertising was midway through the migration from waterfall to real-time bidding, and the legacy path was visibly the losing side of that. So the question was not “how do we fix the waterfall.” It was “how much is it worth building for a system the industry is in the process of leaving.”

What a waterfall actually is

A waterfall sells one impression by asking the same question repeatedly at descending prices. Start at a $100 CPM tier and ask every partner in turn. Nobody bites, drop to $50 and ask again. Then $30, $15, $12, $8, and on down to a penny, at which point somebody will take it.

Every rung of that ladder was a fresh request. One impression opportunity, a dozen or more auctions, each one stateless and unaware of the others. That is what made it non-compliant, and it is also why it was expensive in three separate ways at once: egress on every duplicate request, compute on every duplicate auction, and wall-clock latency, because the rungs ran in sequence and each one had to wait for the last.

The latency point deserves emphasis, because it is where the public spec makes the problem concrete. Unity publishes its Exchange integration docs, and they state the bidder deadline plainly: a bidder gets 200 milliseconds to return a bid or an explicit no-bid. That is a per-request budget. A twelve-rung waterfall does not get 200ms; it gets 200ms twelve times over, in series, while a player waits to see an ad.

The field the whole story turns on

Unity’s Exchange is documented in public as an OpenRTB 2.6 implementation, and that shrinks the disclosure question here to almost nothing: the mechanics below are published partner documentation, not internal detail. All auctions are first-price (at: 1), the minimum CPM the seller will accept travels in imp.bidfloor as a float in USD, and wins are confirmed through the nurl win-notice URL with an ${AUCTION_PRICE} macro the exchange substitutes at serve time.

Here is a trimmed request, and the one field to look at is bidfloor:

{
  "id": "9f2a1c34-7b0e-4c1a-9f31-2d6b8e5a0c77",
  "at": 1,
  "tmax": 200,
  "imp": [
    {
      "id": "1",
      "bidfloor": 12.0,
      "bidfloorcur": "USD",
      "instl": 1,
      "video": {
        "mimes": ["video/mp4"],
        "minduration": 5,
        "maxduration": 30,
        "w": 1080,
        "h": 1920
      }
    }
  ],
  "app": { "bundle": "com.example.racing", "publisher": { "id": "pub-4417" } },
  "device": { "os": "iOS", "osv": "18.2", "lmt": 1 },
  "regs": { "gdpr": 0 }
}

bidfloor: 12.0 is the seller saying twelve dollars CPM or no deal. In the waterfall, a partner did not see one of these per impression. They saw a sequence: the same opportunity arriving at 100.0, then 50.0, then 30.0, then 15.0, then 12.0. And because the sequence was observable, it was information. A buyer could infer where the seller’s real reservation price sat, which rung tended to clear, and how much room there was underneath the one in front of them.

Consolidation deleted that sequence. The field survives — a consolidated request still carries a bidfloor — but the ladder it used to encode, and the hand-set publisher number sitting in it, do not. I did not appreciate up front how much of the revenue depended on that ladder being visible. More on that below, because it turned into the largest piece of follow-on work.

Fetch once, replay often

The architecture change is one sentence: stop treating each rung as an independent auction, and start treating the whole ladder as one stateful pass over a single set of bids.

Interception happens at the very first request of a waterfall. Instead of calling partners sequentially, once per descending tier, that first request triggers a parallel pre-fetch — one bid request fanned out to every programmatic partner at once. The responses land in a low-latency key-value store, and every tier below reads from there instead of going back out. One outbound fan-out, N cache-served tiers.

The ladder does the pricing

Here is the part I find genuinely elegant, and it is the piece most people expect to be a model and are surprised to hear is not.

A waterfall tier is a fixed price — a number the publisher and the partner agreed on in advance. A programmatic bid is a variable price — whatever the buyer decided this impression was worth, arriving fresh each time. Bridging the two means deciding which fixed slot a variable price belongs in.

The ladder answers that for free. Because tiers are ordered from highest price down, walking them in order with a bid already in hand and stopping at the first tier the bid clears lands you, by construction, on the highest-priced tier that bid can satisfy. There is nothing to estimate and no model to train or defend. The tier that maximizes revenue on a cleared bid is simply the first one you reach, so the optimization is a property of how the data is ordered.

In code the resolve path is unremarkable, which is the point — the hard part was everything that had been leaning on the old shape, not the loop itself:

// Resolve sells one impression opportunity. One parallel pre-fetch, then the
// ladder is a local walk over the bids that came back.
func (a *Auction) Resolve(ctx context.Context, opp Opportunity) (*Award, error) {
	// Keyed on the session, so these bids are addressable only from inside
	// the waterfall that fetched them.
	cached, err := a.bids.ForSession(ctx, opp.SessionKey)
	if err != nil {
		return nil, err
	}

	// Tiers are ordered by price, highest first. That ordering is the entire
	// pricing decision: the first tier a bid clears is the highest-priced
	// tier it could satisfy, so there is nothing to estimate.
	for _, tier := range opp.Tiers {
		bid, ok := cached.ClearsAt(tier.CPM)
		if !ok {
			continue // nothing values this opportunity that highly
		}

		// Short-circuit. Every tier below is cheaper, so none can pay more,
		// and none of them are walked.
		return a.award(opp, tier, bid), nil
	}

	return nil, ErrNoFill
}

That single return is doing two jobs at once. It is the pricing decision, and it is the request reduction — the tiers that never get walked are exactly the requests that never get sent. The cost saving and the correct price come out of the same line.

What a cached bid is allowed to promise

The sharpest question I get about this design, and the one I would ask first if someone described it to me: when does a cached bid go stale?

It doesn’t, because it is never given the chance to. A cached bid here is not a general-purpose cache entry with a lifetime. It is scoped to one impression opportunity for one user session, and the key is what enforces that: entries are keyed on a salted hash of the session, so nothing outside that session can even address them. A bid can only be replayed into the waterfall that fetched it, because no other caller can compute where it lives.

That is a better answer than a TTL would be, and worth dwelling on for a second. A time-based bound is a guess about how long a bid stays true, and guesses can be wrong in both directions — too long and you serve against a bid the buyer would no longer stand behind, too short and you throw away demand you already paid a round trip for. A structural bound is not a guess. The reuse window is however long one client takes to descend one ladder, and the question stops being “how stale is this bid” and becomes “is this still the opportunity the bid was fetched for,” which the key answers definitionally.

The settlement path is the other half. Because wins confirm through nurl at the price that actually cleared, the buyer is billed against a serve that happened rather than against a bid we were holding.

Three things consolidation broke

The core change took a fraction of the effort. Consolidation quietly broke three other things, none of which was obvious when the mandate landed.

Partners were pricing against the floors

This is the big one, and it is the one I would flag to anyone doing similar work: the floors were not just prices, they were the pricing signal.

Partners had spent years tuning valuation models against an observable ladder. Take it away and they are bidding into the dark, and a buyer bidding into the dark bids low, because low is the safe direction to be wrong in. Compliance achieved, revenue down. That is not a win.

So I built a pricing guidance module. Instead of a static, hand-set publisher floor, the consolidated request carries a computed target floor whose objective is a single thing: maximize the probability that a partner’s bid actually wins. Worth being precise here, because it is the detail people assume wrong — there is no margin term in that objective. It is a win probability estimate, and it points outbound, to partners, as the replacement for the number consolidation deleted. It has nothing to do with choosing which tier to replay a bid into. Those are two different mechanisms in two different directions, and fusing them describes a system I did not build.

Three layers of signal feed it, plus a knob: a baseline of historical win-floors at the P50 and P90 percentiles sliced by geo and app, a real-time proxy for the value of the user drawn from our own direct bidders, an early ML forecast of how partners were likely to behave, and an alpha parameter for tuning the blend under experiment.

This module is the piece of the project I am proudest of, and not because of what it did for the waterfall. It showed that predictive floor pricing worked at production scale, and it became the architectural blueprint for the ML pricing engine that came after it — a system that outlived the bridge it was built to hold up.

Pre-fetching invents bids that never happened

Pre-fetching created a class of bid that had not existed before: a real bid, from a real partner, that might never be used at all. Most of the pre-fetched bids in any given opportunity lose, and some belong to tiers the short-circuit means we never walk.

That collides with something partners depend on. Auction-outcome metadata is not bookkeeping to them — they feed it back into their valuation models, so the notification path is an input to the pricing behavior of the counterparties whose money you are trying to attract. In a stateless waterfall this came for free, because each tier ran its own auction and fired its own notifications. Going stateful deleted that, and there was no longer a per-tier auction to report on; the outcome only exists once the whole ladder has resolved.

The naive fix is worse than doing nothing. Firing a loss notification for every pre-fetched bid would tell a partner it lost an auction that, from its side, never took place. Phantom losses in a partner’s training data are not neutral noise — they are wrong in the specific direction that teaches the partner to bid down, which is precisely the outcome the whole project existed to avoid.

So notifications became a reconciliation problem rather than a fire-and-forget one. Every pre-fetched bid is buffered in a durable append-only log. When an auction concludes and emits a winner, a consumer matches that outcome against the buffer for that session and fires loss notifications only for the bids that genuinely competed and genuinely lost. It runs asynchronously, off the serving path, so none of it lands inside the bidder’s latency budget.

The general shape: when you consolidate something, audit what other people were reading out of the shape you are about to delete. Some of your outputs are somebody else’s inputs, and they will not be in your design doc.

Parity had to be proven, not asserted

The mandate came with a hard constraint attached: publisher revenue and fill rate could not regress. Not in aggregate — granularly, at treatment level, because an aggregate that holds while a specific geo or app collapses is a publisher escalation waiting to happen.

Which meant this could not ship on a dashboard that moved in the right direction after launch. Ad revenue is far too noisy for that: seasonality, partner budget cycles, supply mix. A dashboard going up the week you launch is not evidence, and on a system this noisy it is not even weak evidence. So parity was measured through the experimentation platform, with controlled treatments and a randomization unit chosen to match the unit revenue is earned in — which turned out to be the constraint that shaped the entire rollout, and is the first item in the rails below.

Where it worked, and where I compromised

Everything above describes the architecture as designed. It did not land the same way in every ad format, and the difference is the most honest thing I can tell you about this project.

In banner, consolidation was complete and it worked as intended. All programmatic demand went through the consolidated path, which meant every partner’s bid really was on the table at the same time, before any tier was walked. That is the precondition for choosing across bids rather than settling for the first one you happen to encounter, and it is where actual margin optimization happened — driven by a machine-learning model that our own first-party demand integration used to bid directly inside the waterfall. Having the full picture early is what made a model worth running at all.

In fullscreen formats, I compromised, and it cost something specific. We consolidated all the programmatic demand except our largest DSP spenders, who stayed on the legacy sequential path. The reasoning was not subtle: those buyers represented too much revenue to put through a rewrite of the pricing mechanism at the same time as everyone else. Protecting them was the right call on risk, and I would make it again.

The cost is that a consolidated auction is only as good as the demand sitting inside it, and we had just removed the biggest bidders from ours. The partners left in the consolidated pool were competing for the opportunities the excluded buyers had not already taken. They rarely won. When they did win, it was at a low floor. Structurally the fullscreen path got the request-reduction benefit and very little of the pricing benefit, because the pricing benefit depends entirely on the strongest demand being in the pool at the moment the ladder is walked.

I think that is worth stating plainly rather than smoothing over. The same architecture delivered its full value in one format and a deliberately degraded version in another, for a defensible reason, with a consequence that showed up in exactly the place the design says it should. Carving your best demand out of a consolidated auction to protect it hollows out the thing you just built — and knowing that in advance would not necessarily have changed the decision, because the risk of not carving them out was worse.

The rails

The rollout is the part of this project I would defend hardest, because a change to how a revenue-critical auction prices itself can lose real money in minutes and nobody wants to be awake for it.

• Deterministic assignment. A salted hash of user session plus game ID decided treatment, which kept a user in the same treatment across the entire multi-tier waterfall. Get this wrong and a user drifts between control and treatment partway down the ladder, and you do not get a noisy measurement, you get a meaningless one — the tiers of a single opportunity are not independent observations of anything.
• Multi-treatment iteration. Several treatments against a control each round, promoting whatever held up to larger traffic shares, rather than one big flip.
• Scale-normalized revenue alerting. A treatment at 1% of traffic and one at 10% do not have the same sensitivity, and a single fixed threshold is either deaf to the small one or constantly crying about the large one.
• Automatic rollback via circuit breakers. A treatment reverted itself to control when margin or fill rate breached a Z-score threshold. Not a page, not a runbook — the system did it and told us afterwards. Worth one point of precision: that rollback was automated at the experiment layer, not the deploy layer. Nothing was being redeployed; a treatment was being sent back to control.
• Low-cardinality telemetry tags, so the on-call could filter live behavior by geo, partner, and treatment without taking out the metrics backend. That is a safety decision rather than a cost one, because the circuit breakers read from the same metrics infrastructure. An observability system that falls over under incident-time query load takes the automated rollback with it.

One small thing I like about this system: the same primitive shows up twice doing two unrelated jobs. The bid cache keys on a salted hash of the user session, which is what scopes a cached bid to the opportunity that fetched it. Experiment assignment uses a salted hash of the session plus the game, which is what keeps a user in one treatment for the whole descent. One hash for isolation, one for deterministic bucketing.

Results, and what they are attached to

Combined networking and compute savings ran $5–8k a day, over $2 million annualized. Publisher revenue parity held, Unity held its target margin per auction, and the consolidation mandate was met — which is the result that mattered most, because the alternative was penalized or rejected bids from our most important demand channel during the exact window when that channel was funding the move off the waterfall and onto real-time bidding.

For scale context rather than as a claim of my own: the Exchange served programmatic auctions at over a million QPS, and Unity’s public docs put the surrounding platform at 2.5 billion users across more than 750,000 apps, averaging 165 million daily actives. Those are the platform’s numbers, not mine. The QPS figure is the one my design had to survive.

Turning it off, and why that is the rare part

By 2025 the industry had finished moving to real-time bidding, and legacy waterfall traffic fell from about 30% of the total to under 5%. I want to be careful about that number: it is not a result of mine. It is the market completing the move from waterfall to real-time bidding that the bridge had been built to span. But it is exactly the signal that the bridge was done, and the system was decommissioned.

The whole arc is real, and it is the reason I still tell this story: a mandate that could not be met by the existing architecture, a bridge built to buy time against it, roughly two years of it earning its keep, and then a teardown that actually happened. I would not claim I set an expiry date at design time — the horizon became clear as the market moved, not on a whiteboard in the first week. What I will claim is the ending, because the ending is the part that usually does not arrive.

Plenty of engineers can describe something they built. Far fewer can describe something they built and then removed. The failure mode here was never building the bridge — the bridge was correct, and it paid for itself many times over. The failure mode is that bridges calcify. Traffic drops to a rounding error, the cutover gets declared done, and the old path sits there for years accruing on-call load and doubling the cost of every feature that has to work on both sides, because deleting it carries all of the remaining risk and none of the credit.

So the lesson I would actually hand to someone is narrower than “plan your sunsets,” which nobody does. It is this: a bridge is legitimate architecture as long as you stay willing to remove it, and staying willing is the hard part, because by the time removal is correct the system has colleagues, dashboards, and a reputation for working. Build it to solve the problem in front of you. Do not let it become the thing you defend.

The one durable artifact was the piece worth keeping, and notably it was not the caching: the pricing guidance module, which proved out predictive floors and went on to become something larger.

---

Unity’s integration spec for the Exchange, including the auction type, the imp.bidfloor field, the 200ms bidder deadline, and nurl win notices, is published at the Unity Exchange integration docs.