Same Request, Two Paths

You cannot take a maintenance window on a live ad exchange. Over a million queries per second. Fill is the product. A bad deploy costs money in minutes. The architecture still has to change, because the next five years of work are trapped in the current shape.

A dashboard that moves the week you launch is not evidence. Ad revenue is too noisy for that: seasonality, partner budget cycles, supply mix. Two treatments can hold revenue and still be different auctions.

So the rewrite is not the new structs. It is a second path that is allowed to be wrong in production because it does not serve. Same params through both implementations. The legacy path answers the player. The new path is scored against it, scope by scope, variable by variable, until the contracts match. Then the business is scored again, sliced by demand partner — the grain where one DSP can collapse without the exchange-wide graph moving.

99.5% of those field comparisons matched. The remaining half a percent could be named. Then we compared downstream impressions, revenue, eCPM, and auction wins per DSP. That is what made the cutover possible. What follows is the system that made it necessary, the plan that made it staffable, and the two proofs that made it shippable.

The shape that had to go

The pipeline was a list of hooks. Each one took the full ad request, mutated it, and handed it to the next. Pre-hooks enriched it — geo, experiments, floors, a couple of outbound calls — each reading and writing the same object. Demand partners already ran in parallel. Post-hooks ran the auction and the reporting in the same list. Then a response went back to the client.

A hook took *Request because that was the only type the pipeline knew about. Every stage could see every field and write any of them. Most hooks needed a handful. All of them could clobber the rest. Side effects on a shared request are how you get a bug that cannot be localized — and how you get a pipeline with no contract to diff. If twelve functions own the same struct, there is nothing to compare a replacement against. There is only production.

// ProcessRequest is the whole auction as two lists that mutate the same
// request. You cannot replace this in pieces. The lists are both the
// architecture and the data race.
func (n *Node) ProcessRequest(req *Request) {
	for _, h := range n.pre {
		h.Run(req)
	}
	req.Bids = n.getBids(req)
	for _, h := range n.post {
		h.Run(req) // auction and reporting, same list
	}
	n.send(req.Response)
}

What the plan actually chose

Before anyone rewrote a stage, the request path was instrumented end to end — the old path and, as it landed, the replacement — so the two could be compared at all. That is more than timing a hot function: spans on the way in and the way out, with fill, revenue, latency, and errors sliced the way an on-call debugs — geo, app, partner.

An early draft of the plan had already decided the answer was parallelism. Pre-auction enrichment — the work that draft proposed to fan out — was about forty milliseconds at p99, almost all of it in one stage, about four percent of the request. Two of those stages made outbound calls. The rest were in-memory. A goroutine per in-memory function is scheduler noise. I wrote the skip down. “Won’t do.” Dated.

The 200 millisecond bidder deadline Unity publishes for Exchange partners is a per-call budget on the way out, not a description of the whole request. Enrichment was a rounding error of that budget too.

The traces split the post-hook list into two jobs. Almost all of that time was not a function call. It was a second auction sitting in the same list as the reporting: a demand call that cannot start until the first round has finished. The client has to wait for it. The second auction belongs on the critical path. Kafka publishes and win notices do not.

So the plan that shipped was narrower. Make the request an input, not a blackboard. Each stage returns a new value and an error. Reporting leaves the critical path. The second auction stays sequential, in the open. Do not WaitGroup the four percent.

func (n *Node) ProcessRequest(req *Request) {
	enriched, err := n.enrich(req)
	if err != nil {
		n.fail(req, err)
		return
	}
	bids, err := n.getBids(enriched)
	if err != nil {
		n.fail(req, err)
		return
	}
	result, err := n.auction(bids)
	if err != nil {
		n.fail(req, err)
		return
	}
	go n.report(result)
	n.send(result.Response)
}

enrich, getBids, and auction are ordinary functions. A stage that needs geo takes geo. A later stage cannot clobber a field an earlier stage already finished computing. That is the property the hook list did not have, and it is the property that makes a replacement diffable: each scope has an output you can name.

Make it comparable, then diff it

Two implementations of an auction are not comparable just because they compile. They are comparable when they consume the same params and you can walk their outputs.

We made the sequential path as deterministic as it could be. Fields that rolled a die — rules evaluation with randomization in it — got a seed so both paths shared the roll. Where that was not possible, those fields were left out of the comparison. Then the same request went through both. The legacy path served the client. The new path did not. It was evaluated scope by scope, variable by variable, through the outputs of the functions.

99.5% matched. The remaining 0.5% was not a rounding error we papered over. It sorted into two buckets: legacy bugs the new path had fixed, so the old contract was the one that was wrong, and results that were never deterministic — the legacy path did not always agree with itself. Both are acceptable. Unexplained drift is not.

I am not going to invent a sample rate or a field count I cannot show. The method is the claim: same params, legacy serves, walk the scopes, account for every mismatch.

Then the money, at DSP grain

A 99.5% field match can still be a different auction economically. Mix shifts. A demand partner stops winning. eCPM holds while impressions die. An exchange-wide revenue graph will not tell you which.

So we also compared the two paths on what actually paid the bills, sliced by DSP: downstream impressions, revenue, eCPM, and auction wins. That is the grain. Aggregate fill can hide a partner collapsing. eCPM can look healthy while volume has left. Wins without revenue is a different lie than revenue without wins.

Those slices ran through the experiment platform, against a control, in waves — not a dashboard that moved the week of launch. The field diff asks whether you rewrote the auction. The DSP-level four-metric check asks whether the rewritten auction is still the business. They are not the same question. We did not ship on either one alone.

I do not have public lifts to hang on those four metrics, so I will not invent them. The claim is the grain and the set: impressions, revenue, eCPM, wins, per DSP.

The plan had to survive my leaving

I wrote the refactor document. I broke the work into epics and milestones and tickets. Then I went on paternity leave and handed implementation to two senior engineers.

When I got back they were almost ready to start testing. That is the correct shape. A design that only works while you are in the room is not a design yet. They built from the plan. I came back for the proof: the comparison, the remainder, the DSP-level checks.

What that unlocked

Revenue was the constraint, and it held.

Latency moved for a reason you can point at: reporting used to sit in the same list as the auction. The client waited on a Kafka publish. It does not anymore.

Error rates moved for a reason you can also point at. A mutating hook list fails by leaving a half-written request for the next function to consume. The new stages fail by returning error. The blast radius of a bad geo lookup is that stage, not an ambient wrong field three hooks later.

Immutable stages are why a later change — the one-shot pre-fetch that taught a waterfall to speak bidding — did not have to audit twelve mutating hooks.

What I would hand someone

Instrument the path you have and the path you are putting next to it. If you cannot score a replacement, you will find out from a publisher.

Make the request a sequence of typed outputs, not a blackboard. The reason to stop mutating a shared struct is not a WaitGroup. It is so a failure has a return value, a stage has a name, and the next person can change one function without auditing twelve.

Then build the comparison before you cut over. Same params. Legacy serves. Walk the scopes. Seed the die or skip it. Name the 0.5%. Then check impressions, revenue, eCPM, and wins per DSP, because a perfect field match can still be the wrong auction.

This project predates pairing with a model. Generating the replacement types is cheap now. A shadow harness that diffs every variable, names the 0.5%, and then checks impressions, revenue, eCPM, and wins per DSP is the rewrite — whether two seniors type it or a model does. When generation is cheap, verification is the product.