Latency is a placement decision, and it is not an infrastructure one

Our platform schedules work across whatever machines happen to be on the network. Peers find each other, advertise how much memory they have free, take a lease on a job and stream the result back. A machine that stops answering leaves the schedule rather than failing the work.

Building that made something obvious that we had previously treated as an infrastructure detail: placement decides what the product is allowed to promise.

The conversation that happens too late

An architecture review, weeks from a launch: the product promises a response inside a few hundred milliseconds, and the inference sits in one region a long way from most users. Somebody is about to break a promise, and it will not be the person who chose the region.

Placement is not a deployment concern that follows the product decision. It is one of the inputs to it.

Three questions, asked early

• Is a human waiting? If so, round trips are the budget you are spending, and every hop needs to justify itself against it.
• Where must the data stay? Residency rules do not complicate architectures, they eliminate them. Better to know which ones are already gone.
• What does idle capacity cost? A workload with a sharp peak, running on always-on infrastructure, is paying for silence eleven months a year.

What tends to win

The interactive path close to the user; the heavy lifting wherever it is cheapest; an explicit contract between them. Not because distributed is fashionable, but because the two paths genuinely have different constraints, and pretending otherwise means one of them quietly loses.

The corollary is worth stating: if you cannot say where a given piece of work runs and why, you do not yet know what your product can promise.