A small, self-hosted gateway and Git-managed control plane for LLM traffic. It is early, readable, and still changing.
Provider-native requests stay in the data plane. Fleet configuration stays in Git. Every figure below says where it came from. The ones marked measured were written down by a run rather than by a person. The ones marked bounded are limits the code refuses to cross. The ones marked chosen are decisions, with the reasoning written down somewhere you can go and disagree with it.
chosen
One data plane, one control plane. A team can run the data plane alone, from a file on disk, and never install the second.
bounded
Runtime state lives in memory and Git holds the truth. Nothing your traffic depends on can be down because a database is down.
measured
Measured on the WASM path, then used as a budget: per-event streaming hooks stay switched off because that cost times every token is not worth paying.
measured
Recorded from live traffic rather than written by hand, because fixtures you author agree with you and traffic does not.
01 · Shape
Platform tools tend to grow until running them is its own job. The budget here was fixed before the first line: one thing in the request path, one thing beside it, and a Git repository they both agree to obey. Anything that wanted to be a third component had to justify itself against that, and nothing has.
01
Sits in the request path and does the work
runs alone
A proxy that reads every provider dialect into one internal event model, meters tokens as they stream, redacts on the way past, and stops a request when its budget is gone. It reads a file, it serves traffic. If the rest of this page disappeared it would keep doing that.
02
Sits beside the path and keeps the fleet honest
It compiles what is in Git into exactly what each data plane should be running, ships it, watches for divergence, and pulls anything that drifted back into line. It is the interesting half, and it is also the half you can decline.
The split is not architectural taste. It means the thing carrying your traffic has nothing to phone home to, so an outage in the half that manages config cannot become an outage in the half that serves requests.
02 · Path
Streaming is where gateways quietly give up. Counting is easier once the whole response has arrived, so a lot of tooling waits, and waiting is indistinguishable from being slow. Everything below happens while the bytes are still moving.
Server-sent events from one vendor, a different event framing from another, raw binary frames with their own checksums from a third. Three shapes for one idea.
holds: No dialect gets a privileged path through the code
Adapters translate each dialect into a single event model. Replay the same response at chunk sizes of one byte, seven, sixty-four, or all at once, and the events that come out are identical.
holds: Chunk boundaries are an accident of the network, never of the meaning
Tokens are tallied as they pass, then reconciled against the provider's own final count. Nothing is held back to be counted at the end, because holding it back would be the same as breaking streaming.
holds: The response is never buffered whole
A budget exhausted halfway through a generation ends that stream deliberately, with a terminal event the operator chose, in the dialect the caller is already speaking.
holds: A cut looks like an ending, never like a crash
03 · Measured
A claim with no stated failure mode has usually not been tested hard enough to have found one. Each row here carries the method that produced it and the part that still does not work, in the same weight of type.
claim
how it was checked
Seventeen recorded transcripts replayed against the provider's own reported usage.
what is still wrong
All but one stream. The misses are structural rather than random: tool-call scaffolding and very short responses are where the estimate is worst.
claim
how it was checked
Each gateway holds a share of the budget, and the shares provably sum to no more than the cap.
what is still wrong
During a network partition, spend can exceed the cap by a bounded amount. The bound is published rather than hidden, because a system that claims a hard cap under partition is claiming something it cannot do.
claim
how it was checked
Extensions run sandboxed with no ambient access, on a fuel budget, interruptible mid-execution. They must be signed.
what is still wrong
This is why per-event hooks are off. The safety costs about eleven microseconds each time, which is affordable once per request and indefensible once per token.
claim
how it was checked
Git is the source of truth, and every rollout is a rendered snapshot tied to the commit that produced it.
what is still wrong
Break-glass exists for the night when Git is not the fastest way to fix production. It is visible, it expires on its own, and it is the exception that the design admits to.
04 · The terms
Six months later, when something breaks and the commit has your name on it, you turn up. You find it, you fix it or you revert it, and the write-up afterward describes the mechanism rather than the person. That is the whole contract, and the technical design exists to make it survivable: truth in Git is what lets you reconstruct a night you were not there for.
Tooling has closed most of the gaps that used to separate engineers, which leaves standing behind your own work as one of the few things still worth anything. Owned by the people who run it, open from the first commit, with nothing held back behind a paywall.