- Go 98.4%
- Dockerfile 1.5%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
An agent resends its whole conversation every turn, which at full price is most of what it costs. Only a model whose rate knows the cache prices is cached, so a deploy of this changes nothing until the rates say what to count. |
||
| internal | ||
| .gitignore | ||
| .woodpecker.yaml | ||
| config.example.yaml | ||
| Containerfile | ||
| go.mod | ||
| go.sum | ||
| justfile | ||
| main.go | ||
| README.md | ||
switchboard
The one address the house's services ask language models through. It speaks
OpenAI's /v1/chat/completions and decides who actually answers, so swapping a
provider is a change to one routing table and none to the services.
What a caller asks for
model |
What happens |
|---|---|
anthropic/claude-haiku-5-5 |
That model and no other. If it is unavailable, the caller gets a 503 |
remote-fast (a pool) |
Members in order, swapping when one is unavailable |
local, local/<model> |
Local only. If it is down, a 503 and the caller retries later |
| Anything else | 404. There is no default route |
The response's model is always the one that really answered, and usage.cost
is what it cost. Anything citing a model must read model and not assume.
Asking for a local model never reaches a remote one. The config refuses to load a pool mixing the two, or a local upstream whose host is not on this network, so there is no failover path for a request to take. Which also means each remote pool's member list is a statement of which companies may see that prompt.
Exact model ids are for anything that has to be able to rely on the model, and pools are for anything that does not.
When a pool swaps
| The upstream says | What happens |
|---|---|
| 429, 503, 529 | Next member. This model rests for its Retry-After, or 30s |
| Timeout, connection failure, other 5xx | Next member. This upstream rests 30s |
| Monthly cap reached, or out of credit | Next member. This upstream rests until the month turns |
| 401, 403 | Next member, logged. This upstream rests 5 minutes |
| 404 | Next member, logged. This model rests 5 minutes |
| 400 and other 4xx | No swap. The caller gets the error, since the request is probably at fault |
| A refusal | No swap. A refusal is an answer, and is not shopped around |
A pool goes back to its first member as soon as that member's rest is over.
A request nobody could answer is always a 503 with a Retry-After, whatever the
last upstream said. Rests are in memory, so a restart clears them, and the monthly
spend is on disk, so a restart does not.
Streams
"stream": true is answered as a stream, which can swap until the first chunk
carrying any of the answer: some text, a tool call, a finish or a count. Up to
there nothing has reached the caller, so a member failing is the table above,
including OpenRouter's habit of answering 200 and then sending an error. Past
there the answer is the caller's:
| Partway, the upstream | What happens |
|---|---|
| Breaks off, or ends with no finish | An error event and no [DONE]. This upstream rests as above, so a retry finds the next member |
Says nothing for its timeout |
The same. For a stream, timeout is the longest silence, not a limit on the whole answer |
| Loses its caller | It is stopped, and nothing rests |
Every stream is asked for its count, and the caller is only sent the usage chunk
if it asked with stream_options.include_usage. One that ends before the count
comes is estimated from the bytes asked and answered, at about four a token, and
logged estimated=true. OpenRouter's models have no rate, so its estimates go
uncounted unless one is given in rates.
Providers
openrouter, openai and local take the request as it came, reshaped a little
for each one's vocabulary. anthropic is translated to the native Messages API,
tools and their calls included, and refuses what it cannot carry (images,
temperature, tools other than functions, mid-conversation system messages)
rather than dropping it. Its thinking is left out of answers and streams alike:
it is the model's working, and Claude only takes it back with its signature, which
OpenAI's shape has nowhere to keep. A Claude stream cut short still counts its
input, which Claude says up front, and estimates only the output.
Caching
OpenAI and OpenRouter cache prompts on their own. Claude caches only what it is
asked to, and is asked for a model once its rate gives cache_read and
cache_write: until then, what a cached token cost could only be a guess. Each
request is marked in two places, for five minutes. After the instructions, for a
caller asking many one-off questions under the same ones, and at the end, for a
conversation whose next turn starts with all of this one. A prompt too short for
the model to cache costs nothing extra.
Every cached token is counted at its rate's cache price, or as any other input
where it has none, and the response's usage.prompt_tokens_details says how many
were read and written.
Running it
switchboard -config config.yaml -db ledger.db -addr :8080
SWITCHBOARD_TOKENS=spis-resolver-llm=…,other=… who may ask
ANTHROPIC_API_KEY=… whatever key_env names
It refuses to start on a missing credential or a bad routing table. See
config.example.yaml. GET /status shows each upstream's spend, cap and rest, and
needs a caller's token. GET /healthz does not.
Each request logs who asked, what answered, the tokens and the cost, and a stream
adds how it ended, when its first chunk came and whether its count is an estimate.
Never the prompt: these are food logs. Lines starting switchboard: are the ones worth an
alert.