No description
  • Go 98.4%
  • Dockerfile 1.5%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Claudia c9986f645a
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
feat: cache prompts on the anthropic route
An agent resends its whole conversation every turn, which at full price is most of what it costs. Only a model whose rate knows the cache prices is cached, so a deploy of this changes nothing until the rates say what to count.
2026-11-15 20:23:11 +01:00
internal feat: cache prompts on the anthropic route 2026-11-15 20:23:11 +01:00
.gitignore feat: load a routing table that cannot mix local and remote 2026-11-02 12:37:25 +01:00
.woodpecker.yaml refactor: move to the meatbag org 2026-11-02 12:42:05 +01:00
config.example.yaml feat: price cached tokens at their own rates 2026-11-15 20:22:33 +01:00
Containerfile build: package the service for the registry 2026-11-02 12:37:25 +01:00
go.mod refactor: move to the meatbag org 2026-11-02 12:42:05 +01:00
go.sum feat: load a routing table that cannot mix local and remote 2026-11-02 12:37:25 +01:00
justfile refactor: move to the meatbag org 2026-11-02 12:42:05 +01:00
main.go feat: cache prompts on the anthropic route 2026-11-15 20:23:11 +01:00
README.md feat: cache prompts on the anthropic route 2026-11-15 20:23:11 +01:00

switchboard

The one address the house's services ask language models through. It speaks OpenAI's /v1/chat/completions and decides who actually answers, so swapping a provider is a change to one routing table and none to the services.

What a caller asks for

model What happens
anthropic/claude-haiku-5-5 That model and no other. If it is unavailable, the caller gets a 503
remote-fast (a pool) Members in order, swapping when one is unavailable
local, local/<model> Local only. If it is down, a 503 and the caller retries later
Anything else 404. There is no default route

The response's model is always the one that really answered, and usage.cost is what it cost. Anything citing a model must read model and not assume.

Asking for a local model never reaches a remote one. The config refuses to load a pool mixing the two, or a local upstream whose host is not on this network, so there is no failover path for a request to take. Which also means each remote pool's member list is a statement of which companies may see that prompt.

Exact model ids are for anything that has to be able to rely on the model, and pools are for anything that does not.

When a pool swaps

The upstream says What happens
429, 503, 529 Next member. This model rests for its Retry-After, or 30s
Timeout, connection failure, other 5xx Next member. This upstream rests 30s
Monthly cap reached, or out of credit Next member. This upstream rests until the month turns
401, 403 Next member, logged. This upstream rests 5 minutes
404 Next member, logged. This model rests 5 minutes
400 and other 4xx No swap. The caller gets the error, since the request is probably at fault
A refusal No swap. A refusal is an answer, and is not shopped around

A pool goes back to its first member as soon as that member's rest is over. A request nobody could answer is always a 503 with a Retry-After, whatever the last upstream said. Rests are in memory, so a restart clears them, and the monthly spend is on disk, so a restart does not.

Streams

"stream": true is answered as a stream, which can swap until the first chunk carrying any of the answer: some text, a tool call, a finish or a count. Up to there nothing has reached the caller, so a member failing is the table above, including OpenRouter's habit of answering 200 and then sending an error. Past there the answer is the caller's:

Partway, the upstream What happens
Breaks off, or ends with no finish An error event and no [DONE]. This upstream rests as above, so a retry finds the next member
Says nothing for its timeout The same. For a stream, timeout is the longest silence, not a limit on the whole answer
Loses its caller It is stopped, and nothing rests

Every stream is asked for its count, and the caller is only sent the usage chunk if it asked with stream_options.include_usage. One that ends before the count comes is estimated from the bytes asked and answered, at about four a token, and logged estimated=true. OpenRouter's models have no rate, so its estimates go uncounted unless one is given in rates.

Providers

openrouter, openai and local take the request as it came, reshaped a little for each one's vocabulary. anthropic is translated to the native Messages API, tools and their calls included, and refuses what it cannot carry (images, temperature, tools other than functions, mid-conversation system messages) rather than dropping it. Its thinking is left out of answers and streams alike: it is the model's working, and Claude only takes it back with its signature, which OpenAI's shape has nowhere to keep. A Claude stream cut short still counts its input, which Claude says up front, and estimates only the output.

Caching

OpenAI and OpenRouter cache prompts on their own. Claude caches only what it is asked to, and is asked for a model once its rate gives cache_read and cache_write: until then, what a cached token cost could only be a guess. Each request is marked in two places, for five minutes. After the instructions, for a caller asking many one-off questions under the same ones, and at the end, for a conversation whose next turn starts with all of this one. A prompt too short for the model to cache costs nothing extra.

Every cached token is counted at its rate's cache price, or as any other input where it has none, and the response's usage.prompt_tokens_details says how many were read and written.

Running it

switchboard -config config.yaml -db ledger.db -addr :8080
SWITCHBOARD_TOKENS=spis-resolver-llm=…,other=…     who may ask
ANTHROPIC_API_KEY=…                                whatever key_env names

It refuses to start on a missing credential or a bad routing table. See config.example.yaml. GET /status shows each upstream's spend, cap and rest, and needs a caller's token. GET /healthz does not.

Each request logs who asked, what answered, the tokens and the cost, and a stream adds how it ended, when its first chunk came and whether its count is an estimate. Never the prompt: these are food logs. Lines starting switchboard: are the ones worth an alert.