one door for every llm call: fallbacks and caching

Every run of this content pipeline makes more than 20 model calls: a topic selector, a hypothesis agent, an outliner, a writer, an editor, a validator, plus a handful of research and metadata steps. Early on, each of those call sites talked to the provider client directly. The model id, the retry count, the caching header, and the cost-tracking write were all copied into a dozen places, and swapping a model or changing a fallback meant editing all of them and hoping I got each one right.

So now everything goes through one function. A call site passes an organization id, a model alias, and the messages, and gets back a string. It does not know which provider it hit, whether a fallback fired, whether the system prompt was cached, or what the call cost. All of that lives in a single module, which is the only place model behavior is allowed to change.

one function, every model call

The alias map is the whole public contract. A caller says claude-opus and never touches a provider path. The map turns an alias into an OpenRouter model id, and a couple of small frozensets record which of those support prompt caching and which are the expensive research models.

MODELS = {
    "claude-opus": "openrouter/anthropic/claude-opus-4-7",
    "claude-sonnet": "openrouter/anthropic/claude-sonnet-4-6",
    "claude-haiku": "openrouter/anthropic/claude-haiku-4.5",
    "deepseek-flash": "openrouter/deepseek/deepseek-v4-flash",
    # image-gen, embedding-small, and three perplexity-sonar aliases too
}
 
async def call_model(org_id, model_alias, messages,
                     temperature=0.3, max_tokens=4096,
                     use_response_cache=False, json_response=False) -> str:
    ...

The value of a single door is that every cross-cutting concern collapses into it. A model swap is one line in the map. Fallback, caching, dev-mode substitution, retries, and spend tracking each appear exactly once. When I later found a bug in how fallbacks were configured, there was one function to fix, not a dozen call sites to audit. The rest of this post is what accumulated inside that one function.

fallback chains that never touch my code

Providers have bad minutes. A model can be rate limited or briefly unavailable, and I did not want a single busy upstream to fail a whole article. OpenRouter supports a server-side fallback list: you send a primary model plus an ordered list of alternates in the request body, and it tries them in order until one answers. The caller issues one request and reads one response.

extra_body = {}
fallback_chain = _FALLBACK_CHAINS.get(model)
if fallback_chain:
    extra_body["models"] = fallback_chain

The alternative is a retry loop in my own code: catch the error, pick the next model, re-issue the call. That works, but it adds a network round trip per attempt and moves failover logic into the application, where it has to be tested and maintained. Pushing the chain into extra_body["models"] means the failover happens inside the router, on the same request, with no second round trip and no retry state for me to hold. Opus falls back to Sonnet and then Haiku; Sonnet falls back to Haiku. The pipeline keeps moving even when the first-choice model is having a rough time.

the prefix bug that ate my fallbacks

This is the part that cost me real time. Every alias in my map is a full OpenRouter path with an openrouter/ prefix, because that is what litellm expects in the primary model field. It turns out litellm strips that prefix off the primary field before sending, but it passes extra_body through completely untouched. My fallback list lived in extra_body["models"], so those ids kept their prefix.

def _or(model: str) -> str:
    return model.removeprefix("openrouter/")
 
_FALLBACK_CHAINS = {
    MODELS["claude-opus"]: [_or(MODELS["claude-sonnet"]), _or(MODELS["claude-haiku"])],
    MODELS["claude-sonnet"]: [_or(MODELS["claude-haiku"])],
    MODELS["deepseek-flash"]: ["google/gemini-3.5-flash", _or(MODELS["claude-haiku"])],
}

The failure was invisible. OpenRouter received fallback ids like openrouter/anthropic/claude-haiku, which are not valid model names, so on a primary failure it had nothing usable to fall back to and the request just failed. Nothing errored at config time; the chain was simply dead. The fix is the tiny _or helper that removes the prefix, so the primary field keeps its prefix and the fallback list is bare. The lesson stuck: a config that can be wrong without failing loudly at startup is a config that will be wrong in production.

caching a system prompt that repeats

The system prompt in this pipeline is large. It is the assembled orchestrator prompt with the org's learned memory stitched in, and it is nearly identical across the 20-plus calls in a run. Anthropic prompt caching exists for exactly this shape: mark a stable prefix once, and reads of it after the first call cost a fraction of the input price.

def _inject_cache_control(messages, model):
    if model not in _CLAUDE_MODELS:
        return messages
    if not messages or messages[0].get("role") != "system":
        return messages
    system_content = messages[0]["content"]
    if isinstance(system_content, str):
        system_content = [{"type": "text", "text": system_content}]
    last_block = system_content[-1]
    if "cache_control" not in last_block:
        system_content = [*system_content[:-1],
                          {**last_block, "cache_control": {"type": "ephemeral"}}]
    return [{"role": "system", "content": system_content}, *messages[1:]]

The economics decide where this is worth doing. A cache write costs about 1.25 times normal input, and a cache read costs about a tenth. On a one-shot call that trade loses. Across a run that reuses the same prefix 20-plus times, the first call pays the write premium and everything after reads at a tenth, which lands around 85 percent off the input cost of that shared prompt. The breakpoint goes on the last block of the system message only, and only for Claude models, since other providers on the router do not honor the same header. Everything else in the message list stays uncached, which is correct, because only the system prefix is actually stable.

a dev mode that does not spend real money

When I am iterating on prompt wording or graph flow, I am running the pipeline over and over, and none of those runs need a frontier model. So the router checks a dev-mode flag before it resolves anything, and in dev mode it quietly redirects the call.

if get_settings().dev_mode:
    if model_alias == "image-gen":
        return ""                       # never bill image generation while iterating
    if model_alias in _SONAR_ALIASES:
        return await _dev_sonar(org_id, messages, max_tokens)
    lm_base = get_settings().lm_studio_api_base
    if lm_base:
        ...                             # route to a local model served by LM Studio
    model = _DEV_MODELS.get(model_alias) or MODELS.get(model_alias, model_alias)
    ...

The important property is that the call sites do not know any of this happened. Image generation returns an empty string so it never bills. The research aliases run through a path that does a plain web search and feeds the results to a cheap model, instead of an expensive deep-research call. Everything else maps to one cheap model, or to a model served locally by LM Studio for zero API cost when a local base URL is set. I get to exercise the entire flow end to end, dozens of times a day, without watching a spend meter, and the production path is completely unaffected because it lives past the same flag.

cheap calls, self-healing json, tracked spend

Not every production call deserves the same treatment either. Background work like validation passes and metadata generation cares about cost more than latency, so there is a second entry point, call_model_cheap, that adds a provider preference telling OpenRouter to sort available providers by price. It is a small addition to the same body:

extra_body = {"provider": {"sort": "price"}}          # call_model_cheap only
...
if json_response:
    extra_body["plugins"] = [{"id": "response-healing"}]
    completion_kwargs["response_format"] = {"type": "json_object"}
...
completion_kwargs["num_retries"] = 2                  # production path only
fire_task(_record_spend(org_id, run_id, model, response))

Two more details close the loop. When a caller wants JSON, the router both sets a JSON response format and turns on OpenRouter's response-healing plugin, which repairs malformed model output before it comes back, so my parser sees valid JSON far more often. And spend is recorded through a fire-and-forget background task rather than inline, so writing the cost row never blocks or slows the response the caller is waiting on. Retries are set to 2, but only on the production path, so a deterministic bad-input error in dev mode is not pointlessly retried.

what i would change

Move the alias map out of code. The alias-to-model map and the dev substitutes are module constants. Making a different model choice for a specific org, or A/B testing two models on the same step, means editing and redeploying. A per-org config in the database would turn model selection into data, which is where a decision that changes this often belongs.

Validate fallback chains at startup. The prefix bug was silent because a malformed fallback id is only rejected when a failover actually fires, which is rare. I would add a startup check that every id in every chain is a known, resolvable model, so a typo fails the boot instead of quietly disarming a safety net that I only notice missing during an outage.

Cache more than the system prefix. The caching breakpoint sits only on the system message. In the per-article loop, a large research bundle is also reused across the write, edit, and validate calls for that one article. Marking that stable context as a second cache breakpoint would extend the same savings to the user side of the prompt, not just the system side.