Routes and fallback

Give a model name a list of providers, and choose how they are tried: in order, with retries, or raced.

A route is a model name that clients ask for, and the candidates it goes to. Without routes, a model goes to the provider whose model list has it (see Providers and models).

[routes.smart]
to = ["anthropic/claude-haiku-5-5", "openai/gpt-6-luna"]

A client that asks for "model": "smart" gets Claude Haiku, and GPT if that call fails. The client’s code doesn’t change when you change the route.

Candidates

to is one candidate or a list. Each candidate is:

provider is a name from your [providers.*] tables.

Fallback

Candidates are tried in order. If one fails, the next one gets the request.

[routes.smart]
to      = ["anthropic/claude-haiku-5-5", "openai/gpt-6-luna"]
tries   = 2          # per candidate, retries included
backoff = "250ms"    # first wait before a retry, doubled after each

A candidate is retried only for transient failures: the provider couldn’t be reached, or it answered 408, 425, 429, 500, 502, 503 or 504. Any failure moves on to the next candidate. A candidate that failed its last 3 calls within 30 seconds is tried last.

Hedge

hedge races the candidates instead of waiting for one to fail. Each candidate starts this long after the one before, if no answer has come yet. The first success wins.

[routes.fast]
to    = ["openai/gpt-6-luna", "anthropic/claude-haiku-5-5"]
hedge = "200ms"

Here GPT gets the request first. If it hasn’t answered after 200 ms, Claude gets it too, and the client gets whichever answers first. Use it to cut tail latency; it can cost two calls for one request. hedge can’t be combined with fallback = false, and tries and backoff don’t apply to a hedged route.

Pick

pick orders the candidates before they are tried:

[routes.cheap]
to    = ["openai/gpt-6-luna", "anthropic/claude-haiku-5-5"]
pick  = "cheapest"
needs = ["tools"]

needs drops candidates whose model lacks a capability, such as tools or vision, as its provider’s model list says. A candidate the list says nothing about is kept.

Weights go on the candidates:

[routes.split]
to   = [{ target = "openai/gpt-6-luna", weight = 3 }, { target = "anthropic/claude-haiku-5-5", weight = 1 }]
pick = "weighted"

Model aliases and globs

A route’s name is any model name, so a route is also an alias. smart, fast and cheap above are names your clients use, and you can change what they point at in one place.

A route name with * is a glob that matches many model names. With {model}, the name the client asked for is passed through:

[routes."claude-*"]
to = "anthropic/{model}"

[routes."llama-*"]
to = "local"

An exact route name is matched first, then provider/model names, then globs in the order they appear in the file.

Limit a route

when limits a route to some requests. Other requests skip it and are routed as if it didn’t exist.

[routes.internal]
to   = "openai/gpt-6-luna"
when = { clients = ["ops"] }       # client keys, by name

clients takes client key names. kinds takes request kinds such as chat or embedding.

Explain

To see where a model would go, without sending anything:

curl "http://localhost:8080/ui/api/routes/explain?model=cheap"

{
  "by": { "type": "rule", "name": "cheap" },
  "pick": "cheapest",
  "mode": "fallback",
  "to": [
    { "provider": "openai", "model": "gpt-6-luna", "price": { "input": 0.25, "output": 2 }, "seen": null },
    { "provider": "anthropic", "model": "claude-haiku-5-5", "price": { "input": 1, "output": 5 }, "seen": null }
  ],
  "passed": []
}

by.type is rule (a route), named (provider/model), listed (a provider’s model list) or default. mode is single, fallback or hedge. passed lists the candidates dropped and why, such as lacks tools. Add &kind=embedding for another kind of request, or &client=NAME for a route limited by when. A model nothing serves returns { "by": null }.

This is part of the admin API: with client keys set, it needs an admin key. The web UI shows the same thing.