Skip to content

Providers, Priority Routing & Automatic Cooldown ​

Clients always send "model": "auto" to POST /v1/proxy/chat/completions. All provider selection, priority ordering, token limit clamping, error classification, cooldown backoff, and automatic recovery happen centrally on the server.

Data Collections ​

upstream_providers ​

FieldTypeDescription
nametext (unique)Provider identifier (e.g. deepseek, siliconflow, vllm)
base_urltextOpenAI-compatible root URL (without /chat/completions, e.g. https://api.deepseek.com/v1)
api_keytextUpstream API key; always masked as •••••••• in API responses and optionally encrypted at rest with --encryptionEnv
enabledboolDisabling a provider immediately skips all models belonging to it

upstream_models ​

FieldTypeDescription
providerrelation → upstream_providersParent provider (cascade delete)
nametext(128)Upstream model ID sent in JSON body (unique(provider, name))
display_nametext(128)Friendly label shown in admin UI relations
priorityintAscending priority: smaller values are tried first (10 before 20 before 100; ties break alphabetically by provider, then name)
context_windowintModel context limit; user's effective limits.context_window is the minimum positive value across enabled candidates (default 128000)
max_output_tokensintOutgoing max_tokens is clamped per candidate to this ceiling (default 8192)
enabledboolMaster switch for this model
unavailable_untildate (UTC)Cooldown expiration timestamp; empty when healthy or permanently blocked by auth
unavailable_reasonselectquota / rate_limit / auth
last_errortext(1024)Sanitized upstream error summary from the latest failure

Multi-Tier Cost & Reliability Strategy ​

Use priority to layer free daily quotas, promotional credits, and paid fallback models seamlessly:

priorityTier ExampleBehavior
10Internal vLLM or daily free-quota providerTried first on every request; automatically cools down when daily quota runs out and resumes automatically after cooldown
20Secondary / trial credit providerAbsorbs overflow when tier 10 is cooling down
100Paid primary provider (e.g. DeepSeek API)Reliable final fallback so users never see downtime

Upstream Error Classification & Cooldown Rules ​

When a candidate returns an error during POST /v1/proxy/chat/completions, the server classifies both the HTTP status code and response JSON (error.code / error.type / message):

CategoryUpstream SignalFallback?Cooldown / Recovery Policy
Quota / Balance Exhausted (quota)HTTP 402, or 429 with insufficient_quota / insufficient_balance / balance markersYes → tries next candidateMarks unavailable_reason = "quota", cools down 6 hours on first hit and 24 hours if a probe after expiry fails again
Rate Limited (rate_limit)HTTP 429 (non-quota)Yes → tries next candidateMarks unavailable_reason = "rate_limit", honors Retry-After header (default 30s, escalating to 2m on consecutive failure, capped at 24h)
Invalid Key / Unauthorized (auth)HTTP 401 or 403Yes → tries next candidateMarks unavailable_reason = "auth" with empty unavailable_until (no automatic retry). Updating the provider's api_key or clearing unavailable_reason in /_/ immediately restores it
Transient Upstream FaultHTTP 5xx or network timeoutYes → tries next candidateDoes not mark model unavailable
Invalid Client RequestHTTP 400 (context length exceeded, safety filter, bad schema)No → returns structured 400 invalid_request immediatelyDoes not waste fallback calls on deterministic client input errors

When a cooled-down model's unavailable_until passes, the next eligible request automatically probes it first: a 2xx response clears unavailable_reason, unavailable_until, and last_error automatically.

Licensed under AGPL-3.0 (Commercial licensing available: contact@aistrun.cc)