Handle failover and retries

A provider going down should not become your outage. The Router already retries across providers inside a single request of yours, so the useful questions are narrower: when is that default silently switched off, and which of the errors that still reach your client are worth retrying yourself. This page answers both and gives a retry policy you can paste in.

Prerequisites

• An account with a funded balance. See Get 0G and fund your account.
• An sk- API key from the 0G Private Computer Console, exported as ZG_API_KEY.

The examples use glm-5.2.

The default already fails over

Send no routing headers and the Router picks a healthy provider for the model by round-robin, moves to the next healthy provider if that one errors, and returns 503 no_available_provider only when every candidate has failed. All of that happens inside one HTTP request; you see a single response, and x_0g_trace.provider tells you who ultimately served it.

For most applications this is the whole answer, and the correct amount of client code is none.

Pinning turns it off

X-0G-Provider-Address routes to one specific provider by on-chain address, and it implies X-0G-Provider-Allow-Fallbacks: false. If that provider fails, the request fails. That is the intended semantic: you asked for that provider, so quietly serving you from another one would defeat the point of asking.

For “prefer this provider, but do not die with it”, re-enable fallback explicitly:

curl https://router-api.0g.ai/v1/chat/completions \
  --max-time 120 \
  --fail-with-body \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ZG_API_KEY" \
  -H "X-0G-Provider-Address: <PROVIDER_ADDRESS>" \
  -H "X-0G-Provider-Allow-Fallbacks: true" \
  -d '{
    "model": "glm-5.2",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Situation	Effective Allow-Fallbacks	Result
No X-0G-Provider-Address	true	Cross-provider retry on failure
X-0G-Provider-Address set	false	Pinned provider fails, request fails
X-0G-Provider-Address plus Allow-Fallbacks: true	true	Start at the pin, fall back to others

X-0G-Provider-Allow-Fallbacks accepts exactly true or false, case-insensitive. 1, 0, and yes are rejected with 400 invalid_provider_header — the Router will not guess what you meant about a reliability control. Absent or blank means unset and takes the default, which is never an error. Full semantics: Provider routing headers.

Two other constraints also survive failover, because both are applied before the candidate pool is sorted: price ceilings (see Cap what a single request may cost) and trust mode (see Trust modes). A fallback path can never cross a line you drew.

The retry matrix

The Router already exhausted its own failover before returning an error to you. Your retry only helps when the underlying condition can change on its own within seconds.

Status	code	Retry	Why
429	rate_limit_exceeded	Yes, after Retry-After seconds	Purely time-based; it will clear
502	provider_error	Yes, briefly	Failover was exhausted, but a provider may come back. Bounded exponential backoff
503	no_available_provider	Not on a tight loop	Nothing healthy to try; switch model or wait
503	no_provider_for_trust_mode	Not on a tight loop	No supply in that tier; retry slowly or switch to a model with a provider in the tier
400	no_provider_within_max_price	No	Structural: the pool is empty by your own filter. Raise the ceiling
400	invalid_provider_header, invalid_trust_mode, invalid_body	No	Malformed request. Fix the value
401 / 402 / 403	—	No	Credentials, funds, or scope. Retrying changes nothing

The 502 versus 503 split is the one to get right. 502 means providers were reachable but erroring, so a short backoff is reasonable. 503 means there was nothing healthy to try at all, and a retry loop against it is just load with extra steps.

Never recover from a 503 by dropping the constraint

Retrying no_provider_for_trust_mode without the X-0G-Provider-Trust-Mode header will succeed — by giving you a weaker guarantee than the one you asked for. The Router never silently downgrades a tier; do not do it on its behalf. See Trust modes.

Implement the policy

Set maxRetries: 0 on the SDK client and classify errors yourself. The SDK’s own retry is status-blind at the level that matters here: it cannot tell no_provider_within_max_price from provider_error, so it will happily re-send a request that is guaranteed to fail the same way. Use the SDK’s counter only if you never inspect error.code.

cURL

#!/bin/sh
# Retry 429 per Retry-After and 502 with bounded backoff. Everything else fails fast.
attempt=0
while [ "$attempt" -lt 3 ]; do
  body=$(mktemp)
  status=$(curl https://router-api.0g.ai/v1/chat/completions \
    --max-time 120 \
    --silent \
    --output "$body" \
    --write-out '%{http_code}' \
    --dump-header /tmp/zg-headers \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $ZG_API_KEY" \
    -d '{"model":"glm-5.2","messages":[{"role":"user","content":"Hello"}]}')

  case "$status" in
    200) cat "$body"; exit 0 ;;
    429)
      wait=$(awk 'tolower($1) == "retry-after:" { print $2 }' /tmp/zg-headers | tr -d '\r')
      sleep "${wait:-5}" ;;
    502)
      sleep $((attempt + 1)) ;;
    *)
      cat "$body" >&2; exit 1 ;;
  esac
  attempt=$((attempt + 1))
done
echo "retries exhausted" >&2
exit 1

TypeScript

import OpenAI from "openai";

const apiKey = process.env.ZG_API_KEY;
if (!apiKey) throw new Error("ZG_API_KEY is not set");

const client = new OpenAI({
  apiKey,
  baseURL: "https://router-api.0g.ai/v1",
  timeout: 120_000,
  maxRetries: 0, // classify below instead of retrying blindly
});

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

async function chat(prompt: string, attempt = 0): Promise<string> {
  try {
    const res = await client.chat.completions.create({
      model: "glm-5.2",
      messages: [{ role: "user", content: prompt }],
    });
    return res.choices[0].message.content ?? "";
  } catch (err) {
    if (!(err instanceof OpenAI.APIError)) throw err;

    // 429: honour the server's own Retry-After.
    if (err.status === 429 && attempt < 3) {
      const wait = Number(err.headers?.["retry-after"] ?? 5) * 1000;
      await sleep(wait);
      return chat(prompt, attempt + 1);
    }

    // 502: failover was exhausted, but a provider may return shortly.
    if (err.status === 502 && attempt < 2) {
      await sleep(1000 * 2 ** attempt);
      return chat(prompt, attempt + 1);
    }

    // 503: do not loop, and do not drop the constraint that caused it.
    // 400/401/402/403: deterministic. Retrying reproduces the failure.
    throw new Error(`router error ${err.status} (${(err.error as { code?: string } | undefined)?.code})`);
  }
}

Python

import os
import time

from openai import APIStatusError, OpenAI

api_key = os.environ["ZG_API_KEY"]

client = OpenAI(
    api_key=api_key,
    base_url="https://router-api.0g.ai/v1",
    timeout=120.0,
    max_retries=0,  # classify below instead of retrying blindly
)


def chat(prompt: str) -> str:
    for attempt in range(3):
        try:
            res = client.chat.completions.create(
                model="glm-5.2",
                messages=[{"role": "user", "content": prompt}],
            )
            return res.choices[0].message.content or ""
        except APIStatusError as err:
            # 429: honour the server's own Retry-After.
            if err.status_code == 429:
                time.sleep(float(err.response.headers.get("Retry-After", "5")))
                continue
            # 502: failover was exhausted, but a provider may return shortly.
            if err.status_code == 502 and attempt < 2:
                time.sleep(2**attempt)
                continue
            # 503: do not loop, and do not drop the constraint that caused it.
            # 400/401/402/403: deterministic. Retrying reproduces the failure.
            raise
    raise RuntimeError("retries exhausted")

Expected output

A 429 carries Retry-After in seconds; sleep for that long rather than picking your own interval.

{
  "error": {
    "message": "Rate limit exceeded",
    "type": "rate_limit_error",
    "code": "rate_limit_exceeded"
  },
  "request_id": "7f3c1e08-9a4b-4d21-8f6e-2b5c9d0a1e34"
}

Reproduced from the error contract — the envelope and code are verified against the live API, but a 429 was not deliberately triggered to capture this body.

A 503 from a trust-mode constraint names the tier it could not satisfy:

{
  "error": {
    "message": "failed to select provider: no provider available for the requested trust mode: tier=private",
    "type": "server_error",
    "code": "no_provider_for_trust_mode"
  },
  "request_id": "2e85451b-cb67-420b-8321-c2edf22ce0bb"
}

Live response, 2026-07-20 — triggered by asking for private on a model with no TEE-backed provider.

Log request_id on every failure. The Router does not store your prompts, so it is the only handle support has on the request.

Common errors

Status	code	Cause	Fix
400	invalid_provider_header	Allow-Fallbacks sent as 1, 0, or yes	Send exactly true or false
400	no_provider_within_max_price	Price ceiling emptied the candidate pool	Raise the ceiling; see Cap what a single request may cost
429	rate_limit_exceeded	Too many requests	Sleep for Retry-After seconds
502	provider_error	Provider failed after failover was exhausted	Short bounded retry
503	no_available_provider	No healthy provider for this model	Back off hard, or switch model
503	no_provider_for_trust_mode	No supply in the requested tier	Wait, or switch to a model with a provider in that tier. Do not drop the header

Full list with retry guidance: Error codes.

Next steps

• Harden for production — the rest of the pre-launch list around this retry policy
• Error codes — every status, type, and code with its retryability