Skip to content
0G Private Computer Cookbook

Harden for production

Everything that works in development also works in production, right up to the first leaked key, the first 429 storm, and the first balance that hits zero at 3am. This page is the list to work through before you point real traffic at the Router. Nothing on it is enforced by the API; these are the defaults that keep a deployment from failing in ways that are tedious to debug under pressure.

Four groups, roughly in the order things go wrong.

Prerequisites

Section titled “Prerequisites”

Key management

Section titled “Key management”

A valid sk- key spends your deposit. It is a spending instrument, not an identifier, and every item below follows from that.

  • Separate keys per environment. A distinct sk- key for staging and for production, not one key with two labels. The point is that revoking one cannot take the other down.
  • Keys come from the environment, never from source. ZG_API_KEY for inference, ZG_MANAGEMENT_KEY for account reads. Two variables, because they have different blast radii and a single shared name invites shipping the management key to a runtime that only needed inference.
  • No key reaches a browser. Proxy client requests through your own backend. Bundler prefixes such as NEXT_PUBLIC_ inline the value into the client bundle.
  • sk- and mk- are used for the right jobs. sk- calls inference; mk- reads balance and manages keys. An sk- key on /v1/account/* returns 403 insufficient_scope by design.
  • mk- keys carry the minimum scope. A balance alert needs account:read and nothing else. It does not need keys:create.
  • A rotation procedure is written down. Undocumented is the failure mode, not un-automated.
  • Audit fields are reviewed on a schedule. Each mk- key records last_used_at and last_source_ip on successful authentication, coalesced to at most one write per key per 60 seconds. A management key unused for months is a key you can revoke; an unexpected source IP is worth a rotation.

Both environments call the same base URL:

Terminal window
# Production and staging differ by credential, not by endpoint.
export ZG_BASE_URL="https://router-api.0g.ai/v1"
export ZG_API_KEY="<YOUR_API_KEY>"
export ZG_MANAGEMENT_KEY="<YOUR_MANAGEMENT_KEY>"

Environment separation here is a property of the credential, not of the endpoint: both environments call the same base URL with different keys. Give each environment its own sk- key so that revoking one cannot take the other down, and put a credit_limit on the staging key so a runaway test cannot drain the balance production depends on. If a workload must be sealed, mint the staging and production keys with trust_mode set, and the guarantee holds regardless of which environment’s code is wrong. See API keys.

Rotate without downtime

Section titled “Rotate without downtime”

Both keys are valid during the overlap, which is what makes a rotation a non-event.

  1. Create the replacement key in the Console. The secret is shown once; copy it then.

  2. Write it to your secret store and deploy. Old and new instances are both working, because nothing has been revoked yet.

  3. Confirm the new key is live — traffic under the new sk-, or last_used_at moving on the new mk-.

  4. Revoke the old key. Revocation is immediate: the next request using it returns 401 invalid_api_key.

Step 3 before step 4 is the entire procedure. Revoking first turns a rotation into an outage, and nothing forces you to hold only one valid key at a time.

Resilience

Section titled “Resilience”
  • Every request has an explicit timeout. SDK defaults are usually too generous for a user-facing path and too tight for a long completion. Set one that matches the call.
  • Retries follow the matrix, not a loop. Honour Retry-After on 429, retry 502 briefly with bounded backoff, do not tight-loop 503, and never retry 400, 401, 402, or 403 unchanged. Full policy and code: Handle failover and retries.
  • Failover behaviour is intentional. Default routing already retries across healthy providers. Pinning with X-0G-Provider-Address turns that off unless you also send X-0G-Provider-Allow-Fallbacks: true. See Provider routing headers.
  • Interrupted streams are handled explicitly. A stream that ends without a finish_reason is a partial answer, whatever the text looks like. There is no resume — a retry re-runs the whole completion and is billed again. Discard partials for structured output; mark them truncated for prose.
  • request_id is logged on every failure. It is in the error body and in x_0g_trace.request_id on success, and it is the only handle support has, because the Router does not store prompts.
  • Unknown response fields are tolerated. A client that rejects unrecognized JSON keys breaks on a routine additive release.
const client = new OpenAI({
  apiKey: process.env.ZG_API_KEY,
  baseURL: process.env.ZG_BASE_URL ?? "https://router-api.0g.ai/v1",
  timeout: 120_000,
  maxRetries: 0, // classify by error.code instead; see the failover recipe
});

Cost and balance

Section titled “Cost and balance”

Providers compete on price, so a model’s cost is a range and a failover can move you within it. Balance is a cliff, not a slope: at zero, inference stops.

  • Keys carry a spending limit where one makes sense. A limit can be set when the key is created in the Console, which bounds the damage a runaway loop or a leaked staging key can do.
  • Price ceilings are set where cost matters. X-0G-Provider-Max-Price-Usd-Prompt and -Completion on chat, -Image on image endpoints. The ceiling is a hard filter that runs before sorting and failover, so an outage can never fail you over to a provider you priced out. See Cap what a single request may cost and Provider routing headers.
  • 400 no_provider_within_max_price has defined handling. An empty pool is structural, not transient. Raise the ceiling or change model; do not retry.
  • Per-request cost lands in your logs. Every non-streaming response carries x_0g_trace.billing with input_cost, output_cost, and total_cost for that exact call. It is the only per-request cost record you get, and logging provider alongside it is what lets you set a ceiling from data later.
  • Balance is polled on a schedule and alerts before zero. GET /v1/account/balance with an mk- key carrying account:read. Alert on runway — balance divided by recent burn — rather than on a fixed number, which stops meaning anything the moment traffic doubles.
  • 402 insufficient_balance has a defined behaviour. Queue, degrade, or page a human. Silently dropping user requests is the worst of the three, and retrying is not an option: no amount of backoff creates funds.
const trace = (res as unknown as { x_0g_trace: { request_id: string; provider: string; billing: { total_cost: string } } }).x_0g_trace;
logger.info("inference", {
  request_id: trace.request_id,
  provider: trace.provider,
  total_cost: trace.billing.total_cost,
});

A balance alert is a scheduled job, not a per-request call:

#!/bin/sh
# Poll every few minutes with a management key. An sk- key returns 403 insufficient_scope.
curl https://router-api.0g.ai/v1/account/balance \
  --max-time 30 \
  --fail-with-body \
  --silent \
  -H "Authorization: Bearer $ZG_MANAGEMENT_KEY"

Explicit trust mode

Section titled “Explicit trust mode”

With no trust mode set, the Router balances across tiers for performance. That is the right default for most traffic and the wrong one for a workload with a stated privacy or verifiability requirement — the difference has to come from your request.

  • Trust mode is stated, not inherited. Send X-0G-Provider-Trust-Mode on the request, or set the mode on the key so every request made with it is constrained regardless of what the calling code sends. Key-level enforcement is stronger, because it does not depend on each caller remembering. See Trust modes and Provider routing headers.
  • The tier matches the requirement. Tiers are a floor, so asking for verified is also satisfied by a private provider.
  • The model has a provider in that tier. Not every model does. Choosing a model that has a qualifying provider does not by itself guarantee you get one — the header or the key setting is what guarantees it.
  • 503 no_provider_for_trust_mode is handled without downgrading. The Router never silently serves a weaker tier; it fails instead. Retry slowly or switch model, but do not retry with the header removed.
  • Verification is a gate where it needs to be. verify_tee: true returns x_0g_trace.tee_verified, and false means a signature was present and failed. Treat it as untrusted, not as a warning. See Verify a response came from the enclave.

Before you flip traffic over

Section titled “Before you flip traffic over”
  • Staging ran against the real Router with a staging key long enough to see a 429 and a 5xx, and handled both.
  • Dashboards break error rate down by error.code. That is the axis that matters, because a 429 and a 503 need opposite responses.
  • A runbook exists for 402, 429, and 503. Each has a different fix and none of them is restarting the service.
  • Someone other than the author has read this list against the actual deployment.

Common errors

Section titled “Common errors”
Status code Cause Fix
401 invalid_api_key Old key revoked before the replacement was confirmed live Deploy the new key first, revoke second
402 insufficient_balance Deposit exhausted Fund the account; do not retry
403 insufficient_scope An sk- key called /v1/account/*, or an mk- key lacks account:read Use an mk- key with the right scope
400 no_provider_within_max_price Price ceiling emptied the candidate pool Raise the ceiling or change model
503 no_provider_for_trust_mode No supply in the requested tier Wait or switch model; do not drop the header

Full list with retry guidance: Error codes.

Next steps

Section titled “Next steps”