# LiteLLM Proxy — troubleshoot cookbook

Captured during real deploys 2026-07-05. Five symptoms that bite fresh
deploys and how to recover quickly.

## 1. Container is healthy but `/health/liveliness` returns 502

**Symptom (nginx perspective):**

```
$ curl -ksS https://<host>/health/liveliness
<html><head><title>502 Bad Gateway</title></head>...
```

**Symptom (direct probe):**

```
$ curl -sS http://127.0.0.1:4000/health/liveliness
{"status":"ok"}
```

nginx → 502, direct → 200 means **the litellm container IS up**, but nginx can't reach it. Causes in order of likelihood:

1. **Port mismatch.** Vhost proxies to `127.0.0.1:4000` but compose binds elsewhere. Verify:
   ```bash
   docker inspect litellm --format '{{(index .HostConfig.PortBindings "4000/tcp")}}' 2>/dev/null
   ss -tln | grep :4000
   ```
2. **Wrong network mode.** If you used `network_mode: host` and bind to `0.0.0.0`, nginx should reach it; if you used bridge without `ports:` mapping, nginx can't. Re-check the compose file.
3. **Container restarted and is still starting.** `start_period: 90s` covers Prisma migrations, but on first start it can take 60-90s before uvicorn binds. Watch `docker logs --tail 20 litellm` until you see "Application startup complete".

## 2. `Authentication Error, Invalid proxy server token passed`

```
{"error":{"message":"Authentication Error, Invalid proxy server token passed.
Received API Key = sk-...c..., Key Hash (Token) =9d51772b...
Unable to find token in cache or LiteLLM_VerificationTokenTable...","code":"401"}}
```

**Fix in this order:**

1. **Confirm the master key in the container matches what you're sending.**

   Read `Config.Env` from the container:
   ```bash
   docker inspect litellm --format '{{range .Config.Env}}{{println .}}{{end}}' \
     | grep LITELLM_MASTER_KEY
   ```
   The output should NOT be a placeholder. If it shows `PLACEHOLDER_...` or any literal placeholder, your `environment:` block is shadowing `.env` — see Pitfall in the umbrella SKILL.md.

2. **Confirm `.env` has a real key:**
   ```bash
   head -1 /root/litellm/.env   # LITELLM_MASTER_KEY=<value>
   stat -c '%a' /root/litellm/.env   # 600
   ```
   If the file is missing or mode != 600, fix:
   ```bash
   printf 'LITELLM_MASTER_KEY=sk-RealKeyHere\n' > /root/litellm/.env
   chmod 600 /root/litellm/.env
   ```
   Then re-create the container (NOT just restart):
   ```bash
   docker compose up -d --force-recreate
   ```

3. **Recreate the master-key entry directly in Postgres** if you suspect stale state (this is rare):
   ```bash
   docker exec -i litellm-db psql -U llmproxy -d litellm -c "
     INSERT INTO \"LiteLLM_VerificationToken\" (key, user_id, max_budget, spend, expires, blocked, \"key_name\")
     VALUES ('sk-RealKeyHere', NULL, 1e30, 0, NULL, false, 'master-key')
     ON CONFLICT DO NOTHING;
   "
   ```

If the master key is fine but `Invalid proxy server token passed` persists for a different virtual key, **the virtual key row doesn't exist in `LiteLLM_VerificationToken`** — recreate it via `/key/generate`:

```bash
curl -X POST -H "Authorization: Bearer ${MASTER}" \
  -H 'Content-Type: application/json' \
  -d '{}' \
  https://<host>/key/generate
```

## 3. `503 No available channel for model X under group default (distributor)`

**Symptom (chat completion via litellm):**

```
{"error":{"message":"litellm.ServiceUnavailableError: ServiceUnavailableError:
OpenAIException - No available channel for model gpt-4o under group default (distributor)
(request id: 202607...). Received Model Group=gpt-4o\nAvailable Model Group Fallbacks=None"}}
```

This means litellm successfully **authenticated** with the upstream, but the upstream (new-api, sub2api, …) says **it doesn't have a channel registered for the literal model name you used**.

**Diagnosis:** look at the upstream's actual model list. For new-api (one-api fork) over Postgres:
```bash
docker exec <newapi-pg> psql -U <user> -d <db> \
  -c "SELECT model_name, vendor_id, status FROM models WHERE status=1;"
```

For sub2api, hit its admin API after authenticating with the admin cookie.

**Fix:** change `model:` in litellm_config.yaml to a name the upstream actually exposes. Common cases on 2026-07-05:
- Upstream exposes only MiniMax family (`MiniMax-M2.7`, `MiniMax-M3`, etc.), not `gpt-4o`
- Upstream has `gpt-4o-mini` but you asked for `gpt-4o`
- Upstream exposes sub-namespaced names like `azure/gpt-4o-deployment-1` which need a deploy-side config, not a `model:` rewrite

After editing `litellm_config.yaml`, you must `docker compose restart litellm` (NOT `up -d` — restart is enough because the config is bind-mounted).

**Prevention:** Before writing `model_list` aliases, peek at the upstream's model table and use exact names. The model's `name` is what clients see; the upstream's `model` is what resolves to a channel — they only need to match if you want clients to send the same string.

## 4. nginx 502 immediately after `acme.sh --install-cert` succeeds

Already covered in `nginx-reverse-proxy-ssl/SKILL.md` → `--install-cert can half-fail`. Brief recap specific to litellm:
- `--install-cert` opens the destination file before validating state. If `~/.acme.sh/<host>_ecc/<host>.cer` is missing (e.g. you re-issued after a `--issue --test`), install-cert writes 0 bytes to `/etc/nginx/ssl/<host>.crt`, leaves it there, then runs the reloadcmd.
- `systemctl reload nginx` succeeds because the master is fine. Workers re-read configs, fail to load the 0-byte cert, log `SSL_CTX_use_PrivateKey_file(...) failed`, drop the affected vhost silently. Public HTTPS for that host 502s.
- **Recovery**: regenerate the placeholder self-signed, re-run install-cert, verify with `openssl s_client`.

## 5. `verify return code: 21 (unable to verify the first certificate)` after a successful install-cert

Also covered in umbrella skill. Recap: the vhost's `ssl_certificate` directive points at `<host>.crt` (leaf only) instead of `<host>.fullchain.crt`. Browsers tolerate this; `openssl s_client` and strict clients don't.

```bash
sed -i 's|/<host>.crt;|/<host>.fullchain.crt;|' /etc/nginx/conf.d/<host>.conf
nginx -s reload
echo | openssl s_client -connect <host>:443 -servername <host> 2>&1 | grep "Verify return"
# expected: Verify return code: 0 (ok)
```

## 6. POST /v1/key/generate returns 200 but no `key` field

```json
{"key_alias":null,"duration":null,"models":[],"spend":0.0, ...}
```

Not an error — that's a successfully created key with all defaults (no model restrictions, no budget, no expiry). To get a usable key that's clearly distinguishable in `/key/info`, pass:

```bash
curl -X POST -H "Authorization: Bearer ${MASTER}" \
  -H 'Content-Type: application/json' \
  -d '{"key_alias":"my-app","models":["MiniMax-M3"],"rpm_limit":60}' \
  https://<host>/key/generate
```

The `key` field at the top of the response is the new virtual key. Store it somewhere safe — there's no way to retrieve it again after the create call returns.

## 7. Litellm container starts then crashes with "Couldn't reach database server at `localhost:5432`"

Means `db` service isn't healthy yet when litellm starts Prisma migrate. Two fixes:
1. **Already in template** — `depends_on: db: condition: service_healthy` plus `db.healthcheck` ensure order.
2. **If still happening**: confirm `db` is actually healthy:
   ```bash
   docker inspect litellm-db --format '{{.State.Health.Status}}'
   ```
   If empty or `starting`, the postgres container hasn't initialized yet. Wait, then retry compose up.

## 8. Bizarre intermittent 503s on chat completion

If the upstream is a **single-channel new-api** with a small set of cached model entries, sustained litellm traffic can deplete the quota faster than expected, and the upstream itself returns 503 to litellm. Litellm surfaces this as `litellm.ServiceUnavailableError` or `litellm.APIConnectionError`. Confirm by:

```bash
# Watch new-api container logs while triggering a chat
docker logs -f --tail 50 newapi-app
# If you see "insufficient quota" or "channel not available", the upstream's account itself is the constraint
```

Fix: top up the upstream account, OR add multiple upstreams in litellm config:

```yaml
model_list:
  - model_name: MiniMax-M3
    litellm_params:
      model: openai/MiniMax-M3
      api_base: "http://127.0.0.1:3000/v1"
      api_key: "sk-account-A..."
  - model_name: MiniMax-M3-alt                 # different alias = different channel
    litellm_params:
      model: openai/MiniMax-M3
      api_base: "http://127.0.0.1:8080/v1"      # different upstream (sub2api)
      api_key: "..."
```

Or use litellm's load-balancing router for one alias across two upstreams (more advanced — see litellm docs on `router_settings.weighted_response_models`).
