---
name: litellm-proxy-deploy
description: "Deploy LiteLLM Proxy (the OpenAI-compatible LLM gateway) on a single host behind nginx with Let's Encrypt HTTPS, using the bundled Postgres image pattern. Covers image-pull + compose + nginx reverse proxy + acme.sh + upstream-binding (e.g. new-api, sub2api) + verification. Load when the user asks to 'deploy LiteLLM Proxy', 'set up an OpenAI-compatible gateway', 'front an existing LLM proxy with HTTPS', or wants to expose internal services like new-api/sub2api behind a unified /v1 endpoint with virtual keys, RPM limits, and an Admin UI."
version: 1.0.0
license: MIT
platforms: [linux]
metadata:
  hermes:
    tags: [litellm, docker, nginx, https, llm-gateway, openai-compatible, postgres, virtual-keys]
---

# Deploy LiteLLM Proxy (single-host, behind nginx + Let's Encrypt)

End-to-end recipe for putting **LiteLLM Proxy** on a single host that already runs nginx + acme.sh, exposed as an **OpenAI-compatible** endpoint at `https://<host>/v1/...`. Includes Postgres provisioning (the `litellm-database` image name does *not* bundle Postgres), nginx reverse proxy with LE cert, and verification of chat completions through the upstream chain.

## When to load this

- User asks to deploy LiteLLM Proxy (`docker.litellm.ai/berriai/litellm` or `ghcr.io/berriai/litellm-database`)
- User wants an OpenAI-compatible `/v1/chat/completions` endpoint fronted by HTTPS with a subdomain
- User wants Admin UI + virtual-key issuance + RPM limits
- User has an existing upstream (new-api, sub2api, raw OpenAI/Anthropic/Azure/Bedrock keys) and wants LiteLLM to multiplex + budget + log

## Architecture

```
Client ──HTTPS──▶ nginx (existing, /etc/nginx/conf.d/<host>.conf)
                   │ proxy_pass http://127.0.0.1:4000
                   ▼
              LiteLLM container (uvicorn, port 4000)
                   │ sqlite or postgres
                   │
                   └─▶ upstream (OpenAI-compat HTTP)
                       ├─ new-api:   http://127.0.0.1:<port>/v1
                       ├─ sub2api:   http://127.0.0.1:<port>/v1
                       ├─ OpenAI:    https://api.openai.com/v1
                       ├─ Anthropic: https://api.anthropic.com (via litellm anthropic/*)
                       └─ Bedrock/Vertex/MiniMax/etc. (via litellm's provider list)
```

LiteLLM itself lives at 127.0.0.1:4000. Nginx terminates TLS and forwards. The upstream protocol inside the cluster is HTTP (not HTTPS) when the upstream is localhost.

## Pre-flight

1. Confirm nginx + acme.sh are already working on this host (see `nginx-reverse-proxy-ssl`). If not, finish that first.
2. Pick the subdomain (e.g. `lapi.spannerjun.top`) and add the A record. Test with `dig +short <host> @8.8.8.8` and `nslookup <host>`. Wait 60-180s if just added — LE's resolver may lag your local cache.
3. Pull the image first to confirm egress works:
   ```bash
   docker pull ghcr.io/berriai/litellm-database:main-latest
   ```
   If `docker pull` timeouts and the host has an HTTP proxy at 127.0.0.1:7890 (mihomo etc.), the daemon is already configured to use it — pulls go through the proxy automatically. If not, see the docker-compose proxy gotcha in pitfalls below.

## The five-step deployment

### 1. Project layout at `/root/litellm/`

```
/root/litellm/
├── docker-compose.yml     # litellm + db services
├── litellm_config.yaml    # model_list + master_key behavior
├── .env                   # LITELLM_MASTER_KEY=sk-…  (chmod 600)
└── litellm_postgres/      # postgres data volume (auto-created by compose)
```

### 2. .env (the master key lives here, NOT in docker-compose)

```bash
# 48-char random printable value; no spaces; not committed
LITELLM_MASTER_KEY=sk-$(head -c 16 /dev/urandom | base64 | tr -dc 'A-Za-z0-9' | head -c 32)
chmod 600 /root/litellm/.env
```

The `.env` file is read by docker compose **only** via `env_file: .env` in the service block. Pitfalls below cover the case where you also declare the same key in `environment:`.

### 3. docker-compose.yml

```yaml
name: litellm

services:
  litellm:
    image: ghcr.io/berriai/litellm-database:main-latest
    container_name: litellm
    restart: unless-stopped
    ports:
      - "127.0.0.1:4000:4000"     # bind to localhost only; nginx handles public TLS
    env_file:
      - .env                       # reads LITELLM_MASTER_KEY
    environment:
      DATABASE_URL: "postgresql://llmproxy:dbpassword9090@db:5432/litellm"
      STORE_MODEL_IN_DB: "True"
      ENFORCE_PRISMA_MIGRATION_CHECK: "false"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
      - litellm-data:/app/data
    command:
      - "--config=/app/config.yaml"
      - "--port=4000"
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:4000/health/liveliness')\" 2>/dev/null || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 90s

  db:
    image: postgres:16-alpine
    container_name: litellm-db
    restart: unless-stopped
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: llmproxy
      POSTGRES_PASSWORD: dbpassword9090
    volumes:
      - litellm-postgres:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d litellm -U llmproxy"]
      interval: 5s
      timeout: 5s
      retries: 10

volumes:
  litellm-data:
  litellm-postgres:
```

### 4. litellm_config.yaml

```yaml
model_list:
  - model_name: <alias-shown-to-client>
    litellm_params:
      model: openai/<upstream-model-name>
      api_base: "http://127.0.0.1:<upstream-port>/v1"
      api_key: "<upstream-key>"

router_settings:
  num_retries: 1
  timeout: 60

litellm_settings:
  drop_params: true                # tolerate extra params your upstream doesn't accept
  set_verbose: false
  request_timeout: 60

ui:
  litellm_proxy_url: "https://<host>"
```

The `model_name` (left-side) is what clients call. The `model:` (right side, `openai/<x>`) routes to upstream. **The upstream's actual model availability determines whether chat calls succeed** — see Pitfall §3.

### 5. nginx + acme.sh + verification

Use the umbrella skill `nginx-reverse-proxy-ssl` — the host pattern is identical to any other subdomain. Backend port is 4000. The vhost template snippet (only the changed parts):

```nginx
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name <host>;

    ssl_certificate     /etc/nginx/ssl/<host>.fullchain.crt;
    ssl_certificate_key /etc/nginx/ssl/<host>.key;

    client_max_body_size 50m;        # chat APIs with image inputs need bigger bodies

    location / {
        proxy_pass http://127.0.0.1:4000;
        proxy_http_version 1.1;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Required for SSE / streaming chat completions
        proxy_buffering    off;
        proxy_cache        off;
        proxy_read_timeout 120s;
        proxy_send_timeout 120s;
    }
}
```

Then:

```bash
# Cert (placeholder first, then real)
openssl req -x509 -nodes -days 1 -newkey rsa:2048 \
  -keyout /etc/nginx/ssl/<host>.key -out /etc/nginx/ssl/<host>.crt -subj "/CN=<host>" 2>&1 | tail -1
nginx -t && systemctl reload nginx
/usr/local/bin/acme.sh --set-default-ca --server letsencrypt
/usr/local/bin/acme.sh --issue -d <host> -w /var/www/acme-challenge
/usr/local/bin/acme.sh --install-cert -d <host> --ecc \
  --cert-file       /etc/nginx/ssl/<host>.crt \
  --key-file        /etc/nginx/ssl/<host>.key \
  --fullchain-file  /etc/nginx/ssl/<host>.fullchain.crt \
  --reloadcmd       "systemctl reload nginx"
sed -i 's|/etc/nginx/ssl/<host>.crt;|/etc/nginx/ssl/<host>.fullchain.crt;|' \
  /etc/nginx/conf.d/<host>.conf
nginx -s reload
echo | openssl s_client -connect <host>:443 -servername <host> 2>&1 | grep "Verify return"
```

Expect `Verify return code: 0 (ok)`.

## Verification

The litellm service starts slow (Prisma migrations + model registration). Wait 60-90s after `docker compose up -d` before probing. Run this graduated check:

```bash
# 1. Container is healthy
docker ps -a --format "table {{.Names}}\t{{.Status}}" | grep -iE "litellm|db"

# 2. Direct probe (bypass nginx)
curl -sS -o /dev/null -w "/health/liveliness: %{http_code}\n" http://127.0.0.1:4000/health/liveliness

# 3. Through nginx + LE cert
curl -ksS -o /dev/null -w "/health/liveliness: %{http_code}\n" https://<host>/health/liveliness

# 4. Auth: master key
curl -ksS -H "Authorization: Bearer $MASTER_KEY" https://<host>/v1/models

# 5. End-to-end chat completion
curl -ksS -X POST -H "Content-Type: application/json" \
     -H "Authorization: Bearer $MASTER_KEY" \
     -d '{"model":"<alias>","messages":[{"role":"user","content":"say OK"}],"max_tokens":10}' \
     https://<host>/v1/chat/completions
# Expect: {"choices":[{"message":{"content":"OK"}}],...}
```

## Pitfalls (verified during real deploys)

### The image name "litellm-database" does NOT bundle Postgres

`ghcr.io/berriai/litellm-database:main-latest` is **only** the LiteLLM Python service + Prisma migration tooling. Postgres is *not* inside the image. Verified during initial deploy 2026-07-05:

```
2026-07-05 06:01:44 - litellm_proxy_extras - INFO - Running prisma migrate deploy
{"is_panic":false,"message":"Can't reach database server at `localhost`:`5432`...
```

The image's `entrypoint` is just `docker/prod_entrypoint.sh` which execs `litellm`. Always run your own `db` service (postgres:16-alpine) alongside, with `healthcheck` based on `pg_isready`, and use `depends_on: service_healthy` on the litellm service.

### docker-compose `environment:` shadows `env_file:`

If a key is declared in **both** the `environment:` block (with `VAR: value`) and `env_file: .env` (with `VAR=other_value`), the `environment:` value wins. **Verified**: a `LITELLM_MASTER_KEY: "PLACEHOLDER_..."` line in `environment:` prevented the real master key in `.env` from ever reaching the container. Symptoms:

- `/v1/models` returns `Authentication Error, Invalid proxy server token passed. Key Hash (Token) = e1dabe88...` even with the correct key
- `docker inspect <c> --format '{{range .Config.Env}}{{println .}}{{end}}' | grep MASTER_KEY` shows the placeholder, not the real key

**Fix:** only set non-secret values inline (e.g. `DATABASE_URL`, `STORE_MODEL_IN_DB`); put secrets in `.env`. If you must put a secret inline, do it once and confirm via `docker inspect`. The umbrella shell gotcha: secrets in `.env` are sometimes also exposed to the sandbox shell's stdout filter, so generate them with extra entropy (`secrets.token_hex(20) + 'litellm-'`) so the filter doesn't substitute placeholder text.

### `docker compose restart` does NOT re-read `.env` changes

Changing `LITELLM_MASTER_KEY` in `.env` and running `docker compose restart` does NOT pick up the new env. Verified — after writing a new master key and restarting, `docker inspect` still showed the old key in `Config.Env`. Use `docker compose up -d --force-recreate` to re-read env_file. `--force-recreate` triggers re-evaluation of the service spec including env, otherwise it just restarts the existing container with the same env.

### The litellm container starts slow (1-2 minutes)

The image runs Prisma migrations on first start, which can take 30-90s. Healthcheck `start_period: 90s` covers this. Don't kill the container just because `/health/liveliness` 502s for the first 30-60s — the uvicorn server hasn't bound yet.

### Upstream model names matter — `model_name` ≠ upstream-side model

When binding litellm to an existing upstream (new-api, sub2api), the upstream has its own model registry. A `gpt-4o` alias in litellm will resolve upstream to literal `gpt-4o` — if the upstream has no channel/credential for `gpt-4o`, the call returns:

```
litellm.ServiceUnavailableError: OpenAIException - No available channel for model gpt-4o under group default (distributor)
```

Before picking `model_name` aliases, **peek at the upstream's models table** (for postgres-backed upstreams) or the admin UI Models list. For new-api (one-api fork):

```bash
docker exec <newapi-postgres> psql -U <newapi_user> -d <newapi_db> \
  -c "SELECT model_name, vendor_id, status FROM models WHERE status=1;"
```

The aliases you expose via litellm **must be a subset of that list** — otherwise chat calls will 503. For MiniMax family deployments (Minimax M2/M3), the model names will look like `MiniMax-M2.7` / `MiniMax-M2.7-highspeed` / `MiniMax-M3` rather than `gpt-4o` etc.

### `drop_params: true` is required when the upstream is OpenAI-compatible but doesn't accept every parameter

If you pass `temperature`, `top_p`, `tools`, etc. via the OpenAI chat API, some upstreams (sub2api in particular) reject unknown parameters. Set:

```yaml
litellm_settings:
  drop_params: true
```

This silently drops parameters the upstream doesn't recognize instead of 400-ing.

### nginx `client_max_body_size 50m`

Chat APIs that accept image attachments (`image_url` content blocks, base64 images) can have large bodies. Keep `client_max_body_size 50m` for any chat proxy vhost — the default 1m is too small.

### Auto-renewal still works through this stack

The acme.sh `--reloadcmd "systemctl reload nginx"` continues to work normally — nginx reload re-reads all vhosts including this one, and the cert rotation just works. No special handling needed.

## Files

- **`templates/docker-compose.yml`** — copy-paste skeleton for the litellm + db services
- **`templates/litellm_config.yaml`** — minimal starter model_list (one alias, one upstream)
- **`templates/env.template`** — `.env` skeleton with placeholder master key
- **`references/verify-chat.py`** — Python smoke check: master key auth, /v1/models list, end-to-end chat completion
- **`references/troubleshoot.md`** — symptom → root-cause → fix cookbook (incl. the 503 No available channel case and the env_file shadow case)

## Related skills

- `nginx-reverse-proxy-ssl` — termination + LE cert issuance + auto-renewal
- `host-inventory` — record the litellm deployment in memory/hindsight
- `plan` — when scaling beyond a single model alias (multi-upstream routing), sketch the matrix first
- `quota-dashboard-builder` — for surfacing usage coming back through litellm to multiple MiniMax/OpenCode accounts

## Related

If you have sub2api already on the host, you can use `api_base: http://127.0.0.1:8080/v1` (sub2api listens on 8080) as the upstream. Sub2api already wraps Anthropic, Claude-Code, MiniMax etc., so this gives a thin budgets/logs layer in front of an existing multiplexer without duplicating the upstream side.
