---
name: nginx-reverse-proxy-ssl
description: "Deploy nginx as a reverse proxy for multiple backend services with per-domain vhost configs, automatic Let's Encrypt HTTPS via acme.sh, and 90-day auto-renewal with reload hooks. Load when the user asks for nginx reverse proxy + SSL + multi-domain HTTPS termination, or wants to add a new domain to an existing nginx-fronted stack."
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux]
metadata:
  hermes:
    tags: [nginx, reverse-proxy, ssl, letsencrypt, acme, https, multi-domain]
---

# nginx Reverse Proxy + Let's Encrypt (multi-domain)

End-to-end procedure for putting one or more backend services behind nginx with automatic HTTPS, on a single host that already has nginx installed and one domain already working (so the conventions of the existing vhost — config dir, cert dir, webroot, ACME cron — are established and you just need to follow them).

## When to load this

- User adds a new domain and asks "set up reverse proxy + SSL + auto-renew"
- User has multiple backends (e.g. `api1.example`, `api2.example` → different upstreams) and wants each on its own domain with its own cert
- User wants to front an existing Docker service (sub2api, new-api, etc.) with HTTPS
- Service exposes WebSocket / SSE / streaming (chat APIs, log tails) — proxy config must preserve connection upgrades and disable buffering
- User asks "deploy frps" / "set up frp server" / "expose a tunnel server" — for the dashboard, this skill covers the nginx + acme.sh side; see `references/frps-deployment.md` for the frps-specific bits

## Pre-flight (always do these before touching config)

1. **Confirm no CDN in front.** Curl `http://<domain>/` with a known Host header. If you see `Server: cloudflare` or a 301 from Cloudflare IPs — STOP and ask. Behind CF, use Cloudflare Origin Certificate (long-lived, no renew), not Let's Encrypt. Verified confusion trap on 2026-07-05 (user thought "no Cloudflare" but logs showed CF redirect; re-checked: it was nginx itself).
2. **Find the existing convention.** Look at `/etc/nginx/conf.d/*.conf` (or `sites-enabled/`). One file per domain. Certs in `/etc/nginx/ssl/<host>.{crt,key,fullchain.crt}`. Webroot at `/var/www/acme-challenge/<token>`. acme.sh at `/usr/local/bin/acme.sh` with cron `*/6h`. **Follow it** — don't introduce a parallel convention.
3. **`nginx -t` works** before you do anything. If it complains, do NOT proceed.
4. **Snap memory + listener state.** Save the current config so revert is one `cp` away: `cp <file> <file>.bak.<timestamp>`.

## The five-step deployment

### 1. Write vhost config to `/etc/nginx/conf.d/<host>.conf`

Two server blocks per file: HTTP (port 80) for ACME + force-redirect, HTTPS (port 443) for the actual proxy. Pattern (verified working 2026-07-05):

```nginx
server {
    listen 80;
    listen [::]:80;
    server_name <host>;                     # CRITICAL: exact name — see pitfalls

    location /.well-known/acme-challenge/ {
        root /var/www/acme-challenge;
        default_type "text/plain";
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name <host>;

    ssl_certificate     /etc/nginx/ssl/<host>.fullchain.crt;
    ssl_certificate_key /etc/nginx/ssl/<host>.key;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_ciphers         HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 1d;

    client_max_body_size 50m;               # adjust for upstream (chat APIs need big bodies)

    location / {
        proxy_pass http://127.0.0.1:<backend-port>;
        proxy_http_version 1.1;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host  $host;

        # Streaming/SSE — required for AI chat API relays
        proxy_buffering    off;
        proxy_cache        off;
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }
}
```

### 2. Place a placeholder cert so `nginx -t` passes before signing

```bash
mkdir -p /etc/nginx/ssl
openssl req -x509 -nodes -days 1 -newkey rsa:2048 \
  -keyout /etc/nginx/ssl/<host>.key \
  -out /etc/nginx/ssl/<host>.crt \
  -subj "/CN=<host>"
```

Then `nginx -t && systemctl reload nginx`. After reload, the new domain should:
- `http://<host>/` → 301 to `https://<host>/`
- `https://<host>/` → certificate error (expected — self-signed)
- The host header must reach only this vhost, not the old default site

### 3. Sign the cert with acme.sh

```bash
# Default CA may be ZeroSSL — switch once globally:
/usr/local/bin/acme.sh --set-default-ca --server letsencrypt

# Issue (webroot mode — placeholder cert at :80 already serves the challenge path):
/usr/local/bin/acme.sh --issue --force -d <host> -w /var/www/acme-challenge

# Install (copies to /etc/nginx/ssl/ and registers reload hook):
/usr/local/bin/acme.sh --install-cert -d <host> --ecc \
  --cert-file       /etc/nginx/ssl/<host>.crt \
  --key-file        /etc/nginx/ssl/<host>.key \
  --fullchain-file  /etc/nginx/ssl/<host>.fullchain.crt \
  --reloadcmd       "systemctl reload nginx"
```

`--ecc` produces an ECC cert (smaller, faster). LE signs both. Stick with one mode per host — mixing ECC and RSA creates two separate acme.sh state dirs (`<host>_ecc/` vs `<host>/`).

**If the issue fails:** look at `~/.acme.sh/<host>*/` for the most recent attempt's stderr. Common causes:
- "no AAAA record" — fix the DNS first, wait for propagation, retry
- "timeout during connect" — firewall on 80 (LE requires :80 reachable from outside)
- "challenge mismatch" — webroot path is wrong; see pitfalls section below

### 4. Verify

```bash
nginx -t && systemctl reload nginx
openssl x509 -in /etc/nginx/ssl/<host>.crt -noout -issuer -subject -dates
curl -kS https://<host>/api/status         # -k if local CA bundle is missing
openssl s_client -servername <host> -connect <host>:443 </dev/null 2>&1 | grep "Verify return code"
```

Expect: `issuer=... Let's Encrypt`, dates ~90 days out, `Verify return code: 0 (ok)`.

### 5. Confirm auto-renew is in place

```bash
crontab -l | grep acme.sh       # should exist and run every 6h
grep ReloadCmd ~/.acme.sh/<host>*/<host>.conf  # should print systemctl reload nginx
```

If the cron doesn't exist, install it: `crontab -l > /tmp/cron && echo '40 1,7,13,19 * * * /usr/local/bin/acme.sh --cron --home "/root/.acme.sh" > /dev/null' >> /tmp/cron && crontab /tmp/cron`.

## Pitfalls (verified 2026-07-05)

### Brand-new subdomain is not auto-wildcarded at the registrar

Before running `acme.sh --issue -d <new-sub>.example.com`, verify DNS resolves. A common false assumption: "the parent domain works, so any subdomain works." It does not — `*.example.com` is rarely a wildcard zone. Each new sub needs its own A (or AAAA) record. Quick check before issuing:

```bash
dig +short <host> @8.8.8.8      # external resolver, no local cache
nslookup <host>                  # verbose
```

If empty / NXDOMAIN: stop, ask the user to add the A record at their registrar, wait ~60 s, retry. Don't burn `--issue` attempts on broken DNS. See `references/troubleshoot.md` → "ACME issue returns NXDOMAIN" for the verified-2026-07-05 transcript and cleanup steps.

**Verified 2026-07-05 (lapi.spannerjun.top + quota.spannerjun.top):** even after `dig +short <host> @8.8.8.8` returns the right IP from your local resolver, **Let's Encrypt's authoritative DNS resolvers can lag the local cache by 30-180 s**. The first `--issue` call often returns `NXDOMAIN looking up A for <host>`, and a second `--issue` call a few minutes later succeeds with no further user action. Concretely, this fails-then-works pattern is normal during the first ~3 minutes after the user adds an A record. **Don't burn 5+ retry attempts in fast succession** — sleep ~60 s between retries and accept that the first attempt may fail. Only treat a sustained `NXDOMAIN` (>5 min) as a real DNS error.

### nginx: the `listen 80 default_server` pitfall

Multiple vhosts all on `:80`. Nginx picks the **default server** by:
1. Explicit `default_server` flag (highest priority)
2. **First `listen 80` in config load order** otherwise

If you have an older vhost like mihomo.conf that was the first `listen 80` before your new file, **it is the default** and silently intercepts requests for ANY hostname — returning its own `return 301 https://$host`. You will see this as "my new domain 301s to https:// itself but the wrong cert" or "curl returns 200 welcome-page for unknown hosts."

**Fix:** ensure `/etc/nginx/sites-enabled/default` exists with `listen 80 default_server;` symlinked from `sites-available/default`. The default site serves only the ACME challenge path (or 404 if you follow the simpler pattern below). This makes the default site the explicit default, demoting the older vhost to "only handles requests for its server_name."

**Simpler pattern that avoids the whole trap:** in every per-domain `*.conf`, give the 80 server block an explicit `server_name <host>;` (never `_`). Add **only** one server with `default_server` — the sites-enabled/default that handles 80 fallback cleanly. Verified nginx config:
```
/etc/nginx/conf.d/mihomo.conf            # 80 server_name mihome.spannerjun.top
/etc/nginx/conf.d/2api.spannerjun.top.conf   # 80 server_name 2api.spannerjun.top
/etc/nginx/conf.d/api.spannerjun.top.conf    # 80 server_name api.spannerjun.top
/etc/nginx/sites-enabled/default         # listen 80 default_server → 404
```
After reload, each Host header lands on its own block. Confirmed `curl -H "Host: 2api.spannerjun.top" http://127.0.0.1/` returns 301 from the right vhost.

### nginx: `location /dir/ { root /x; }` resolves root + URI, NOT root

`location /.well-known/acme-challenge/ { root /var/www/acme-challenge; }` makes nginx look for `/var/www/acme-challenge/.well-known/acme-challenge/<token>`, not `/var/www/acme-challenge/<token>`. acme.sh writes the token at `<webroot>/.well-known/acme-challenge/<token>` by default. Mismatch → 404 → validation fails.

**Three ways to fix:**
- (A) `location /.well-known/acme-challenge/ { root /var/www; }` — root is one level up so root + URI hits `/var/www/.well-known/acme-challenge/<token>`. **Cleanest.**
- (B) Use `acme.sh --nginx` mode (acme.sh parses nginx config, sets up challenge paths correctly).
- (C) Use `acme.sh --standalone` mode (acme.sh binds :80 itself, nginx temporarily off). Like mihomo — but requires nginx downtime.

The host already using webroot mode successfully means (A) is the path of least resistance if you're adding a second ACME-aware vhost.

### nginx: curl can't verify cert, "SSL certificate problem"

Server cert is fine (Let's Encrypt chain). Client issue — local CA bundle missing. Use `curl -k` for an HTTP-status probe, or `openssl s_client` to check the chain. Don't try to "fix" the server.

### nginx: `openssl s_client` returns `Verify return code: 21 (unable to verify the first certificate)`

This is **not** the same problem as the previous pitfall. Code 21 means the server is shipping only the **leaf** certificate, not the full chain. Browsers tolerate this because they cache LE intermediates from other sites; `openssl s_client` and strict HTTP clients don't. Two ways to land at code 21:

1. The vhost's `ssl_certificate` directive points at `<host>.crt` (leaf) instead of `<host>.fullchain.crt` (leaf + LE intermediate). The Step 1 template above uses `.fullchain.crt` precisely to avoid this — but older vhosts written before this update will have the leaf-only path.
2. `acme.sh --install-cert` was missing the `--fullchain-file` flag, so `fullchain.crt` was never written.

Fix per vhost:

```bash
# Confirm the file exists and is bigger than the leaf (chain + leaf ≈ 4–5 KB on ECC)
wc -c /etc/nginx/ssl/<host>*.crt /etc/nginx/ssl/<host>*.fullchain.crt 2>/dev/null

# Point nginx at the fullchain
sed -i 's|/etc/nginx/ssl/<host>.crt;|/etc/nginx/ssl/<host>.fullchain.crt;|' \
  /etc/nginx/conf.d/<host>.conf
nginx -t && systemctl reload nginx

# Verify
echo | openssl s_client -connect <host>:443 -servername <host> 2>&1 | grep "Verify return"
# expect: Verify return code: 0 (ok)
```

Verify-after-rotate-loop: `acme.sh --renew --force` writes a new cert, runs reloadcmd, but if reloadcmd fails (e.g. nginx config error), nginx keeps the OLD config and continues serving whatever cert was loaded last — including a leaf-only chain if you'd never switched. Always `nginx -t` after renewals in the first 24h of a new vhost, until you've confirmed a clean `s_client` cycle through a real expiry.

### acme.sh: `--issue` defaults to ZeroSSL since v3.x

ZeroSSL requires EAB credentials and an email. For a minimal install with no prior account, signing fails with `Please add '--debug' or '--log' to see more information`. Run this **once** before your first `--issue`:

```bash
/usr/local/bin/acme.sh --set-default-ca --server letsencrypt
```

After that, all `--issue` calls go to Let's Encrypt.

### acme.sh: `--install-cert` can half-fail and leave nginx serving a placeholder/zero-byte cert

Verified 2026-07-05 during quota.spannerjun.top rollout. The chain looks like this:

1. You ran `acme.sh --issue -d <host>` — succeeded.
2. You ran `acme.sh --install-cert ... --cert-file /etc/nginx/ssl/<host>.crt ...` — `--install-cert` requires the leaf `.cer` to exist at `~/.acme.sh/<host>*/<host>.cer`. If the issue was issued as **staging** first then re-issued to prod, or state dir was cleared mid-flight, the leaf file is missing. `--install-cert` exits non-zero **after** it has already opened `/etc/nginx/ssl/<host>.crt` for writing — leaving a **0-byte file** behind.
3. `systemctl reload nginx` runs as the reloadcmd. Master is fine, but workers re-read config: `ssl_certificate` now points at a 0-byte file. nginx emits `SSL_CTX_use_PrivateKey_file("/etc/nginx/ssl/<host>.key") failed ... BIO_new_file() failed`. Workers in this branch **will refuse to handle TLS for the affected vhost** while still serving HTTP/80 and any other healthy vhost on 443.
4. The reloadcmd exit code can be misleading: nginx prints "signal process started" before the workers actually re-parse — a `[notice]` line — and a **deferred** `bind() ... failed` error in `nginx -t` mode only surfaces when you run `nginx -t` explicitly. `nginx -s reload` exit code alone won't catch it.

**Symptoms in practice:**

```text
$ curl -kS https://<host>/api/health        # returns 200 from another vhost via SNI fallback / wrong cert
$ curl -S https://<host>/api/health         # 502 — handshake fails
$ nginx -t
nginx: [emerg] SSL_CTX_use_PrivateKey_file(...) failed
$ systemctl status nginx                    # Active: active (running) — but `Process: ... ExecReload=... (code=exited, status=1/FAILURE)`
```

`Active: active (running)` is **misleading** — the master is alive, but the failed vhost branch has no workers reading it. The whole service *looks* up.

**Recovery (verified 2026-07-05):**

```bash
# 1. Confirm and clean the zero-byte cert
ls -la /etc/nginx/ssl/<host>.crt /etc/nginx/ssl/<host>.fullchain.crt
# If .crt or .fullchain.crt is 0 bytes, regenerate the placeholder first:
openssl req -x509 -nodes -days 1 -newkey rsa:2048 \
  -keyout /etc/nginx/ssl/<host>.key \
  -out /etc/nginx/ssl/<host>.fullchain.crt \
  -subj "/CN=<host>"
chmod 600 /etc/nginx/ssl/<host>.key
nginx -t && systemctl reload nginx
# 2. Diagnose why the install-cert failed
cat /tmp/acme-install.log 2>/dev/null
# Look for "cat: ...cer: No such file or directory"  → state dir mismatch
# Look for "Couldn't get title from dir" → permission denied
# 3. Reinstall cert cleanly
/usr/local/bin/acme.sh --install-cert -d <host> --ecc \
  --cert-file       /etc/nginx/ssl/<host>.crt \
  --key-file        /etc/nginx/ssl/<host>.key \
  --fullchain-file  /etc/nginx/ssl/<host>.fullchain.crt \
  --reloadcmd       "systemctl reload nginx"
# 4. Confirm workers actually picked up the new cert
echo | openssl s_client -connect <host>:443 -servername <host> 2>&1 | grep -E "Verify return|issuer="
```

**General rule:** treat `--install-cert` exit code as **the source of truth**, not `nginx -s reload`'s exit code. If the install-cert exit was nonzero, run it again before reloading nginx. Adding `set -e` to a deployment script is not enough — `--install-cert` exits mid-write before validating the cert-chain end-to-end, so `--reloadcmd "nginx -t && systemctl reload nginx"` instead of plain reload is the safe form.

### acme.sh: `--install-cert` reloadcmd expects exact service name

`--reloadcmd "systemctl reload nginx"` works when systemd unit is `nginx`. Verify: `systemctl status nginx` shows `Loaded: loaded (/lib/systemd/system/nginx.service; ...)`. If user has `nginx-extras` or custom build, the unit name differs.

### acme.sh: ECC + RSA state dirs don't share

Each cert lives in `~/.acme.sh/<host>/` (RSA) or `~/.acme.sh/<host>_ecc/` (ECC). If you ran an earlier issue without `--ecc`, then this one with `--ecc`, you'll have **both** state dirs. The `--install-cert` only touches the dir matching the flag you pass. Delete the unused one if you don't want both renewals.

### vhost: `server_name` with trailing dot or wildcard

Use FQDN: `2api.spannerjun.top`. Don't include port, scheme, wildcards (`*.example.com` needs DNS-01 mode, not webroot).

### Multi-domain stack: each host in its own `.conf` file

Don't try to share server blocks across hosts even when they proxy to the same upstream. nginx's SNI routing on TLS is per-cert, and sharing causes ghost 421 Misdirected Request errors when a client sends SNI for `host-A` but a stale connection.

### Write tools get blocked on `/etc/nginx/*`, `/etc/letsencrypt/*`, etc.

`write_file` / `patch` will refuse paths under system config dirs. Workaround: write via `terminal` + heredoc:

```bash
cat > /etc/nginx/conf.d/<host>.conf <<'NGINX_CONF'
... full vhost content ...
NGINX_CONF
```

`terminal` `mkdir`, `cp`, `tee`, `openssl`, `nginx -t`, `systemctl reload` all work fine. Only the abstract write/edit layer has the path heuristic.

## Files

- `references/vhost-template.conf` — copy-paste template for one domain, fill in three placeholders (`<host>`, `<backend-port>`, optionally `client_max_body_size`)
- `references/issue-and-install.sh` — the exact acme.sh incantation, idempotent
- `references/troubleshoot.md` — symptom → root-cause → fix cookbook for the top failure modes
- `references/frps-deployment.md` — companion guide: installing frps (or any public-TCP-plus-localhost-dashboard service) and fronting its dashboard through this same nginx + acme.sh stack. Includes v0.69 config-field rename pitfalls, systemd hardening, and a verified skeleton.
- `templates/frps.toml` — placeholder frps config (fill `__AUTH_TOKEN__`, `__DASH_USER__`, `__DASH_PASSWORD__`)
- `templates/frps.service` — systemd unit with hardening directives

## Related skills

- `host-inventory` — capture host state before / after deploy so future sessions know which files were added.
- `hermes-memory-strategy` — durable facts about hosts belong in hindsight; do NOT spam built-in memory with per-domain config details.
- `hermes-agent` — for raising config limits if memory hits a hard cap during a multi-step deploy.

## Related

- Caddy (`caddy`, `caddy-server`) automates all of this for free if you're starting from scratch. Reach for nginx only when it's already on the host.
