Reverse Proxy (Caddy)

Caddy serves as the reverse proxy for all web-facing services. It runs as a Docker container with a custom-built image that includes the Cloudflare DNS plugin for automatic wildcard TLS certificates.

File Locations

File	Purpose
ansible/playbooks/infrastructure/networking/tasks/caddy.yml	Installation and configuration task
ansible/playbooks/infrastructure/networking/templates/Caddyfile.j2	Reverse proxy configuration
ansible/playbooks/infrastructure/networking/templates/compose.yaml.j2	Docker Compose service definition
ansible/playbooks/infrastructure/networking/templates/Dockerfile.j2	Custom xcaddy build with Cloudflare plugin
ansible/environments/<env>/group_vars/all/proxy/*.yml	Manual per-domain service definitions (non-catalog services)
ansible/environments/<env>/group_vars/all/proxy/_services.yml	Registry entry generation and service aggregation
ansible/environments/<env>/group_vars/all/apps.yml	App registry — proxy: blocks generate service entries

Architecture

The Caddyfile is generated from the unified services list. For each domain, a wildcard server block (*.domain.tld) is created with:

1. TLS configuration — Cloudflare DNS-01 ACME challenge for wildcard certificates
2. Per-service matchers — named host matchers (@servicename) route requests to the correct backend
3. Reverse proxy directives — each service gets a handle block with optional headers, encoding, and transport configuration

Only services with proxied: true and enabled: true (default) generate Caddy entries.

*.5am.video {
    tls { dns cloudflare ... }

    @plex host plex.5am.video
    handle @plex {
        reverse_proxy 10.2.0.5:32400
    }

    @sonarr host sonarr.5am.video
    handle @sonarr {
        reverse_proxy 10.2.0.5:8989
    }
}

TLS Certificate Management

Caddy obtains wildcard certificates for each domain using the Cloudflare DNS-01 ACME challenge. This means:

• No ports need to be publicly open for certificate validation
• One wildcard cert covers all subdomains per domain (e.g., *.5am.video)
• Certificates auto-renew before expiry

External traffic reaches Caddy via port forwarding on the UDM Pro (ports 80 and 443).

The custom Docker image is built with xcaddy to include the caddy-dns/cloudflare plugin. Two environment variables authenticate with Cloudflare:

Variable	Source	Purpose
CF_API_TOKEN	SOPS-encrypted secrets	Cloudflare API token with DNS edit permissions
CF_EMAIL	SOPS-encrypted secrets	Cloudflare account email

The Caddy container exposes ports 80, 443, and 2019 (admin API), and mounts persistent volumes for certificate storage.

Service Definition Reference

Service entries come from two sources:

1. App registry — apps in group_vars/all/apps.yml declare a proxy: block, and _services.yml generates their entries automatically. See Registry-Generated Entries.
2. Manual per-domain files — YAML files under ansible/environments/<env>/group_vars/all/proxy/ for services outside the app registry: third-party devices (Proxmox, NAS, KVM, Haivision), the media stack, monitoring, Frigate, and cloud gaming. Each file defines a list variable (e.g., wil_services, video_services) that is aggregated by _services.yml.

The fields below apply to entries from both sources.

Required Fields

---

name

Subdomain name. Combined with the domain to form the FQDN (e.g., plex becomes plex.5am.video).

Type: string

name: plex

---

backend_host

IP address of the backend service.

Type: string

backend_host: 10.2.0.5

---

backend_port

Port number of the backend service.

Type: integer

backend_port: 32400

---

proxied

Controls both DNS resolution and Caddy proxy behavior:

• true — DNS resolves to reverse_proxy_ip, Caddy reverse proxies to the backend
• false — DNS resolves directly to backend_host, no Caddy entry generated

Type: boolean

proxied: true

Optional Fields

---

enabled

Set to false to disable both the DNS record and Caddy entry for this service. Useful for temporarily taking a service offline without removing its definition.

Type: boolean

Default: true

enabled: false

---

dns

Controls DNS A record generation. Set to "external" to skip internal A record creation — useful for services that only need Cloudflare DNS records.

Type: string

Default: "internal"

dns: external

---

tls_skip_verify

Skip TLS certificate verification when proxying to the backend. Use when the backend serves HTTPS with a self-signed certificate (e.g., Proxmox, Kasm).

Type: boolean

Default: false

tls_skip_verify: true

---

forward_headers

Add X-Real-IP, X-Forwarded-For, and X-Forwarded-Proto headers to proxied requests. Enable when the backend needs the client's real IP address.

Type: boolean

Default: false

forward_headers: true

---

host_header

Set to "upstream" to override the Host header with the upstream host and port. Required by services that validate the Host header (e.g., Plex).

Type: string

Default: not set

host_header: upstream

---

encode

Enable response encoding. Reduces bandwidth for content-heavy services.

Type: string

Default: not set

encode: gzip

---

read_buffer

Transport read buffer size in bytes. Increase for services with large response headers or streaming payloads.

Type: integer

Default: not set

read_buffer: 8192

Service Configuration Examples

Minimal service

A basic proxied web application:

- name: tools
  backend_host: 10.2.20.60
  backend_port: 8070
  proxied: true

Media service with headers and encoding

Plex requires header forwarding, host header override, gzip encoding, and an increased read buffer:

- name: plex
  backend_host: 10.2.0.5
  backend_port: 32400
  proxied: true
  encode: gzip
  forward_headers: true
  host_header: upstream
  read_buffer: 8192

Self-signed backend

Proxmox serves HTTPS with a self-signed certificate:

- name: vm
  backend_host: 10.2.20.7
  backend_port: 8006
  proxied: true
  tls_skip_verify: true

DNS-only (non-proxied) service

A service that gets a DNS record pointing directly to its IP, with no Caddy proxy:

- name: hmg
  backend_host: 10.2.20.186
  backend_port: 443
  proxied: false

Registry-Generated Entries

Apps in the app registry (group_vars/all/apps.yml) declare their DNS and proxy configuration inline via a proxy: block:

kasm:
  host_group: app_kasm
  # renovate: datasource=docker
  image: "lscr.io/linuxserver/kasm:1.18.1"
  port: 4443
  proxy:
    name: kasm             # Subdomain → kasm.wil.5am.cloud
    domain: wil.5am.cloud
    proxied: true          # Required — no default in the zone template
    tls_skip_verify: true

_services.yml builds a registry_services list from every proxy: block — on a top-level app or a one-level-nested sub-service (e.g., apps.work.convertx.proxy). For each block:

• backend_host is derived — the first inventory host of the app's host_group. If the group has no hosts in the environment's inventory, the render fails loudly rather than silently dropping the entry.
• backend_port defaults to the sibling port — set backend_port inside the proxy: block to override it.
• name, domain, and proxied are required — proxied has no default in the zone template.
• Optional fields pass through unchanged — tls_skip_verify, forward_headers, host_header, encode, read_buffer, dns, and enabled behave exactly as in manual entries.

Apps without a proxy: block (e.g., openbooks, games-server) get no DNS record and no Caddy entry.

Service Aggregation

The _services.yml file in each environment aggregates the manual per-domain service lists (injecting the domain field) and appends the registry-generated entries, which already carry their domain from the proxy: block. Manual lists come first to keep zone and Caddyfile ordering stable:

WILLDN

services: >-
  {{
    (video_services | default([]) | map('combine', {'domain': '5am.video'}) | list) +
    (cloud_services | default([]) | map('combine', {'domain': '5am.cloud'}) | list) +
    (wil_services | default([]) | map('combine', {'domain': 'wil.5am.cloud'}) | list) +
    (ext_services | default([]) | map('combine', {'domain': 'ext.5am.cloud'}) | list) +
    (sfc_services | default([]) | map('combine', {'domain': 'sfc.al'}) | list) +
    registry_services
  }}

services: >-
  {{
    (ldn_services | default([]) | map('combine', {'domain': 'ldn.5am.cloud'}) | list) +
    registry_services
  }}

A list may aggregate even when no domain file defines it — the default([]) keeps unused lists (e.g., sfc_services in WIL, whose services all live in the registry) as extension points.

Both BIND9 zone templates and the Caddyfile template consume the resulting services list.

Best Practices

Scenario	Configuration
Modern web app with standard HTTP backend	proxied: true (no optional fields needed)
Backend with self-signed HTTPS (Proxmox, Kasm)	Add tls_skip_verify: true
Backend needs client IP (analytics, rate limiting)	Add forward_headers: true
Backend validates Host header (Plex)	Add host_header: upstream
Bandwidth-heavy streaming service	Add encode: gzip
Large response headers or websocket streams	Add read_buffer: 8192 (or higher)
Service only accessible externally via Cloudflare	Add dns: external
Temporarily take a service offline	Set enabled: false

Common Tasks

Add a new service to an existing domain

Registry apps — add a proxy: block to the app's entry in ansible/environments/<env>/group_vars/all/apps.yml:

myapp:
  host_group: app_myapp
  # renovate: datasource=docker
  image: "myimage:latest"
  port: 8080
  proxy:
    name: myapp
    domain: wil.5am.cloud
    proxied: true

backend_host and backend_port are derived from the app's host_group and port — no manual entry is needed.

Non-catalog services (third-party devices, media stack, monitoring) — add an entry to the domain file, e.g., ansible/environments/wil/group_vars/all/proxy/wil.5am.cloud.yml:

- name: myapp
  backend_host: 10.2.20.60
  backend_port: 8080
  proxied: true

Then, for either path:

1. Deploy:

   task ansible:deploy-networking ENV=wil
   

2. Verify:

   # Check DNS
   dig myapp.wil.5am.cloud @10.2.20.53
   
   # Check HTTPS
   curl -I https://myapp.wil.5am.cloud
   

Add a new domain

1. Add the domain to ansible/environments/<env>/group_vars/all/vars.yml:

   domains:
     # ... existing domains
     - name: "new.5am.cloud"
   

2. Create a service file at ansible/environments/<env>/group_vars/all/proxy/new.5am.cloud.yml:

   ---
   new_services:
     - name: app
       backend_host: 10.2.20.60
       backend_port: 8080
       proxied: true
   

3. Update ansible/environments/<env>/group_vars/all/proxy/_services.yml to include the new list:

   services: >-
     {{
       ... +
       (new_services | default([]) | map('combine', {'domain': 'new.5am.cloud'}) | list)
     }}
   

4. Ensure the domain is registered with Cloudflare (required for TLS certificates)

5. Deploy:

   task ansible:deploy-networking ENV=wil
   

Disable a service temporarily

Set enabled: false on the service entry (for registry apps, inside the proxy: block in apps.yml). This removes both the DNS record and the Caddy proxy entry without deleting the configuration:

- name: myapp
  backend_host: 10.2.20.60
  backend_port: 8080
  proxied: true
  enabled: false

Troubleshooting

502 Bad Gateway — Caddy can reach the backend but it's not responding. Check the backend container is running: ssh <host> docker ps. Verify backend_port matches the container's exposed port.

Certificate not issuing — Check Caddy logs: ssh <networking-ip> docker logs caddy. Common causes: Cloudflare API token expired, domain not on Cloudflare, or Let's Encrypt rate limit hit.

Service accessible via IP but not hostname — The DNS record is missing or pointing to the wrong IP. Check with dig <service>.<domain> @10.2.20.53 and verify the service entry in the domain file (or the app's proxy: block in apps.yml for registry apps).

Headers not forwarded — Ensure forward_headers: true is set on the service entry and redeploy networking.