The Header Trust Problem in Split-Horizon Topologies

Split-horizon DNS failover breaks the implicit assumption that your load balancer or reverse proxy is the sole source of client IP information. When internal clients resolve a public domain to a private IP, and external clients resolve it to a public IP, the traffic paths diverge. If Caddy trusts headers like X-Forwarded-For indiscriminately, an attacker who can reach the backend directly can spoof their IP address. This is not a theoretical risk; it is a header trust misconfiguration that exposes your infrastructure to log poisoning and rate-limit bypasses.

The core issue is that Caddy must distinguish between trusted upstream proxies and untrusted clients. In a split-horizon setup, the “trusted” set of proxies may change during failover events. If your configuration does not explicitly define which IPs are allowed to set forwarding headers, Caddy will either ignore legitimate headers (breaking geo-blocking or audit logs) or accept forged headers (breaking security controls). This article details how to configure Caddy’s trusted_proxies directive to handle these dynamic scenarios safely.

For context, this guidance applies to Caddy 2.x and later, where the trusted_proxies directive was introduced to replace the older, less granular header handling. The asset reference for this runbook is 20260909c, which documents the baseline configuration patterns for this specific topology.

Configuring Caddy’s trusted_proxies Correctly

Caddy’s trusted_proxies directive defines a list of IP addresses or CIDR ranges that are permitted to set forwarding headers. By default, Caddy trusts no proxies unless explicitly configured. This is the correct default: trust should be explicit, not implicit.

In a split-horizon DNS failover scenario, you must account for both internal and external proxy IPs. For example, if your internal load balancer uses 10.0.0.5 and your external load balancer uses 203.0.113.10, both must be listed. If you omit the internal IP, Caddy will ignore the X-Forwarded-For header from internal traffic, causing the backend to see the internal load balancer’s IP instead of the original client’s IP.

Here is a minimal Caddyfile configuration that demonstrates correct usage:

example.com {
    reverse_proxy backend:8080 {
        trusted_proxies 10.0.0.5 203.0.113.10
    }
}

Key points:

• Explicit IPs only: Avoid using 0.0.0.0/0 or broad CIDR ranges unless absolutely necessary. Broad ranges increase the attack surface for header spoofing.
• Failover awareness: If your failover mechanism rotates proxy IPs dynamically, consider using a DNS name that resolves to the current proxy IP, or use a script to update the Caddy configuration and reload it during failover events.
• No implicit trust: Caddy does not infer trust from the network interface or routing table. You must explicitly list every trusted proxy.

If you are using a service mesh or a service discovery mechanism, ensure that the proxy IPs reported by the mesh are included in the trusted_proxies list. Otherwise, the mesh’s forwarding headers will be ignored, and you will lose visibility into the original client IP.

Handling X-Forwarded-For and X-Real-IP Headers

Caddy processes X-Forwarded-For and X-Real-IP headers only if the connection originates from a trusted proxy. If the connection does not come from a trusted proxy, Caddy overwrites these headers with the actual client IP. This behavior is critical for security: it prevents untrusted clients from injecting arbitrary values into these headers.

However, this behavior can lead to confusion if your upstream application (e.g., a WordPress site or a Python API) expects to see the original client IP in X-Forwarded-For but receives the proxy IP instead. This happens when the proxy is not listed in trusted_proxies. To avoid this, ensure that all proxies in the chain are trusted.

Consider a chain: Client → External LB (203.0.113.10) → Internal LB (10.0.0.5) → Caddy → Backend. In this case, both 203.0.113.10 and 10.0.0.5 must be in trusted_proxies. If only 203.0.113.10 is trusted, Caddy will see the connection from 10.0.0.5, which is not trusted, and will overwrite X-Forwarded-For with 10.0.0.5. The backend will then see 10.0.0.5 as the client IP, losing the original client IP.

To handle this correctly:

1. Identify all proxies in the chain.
2. List all of them in trusted_proxies.
3. Verify that the upstream application reads X-Forwarded-For (not REMOTE_ADDR) to determine the client IP.

If you cannot trust all proxies in the chain (e.g., due to a compromised intermediate proxy), consider using a different header or a custom protocol that includes a signed token to verify the client IP. This is outside the scope of Caddy’s built-in features but is a common pattern in high-security environments.

Validating Upstream Behavior During Failover

During a split-horizon DNS failover, the set of trusted proxies may change. For example, if the external LB fails over to a new IP, the trusted_proxies list must be updated to include the new IP. If you do not update the list, Caddy will ignore the forwarding headers from the new LB, and the backend will see the new LB’s IP instead of the original client IP.

To validate upstream behavior during failover:

1. Monitor proxy IPs: Use a monitoring tool to track the IPs of your load balancers. If an IP changes, trigger a configuration update.
2. Automate configuration updates: Use a script to update the Caddy configuration and reload it. For example, a cron job or a systemd timer can check for changes in the proxy IP list and apply them.
3. Test failover scenarios: Simulate a failover event and verify that the backend receives the correct client IP. Use a test client to send requests with known X-Forwarded-For headers and check the backend logs.

Example script to update trusted_proxies and reload Caddy:

#!/bin/bash
# Get current proxy IPs from a config file or API
NEW_IPS=$(cat /etc/proxy_ips.txt)
# Update Caddyfile
sed -i "s/trusted_proxies.*/trusted_proxies $NEW_IPS/" /etc/caddy/Caddyfile
# Reload Caddy
caddy reload --config /etc/caddy/Caddyfile

Ensure that the script is idempotent and that the Caddyfile backup is retained in case of errors. If the reload fails, Caddy will continue serving traffic with the old configuration, which is a safe fallback.

Common Misconfigurations and Pitfalls

Several common misconfigurations can lead to header trust issues in split-horizon topologies:

• Forgetting internal proxies: Only listing external proxy IPs and omitting internal proxies. This causes the backend to see internal proxy IPs instead of client IPs.
• Using broad CIDR ranges: Listing 10.0.0.0/8 instead of specific IPs. This allows any host in the internal network to spoof headers, increasing the risk of log poisoning.
• Not updating during failover: Failing to update the trusted_proxies list when proxy IPs change. This causes the backend to see the wrong client IP after failover.
• Mixing REMOTE_ADDR and X-Forwarded-For: Some applications read REMOTE_ADDR for the client IP, which will always be the proxy IP. Ensure your application reads X-Forwarded-For instead.
• Ignoring header overwriting: Assuming that Caddy will preserve X-Forwarded-For headers from untrusted proxies. Caddy overwrites these headers, so untrusted proxies cannot inject arbitrary values.

To avoid these pitfalls, use a configuration management tool (e.g., Ansible, Puppet) to manage the Caddy configuration. This ensures that the trusted_proxies list is consistent across all servers and is updated automatically during failover events.

Verification Steps for Disposable Assets

Before deploying a new Caddy configuration to production, verify it in a disposable environment. This prevents breaking production traffic due to a misconfigured trusted_proxies list.

Steps to verify:

1. Set up a test environment: Deploy Caddy and a backend application in a disposable VM or container. Configure the test environment to mimic your production split-horizon topology.
2. Configure trusted_proxies: Add the test proxy IPs to the trusted_proxies list.
3. Send test requests: Use curl to send requests with known X-Forwarded-For headers. For example:

   curl -H "X-Forwarded-For: 192.168.1.100" http://test.example.com/api

4. Check backend logs: Verify that the backend logs show 192.168.1.100 as the client IP. If it shows the proxy IP, the trusted_proxies list is incorrect.
5. Test untrusted proxies: Send requests from an untrusted IP with a forged X-Forwarded-For header. Verify that the backend logs show the untrusted IP, not the forged IP. This confirms that Caddy is overwriting the header correctly.

If the verification steps pass, deploy the configuration to production. If they fail, debug the trusted_proxies list and the upstream application’s header handling before proceeding.

Limitations: This runbook assumes that your upstream application reads X-Forwarded-For for the client IP. If your application uses a different mechanism (e.g., a custom header or a signed token), you will need to adjust the configuration accordingly. And, this runbook does not cover certificate management, TLS termination, or other Caddy features. For those topics, refer to the Caddy documentation or a dedicated runbook.