DEV Community

Alex Georgiev
Alex Georgiev

Posted on AI-assisted

Caddy 2.11's default post-quantum key exchange sends six times more handshake bytes

Packet size crosses a TCP boundary

Caddy 2.10 switched on a post-quantum key exchange by default back in April, and I'd assumed, without checking, that "post-quantum" meant "slower". I ran Caddy 2.11.4 against a modern TLS client and measured the handshake instead of guessing.

Post-quantum key exchange matters because a sufficiently capable quantum computer could break the classical Diffie-Hellman maths (X25519, the elliptic-curve algorithm nearly everything uses today) that TLS relies on to agree a shared secret. ML-KEM, standardised by NIST in 2024, resists that attack. Caddy now defaults to a hybrid group called X25519MLKEM768, which runs classical X25519 and ML-KEM-768 side by side and combines both outputs, so a connection stays as safe as before even if one of the two turns out to be broken later.

The usual justification for doing this now, years before a working quantum computer exists, is "harvest now, decrypt later": someone records encrypted traffic today and keeps it until they have the hardware to break the classical key exchange retroactively. Hybrid key exchange closes that off going forward, provided the recording started after the switch. It does nothing for traffic already recorded under a pure classical handshake, which is a reason to check whether your own traffic has actually moved rather than assuming an upgraded proxy took care of it.

I stood up Caddy 2.11.4 in Docker with a self-signed certificate and pointed a client with OpenSSL 3.5.7 at it, since that's the first stable OpenSSL release with native ML-KEM support.

The default is real

$ openssl s_client -connect caddytest:443 -servername caddytest.local | grep -i "negotiated\|protocol"
Negotiated TLS1.3 group: X25519MLKEM768
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Protocol: TLSv1.3
Enter fullscreen mode Exit fullscreen mode

That confirms Caddy's own announcement: with no configuration at all, a capable client and Caddy agree on the hybrid post-quantum group. I didn't have to opt in to anything.

What it actually costs: bytes, not time

ML-KEM-768 public keys and ciphertexts are much larger than an X25519 point (32 bytes), so I expected the handshake messages to grow. I used openssl s_client -msg to print the exact record sizes, once forcing classical X25519 and once forcing the new hybrid group.

Message X25519 (classical) X25519MLKEM768 (default) Growth
ClientHello 318 bytes 1494 bytes 4.7x
ServerHello 122 bytes 1210 bytes 9.9x
Combined 440 bytes 2704 bytes 6.1x
>>> TLS 1.3, Handshake [length 013e], ClientHello        # classical: 0x013e = 318
>>> TLS 1.3, Handshake [length 05d6], ClientHello        # PQC:       0x05d6 = 1494
<<< TLS 1.3, Handshake [length 007a], ServerHello         # classical: 0x007a = 122
<<< TLS 1.3, Handshake [length 04ba], ServerHello         # PQC:       0x04ba = 1210
Enter fullscreen mode Exit fullscreen mode

Six times more bytes for the two messages that carry the key exchange is a real cost, and it lands before the connection is even useful. On a slow or lossy link — mobile, satellite, anything where a TCP segment might not fit the initial congestion window — a bigger ClientHello can mean an extra round trip that wasn't there before. That's the number I'd actually watch if I were running this in front of a mobile-heavy audience, not "did the CPU get slower".

The CPU cost, measured separately, is close to nothing

I didn't want to guess about compute cost from the network test, since network noise swamps anything sub-millisecond. openssl speed exercises the raw KEM operations with no network or process-spawn overhead in the loop:

$ openssl speed -seconds 2 ML-KEM-768
                keygen    encaps    decaps  keygens/s  encaps/s  decaps/s
ML-KEM-768    0.000063s 0.000038s 0.000059s   15789.5   26176.0   16915.0

$ openssl speed -seconds 2 X25519
                keygen    encaps    decaps  keygens/s  encaps/s  decaps/s
X25519        0.000046s 0.000095s 0.000043s   21709.5   10526.5   23497.0
Enter fullscreen mode Exit fullscreen mode

ML-KEM-768's keygen is about 37% slower than X25519's, but its encapsulation step is over twice as fast. Every operation on both sides finishes in under 65 microseconds. That's roughly three orders of magnitude below the millisecond-scale cost of a network round trip, so on a normal connection it's not going to be what you notice.

I also tried to measure this end to end: 50 sequential handshakes over the loopback-speed Docker network, three trials each, forcing one group or the other.

Group Fastest of 3 trials (50 handshakes) Per handshake
X25519 1.038s 20.8 ms
X25519MLKEM768 1.038s 20.8 ms

Identical to the millisecond. I don't trust that as a real "no difference" result — it's process-spawn and TCP-connect overhead completely swamping a sub-100-microsecond crypto operation, which is exactly what the openssl speed numbers predict it should do. The honest reading is that the wall-clock cost of the new default is too small to measure with a per-connection harness like this one, not that it's zero. The byte counts above are the number that's actually attributable to the feature.

Old clients still connect

I wanted to know what happens when a client has never heard of ML-KEM at all, since that's most of the installed base right now. I ran OpenSSL 1.1.1f, released in March 2020, against the same server.

$ openssl version
OpenSSL 1.1.1f  31 Mar 2020
$ openssl s_client -connect caddytest:443 -servername caddytest.local
...
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Verify return code: 20 (unable to get local issuer certificate)
Enter fullscreen mode Exit fullscreen mode

It completed a full TLS 1.3 handshake. The certificate warning is just my self-signed CA not being trusted by that container, unrelated to key exchange. Caddy fell back to a group the old client actually understood without any error, warning, or configuration on my part. Nothing about turning this default on breaks a client that predates it, which is the whole point of a hybrid group over a PQC-only one.

You can't see any of this in Caddy's own logs

This is the part I didn't expect. Caddy's structured JSON access log records a tls object on every request:

"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"caddytest.local","ech":false}
Enter fullscreen mode Exit fullscreen mode

Version and cipher suite are there. Whether Encrypted Client Hello was used is there. The negotiated key-exchange group — the actual subject of this whole feature — is not, in either the default log format or the JSON one shown above. I ran the same request forced onto classical X25519 and onto the PQC hybrid group and diffed the two log lines: aside from timestamps and the source port, they were identical. If you want to know what fraction of your real traffic is landing on the post-quantum group versus falling back to classical, this access log won't tell you. You'd need a packet capture or a change on the client side to find out, which is a strange gap for a feature whose whole point is a migration you'd want to track.

What I got wrong on the way

My first handshake-timing script used openssl s_client piped from echo -n with no HTTP request, expecting it to exit as soon as stdin hit EOF. It didn't — s_client keeps the TLS connection open waiting for the server to close first, and Caddy doesn't close an idle connection quickly. Fifty of those in a loop took over two minutes and I assumed I'd found some bizarre PQC-related slowdown, right up until I ran a single one under timeout 5 and watched it use the entire five seconds doing nothing. Sending a real GET / HTTP/1.1 with Connection: close fixed it: Caddy answers and closes the socket, and the true handshake cost turned out to be about 20 milliseconds, not two-plus seconds.

Run it yourself

This needs Docker and a client built against OpenSSL 3.5 or later (Debian trixie ships it; Ubuntu 24.04's stock OpenSSL, 3.0.13, does not know ML-KEM and will silently never negotiate it).

docker network create pqctest

cat > Caddyfile << 'EOF'
https://caddytest.local:443 {
    tls internal
    respond "hello"
}
EOF
docker run -d --name caddytest --network pqctest \
  -v "$PWD/Caddyfile:/etc/caddy/Caddyfile" caddy:latest

docker run --rm --network pqctest debian:trixie-slim bash -c "
  apt-get update -qq && apt-get install -y -qq openssl >/dev/null
  echo | openssl s_client -connect caddytest:443 -servername caddytest.local 2>&1 | grep 'Negotiated'
  echo | openssl s_client -connect caddytest:443 -servername caddytest.local -msg -groups X25519 2>&1 | grep 'ClientHello'
  echo | openssl s_client -connect caddytest:443 -servername caddytest.local -msg -groups X25519MLKEM768 2>&1 | grep 'ClientHello'
"
Enter fullscreen mode Exit fullscreen mode

If you're running Caddy in front of anything latency-sensitive on constrained links, check your actual ClientHello size rather than assuming the CPU cost is the thing to worry about — it isn't. A six-times-larger ClientHello is unlikely to matter on a normal broadband path, but it's worth a real check if you serve clients on satellite links, congested mobile networks, or anywhere else a single extra round trip shows up in your latency numbers.

And if you need to know how much of your traffic is actually using the new default in production, don't expect Caddy's access log to tell you; you'll have to capture it yourself, with something like tshark watching for the key_share extension in the ClientHello, or a check on the client side before it ever reaches your proxy.

Top comments (8)

Collapse
 
_firelinks profile image
Mike Dabydeen •

The extra round trip has a mechanism, and your own byte count gets most of the way to it without a lossy trace.

1494 bytes of ClientHello, plus the four byte handshake header and the five byte TLS record header, is 1503 bytes of TCP payload before you have added a 20 byte TCP header or a 20 byte IP header. That is already past a 1500 byte Ethernet MTU on its own, so the message no longer fits in one segment. On a PPPoE path at 1492, or a tunnel nearer 1400, it splits further. The classical ClientHello at 318 bytes was never close to this.

What makes that worse than one extra packet is how the loss gets found rather than how many bytes move. The initial window is ten segments under RFC 6928, so two segments are not a window problem. But a split ClientHello is the entire flight, and there is nothing queued behind it to generate duplicate acknowledgements. Fast retransmit needs three of them. RFC 6928 makes the same point in its own justification for the larger window, that it exists partly so loss can be recovered by fast retransmit instead of the initial RTO. At two segments you are on the wrong side of that. Losing either half is found by timeout, on a connection that has no measured round trip time yet, so the fallback is the conservative initial value rather than anything your latency numbers would predict.

On the logging gap I would go one step further than you did. An access log only contains connections that completed. A client or a middlebox that breaks on a two segment ClientHello never reaches Caddy and never becomes a line, so the ratio you want has a third bucket that is invisible by construction, and it is the only one that represents harm. Classical fallback is the success story. The clients that disappeared are the migration risk, and they are only visible from the client side or a capture, which is where you ended up anyway.

Collapse
 
alexgeorgiev17 profile image
Alex Georgiev •

I checked this against a packet capture rather than trusting either set of numbers. One correction: the 1494 I measured already includes the 4-byte handshake header, it's not the body alone. The record's own length field on the wire is 0x05d6 = 1494, and the record header adds 5 more bytes on top, so 1499 total, not 1503. Doesn't change your conclusion, 1499 is still well past 1460, so it still needs two segments on a standard path.

One thing I couldn't confirm here: my capture showed the whole ClientHello as a single 1499-byte packet even though MSS was 1460. That's GSO deferring the real segmentation past where tcpdump grabs it on a Docker bridge, so I can back the byte count directly but not the literal two-packet split, not on this setup anyway.

The logging point is the sharper version of something I only gestured at. Good catch and thanks for joining the post and your time to share your findings mate :)

Collapse
 
_firelinks profile image
Mike Dabydeen •

You're right, and thanks for checking it on the wire. I added the handshake header to a figure that already included it, so 1499 is the number. The conclusion holds, since 1499 is still past 1460, but my arithmetic was wrong.

On the capture, I don't think a Docker bridge will ever show you the split, because nothing on that path needs to make it. GSO holds segmentation back until the packet reaches a device that can't take it whole, and a veth pair into a local bridge can pass it along whole. The two segments only exist once the packet leaves through an interface with a real 1500 MTU.

Two ways to see them: capture on the client with the server on another machine, or turn the offloads off on the sending interface with ethtool -K eth0 tso off gso off so the kernel segments before tcpdump looks. If you capture on the receiving side, turn gro off there too, or it will stitch the halves back into one packet and show you the same illusion from the other end.

Thread Thread
 
alexgeorgiev17 profile image
Alex Georgiev •

Thanks, and no worries on the arithmetic, 1499 is right and everything downstream of it still holds.

You called the Docker point exactly right: a veth pair into a local bridge never reaches a device with a real MTU, so GSO has no reason to ever segment anything there, whatever tcpdump shows. I ended up doing pretty much what you described, rented a small external VM and connected over the real internet instead. First capture on the server side still showed one packet, 1490 bytes, MSS 1460, so still not enough on its own. Turned out GRO on the receiving side was reassembling the halves before tcpdump ever saw them. ethtool -K eth0 gro off gso off tso off on the server fixed it: two distinct segments, 1448 and 42, same total. The classical ClientHello stayed one packet regardless of the flags. That's the split confirmed on a real external path, not inferred and not a container artifact.

Writing it up properly, I'll drop the link here once it's live. Appreciate you pointing at exactly where to look, saved me a few more dead-end captures

Thread Thread
 
_firelinks profile image
Mike Dabydeen •

The 1448 is TCP timestamps. MSS 1460 is announced without options, and with timestamps on (the Linux default) every segment carries 12 bytes of them, so each full segment holds 1448 bytes of payload. Worth one line in the write-up, because readers will see 1448 against a 1460 MSS and wonder where the 12 bytes went.

The other number I'd pin down before publishing: this capture's ClientHello is 1490 and the Docker one was 1499. My guess is the server name. SNI carries the hostname, and the VM's name is probably shorter than the first host's. Stating the hostname, or quoting the record length field from each capture, stops anyone reading the 9 bytes as a measurement error.

Looking forward to the post.

Thread Thread
 
alexgeorgiev17 profile image
Alex Georgiev •

Good catch on both, and one of them's a clean win, the other needs more digging before I'd put a number on it.

1448 = 1460 minus the 12 bytes Linux tacks onto basically every data segment for RFC 7323 timestamps by default (10 bytes value, 2 bytes NOP padding to stay 4-byte aligned). That's exactly where the 12 bytes went, not a mystery once you know timestamps are on, and yeah, worth a line in the post so nobody reads it as an error.

On the 1490 vs 1499, decent instinct on the SNI, but I went back and checked and both tests actually used the literal same -servername caddytest.local, so it looks like the hostnames were already matched going in. Still an open question though, I didn't pin down whether the laptop I used for the droplet test was running the same OpenSSL build as the 3.5.7 I named for the Docker test, and that's probably the more likely suspect now, different client versions can add or drop an extension without much fanfare. Gonna diff the two -msg outputs directly and get a real answer before calling it a measurement error either way. Appreciate you pushing on this instead of letting it slide.

Collapse
 
compoundlabs profile image
Compound Labs •

The byte increase is clear, but the extra-round-trip claim still rests on inference from message sizes. Did you capture a lossy mobile or satellite trace where the hybrid ClientHello crossed a segment or congestion-window boundary?

Collapse
 
alexgeorgiev17 profile image
Alex Georgiev •

No, the post didn't capture that, it was inference from message size. Tested it directly instead: netem on the same Docker link as the repro above, 8% loss, 60ms latency with 10ms jitter, roughly standing in for a rough mobile or satellite path. 40 handshakes each, classical vs the hybrid group.

Classical: 1 of 40 runs over 500ms, worst case 516ms. Hybrid: 8 of 40 over 500ms, two around 2.3 seconds, one at 1.2 seconds. Medians were close, most connections don't hit it, the tail is where it shows up, roughly 8x the stall rate and 4-5x the worst case.

That matches what Mike worked out from the segment math in the other comment: when the second half of a split ClientHello is the one that gets lost, there's nothing behind it to trigger fast retransmit, so it falls back to a timeout. Synthetic netem link, not a captured trace off a real device, but same mechanism, and now it's measured instead of inferred.