Linux kTLS (Kernel TLS) Deployment Playbook

Date: 2026-03-28
Category: knowledge
Audience: Infra/platform engineers optimizing high-TPS TLS services on Linux

1) What kTLS actually gives you

kTLS moves the TLS record data path (encrypt/decrypt of application records) into the Linux kernel after userspace TLS handshake is done.

Important boundary:

• Handshake, cert validation, session negotiation: still userspace TLS library
• Record protection for application data (TX/RX): kernel path (software or NIC offload)

So kTLS is not "TLS in kernel from start"; it is a post-handshake data-path optimization.

2) Why operators adopt kTLS

Primary wins:

• Fewer userspace↔kernel copies in hot path
• Better integration with sendfile()-style delivery
• Potential CPU savings at high throughput/connection counts
• Optional NIC offload path where hardware supports it

Most realistic gains show up for:

• static/object-heavy HTTPS egress,
• large streaming responses,
• high-concurrency edge/API workloads where TLS record work is material.

3) Modes you need to reason about

From Linux kernel docs, kTLS can run in:

1. TLS_SW (software crypto in kernel)
2. TLS_HW (packet-based NIC TLS offload)
3. TLS_HW_RECORD (full TCP offload mode; usually not desirable for general Linux network-stack features)

Operationally, you typically target:

• TLS_SW first (portable baseline),
• then selective TLS_HW on supported NICs/drivers.

4) Preconditions checklist (before rollout)

4.1 Kernel + userspace stack readiness

• Linux with kTLS support and relevant cipher support
• TLS library/runtime with kTLS integration (e.g., OpenSSL features in your deployed version)
• App/server hooks to enable kTLS on established sockets

4.2 Cipher/protocol reality

kTLS support is cipher/protocol-path dependent. Don’t assume every negotiated suite lands on kTLS. Measure actual enable ratio under production cipher mix.

4.3 Network/NIC readiness (for hardware offload)

• NIC/driver support for tls-hw-tx-offload / tls-hw-rx-offload
• Verify offload features via ethtool and runtime counters
• Plan explicit fallback expectations (software path on offload add failure)

5) The biggest correctness foot-gun: zero-copy + mutable files

Kernel docs explicitly warn: zero-copy sendfile-style optimizations require source data to remain immutable until transmission completes.

If data changes mid-flight, retransmissions may carry different bytes, causing receiver auth failures (looks like record tampering).

Rule:

• Enable sendfile/zerocopy paths only for immutable content (or tightly controlled write discipline).

6) TLS 1.3 KeyUpdate behavior you must monitor

With kTLS RX path, when TLS 1.3 KeyUpdate arrives, kernel can pause decryption until userspace installs new RX key material.

Practical symptoms:

• reads can fail with EKEYEXPIRED
• read readiness may pause until updated keys are set

If your TLS stack/app isn’t robust around this, you can create intermittent stalls under long-lived connections.

7) Observability that separates "enabled" from "useful"

Minimum production signals:

• /proc/net/tls_stat counters (SW/HW TX/RX sessions, decrypt errors, rekey stats, retry stats)
• app-side kTLS activation ratio by protocol/cipher
• CPU per request byte, especially at p95/p99 traffic mixes
• TLS error taxonomy (EBADMSG, EMSGSIZE, EKEYEXPIRED patterns)
• offload fallback rate (attempted HW vs actually installed HW)

If you can’t answer "what percentage of bytes used kTLS vs userspace TLS?" you’re flying blind.

8) Rollout strategy (safe and boring wins)

Phase 0 — Instrument first

• Add kTLS on/off labels in metrics before enabling globally
• Baseline CPU, latency, TLS error rates

Phase 1 — TX-only canary

• Start with TX path on one service slice
• Keep conservative cipher policy
• Validate no error-rate regressions and real CPU improvement

Phase 2 — Controlled RX enablement

• Expand to RX path after KeyUpdate/decrypt behavior is proven
• Stress long-lived and bidirectional flows

Phase 3 — Hardware offload pilots

• Enable only on known-good NIC fleet segment
• Track software fallback and resync/decrypt anomalies
• Keep instant rollback switch

9) Fast diagnostic playbook

Symptom A: "kTLS enabled" but no throughput/CPU gain

Likely causes:

• workload handshake-bound, not record-path-bound
• cipher mix falling back out of kTLS path
• insufficient sendfile/zero-copy applicability

Symptom B: intermittent read failures on long sessions

Likely causes:

• TLS 1.3 KeyUpdate handling gap (EKEYEXPIRED path)
• delayed RX key installation by userspace

Symptom C: hardware offload unstable

Likely causes:

• NIC/driver limitations under reordering/resync cases
• mixed fleet capabilities with silent path variance
• overly aggressive zerocopy assumptions

10) Practical defaults for first production attempt

1. Treat kTLS as a data-path optimization, not a security model change.
2. Launch with TX-focused canary, then broaden.
3. Keep immutable-content discipline before zerocopy/sendfile acceleration.
4. Watch /proc/net/tls_stat + app labels together.
5. Add a one-flag rollback to userspace TLS path.

Do this well and you get measurable efficiency gains without mysterious TLS regressions.

References

1. Linux Kernel Docs — Kernel TLS
   https://docs.kernel.org/networking/tls.html
2. Linux Kernel Docs — Kernel TLS offload
   https://docs.kernel.org/networking/tls-offload.html
3. OpenSSL 3.3 docs — openssl s_server (-ktls, -sendfile, -zerocopy_sendfile)
   https://docs.openssl.org/3.3/man1/openssl-s_server/
4. OpenSSL manpage mirror — SSL_sendfile behavior/availability notes
   https://manpages.opensuse.org/Leap-15.6/openssl-3-doc/SSL_sendfile.33ssl.en.html
5. GnuTLS Manual — kTLS overview and enablement notes
   https://www.gnutls.org/manual/html_node/kTLS-_0028Kernel-TLS_0029.html