RoCEv2 Congestion Control Playbook (PFC + ECN + DCQCN)

Date: 2026-03-17
Category: knowledge

Why this matters

RoCEv2 can deliver very low latency and high throughput, but only if the congestion loop is tuned as a system, not as isolated knobs.

Most outages in RDMA fabrics are not caused by one broken switch setting; they come from bad interaction between:

• link-level pause (PFC),
• queue marking (ECN/WRED),
• endpoint rate control (DCQCN/TIMELY/HPCC),
• traffic mix (incast + elephants + control traffic).

If this loop is misaligned, you get familiar symptoms:

• p99 latency spikes,
• pause storms,
• unfair throughput across senders,
• congestion spreading and occasional deadlock-like behavior.

1) The mental model: three control planes

Think in layers:

1. Endpoint control (DCQCN/TIMELY/HPCC)

   • controls send rate over time.

2. Queue signal (ECN)

   • marks packets before loss to say “slow down soon”.

3. Emergency brake (PFC)

   • per-priority pause when buffers approach unsafe levels.

Healthy design principle:

Mark before pause, pause before drop.

If PFC fires before ECN can provide early feedback, you end up with pause-heavy behavior and unstable tails.

2) Quick algorithm selection matrix

A) Default, production-safe RoCEv2 in mixed datacenter traffic

Start with: DCQCN + ECN + bounded PFC

Why:

• widely deployed model,
• NIC/switch ecosystem support is mature,
• predictable in large Clos fabrics when thresholds are tuned.

B) RTT-sensitive fabrics with strong timestamp fidelity and NIC support

Try: TIMELY

Why:

• uses RTT signal rather than ECN mark fraction,
• can reduce dependency on ECN-mark behavior quality.

Caveat: noisy RTT measurement can destabilize control.

C) INT-capable fabrics seeking near-zero queue behavior

Try: HPCC

Why:

• uses in-band telemetry for precise load feedback,
• often best latency under heavy incast in supported environments.

Caveat: requires stronger fabric feature assumptions and validation discipline.

3) Baseline observability (before tuning anything)

Collect these per-priority and per-port metrics first:

• ECN mark rate (and mark burstiness),
• PFC pause frames / pause duration,
• queue occupancy percentiles,
• CNP rate (for DCQCN fabrics),
• throughput fairness (e.g., Jain’s index across competing senders),
• FCT p50/p95/p99 for short flows,
• retransmissions / drops (should remain rare in lossless class).

Without this baseline, tuning is guesswork.

4) Tuning order that avoids most failures

Step 1 — Traffic class hygiene

• Put RoCE data and control traffic in intentional classes.
• Ensure CNP/control traffic is never starved by bulk data.
• Confirm DSCP/PCP mapping is consistent end-to-end.

Many "mystery" problems are just classification drift.

Step 2 — ECN thresholds before PFC thresholds

Set ECN/WRED mark region to engage earlier than PFC headroom exhaustion.

Goal:

• endpoint sender backs off from ECN feedback,
• PFC remains a rare safeguard, not the normal operating mechanism.

If PFC activates frequently while ECN marks are low, your marking starts too late.

Step 3 — PFC headroom correctness

• Size headroom for worst-case in-flight bytes during pause propagation.
• Validate per-speed/per-cable assumptions for your topology.
• Keep PFC scope minimal (only required priorities).

Over-broad PFC is how small congestion turns into fabric-wide pain.

Step 4 — Endpoint controller tuning

For DCQCN fabrics:

• start conservative on additive increase,
• avoid overly aggressive recovery rates,
• tune around fairness + p99, not just average throughput.

For TIMELY/HPCC fabrics:

• validate signal quality under production-like load,
• explicitly test noisy feedback and control-loop stability.

5) Practical health states (operator-friendly)

Use a simple state model:

GREEN

• moderate ECN marks,
• low PFC events,
• stable p99 latency and fairness.

Action: normal operation.

AMBER

• rising ECN burstiness,
• occasional PFC bursts,
• p99 starting to wobble under incast windows.

Action:

• reduce sender aggressiveness slightly,
• verify queue/threshold alignment,
• check hot-spots in oversubscribed paths.

RED

• persistent PFC storms,
• congestion spreading across ports/classes,
• major p99/FCT degradation and fairness collapse.

Action:

• enter safe profile (lower send ramps / stricter caps),
• quarantine worst offenders,
• validate ECN-before-PFC ordering immediately.

SAFE_RECOVERY

• gradual rollback from emergency profile,
• only after PFC and queue tails normalize.

Action: stepwise recovery with rollback gates.

6) Incast test plan (must-have)

Before rollout, run these repeatedly:

1. N:1 short-flow incast (RPC-like)
2. Elephant + mice coexistence
3. Asymmetric path contention
4. Failure drill (one switch under constrained buffers)

Promotion gate (example):

• p99 FCT improved or unchanged,
• PFC event rate not increased materially,
• no fairness regression beyond agreed bound.

If only throughput improved but p99 worsened, reject.

7) Fast diagnosis cheatsheet

Symptom: PFC high, ECN low

Likely cause:

• ECN threshold too high / mark region too late.

First action:

• move ECN marking earlier; re-check queue traces.

Symptom: ECN very high, throughput collapses, PFC low

Likely cause:

• over-marking / overly aggressive endpoint decrease.

First action:

• relax mark aggressiveness or endpoint decrease dynamics.

Symptom: one sender dominates, others starve

Likely cause:

• unfair endpoint loop parameters,
• class misconfiguration,
• hot-spot hash imbalance.

First action:

• inspect per-flow fairness and hash distribution before changing global thresholds.

Symptom: tail latency random spikes despite good averages

Likely cause:

• incast burst sensitivity,
• control-traffic starvation,
• short-term queue microbursts.

First action:

• isolate short-flow p99 + burst windows, not 1-minute averages.

8) Rollout strategy (safe)

1. Lab calibration with realistic traffic mix.
2. Canary rack(s) with full telemetry.
3. Diurnal validation (at least 24h, ideally 72h).
4. Gradual expansion with automatic rollback trigger.
5. Post-change audit: ensure config drift detection is active.

Never roll congestion policy globally in one shot.

9) Common mistakes

1. Treating PFC as normal flow control

   • PFC is emergency protection, not daily traffic shaping.

2. Tuning only average throughput

   • user pain lives in p99 tails and fairness collapse.

3. Ignoring classification correctness

   • perfect thresholds cannot fix wrong traffic class mapping.

4. No canary rollback criteria

   • congestion changes without hard rollback gates are outage bait.

5. Cross-vendor knob translation by name only

   • same label can have different semantics; validate behavior, not wording.

10) Minimal default stance (good starting point)

If you need a pragmatic baseline:

• RoCE class: lossless + PFC enabled, but tightly scoped.
• ECN: enabled with marking thresholds below PFC-trigger range.
• Endpoint: DCQCN default profile, conservative ramp.
• Ops: always track ECN/PFC/CNP/FCT p99 together.

Then iterate with measured evidence.

References

• RFC 3168 — The Addition of ECN to IP: https://datatracker.ietf.org/doc/html/rfc3168
• RFC 8257 — Data Center TCP (DCTCP): https://datatracker.ietf.org/doc/html/rfc8257
• SIGCOMM 2015 — DCQCN (Congestion Control for Large-Scale RDMA Deployments): https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p523.pdf
• SIGCOMM 2015 — TIMELY: RTT-based Congestion Control for the Datacenter: https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p537.pdf
• SIGCOMM 2019 — HPCC: High Precision Congestion Control: https://dl.acm.org/doi/10.1145/3341302.3342085
• IEEE 802.1Qbb (Priority-based Flow Control): https://1.ieee802.org/dcb/802-1qbb/
• NVIDIA Cumulus RoCE operational guide (example vendor implementation details): https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-42/Network-Solutions/RDMA-over-Converged-Ethernet-RoCE/
• Juniper DCQCN operational notes (ECN/PFC interaction): https://www.juniper.net/documentation/us/en/software/junos/traffic-mgmt-qfx/topics/topic-map/cos-qfx-series-DCQCN.html