Linux CPU Isolation + NOHZ Full Playbook (Tickless Cores for Low-Latency Workloads)

Date: 2026-03-18
Category: knowledge

Why this matters

For latency-sensitive services, the biggest tail spikes are often not your app logic. They are background kernel noise on the same core:

• scheduler ticks,
• RCU callback work,
• random kthreads,
• IRQ bursts from unrelated devices.

If your hot thread shares cores with that noise, p99/p999 jitter can stay ugly even when average CPU usage looks fine.

CPU isolation is about creating two worlds:

1. housekeeping cores for general OS work,
2. isolated cores for critical threads with minimal interruption.

1) Mental model: isolate work, not just CPU percentages

A core can be 30% busy and still terrible for low latency if that 30% is fragmented by interrupts and scheduler activity.

Think in terms of interruption budget:

• how often does your hot thread get descheduled?
• how many interrupts hit its core?
• how much runnable-queue contention exists on that core?

The objective is simple:

• keep critical threads on isolated cores,
• push periodic/background kernel activity to housekeeping cores,
• keep IRQ and RCU noise off the critical set.

2) Core building blocks

nohz_full=

Enables full dynticks on target CPUs so periodic scheduler ticks are minimized when one runnable task is active.

rcu_nocbs=

Offloads RCU callback processing from target CPUs to RCU kthreads (typically on housekeeping CPUs).

IRQ affinity

Pins hardware interrupts to housekeeping cores so isolated cores are not interrupted by unrelated device traffic.

cpuset/cgroup pinning

Pins the application’s hot threads to isolated cores and prevents accidental co-location.

Housekeeping cores

A small set of non-isolated cores that absorb:

• IRQ handling,
• unbound kernel workqueues,
• miscellaneous OS tasks.

3) Practical baseline topology

For an 8-core machine (0-7), a common low-risk pattern:

• Housekeeping: 0-1
• Isolated/latency cores: 2-7

Why reserve 2 housekeeping cores?

• one core can saturate under IRQ bursts,
• two gives room for OS + network + storage background work,
• reduces collapse risk when the system is busy.

4) Boot-time baseline (example)

Example kernel cmdline concept:

nohz_full=2-7 rcu_nocbs=2-7 irqaffinity=0-1

Notes:

• irqaffinity=0-1 biases unmanaged IRQs to housekeeping cores.
• Keep this conservative first; tune after observing real workload behavior.
• isolcpus= can be used in special cases, but cpuset/cgroup-based placement is generally easier to operate and adjust.

5) Runtime placement and affinity

Pin critical process/threads

# Example: pin process to isolated cores
taskset -cp 2-7 <pid>

Verify CPU placement

ps -eLo pid,tid,psr,comm | egrep "<your_process>|ksoftirqd|rcu|migration"

You want your hot threads on isolated CPUs consistently, with OS background threads mostly on housekeeping CPUs.

6) IRQ hygiene checklist

Check interrupt distribution:

cat /proc/interrupts

If NIC/storage IRQs are landing on isolated CPUs, move them.

Manual example:

# Pin a specific IRQ (example IRQ 123) to housekeeping CPUs 0-1
echo 0-1 | sudo tee /proc/irq/123/smp_affinity_list

Also verify NIC queue steering (RSS/XPS/RPS) is not reintroducing traffic onto isolated cores unintentionally.

7) Observability: what to watch

Scheduling pressure

cat /proc/pressure/cpu

Softirq pressure indicators

cat /proc/softirqs

Tick/interrupt noise and placement

cat /proc/interrupts

Per-thread CPU stickiness

ps -eLo pid,tid,psr,comm | sort -k3n

Success pattern:

• critical threads stay on isolated CPUs,
• IRQ/softirq growth concentrates on housekeeping CPUs,
• fewer unexpected migrations/deschedules,
• p99/p999 latency narrows.

8) Safe rollout plan

1. Measure baseline
   • p50/p95/p99 latency, context switches, interrupt distribution.
2. Introduce housekeeping + nohz_full + rcu_nocbs on canary hosts
3. Apply IRQ affinity discipline
4. Pin critical threads explicitly
5. Load test busy-window behavior
   • include burst traffic and storage activity.
6. Scale gradually by service class
7. Document rollback
   • return to previous kernel cmdline + affinity profile quickly.

9) Common footguns

1. Isolating too many cores

   • housekeeping set becomes overloaded, causing indirect jitter everywhere.

2. No thread pinning

   • kernel isolation settings alone do not guarantee app thread placement.

3. Ignoring IRQ routing

   • one misrouted high-rate IRQ can destroy isolation gains.

4. Changing too many knobs at once

   • makes regressions impossible to attribute.

5. Assuming one profile fits all hosts

   • NIC/storage topology and workload mix change optimal core partitioning.

10) What “good” looks like

After tuning, you should observe:

• lower runqueue jitter for critical threads,
• tighter tail latency (especially p99/p999),
• fewer involuntary deschedules on isolated cores,
• stable behavior during burst windows.

If tail improves but throughput drops too much, reduce isolation aggressiveness (more housekeeping capacity) before abandoning the model.

Closing

CPU isolation is a latency control system, not a one-time kernel flag tweak.

Treat nohz_full + rcu_nocbs + IRQ affinity + thread pinning as one coordinated stack.
When that stack is aligned, tail latency usually improves more than any micro-optimization inside the application code.