Fixing Head-of-Line Blocking: Why Topology Matters More Than Queues

In the previous article, we showed how the CPU port becomes the bottleneck in a DSA-based system.

The obvious fix is “add more queues”.

But that’s not enough.

Because the real problem isn’t the number of queues — it’s what you put into them.

The Classic Mistake: Treating All Traffic Equally

Most multi-queue NIC designs do this:

• take all traffic
• hash it
• spread it across queues

This improves throughput.

But it does not solve head-of-line blocking.

Because fundamentally different traffic still competes.

Not All Traffic Is The Same

On a router, traffic falls into different categories:

• bridged traffic (LAN ↔ LAN)
• routed traffic (LAN ↔ WAN)
• Wi-Fi traffic

These behave differently:

• different latency sensitivity
• different processing paths
• different burst patterns

Putting them into the same queue is the root of the problem.

Step 1: Split by Topology

The first step is not adding queues.

It is separating traffic by topology.

We split traffic into two pipelines:

• bridged traffic
• standalone (routed) traffic

Each pipeline gets its own RX/TX path.

This removes structural contention.

Step 2: Then Distribute by Flow

Only after separation do we distribute traffic:

skb->hash % 2

This ensures:

• flows don’t block each other
• queues stay balanced

Now we have two layers:

• topology separation
• flow distribution

This is fundamentally different from generic multi-queue designs.

Why This Works

This mirrors how real switches behave:

• first classify traffic
• then schedule it

Instead of:

“spread everything and hope for the best”

we do:

“separate first, then distribute”

The Result

• reduced latency spikes
• predictable behavior
• no cross-class blocking

And most importantly:

the CPU port stops being chaotic.

Conclusion

Head-of-line blocking is not solved by adding queues.

It is solved by structuring traffic correctly.

Topology first.

Then flow.