Netconf 2006
Tokyo, .jp
Jamal Hadi Salim

Overview

Main presentation focus on

adventures experimenting on changes by Herbert on qdisc path

And other related changes i pursued as a result

If time permits

A quick summary of other WIP i have in some oven or/and freezer

In Pursuit of Herbert Xu

Motivation

Herbert's work on GSO resulted in a subtle change in the qdisc path

Some races were made more obvious by GSO

Worry-ier me has some alarm bells ringing

endless emails

All of Herbert's responses concise and patient

Motivation

But Jamal's gut feeling still not happy

Time constraints on my part

i tried to convince Robert to test (he was busy-ed out elsewhere)

So one summer morning in August (Ontario Civic holiday) Jamal went berserk

then got inspired to go on a tangent ;->

Subject of part of this presentation

Presentation contents

Part 1

Demonstrate how things worked before change

Demonstrate how things work now (2.6.18-rcx)

Quantify with observations

Talk about ensuing experiments based on observation

Part 2

Other WIP related to the above

. . .

Qdisc

Driver tx path

Wire

Taxonomy for Demonstration

a codepath

Yellow CPU executing

on a codepath

lock

(unlocked)

queue

packet enqueue by (blue CPU)

lock acquired by (blue CPU)

Blue CPU in driver txpath

Enqueue

Dequeue/

Requeue

Before changes in 2.6.18

Demonstrate

qdisc already occupied with 3 packets

two cpus coming in with two new packets

To be melodramatic, dont show any failure to grab txlock

which normally results in a requeue

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

Incorporating the new change

Repeat the same scenario as before

qdisc already occupied with 3 packets

two cpus coming in with two new packets

To be melodramatic, dont show any failure to grab txlock

which results in a requeue

represent qdisc_is_running as occupied flag

again trying to be melodramatic to make a point;->

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

OCCUPIED

occupied stops other CPUs from attempting

to dequeue (and hence attempting to grab txlock)

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

. . .

Qdisc

Driver

Wire

Observations: number of locks

old scheme

7 qdisc locks, 7 tx locks

4 by one CPU and 3 by other

new scheme

8 qdisc locks

2 by one CPU 6 by other

7 tx locks

all by one CPU

one bit set and another unset

Observations

Not much difference in the grand scheme of things

i.e the creation of extra lock is mitigated by the fact tx lock is grabbed by only one CPU

cache utilization improved

Indeed, tests at high speed validate this theory

The big change is that a single CPU could theoretically be locked processing tx packets

at the expense of other work it has to do

DOS/starvation of work potential

DOS Test scenario

Note: I said theoretical because i dont have the equipment to produce the test scenario

Assume you have

multiple CPUs each bound to at least one ethernet

all CPUs both receiving and sending a lot of traffic

all CPUs funneling to one ethernet

Then it is possible to lock a single CPU into just processing packets

Lets demonstrate

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

. . .

Qdisc

Driver

Wire

OCCUPIED

Conclusion 1

Could not reproduce the DOS setup in my VIA C3 dual processor

It is probably producible in a setup with many more CPUs

I did not see performance degradation in the case of 2 CPUs

It may be possible to see it in the many CPUs case

Opportunity sometimes knocks twice

Test equipment

IXIA traffic generator

Sends traffic at wire rate @64B

e1000 dual port Gige NIC

Main boards

A dual 1.2Ghz VIA C3 mainboard (VT-310DP)

Not very impressive: PCI on southbridge (VT8237R)

Forwards about 200Kpps

A dual XEON 2.4 Ghz, 32/33Mhz PCI bus ;->

Aptly named speedopolis

With high speed packet input this machine starves xmit

Test setup

eth0

eth1

CPU1

CPU0

IXIA traffic generator

More tests

Bind CPU1 to eth0 and CPU0 to eth1

Created a test tool to collect data on the transmit state transitions

genetlink control and relayfs bulk data collection

observe the tx path state transitions

Observation

eth1

eth0

netif_stop

(by CPU1)

netif_stopped

netif_sched

(by CPU0)

netif scheduled

packets

not going

for DMA

packets

going

for tx DMA

More tests

Essentially interesting attribute is:

When the tx path is shutdown

upto 64 packets could be enqueued in the qdisc before the tx path opens up again

Turns out to be an e1000 change that creates this effect ;->

But aggravated more now by qdisc_is_locking change

This is positive

We know theres always space on tx ring

Attempt to batch when the valve opens

Approach taken

When first packet goes to tx DMA, the return code from the driver returns the amount of space available on ring

Theres contention for semantics of this code (fixable)

The CPU owning qdisc_is_running will dequeue from qdisc up to a number of packets that the tx dma can take then send to the driver

Approach taken

No packets sent to the driver get re-queued

Clearly there's amortization of tx lock as well as qdisc lock

for 64 packets we have 2 queue lock grabs and 1 tx lock as opposed to 65 queue locks and 64 tx locks

Clearly improved cache utilization

Seems like good mitigation for the DOS attack

Demo with the previous DOS slides

Seems like it will improve throughput

Approach taken

Added several netdev methods

hard_xmit_batch(*skbuff list, *netdev)

Batches packets onto ring from qdisc

It is invoked when we have enough space in the TX ring

We find out by return code from hard_xmit()

hard_prep_xmit(*skb, *netdev)

All it does is massage the skb ready to go into ring

invoked by both hard_xmit() and hard_xmit_batch()

Added a netdev->blist used to carry batched skbs

Results!

VIA C3 dual processor showed no improvement

WTF!!

e1000 hooked on southbridge?

speedopolis showed remarkable improvement

forwarding performance almost doubled

Need to sit down and reflect

Need to test on faster bus machine (opteron?)

Next Steps

Need to sit down and reflect

If the driver TX path is bus starved we do observe improvements

Is the way the e1000 queues frames for DMA the most optimal?

Need to test on faster bus machine (opteron?)

Hookup a bus analyzer and watch

The struggle continues

Other WIP

WIP

Old stuff (some code in place)

Rabin Fingerprint

meta action

checksum action

genetlink small code changes

genetlink doc in progress

Idle action (design complete, no code)

user space state tracking of tc classifier/action

Some code in place

2006-09-30T17:59:24