BSD firewall / Router Performance

Goals of this testing was to understand the following :

1) the new em driver to make sure it works well for me and give it a good shake out under load for RELENG_6

2) understand the implications of various configs on forwarding performance of SMP vs UP vs Polling vs Fast Interrupts and to avoid livelock when there is a lot of pps.

 

For more discussion, see http://lists.freebsd.org/pipermail/freebsd-performance/2006-November/002320.html

dmesg from box acting as router (r2)
r2 kernel config for freebsd
dmesg from 192.168.44.1
dmesg from 192.168.88.176
dmesg from 192.168.44.244
dmesg from 192.168.88.218
 

switch is Netgear GSM7212 using port based vlans.  All tests are done with a dual port Intel PCIe PT1000 NIC unless otherwise stated. There are a couple of tests that use a pair of Broadcom PCIe single port NICs (see dmesg for more detail)

 

 

 

 

 

 

Added Nov 24--

Test on HEAD from Nov 23rd with PMC in the kernel. UP results at  FastForward Off   FastForward On  and with ipfw in the kernel
and SMP results with ipfw in the kernel  and one firewall rule:  fastforward on, fastforward off (aka just HEAD) and net.isr.direct=1

 

Platform

 

10 bytes 64 bytes 128 bytes 200 bytes
Linux 2.18.2 NAPI 46.79 297.65 531.00 706.00
FreeBSD HEAD 46.75 297.82 530.70 728.01
RELENG6 i386 46.70 296.32 529.12 721.80
RELENG6 i386 FastFWD 46.37 295.88 529.72 722.02
FreeBSD HEAD w Patch 46.39 293.78 529.41 728.17
HEAD w Patch FastFWD 46.52 295.71 529.81 718.32
AMD64 RELENG6 w FastFWD 46.27 295.85 529.44 721.96

 

Interface is always EM (Intel dualport Pro 1000PT PCIe 4x in 16x slot ) unless stated otherwise. All RELENG_4 tests are UP as SMP does not get detected on an AMD X2

HEAD is Always from Nov24th sources, unless otherwise noted.
firewall tests with NetBSD are done with ipfilter

em
Straight Routing test One Stream pps 194K Routes bgp clear ipfw 1 ipfw 5 ipfw 10 pf 1
Linux 581,310 590,584 579,833 582,963 575,718 579,442
LINUX 2.18.2 non NAPI 552,824
FreeBSD HEAD Nov 20 +FastFWD 540,695 529,425 439,980 398,283 370,458
FreeBSD HEAD Nov 11 441,560
RELENG6 i386 407,403
RELENG6 i386 FastFWD 557,589 562,547 484,250 425,791 386,644 353,856 333,294
FreeBSD HEAD w Patch 422,294
FreeBSD HEAD FastFWD bge PCIe (Nov 23) w patch* 509,843 468,341 448,613 384,083 358,451 336,582
FreeBSD HEAD w Patch FastFWD 567,290 564,340 482,093 436,205 381,459 359,248 361,458
FreeBSD HEAD Nov 24 CPUi686 in kernel FastFWD 459,442 410,136 372,266
FreeBSD HEAD Nov 24 CPUi586,i686 in kernel FastFWD 466,329 408,406 369,802
FreeBSD HEAD Nov 24 CPUi486,i586,i686 in kernel FastFWD 461,927 408,536 366,254
AMD64 RELENG6 w FastFWD 574,592 549,233 486,737 400,050 320,129 296,760 273,824
AMD64 RELENG6 polling 285,917
AMD64 RELENG6 polling FastFWD 512,042
RELENG6 i386 polling FastFWD 558,603 550,041
RELENG6 i386 polling FastFWD HZ=2000 565,523 563,068 373,904
RELENG6 i386 INTR_FAST 596,431
RELENG6 i386 INTR_FAST (tuneable1) 591,283
RELENG6 i386 INTR_FAST (tuneable2) 585,231
RELENG6 i386 (tuneable3) 598,684
RELENG6 i386 (tuneable2) 597,831
RELENG_6 UP i386 400,188
RELENG_6 UP i386 FastFWD 584,909 582,534 560,033 560,323
RELENG_6 UP i386 FastFWD Polling 585,934 476,629 422,633 393,301
RELENG_4 Polling FastWD (two sources used to generate rate) 925,046 924,652   576,045 577,884
DragonFly FastFWD 1.6.0  bge UP FastFWD ** 148,935 148,938 148,933 148,933
DragonFly 1.7.0-DEVELOPMENT UP, bge fastFWD ** 481,494 481,228 463,668 459,491
DragonFly 1.7-Dev SMP with EM patch, FastFWD, polling, HZ=4000 ** 588,514
DragonFly 1.7-Dev UP with EM patch, fastwd, polling HZ=4000 (two sources used to generate packets) 807,048
OpenBSD 4.0 UP, default install pf disabled 219,080
OpenBSD 4.0 SMP, default install pf disabled 142,652
OpenBSD CURRENT, SMP em1, vlan44, vlan88 284,703            
OpenBSD CURRENT, UP, em1, vlan44, vlan88 293,428            
OpenSolaris, Default install (console locks up with one stream) 427,459 269,536
HEAD, FastFWD, vlan44 and vlan88 off single int, em1 (Nov 24th sources) 568,023     478,961 416,521 390,260  
RELENG_4, FastFWD, vlan44 and vlan88 off single int, em1 Polling, HZ=2000       593,017 593,817 590,086  
HEAD, FastFWD, vlan44 and vlan88 off single int, bge0 (Nov 24th sources) 542,007     463,624 409,238    
RELENG_6, FastFWD, INTR_FAST, vlan44 and vlan44 off single int, em1 590,070     457,208 404,974 367,736  
NetBSD 3.1, UP, ipfilter vlan44 and vlan88 off single int bge1 (ipfilter) (live lock with one stream) 482,491     421,642 305,683 302,289  
NetBSD 3.1, SMP, ipfilter vlan44 and vlan88 off single int bge1 (ipfilter) (live lock with one stream) 428,767       285,981 287,939  
NetBSD Current SMP kernel Intel NICs (wm driver) Options GATEWAY in kernel 129,142            
NetBSD Current SMP kernel Intel NICs (bge driver) Options GATEWAY in kernel*** 593,687       218,310 214,946  
NetBSD Current SMP kernel Intel NICs (bge driver) Options GATEWAY in kernel
hw.bge.rx_lvl = 5
589,445       345,317 336,221  

*BGE Patch http://lists.freebsd.org/pipermail/freebsd-net/2006-November/012389.html
** ipfw might not be working properly on DragonFly Dev
*** Driver did wedge at one point in testing, but speed looks promising. See dmesg for detail

 

Straight Routing test 2 streams opposite direction   194K Routes bgp clear ipfw 1 ipfw 5 ipfw 10
Linux 473,814 452,205       408,932
Linux non NAPI 272,313          
FreeBSD HEAD Nov 11 204,043          
FreeBSD HEAD nov 20 + fastFWD 312,140     250,277 223,289 208,551
RELENG6 i386 165,461          
RELENG6 i386 FastFWD 368,967 353,437   216,597 206,077 194,476
FreeBSD HEAD w Patch 127,832          
FreeBSD HEAD w Patch FastFWD 346,220 404,785   249,617 234,047 194,357
FreeBSD HEAD FastFWD bge PCIe (Nov 23) w patch* 269,469 249,551     207,491  
AMD64 RELENG6 w Polling 155,659          
AMD64 RELENG6 w Polling FastFWD 231,541          
RELENG6 i386 polling FastFWD 319,352 312,621        
RELENG6 i386 polling FastFWD HZ=2000 300,280 299,018        
RELENG6 i386 INTR_FAST 348,158          
RELENG6 i386 INTR_FAST (tuneable1 ) 367,312          
RELENG6 i386 INTR_FAST (tuneable2 ) 335,061          
RELENG6 i386  (tuneable3 ) 230,049          
RELENG6 i386  (tuneable2 ) 224,861          
RELENG_6 UP i386 FastFWD Polling 342,551       229,652 205,322
RELENG_4 Polling FastWD 595,040       540,765 532,413
DragonFly FastFWD 1.6.0  bge UP 148,940     148,940 148,933 148,933
DragonFly 1.7.0-DEVELOPMENT UP, bge fastFWD 217,982     217,288 216,590 207,499
DragonFly 1.7-Dev SMP with EM patch, FastFWD, polling, HZ=4000 ** 391,615          
OpenBSD 4.0 SMP, default install pf disabled 92,299          
OpenBSD Current SMP, vlan44 and vlan88 of em1 54,525          
55,637          
OpenSolaris, Default install 4098          
HEAD, SMP FastFWD, vlan44 and vlan88 off single int, em1 321,346     275,163 237,699 218,230
RELENG_4, FastFWD, vlan44 and vlan88 off single int, em1 Polling, HZ=2000       379,636 369,468 367,471
HEAD, SMP FastFWD, vlan44 and vlan88 off single int, bge0 306,769     263,529 233,228  
RELENG_6, FastFWD, INTR_FAST, vlan44 and vlan44 off single int, em1 342,396     256,645 222,903 206,614
NetBSD 3.1, UP, ipfilter vlan44 and vlan88 off single int bge1 (ipfilter) 136,433     83,279 82,515 81,914
NetBSD 3.1, SMP, ipfilter vlan44 and vlan88 off single int bge1 (ipfilter) 102,897       71,880 69,099
NetBSD Current SMP kernel Intel NICs (bge driver) Options GATEWAY in kernel 272,007       6,491 5,640
NetBSD Current SMP kernel Intel NICs (bge driver) Options GATEWAY in kernel  hw.bge.rx_lvl = 5 272,593       135,011 131,764