Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#3762 - WAN performance regression on MT7621 21.02-rc1 vs 19.07.7 #8778

Open
openwrt-bot opened this issue Apr 29, 2021 · 13 comments
Open
Labels

Comments

@openwrt-bot
Copy link

stevenj:

There is an issue with Wan throughput on the MT7621 and OpenWrt 21.02-rc1 vs 19.07.7

The test setup is an Iperf3 server running locally on the wan side of the router (A NeWifi-D2) and running Iperf3 as client on the router. This is stock firmware with no changes to the network settings from stock.

19.07.7 - TX = 531Mbps RX = 629Mbps
21.02-Rc1 - TX = 792Mbps Rx = 523Mbps

So while it appears TX has improved, RX is suffering and it is noticeable with non synthetic transfers.

Noticably the RX test on 21.02-RC1 is reporting rampant Retransmits by the Iperf server. These are not present in 19.07.7 in either direction, OR in 21.02-RC1 when the router is sending, only when its receiving. I also ran mpstat with an interval of 3 seconds to see processor load during the test. In the result sets below, iperf3 test was started during interval 3.

The command line on the router is
TX =
iperf3 -c -P 2
RX =
iperf3 -c -P 2 -R

19.07.7 - TX

iperf3 -c 192.168.69.102 -P 2
Connecting to host 192.168.69.102, port 5201
[ 5] local 192.168.69.127 port 50742 connected to 192.168.69.102 port 5201
[ 7] local 192.168.69.127 port 50744 connected to 192.168.69.102 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.04 sec 28.7 MBytes 231 Mbits/sec 0 130 KBytes
[ 7] 0.00-1.04 sec 28.7 MBytes 231 Mbits/sec 0 123 KBytes
[SUM] 0.00-1.04 sec 57.3 MBytes 462 Mbits/sec 0


[ 5] 1.04-2.00 sec 28.8 MBytes 251 Mbits/sec 0 130 KBytes
[ 7] 1.04-2.00 sec 28.8 MBytes 251 Mbits/sec 0 130 KBytes
[SUM] 1.04-2.00 sec 57.5 MBytes 502 Mbits/sec 0


[ 5] 2.00-3.01 sec 30.0 MBytes 251 Mbits/sec 0 130 KBytes
[ 7] 2.00-3.01 sec 30.0 MBytes 251 Mbits/sec 0 140 KBytes
[SUM] 2.00-3.01 sec 60.0 MBytes 502 Mbits/sec 0


[ 5] 3.01-4.00 sec 30.0 MBytes 252 Mbits/sec 0 130 KBytes
[ 7] 3.01-4.00 sec 30.0 MBytes 252 Mbits/sec 0 140 KBytes
[SUM] 3.01-4.00 sec 60.0 MBytes 505 Mbits/sec 0


[ 5] 4.00-5.03 sec 31.2 MBytes 256 Mbits/sec 0 130 KBytes
[ 7] 4.00-5.03 sec 31.2 MBytes 256 Mbits/sec 0 140 KBytes
[SUM] 4.00-5.03 sec 62.5 MBytes 511 Mbits/sec 0


[ 5] 5.03-6.01 sec 30.0 MBytes 257 Mbits/sec 0 130 KBytes
[ 7] 5.03-6.01 sec 30.0 MBytes 257 Mbits/sec 0 140 KBytes
[SUM] 5.03-6.01 sec 60.0 MBytes 513 Mbits/sec 0


[ 5] 6.01-7.02 sec 31.2 MBytes 258 Mbits/sec 0 130 KBytes
[ 7] 6.01-7.02 sec 31.2 MBytes 258 Mbits/sec 0 140 KBytes
[SUM] 6.01-7.02 sec 62.5 MBytes 517 Mbits/sec 0


[ 5] 7.02-8.01 sec 35.0 MBytes 298 Mbits/sec 0 130 KBytes
[ 7] 7.02-8.01 sec 35.0 MBytes 298 Mbits/sec 0 140 KBytes
[SUM] 7.02-8.01 sec 70.0 MBytes 595 Mbits/sec 0


[ 5] 8.01-9.01 sec 36.2 MBytes 305 Mbits/sec 0 139 KBytes
[ 7] 8.01-9.01 sec 36.2 MBytes 305 Mbits/sec 0 140 KBytes
[SUM] 8.01-9.01 sec 72.5 MBytes 611 Mbits/sec 0


[ 5] 9.01-10.03 sec 37.5 MBytes 306 Mbits/sec 0 139 KBytes
[ 7] 9.01-10.03 sec 37.5 MBytes 306 Mbits/sec 0 140 KBytes
[SUM] 9.01-10.03 sec 75.0 MBytes 612 Mbits/sec 0


[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.03 sec 319 MBytes 266 Mbits/sec 0 sender
[ 5] 0.00-10.07 sec 319 MBytes 265 Mbits/sec receiver
[ 7] 0.00-10.03 sec 319 MBytes 266 Mbits/sec 0 sender
[ 7] 0.00-10.07 sec 319 MBytes 265 Mbits/sec receiver
[SUM] 0.00-10.03 sec 637 MBytes 533 Mbits/sec 0 sender
[SUM] 0.00-10.07 sec 637 MBytes 531 Mbits/sec receiver

iperf Done.

mpstat 3 10

Linux 4.14.221 (OpenWrt) 04/29/21 mips (4 CPU)

05:45:14 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
05:45:17 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:45:20 all 0.66 0.00 0.17 0.00 0.00 0.00 0.00 0.00 0.00 99.17
05:45:23 all 0.08 0.00 18.11 0.00 0.00 10.38 0.00 0.00 0.00 71.43
05:45:26 all 1.16 0.00 22.92 0.00 0.00 13.62 0.00 0.00 0.00 62.29
05:45:29 all 0.33 0.00 24.50 0.00 0.00 14.62 0.00 0.00 0.00 60.55
05:45:32 all 0.58 0.00 14.04 0.00 0.00 10.38 0.00 0.00 0.00 75.00
05:45:35 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:45:38 all 0.50 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 99.17

19.07.7 - RX (Server Iperf3 Stats)

Accepted connection from 192.168.69.127, port 50746
[ 5] local 192.168.69.102 port 5201 connected to 192.168.69.127 port 50748
[ 8] local 192.168.69.102 port 5201 connected to 192.168.69.127 port 50750
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 38.9 MBytes 326 Mbits/sec 0 334 KBytes
[ 8] 0.00-1.00 sec 36.5 MBytes 306 Mbits/sec 0 266 KBytes
[SUM] 0.00-1.00 sec 75.4 MBytes 633 Mbits/sec 0


[ 5] 1.00-2.00 sec 37.5 MBytes 314 Mbits/sec 0 334 KBytes
[ 8] 1.00-2.00 sec 37.9 MBytes 318 Mbits/sec 0 266 KBytes
[SUM] 1.00-2.00 sec 75.4 MBytes 632 Mbits/sec 0


[ 5] 2.00-3.00 sec 37.4 MBytes 314 Mbits/sec 0 334 KBytes
[ 8] 2.00-3.00 sec 37.0 MBytes 310 Mbits/sec 0 266 KBytes
[SUM] 2.00-3.00 sec 74.4 MBytes 624 Mbits/sec 0


[ 5] 3.00-4.00 sec 37.5 MBytes 314 Mbits/sec 0 334 KBytes
[ 8] 3.00-4.00 sec 37.5 MBytes 315 Mbits/sec 0 266 KBytes
[SUM] 3.00-4.00 sec 75.0 MBytes 629 Mbits/sec 0


[ 5] 4.00-5.00 sec 37.5 MBytes 315 Mbits/sec 0 334 KBytes
[ 8] 4.00-5.00 sec 37.7 MBytes 316 Mbits/sec 0 266 KBytes
[SUM] 4.00-5.00 sec 75.3 MBytes 631 Mbits/sec 0


[ 5] 5.00-6.00 sec 37.5 MBytes 314 Mbits/sec 0 334 KBytes
[ 8] 5.00-6.00 sec 37.5 MBytes 315 Mbits/sec 0 266 KBytes
[SUM] 5.00-6.00 sec 75.0 MBytes 629 Mbits/sec 0


[ 5] 6.00-7.00 sec 37.2 MBytes 312 Mbits/sec 0 430 KBytes
[ 8] 6.00-7.00 sec 36.4 MBytes 305 Mbits/sec 0 322 KBytes
[SUM] 6.00-7.00 sec 73.5 MBytes 617 Mbits/sec 0


[ 5] 7.00-8.00 sec 38.3 MBytes 321 Mbits/sec 0 430 KBytes
[ 8] 7.00-8.00 sec 37.9 MBytes 318 Mbits/sec 0 322 KBytes
[SUM] 7.00-8.00 sec 76.2 MBytes 639 Mbits/sec 0


[ 5] 8.00-9.00 sec 37.1 MBytes 311 Mbits/sec 0 430 KBytes
[ 8] 8.00-9.00 sec 37.0 MBytes 311 Mbits/sec 0 322 KBytes
[SUM] 8.00-9.00 sec 74.1 MBytes 622 Mbits/sec 0


[ 5] 9.00-10.00 sec 37.8 MBytes 317 Mbits/sec 0 430 KBytes
[ 8] 9.00-10.00 sec 37.8 MBytes 317 Mbits/sec 0 322 KBytes
[SUM] 9.00-10.00 sec 75.7 MBytes 635 Mbits/sec 0


[ 5] 10.00-10.04 sec 954 KBytes 214 Mbits/sec 0 430 KBytes
[ 8] 10.00-10.04 sec 1.55 MBytes 356 Mbits/sec 0 322 KBytes
[SUM] 10.00-10.04 sec 2.49 MBytes 570 Mbits/sec 0


[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 378 MBytes 316 Mbits/sec 0 sender
[ 8] 0.00-10.04 sec 375 MBytes 313 Mbits/sec 0 sender
[SUM] 0.00-10.04 sec 752 MBytes 629 Mbits/sec 0 sender

mpstat 3 10

Linux 4.14.221 (OpenWrt) 04/29/21 mips (4 CPU)

05:46:28 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
05:46:31 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
05:46:34 all 0.58 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 99.09
05:46:37 all 0.42 0.00 13.95 0.00 0.00 14.37 0.00 0.00 0.00 71.26
05:46:40 all 0.58 0.00 14.37 0.00 0.00 27.57 0.00 0.00 0.00 57.48
05:46:43 all 1.00 0.00 17.77 0.00 0.00 26.58 0.00 0.00 0.00 54.65
05:46:46 all 0.33 0.00 12.46 0.00 0.00 15.20 0.00 0.00 0.00 72.01
05:46:49 all 0.50 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 99.17
05:46:52 all 0.00 0.00 0.00 0.00 0.00 0.17 0.00 0.00 0.00 99.83

21.02-RC1 - TX

iperf3 -c 192.168.69.102 -P 2

Connecting to host 192.168.69.102, port 5201
[ 5] local 192.168.69.128 port 54862 connected to 192.168.69.102 port 5201
[ 7] local 192.168.69.128 port 54864 connected to 192.168.69.102 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.02 sec 48.7 MBytes 400 Mbits/sec 0 134 KBytes
[ 7] 0.00-1.02 sec 48.6 MBytes 400 Mbits/sec 0 134 KBytes
[SUM] 0.00-1.02 sec 97.3 MBytes 799 Mbits/sec 0


[ 5] 1.02-2.02 sec 47.5 MBytes 401 Mbits/sec 0 134 KBytes
[ 7] 1.02-2.02 sec 47.5 MBytes 401 Mbits/sec 0 146 KBytes
[SUM] 1.02-2.02 sec 95.0 MBytes 801 Mbits/sec 0


[ 5] 2.02-3.01 sec 47.5 MBytes 399 Mbits/sec 0 134 KBytes
[ 7] 2.02-3.01 sec 47.5 MBytes 399 Mbits/sec 0 146 KBytes
[SUM] 2.02-3.01 sec 95.0 MBytes 799 Mbits/sec 0


[ 5] 3.01-4.03 sec 46.8 MBytes 388 Mbits/sec 0 214 KBytes
[ 7] 3.01-4.03 sec 45.7 MBytes 379 Mbits/sec 0 164 KBytes
[SUM] 3.01-4.03 sec 92.5 MBytes 767 Mbits/sec 0


[ 5] 4.03-5.01 sec 46.2 MBytes 396 Mbits/sec 0 214 KBytes
[ 7] 4.03-5.01 sec 46.2 MBytes 396 Mbits/sec 0 164 KBytes
[SUM] 4.03-5.01 sec 92.5 MBytes 792 Mbits/sec 0


[ 5] 5.01-6.01 sec 47.5 MBytes 397 Mbits/sec 0 214 KBytes
[ 7] 5.01-6.01 sec 47.5 MBytes 397 Mbits/sec 0 164 KBytes
[SUM] 5.01-6.01 sec 95.0 MBytes 794 Mbits/sec 0


[ 5] 6.01-7.01 sec 47.5 MBytes 398 Mbits/sec 0 214 KBytes
[ 7] 6.01-7.01 sec 47.5 MBytes 398 Mbits/sec 0 164 KBytes
[SUM] 6.01-7.01 sec 95.0 MBytes 796 Mbits/sec 0


[ 5] 7.01-8.01 sec 47.5 MBytes 397 Mbits/sec 0 226 KBytes
[ 7] 7.01-8.01 sec 47.5 MBytes 397 Mbits/sec 0 174 KBytes
[SUM] 7.01-8.01 sec 95.0 MBytes 795 Mbits/sec 0


[ 5] 8.01-9.02 sec 47.5 MBytes 396 Mbits/sec 0 226 KBytes
[ 7] 8.01-9.02 sec 47.5 MBytes 396 Mbits/sec 0 180 KBytes
[SUM] 8.01-9.02 sec 95.0 MBytes 792 Mbits/sec 0


[ 5] 9.02-10.00 sec 46.2 MBytes 395 Mbits/sec 0 226 KBytes
[ 7] 9.02-10.00 sec 46.2 MBytes 395 Mbits/sec 0 180 KBytes
[SUM] 9.02-10.00 sec 92.5 MBytes 790 Mbits/sec 0


[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 473 MBytes 397 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 473 MBytes 396 Mbits/sec receiver
[ 7] 0.00-10.00 sec 472 MBytes 396 Mbits/sec 0 sender
[ 7] 0.00-10.01 sec 472 MBytes 396 Mbits/sec receiver
[SUM] 0.00-10.00 sec 945 MBytes 792 Mbits/sec 0 sender
[SUM] 0.00-10.01 sec 945 MBytes 792 Mbits/sec receiver

iperf Done.

mpstat 3 10

Linux 5.4.111 (OpenWrt) 04/29/21 mips (4 CPU)

06:14:41 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
06:14:44 all 0.75 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.00 99.00
06:14:47 all 0.00 0.00 0.00 0.00 0.00 0.08 0.00 0.00 0.00 99.92
06:14:50 all 0.08 0.00 16.97 0.00 0.00 10.40 0.00 0.00 0.00 72.55
06:14:53 all 1.33 0.00 25.48 0.00 0.00 15.24 0.00 0.00 0.00 57.95
06:14:56 all 0.17 0.00 25.02 0.00 0.00 15.21 0.00 0.00 0.00 59.60
06:14:59 all 0.58 0.00 17.00 0.00 0.00 9.83 0.00 0.00 0.00 72.58
06:15:02 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
06:15:05 all 0.83 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.00 98.92

21.02-RC1 - RX (Server Iperf3 stats)

Accepted connection from 192.168.69.128, port 54872 [ 5] local 192.168.69.102 port 5201 connected to 192.168.69.128 port 54874 [ 8] local 192.168.69.102 port 5201 connected to 192.168.69.128 port 54876 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 28.7 MBytes 241 Mbits/sec 20 342 KBytes [ 8] 0.00-1.00 sec 34.5 MBytes 290 Mbits/sec 1 443 KBytes [SUM] 0.00-1.00 sec 63.3 MBytes 531 Mbits/sec 21 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 28.8 MBytes 241 Mbits/sec 0 403 KBytes [ 8] 1.00-2.00 sec 31.3 MBytes 262 Mbits/sec 4 362 KBytes [SUM] 1.00-2.00 sec 60.0 MBytes 504 Mbits/sec 4 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 27.8 MBytes 234 Mbits/sec 7 339 KBytes [ 8] 2.00-3.00 sec 35.5 MBytes 298 Mbits/sec 0 428 KBytes [SUM] 2.00-3.00 sec 63.4 MBytes 532 Mbits/sec 7 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 27.9 MBytes 234 Mbits/sec 2 291 KBytes [ 8] 3.00-4.00 sec 35.2 MBytes 296 Mbits/sec 17 356 KBytes [SUM] 3.00-4.00 sec 63.1 MBytes 530 Mbits/sec 19 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 27.8 MBytes 234 Mbits/sec 0 356 KBytes [ 8] 4.00-5.00 sec 34.4 MBytes 289 Mbits/sec 0 424 KBytes [SUM] 4.00-5.00 sec 62.3 MBytes 522 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 25.2 MBytes 212 Mbits/sec 1 297 KBytes [ 8] 5.00-6.00 sec 37.7 MBytes 316 Mbits/sec 0 486 KBytes [SUM] 5.00-6.00 sec 62.9 MBytes 528 Mbits/sec 1 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 26.1 MBytes 219 Mbits/sec 0 359 KBytes [ 8] 6.00-7.00 sec 36.5 MBytes 307 Mbits/sec 7 395 KBytes [SUM] 6.00-7.00 sec 62.6 MBytes 525 Mbits/sec 7 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 30.7 MBytes 258 Mbits/sec 0 420 KBytes [ 8] 7.00-8.00 sec 30.0 MBytes 252 Mbits/sec 4 321 KBytes [SUM] 7.00-8.00 sec 60.7 MBytes 509 Mbits/sec 4 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 33.9 MBytes 284 Mbits/sec 10 232 KBytes [ 8] 8.00-9.00 sec 27.7 MBytes 232 Mbits/sec 0 383 KBytes [SUM] 8.00-9.00 sec 61.5 MBytes 516 Mbits/sec 10 - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 25.4 MBytes 213 Mbits/sec 0 303 KBytes [ 8] 9.00-10.00 sec 38.7 MBytes 324 Mbits/sec 0 451 KBytes [SUM] 9.00-10.00 sec 64.0 MBytes 537 Mbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 282 MBytes 237 Mbits/sec 40 sender [ 8] 0.00-10.01 sec 342 MBytes 286 Mbits/sec 33 sender [SUM] 0.00-10.01 sec 624 MBytes 523 Mbits/sec 73 sender

mpstat 3 10

Linux 5.4.111 (OpenWrt) 04/29/21 mips (4 CPU)

06:17:59 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
06:18:02 all 0.00 0.00 0.00 0.00 0.00 0.17 0.00 0.00 0.00 99.83
06:18:05 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
06:18:08 all 1.33 0.00 16.81 0.00 0.00 31.86 0.00 0.00 0.00 50.00
06:18:11 all 0.83 0.00 22.55 0.00 0.00 44.43 0.00 0.00 0.00 32.20
06:18:14 all 1.66 0.00 23.13 0.00 0.00 43.68 0.00 0.00 0.00 31.53
06:18:17 all 0.50 0.00 12.51 0.00 0.00 24.85 0.00 0.00 0.00 62.14
06:18:20 all 0.67 0.00 0.17 0.00 0.00 0.08 0.00 0.00 0.00 99.09
06:18:23 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00

@openwrt-bot
Copy link
Author

netdsg:

I'm also seeing this issue. On 21.02 I'm only getting half of the line rate. Looks to be related to software interrupts.

@openwrt-bot
Copy link
Author

kristrev:

RPS/XPS is configured by default in 19.07, but has to be explicitly enabled in 21.02. In order to enable RPS/XPS, set packet_steering in /etc/config/network to 1. I haven't tested the effect of enabling RPS/XPS on mt7621, but it gave nice performance boost on mt7622.

@openwrt-bot
Copy link
Author

stevenj:

Thanks for the suggestion, i tried it unfortunately it didn't yield any different results.

The issue is characterized by lots of dropped packets being reported on RX on the WAN. The same test over any LAN port runs at near Line rate with no dropped packets.

@openwrt-bot
Copy link
Author

DasTestament:

Upvote this. I have Xiaomi Mi Router 3G v1 and a 500mbps line over pppoe.

A 21.02.1 without fine tuning, i.e. stock settings pushes about 350-380mbps stable. Same settings on 19.08.7 shows 500-520 stable. Tested on 2 3Gv1 routers with the very same results. 21.02.1 is definitely has performance issues. Almost a year has passed and still no fix?

@openwrt-bot
Copy link
Author

DasTestament:

Upvote!

@openwrt-bot
Copy link
Author

DasTestament:

Upvote!
I wonder if this bug tracker is alive at all?

@openwrt-bot
Copy link
Author

easyteacher:

Unfortunately there seems to be not enough developers to fix this bug. So unless someone volunteers, the bug will stay open for a foreseeable long time.

@openwrt-bot
Copy link
Author

DasTestament:

That's hurts to hear :-( Feels like I will stick with 19.08.7 for a unforeseeable long time ...

@rmilecki
Copy link
Contributor

  • OpenWrt 19.07 uses Linux kernel 4.14
  • OpenWrt 21.01 uses Linux kernel 5.4

Few days ago I noticed a change in KBUILD_CFLAGS used for kernel 5.4 and newer. See Testing network / NAT performance. Few years ago I noticed that -fno-reorder-blocks is important for mt7621 NAT performance (see [PATCH RFC] kernel: drop -fno-reorder-blocks).

I'll provide a change to test next week based on above.

@rmilecki
Copy link
Contributor

rmilecki commented Jun 21, 2022

Can someone give following patch a try, please?
[PATCH RFT] ramips: use -fno-reorder-blocks for kernel compilation

  1. Compile OpenWrt from master & test NAT speed
  2. Apply above patch, refresh config (sed -i '/KERNEL_CFLAGS/d' .config), recompile & test NAT speed

@arinc9
Copy link
Contributor

arinc9 commented Jun 26, 2022

I'll test this on a Unielec U7621-06. However, I want to take out a few variables that may potentially affect the NAT performance. First of all, I want to do NAT between the interfaces of two CPU ports, gmac0 and gmac1. I can mux a switch port to gmac1 which will allow me to have the port directly connected to that CPU port. For gmac0, I want to disable DSA on the MT7530 switch. The firmware on the bootloader should load the switch to act like a dumb switch. That should prevent any overhead DSA may cause.

I will connect the muxed port to my PC and set up a network with iperf3 listening.
Any other switch port will be connected to my second gigabit port my motherboard's got with a separate network. (I don't have a second device free of DSA to test and my PC is very powerful to maintain 2Gbps full duplex traffic. I'll see if I can make iperf3 send data over the interface as both IPs being bind on the same computer may cause iperf3 client to send traffic directly to the local process, iperf3 server.)
Firewall zone for gmac0 will have masquerade enabled.

I'll follow your beautifully written Testing network / NAT performance mail to produce proper test results.

@arinc9
Copy link
Contributor

arinc9 commented Jun 28, 2022

Update: I have updated the results, previous one wasn't really done properly.

Network Structure

# Client Network
iperf client: 192.168.2.2/24
router: 192.168.2.1/24

# Server Network
router: 192.168.3.2/24, gateway 192.168.3.1
iperf server: 192.168.3.1/24
# Router
tc qdisc replace dev lan1 root pfifo
tc qdisc replace dev wan root pfifo
tc qdisc replace dev eth0 root pfifo

# iperf Client
ip a add 192.168.2.2/24 dev enp9s0
ip l set up enp9s0
tc qdisc replace dev enp9s0 root pfifo
ip route add 192.168.3.1 via 192.168.2.1

# iperf Server
ip netns add iperfserver
ip link set dev eno1 netns iperfserver
ip netns exec iperfserver ip a add 192.168.3.1/24 dev eno1
ip netns exec iperfserver ip l set up eno1
ip netns exec iperfserver tc qdisc replace dev eno1 root pfifo
ip netns exec iperfserver iperf3 -s

---

# No NAT
# Disable masquerade on the router
ip netns exec iperfserver ip route add 192.168.2.0/24 via 192.168.3.2

Test Information

  • Compiled the current master snapshot.

  • squashfs-sysupgrade image is installed and booted from the device’s flash.

  • All tests are done right after boot. Router is rebooted for each test.

  • MSS clamping, software and hardware offloading is disabled.

  • No other traffic was being sent to the router, no active SSH sessions, LuCI pages, etc.

  • Command run for TX test:

    for i in $(seq 1 5); do
        date
        iperf3 -t 80 -i 10 -c 192.168.3.1 | head -n -1 | sed -n 's/.* \([0-9][0-9]*\) Mbits\/sec.*/\1/p' | sort -n
        echo
        sleep 15
    done
    
  • Command run for RX test:

    for i in $(seq 1 5); do
        date
        iperf3 -R -t 80 -i 10 -c 192.168.3.1 | head -n -1 | sed -n 's/.* \([0-9][0-9]*\) Mbits\/sec.*/\1/p' | sort -n
        echo
        sleep 15
    done
    

Test Results

  • TX at first performs bad but it progressively gets better.
  • RX always performs good.
  • NAT performs very similar to just routing on TX and RX.
  • I don’t see this issue to be related to NAT as it performs very similar to just routing.

TX - lan1 <-> eth1 (two links to CPU)

Çrş 29 Haz 2022 11:40:21 +03
533
560
577
616
616
618
621
626
635
762

Çrş 29 Haz 2022 11:41:56 +03
626
639
653
660
665
672
672
676
689
768

Çrş 29 Haz 2022 11:43:31 +03
629
630
634
641
659
659
667
671
673
727

Çrş 29 Haz 2022 11:45:06 +03
742
743
744
744
746
746
748
749
750
751

Çrş 29 Haz 2022 11:46:41 +03
590
605
641
703
704
756
758
758
759
761
  • Just routing
Çrş 29 Haz 2022 11:50:08 +03
774
775
776
776
777
777
778
778
778
782

Çrş 29 Haz 2022 11:51:43 +03
787
789
792
792
793
795
795
800
801
808

Çrş 29 Haz 2022 11:53:18 +03
773
773
774
776
776
777
777
777
778
780

Çrş 29 Haz 2022 11:54:53 +03
635
654
662
662
665
665
668
676
683
683

Çrş 29 Haz 2022 11:56:28 +03
532
533
681
695
695
762
762
763
763
765

RX - lan1 <-> eth1 (two links to CPU)

Çrş 29 Haz 2022 11:30:50 +03
766
770
772
773
773
773
774
775
775
777

Çrş 29 Haz 2022 11:32:25 +03
772
774
775
775
775
776
776
776
777
778

Çrş 29 Haz 2022 11:34:00 +03
701
706
706
706
707
707
708
709
711
712

Çrş 29 Haz 2022 11:35:35 +03
766
768
774
774
775
775
777
778
778
778

Çrş 29 Haz 2022 11:37:10 +03
774
774
774
776
776
776
777
777
778
778
  • Just routing
Çrş 29 Haz 2022 11:59:57 +03
721
722
722
723
723
723
723
723
723
724

Çrş 29 Haz 2022 12:01:32 +03
715
754
766
766
770
775
777
777
778
778

Çrş 29 Haz 2022 12:03:07 +03
775
775
776
776
776
777
777
777
777
777

Çrş 29 Haz 2022 12:04:42 +03
711
741
763
763
774
775
775
776
776
776

Çrş 29 Haz 2022 12:06:17 +03
708
709
710
711
712
713
714
715
716
716

@arinc9
Copy link
Contributor

arinc9 commented Jun 29, 2022

After Patch

  • I’ve made sure CONFIG_KERNEL_CFLAGS="-fno-reorder-blocks” is there on the makefile config.
  • Doesn’t seem to affect the performance.

TX - lan1 <-> eth1 (two links to CPU)

Çrş 29 Haz 2022 13:08:34 +03
551
608
613
614
614
615
620
621
628
663

Çrş 29 Haz 2022 13:10:09 +03
495
514
519
519
525
535
535
546
565
599

Çrş 29 Haz 2022 13:11:44 +03
445
451
451
454
495
497
498
552
555
580

Çrş 29 Haz 2022 13:13:19 +03
526
533
649
653
653
674
675
717
726
726

Çrş 29 Haz 2022 13:14:55 +03
545
558
562
565
581
591
591
599
657
662
  • Just routing
Çrş 29 Haz 2022 13:18:51 +03
479
490
501
501
502
505
527
530
530
733

Çrş 29 Haz 2022 13:20:26 +03
696
721
725
729
729
734
737
739
740
740

Çrş 29 Haz 2022 13:22:02 +03
714
720
722
725
725
728
728
729
730
731

Çrş 29 Haz 2022 13:23:37 +03
476
506
527
642
642
718
720
728
731
733

Çrş 29 Haz 2022 13:25:12 +03
725
729
734
734
735
737
737
737
738
739

RX - lan1 <-> eth1 (two links to CPU)

Çrş 29 Haz 2022 13:44:03 +03
710
753
757
759
760
766
768
773
773
776

Çrş 29 Haz 2022 13:45:38 +03
755
768
770
771
771
771
772
773
774
776

Çrş 29 Haz 2022 13:47:13 +03
772
772
774
774
775
775
775
776
776
778

Çrş 29 Haz 2022 13:48:48 +03
765
769
770
772
772
773
775
776
776
776

Çrş 29 Haz 2022 13:50:24 +03
685
695
753
753
767
769
774
777
778
778
  • Just routing
Çrş 29 Haz 2022 13:34:13 +03
748
749
750
750
751
751
752
753
754
754

Çrş 29 Haz 2022 13:35:48 +03
731
759
760
764
764
768
773
773
774
775

Çrş 29 Haz 2022 13:37:23 +03
762
770
773
773
774
775
776
777
777
778

Çrş 29 Haz 2022 13:38:58 +03
765
773
775
775
776
777
777
777
778
778

Çrş 29 Haz 2022 13:40:33 +03
680
701
751
751
756
768
774
774
777
777

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants