OpenWrt/LEDE Project

  • Status Unconfirmed
  • Percent Complete
    0%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Low
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Martin Tippmann - 27.02.2017

FS#563 - cake qdisc high cpu usage on tp-link 841n

So, I had this crazy idea to just make cake the default qdisk after seeing the battlemesh presentation. Now we are seeing high spikes in CPU usage after some time - the peaks appear to be of periodic nature. I only have some nmeter plots (basically vmstat/sar with dots - it’s a busybox applet) - the high load immediatly vanishes after switching the qdiscs back to fq_codel.

I’ve seen this on 2 841n routers one only using ap+ibss mode the other having wan+lan+ap+ibss+tap interfaces with cake running. Both also run a bridge for the AP.

Is the high system cpu usage expacted behavoir? cake qdisc is just using defaults:

This script does the activation:

# cat /etc/hotplug.d/iface/00-cake-qdisc 
#!/bin/sh

case "$ACTION" in
                ifup)
                grep -q '^sch_cake' '/proc/modules' && {
                        command -V tc > /dev/null || return 
                        tc qdisc replace dev "$DEVICE" root cake
                }
        ;;
esac
# vim: set filetype=sh ai noet ts=4 sw=4 sts=4 :

The results (1 line = 1 second):

root@weimarnetz-188:~ :) nmeter '%40c wlan: %[nbr-vap] ibss: %[nwlan0]'

cake 

SSSSSSSSUii............................. wlan:  8856  756 ibss:  4026 9162
SSSSSSSSSSSSSSSSSSSSU................... wlan:   44k 1736 ibss:  8726  49k
SSSSSSSi................................ wlan:   58k 2130 ibss:  5974  62k
SSSUUUUi................................ wlan:   58k 1296 ibss:   15k  68k
SSSSSSSSi............................... wlan:   49k 2532 ibss:  5298  53k
SSSSSSSSSSiii........................... wlan:   45k 2082 ibss:  8282  49k
SSSSSSSSSSSSSSSSSSSSSSU................. wlan:   39k 1794 ibss:   14k  45k
SSSSSSSSSSSSSSUi........................ wlan:   34k 2112 ibss:   11k  29k
SSSSSSSSSSSSSSSSSSSSSUUi................ wlan:   47k 2172 ibss:  7614  60k
U....................................... wlan:   58k 2934 ibss:  7496  61k
SSUi.................................... wlan:   53k 1944 ibss:  9.8k  58k
SSSSSSSSSSSSSSSSSSU..................... wlan:   47k 1698 ibss:  6552  51k
SSSSSSSSSSSSSUi......................... wlan:   57k 2704 ibss:  9.9k  62k
UUUiii.................................. wlan:   57k 1512 ibss:  8956  63k
SSSSSSSSSSSSSSSSSSSSSSSSii.............. wlan:   77k 4010 ibss:   17k  85k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSUii wlan:  6400  318 ibss:   450 6784
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSi wlan:   47k 1296 ibss:   12k  55k
SUUii................................... wlan:   49k 2542 ibss:  5604  53k
SSSSSSSSUUi............................. wlan:   56k 2354 ibss:   10k  62k
SSSSSSSSSSSSSSSSSSSSSi.................. wlan:   58k 1926 ibss:   11k  64k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSi wlan:   59k 2464 ibss:   22k  67k
SSSSSSSSSSSSSSUi........................ wlan:   41k  972 ibss:   10k  50k
SUi..................................... wlan:   44k 2132 ibss:  6310  47k
SUUi.................................... wlan:   52k 1642 ibss:   11k  58k
SSSSSSSSSSSSSSSSSSSSSSUi................ wlan:   54k 3826 ibss:   11k  60k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSUi wlan:   44k 1482 ibss:  6580  48k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSi.......... wlan:   32k 2396 ibss:  9732  38k
SSSSSSSSSSSSSSSSUUi..................... wlan:   18k 1164 ibss:  6888  20k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSUi... wlan:   37k  864 ibss:   13k  46k
SSSSSSSSSSSi............................ wlan:   57k 2112 ibss:  4166  60k
SSi..................................... wlan:   56k 1746 ibss:   11k  62k
SSSSSSSSSSSSSSSSSSSSi................... wlan:   43k  28k ibss:   39k  48k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSUU....... wlan:   15k  18k ibss:   25k  21k
SSSSSSSSSSSSSSSSUiiii................... wlan:   42k 255k ibss:  261k  52k
SUiiii.................................. wlan:   39k 393k ibss:  398k  54k
SSSSSSSSSSSSSSSSSSSSSSSSiii............. wlan:   26k 244k ibss:  251k  33k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSii........ wlan:   44k  86k ibss:   93k  54k
SSSSSSSSSSSSSSSSSSSSSSSSSSSUUii......... wlan:   27k  33k ibss:   41k  33k
SSSSSUiii............................... wlan:   54k  36k ibss:   40k  60k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSii wlan:   30k  46k ibss:   52k  35k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSi wlan:   23k  13k ibss:   20k  26k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSUii.... wlan:   53k  31k ibss:   40k  66k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSU......... wlan:   31k  18k ibss:   29k  35k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSi wlan:   13k 1262 ibss:  8370  16k
SSSSSSSSSSSSUi.......................... wlan:   46k  35k ibss:   54k  63k
SU...................................... wlan:   53k  87k ibss:   93k  59k
SSSSSSSSSSUUi........................... wlan:   43k  23k ibss:   32k  50k
SSSUUUiii............................... wlan:   46k 164k ibss:  172k  54k
SSSSSSSSSUiii........................... wlan:   48k 245k ibss:  250k  59k
SSSSSSSSSSSSSSS......................... wlan:   38k 5959 ibss:   10k  44k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSi wlan:   38k 1838 ibss:  9944  41k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSi wlan:   25k 1344 ibss:  2774  25k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS wlan:   43k 2442 ibss:   10k  44k
SSSSSSSSSSSSSSSSSSSSUUii................ wlan:   59k 1968 ibss:   11k  73k
Si...................................... wlan:   60k 2354 ibss:   10k  67k
SU...................................... wlan:   57k 1680 ibss:   11k  63k
Siii.................................... wlan:   51k 3560 ibss:  8042  54k
SSSSSSSSSSSi............................ wlan:   50k 1080 ibss:   14k  58k
SSSSSSSSSSSSSUU......................... wlan:   53k 2548 ibss:   13k  59k
SSSSSSSSSSSSi........................... wlan:   46k 2309 ibss:  9093  52k
SSSSSSSSSSSSSSSSSSSSi................... wlan:   83k 3816 ibss:   14k  91k
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSi........ wlan:   25k 3210 ibss:  5668  26k
SSSSSSSSSSSSSSSSSSSSSSSUii.............. wlan:   53k 2478 ibss:   11k  59k
SSSSSSSSSSSSSSSSSSSSSSSSSUi............. wlan:   15k 1308 ibss:  8142  20k
SSUUii.................................. wlan:   799  607 ibss:   11k 5339
SUi..................................... wlan:  1449 1436 ibss:   11k 9399
SSSSSSSSSSSSSSSSSSSSSSSSSSSSU........... wlan:   240 2738 ibss:   10k 3224
SSSSSSSSSU.............................. wlan:   132   66 ibss:   10k 4804
UUi..................................... wlan:   320  347 ibss:   11k 6020


-----> fq_codel 

S....................................... wlan:     0    0 ibss:  8664 3682
SSSSSSSSSU.............................. wlan:     0    0 ibss:  9022 6566
SS...................................... wlan:   703 6076 ibss:   14k 3565
S....................................... wlan:  2971 1862 ibss:  8554 7219
SUi..................................... wlan:     0    0 ibss:  7810 3732
SSUUi................................... wlan:     0    0 ibss:  8440 2920
SSii.................................... wlan:  1449 1436 ibss:   10k 7771
Ui...................................... wlan:     0    0 ibss:  8068 3258
Ui...................................... wlan:     0    0 ibss:  7910 3614
SSUi.................................... wlan:   252  185 ibss:   12k 5022
SUi..................................... wlan:  4742 1368 ibss:  9.8k  10k
SU...................................... wlan:     0    0 ibss:   11k 4646
Ui...................................... wlan:     0    0 ibss:   15k 5802
SS...................................... wlan:     0    0 ibss:  6918 3742
SU...................................... wlan:   122 2576 ibss:  4978  404
SSUii................................... wlan:     0    0 ibss:  8856 5890
SU...................................... wlan:  1449 1436 ibss:  6598 4791
SU...................................... wlan:    41   54 ibss:  4234 2101
UUii.................................... wlan:     0    0 ibss:   12k 5438
SUi..................................... wlan:   280  239 ibss:  7599 1644
U....................................... wlan:     0    0 ibss:  5962 3606
U....................................... wlan:     0    0 ibss:   10k 5502
SU...................................... wlan:     0    0 ibss:   14k 7486
Ui...................................... wlan:     0    0 ibss:   14k 5030
SU...................................... wlan:     0    0 ibss:  7930 4702
Ui...................................... wlan:   799  606 ibss:   10k 5400
SSSSSSSSSSSSSU.......................... wlan:  1449 1436 ibss:  9936 5653
S....................................... wlan:     0    0 ibss:  5818 2964
SU...................................... wlan:     0    0 ibss:   11k 7547
SU...................................... wlan:   252  185 ibss:  6264 3875
Ui...................................... wlan:  4159  995 ibss:  8313 8409
U....................................... wlan:     0    0 ibss:  3912 1240
SU...................................... wlan:     0    0 ibss:  4600 3784
S....................................... wlan:   122 2604 ibss:  9706 3374
SUUUi................................... wlan:     0    0 ibss:   13k 6328
U....................................... wlan:     0    0 ibss:   10k 6940
ii...................................... wlan:  1504 1589 ibss:  8605 4736
SUUi.................................... wlan:   108    0 ibss:   13k 8516
SUi..................................... wlan:  5241  28k ibss:   32k 9431
SSUUi................................... wlan:  9.7k  24k ibss:   33k  14k
SUi..................................... wlan:   743  456 ibss:   14k 7591
U....................................... wlan:     0    0 ibss:  2212 2732
SSSSSSSSSSU............................. wlan:     0    0 ibss:  9750 6434
........................................ wlan:  4585  492 ibss:  4616 6971
SUi..................................... wlan:   326 1155 ibss:   11k 4874
Si...................................... wlan:     0    0 ibss:  2330  214
Si...................................... wlan:  4336 3047 ibss:   18k  11k
SUi..................................... wlan:   100    0 ibss:  9838 5264
S....................................... wlan:     0    0 ibss:   14k 6298
SSSU.................................... wlan:   644  511 ibss:  6719 4294
SUii.................................... wlan:     0    0 ibss:   13k 6274
S....................................... wlan:     0    0 ibss:  6822 1738
U....................................... wlan:     0    0 ibss:  4790 1730
Ui...................................... wlan:   124  110 ibss:  5518 3544
SU...................................... wlan:   229   42 ibss:  8218 4972
Si...................................... wlan:  1000  607 ibss:  4750 2432
........................................ wlan:  1650 1436 ibss:  8372 5635
UUi..................................... wlan:   201    0 ibss:  9418 5422
Ui...................................... wlan:     0    0 ibss:   14k 7438
SSUi.................................... wlan:  3268 1862 ibss:   14k 9690
SUU..................................... wlan:     0  108 ibss:  9030 4918
SSi..................................... wlan:    80  108 ibss:  9442 5162
Ui...................................... wlan:    41   54 ibss:  9768 4757
Sii..................................... wlan:    58  104 ibss:  5570 3562
UU...................................... wlan:     0    0 ibss:  7098 4134
SU...................................... wlan:     0    0 ibss:  8142 2818
S....................................... wlan:  2941 2271 ibss:  7081 6303
SUi..................................... wlan:   315    0 ibss:  8524 3738
Ui...................................... wlan:  1309  757 ibss:   10k 7143
SUU..................................... wlan:  1450  428 ibss:  7150 5470
SUi..................................... wlan:  3187 8952 ibss:   16k 7107
U....................................... wlan:  4559 2197 ibss:  7959 8811
Ui...................................... wlan:  5564 2171 ibss:  8323 9118
i....................................... wlan:   11k  41k ibss:   50k  18k
root@weimarnetz-188:~ 8( cat /tmp/sysinfo/model 
TP-Link TL-WR841N/ND v11
root@weimarnetz-188:~ :) cat /etc/os-release 
NAME="weimarnetz"
VERSION="SNAPSHOT, Reboot"
ID="weimarnetz"
ID_LIKE="lede openwrt"
PRETTY_NAME="weimarnetz Reboot SNAPSHOT"
VERSION_ID="snapshot"
HOME_URL="http://lede-project.org/"
BUG_URL="http://bugs.lede-project.org/"
SUPPORT_URL="http://forum.lede-project.org/"
BUILD_ID="v0.10.0-dirty+lede-r3598-eb09d79c16"
LEDE_BOARD="ar71xx/generic"
LEDE_ARCH="mips_24kc"
LEDE_TAINTS="no-all busybox"
LEDE_DEVICE_MANUFACTURER="LEDE"
LEDE_DEVICE_MANUFACTURER_URL="http://lede-project.org/"
LEDE_DEVICE_PRODUCT="weimarnetz"
LEDE_DEVICE_REVISION="v0"
LEDE_RELEASE="weimarnetz Reboot SNAPSHOT v0.10.0-dirty+lede-r3598-eb09d79c16"

(this is basically stock release just with a versioning patch and the symbols added:

CONFIG_KERNEL_KALLSYMS=y
CONFIG_BUSYBOX_CUSTOM=y
CONFIG_BUSYBOX_CONFIG_MKSWAP=n
CONFIG_BUSYBOX_CONFIG_SWAPONOFF=n
CONFIG_BUSYBOX_CONFIG_MKPASSWD=y
CONFIG_BUSYBOX_CONFIG_ARPING=y
CONFIG_BUSYBOX_CONFIG_NMETER=y
CONFIG_PACKAGE_MAC80211_MESH=n
CONFIG_TARGET_SQUASHFS_BLOCK_SIZE=256
CONFIG_STRIP_KERNEL_EXPORTS=y
Martin Tippmann commented on 27.02.2017 22:29

Can't edit the ticket:
- forget to add after a reboot it's normal - this seems to appear after some memory pressure
- it's not related to traffic afaik - both routers had very little traffic

Weedy commented on 08.03.2017 22:18

I have a tp-link 842nd using qos-scripts.

I think this is more of a ath9k/memory bug.
My box will grind to a halt after a good chunk of wifi traffic. And haing QoS running or darkstat only makes it happen faster.
Slab fragmentation and ath9k buffers?

I'll have the device back soon and I'll try to bisect it while hammering with iperf/flent.

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing