OpenWrt/LEDE Project

  • Status Closed
  • Percent Complete
    100%
  • Task Type Bug Report
  • Category Kernel
  • Assigned To No-one
  • Operating System All
  • Severity Medium
  • Priority Very Low
  • Reported Version All
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Jaap Buurman - 06.05.2017
Last edited by Ted Hess - 11.02.2018

FS#764 - MT7621: Any traffic shaping results in crashes/stack traces

There has been a large number of reports of bugs with MT7621 devices in combination with SQM. Debugging is difficult, because it often results in a hardcrash which leaves no log files. I believe I have some interesting details that might make it easier to debug.

Device: DIR-860L rev B1, but according to reports all MT7621 devices are affected.
LEDE Version: LEDE Reboot SNAPSHOT r4094-961c0ea
Steps to reproduce: Run a dslreports.com speedtest with a large number of upload and download streams (32/32) with either SQM or QOS enabled on your WAN interface.

Observations:

  • It happens both with SQM-scripts _and_ QOS. So I don’t believe it is an issue with the SQM package specifically. These two packages have in common that they both shape traffic.
  • It seems to be load dependent. 100/100 and 200/200 mbit egress/ingress limits crash less often than 300/300 or higher limits
  • It happens with all qdiscs: Cake + piece of cake, fq_codel + simple, fq_codel + simplest

Crash log:

There is usually no crash log because the router hardlocks and then reboots. But I got very lucky once and managed to get a log of the event:

[  710.140000] INFO: rcu_sched detected stalls on CPUs/tasks:
[  710.150000] 	1-...: (257 GPs behind) idle=dfc/0/0 softirq=48167/48179 fqs=1 
[  710.160000] 	(detected by 2, t=6004 jiffies, g=13114, c=13113, q=1063)
[  710.170000] Task dump for CPU 1:
[  710.180000] swapper/1       R running      0     0      1 0x00100000
[  710.190000] Stack : 00000000 5b6c286a 000000a3 ffffffff 00000090 773742c0 804df2a4 80490000
[  710.190000] 	  8048c75c 00000001 00000001 8048c540 8048c724 80490000 00000000 800135e4
[  710.190000] 	  00000000 00000001 87c70000 87c71ec0 80490000 8005ec74 1100fc03 00000001
[  710.190000] 	  00000000 80490000 804df2a4 8005ec6c 80490000 8001b1a8 1100fc03 00000000
[  710.190000] 	  00000004 8048c4a0 000000a0 8001b1b0 8c94e220 00008018 dc124877 a0020044
[  710.190000] 	  ...
[  710.260000] Call Trace:
[  710.270000] [<8000be98>] __schedule+0x574/0x758
[  710.280000] [<800135e4>] r4k_wait_irqoff+0x0/0x20
[  710.290000] 
[  710.290000] rcu_sched kthread starved for 6016 jiffies! g13114 c13113 f0x0 s3 ->state=0x1
[  782.470000] INFO: rcu_sched detected stalls on CPUs/tasks:
[  782.470000] 	1-...: (0 ticks this GP) idle=12c/0/0 softirq=48179/48179 fqs=0 
[  782.470000] 	(detected by 0, t=6002 jiffies, g=13324, c=13323, q=1260)
[  782.470000] Task dump for CPU 1:
[  782.470000] swapper/1       R running      0     0      1 0x00100000
[  782.470000] Stack : 00000000 00000001 0000000a 00000000 00000000 00000001 804df2a4 80490000
[  782.470000] 	  8048c75c 00000001 00000001 8048c540 8048c724 80490000 00000000 800135e4
[  782.470000] 	  00000000 00000001 87c70000 87c71ec0 80490000 8005ec74 1100fc03 00000001
[  782.470000] 	  00000000 80490000 804df2a4 8005ec6c 80490000 8001b1a8 1100fc03 00000000
[  782.470000] 	  00000004 8048c4a0 000000a0 8001b1b0 8c94e220 00008018 dc124877 a0020044
[  782.470000] 	  ...
[  782.470000] Call Trace:
[  782.470000] [<8000be98>] __schedule+0x574/0x758
[  782.470000] [<800135e4>] r4k_wait_irqoff+0x0/0x20
[  782.470000] 
[  782.470000] rcu_sched kthread starved for 6002 jiffies! g13324 c13323 f0x0 s3 ->state=0x1
[  860.040000] INFO: rcu_sched detected stalls on CPUs/tasks:
[  860.050000] 	1-...: (0 ticks this GP) idle=5a8/0/0 softirq=48179/48179 fqs=0 
[  860.060000] 	(detected by 3, t=6004 jiffies, g=13501, c=13500, q=2389)
[  860.070000] Task dump for CPU 1:
[  860.080000] swapper/1       R running      0     0      1 0x00100000
[  860.090000] Stack : 00000000 00002cd1 00000000 777882c0 00000000 00000000 804df2a4 80490000
[  860.090000] 	  8048c75c 00000001 00000001 8048c540 8048c724 80490000 00000000 800135e4
[  860.090000] 	  00000000 00000001 87c70000 87c71ec0 80490000 8005ec74 1100fc03 00000001
[  860.090000] 	  00000000 80490000 804df2a4 8005ec6c 80490000 8001b1a8 1100fc03 00000000
[  860.090000] 	  00000004 8048c4a0 000000a0 8001b1b0 8c94e220 00008018 dc124877 a0020044
[  860.090000] 	  ...
[  860.160000] Call Trace:
[  860.170000] [<8000be98>] __schedule+0x574/0x758
[  860.180000] [<800135e4>] r4k_wait_irqoff+0x0/0x20
[  860.190000] 
[  860.190000] rcu_sched kthread starved for 6017 jiffies! g13501 c13500 f0x0 s3 ->state=0x1

I hope it contains useful information for tracking down this bug. If there is anything else I can supply or test in order to help the debugging process, please let me know.

Closed by  Ted Hess
11.02.2018 14:04
Reason for closing:  Fixed
Jaap Buurman commented on 06.05.2017 07:09

There is a typo in the task title. MT6721 should be MT7621. I can't seem to be able to edit it. Could a mod please edit the title? Thanks :)

edit 1: I forgot to mention: Disabling offloading with ethtool does not fix it for me. It only delays the crashes. It does not eliminate them.

edit 2: Another important detail: the crashes always seem to happen during the upload part of the speedtest

Bartvz commented on 06.05.2017 13:56

Possibly related, if running SQM QoS the following stack traces appear in the kernel and system logs:

Kernel log:

[81578.960000] ------------[ cut here ]------------
[81578.970000] WARNING: CPU: 2 PID: 0 at net/core/skbuff.c:4196 skb_try_coalesce+0x228/0x35c()
[81578.990000] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TEE xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_dup_ipv6 nf_dup_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt sch_cake nf_conntrack act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_tbf sch_htb sch_hfsc sch_ingress mt76x2e mt7603e ledtrig_usbport mt76 mac80211 cfg80211 compat xt_set ip_set_list_set ip_set_hash_netiface ip_set_hash_netport ip_set_hash_netnet ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables ifb leds_gpio xhci_mtk xhci_plat_hcd xhci_pci xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[81579.220000] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.4.61 #0
[81579.230000] Stack : 00000000 00000000 804c6862 00000033 00000000 00000000 80470000 804e0000
[81579.230000] 	  87c4b9ec 80465c83 803e352c 00000002 00000000 804c367c 87c1dd3f 804fade0
[81579.230000] 	  00000000 80063258 80470000 804e0000 8046a188 8046a18c 803e8160 87c1dbec
[81579.230000] 	  00000003 80061014 87c1dd3f 804fade0 00000000 00000125 00000000 00c1dbec
[81579.230000] 	  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[81579.230000] 	  ...
[81579.300000] Call Trace:
[81579.310000] [<80016718>] show_stack+0x6c/0x88
[81579.320000] [<801b5640>] dump_stack+0x8c/0xc0
[81579.330000] [<8002b954>] warn_slowpath_common+0xa0/0xd0
[81579.340000] [<8002ba0c>] warn_slowpath_null+0x18/0x24
[81579.350000] [<80290690>] skb_try_coalesce+0x228/0x35c
[81579.360000] [<802ebab4>] tcp_try_coalesce+0x70/0xd4
[81579.370000] 
[81579.370000] ---[ end trace 2800734bc845e5bf ]---

System log:

Sat May  6 14:11:36 2017 authpriv.info dropbear[8510]: Child connection from 192.168.0.147:56580
Sat May  6 14:11:37 2017 kern.warn kernel: [81578.960000] ------------[ cut here ]------------
Sat May  6 14:11:37 2017 kern.warn kernel: [81578.970000] WARNING: CPU: 2 PID: 0 at net/core/skbuff.c:4196 skb_try_coalesce+0x228/0x35c()
Sat May  6 14:11:37 2017 kern.warn kernel: [81578.990000] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TEE xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipvSat May  6 14:11:37 2017 kern.warn kernel: [81579.220000] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.4.61 #0
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.230000] Stack : 00000000 00000000 804c6862 00000033 00000000 00000000 80470000 804e0000
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.230000] 	  87c4b9ec 80465c83 803e352c 00000002 00000000 804c367c 87c1dd3f 804fade0
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.230000] 	  00000000 80063258 80470000 804e0000 8046a188 8046a18c 803e8160 87c1dbec
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.230000] 	  00000003 80061014 87c1dd3f 804fade0 00000000 00000125 00000000 00c1dbec
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.230000] 	  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.230000] 	  ...
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.300000] Call Trace:
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.310000] [<80016718>] show_stack+0x6c/0x88
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.320000] [<801b5640>] dump_stack+0x8c/0xc0
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.330000] [<8002b954>] warn_slowpath_common+0xa0/0xd0
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.340000] [<8002ba0c>] warn_slowpath_null+0x18/0x24
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.350000] [<80290690>] skb_try_coalesce+0x228/0x35c
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.360000] [<802ebab4>] tcp_try_coalesce+0x70/0xd4
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.370000] 
Sat May  6 14:11:37 2017 kern.warn kernel: [81579.370000] ---[ end trace 2800734bc845e5bf ]---

If necessary I can provide more information.

Borromini commented on 06.05.2017 21:15

I can confirm this behaviour. Disabling offloading improves stability but will not eliminate the reboots, they just become less frequent.

Bartvz commented on 07.05.2017 17:47

Disabling SMT *seems* to fix the problem. My build with SMT disabled has been running stable for the last 19 hours. I will keep testing it and report back.
For the people wanting to try this build, you can find it in the link below. Weirdly enough, "cat /proc/cpuinfo" shows 4 cpu cores.

And it failed...

Kernel log:
[76720.940000] ————[ cut here ]———— [76720.950000] WARNING: CPU: 0 PID: 2988 at net/core/skbuff.c:4196 skb_try_coalesce+0x228/0x35c()
[76720.970000] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TEE xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_dup_ipv6 nf_dup_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt sch_cake nf_conntrack act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_tbf sch_htb sch_hfsc sch_ingress mt76x2e mt7603e ledtrig_usbport mt76 mac80211 cfg80211 compat xt_set ip_set_list_set ip_set_hash_netiface ip_set_hash_netport ip_set_hash_netnet ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables ifb leds_gpio xhci_mtk xhci_plat_hcd xhci_pci xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[76721.200000] CPU: 0 PID: 2988 Comm: dropbear Not tainted 4.4.61 #0
[76721.200000] Stack : 00000000 00000000 804c6862 00000035 00000000 00000000 80470000 804e0000
[76721.200000] 87e1606c 80465c83 803e351c 00000000 00000bac 804c367c 854a9cf7 00000010
[76721.200000] 872613f8 8006323c 80470000 804e0000 8046a168 8046a16c 803e8150 854a9ba4
[76721.200000] 00000003 80060ff8 854a9cf7 00000010 872613f8 00000125 00000000 004a9ba4
[76721.200000] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[76721.200000] ...
[76721.200000] Call Trace:
[76721.200000] [<80016718>] show_stack+0x6c/0x88
[76721.200000] [<801b5624>] dump_stack+0x8c/0xc0
[76721.200000] [<8002b954>] warn_slowpath_common+0xa0/0xd0
[76721.200000] [<8002ba0c>] warn_slowpath_null+0x18/0x24
[76721.200000] [<80290678>] skb_try_coalesce+0x228/0x35c
[76721.200000] [<802eba9c>] tcp_try_coalesce+0x70/0xd4
[76721.200000]
[76721.350000] —[ end trace 494c6962df42b073 ]—

System log:
Sun May 7 21:01:51 2017 kern.warn kernel: [76720.940000] ————[ cut here ]———— Sun May 7 21:01:51 2017 kern.warn kernel: [76720.950000] WARNING: CPU: 0 PID: 2988 at net/core/skbuff.c:4196 skb_try_coalesce+0x228/0x35c()
Sun May 7 21:01:51 2017 kern.warn kernel: [76720.970000] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TEE xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipvSun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] CPU: 0 PID: 2988 Comm: dropbear Not tainted 4.4.61 #0
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] Stack : 00000000 00000000 804c6862 00000035 00000000 00000000 80470000 804e0000
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] 87e1606c 80465c83 803e351c 00000000 00000bac 804c367c 854a9cf7 00000010
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] 872613f8 8006323c 80470000 804e0000 8046a168 8046a16c 803e8150 854a9ba4
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] 00000003 80060ff8 854a9cf7 00000010 872613f8 00000125 00000000 004a9ba4
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] ...
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] Call Trace:
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] [<80016718>] show_stack+0x6c/0x88
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] [<801b5624>] dump_stack+0x8c/0xc0
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] [<8002b954>] warn_slowpath_common+0xa0/0xd0
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] [<8002ba0c>] warn_slowpath_null+0x18/0x24
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] [<80290678>] skb_try_coalesce+0x228/0x35c
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000] [<802eba9c>] tcp_try_coalesce+0x70/0xd4
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.200000]
Sun May 7 21:01:51 2017 kern.warn kernel: [76721.350000] —[ end trace 494c6962df42b073 ]—

Zhong Jianxin commented on 08.05.2017 08:44

I have the some issue with a different MT7621 device, ZBT-WG3526.

Jaap Buurman commented on 12.05.2017 22:58

A compile of the latest master branch with the 4.9 kernels seems to have fixed the issue for me. Initial testing has shown no crashes whatsoever. I will monitor this over a longer period of time to see whether the issue is completely gone or not. Backporting some code from 4.9 to 4.4 might be needed to fix this issue once and for all.

Unfortunately, I am also running into a few new issues with the build with the 4.9 kernel. After a while, the router stops responding to a large number of commands in SSH sessions, and LUCI is also not able to apply changed settings (presumably because the restart command for the network is not working either). I am not sure if that is a widespread issue, or whether more people have the same issue.

I will keep this thread updated as more information comes in about the current situation. Hopefully more people will be able to join in on the testing.

Borromini commented on 13.05.2017 12:43

That's good to know Jaap, but from what you're saying (and from what I saw committed to trunk) it looks like 4.9 support isn't fully done yet for mt7621?

Jaap Buurman commented on 15.05.2017 17:31

I haven't checked the status in the commits, so I am not sure where the support for 4.9 currently stands. But the router running the 4.9 kernel now has an uptime of 2.5 days, so it's not that bad. Still far from perfect though. I'm still suspecting the drivers of being buggy and causing all the SQM/QOS and aforementioned issues.

edit: This is with SQM disabled btw. The entire platform is still very unstable with SQM enabled, just like on the 4.4 kernel. The mt76 probably needs some work to fix this for once and for all.

Jaap Buurman commented on 28.05.2017 18:09

Is there anything I can do to help pinpoint the exact cause of this bug? Would love to hear if anybody has already found some time to take a look at this issue :)

camel commented on 29.05.2017 12:59

try to disable the wlan 2,4GHZ ... this is buggy as hell, and for me it is useless to use.
(also other crashes on the drivers - i have no clue why, but since i completely disabled the 2,4GHZ WLAN - it is stable for me)
try again if you still have the problems without 2,4GHZ wlan.
cu Erwin

Jaap Buurman commented on 29.05.2017 15:28

@camel Are you running SQM and is it stable for you, even with cake? And what do you mean with completely disabling 2.4ghz wlan? Just disabling it in luci is enough, or something else as well?

camel commented on 29.05.2017 15:46

i disabled it completely via web (disable all SSID from 2,4GHZ ..
since that time, i had no crashed .. also traffic shapping running.
cake + default
I'M using it to limit the bandwidth of my internet:
30mbit DL / 5mbit UL
also using it for my br-lan_guest on which i have multiSSID for guest network with own IP range and wifi guest (now 5GZ only)
and limit on 1mbit / 512kbit

currently using another router for 2,4Ghz (what a shame ...)
and did not had any kernel crashes since 2 days ... still testing ...
not sure if it is really related to it.
also with the mt7621 - i have seen max: 3MB/sec.
with a mt7620 device (zbt826) i have throughput 7MB/sec

so, this mt7621 is really totally unstable and useless.. and maybe it is really related to the mt7621 driver parts of 2,4GHZ. pls test it

camel commented on 02.06.2017 10:35

meanwhile tested ... with default SQM it is OK
the SQM "cake" is not stable for me

Jaap Buurman commented on 03.06.2017 11:44

@camel And that is with Wifi enabled or disabled?

camel commented on 03.06.2017 12:05

i guess it is one of the most wanted parts to get a proper stable 2,4GHZ wifi wifi 2,4GHZ
but currently this 2,4wifi with chipset mt7603* is useless...
i have it disabled !
as told ... the mt7603 open source driver part is buggy as hell :(
best if you use for 2,4 a different switch :(
and seems to be, that it is not that easy to fix ...

Borromini commented on 05.06.2017 13:31

It might be premature, but with kernel 4.4.70, my uptime is approaching one week. Usually, my DIR-860L would reboot once ever two days or sometimes even multiple times during the day.

SQM is still enabled (with cake).

Will follow up.

jordipalet commented on 25.06.2017 09:47

I'm having the same troubles, tried with two different MT7621 device, ZBT-WG3526 and SK-WB8.

The problem happens to me several times per day if I've SQM activated. Tested with Lede 17.01.0, 17.01.1 and 17.01.2.

I've tried also disabling the WiFi interfaces, no difference.

Of course the 2.4 WiFi has its own problems ... but I believe they are not related to SQM.

Jaap Buurman commented on 29.06.2017 19:33

This commit supposedly fixes this longstanding issue:

https://git.lede-project.org/?p=lede/blogic/staging.git;a=commit;h=c05efda56aecea0f0f52a000a3ce271775b5fb24

I will personally test this commit this weekend. For people that have time earlier, feel free to test this :)

dchard commented on 16.07.2017 11:20

This happnes under kernel 4.9.37 and I agree this is load depenedent. I am not using any QoS like SQM, but when I run an iperf3 server on the router itself, after a few seconds I got exactly the same kernel trace as in the first post. However I never seen a reboot of my device.

Project Manager
Baptiste Jonglez commented on 16.07.2017 11:44

This is very likely the same issue as  FS#804 .

Please test the patch mentioned above or the pre-compiled images here https://pub.polyno.me/lede-ramips-FS804/ (17.01 + the above patch, version is r3464+1-82b20d74cb)

dchard commented on 16.07.2017 17:59

I installed the patch and run CPU benchmarks for about an hour. So far so good, I was not able to reproduce this issue any way I did it before. I will test it for 2-3 days and report back with the result.

camel commented on 17.07.2017 08:33

it is looking very good, but i can not fully test it, as many packages can'T be installed on this patched version, as:
Collected errors:
* satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-fs-ext4:
* kernel (= 4.4.74-1-05c37ed15d4e3c3a6882003f8dd4d104) * kernel (= 4.4.74-1-05c37ed15d4e3c3a6882003f8dd4d104) * kernel (= 4.4.74-1-05c37ed15d4e3c3a6882003f8dd4d104) * kernel (= 4.4.74-1-05c37ed15d4e3c3a6882003f8dd4d104) *
* opkg_install_cmd: Cannot install package kmod-fs-ext4.

pls can devel team deploy this patch to current trunk ?
then we can test it better.
in any case, it can not be more worst as it is currently in trunk, on which i get 5x rcu stalls per hour, and a lot of things are not working afterwards anymore. basically on rcu stalls i need to do a reboot..

PLS commit to current TRUNK, and let us know when done then i will ASAP doing more testing.

dchard commented on 18.07.2017 13:29

After 3 days of testing I can tell that the kernel warning is not presenting itself anymore. I did not used SQM or QoS at all, but I tested the SoC under heavy load for hours and nothing happened: no hardlocks, no crash, no kernel errors/warnings, no restarts. Everything looks fine. Previously it took about 5-10 minutes to present the RCU error under heavy load.

I agree that this patch could be merged to trunk, would be a lot easier for us to further test it.

Jaap Buurman commented on 19.07.2017 10:53

Sorry for my late reply. Unfortunately, during traffic shaping the Dir-860l still crashes with that patch applied. So it does not seem to be a complete fix. It does look like it takes longer for it to manifest, so I believe we're getting closer to the solution for our issues :)

dchard commented on 20.07.2017 17:13

After 4 days of running, I found this in kernel log:

[168991.950000] ------------[ cut here ]------------
[168991.950000] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:306 dev_watchdog+0x258/0x2fc()
[168991.970000] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[168991.990000] Modules linked in: pppoe ppp_async iptable_nat pppox ppp_generic nf_nat_ipv4 nf_conntrack_ipv6 nf_conntrack_ipv4 ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt mt76x2e mt7603e ledtrig_usbport mt76 mac80211 cfg80211 compat ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables leds_gpio xhci_mtk xhci_plat_hcd xhci_pci xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[168992.110000] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.4.74 #0
[168992.130000] Stack : 00000000 00000000 804c6862 00000033 00000000 00000000 80470000 804e0000
[168992.130000] 	  87c4b9ec 80465c63 803e2840 00000002 00000000 804c367c ffffffff 00000200
[168992.130000] 	  00100000 800650a4 80470000 804e0000 8046a198 8046a19c 803e7160 87c1de04
[168992.130000] 	  00000003 80062df0 ffffffff 00000200 00100000 00000000 00000006 00c1de04
[168992.130000] 	  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[168992.130000] 	  ...
[168992.200000] Call Trace:
[168992.200000] [<80016960>] show_stack+0x50/0x84
[168992.210000] [<801b5790>] dump_stack+0x84/0xbc
[168992.220000] [<8002d020>] warn_slowpath_common+0xa0/0xd0
[168992.230000] [<8002d07c>] warn_slowpath_fmt+0x2c/0x38
[168992.240000] [<802c3298>] dev_watchdog+0x258/0x2fc
[168992.250000] [<80074158>] call_timer_fn.isra.4+0x24/0x80
[168992.260000] [<800743b0>] run_timer_softirq+0x1fc/0x25c
[168992.270000] [<8002fc34>] __do_softirq+0x294/0x2e0
[168992.280000] [<8002ff1c>] irq_exit+0x78/0x94
[168992.290000] [<801dfb64>] plat_irq_dispatch+0xb4/0xdc
[168992.300000] 
[168992.300000] ---[ end trace cc72877aa6acaf37 ]---
[168992.310000] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
[168992.320000] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
[168992.340000] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=06cf0000, max=512, ctx=449, dtx=449, fdx=448, next=449
[168992.360000] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=06cf2000, max=512, calc=46, drx=47
[218747.950000] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
[218747.960000] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
[218747.970000] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=064e6000, max=512, ctx=508, dtx=508, fdx=507, next=508
[218747.990000] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=06dd6000, max=512, calc=485, drx=486
[218757.950000] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
[218757.960000] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
[218757.970000] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=06dd6000, max=512, ctx=8, dtx=8, fdx=7, next=8
[218757.990000] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=064e6000, max=512, calc=279, drx=280

The system log is empty. No restarts or any other way of noticing this except the log entry.

This is with the patched version, maybe completely unrelated.

Pierre Parent commented on 26.07.2017 08:46

Can you test with these images containing latest patch for this issue:

https://www.own-mailbox.com/lede/

It will also be in next trunk built (in 1 day max).

dchard commented on 02.08.2017 17:38

I am testing with latest trunk (patches already included), but after 6 days of error free operation, today I got this:

[509232.226713] ------------[ cut here ]------------
[509232.236114] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:316 dev_watchdog+0x254/0x2f4
[509232.252761] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[509232.266800] Modules linked in: nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_amanda nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast nf_conntrack_amanda ts_kmp ts_fsm ts_bm pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_CT slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt mt76x2e ledtrig_usbport mt76 mac80211
[509232.407717]  cfg80211 compat ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables leds_gpio xhci_mtk xhci_plat_hcd xhci_pci xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[509232.446222] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.9.37 #0
[509232.458159] Stack : 00000000 00000000 80547b2a 00000033 80402a44 00000000 00000000 80540000
[509232.474972]         87c4c3bc 804e7ea7 8047e9ec 00000002 00000000 80543824 ffffffff 00000200
[509232.491781]         00200000 800699b0 00000001 80540000 804edfc4 804edfc8 8048361c 87c1dddc
[509232.508599]         00000003 800a7888 ffffffff 00000200 00200000 00000000 00000006 00c1dddc
[509232.525409]         00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[509232.542215]         ...
[509232.547243] Call Trace:
[509232.552294] [<8000f764>] show_stack+0x54/0x88
[509232.561142] [<801db424>] dump_stack+0x84/0xc0
[509232.569975] [<8002a3e4>] __warn+0xe4/0x118
[509232.578288] [<8002a448>] warn_slowpath_fmt+0x30/0x3c
[509232.588345] [<8030b81c>] dev_watchdog+0x254/0x2f4
[509232.597872] [<8007bd20>] call_timer_fn.isra.3+0x24/0x84
[509232.608432] [<8007bf60>] run_timer_softirq+0x1e0/0x240
[509232.618825] [<8002deac>] __do_softirq+0x294/0x2e0
[509232.628347] [<8002e1a0>] irq_exit+0x7c/0x98
[509232.636843] [<8020a5a0>] plat_irq_dispatch+0xb4/0xdc
[509232.646998] ---[ end trace 30d36c65571c506e ]---
[509232.656368] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
[509232.668868] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
[509232.681021] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=06c30000, max=512, ctx=210, dtx=210, fdx=209, next=210
[509232.702505] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=073c0000, max=512, calc=442, drx=443

It is intersting, as I tortured the patched version for hours with both 100% CPU load and iperf3 tests in the same time, yet I got nothing. And today, out of the blue I got this error again. No crashes, no reboot, no any other sign of this event except the kernel log entry.

Nishant Sharma commented on 01.11.2017 06:42

It is crashing on x86_64 APU2 with 17.01.4 kernel version 4.4.92 as well. Created a bug report FS#1136

camel commented on 10.12.2017 18:54

is that bug still existing ?
pls can anyone test this again with current trunk ?

Jaap Buurman commented on 02.02.2018 22:07

No, this bug has been fixed in 17.01.4 and in the master branch.

ds_shadof commented on 05.11.2018 03:54

ZBT-WG3526 (16M)
In dumb AP mode. 2.4 5ghz wifi
OpenWrt SNAPSHOT, r8385-374e646724
after some time

[Mon Nov  5 03:06:57 2018] ------------[ cut here ]------------
[Mon Nov  5 03:06:57 2018] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x1ac/0x324
[Mon Nov  5 03:06:57 2018] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[Mon Nov  5 03:06:57 2018] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD xt_CT slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt compat ledtrig_usbport ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables ext4 jbd2 mbcache crc32c_generic mmc_block mtk_sd mmc_core leds_gpio xhci_mtk xhci_plat_hcd xhci_pci
[Mon Nov  5 03:06:57 2018]  xhci_hcd ahci libahci libata sd_mod scsi_mod gpio_button_hotplug usbcore nls_base usb_common
[Mon Nov  5 03:06:57 2018] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.78 #0
[Mon Nov  5 03:06:57 2018] Stack : 00000000 8fd30240 805a0000 8006f804 805d0000 8056cd38 00000000 00000000
[Mon Nov  5 03:06:57 2018]         80537728 8fc11dc4 8fc4483c 805a8947 80532818 00000001 8fc11d68 ac07f557
[Mon Nov  5 03:06:57 2018]         00000000 00000000 80610000 00007be8 00000000 0000016c 00000007 00000000
[Mon Nov  5 03:06:57 2018]         00000000 805b0000 000b7a02 70617773 00000000 00000000 805d0000 8037ff98
[Mon Nov  5 03:06:57 2018]         00000009 00000140 00000002 8fd30240 00000000 802a2ca8 00000008 80610008
[Mon Nov  5 03:06:57 2018]         ...
[Mon Nov  5 03:06:57 2018] Call Trace:
[Mon Nov  5 03:06:57 2018] [<800106c0>] show_stack+0x58/0x100
[Mon Nov  5 03:06:57 2018] [<8047244c>] dump_stack+0x9c/0xe0
[Mon Nov  5 03:06:57 2018] [<8002e408>] __warn+0xe0/0x114
[Mon Nov  5 03:06:57 2018] [<8002e46c>] warn_slowpath_fmt+0x30/0x3c
[Mon Nov  5 03:06:57 2018] [<8037ff98>] dev_watchdog+0x1ac/0x324
[Mon Nov  5 03:06:57 2018] [<80086774>] call_timer_fn.isra.3+0x24/0x84
[Mon Nov  5 03:06:57 2018] [<80086990>] run_timer_softirq+0x1bc/0x248
[Mon Nov  5 03:06:57 2018] [<8048f920>] __do_softirq+0x128/0x2ec
[Mon Nov  5 03:06:57 2018] [<80032b34>] irq_exit+0xac/0xc8
[Mon Nov  5 03:06:57 2018] [<802511ac>] plat_irq_dispatch+0xfc/0x138
[Mon Nov  5 03:06:57 2018] [<8000b5e8>] except_vec_vi_end+0xb8/0xc4
[Mon Nov  5 03:06:57 2018] [<8000cfb0>] r4k_wait_irqoff+0x1c/0x24
[Mon Nov  5 03:06:57 2018] [<8006687c>] do_idle+0xe4/0x168
[Mon Nov  5 03:06:57 2018] [<80066af8>] cpu_startup_entry+0x24/0x2c
[Mon Nov  5 03:06:57 2018] [<800180b4>] not_nmi+0x8c/0x1d8
[Mon Nov  5 03:06:57 2018] ---[ end trace 50bc01918e559e2d ]---
[Mon Nov  5 03:06:57 2018] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
[Mon Nov  5 03:06:57 2018] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
[Mon Nov  5 03:06:57 2018] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0ec50000, max=0, ctx=3963, dtx=3963, fdx=3962, next=3963
[Mon Nov  5 03:06:57 2018] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0e630000, max=0, calc=1466, drx=1467
[Mon Nov  5 03:06:57 2018] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818
[Mon Nov  5 03:06:57 2018] mtk_soc_eth 1e100000.ethernet: PPE started
Nick commented on 12.11.2018 08:01

Hi There,

I also have the same issue running snapshot r8378-9ac7350240 on ubiquiti edgerouter x sfp. This also occurs on stable 18.06.x builds as well.

This is random in it behaviour and can occur at any time. No particular pattern appears to trigger this failure.

[178873.280538] ------------[ cut here ]------------
[178873.289945] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x1ac/0x324
[178873.306578] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[178873.320602] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD xt_CT slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt i2c_gpio i2c_algo_pca i2c_algo_bit gpio_pca953x i2c_dev ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables leds_gpio gpio_button_hotplug
[178873.444409] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.78 #0
[178873.456520] Stack : 00000000 8ff1e540 805a0000 8006f804 805d0000 8056cd38 00000000 00000000
[178873.473323]         80537728 8fc0ddc4 8fc441fc 805a8947 80532818 00000001 8fc0dd68 532616af
[178873.490124]         00000000 00000000 80610000 00003f58 00000000 000000d3 00000008 00000000
[178873.506924]         00000000 805b0000 0006c7f9 70617773 00000000 00000000 805d0000 8037ff98
[178873.523724]         00000009 00000140 00000001 8ff1e540 00000000 802a2ca8 00000004 80610004
[178873.540527]         ...
[178873.545555] Call Trace:
[178873.550603] [<800106c0>] show_stack+0x58/0x100
[178873.559632] [<8047244c>] dump_stack+0x9c/0xe0
[178873.568468] [<8002e408>] __warn+0xe0/0x114
[178873.576783] [<8002e46c>] warn_slowpath_fmt+0x30/0x3c
[178873.586828] [<8037ff98>] dev_watchdog+0x1ac/0x324
[178873.596369] [<80086774>] call_timer_fn.isra.3+0x24/0x84
[178873.606931] [<80086990>] run_timer_softirq+0x1bc/0x248
[178873.617322] [<8048f920>] __do_softirq+0x128/0x2ec
[178873.626853] [<80032b34>] irq_exit+0xac/0xc8
[178873.635344] [<802511ac>] plat_irq_dispatch+0xfc/0x138
[178873.645560] [<8000b5e8>] except_vec_vi_end+0xb8/0xc4
[178873.655602] [<8000cfb0>] r4k_wait_irqoff+0x1c/0x24
[178873.665322] [<8006687c>] do_idle+0xe4/0x168
[178873.673812] [<80066af8>] cpu_startup_entry+0x24/0x2c
[178873.683935] ---[ end trace 83be30e64239c52f ]---

Happy to help debug this one if anyone can assist? Can we get this ticket re-opened or should a new ticket be created?

Cheers.

cwbsw commented on 16.12.2018 02:29

this problem still exists, trunk version.

Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.190653] ------------[ cut here ]------------
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.195381] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x1ac/0x324
Sat Dec 15 20:15:10 2018 kern.info kernel: [157865.203721] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.210744] Modules linked in: pppoe ppp_async pppox ppp_generic iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_policy xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY wireguard slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack_netlink macvlan iptable_raw iptable_mangle iptable_filter ipt_ah ipt_ECN ip_tables crc_ccitt sch_cake tcp_bbr act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_tbf sch_htb sch_hfsc sch_ingress
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.282057]  xt_set ip_set_list_set ip_set_hash_netiface ip_set_hash_netport ip_set_hash_netnet ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_NPT ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables ifb ip6_udp_tunnel udp_tunnel sit ipcomp6 xfrm6_tunnel xfrm6_mode_tunnel xfrm6_mode_transport xfrm6_mode_beet esp6 ah6 ipcomp xfrm4_tunnel xfrm4_mode_tunnel xfrm4_mode_transport xfrm4_mode_beet esp4 ah4 tunnel6 tunnel4 ip_tunnel af_key xfrm_user xfrm_ipcomp
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.352632]  xfrm_algo sha256_generic sha1_generic jitterentropy_rng drbg md5 hmac echainiv des_generic cbc authenc leds_gpio xhci_mtk xhci_plat_hcd xhci_pci xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.371747] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.87 #0
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.377810] Stack : 00000000 8ff60240 80560000 8006f5b4 80580000 80542e18 00000000 00000000
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.386229]         8050f05c 8fc11dc4 8fc4483c 80562927 8050a128 00000001 8fc11d68 ac07f54e
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.394650]         00000000 00000000 805c0000 00004820 00000000 000000ed 00000008 00000000
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.403069]         00000000 80560000 0005ac23 70617773 00000000 00000000 80580000 80358be8
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.411489]         00000009 00000140 00000002 8ff60240 00000000 802836c0 00000008 805c0008
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.419908]         ...
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.422429] Call Trace:
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.424977] [<800106a0>] show_stack+0x58/0x100
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.429500] [<8044db2c>] dump_stack+0x9c/0xe0
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.433935] [<8002e368>] __warn+0xe0/0x114
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.438099] [<8002e3cc>] warn_slowpath_fmt+0x30/0x3c
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.443132] [<80358be8>] dev_watchdog+0x1ac/0x324
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.447918] [<800865f4>] call_timer_fn.isra.3+0x24/0x84
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.453208] [<80086810>] run_timer_softirq+0x1bc/0x248
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.458424] [<8046ade0>] __do_softirq+0x128/0x2ec
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.463200] [<80032aa4>] irq_exit+0xac/0xc8
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.467471] [<8023996c>] plat_irq_dispatch+0xfc/0x138
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.472589] [<8000b5c8>] except_vec_vi_end+0xb8/0xc4
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.477617] [<8000cf90>] r4k_wait_irqoff+0x1c/0x24
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.482497] [<8006662c>] do_idle+0xe4/0x168
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.486749] [<800668a8>] cpu_startup_entry+0x24/0x2c
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.491791] [<800180b4>] not_nmi+0x8c/0x1d8
Sat Dec 15 20:15:10 2018 kern.warn kernel: [157865.496096] ---[ end trace 353462105a243cec ]---
Sat Dec 15 20:15:10 2018 kern.err kernel: [157865.500794] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Sat Dec 15 20:15:10 2018 kern.info kernel: [157865.507035] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
Sat Dec 15 20:15:10 2018 kern.info kernel: [157865.513121] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0f220000, max=0, ctx=2526, dtx=2526, fdx=2525, next=2526
Sat Dec 15 20:15:10 2018 kern.info kernel: [157865.524049] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0ddb0000, max=0, calc=188, drx=189
Sat Dec 15 20:15:10 2018 kern.info kernel: [157865.538798] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818
Sat Dec 15 20:15:10 2018 kern.info kernel: [157865.551414] mtk_soc_eth 1e100000.ethernet: PPE started

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing