Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2628 - Kernel warning / crash on ramips: eth0 (mtk_soc_eth): transmit queue 0 timed out #8660

Closed
openwrt-bot opened this issue Nov 25, 2019 · 33 comments
Labels
flyspray kernel pull request/issue with Linux kernel related changes release/19.07 pull request/issue targeted (also) for OpenWrt 19.07 release

Comments

@openwrt-bot
Copy link

jarekera:

Model ZBT-WG3526 (16M)
Architecture - MediaTek MT7621 ver:1 eco:3
Firmware version - OpenWrt 19.07-SNAPSHOT r10731-e68d589e7b / LuCI openwrt-19.07 branch git-19.326.61751-179c5e8
Kernel version- 4.14.155

Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.762093] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.770348] br-VLAN8: port 2(wlan1) entered blocking state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.775981] br-VLAN8: port 2(wlan1) entered forwarding state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.784594] br-VLAN6: port 3(wlan1-1) entered blocking state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.790393] br-VLAN6: port 3(wlan1-1) entered disabled state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.796955] device wlan1-1 entered promiscuous mode Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.803794] IPv6: ADDRCONF(NETDEV_UP): wlan1-1: link is not ready Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.810026] br-VLAN6: port 3(wlan1-1) entered blocking state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 106.815820] br-VLAN6: port 3(wlan1-1) entered forwarding state Mon Nov 25 10:11:25 2019 kern.info kernel: [ 107.343567] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1-1: link becomes ready Mon Nov 25 11:10:20 2019 kern.info kernel: [ 3641.266434] nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead. Mon Nov 25 11:37:14 2019 kern.info kernel: [ 5255.651116] TCP: request_sock_TCP: Possible SYN flooding on port 88. Sending cookies. Check SNMP counters. Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5510.958004] ------------[ cut here ]------------ Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5510.962647] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:320 0x8038e1c0 Mon Nov 25 11:41:29 2019 kern.info kernel: [ 5510.969709] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5510.976660] Modules linked in: qcserial pppoe ppp_async option cdc_mbim usb_wwan sierra_net sierra rndis_host qmi_wwan pppox ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE huawei_cdc_ncm cfg80211 cdc_ncm cdc_ether xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY wireguard usbserial usbnet usblp ts_fsm ts_bm slhc nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5511.047339] nf_nat_h323 nf_nat_amanda nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conntrack iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat cdc_wdm cdc_acm fuse ledtrig_usbport xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5511.118581] x_tables nf_reject_ipv6 ip6_udp_tunnel udp_tunnel tun vfat fat nls_utf8 nls_iso8859_1 nls_cp437 uas mmc_block usb_storage mtk_sd mmc_core leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd ahci libahci libata ehci_platform sd_mod scsi_mod ehci_hcd gpio_button_hotplug ext4 mbcache jbd2 usbcore nls_base usb_common mii crc32c_generic Mon Nov 25 11:41:29 2019 kern.warn kernel: [ 5511.149081] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.155 #0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.155177] Stack : 00000000 00000000 00000000 8fe6d540 00000000 00000000 00000000 00000000 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.163556] 00000000 00000000 00000000 00000000 00000000 00000001 8fc0bd60 ac07f582 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.171942] 8fc0bdf8 00000000 00000000 000093c8 00000038 8049e458 00000008 00000000 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.180305] 00000000 80550000 00024659 00000000 8fc0bd40 00000000 00000000 8050c830 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.188647] 8038e1c0 00000140 00000001 8fe6d540 00000000 802b02b8 00000004 805b0004 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.196980] ... Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.199415] Call Trace: Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.199535] [<8049e458>] 0x8049e458 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.205440] [<8038e1c0>] 0x8038e1c0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.208923] [<802b02b8>] 0x802b02b8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.212412] [<800101a0>] 0x800101a0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.215881] [<800101a8>] 0x800101a8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.219354] [<804873a4>] 0x804873a4 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.222825] [<800759a0>] 0x800759a0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.226315] [<800325b8>] 0x800325b8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.229802] [<8038e1c0>] 0x8038e1c0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.233316] [<80032640>] 0x80032640 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.236785] [<800d20e8>] 0x800d20e8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.240287] [<8038e1c0>] 0x8038e1c0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.243757] [<8009d860>] 0x8009d860 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.247237] [<8038e014>] 0x8038e014 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.250711] [<8008c3dc>] 0x8008c3dc Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.254181] [<80063108>] 0x80063108 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.257662] [<8008c698>] 0x8008c698 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.261136] [<8007cfe8>] 0x8007cfe8 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.264627] [<804a5240>] 0x804a5240 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.268102] [<80036f74>] 0x80036f74 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.271573] [<8025daf0>] 0x8025daf0 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.275054] [<8000b488>] 0x8000b488 Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.278548] Mon Nov 25 11:41:30 2019 kern.warn kernel: [ 5511.280144] ---[ end trace f0b0ca1dd55db7a7 ]--- Mon Nov 25 11:41:30 2019 kern.err kernel: [ 5511.284782] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.291016] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065 Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.297053] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0dc00000, max=0, ctx=1789, dtx=1789, fdx=1788, next=1789 Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.307977] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0db60000, max=0, calc=893, drx=894 Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.321091] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818 Mon Nov 25 11:41:30 2019 kern.info kernel: [ 5511.334877] mtk_soc_eth 1e100000.ethernet: PPE started
@openwrt-bot
Copy link
Author

paravz:

Similar issue on 19.07.3, kernel 4.14.180. This was not "fatal", i did not have to reboot (at least so far:)

# uname -a
Linux gw 4.14.180 #0 SMP Sat May 16 18:32:20 2020 mips GNU/Linux

cat /etc/openwrt_release

DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='19.07.3'
DISTRIB_REVISION='r11063-85e04e9f46'
DISTRIB_TARGET='ramips/mt7621'
DISTRIB_ARCH='mipsel_24kc'
DISTRIB_DESCRIPTION='OpenWrt 19.07.3 r11063-85e04e9f46'
DISTRIB_TAINTS=''

[572667.056986] mtk_soc_eth 1e100000.ethernet eth0: port 0 link down [572670.258904] mtk_soc_eth 1e100000.ethernet eth0: port 0 link up [650033.438718] ------------[ cut here ]------------ [650033.443441] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:320 0x8038ba10 [650033.450587] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out [650033.457605] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY wireguard slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat tcp_bbr ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 [650033.529064] ip6_udp_tunnel udp_tunnel mmc_block mtk_sd mmc_core leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd ahci libahci libata sd_mod scsi_mod gpio_button_hotplug usbcore nls_base usb_common [650033.546941] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.180 #0 [650033.553094] Stack : 00000000 00000000 00000000 8fde6d40 00000000 00000000 00000000 00000000 [650033.561515] 00000000 00000000 00000000 00000000 00000000 00000001 8fc0dd60 ac07f5ad [650033.569934] 8fc0ddf8 00000000 00000000 00006f40 00000038 8049c0d8 00000008 00000000 [650033.578351] 00000000 80550000 0008587d 70617773 8fc0dd40 00000000 00000000 8050ae44 [650033.586769] 8038ba10 00000140 00000002 8fde6d40 00000008 802accb0 00000008 806b0008 [650033.595192] ... [650033.597718] Call Trace: [650033.597781] [<8049c0d8>] 0x8049c0d8 [650033.603856] [<8038ba10>] 0x8038ba10 [650033.607415] [<802accb0>] 0x802accb0 [650033.610980] [<8000c1a0>] 0x8000c1a0 [650033.614535] [<8000c1a8>] 0x8000c1a8 [650033.618090] [<80484f34>] 0x80484f34 [650033.621647] [<80071a90>] 0x80071a90 [650033.625212] [<8002e608>] 0x8002e608 [650033.628769] [<8038ba10>] 0x8038ba10 [650033.632335] [<8002e690>] 0x8002e690 [650033.635893] [<8036dee0>] 0x8036dee0 [650033.639463] [<8038ba10>] 0x8038ba10 [650033.643027] [<8038b864>] 0x8038b864 [650033.646584] [<8008850c>] 0x8008850c [650033.650140] [<8005f1fc>] 0x8005f1fc [650033.653702] [<800887c8>] 0x800887c8 [650033.657256] [<800790f8>] 0x800790f8 [650033.660818] [<804a2ed8>] 0x804a2ed8 [650033.664378] [<80032fb4>] 0x80032fb4 [650033.667933] [<8025a2f0>] 0x8025a2f0 [650033.671496] [<80007488>] 0x80007488 [650033.675064] [650033.676725] ---[ end trace 9e4a72dbc7b605fd ]--- [650033.681441] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out [650033.687689] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065 [650033.693795] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0f180000, max=0, ctx=2624, dtx=2624, fdx=2623, next=2624 [650033.704744] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0e520000, max=0, calc=3016, drx=3018 [650033.741926] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818 [650033.754545] mtk_soc_eth 1e100000.ethernet: PPE started [1057747.396763] mtk_soc_eth 1e100000.ethernet eth0: port 0 link down [1057750.653587] mtk_soc_eth 1e100000.ethernet eth0: port 0 link up

@openwrt-bot
Copy link
Author

Gektor:

Same issue on MikroTik RBM33G
SoC Type: MediaTek MT7621 ver:1 eco:3
Linux OpenWrt 4.14.180 #0 SMP Sat May 16 18:32:20 2020 mips GNU/Linux
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='19.07.3'
DISTRIB_REVISION='r11063-85e04e9f46'
DISTRIB_TARGET='ramips/mt7621'
DISTRIB_ARCH='mipsel_24kc'
DISTRIB_DESCRIPTION='OpenWrt 19.07.3 r11063-85e04e9f46'
DISTRIB_TAINTS=''

[140121.189586] ------------[ cut here ]------------
[140121.194302] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 0x8038ba10
[140121.201458] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[140121.208487] Modules linked in: qcserial pppoe ppp_async option ath10k_pci ath10k_core ath usb_wwan qmi_wwan pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY usbserial usbnet slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat cdc_wdm xt_set ip_set_list_set
[140121.279359] ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 mii leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[140121.323813] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.180 #0
[140121.329970] Stack : 00000000 00000000 00000000 8ff20a40 00000000 00000000 00000000 00000000
[140121.338393] 00000000 00000000 00000000 00000000 00000000 00000001 8fc09d60 ac07f5ad
[140121.346809] 8fc09df8 00000000 00000000 00007330 00000038 8049c0d8 00000008 00000000
[140121.355226] 00000000 80550000 0004f0e5 70617773 8fc09d40 00000000 00000000 8050ae44
[140121.363644] 8038ba10 00000140 00000000 8ff20a40 00000008 802accb0 00000000 806b0000
[140121.372061] ...
[140121.374583] Call Trace:
[140121.374600] [<8049c0d8>] 0x8049c0d8
[140121.380668] [<8038ba10>] 0x8038ba10
[140121.384223] [<802accb0>] 0x802accb0
[140121.387778] [<8000c1a0>] 0x8000c1a0
[140121.391334] [<8000c1a8>] 0x8000c1a8
[140121.394889] [<80484f34>] 0x80484f34
[140121.398443] [<80071a90>] 0x80071a90
[140121.402000] [<8002e608>] 0x8002e608
[140121.405556] [<8038ba10>] 0x8038ba10
[140121.409111] [<8002e690>] 0x8002e690
[140121.412668] [<8038ba10>] 0x8038ba10
[140121.416223] [<80099940>] 0x80099940
[140121.419780] [<8038b864>] 0x8038b864
[140121.423334] [<8008850c>] 0x8008850c
[140121.426887] [<8035ba9c>] 0x8035ba9c
[140121.430444] [<800887c8>] 0x800887c8
[140121.433999] [<800790f8>] 0x800790f8
[140121.437556] [<804a2ed8>] 0x804a2ed8
[140121.441115] [<80032fb4>] 0x80032fb4
[140121.444671] [<8025a2f0>] 0x8025a2f0
[140121.448227] [<80007488>] 0x80007488
[140121.451780]
[140121.453491] ---[ end trace ea6e6111189c7d38 ]---
[140121.458205] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
[140121.464502] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
[140121.470664] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0ec00000, max=0, ctx=2623, dtx=2623, fdx=2622, next=2623
[140121.481650] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0e2d0000, max=0, calc=3252, drx=3257
[140121.494774] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818
[140121.509194] mtk_soc_eth 1e100000.ethernet: PPE started

@openwrt-bot
Copy link
Author

Linaro1985:

Same issue on TP-Link Archer C20 V4 since OpenWrt 19.07.4

system type : MediaTek MT7628AN ver:1 eco:2
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='19.07.4'
DISTRIB_REVISION='r11208-ce6496d796'
DISTRIB_TARGET='ramips/mt76x8'
DISTRIB_ARCH='mipsel_24kc'
DISTRIB_DESCRIPTION='OpenWrt 19.07.4 r11208-ce6496d796'
DISTRIB_TAINTS=''

Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.053373] ------------[ cut here ]------------
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.062545] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 0x802c56b0
Tue Sep 15 08:11:57 2020 kern.info kernel: [ 3059.076526] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.090320] Modules linked in: pppoe ppp_async l2tp_ppp pptp pppox ppp_mppe ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 mt7603e mt76 mac80211 lzo lz4 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY ts_fsm ts_bm slhc nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_amanda nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.230994] nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conntrack lzo_decompress lzo_compress lz4_decompress lz4_compress iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ip_gre gre l2tp_ip6 l2tp_ip l2tp_eth l2tp_netlink l2tp_core udp_tunnel ip6_udp_tunnel ip_tunnel zram zsmalloc sha1_generic ecb crypto_acompress leds_gpio ehci_platform ehci_hcd gpio_button_hotplug usbcore nls_base usb_common aead cryptomgr crypto_null crypto_hash
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.337383] CPU: 0 PID: 0 Comm: swapper Not tainted 4.14.195 #0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.349111] Stack : 8046fee0 80054358 80480000 8042ed7c 00000000 00000000 00000000 00000000
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.365687] 00000000 00000000 00000000 00000000 00000000 00000001 83807de0 53261622
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.382255] 83807e78 00000000 00000000 00003a88 00000038 803cea38 00000007 00000000
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.398826] 00000000 80460000 000525e7 00000000 83807dc0 00000000 00000000 80428c30
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.415397] 802c56b0 00000140 8046ff34 8046fef4 00000002 8021cfc0 00000000 805b0000
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.431981] ...
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.436835] Call Trace:
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.436848] [<80054358>] 0x80054358
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.448642] [<803cea38>] 0x803cea38
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.455580] [<802c56b0>] 0x802c56b0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.462486] [<8021cfc0>] 0x8021cfc0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.469429] [<8000aa50>] 0x8000aa50
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.476350] [<8000aa58>] 0x8000aa58
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.483275] [<80026480>] 0x80026480
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.490210] [<802c56b0>] 0x802c56b0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.497156] [<80026508>] 0x80026508
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.504092] [<8026de70>] 0x8026de70
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.511020] [<802c56b0>] 0x802c56b0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.517957] [<802c554c>] 0x802c554c
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.524876] [<8005ffcc>] 0x8005ffcc
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.531773] [<8000e0a4>] 0x8000e0a4
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.538715] [<800601a0>] 0x800601a0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.545649] [<8029f198>] 0x8029f198
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.552562] [<803d44e8>] 0x803d44e8
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.559481] [<8005a554>] 0x8005a554
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.566419] [<80055ab0>] 0x80055ab0
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.573345] [<80005e90>] 0x80005e90
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.580239]
Tue Sep 15 08:11:57 2020 kern.warn kernel: [ 3059.583176] ---[ end trace 7c22dc092efa42ba ]---
Tue Sep 15 08:11:57 2020 kern.err kernel: [ 3059.592336] mtk_soc_eth 10100000.ethernet eth0: transmit timed out
Tue Sep 15 08:11:57 2020 kern.info kernel: [ 3059.604601] mtk_soc_eth 10100000.ethernet eth0: dma_cfg:00000055
Tue Sep 15 08:11:57 2020 kern.info kernel: [ 3059.616525] mtk_soc_eth 10100000.ethernet eth0: tx_ring=0, base=02cc8000, max=1024, ctx=507, dtx=507, fdx=455, next=507
Tue Sep 15 08:11:57 2020 kern.info kernel: [ 3059.637903] mtk_soc_eth 10100000.ethernet eth0: rx_ring=0, base=02ccc000, max=1024, calc=363, drx=369

@openwrt-bot
Copy link
Author

Internet:

same issue - router useless
Model Asus RT-AC51U
Architecture MediaTek MT7620A ver:2 eco:6
Firmware Version OpenWrt 19.07.4 r11208-ce6496d796 / LuCI openwrt-19.07 branch git-20.265.67308-c1fc83b

@openwrt-bot
Copy link
Author

Internet:

it seems it happens more often when there is less free memory when more processes are running, but free memory is still about 20 MB according to "Overview" page,

@openwrt-bot
Copy link
Author

mrakotiq:

Hi,
I hit this issue too with:
SoC Type: MediaTek MT7621 ver:1 eco:3
MIPS: machine is UBNT-ERX
Kernel: 4.14.209
openwrt: 19.7 d830403

I have this version deployed on approximately 50 routers in our network. It looks like the network traffic is the trigger, but I'm not sure. Before applying this patch https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=34a96529041d4e9502c490c66f8af0154187c6d2 I can reliably reproduce this in lab environment with sending trafic out "dd if=/dev/zero bs=4k |nc -u 10.19.19.194 887" and simultaneously send many small packet with random dst port through the router from packet generator.
This conditions raliably generate message like log below. With last version from 19.7 branch I can't reproduce this in lab environment but in production environment this occurs randomly several times a day.

Dec 6 16:26:58 doubrava711 kern.warn kernel: [21424.183592] ------------[ cut here ]------------
Dec 6 16:26:58 doubrava711 kern.warn kernel: [21424.183669] WARNING: CPU: 0 PID: 29230 at net/sched/sch_generic.c:320 dev_watchdog+0x1ac/0x324
Dec 6 16:26:58 doubrava711 kern.info kernel: [21424.183676] NETDEV WATCHDOG: eth1 (mtk_soc_eth): transmit queue 0 timed out
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.183680] Modules linked in: pppoe ppp_async pppox ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE ebtable_nat ebtable_filter ebtable_broute xt_time xt_tcpudp xt_tcpmss xt_s
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.183907] nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amand
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184131] ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_se
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184256] CPU: 0 PID: 29230 Comm: bash Not tainted 4.14.195 #0
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184262] Stack : 00000000 8fe06040 80580000 800720e4 805b0000 8054b71c 00000000 00000000
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184293] 805168bc 8fc09dc4 8e594ffc 80584907 80511608 00000001 8fc09d68 53261622
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184322] 00000000 00000000 806e0000 00004280 00000000 000000dc 00000000 00000000
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184350] 00000000 80590000 000000db 80710000 80000000 805b0000 00000000 80544604
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184379] 8036289c 00000140 00000000 8fe06040 00000000 80580000 00000000 806e0000
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184408] ...
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184417] Call Trace:
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184445] [<8000c6d0>] show_stack+0x58/0x100
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184477] [<804557b4>] dump_stack+0xa4/0xe0
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184498] [<8002f118>] __warn+0xe0/0x138
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184506] [<8002f1a0>] warn_slowpath_fmt+0x30/0x3c
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184517] [<8036289c>] dev_watchdog+0x1ac/0x324
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184551] [<80088dc8>] call_timer_fn.isra.25+0x24/0x84
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184561] [<8008907c>] run_timer_softirq+0x1b4/0x240
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184591] [<80473750>] __do_softirq+0x128/0x2ec
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184607] [<80033acc>] irq_exit+0xac/0xc8
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184625] [<80240e80>] plat_irq_dispatch+0xfc/0x138
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184634] [<800074c8>] except_vec_vi_end+0xb8/0xc4
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184643] [<8002ddf8>] copy_process.part.91+0x1250/0x1554
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184652] [<8002e290>] _do_fork+0xe0/0x2fc
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184660] [<8002e50c>] sys_fork+0x24/0x30
Dec 6 16:26:58 doubrava711 kern.alert kernel: [21424.184682] [<80015578>] syscall_common+0x34/0x58
Dec 6 16:26:58 doubrava711 kern.warn kernel: [21424.184697] ---[ end trace 8d819b188bbe37e8 ]---
Dec 6 16:26:58 doubrava711 kern.err kernel: [21424.184712] mtk_soc_eth 1e100000.ethernet eth1: transmit timed out
Dec 6 16:26:58 doubrava711 kern.info kernel: [21424.184722] mtk_soc_eth 1e100000.ethernet eth1: dma_cfg:80000067
Dec 6 16:26:58 doubrava711 kern.info kernel: [21424.184736] mtk_soc_eth 1e100000.ethernet eth1: tx_ring=0, base=0e950000, max=0, ctx=308, dtx=273, fdx=273, next=308
Dec 6 16:26:58 doubrava711 kern.info kernel: [21424.184748] mtk_soc_eth 1e100000.ethernet eth1: rx_ring=0, base=0d8e0000, max=0, calc=2748, drx=2749
Dec 6 16:26:59 doubrava711 kern.info kernel: [21424.586508] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3560170c, 0x10c = 0x80818
Dec 6 16:26:59 doubrava711 kern.info kernel: [21424.586527] mtk_soc_eth 1e100000.ethernet: reset pse
Dec 6 16:26:59 doubrava711 kern.info kernel: [21424.591792] mtk_soc_eth 1e100000.ethernet: PPE started
Dec 6 16:27:08 doubrava711 kern.err kernel: [21434.183552] mtk_soc_eth 1e100000.ethernet eth1: transmit timed out
Dec 6 16:27:08 doubrava711 kern.info kernel: [21434.183579] mtk_soc_eth 1e100000.ethernet eth1: dma_cfg:80000067
Dec 6 16:27:08 doubrava711 kern.info kernel: [21434.183594] mtk_soc_eth 1e100000.ethernet eth1: tx_ring=0, base=0ef50000, max=0, ctx=1, dtx=0, fdx=0, next=1
Dec 6 16:27:08 doubrava711 kern.info kernel: [21434.183606] mtk_soc_eth 1e100000.ethernet eth1: rx_ring=0, base=0c8d0000, max=0, calc=722, drx=723
Dec 6 16:27:09 doubrava711 kern.info kernel: [21434.586354] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x4a600a0c, 0x10c = 0x80818
Dec 6 16:27:09 doubrava711 kern.info kernel: [21434.591715] mtk_soc_eth 1e100000.ethernet: PPE started
Dec 6 16:27:18 doubrava711 kern.err kernel: [21444.183543] mtk_soc_eth 1e100000.ethernet eth1: transmit timed out
Dec 6 16:27:18 doubrava711 kern.info kernel: [21444.183570] mtk_soc_eth 1e100000.ethernet eth1: dma_cfg:80000067
Dec 6 16:27:18 doubrava711 kern.info kernel: [21444.183584] mtk_soc_eth 1e100000.ethernet eth1: tx_ring=0, base=0edb0000, max=0, ctx=1, dtx=0, fdx=0, next=1
Dec 6 16:27:18 doubrava711 kern.info kernel: [21444.183596] mtk_soc_eth 1e100000.ethernet eth1: rx_ring=0, base=0c860000, max=0, calc=567, drx=568
Dec 6 16:27:19 doubrava711 kern.info kernel: [21444.586349] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x3f60150c, 0x10c = 0x80818
Dec 6 16:27:19 doubrava711 kern.info kernel: [21444.586368] mtk_soc_eth 1e100000.ethernet: reset pse
Dec 6 16:27:19 doubrava711 kern.info kernel: [21444.591657] mtk_soc_eth 1e100000.ethernet: PPE started
Dec 6 16:27:28 doubrava711 kern.err kernel: [21454.183606] mtk_soc_eth 1e100000.ethernet eth1: transmit timed out
Dec 6 16:27:28 doubrava711 kern.info kernel: [21454.183633] mtk_soc_eth 1e100000.ethernet eth1: dma_cfg:80000067
Dec 6 16:27:28 doubrava711 kern.info kernel: [21454.183647] mtk_soc_eth 1e100000.ethernet eth1: tx_ring=0, base=0c9b0000, max=0, ctx=1, dtx=0, fdx=0, next=1
Dec 6 16:27:28 doubrava711 kern.info kernel: [21454.183659] mtk_soc_eth 1e100000.ethernet eth1: rx_ring=0, base=0ee10000, max=0, calc=687, drx=688
Dec 6 16:27:29 doubrava711 kern.info kernel: [21454.586350] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x5860000c, 0x10c = 0x80818
Dec 6 16:27:29 doubrava711 kern.info kernel: [21454.591654] mtk_soc_eth 1e100000.ethernet: PPE started
...

And this sometimes leads to kernel panic in (skbuff: skb_over_panic:) and reboot.
Crashlog from two routers in production is attached.

If anyone has an idea how to fix it. I can check in a very short time if the error is fixed. Or extract more debug information.

@openwrt-bot
Copy link
Author

bjonglez:

I can confirm I saw this warning on UBNT-ERX with 19.07.2 / kernel 4.14.171:

DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='19.07.2'
DISTRIB_REVISION='r10947-65030d81f3'
DISTRIB_TARGET='ramips/mt7621'
DISTRIB_ARCH='mipsel_24kc'
DISTRIB_DESCRIPTION='OpenWrt 19.07.2 r10947-65030d81f3'
DISTRIB_TAINTS=''

We also had one unexplained crash in production in the past year, so it's probably related. The device doesn't see a lot of traffic.

@openwrt-bot
Copy link
Author

mrakotiq:

And sometimes this "transmit timed out" leads to the sending of the malformed packets just after "mtk_soc_eth 1e100000.ethernet: PPE started" appears in log. For example, the packet type is 0x8100 but 4B defining the vlan tag is missed and this leads to misinterpretation of the ip header as a vlan tag.

@openwrt-bot
Copy link
Author

mrakotiq:

I can reproduce this in lab environment very reliably again.

Please, if anyone has an idea of what I should try or what I should focus on. Let me know.

@openwrt-bot
Copy link
Author

mrakotiq:

Can someone please confirm that this is the right place to report issues like this?

May someone notice authors of this driver (John Crispin, Felix Fietkau, Michael Lee) to this thread. Because I thing no one else here seems to have enough experience to solve this strange problem. :(

@openwrt-bot
Copy link
Author

mrakotiq:

This little patch solved the "mtk_soc_eth 1e100000.ethernet eth0: transmit timed out". At least for now. Now it cannot be reproduced in a lab or on a 75pcs routers in network.

Please try someone too and confirm if it helped.

diff --git a/target/linux/ramips/files-4.14/drivers/net/ethernet/mediatek/soc_mt7621.c b/target/linux/ramips/files-4.14/drivers
index 26a198f..597d50a 100644
--- a/target/linux/ramips/files-4.14/drivers/net/ethernet/mediatek/soc_mt7621.c
+++ b/target/linux/ramips/files-4.14/drivers/net/ethernet/mediatek/soc_mt7621.c
@@ -142,8 +142,9 @@ static void mt7621_init_data(struct fe_soc_data *data,
FE_FLAG_HAS_SWITCH | FE_FLAG_JUMBO_FRAME;

    netdev->hw_features = NETIF_F_IP_CSUM | NETIF_F_RXCSUM |
  •           NETIF_F_HW_VLAN_CTAG_TX | NETIF_F_HW_VLAN_CTAG_RX |
    
  •           NETIF_F_SG | NETIF_F_TSO |
    

+// NETIF_F_HW_VLAN_CTAG_TX | NETIF_F_HW_VLAN_CTAG_RX |

  •           NETIF_F_SG |
    

+// NETIF_F_TSO |
NETIF_F_TSO6 | NETIF_F_IPV6_CSUM |
NETIF_F_TSO_MANGLEID;
}

BTW: i removeNETIF_F_HW_VLAN_CTAG_TX | NETIF_F_HW_VLAN_CTAG_RX because in our vlan configuration it also tagged packets that it did not have. The key is remove TSO offloading, I thing.

@openwrt-bot
Copy link
Author

bjonglez:

Good catch, well done!

I had a look and this driver was removed upstream: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=bd31342f0046077e92062a6c09eae6c8f1676916

Upstream is using the generic mediatek driver, and this is also what is used in OpenWrt master with kernel 5.4. The generic driver does enable TSO, so TSO is likely a symptom and not the root cause.

Have you noticed any performance regression when disabling TSO? Did you try to reproduce the issue on latest OpenWrt master?

@openwrt-bot
Copy link
Author

mrakotiq:

//so TSO is likely a symptom and not the root cause. //
I agree, but obviously it has something to do with it.

//performance regression when disabling TSO//
It's too fresh and I don't do any performance tests yet, but there is no visible difference in the avg load and cpu usage graphs. So if there is an impact, then only insignificant

@openwrt-bot
Copy link
Author

mrakotiq:

//Did you try to reproduce the issue on latest OpenWrt master?//

Unfortunately, I don't have time for this yet.

@openwrt-bot
Copy link
Author

bjonglez:

I'm trying to reproduce the crash to test your fix. Are you still using the same method to reproduce? ("dd if=/dev/zero bs=4k |nc -u 10.19.19.194 887 and simultaneously send many small packet with random dst port through the router from packet generator")

Also, did you have flow offloading enabled? (soft, hard?)

It's a bit strange that you reproduced this with UDP traffic if the issue is related to TSO (because it should only apply to TCP).

I'm hoping we can integrate a fix (or at least a workaround) for the upcoming 19.07.7 point release.

@openwrt-bot
Copy link
Author

bjonglez:

I am not able to reproduce with a current 19.07 snapshot (self-build, r11297-4465b44fc1). I have been sending 32 iperf3 UDP flows with small packets + iperf3 TCP flows through the router, and also sending UDP packets from the router itself with "dd if=/dev/zero bs=4k |nc -u ". No crash in 24h, with and without software flow offloading.

So, either:

  1. it has been fixed (but I don't see how, except maybe for a kernel bump)
  2. it is hard to reproduce
  3. it depends on build parameters
  4. it depends on runtime parameters like VLAN configuration

@openwrt-bot
Copy link
Author

mrakotiq:

I'm currently in time press, so only briefly

Later I reliably reproduced the problem by sending a UDP packet with:
dd if=/dev/urandom bs=4k |nc -u 10.19.19.194 888
And replayig pcap through router from my workstation (eth1: RTL8168evl/8111evl, gso,tso off ) with command:
tcpreplay-edit -l 50 -F pad --enet-dmac=74:ac:b9:a2:5a:5a,74:ac:b9:a2:5a:5a -i eth1 ./linuxbox.pcap

Pcap was captured in real network where he caused this error. Because it may contain sensitive data, I will send it to you by email.
By this procedure I was able to reproduce the error every time tcpreplay was run.

//For udp stream it is necessary to pay attention so that an icmp packet does not return from somewhere, which could stop the flow. And verify that the packets from tcpreplay are actually passing through the router //

BTW: When I have time, I will try to reproduce this with the current build.

@openwrt-bot
Copy link
Author

pchernikhowsky:

Still seeing this bug in the (near) latest snapshot. Software offloading enabled, hardware offloading disabled.

Any suggestions to try?

Device: UBNT-ERX-SFP

DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='19.07-SNAPSHOT'
DISTRIB_REVISION='r11294-1e90091c5d'
DISTRIB_TARGET='ramips/mt7621'
DISTRIB_ARCH='mipsel_24kc'
DISTRIB_DESCRIPTION='OpenWrt 19.07-SNAPSHOT r11294-1e90091c5d'
DISTRIB_TAINTS='no-all busybox'

Mon Feb 8 13:34:58 2021 kern.warn kernel: [157077.481513] ------------[ cut here ]------------
Mon Feb 8 13:34:58 2021 kern.warn kernel: [157077.490922] WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x1ac/0x324
Mon Feb 8 13:34:58 2021 kern.info kernel: [157077.507557] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.521614] Modules linked in: nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt i2c_gpio i2c_algo_pca i2c_algo_bit gpio_pca953x i2c_dev xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.665156] ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 tun leds_gpio gpio_button_hotplug
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.698268] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.14.218 #0
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.710558] Stack : 00000000 8fe55740 80580000 8007297c 805a0000 80549e74 00000000 00000000
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.727365] 8051595c 8fc0fdc4 8fc3cffc 80582907 805106f0 00000001 8fc0fd68 5326163b
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.744174] 00000000 00000000 806e0000 00003c10 00000000 000000cb 00000008 00000000
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.760981] 00000000 80580000 000aa79c 70617773 00000000 805a0000 00000000 8054407c
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.777787] 803723a0 00000140 00000003 8fe55740 00000008 8029aef8 0000000c 806e000c
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.794592] ...
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.799624] Call Trace:
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.804684] [<8000c7b0>] show_stack+0x58/0x100
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.813724] [<804523e4>] dump_stack+0xa4/0xe0
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.822568] [<8002f798>] __warn+0xe0/0x138
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.830885] [<8002f820>] warn_slowpath_fmt+0x30/0x3c
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.840941] [<803723a0>] dev_watchdog+0x1ac/0x324
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.850494] [<80089758>] call_timer_fn.isra.25+0x24/0x84
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.861231] [<80089a14>] run_timer_softirq+0x1bc/0x248
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.871625] [<804702b0>] __do_softirq+0x128/0x2ec
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.881159] [<80034164>] irq_exit+0xac/0xc8
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.889667] [<8024d9f0>] plat_irq_dispatch+0xfc/0x138
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.899890] [<80007588>] except_vec_vi_end+0xb8/0xc4
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.909937] [<80008f50>] r4k_wait_irqoff+0x1c/0x24
Mon Feb 8 13:34:59 2021 kern.warn kernel: [157077.919724] ---[ end trace 9c3e786e34da7819 ]---
Mon Feb 8 13:34:59 2021 kern.err kernel: [157077.929096] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Mon Feb 8 13:34:59 2021 kern.info kernel: [157077.941582] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
Mon Feb 8 13:34:59 2021 kern.info kernel: [157077.953721] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0f1a0000, max=0, ctx=162, dtx=162, fdx=161, next=162
Mon Feb 8 13:34:59 2021 kern.info kernel: [157077.974850] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0eb30000, max=0, calc=2099, drx=2100
Mon Feb 8 13:34:59 2021 kern.info kernel: [157078.000163] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818
Mon Feb 8 13:34:59 2021 kern.info kernel: [157078.020303] mtk_soc_eth 1e100000.ethernet: PPE started

@openwrt-bot
Copy link
Author

bjonglez:

Thanks, with your input I can reproduce a crash (RCU stall). It takes a few hours. I did not have any offloading enabled.

I am unable to reproduce on latest master, but that is not surprising, it's using a different driver.

I'm now testing a variant of your patch.

@openwrt-bot
Copy link
Author

bjonglez:

Disabling TSO makes things more stable, with a small hit on forwarding performance.

While it's probably not a complete fix, I have pushed this as a workaround for the next 19.07.7 release: https://git.openwrt.org/f8b849103dfe1b21a6462837efc9460ca3cca02d

@openwrt-bot
Copy link
Author

pchernikhowsky:

I built a custom 19.07-SNAPSHOT r11306 for my UBNT-ERX-SFP and installed it last night. No hiccups yet in 20 hours of uptime (but it would often run for several days with no errors before). I'm on gigabit symmetrical GPON and there has been some impact on performance, but I prefer the increased stability (it it works!).

@openwrt-bot
Copy link
Author

pchernikhowsky:

Booo!! It's still there - see below. The router didn't crash though - in fact, I never even noticed a blip (I was online working at the time), but something bad is still going on. So this is not the fix either. I'm starting to think this driver is hopeless until I switch to the 21.0x branch and the new kernel.

Model UBNT-ERX-SFP
Architecture MediaTek MT7621 ver:1 eco:3
Firmware Version OpenWrt 19.07-SNAPSHOT r11306-c4a6851c72 / LuCI openwrt-19.07 branch git-21.036.67598-2936a19
Kernel Version 4.14.221
Uptime 4d 16h 34m 16s

Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.039911] ------------[ cut here ]------------
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.049315] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x1ac/0x324
Sat Feb 20 11:16:20 2021 kern.info kernel: [394842.065955] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.079985] Modules linked in: nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt i2c_gpio i2c_algo_pca i2c_algo_bit gpio_pca953x i2c_dev xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.223523] ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 tun leds_gpio gpio_button_hotplug
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.256659] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.221 #0
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.268951] Stack : 00000000 8fd72740 80580000 80072a6c 805a0000 80549e34 00000000 00000000
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.285756] 8051591c 8fc0ddc4 8fc3c99c 80582907 805106b0 00000001 8fc0dd68 5326163c
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.302558] 00000000 00000000 806e0000 000049a0 00000000 000000e6 00000008 00000000
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.319358] 00000000 80580000 0003ea93 70617773 00000000 805a0000 00000000 8054403c
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.336162] 80372440 00000140 00000002 8fd72740 00000008 8029af38 00000008 806e0008
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.352970] ...
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.358003] Call Trace:
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.363061] [<8000c7b0>] show_stack+0x58/0x100
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.372095] [<80452494>] dump_stack+0xa4/0xe0
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.380937] [<8002f798>] __warn+0xe0/0x138
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.389248] [<8002f820>] warn_slowpath_fmt+0x30/0x3c
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.399300] [<80372440>] dev_watchdog+0x1ac/0x324
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.408846] [<80089848>] call_timer_fn.isra.25+0x24/0x84
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.419580] [<80089b04>] run_timer_softirq+0x1bc/0x248
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.429971] [<80470370>] __do_softirq+0x128/0x2ec
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.439509] [<80034164>] irq_exit+0xac/0xc8
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.448019] [<8024da30>] plat_irq_dispatch+0xfc/0x138
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.458238] [<80007588>] except_vec_vi_end+0xb8/0xc4
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.468281] [<80008f50>] r4k_wait_irqoff+0x1c/0x24
Sat Feb 20 11:16:20 2021 kern.warn kernel: [394842.478062] ---[ end trace 97a6b717df71c27c ]---
Sat Feb 20 11:16:20 2021 kern.err kernel: [394842.487430] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Sat Feb 20 11:16:20 2021 kern.info kernel: [394842.499915] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
Sat Feb 20 11:16:20 2021 kern.info kernel: [394842.512050] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0eec0000, max=0, ctx=2473, dtx=2473, fdx=2472, next=2473
Sat Feb 20 11:16:20 2021 kern.info kernel: [394842.533875] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0f350000, max=0, calc=734, drx=735
Sat Feb 20 11:16:20 2021 kern.info kernel: [394842.557254] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818
Sat Feb 20 11:16:20 2021 kern.info kernel: [394842.577540] mtk_soc_eth 1e100000.ethernet: PPE started

@openwrt-bot
Copy link
Author

mrakotiq:

Yes, I can confirm that.

NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
will occasionally occur but will not cause any damage.

@openwrt-bot
Copy link
Author

nelgin:

I'm using a Netgear R6220 router and having the same issues.

I was using a snapshot from probably back in April last year and everything was working fine. I decided to experiment with i2pd, unfortunately my file system was rendered read-only due to filling up. I decided I was going to do 2 things.

  1. Upgraded to the latest 19.07.7
  2. extended the filesystem to a USB stick following the instructions at: https://openwrt.org/docs/guide-user/additional-software/extroot_configuration

[code]
[Sun Feb 28 13:38:23 2021] ------------[ cut here ]------------
[Sun Feb 28 13:38:23 2021] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 0x8038d700
[Sun Feb 28 13:38:23 2021] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[Sun Feb 28 13:38:23 2021] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD xt_CT wireguard slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt compat ledtrig_usbport xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport
[Sun Feb 28 13:38:23 2021] ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ehci_platform ip6_gre ip_gre gre ehci_hcd ip6_udp_tunnel udp_tunnel sit ip6_tunnel tunnel6 tunnel4 ip_tunnel usb_storage uhci_hcd ohci_platform ohci_hcd sd_mod scsi_mod ext4 mbcache jbd2 crc32c_generic leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[Sun Feb 28 13:38:23 2021] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.221 #0
[Sun Feb 28 13:38:23 2021] Stack : 00000000 00000000 00000000 87f77540 00000000 00000000 00000000 00000000
[Sun Feb 28 13:38:24 2021] 00000000 00000000 00000000 00000000 00000000 00000001 87c09d60 5326163c
[Sun Feb 28 13:38:24 2021] 87c09df8 00000000 00000000 000075e8 00000038 8049e2b8 00000008 00000000
[Sun Feb 28 13:38:24 2021] 00000000 80550000 000f0274 00000000 87c09d40 00000000 00000000 8050d500
[Sun Feb 28 13:38:24 2021] 8038d700 00000140 00000000 87f77540 00000000 802ae4c0 00000000 806b0000
[Sun Feb 28 13:38:24 2021] ...
[Sun Feb 28 13:38:24 2021] Call Trace:
[Sun Feb 28 13:38:24 2021] [<8049e2b8>] 0x8049e2b8
[Sun Feb 28 13:38:24 2021] [<8038d700>] 0x8038d700
[Sun Feb 28 13:38:24 2021] [<802ae4c0>] 0x802ae4c0
[Sun Feb 28 13:38:24 2021] [<8000c1a0>] 0x8000c1a0
[Sun Feb 28 13:38:24 2021] [<8000c1a8>] 0x8000c1a8
[Sun Feb 28 13:38:24 2021] [<804870f4>] 0x804870f4
[Sun Feb 28 13:38:24 2021] [<80071e00>] 0x80071e00
[Sun Feb 28 13:38:24 2021] [<8002e798>] 0x8002e798
[Sun Feb 28 13:38:24 2021] [<8038d700>] 0x8038d700
[Sun Feb 28 13:38:24 2021] [<8002e820>] 0x8002e820
[Sun Feb 28 13:38:24 2021] [<86d20104>] 0x86d20104 [mt76x02_lib@86d20000+0x9760]
[Sun Feb 28 13:38:24 2021] [<8038d700>] 0x8038d700
[Sun Feb 28 13:38:24 2021] [<8038d554>] 0x8038d554
[Sun Feb 28 13:38:24 2021] [<80088948>] 0x80088948
[Sun Feb 28 13:38:24 2021] [<8035d6bc>] 0x8035d6bc
[Sun Feb 28 13:38:24 2021] [<80088c04>] 0x80088c04
[Sun Feb 28 13:38:24 2021] [<800794a8>] 0x800794a8
[Sun Feb 28 13:38:24 2021] [<804a50b8>] 0x804a50b8
[Sun Feb 28 13:38:24 2021] [<80033164>] 0x80033164
[Sun Feb 28 13:38:24 2021] [<80050818>] 0x80050818
[Sun Feb 28 13:38:24 2021] [<8025b710>] 0x8025b710
[Sun Feb 28 13:38:24 2021] [<80007488>] 0x80007488

[Sun Feb 28 13:38:24 2021] ---[ end trace edf46818e20cf603 ]---
[Sun Feb 28 13:38:24 2021] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
[Sun Feb 28 13:38:24 2021] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
[Sun Feb 28 13:38:24 2021] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=06b70000, max=0, ctx=396, dtx=396, fdx=395, next=396
[Sun Feb 28 13:38:24 2021] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=05f10000, max=0, calc=296, drx=373
[Sun Feb 28 13:38:24 2021] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818
[Sun Feb 28 13:38:24 2021] mtk_soc_eth 1e100000.ethernet: PPE started
[/code]

The router seems to lock up overnight when it's not being used as much but it's random.

@openwrt-bot
Copy link
Author

Thor:

Problem still occurs with 19.07.7 on Mikrotik RB750gr3. I see these errors on my rsyslog server on a daily basis. They don't seem to cause a problem most of the time, but occasionally a router will become unresponsive and needs to be power cycled. I can't verify whether this error is also what causes the hangs, but given how many people are seeing this error message in connection with such hangs, it seems to be the likely culprit:

Feb 23 08:01:19 kernel: [26984.986368] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x1ac/0x
324
Feb 23 08:01:19 kernel: [26984.994619] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
Feb 23 08:01:19 kernel: [26985.001559] Modules linked in: pppoe ppp_async pppox ppp_generic iptable_nat ipt_REJECT
ipt_MASQUERADE xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl
xt_ecn xt_dscp xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CLASSIFY slhc nf_reject_ip
v4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_log_common nf_flow_table_hw nf_f
low_table nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt gpio_beep
er input_core xt_set x_tables ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netifa
ce ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport
Feb 23 08:01:19 kernel: [26985.072669] ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink tun leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
Feb 23 08:01:19 kernel: [26985.092424] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.221 #0
Feb 23 08:01:20 kernel: [26985.098489] Stack : 00000000 8ff1b240 80530000 80072a6c 80560000 804fcfa4 00000000 00000000
Feb 23 08:01:20 kernel: [26985.106829] 804ca4cc 8fc09dc4 8053673c 805362c7 804c5260 00000001 8fc09d68 ac07f5c4
Feb 23 08:01:20 kernel: [26985.115169] 00000000 00000000 80690000 000039b0 00000000 000000bd 00000008 00000000
Feb 23 08:01:20 kernel: [26985.123502] 00000000 80540000 00016908 00000000 00000000 80560000 00000000 804f8c30
Feb 23 08:01:20 kernel: [26985.131834] 80372b50 00000140 00000000 8ff1b240 00000000 8029af38 00000000 80690000
Feb 23 08:01:20 kernel: [26985.140165] ...
Feb 23 08:01:20 kernel: [26985.142602] Call Trace:
Feb 23 08:01:20 kernel: [26985.145063] [<8000c7b0>] show_stack+0x58/0x100
Feb 23 08:01:20 kernel: [26985.149494] [<8040f1f4>] dump_stack+0xa4/0xe0
Feb 23 08:01:20 kernel: [26985.153840] [<8002f798>] __warn+0xe0/0x138
Feb 23 08:01:20 kernel: [26985.157917] [<8002f820>] warn_slowpath_fmt+0x30/0x3c
Feb 23 08:01:20 kernel: [26985.162867] [<80372b50>] dev_watchdog+0x1ac/0x324
Feb 23 08:01:20 kernel: [26985.167573] [<80089848>] call_timer_fn.isra.25+0x24/0x84
Feb 23 08:01:20 kernel: [26985.172862] [<80089b04>] run_timer_softirq+0x1bc/0x248
Feb 23 08:01:20 kernel: [26985.177981] [<8042ca30>] __do_softirq+0x128/0x2ec
Feb 23 08:01:20 kernel: [26985.182671] [<80034164>] irq_exit+0xac/0xc8
Feb 23 08:01:20 kernel: [26985.186853] [<8024da30>] plat_irq_dispatch+0xfc/0x138
Feb 23 08:01:20 kernel: [26985.191883] [<80007588>] except_vec_vi_end+0xb8/0xc4
Feb 23 08:01:20 kernel: [26985.196824] [<80008f50>] r4k_wait_irqoff+0x1c/0x24
Feb 23 08:01:20 kernel: [26985.201698] ---[ end trace 3a4bde47433ec4d2 ]---
Feb 23 08:01:20 kernel: [26985.206323] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Feb 23 08:01:20 kernel: [26985.212524] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
Feb 23 08:01:20 kernel: [26985.218527] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0eee0000, max=0, ctx=3157, dtx=3157, fdx=3156, next=3157
Feb 23 08:01:20 kernel: [26985.229388] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0f380000, max=0, calc=3148, drx=3153
Feb 23 08:01:20 kernel: [26985.242005] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818
Feb 23 08:01:20 kernel: [26985.255979] mtk_soc_eth 1e100000.ethernet: PPE started

@openwrt-bot
Copy link
Author

Thor:

I might attach a serial console to one of my routers to see if I can catch it red-handed. I hear this isn't an issue with the new kernel on the master branch, but I would much prefer to wait until there is a tagged release with a seamless migration path away from swconfig, with a working LUCI GUI for the switch.

@openwrt-bot
Copy link
Author

nelgin:

I've updated my Netgear R6220 router to the latest snapshot at the time of compilation and it's been up and running with no issues for 48 hours apart from one of the ports flapping. I m moved the cable to another port and that stopped it so everything is stable including ddns updates, ipv6 in ipv4 tunnel with HE, etc.

@openwrt-bot
Copy link
Author

Thor:

Snapshots are taken from the master development branch, which has migrated to a new Linux kernel that uses a completely different Ethernet driver. It doesn't have the same issue. As I understand it, the official releases are largely backports at this point. I expect that there will be an official release based on the master branch at some point.

The bug will disappear when this migration happens. In the meanwhile, I hope they'll fix this in an official release, because it's a major pain point for me.

@openwrt-bot
Copy link
Author

compassnet:

Device: Mikrotik RB750Gr3

This problem still persists with OpenWrt 19.07.7 without crashing the router.
In my case it's rare to see it happen, but this bug is not gone.

Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.684019] ------------[ cut here ]------------
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.688743] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:320 0x8038d700
Sun Mar 14 00:26:24 2021 kern.info kernel: [618046.695866] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.702878] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD xt_CT wireguard slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt gpio_beeper input_core nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ip6_udp_tunnel udp_tunnel leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.771181] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.14.221 #0
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.777351] Stack : 00000000 00000000 00000000 8ffd2540 00000000 00000000 00000000 00000000
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.785791] 00000000 00000000 00000000 00000000 00000000 00000001 8fc0dd60 ac07f5c4
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.794213] 8fc0ddf8 00000000 00000000 00003a50 00000038 8049e2b8 00000008 00000000
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.802633] 00000000 80550000 000bc46d 70617773 8fc0dd40 00000000 00000000 8050d500
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.811055] 8038d700 00000140 00000002 8ffd2540 00000008 802ae4c0 00000008 806b0008
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.819479] ...
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.822002] Call Trace:
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.822019] [<8049e2b8>] 0x8049e2b8
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.828090] [<8038d700>] 0x8038d700
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.831649] [<802ae4c0>] 0x802ae4c0
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.835210] [<8000c1a0>] 0x8000c1a0
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.838770] [<8000c1a8>] 0x8000c1a8
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.842326] [<804870f4>] 0x804870f4
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.845883] [<80071e00>] 0x80071e00
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.849443] [<8002e798>] 0x8002e798
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.852999] [<8038d700>] 0x8038d700
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.856560] [<8002e820>] 0x8002e820
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.860117] [<800553b8>] 0x800553b8
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.863672] [<8038d700>] 0x8038d700
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.867230] [<80099da0>] 0x80099da0
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.870788] [<8038d554>] 0x8038d554
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.874348] [<80088948>] 0x80088948
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.877908] [<80088c04>] 0x80088c04
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.881465] [<800794a8>] 0x800794a8
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.885055] [<804a50b8>] 0x804a50b8
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.888614] [<80033164>] 0x80033164
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.892168] [<8025b710>] 0x8025b710
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.895729] [<80007488>] 0x80007488
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.899284]
Sun Mar 14 00:26:24 2021 kern.warn kernel: [618046.900928] ---[ end trace 7a5a44bbe7aa90ab ]---
Sun Mar 14 00:26:24 2021 kern.err kernel: [618046.905681] mtk_soc_eth 1e100000.ethernet eth0: transmit timed out
Sun Mar 14 00:26:24 2021 kern.info kernel: [618046.911953] mtk_soc_eth 1e100000.ethernet eth0: dma_cfg:80000065
Sun Mar 14 00:26:24 2021 kern.info kernel: [618046.918056] mtk_soc_eth 1e100000.ethernet eth0: tx_ring=0, base=0e9e0000, max=0, ctx=2470, dtx=2470, fdx=2469, next=2470
Sun Mar 14 00:26:24 2021 kern.info kernel: [618046.929017] mtk_soc_eth 1e100000.ethernet eth0: rx_ring=0, base=0ee70000, max=0, calc=3218, drx=3219
Sun Mar 14 00:26:24 2021 kern.info kernel: [618046.941465] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818
Sun Mar 14 00:26:24 2021 kern.info kernel: [618046.954454] mtk_soc_eth 1e100000.ethernet: PPE started

Compass.

@openwrt-bot
Copy link
Author

hunterzero99:

I had a hard lock last night on the following hardware/software version. System froze, not allowing connections via http or ssh until I unplugged it to reboot. Stability is greatly improved from the previous release, though.
I remember testing master about 3 months ago and still having hard locks on this hardware as well.

Model: UBNT-ERX
Architecture: MediaTek MT7621 ver:1 eco:3
Firmware Version: OpenWrt 19.07.7 r11306-c4a6851c72 / LuCI openwrt-19.07 branch git-21.050.37945-c33df8f

Mar 24 19:52:02 OpenWrt dnsmasq[2322]: reducing DNS packet size for nameserver 127.0.0.1 to 1280
Mar 24 19:52:05 OpenWrt dnsmasq[2322]: reducing DNS packet size for nameserver ::1 to 1280
Mar 24 19:52:07 OpenWrt dnsmasq-dhcp[2322]: DHCPDISCOVER(br-lan) 72:f7:07:13:2e:68
Mar 24 19:52:07 OpenWrt dnsmasq-dhcp[2322]: DHCPOFFER(br-lan) 192.168.1.118 72:f7:07:13:2e:68
Mar 24 19:52:07 OpenWrt dnsmasq-dhcp[2322]: DHCPREQUEST(br-lan) 192.168.1.118 72:f7:07:13:2e:68
Mar 24 19:52:07 OpenWrt dnsmasq-dhcp[2322]: DHCPACK(br-lan) 192.168.1.118 72:f7:07:13:2e:68
Mar 24 19:53:04 OpenWrt kernel: [877339.654753] INFO: rcu_sched self-detected stall on CPU
Mar 24 19:53:04 OpenWrt kernel: [877339.665213] 3-...: (1 GPs behind) idle=c7e/140000000000001/0 softirq=20489311/20489313 fqs=1797
Mar 24 19:53:04 OpenWrt kernel: [877339.674748] INFO: rcu_sched detected stalls on CPUs/tasks:
Mar 24 19:53:04 OpenWrt kernel: [877339.683044]
Mar 24 19:53:04 OpenWrt kernel: [877339.694140] (t=6003 jiffies g=9063883 c=9063882 q=1017)
Mar 24 19:53:04 OpenWrt kernel: [877339.697461] 3-...: (1 GPs behind) idle=c7e/140000000000001/0 softirq=20489311/20489313 fqs=1797
Mar 24 19:53:04 OpenWrt kernel: [877339.697481] rcu_sched kthread starved for 2409 jiffies! g9063883 c9063882 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1
Mar 24 19:53:04 OpenWrt kernel: [877339.697496] rcu_sched I
Mar 24 19:53:04 OpenWrt kernel: [877339.715331]
Mar 24 19:53:04 OpenWrt kernel: [877339.736938] 0 8 2 0x00100000
Mar 24 19:53:04 OpenWrt kernel: [877339.736953] (detected by 1, t=6008 jiffies, g=9063883, c=9063882, q=1024)
Mar 24 19:53:04 OpenWrt kernel: [877339.740235] Stack :
Mar 24 19:53:04 OpenWrt kernel: [877339.740245] Sending NMI from CPU 1 to CPUs 3:
Mar 24 19:53:04 OpenWrt kernel: [877339.740249] 805aada0 8fc3aca0 8fc57e38 8fc57e38 8fc57e38 80550000 81222380 00000000
Mar 24 19:53:04 OpenWrt kernel: [877339.740277] 8123fda0 80052d18 053a3714 81222380 8fc57e38 80550000 81222380 00000000
Mar 24 19:53:04 OpenWrt kernel: [877339.740307] 00000003 8054c1e8 80554610 804a0638 8fc57e38 80550000 80550000 00000000
Mar 24 19:53:04 OpenWrt kernel: [877339.740336] 00000003 804a3e50 80550000 00000000 8fc57ebc 800683a8 81216420 00000001
Mar 24 19:53:04 OpenWrt kernel: [877339.740365] 00000000 8122242c 053a3714 800888b0 8fc3aca0 05400001 80554500 00000000
Mar 24 19:53:04 OpenWrt kernel: [877339.740394] ...
Mar 24 19:53:04 OpenWrt kernel: [877339.740402] Call Trace:
Mar 24 19:53:04 OpenWrt kernel: [877339.740417] [<80052d18>] 0x80052d18
Mar 24 19:53:04 OpenWrt kernel: [877339.740424] [<804a0638>] 0x804a0638
Mar 24 19:53:04 OpenWrt kernel: [877339.740430] [<804a3e50>] 0x804a3e50
Mar 24 19:53:04 OpenWrt kernel: [877339.740436] [<800683a8>] 0x800683a8
Mar 24 19:53:04 OpenWrt kernel: [877339.740442] [<800888b0>] 0x800888b0
Mar 24 19:53:04 OpenWrt kernel: [877339.740448] [<80084dd4>] 0x80084dd4
Mar 24 19:53:04 OpenWrt kernel: [877339.740455] [<80080000>] 0x80080000
Mar 24 19:53:04 OpenWrt kernel: [877339.740461] [<80081e1c>] 0x80081e1c
Mar 24 19:53:04 OpenWrt kernel: [877339.740469] [<80084874>] 0x80084874
Mar 24 19:53:04 OpenWrt kernel: [877339.740475] [<8004c108>] 0x8004c108
Mar 24 19:53:04 OpenWrt kernel: [877339.740480] [<8004bfd8>] 0x8004bfd8
Mar 24 19:53:04 OpenWrt kernel: [877339.740486] [<8004bfd8>] 0x8004bfd8
Mar 24 19:53:04 OpenWrt kernel: [877339.740492] [<8004bfd8>] 0x8004bfd8
Mar 24 19:53:04 OpenWrt kernel: [877339.740498] [<80006f78>] 0x80006f78
Mar 24 19:53:04 OpenWrt kernel: [877339.740509]
Mar 24 19:53:04 OpenWrt kernel: [877349.723765] NMI backtrace for cpu 3
Mar 24 19:53:04 OpenWrt kernel: [877349.723788] rcu_sched kthread starved for 998 jiffies! g9063883 c9063882 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=2
Mar 24 19:53:04 OpenWrt kernel: [877349.723804] rcu_sched R running task
Mar 24 19:53:04 OpenWrt kernel: [877349.745078] CPU: 3 PID: 18282 Comm: kworker/3:1 Tainted: G W 4.14.221 #0
Mar 24 19:53:04 OpenWrt kernel: [877349.745089] 0 8 2 0x00100000
Mar 24 19:53:04 OpenWrt kernel: [877349.745124] Workqueue: events_power_efficient 0x8e948cac [nf_flow_table@8e948000+0x3370]
Mar 24 19:53:04 OpenWrt kernel: [877349.745130] Stack : 00000101
Mar 24 19:53:04 OpenWrt kernel: [877349.745135] Stack :
Mar 24 19:53:04 OpenWrt kernel: [877349.745139] 00000101
Mar 24 19:53:04 OpenWrt kernel: [877349.745142] ffffffff
Mar 24 19:53:04 OpenWrt kernel: [877349.745146] 80554500
Mar 24 19:53:04 OpenWrt kernel: [877349.745148] 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745153] 81232320
Mar 24 19:53:04 OpenWrt kernel: [877349.745155] 804d8568
Mar 24 19:53:04 OpenWrt kernel: [877349.745160] 8054c1e8
Mar 24 19:53:04 OpenWrt kernel: [877349.745162] 8fc0fd24
Mar 24 19:53:04 OpenWrt kernel: [877349.745167] 8006c010
Mar 24 19:53:04 OpenWrt kernel: [877349.745169] 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745173] 81222380
Mar 24 19:53:04 OpenWrt kernel: [877349.745175] 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745180] 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745182] 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745186] 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745190]
Mar 24 19:53:04 OpenWrt kernel: [877349.745193]
Mar 24 19:53:04 OpenWrt kernel: [877349.745196] 00000003 80088804
Mar 24 19:53:04 OpenWrt kernel: [877349.745201] 00000000 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745208] 053a3714 81222380
Mar 24 19:53:04 OpenWrt kernel: [877349.745213] 00000000 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745219] 00000001 80554500
Mar 24 19:53:04 OpenWrt kernel: [877349.745224] 00000000 00000001
Mar 24 19:53:04 OpenWrt kernel: [877349.745230] 00000000 80554500
Mar 24 19:53:04 OpenWrt kernel: [877349.745236] 8fc0fce0 5326163c
Mar 24 19:53:04 OpenWrt kernel: [877349.745244]
Mar 24 19:53:04 OpenWrt kernel: [877349.745249] 8054c1e8
Mar 24 19:53:04 OpenWrt kernel: [877349.745251]
Mar 24 19:53:04 OpenWrt kernel: [877349.745256] 804a4bb4
Mar 24 19:53:04 OpenWrt kernel: [877349.745260] 8fc0fd78
Mar 24 19:53:04 OpenWrt kernel: [877349.745263] 81222380 80088894
Mar 24 19:53:04 OpenWrt kernel: [877349.745268] 00000000 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745275] 8fc57e38 80550000
Mar 24 19:53:04 OpenWrt kernel: [877349.745280] 00005c28 00000038
Mar 24 19:53:04 OpenWrt kernel: [877349.745286] 80550000 00000101
Mar 24 19:53:04 OpenWrt kernel: [877349.745291] 8049e2b8 00000004
Mar 24 19:53:04 OpenWrt kernel: [877349.745299]
Mar 24 19:53:04 OpenWrt kernel: [877349.745302] 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745307] 80554500
Mar 24 19:53:04 OpenWrt kernel: [877349.745312] 80084758 80550000
Mar 24 19:53:04 OpenWrt kernel: [877349.745315]
Mar 24 19:53:04 OpenWrt kernel: [877349.745320] 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745322] 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745327] 8fc57ebc
Mar 24 19:53:04 OpenWrt kernel: [877349.745329] 80550000
Mar 24 19:53:04 OpenWrt kernel: [877349.745334] 800683a8
Mar 24 19:53:04 OpenWrt kernel: [877349.745336] 00000136
Mar 24 19:53:04 OpenWrt kernel: [877349.745341] 81216420
Mar 24 19:53:04 OpenWrt kernel: [877349.745343] 61745f77
Mar 24 19:53:04 OpenWrt kernel: [877349.745347] 00000001
Mar 24 19:53:04 OpenWrt kernel: [877349.745350] 8fc0fcc0
Mar 24 19:53:04 OpenWrt kernel: [877349.745355] 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745358]
Mar 24 19:53:04 OpenWrt kernel: [877349.745360] 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745365] 00000200
Mar 24 19:53:04 OpenWrt kernel: [877349.745367] 00000003
Mar 24 19:53:04 OpenWrt kernel: [877349.745371] 80550000
Mar 24 19:53:04 OpenWrt kernel: [877349.745376] 80554500
Mar 24 19:53:04 OpenWrt kernel: [877349.745377] 80554500
Mar 24 19:53:04 OpenWrt kernel: [877349.745384] 00000000 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745389] 8054c0cc 000000e0
Mar 24 19:53:04 OpenWrt kernel: [877349.745396] 80550000 80550000
Mar 24 19:53:04 OpenWrt kernel: [877349.745401] 80550000 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745407] 00000000
Mar 24 19:53:04 OpenWrt kernel: [877349.745411] 80550000
Mar 24 19:53:04 OpenWrt kernel: [877349.745415]
Mar 24 19:53:04 OpenWrt kernel: [877349.745417] 0000000c
Mar 24 19:53:04 OpenWrt kernel: [877349.745421] ...
Mar 24 19:53:04 OpenWrt kernel: [877349.745423] 806b000c
Mar 24 19:53:04 OpenWrt kernel: [877349.745430] Call Trace:
Mar 24 19:53:04 OpenWrt kernel: [877349.745432] ...
Mar 24 19:53:04 OpenWrt kernel: [877349.745445] [<8006c010>] 0x8006c010
Mar 24 19:53:04 OpenWrt kernel: [877349.745450] [<80088804>] 0x80088804
Mar 24 19:53:04 OpenWrt kernel: [877349.745453] Call Trace:
Mar 24 19:53:04 OpenWrt kernel: [877349.745457] [<804a4bb4>] 0x804a4bb4
Mar 24 19:53:04 OpenWrt kernel: [877349.745462] [<80088894>] 0x80088894
Mar 24 19:53:04 OpenWrt kernel: [877349.745469] [<80084758>] 0x80084758
Mar 24 19:53:04 OpenWrt kernel: [877349.745473] [<8049e2b8>] 0x8049e2b8
Mar 24 19:53:04 OpenWrt kernel: [877349.745480] [<800683a8>] 0x800683a8
Mar 24 19:53:04 OpenWrt kernel: [877349.745487] [<8000c1a0>] 0x8000c1a0
Mar 24 19:53:04 OpenWrt kernel: [877349.745492] [<80084e50>] 0x80084e50
Mar 24 19:53:04 OpenWrt kernel: [877349.745498] [<80080000>] 0x80080000
Mar 24 19:53:04 OpenWrt kernel: [877349.745503] [<8000c1a8>] 0x8000c1a8
Mar 24 19:53:04 OpenWrt kernel: [877349.745508] [<80081e1c>] 0x80081e1c
Mar 24 19:53:04 OpenWrt kernel: [877349.745513] [<804870f4>] 0x804870f4
Mar 24 19:53:04 OpenWrt kernel: [877349.745521] [<80072dd4>] 0x80072dd4
Mar 24 19:53:04 OpenWrt kernel: [877349.745525] [<80084874>] 0x80084874
Mar 24 19:53:04 OpenWrt kernel: [877349.745531] [<8004c108>] 0x8004c108
Mar 24 19:53:04 OpenWrt kernel: [877349.745535] [<8048e114>] 0x8048e114
Mar 24 19:53:04 OpenWrt kernel: [877349.745542] [<8004bfd8>] 0x8004bfd8
Mar 24 19:53:04 OpenWrt kernel: [877349.745546] [<80008f30>] 0x80008f30
Mar 24 19:53:04 OpenWrt kernel: [877349.745553] [<8004bfd8>] 0x8004bfd8
Mar 24 19:53:04 OpenWrt kernel: [877349.745557] [<80008f30>] 0x80008f30
Mar 24 19:53:04 OpenWrt kernel: [877349.745565] [<8048e200>] 0x8048e200
Mar 24 19:53:04 OpenWrt kernel: [877349.745569] [<8004bfd8>] 0x8004bfd8
Mar 24 19:53:04 OpenWrt kernel: [877349.745574] [<80006f78>] 0x80006f78
Mar 24 19:53:04 OpenWrt kernel: [877349.745578] [<804a4bb4>] 0x804a4bb4
Mar 24 19:53:04 OpenWrt kernel: [877349.745588] [<80086768>] 0x80086768
Mar 24 19:53:04 OpenWrt kernel: [877349.745591]
Mar 24 19:53:04 OpenWrt kernel: [877349.745598] [<8008667c>] 0x8008667c
Mar 24 19:53:04 OpenWrt kernel: [877349.745610] [<80085bb8>] 0x80085bb8
Mar 24 19:53:04 OpenWrt kernel: [877349.745633] [<8e950000>] 0x8e950000 [nf_nat@8e950000+0x37b0]
Mar 24 19:53:04 OpenWrt kernel: [877349.745654] [<8e950000>] 0x8e950000 [nf_nat@8e950000+0x37b0]
Mar 24 19:53:04 OpenWrt kernel: [877349.745662] [<80089328>] 0x80089328
Mar 24 19:53:04 OpenWrt kernel: [877349.745673] [<8009a378>] 0x8009a378
Mar 24 19:53:04 OpenWrt kernel: [877349.745685] [<80332aa4>] 0x80332aa4
Mar 24 19:53:04 OpenWrt kernel: [877349.745695] [<800794a8>] 0x800794a8
Mar 24 19:53:04 OpenWrt kernel: [877349.745707] [<80073760>] 0x80073760
Mar 24 19:53:04 OpenWrt kernel: [877349.745717] [<8025c008>] 0x8025c008
Mar 24 19:53:04 OpenWrt kernel: [877349.745726] [<8025beb4>] 0x8025beb4
Mar 24 19:53:04 OpenWrt kernel: [877349.745737] [<8025c074>] 0x8025c074
Mar 24 19:53:04 OpenWrt kernel: [877349.745747] [<80073760>] 0x80073760
Mar 24 19:53:04 OpenWrt kernel: [877349.745758] [<804a4f7c>] 0x804a4f7c
Mar 24 19:53:04 OpenWrt kernel: [877349.745768] [<8025b710>] 0x8025b710
Mar 24 19:53:04 OpenWrt kernel: [877349.745780] [<80007488>] 0x80007488
Mar 24 19:53:04 OpenWrt kernel: [877349.745785]
Mar 24 19:53:08 OpenWrt dnsmasq-dhcp[2322]: DHCPDISCOVER(br-lan) 192.168.1.147 14:4f:8a:00:80:16
Mar 24 19:53:08 OpenWrt dnsmasq-dhcp[2322]: DHCPOFFER(br-lan) 192.168.1.147 14:4f:8a:00:80:16
Mar 24 19:53:08 OpenWrt dnsmasq-dhcp[2322]: DHCPDISCOVER(br-lan) 192.168.1.147 14:4f:8a:00:80:16
Mar 24 19:53:08 OpenWrt dnsmasq-dhcp[2322]: DHCPOFFER(br-lan) 192.168.1.147 14:4f:8a:00:80:16
Mar 24 19:53:08 OpenWrt dnsmasq-dhcp[2322]: DHCPDISCOVER(br-lan) 72:f7:07:13:2e:68
Mar 24 19:53:08 OpenWrt dnsmasq-dhcp[2322]: DHCPOFFER(br-lan) 192.168.1.118 72:f7:07:13:2e:68
Mar 24 19:53:08 OpenWrt dnsmasq-dhcp[2322]: DHCPREQUEST(br-lan) 192.168.1.147 14:4f:8a:00:80:16
Mar 24 19:53:08 OpenWrt dnsmasq-dhcp[2322]: Ignoring domain americas.swk.pri for DHCP host name EAS-CRD0322-02
Mar 24 19:53:08 OpenWrt dnsmasq-dhcp[2322]: DHCPACK(br-lan) 192.168.1.147 14:4f:8a:00:80:16 EAS-CRD0322-02
Mar 24 19:53:08 OpenWrt dnsmasq-dhcp[2322]: DHCPREQUEST(br-lan) 192.168.1.118 72:f7:07:13:2e:68
Mar 24 19:53:08 OpenWrt dnsmasq-dhcp[2322]: DHCPACK(br-lan) 192.168.1.118 72:f7:07:13:2e:68
Mar 15 15:50:06 OpenWrt netifd: VLAN 'eth0.1' link is up
Mar 15 15:50:06 OpenWrt logread[782]: Logread connected to 192.168.1.10:514
Mar 15 15:50:06 OpenWrt netifd: Network device 'lo' link is up
Mar 15 15:50:06 OpenWrt netifd: Interface 'loopback' has link connectivity
Mar 15 15:50:06 OpenWrt netifd: VLAN 'eth0.2' link is up
Mar 15 15:50:06 OpenWrt netifd: Interface 'wan' has link connectivity
Mar 15 15:50:06 OpenWrt netifd: Interface 'wan' is setting up now

@aparcar aparcar added release/19.07 pull request/issue targeted (also) for OpenWrt 19.07 release kernel pull request/issue with Linux kernel related changes labels Feb 22, 2022
@rozhuk-im
Copy link
Contributor

Got similar issue with DIR-506L: http://netlab.dhis.org/wiki/hardware:d-link:dir-506l
Happen only on boot or on network settings change.

Workaround 1: unplug net cable untill device started
Workaround 2: remove switch from existing network configuration or build f/w without defined switch in 02_network and reset config on/after update

@Camis
Copy link

Camis commented Mar 3, 2022

Router crashes every 1-3 days or so, unresponsive. Sometimes it will reset itself after this error. This problem has been occurring since early version 19.07.

Model	Netgear R6220
Architecture	MediaTek MT7621 ver:1 eco:3
Firmware Version	OpenWrt 19.07.8 r11364-ef56c85848 / LuCI openwrt-19.07 branch git-21.279.78820-fde7889
Kernel version-  4.14.241
[76219.076181] ------------[ cut here ]------------
[76219.085402] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:320 0x8038f110
[76219.099443] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[76219.113310] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD xt_CT wireguard slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt compat ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ip6_udp_tunnel udp_tunnel leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug
[76219.255846]  usbcore nls_base usb_common
[76219.263689] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.241 #0
[76219.275803] Stack : 00000000 00000000 00000000 87f25340 00000000 00000000 00000000 00000000
[76219.292434]         00000000 00000000 00000000 00000000 00000000 00000001 87c0bd60 53261610
[76219.309066]         87c0bdf8 00000000 00000000 000071f0 00000038 804a02f8 00000008 00000000
[76219.325698]         00000000 80550000 00040609 00000000 87c0bd40 00000000 00000000 8050f6e4
[76219.342330]         8038f110 00000140 00000001 87f25340 00000000 802af880 00000004 806b0004
[76219.358963]         ...
[76219.363820] Call Trace:
[76219.363836] [<804a02f8>] 0x804a02f8
[76219.375609] [<8038f110>] 0x8038f110
[76219.382539] [<802af880>] 0x802af880
[76219.389470] [<8000c1a0>] 0x8000c1a0
[76219.396402] [<8000c1a8>] 0x8000c1a8
[76219.403329] [<80580000>] 0x80580000
[76219.410257] [<80489024>] 0x80489024
[76219.417189] [<80071eb0>] 0x80071eb0
[76219.424119] [<8002e788>] 0x8002e788
[76219.431048] [<8038f110>] 0x8038f110
[76219.437980] [<8002e810>] 0x8002e810
[76219.444914] [<877c1b04>] 0x877c1b04 [mt7603e@877c0000+0x9100]
[76219.456337] [<800553e8>] 0x800553e8
[76219.463264] [<8038f110>] 0x8038f110
[76219.470195] [<8038ef64>] 0x8038ef64
[76219.477128] [<80088a48>] 0x80088a48
[76219.484055] [<80088d04>] 0x80088d04
[76219.490982] [<800795a8>] 0x800795a8
[76219.497917] [<804a7108>] 0x804a7108
[76219.504847] [<80033154>] 0x80033154
[76219.511777] [<8025c6f0>] 0x8025c6f0
[76219.518711] [<80007488>] 0x80007488
[76219.525636] 
[76219.528692] ---[ end trace 911f450294183722 ]---
[ 9689.089757] ------------[ cut here ]------------
[ 9689.098972] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:320 0x8038f110
[ 9689.113045] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[ 9689.126902] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD xt_CT wireguard slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt compat ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ip6_udp_tunnel udp_tunnel leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug
[ 9689.269432]  usbcore nls_base usb_common
[ 9689.277262] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.241 #0
[ 9689.289376] Stack : 00000000 00000000 00000000 87f36340 00000000 00000000 00000000 00000000
[ 9689.306009]         00000000 00000000 00000000 00000000 00000000 00000001 87c0bd60 53261610
[ 9689.322642]         87c0bdf8 00000000 00000000 00006750 00000038 804a02f8 00000008 00000000
[ 9689.339274]         00000000 80550000 00043b0e 00000000 87c0bd40 00000000 00000000 8050f6e4
[ 9689.355906]         8038f110 00000140 00000001 87f36340 00000000 802af880 00000004 806b0004
[ 9689.372539]         ...
[ 9689.377396] Call Trace:
[ 9689.377411] [<804a02f8>] 0x804a02f8
[ 9689.389187] [<8038f110>] 0x8038f110
[ 9689.396118] [<802af880>] 0x802af880
[ 9689.403050] [<8000c1a0>] 0x8000c1a0
[ 9689.409983] [<8000c1a8>] 0x8000c1a8
[ 9689.416914] [<80580000>] 0x80580000
[ 9689.423843] [<80489024>] 0x80489024
[ 9689.430775] [<80071eb0>] 0x80071eb0
[ 9689.437701] [<8002e788>] 0x8002e788
[ 9689.444632] [<8038f110>] 0x8038f110
[ 9689.451593] [<8002e810>] 0x8002e810
[ 9689.458527] [<86d01b04>] 0x86d01b04 [mt7603e@86d00000+0x9100]
[ 9689.469953] [<800553e8>] 0x800553e8
[ 9689.476884] [<8038f110>] 0x8038f110
[ 9689.483817] [<8038ef64>] 0x8038ef64
[ 9689.490749] [<80088a48>] 0x80088a48
[ 9689.497680] [<80088d04>] 0x80088d04
[ 9689.504614] [<800795a8>] 0x800795a8
[ 9689.511548] [<804a7108>] 0x804a7108
[ 9689.518477] [<80033154>] 0x80033154
[ 9689.525407] [<8025c6f0>] 0x8025c6f0
[ 9689.532341] [<80007488>] 0x80007488
[ 9689.539267] 
[ 9689.542327] ---[ end trace 3e5b360a4f1120a4 ]---
[192581.462424] ------------[ cut here ]------------
[192581.471810] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:320 0x8038f110
[192581.486031] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[192581.500053] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD xt_CT wireguard slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt compat ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ip6_udp_tunnel udp_tunnel leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug
[192581.642763]  usbcore nls_base usb_common
[192581.650745] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.241 #0
[192581.663028] Stack : 00000000 00000000 00000000 87f38840 00000000 00000000 00000000 00000000
[192581.679832]         00000000 00000000 00000000 00000000 00000000 00000001 87c0bd60 53261610
[192581.696635]         87c0bdf8 00000000 00000000 00006ef0 00000038 804a02f8 00000008 00000000
[192581.713437]         00000000 80550000 0009edf9 70617773 87c0bd40 00000000 00000000 8050f6e4
[192581.730240]         8038f110 00000140 00000001 87f38840 00000008 802af880 00000004 806b0004
[192581.747044]         ...
[192581.752075] Call Trace:
[192581.752092] [<804a02f8>] 0x804a02f8
[192581.764217] [<8038f110>] 0x8038f110
[192581.771315] [<802af880>] 0x802af880
[192581.778414] [<8000c1a0>] 0x8000c1a0
[192581.785518] [<8000c1a8>] 0x8000c1a8
[192581.792621] [<80580000>] 0x80580000
[192581.799718] [<80489024>] 0x80489024
[192581.806818] [<80071eb0>] 0x80071eb0
[192581.813920] [<8002e788>] 0x8002e788
[192581.821018] [<8038f110>] 0x8038f110
[192581.828144] [<8002e810>] 0x8002e810
[192581.835252] [<86c90104>] 0x86c90104 [mt76x02_lib@86c90000+0x9760]
[192581.847539] [<800553e8>] 0x800553e8
[192581.854645] [<8038f110>] 0x8038f110
[192581.861750] [<8038ef64>] 0x8038ef64
[192581.868855] [<80088a48>] 0x80088a48
[192581.875959] [<80088d04>] 0x80088d04
[192581.883061] [<800795a8>] 0x800795a8
[192581.890161] [<804a7108>] 0x804a7108
[192581.897267] [<80033154>] 0x80033154
[192581.904370] [<8025c6f0>] 0x8025c6f0
[192581.911469] [<80007488>] 0x80007488
[192581.918565] 
[192581.921794] ---[ end trace 58f996f8e573c3bc ]---

@ynezz
Copy link
Member

ynezz commented May 23, 2022

Try to reproduce it with latest 22.03-rc2 or development snapshot and report back, we can re-open the ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flyspray kernel pull request/issue with Linux kernel related changes release/19.07 pull request/issue targeted (also) for OpenWrt 19.07 release
Projects
None yet
Development

No branches or pull requests

5 participants