- Status Unconfirmed
- Percent Complete
- Task Type Bug Report
- Category Base system
- Assigned To No-one
- Operating System All
- Severity Critical
- Priority Very Low
- Reported Version Trunk
- Due in Version Undecided
-
Due Date
Undecided
- Private
FS#1074 - NETDEV WATCHDOG: wlan2 (mwifiex_sdio): transmit queue 1 timed out - WRT3200ACM
- Device problem occurs on: WRT3200ACM
- Software versions of LEDE release, packages, etc.: LEDE Reboot SNAPSHOT r5096-a29848c / LuCI Master (git-17.260.43176-678c15c)
- Steps to reproduce: I’ve been consistently having problems with wlan2 going offline after the router has been up for a day or so, but just stumbled across this in dmesg. I’ll see if I can figure out the circumstances leading to it.
[85338.316517] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:316 dev_watchdog+0x1b4/0×234 [85338.324820] NETDEV WATCHDOG: wlan2 (mwifiex_sdio): transmit queue 1 timed out
[85338.331987] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc rfcomm nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack mwifiex_sdio mwifiex iptable_mangle iptable_filter ipt_ECN ip_tables hidp hci_uart crc_ccitt btusb btmrvl_sdio btmrvl btintel bnep bluetooth hid evdev input_core mwlwifi mac80211 cfg80211 compat xt_set ip_set_list_set ip_set_hash_netiface ip_set_hash_netport
[85338.404541] ip_set_hash_netnet ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables tun regmap_mmio ecb cmac uhci_hcd ohci_platform ohci_hcd gpio_button_hotplug
[85338.442106] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.9.54 #0
[85338.448049] Hardware name: Marvell Armada 380/385 (Device Tree)
[85338.454003] [<c0016010>] (unwind_backtrace) from [<c0012220>] (show_stack+0×10/0×14)
[85338.461782] [<c0012220>] (show_stack) from [<c020c1a0>] (dump_stack+0x7c/0x9c)
[85338.469036] [<c020c1a0>] (dump_stack) from [<c0029218>] (warn+0xbc/0xec)
[85338.475939] [<c0029218>] (warn) from [<c002927c>] (warn_slowpath_fmt+0×34/0×44)
[85338.483455] [<c002927c>] (warn_slowpath_fmt) from [<c03b6c38>] (dev_watchdog+0x1b4/0×234)
[85338.491661] [<c03b6c38>] (dev_watchdog) from [<c006ffe8>] (call_timer_fn+0×20/0×94)
[85338.499349] [<c006ffe8>] (call_timer_fn) from [<c0070220>] (run_timer_softirq+0x1c4/0×200)
[85338.507650] [<c0070220>] (run_timer_softirq) from [<c002d2b4>] (do_softirq+0xd0/0×204)
[85338.515775] [<c002d2b4>] (do_softirq) from [<c002d66c>] (irq_exit+0×94/0xb8)
[85338.523029] [<c002d66c>] (irq_exit) from [<c0061e04>] (handle_domain_irq+0×90/0xb4)
[85338.530892] [<c0061e04>] (handle_domain_irq) from [<c0009428>] (gic_handle_irq+0×50/0×94)
[85338.539277] [<c0009428>] (gic_handle_irq) from [<c0012c8c>] (irq_svc+0x6c/0×90)
[85338.546789] Exception stack(0xdf475f90 to 0xdf475fd8)
[85338.551860] 5f80: 00000001 00000000 00000000 c001b1c0
[85338.560071] 5fa0: 00000000 df474000 c0604fe4 00000002 c05ff168 00000000 df475fe8 00000001
[85338.568282] 5fc0: 00000000 df475fe0 c000f808 c000f80c 60000013 ffffffff
[85338.574925] [<c0012c8c>] (irq_svc) from [<c000f80c>] (arch_cpu_idle+0x2c/0×38)
[85338.582355] [<c000f80c>] (arch_cpu_idle) from [<c005b154>] (cpu_startup_entry+0xf0/0x19c)
[85338.590567] [<c005b154>] (cpu_startup_entry) from [<000095ac>] (0x95ac)
[85338.597214] —[ end trace 7ee504210e2aead0 ]— [85338.601865] mwifiex_sdio mmc0:0001:1: 8504333 : Tx timeout(#1), bss_type-num = 1-0
[85344.871433] mwifiex_sdio mmc0:0001:1: 8504960 : Tx timeout(#2), bss_type-num = 1-0
[85350.631045] mwifiex_sdio mmc0:0001:1: 8505536 : Tx timeout(#3), bss_type-num = 1-0
[85356.400657] mwifiex_sdio mmc0:0001:1: 8506113 : Tx timeout(#4), bss_type-num = 1-0
[85366.309996] mwifiex_sdio mmc0:0001:1: 8507104 : Tx timeout(#5), bss_type-num = 1-0
[85376.309327] mwifiex_sdio mmc0:0001:1: 8508104 : Tx timeout(#6), bss_type-num = 1-0
[85386.318652] mwifiex_sdio mmc0:0001:1: 8509105 : Tx timeout(#7), bss_type-num = 1-0
[85386.326257] mwifiex_sdio mmc0:0001:1: tx_timeout_cnt exceeds threshold. Triggering card reset!
[85386.356950] mwifiex_sdio mmc0:0001:1: info: shutdown mwifiex...
[85386.362967] mwifiex_sdio mmc0:0001:1: PREP_CMD: card is removed
[85386.368924] mwifiex_sdio mmc0:0001:1: Failed to delete mgmt IEs!
[85386.374978] mwifiex_sdio mmc0:0001:1: PREP_CMD: card is removed
[85386.380928] mwifiex_sdio mmc0:0001:1: Failed to stop the BSS
[85386.387009] br-lan: port 4(wlan2) entered disabled state
[85386.394495] device wlan2 left promiscuous mode
[85386.398975] br-lan: port 4(wlan2) entered disabled state
[85386.404593] mwifiex_sdio mmc0:0001:1: PREP_CMD: card is removed
[85386.410562] mwifiex_sdio mmc0:0001:1: PREP_CMD: card is removed
[85386.416507] mwifiex_sdio mmc0:0001:1: PREP_CMD: card is removed
[85386.603147] mwifiex_sdio mmc0:0001:1: WLAN FW already running! Skip FW dnld
[85386.610161] mwifiex_sdio mmc0:0001:1: WLAN FW is active
[85396.707953] mwifiex_sdio mmc0:0001:1: mwifiex_cmd_timeout_func: Timeout cmd id = 0xa9, act = 0×0 [85396.716787] mwifiex_sdio mmc0:0001:1: num_data_h2c_failure = 0
[85396.722652] mwifiex_sdio mmc0:0001:1: num_cmd_h2c_failure = 0
[85396.728423] mwifiex_sdio mmc0:0001:1: is_cmd_timedout = 1
[85396.733842] mwifiex_sdio mmc0:0001:1: num_tx_timeout = 0
[85396.739178] mwifiex_sdio mmc0:0001:1: last_cmd_index = 3
[85396.744512] mwifiex_sdio mmc0:0001:1: last_cmd_id: b3 00 b3 00 b2 00 a9 00 b3 00
[85396.751940] mwifiex_sdio mmc0:0001:1: last_cmd_act: 00 00 00 00 00 00 00 00 00 00
[85396.759457] mwifiex_sdio mmc0:0001:1: last_cmd_resp_index = 2
[85396.765225] mwifiex_sdio mmc0:0001:1: last_cmd_resp_id: b3 80 b3 80 b2 80 b3 80 b3 80
[85396.773093] mwifiex_sdio mmc0:0001:1: last_event_index = 2
[85396.778603] mwifiex_sdio mmc0:0001:1: last_event: 37 00 55 00 43 00 55 00 55 00
[85396.785944] mwifiex_sdio mmc0:0001:1: data_sent=1 cmd_sent=1
[85396.791630] mwifiex_sdio mmc0:0001:1: ps_mode=0 ps_state=0
[85396.797372] mwifiex_sdio mmc0:0001:1: info: _mwifiex_fw_dpc: unregister device
Second failure today:
[62698.039681] ————[ cut here ]———— [62698.044334] WARNING: CPU: 1 PID: 14 at net/sched/sch_generic.c:316 dev_watchdog+0x1b4/0x234
[62698.052727] NETDEV WATCHDOG: wlan2 (mwifiex_sdio): transmit queue 1 timed out
[62698.059893] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_DSCP xt_CLASSIFY slhc rfcomm nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack mwifiex_sdio mwifiex iptable_mangle iptable_filter ipt_ECN ip_tables hidp hci_uart crc_ccitt btusb btmrvl_sdio btmrvl btintel bnep bluetooth hid evdev input_core mwlwifi mac80211 cfg80211 compat xt_set ip_set_list_set ip_set_hash_netiface ip_set_hash_netport
[62698.132465] ip_set_hash_netnet ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables tun regmap_mmio ecb cmac uhci_hcd ohci_platform ohci_hcd gpio_button_hotplug
[62698.170034] CPU: 1 PID: 14 Comm: ksoftirqd/1 Not tainted 4.9.54 #0
[62698.176238] Hardware name: Marvell Armada 380/385 (Device Tree)
[62698.182192] [<c0016010>] (unwind_backtrace) from [<c0012220>] (show_stack+0x10/0x14)
[62698.189973] [<c0012220>] (show_stack) from [<c020c1a0>] (dump_stack+0x7c/0x9c)
[62698.197227] [<c020c1a0>] (dump_stack) from [<c0029218>] (warn+0xbc/0xec)
[62698.204130] [<c0029218>] (warn) from [<c002927c>] (warn_slowpath_fmt+0x34/0x44)
[62698.211647] [<c002927c>] (warn_slowpath_fmt) from [<c03b6c38>] (dev_watchdog+0x1b4/0x234)
[62698.219861] [<c03b6c38>] (dev_watchdog) from [<c006ffe8>] (call_timer_fn+0x20/0x94)
[62698.227549] [<c006ffe8>] (call_timer_fn) from [<c0070220>] (run_timer_softirq+0x1c4/0x200)
[62698.235848] [<c0070220>] (run_timer_softirq) from [<c002d2b4>] (do_softirq+0xd0/0x204)
[62698.243973] [<c002d2b4>] (do_softirq) from [<c002d414>] (run_ksoftirqd+0x2c/0x50)
[62698.251663] [<c002d414>] (run_ksoftirqd) from [<c0045140>] (smpboot_thread_fn+0x16c/0x184)
[62698.259963] [<c0045140>] (smpboot_thread_fn) from [<c00422b4>] (kthread+0xd8/0xec)
[62698.267566] [<c00422b4>] (kthread) from [<c000edf8>] (ret_from_fork+0x14/0x3c)
[62698.274822] —[ end trace 13b6a7d4f3c0a950 ]— [62698.279461] mwifiex_sdio mmc0:0001:1: 6240127 : Tx timeout(#1), bss_type-num = 1-0
[62704.039294] mwifiex_sdio mmc0:0001:1: 6240704 : Tx timeout(#2), bss_type-num = 1-0
[62714.038674] mwifiex_sdio mmc0:0001:1: 6241704 : Tx timeout(#3), bss_type-num = 1-0
[62724.038054] mwifiex_sdio mmc0:0001:1: 6242704 : Tx timeout(#4), bss_type-num = 1-0
[62734.047454] mwifiex_sdio mmc0:0001:1: 6243705 : Tx timeout(#5), bss_type-num = 1-0
[62744.036836] mwifiex_sdio mmc0:0001:1: 6244704 : Tx timeout(#6), bss_type-num = 1-0
[62754.036216] mwifiex_sdio mmc0:0001:1: 6245704 : Tx timeout(#7), bss_type-num = 1-0
[62754.043823] mwifiex_sdio mmc0:0001:1: tx_timeout_cnt exceeds threshold. Triggering card reset!
[62754.228690] mwifiex_sdio mmc0:0001:1: info: shutdown mwifiex...
[62754.234775] mwifiex_sdio mmc0:0001:1: PREP_CMD: card is removed
[62754.240738] mwifiex_sdio mmc0:0001:1: Failed to delete mgmt IEs!
[62754.246776] mwifiex_sdio mmc0:0001:1: PREP_CMD: card is removed
[62754.252747] mwifiex_sdio mmc0:0001:1: Failed to stop the BSS
[62754.258851] br-lan: port 4(wlan2) entered disabled state
[62754.266263] device wlan2 left promiscuous mode
[62754.270765] br-lan: port 4(wlan2) entered disabled state
[62754.276246] mwifiex_sdio mmc0:0001:1: PREP_CMD: card is removed
[62754.282222] mwifiex_sdio mmc0:0001:1: PREP_CMD: card is removed
[62754.288181] mwifiex_sdio mmc0:0001:1: PREP_CMD: card is removed
[62754.500727] mwifiex_sdio mmc0:0001:1: WLAN FW already running! Skip FW dnld
[62754.507737] mwifiex_sdio mmc0:0001:1: WLAN FW is active
[62764.835522] mwifiex_sdio mmc0:0001:1: mwifiex_cmd_timeout_func: Timeout cmd id = 0xa9, act = 0x0
[62764.844350] mwifiex_sdio mmc0:0001:1: num_data_h2c_failure = 0
[62764.850213] mwifiex_sdio mmc0:0001:1: num_cmd_h2c_failure = 0
[62764.855988] mwifiex_sdio mmc0:0001:1: is_cmd_timedout = 1
[62764.861408] mwifiex_sdio mmc0:0001:1: num_tx_timeout = 0
[62764.866745] mwifiex_sdio mmc0:0001:1: last_cmd_index = 2
[62764.872082] mwifiex_sdio mmc0:0001:1: last_cmd_id: 1e 00 b2 00 a9 00 b3 00 b3 00
[62764.879513] mwifiex_sdio mmc0:0001:1: last_cmd_act: 00 00 00 00 00 00 00 00 00 00
[62764.887030] mwifiex_sdio mmc0:0001:1: last_cmd_resp_index = 1
[62764.892799] mwifiex_sdio mmc0:0001:1: last_cmd_resp_id: 1e 80 b2 80 b3 80 b3 80 b3 80
[62764.900664] mwifiex_sdio mmc0:0001:1: last_event_index = 0
[62764.906176] mwifiex_sdio mmc0:0001:1: last_event: 43 00 37 00 55 00 55 00 55 00
[62764.913515] mwifiex_sdio mmc0:0001:1: data_sent=1 cmd_sent=1
[62764.919202] mwifiex_sdio mmc0:0001:1: ps_mode=0 ps_state=0
[62764.926512] mwifiex_sdio mmc0:0001:1: info: _mwifiex_fw_dpc: unregister device
Do you know what version of mwifiex you are using? e.g. in dmesg: "[ 8.289013] «Marvell Mac80211 Wireless PCIE Network Driver version 10.3.4.0-20170606»"
Interesting, I am experiencing exaclty the same problem, on the same device (WRT3200ACM), on the same interface (wlan2). So I can indeed confirm this issue.
The interface wlan2 (Marvell 88W8887) causes trouble ever since. On the other hand, wlan0 and wlan1 (Marvell 88W8964) seem to work just fine. I tried with both 2G and 5G, it does not make any difference.
I am facing this problem since LEDE 17.01.0 - currently I have installed LEDE 18.06.0
I just noticed that LEDE 18.06.1 is out, but however I don't expect this problem to be solved. Well, I will give it a try...
Installed wifi modules:
Output from iwinfo:
wlan0 ESSID: "LEDE-0" Access Point: 60:38:E0:BD:7B:BA Mode: Master Channel: 44 (5.220 GHz) Tx-Power: 20 dBm Link Quality: unknown/70 Signal: unknown Noise: -97 dBm Bit Rate: unknown Encryption: WPA2 PSK (CCMP) Type: nl80211 HW Mode(s): 802.11nac Hardware: 11AB:2B40 11AB:0000 [Marvell 88W8964] TX power offset: none Frequency offset: none Supports VAPs: yes PHY name: phy0 wlan1 ESSID: "LEDE-1" Access Point: 60:38:E0:BD:7B:B9 Mode: Master Channel: 11 (2.462 GHz) Tx-Power: 20 dBm Link Quality: unknown/70 Signal: unknown Noise: -49 dBm Bit Rate: unknown Encryption: WPA2 PSK (CCMP) Type: nl80211 HW Mode(s): 802.11bgn Hardware: 11AB:2B40 11AB:0000 [Marvell 88W8964] TX power offset: none Frequency offset: none Supports VAPs: yes PHY name: phy1 wlan2 ESSID: "LEDE-2" Access Point: 60:38:E0:BD:7B:BB Mode: Master Channel: 1 (2.412 GHz) Tx-Power: 0 dBm Link Quality: unknown/70 Signal: unknown Noise: -90 dBm Bit Rate: unknown Encryption: WPA2 PSK (CCMP) Type: nl80211 HW Mode(s): 802.11bgnac Hardware: 02DF:9135 0000:0000 [Marvell 88W8887] TX power offset: none Frequency offset: none Supports VAPs: yes PHY name: phy2ifconfig shows many TX packet errors. The other interfaces don't have such errors:
wlan2 Link encap:Ethernet HWaddr 60:38:E0:BD:7B:BB inet6 addr: fe80::6238:e0ff:febd:7bbb/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:11707 errors:0 dropped:0 overruns:0 frame:0 TX packets:17974 errors:5858 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1937866 (1.8 MiB) TX bytes:6569991 (6.2 MiB)After this bug triggers, wlan2 stops working, the interface disappears.
Here the output from syslog (logread)
Any help would much be appreciated.
Hello i am experiencing exactly the same error on the same device and also on wlan 2 network adapter. I am using it with the 2G frequency and using wlan1 instead is not an option because i have an ESP8266 based wifi plug which is not connecting to wlan1 for some unknown reason. Does anybody have some help?
+1 to franco's detailed report.
root@francis:~# uname -a
Linux francis 4.14.54 #0 SMP Sat Jul 14 11:10:08 2018 armv7l GNU/Linux
root@francis:~# opkg list-installed|grep wifi
kmod-mwifiex-sdio - 4.14.54+2017-11-01-9
kmod-mwlwifi - 4.14.54+2018-03-30-fcaea79a-1
mwifiex-sdio-firmware - 2017-09-06-a61ac5cf-1
mwlwifi-firmware-88w8964 - 2018-03-30-fcaea79a-1
wrtac3200acm using 5ghz 20Mhz on wlan2 radio. drops off rather quickly. logread is same as franco's above.
I am also experiencing this problem on the WRTAC3200ACM with OpenWrt 18.06.1 r7258-5eb055306f.
The wlan2 interface drops down and can't be brought back without rebooting. From the kernel log I would guess that there's some kind of bug in the device driver that gets triggered on this radio.
I also sometimes experience similar issues on wlan1 (which I have set up as my 2.4Ghz radio) which tends to drop off at random sometimes. Not sure if it is really the same issue but it does seem fishy since the drivers should be very closely related.
Hello,
i can confirm that i have the same bug and it very frustate.
I have upgraded to OpenWrt 18.06.2, hoping that the problem will be solved with a new kmod mwifiex_sdio, but i'm experiencing the same problem.
So my two ESP devices lost every day connection because the driver mwifiex_sdio crashed, impossible to unload the module mwifiex_sdio so the only solution for the moment is to reboot my wrtac3200acm.
Could someone knows if there is a workaround, i mean detect when the wlan2 is removed and reinstate it ?
I try restart the wifi from luci but unsucessfull.
I try wifi stop & wifi start, impossible.
Before crashing i see some Tx errors increasing with ifconfig on wlan2.
I had the exact same issue with 18.06.2 on my 3200ACM.
The solution for me was build&use the git master for the firmware image and packages.