Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2053 - Regular crashes of ath10k-ct driver on ZyXEL NBG6817 #6924

Open
openwrt-bot opened this issue Jan 10, 2019 · 8 comments
Open

FS#2053 - Regular crashes of ath10k-ct driver on ZyXEL NBG6817 #6924

openwrt-bot opened this issue Jan 10, 2019 · 8 comments
Labels
flyspray kernel pull request/issue with Linux kernel related changes

Comments

@openwrt-bot
Copy link

msoltyspl:

Model ZyXEL NBG6817
Architecture ARMv7 Processor rev 0 (v7l)
Firmware Version OpenWrt SNAPSHOT r9010-adc8b374e3 / LuCI Master (git-19.007.66460-4edac36)
Kernel Version 4.14.91
Local Time Thu Jan 10 10:57:42 2019
Uptime 16h 32m 57s
Load Average 0.40, 0.34, 0.18

Sadly there isn't any simple way to reproduce, but they seem to be happening somewhat regularly. Since I put the device to use, 1 yesterday, 2 today.

Jan 10 09:26:10 hs14 kernel: [54084.922093] ath10k_pci 0001:01:00.0: firmware crashed! (guid ec43bc35-1369-4765-b62f-6ccef1fb3d02) Jan 10 09:26:10 hs14 kernel: [54084.922155] ath10k_pci 0001:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe Jan 10 09:26:10 hs14 kernel: [54084.929958] ath10k_pci 0001:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0 Jan 10 09:26:10 hs14 kernel: [54084.942242] ath10k_pci 0001:01:00.0: firmware ver 10.4b-ct-9984-fW-012-81e1edd54 api 5 features mfp,peer-flow-ctrl,txstatus-noack,wmi-10.x-CT,ratemask-CT,reg dump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,tx-rc-CT,cust-stats-CT,txrate2-CT crc32 0391c067 Jan 10 09:26:10 hs14 kernel: [54084.951289] ath10k_pci 0001:01:00.0: board_file api 2 bmi_id 0:2 crc32 cf58c3bc Jan 10 09:26:10 hs14 kernel: [54084.972926] ath10k_pci 0001:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 32 raw 0 hwcrypto 1 Jan 10 09:26:10 hs14 kernel: [54084.982076] ath10k_pci 0001:01:00.0: firmware register dump: Jan 10 09:26:10 hs14 kernel: [54084.990109] ath10k_pci 0001:01:00.0: [00]: 0x0000000A 0x000015B3 0x009A582B 0x00975B31 Jan 10 09:26:10 hs14 kernel: [54084.996013] ath10k_pci 0001:01:00.0: [04]: 0x009A582B 0x00060B30 0x00000000 0x00000000 Jan 10 09:26:10 hs14 kernel: [54085.003740] ath10k_pci 0001:01:00.0: [08]: 0x00000000 0x004602A4 0x00406764 0x00000000 Jan 10 09:26:10 hs14 kernel: [54085.011640] ath10k_pci 0001:01:00.0: [12]: 0x00000009 0x00000000 0x00973ABC 0x00973AD2 Jan 10 09:26:10 hs14 kernel: [54085.019537] ath10k_pci 0001:01:00.0: [16]: 0x00973AB0 0x0096398A 0x009606CA 0x00000000 Jan 10 09:26:10 hs14 kernel: [54085.027437] ath10k_pci 0001:01:00.0: [20]: 0x409A582B 0x0040660C 0x00000005 0x000000FF Jan 10 09:26:10 hs14 kernel: [54085.035336] ath10k_pci 0001:01:00.0: [24]: 0x809A698A 0x0040666C 0x00000005 0xC09A582B Jan 10 09:26:10 hs14 kernel: [54085.043235] ath10k_pci 0001:01:00.0: [28]: 0x809A7EF9 0x0040669C 0x0045F0D0 0x0045FCA8 Jan 10 09:26:10 hs14 kernel: [54085.051135] ath10k_pci 0001:01:00.0: [32]: 0x809A8D32 0x0040672C 0x0045FCE4 0x00406850 Jan 10 09:26:10 hs14 kernel: [54085.059035] ath10k_pci 0001:01:00.0: [36]: 0x809A8D62 0x0040684C 0x0042D864 0x0042EF70 Jan 10 09:26:10 hs14 kernel: [54085.066934] ath10k_pci 0001:01:00.0: [40]: 0x80985E4B 0x0040689C 0x00000000 0x0045A9F4 Jan 10 09:26:10 hs14 kernel: [54085.074833] ath10k_pci 0001:01:00.0: [44]: 0x80994A73 0x004068BC 0x0042ED20 0x0045A9F4 Jan 10 09:26:10 hs14 kernel: [54085.082732] ath10k_pci 0001:01:00.0: [48]: 0x8098FC30 0x004068DC 0x0042ED20 0x00000000 Jan 10 09:26:10 hs14 kernel: [54085.090631] ath10k_pci 0001:01:00.0: [52]: 0x80963AD3 0x00406A7C 0x0042ED20 0x0098FC28 Jan 10 09:26:10 hs14 kernel: [54085.098530] ath10k_pci 0001:01:00.0: [56]: 0x80960E80 0x00406A9C 0x0000001F 0x00400000 Jan 10 09:26:10 hs14 kernel: [54085.106429] ath10k_pci 0001:01:00.0: Copy Engine register dump: Jan 10 09:26:10 hs14 kernel: [54085.114333] ath10k_pci 0001:01:00.0: [00]: 0x0004a000 13 13 3 3 Jan 10 09:26:10 hs14 kernel: [54085.120150] ath10k_pci 0001:01:00.0: [01]: 0x0004a400 10 10 432 433 Jan 10 09:26:10 hs14 kernel: [54085.126747] ath10k_pci 0001:01:00.0: [02]: 0x0004a800 46 46 109 110 Jan 10 09:26:10 hs14 kernel: [54085.133170] ath10k_pci 0001:01:00.0: [03]: 0x0004ac00 21 21 23 21 Jan 10 09:26:10 hs14 kernel: [54085.139594] ath10k_pci 0001:01:00.0: [04]: 0x0004b000 2043 1937 30 30 Jan 10 09:26:10 hs14 kernel: [54085.146018] ath10k_pci 0001:01:00.0: [05]: 0x0004b400 1 1 128 129 Jan 10 09:26:10 hs14 kernel: [54085.152441] ath10k_pci 0001:01:00.0: [06]: 0x0004b800 10 10 10 10 Jan 10 09:26:10 hs14 kernel: [54085.159039] ath10k_pci 0001:01:00.0: [07]: 0x0004bc00 1 1 1 1 Jan 10 09:26:10 hs14 kernel: [54085.165462] ath10k_pci 0001:01:00.0: [08]: 0x0004c000 0 0 127 0 Jan 10 09:26:10 hs14 kernel: [54085.171886] ath10k_pci 0001:01:00.0: [09]: 0x0004c400 1 1 1 1 Jan 10 09:26:10 hs14 kernel: [54085.178310] ath10k_pci 0001:01:00.0: [10]: 0x0004c800 0 0 0 0 Jan 10 09:26:10 hs14 kernel: [54085.184733] ath10k_pci 0001:01:00.0: [11]: 0x0004cc00 0 0 0 0 Jan 10 09:26:10 hs14 kernel: [54085.193175] ath10k_pci 0001:01:00.0: debug log header, dbuf: 0x423818 dropped: 0 Jan 10 09:26:10 hs14 kernel: [54085.198590] ath10k_pci 0001:01:00.0: [0] next: 0x423800 buf: 0x419610 sz: 1500 len: 252 count: 9 free: 0 Jan 10 09:26:10 hs14 kernel: [54085.206060] ath10k_pci 0001:01:00.0: ath10k_pci ATH10K_DBG_BUFFER: Jan 10 09:26:10 hs14 kernel: [54085.214587] ath10k: [0000]: 034C7883 14004C01 71104441 01050F98 00019003 00000000 0000FFFF 034C7884 Jan 10 09:26:10 hs14 kernel: [54085.220581] ath10k: [0008]: 14004C01 71108880 00000000 00D80000 00000000 00000FF0 034C7884 14004C01 Jan 10 09:26:10 hs14 kernel: [54085.229521] ath10k: [0016]: 71108880 00010000 00D80000 00000000 00000000 034C7884 14004C01 71108880 Jan 10 09:26:10 hs14 kernel: [54085.238548] ath10k: [0024]: 00020000 00D80000 00000000 00000000 034C7884 14004C01 71108880 00030000 Jan 10 09:26:10 hs14 kernel: [54085.247576] ath10k: [0032]: 00D80000 00000000 00000000 034C7884 14004C01 71108880 00040000 00D80000 Jan 10 09:26:10 hs14 kernel: [54085.256604] ath10k: [0040]: 00000000 00000000 034C7884 14004C01 71108880 00050000 00D80000 00000000 Jan 10 09:26:10 hs14 kernel: [54085.265642] ath10k: [0048]: 00000000 034C7884 14004C01 71108880 00060000 00D80000 00000000 00000000 Jan 10 09:26:10 hs14 kernel: [54085.274659] ath10k: [0056]: 034C7885 17FC0001 009A582B 000015B3 000015B3 004064FC 91104569 Jan 10 09:26:10 hs14 kernel: [54085.283686] ath10k_pci 0001:01:00.0: ATH10K_END Jan 10 09:26:10 hs14 kernel: [54085.293030] ath10k_pci 0001:01:00.0: [1] next: 0x423818 buf: 0x419020 sz: 1500 len: 0 count: 0 free: 0 Jan 10 09:26:10 hs14 kernel: [54085.336026] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon Jan 10 09:26:10 hs14 kernel: [54085.361017] ath10k_pci 0001:01:00.0: removing peer, cleanup-all, deleting: peer da1bf400 vdev: 0 addr: 48:45:20:c2:27:84 Jan 10 09:26:10 hs14 kernel: [54085.361055] ath10k_pci 0001:01:00.0: removing peer, cleanup-all, deleting: peer da1be400 vdev: 0 addr: 64:a2:f9:31:0f:98 Jan 10 09:26:10 hs14 kernel: [54085.371034] ath10k_pci 0001:01:00.0: removing peer, cleanup-all, deleting: peer da1be200 vdev: 0 addr: 84:c7:ea:8f:41:c7 Jan 10 09:26:10 hs14 kernel: [54085.381962] ath10k_pci 0001:01:00.0: removing peer, cleanup-all, deleting: peer da1bf000 vdev: 0 addr: 6c:c7:ec:46:de:44 Jan 10 09:26:10 hs14 kernel: [54085.392896] ath10k_pci 0001:01:00.0: removing peer, cleanup-all, deleting: peer da1bfc00 vdev: 0 addr: 88:d7:f6:9f:51:10 Jan 10 09:26:10 hs14 kernel: [54085.403829] ath10k_pci 0001:01:00.0: removing peer, cleanup-all, deleting: peer dbd42e00 vdev: 1 addr: 5e:e2:8c:f2:fe:80 Jan 10 09:26:10 hs14 kernel: [54085.414796] ath10k_pci 0001:01:00.0: removing peer, cleanup-all, deleting: peer dbbdc200 vdev: 0 addr: 5c:e2:8c:f2:fe:80 Jan 10 09:26:10 hs14 kernel: [54085.516577] ieee80211 phy1: Hardware restart was requested Jan 10 09:26:11 hs14 kernel: [54085.597874] ath10k_pci 0001:01:00.0: Invalid state: 3 in ath10k_htt_tx_32, warning will not be repeated. Jan 10 09:26:11 hs14 kernel: [54085.597905] ------------[ cut here ]------------ Jan 10 09:26:11 hs14 kernel: [54085.606562] WARNING: CPU: 1 PID: 0 at /var/lib/buildbot/slaves/tictex-02/ipq806x_generic/build/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/linux-ipq806x/ath10k-ct-2018-12-20-118e16da/ath10k-4.19/htt_tx.c:1250 ath10k_htt_tx_32+0xf0/0x9e0 [ath10k_core] Jan 10 09:26:11 hs14 kernel: [54085.611507] Modules linked in: pppoe ppp_async ath10k_pci ath10k_core ath pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD xt_CT slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt compat act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_tbf sch_htb sch_hfsc sch_ingress ledtrig_usbport ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables leds_gpio Jan 10 09:26:11 hs14 kernel: [54085.682720] xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_of_simple ohci_platform ohci_hcd phy_qcom_dwc3 ahci ehci_platform sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd gpio_button_hotplug ext4 jbd2 mbcache crc32c_generic Jan 10 09:26:11 hs14 kernel: [54085.704952] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.91 #0 Jan 10 09:26:11 hs14 kernel: [54085.726639] Hardware name: Generic DT based system Jan 10 09:26:11 hs14 kernel: [54085.732654] [] (unwind_backtrace) from [] (show_stack+0x14/0x20) Jan 10 09:26:11 hs14 kernel: [54085.737240] [] (show_stack) from [] (dump_stack+0x88/0x9c) Jan 10 09:26:11 hs14 kernel: [54085.745143] [] (dump_stack) from [] (__warn+0xf0/0x11c) Jan 10 09:26:11 hs14 kernel: [54085.752167] [] (__warn) from [] (warn_slowpath_null+0x20/0x28) Jan 10 09:26:11 hs14 kernel: [54085.759059] [] (warn_slowpath_null) from [] (ath10k_htt_tx_32+0xf0/0x9e0 [ath10k_core]) Jan 10 09:26:11 hs14 kernel: [54085.766772] [] (ath10k_htt_tx_32 [ath10k_core]) from [] (ath10k_mac_op_set_bitrate_mask+0xc40/0xdac [ath10k_core]) Jan 10 09:26:11 hs14 kernel: [54085.776369] [] (ath10k_mac_op_set_bitrate_mask [ath10k_core]) from [] (ath10k_mac_tx_push_txq+0x234/0x290 [ath10k_core]) Jan 10 09:26:11 hs14 kernel: [54085.788524] [] (ath10k_mac_tx_push_txq [ath10k_core]) from [] (ath10k_mac_op_wake_tx_queue+0x88/0x12c [ath10k_core]) Jan 10 09:26:11 hs14 kernel: [54085.801339] [] (ath10k_mac_op_wake_tx_queue [ath10k_core]) from [] (ieee80211_unreserve_tid+0x658/0x718 [mac80211]) Jan 10 09:26:11 hs14 kernel: [54085.813569] [] (ieee80211_unreserve_tid [mac80211]) from [] (__ieee80211_subif_start_xmit+0x728/0x978 [mac80211]) Jan 10 09:26:11 hs14 kernel: [54085.825444] [] (__ieee80211_subif_start_xmit [mac80211]) from [] (ieee80211_subif_start_xmit+0x2f0/0x310 [mac80211]) Jan 10 09:26:11 hs14 kernel: [54085.837553] [] (ieee80211_subif_start_xmit [mac80211]) from [] (dev_hard_start_xmit+0xc8/0x154) Jan 10 09:26:11 hs14 kernel: [54085.849828] [] (dev_hard_start_xmit) from [] (__dev_queue_xmit+0x630/0x7b0) Jan 10 09:26:11 hs14 kernel: [54085.859983] [] (__dev_queue_xmit) from [] (br_dev_queue_push_xmit+0x144/0x168) Jan 10 09:26:11 hs14 kernel: [54085.868662] [] (br_dev_queue_push_xmit) from [] (br_forward_finish+0x38/0xa0) Jan 10 09:26:11 hs14 kernel: [54085.877688] [] (br_forward_finish) from [] (__br_forward+0xe8/0x144) Jan 10 09:26:11 hs14 kernel: [54085.886629] [] (__br_forward) from [] (br_handle_frame_finish+0x4fc/0x56c) Jan 10 09:26:11 hs14 kernel: [54085.894788] [] (br_handle_frame_finish) from [] (br_handle_frame+0x328/0x3c0) Jan 10 09:26:11 hs14 kernel: [54085.903218] [] (br_handle_frame) from [] (__netif_receive_skb_core+0x71c/0xbdc) Jan 10 09:26:11 hs14 kernel: [54085.912154] [] (__netif_receive_skb_core) from [] (process_backlog+0xb0/0x164) Jan 10 09:26:11 hs14 kernel: [54085.921008] [] (process_backlog) from [] (net_rx_action+0x144/0x31c) Jan 10 09:26:11 hs14 kernel: [54085.930032] [] (net_rx_action) from [] (__do_softirq+0xf0/0x264) Jan 10 09:26:11 hs14 kernel: [54085.938285] [] (__do_softirq) from [] (irq_exit+0xdc/0x148) Jan 10 09:26:11 hs14 kernel: [54085.946006] [] (irq_exit) from [] (__handle_domain_irq+0xa8/0xc8) Jan 10 09:26:11 hs14 kernel: [54085.953036] [] (__handle_domain_irq) from [] (gic_handle_irq+0x6c/0xb8) Jan 10 09:26:11 hs14 kernel: [54085.961022] [] (gic_handle_irq) from [] (__irq_svc+0x6c/0x90) Jan 10 09:26:11 hs14 kernel: [54085.969176] Exception stack(0xdd45ff80 to 0xdd45ffc8) Jan 10 09:26:11 hs14 kernel: [54085.976838] ff80: 00000001 00000000 00000000 c0315300 ffffe000 c0b03c74 c0b03c28 00000000 Jan 10 09:26:11 hs14 kernel: [54085.981876] ffa0: 00000000 512f04d0 00000000 00000000 dd45ffc8 dd45ffd0 c030884c c0308850 Jan 10 09:26:11 hs14 kernel: [54085.990018] ffc0: 60000013 ffffffff Jan 10 09:26:11 hs14 kernel: [54085.998171] [] (__irq_svc) from [] (arch_cpu_idle+0x38/0x44) Jan 10 09:26:11 hs14 kernel: [54086.001477] [] (arch_cpu_idle) from [] (do_idle+0xe8/0x1bc) Jan 10 09:26:11 hs14 kernel: [54086.009111] [] (do_idle) from [] (cpu_startup_entry+0x1c/0x20) Jan 10 09:26:11 hs14 kernel: [54086.016142] [] (cpu_startup_entry) from [<423017cc>] (0x423017cc) Jan 10 09:26:11 hs14 kernel: [54086.023843] ---[ end trace 452feb3a499a0195 ]--- Jan 10 09:26:11 hs14 kernel: [54086.030636] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:11 hs14 kernel: [54086.035568] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:11 hs14 kernel: [54086.042594] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:11 hs14 kernel: [54086.048636] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:11 hs14 kernel: [54086.054310] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:11 hs14 kernel: [54086.061524] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:11 hs14 kernel: [54086.081184] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon Jan 10 09:26:11 hs14 kernel: [54086.081230] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon Jan 10 09:26:11 hs14 kernel: [54086.087310] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon Jan 10 09:26:11 hs14 kernel: [54086.094688] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon Jan 10 09:26:11 hs14 kernel: [54086.101966] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon Jan 10 09:26:11 hs14 kernel: [54086.398043] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:11 hs14 kernel: [54086.398093] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:11 hs14 kernel: [54086.404163] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:11 hs14 kernel: [54086.410265] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:11 hs14 kernel: [54086.415885] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:11 hs14 kernel: [54086.423100] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:12 hs14 kernel: [54086.859493] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:12 hs14 kernel: [54086.859541] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:12 hs14 kernel: [54086.865611] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:12 hs14 kernel: [54087.668435] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:12 hs14 kernel: [54087.668484] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:12 hs14 kernel: [54087.674553] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:13 hs14 kernel: [54087.998516] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:13 hs14 kernel: [54087.998564] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:13 hs14 kernel: [54088.004648] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:13 hs14 kernel: [54088.010739] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:13 hs14 kernel: [54088.016359] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:13 hs14 kernel: [54088.023559] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:13 hs14 kernel: [54088.315168] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:13 hs14 kernel: [54088.315215] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:13 hs14 kernel: [54088.321286] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.199325] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:16 hs14 kernel: [54091.199378] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.205447] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.317048] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:16 hs14 kernel: [54091.317096] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.323165] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.399734] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:16 hs14 kernel: [54091.399767] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.405938] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.414260] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:16 hs14 kernel: [54091.417567] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.424841] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.442523] ath10k_pci 0001:01:00.0: 10.4 wmi init: vdevs: 16 peers: 48 tid: 96 Jan 10 09:26:16 hs14 kernel: [54091.442587] ath10k_pci 0001:01:00.0: msdu-desc: 2500 skid: 32 Jan 10 09:26:16 hs14 kernel: [54091.474435] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:16 hs14 kernel: [54091.474465] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.480544] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.492287] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:16 hs14 kernel: [54091.492316] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.499467] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.524925] ath10k_pci 0001:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186 msdu-desc: 2500 sw-crypt: 0 ct-sta: 0' Jan 10 09:26:16 hs14 kernel: [54091.525752] ath10k_pci 0001:01:00.0: wmi print 'free: 87020 iram: 26788 sram: 18240' Jan 10 09:26:16 hs14 kernel: [54091.578225] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:16 hs14 kernel: [54091.578254] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:16 hs14 kernel: [54091.584413] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:17 hs14 kernel: [54091.941028] ath10k_pci 0001:01:00.0: Firmware lacks feature flag indicating a retry limit of > 2 is OK, requested limit: 4 Jan 10 09:26:17 hs14 kernel: [54092.060188] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:17 hs14 kernel: [54092.060235] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.066307] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.072884] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:17 hs14 kernel: [54092.078031] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.085315] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.092632] ath10k_pci 0001:01:00.0: failed to transmit packet, dropping: -19 Jan 10 09:26:17 hs14 kernel: [54092.096943] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.104226] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.110805] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.115940] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.122235] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.127668] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.194155] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.194197] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.199984] ath10k_pci 0001:01:00.0: failed to submit frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.204944] ath10k_pci 0001:01:00.0: failed to push frame: -19 Jan 10 09:26:17 hs14 kernel: [54092.263526] ath10k_pci 0001:01:00.0: device successfully recovered
@openwrt-bot
Copy link
Author

msoltyspl:

The versions of packages in use:

ath10k-firmware-qca9984-ct - 2018-10-10-d366b80d-1
kmod-ath - 4.14.91+4.19.7-1-1
kmod-ath10k-ct - 4.14.91+2018-12-20-118e16da-1

The router (haven't mentioned it above) has been in use since yesterday. We will probably switch to standard (non -ct) firmware and see if it helps.

@openwrt-bot
Copy link
Author

adrianschmutzler:

Can you try again with a more recent version of OpenWrt (master or 19.07.3)? ath10k-ct has seen a lot of improvements ...

@openwrt-bot
Copy link
Author

pauljb:

I am on 19.07.4 and I cannot go more than a few days without it freezing. Not sure if this is the same bug here as it locks up so I cannot see any logs.

@openwrt-bot
Copy link
Author

digitalcircuit:

I have run into what appears to be a similar issue with the NBG6817 as well - it seems to have gotten a bit worse since updating to 19.07.4 with the new ath10k firmware (less automatic reboots, more lockups).

SQM QoS is enabled, 3 virtual WiFi networks (primary, guest, open), with VLANs over Ethernet to another access point (an RT-AC68U running FreshTomato).

With UDP syslog messages sent to another machine on my network, I've managed to capture a reboot (better), and a hang (requires manual hard reboot to recover):

Versions

ath10k-firmware-qca9984-ct - 2020-07-02-1
kmod-ath - 4.14.195+4.19.137-1-2
kmod-ath10k-ct - 4.14.195+2019-09-09-5e8cd86f-1
OpenWrt 19.07.4 r11208-ce6496d796 / LuCI openwrt-19.07 branch git-20.247.75781-0d0ab01

Kernel panic and reboot

Oct 14 15:49:09 OpenWRT kernel: [497781.448640] Unable to handle kernel paging request at virtual address dd79f4c0
Oct 14 15:49:09 OpenWRT kernel: [497781.448664] pgd = d3624000
Oct 14 15:49:09 OpenWRT kernel: [497781.454739] [dd79f4c0] *pgd=5f61141e(bad)
Oct 14 15:49:09 OpenWRT kernel: [497781.457524] Internal error: Oops: 8000000d [#1] SMP ARM
Oct 14 15:49:09 OpenWRT kernel: [497781.461686] Modules linked in: pppoe ppp_async ath10k_pci ath10k_core ath pppox ppp_generic nf_conntrack_netlink nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE c
fg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_commen
t xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY slhc nfnetlink nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw
nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat sch_cake nf_conntrack sch_tbf sch_ingress sch_htb sch_hfsc em_u32
Oct 14 15:49:09 OpenWRT kernel: [497781.515843] cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter
ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb tun usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_of_simple ohci_platform ohci_hcd phy_qcom_dwc3 ahci ehci_platform sd_mod ahci_platform libahci
_platform libahci libata scsi_mod ehci_hcd gpio_button_hotplug ext4 mbcache jbd2 crc32c_generic
Oct 14 15:49:09 OpenWRT kernel: [497781.558730] CPU: 1 PID: 30328 Comm: kworker/1:1 Not tainted 4.14.195 #0
Oct 14 15:49:09 OpenWRT kernel: [497781.580960] Hardware name: Generic DT based system
Oct 14 15:49:09 OpenWRT kernel: [497781.587655] Workqueue: events 0xc0654ce0
Oct 14 15:49:09 OpenWRT kernel: [497781.592241] task: d8153700 task.stack: c8ab4000
Oct 14 15:49:09 OpenWRT kernel: [497781.596409] pc : [] lr : [] psr: a0000113
Oct 14 15:49:09 OpenWRT kernel: [497781.601098] sp : c8ab5e5c ip : 00000002 fp : dd79f4e8
Oct 14 15:49:09 OpenWRT kernel: [497781.607433] r10: 66d16940 r9 : 16e36000 r8 : dd79f240
Oct 14 15:49:09 OpenWRT kernel: [497781.612729] r7 : ddc12010 r6 : 00000000 r5 : c050e0e4 r4 : c05ad3d0
Oct 14 15:49:09 OpenWRT kernel: [497781.618025] r3 : 00000002 r2 : 00000001 r1 : 00000000 r0 : 00000000
Oct 14 15:49:09 OpenWRT kernel: [497781.624363] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
Oct 14 15:49:09 OpenWRT kernel: [497781.630961] Control: 10c5787d Table: 5562406a DAC: 00000051
Oct 14 15:49:09 OpenWRT kernel: [497781.638251] Process kworker/1:1 (pid: 30328, stack limit = 0xc8ab4210)
Oct 14 15:49:09 OpenWRT kernel: [497781.644067] Stack: (0xc8ab5e5c to 0xc8ab6000)
Oct 14 15:49:09 OpenWRT kernel: [497781.650578] 5e40: c05adda8
Oct 14 15:49:09 OpenWRT kernel: [497781.655104] 5e60: c0a61004 00000000 dcc65000 dd79f268 66d16940 16e36000 00009c40 dcc6d400
Oct 14 15:49:09 OpenWRT kernel: [497781.663354] 5e80: 66d16940 dcc65100 dcc6511c 00000005 dcc65140 00000000 ddc18540 c0655c2c
Oct 14 15:49:09 OpenWRT kernel: [497781.671599] 5ea0: 00000000 c065566c 00009c40 dcc6d400 00000000 c0a61004 00000000 00000005
Oct 14 15:49:09 OpenWRT kernel: [497781.679845] 5ec0: 001a5248 00000000 ddc18540 c0651d5c 00000002 0005dc00 001a5248 00000021
Oct 14 15:49:09 OpenWRT kernel: [497781.688092] 5ee0: dcc63380 dcc6d400 dcc5ecc0 dcc63200 dcc63200 dcc5ecc0 00000040 c0654118
Oct 14 15:49:09 OpenWRT kernel: [497781.696339] 5f00: dcc633b8 00000000 dcc63384 dcc6d400 c0a1f0bc 00000000 00000040 c0654d18
Oct 14 15:49:09 OpenWRT kernel: [497781.704585] 5f20: dcc633b8 c4c9db80 ddc18540 ddc1b400 00000000 c033703c ddc18558 ffffe000
Oct 14 15:49:09 OpenWRT kernel: [497781.712832] 5f40: c4c9db80 ddc18540 c4c9db98 ddc18558 ffffe000 c0a02d00 00000008 c0337538
Oct 14 15:49:09 OpenWRT kernel: [497781.721078] 5f60: cca7bebc c8177f40 c8ab4000 c75eaac0 cca7bebc c8177f5c c4c9db80 c03371f4
Oct 14 15:49:09 OpenWRT kernel: [497781.729324] 5f80: 00000000 c033d1d8 00000000 c75eaac0 c033d090 00000000 00000000 00000000
Oct 14 15:49:09 OpenWRT kernel: [497781.737572] 5fa0: 00000000 00000000 00000000 c0307c48 00000000 00000000 00000000 00000000
Oct 14 15:49:09 OpenWRT kernel: [497781.745817] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Oct 14 15:49:09 OpenWRT kernel: [497781.754063] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
Oct 14 15:49:09 OpenWRT kernel: [497781.762305] Function entered at [] from []
Oct 14 15:49:09 OpenWRT kernel: [497781.770551] Code: 000ed1f2 00000000 00000000 00000000 (dd79f440)
Oct 14 15:49:09 OpenWRT kernel: [497781.776361] ---[ end trace db7165d341c15beb ]---
Oct 14 15:49:09 OpenWRT kernel: [497781.782606] Kernel panic - not syncing: Fatal exception
Oct 14 15:49:09 OpenWRT kernel: [497781.787388] CPU0: stopping
Oct 14 15:49:09 OpenWRT kernel: [497781.792764] CPU: 0 PID: 1058 Comm: netifd Tainted: G D 4.14.195 #0
Oct 14 15:49:09 OpenWRT kernel: [497781.795283] Hardware name: Generic DT based system
Oct 14 15:49:09 OpenWRT kernel: [497781.802921] Function entered at [] from []
Oct 14 15:49:09 OpenWRT kernel: [497781.807521] Function entered at [] from []
Oct 14 15:49:09 OpenWRT kernel: [497781.813426] Function entered at [] from []
Oct 14 15:49:09 OpenWRT kernel: [497781.819326] Function entered at [] from []
Oct 14 15:49:09 OpenWRT kernel: [497781.825229] Function entered at [] from []
Oct 14 15:49:09 OpenWRT kernel: [497781.831132] Exception stack(0xdc2ebfb0 to 0xdc2ebff8)
Oct 14 15:49:09 OpenWRT kernel: [497781.837039] bfa0: be8e45d4 00000078 00000058 b6ef0954
Oct 14 15:49:09 OpenWRT kernel: [497781.842257] bfc0: be8e4998 b6ef0953 be8e4a88 ffffffff b6ef0953 00000000 00000000 00000000
Oct 14 15:49:09 OpenWRT kernel: [497781.850502] bfe0: 00012889 be8e4580 b6ea3e28 b6ecfb58 20000010 ffffffff
Oct 14 15:49:09 OpenWRT kernel: [497781.858743] Rebooting in 3 seconds..

Device freeze, no automatic recovery (device stopped responding after all this)

Oct 15 16:01:06 OpenWRT kernel: [87095.946889] INFO: rcu_sched detected stalls on CPUs/tasks:
Oct 15 16:01:06 OpenWRT kernel: [87095.946925] #0111-...: (2 ticks this GP) idle=30e/140000000000001/0 softirq=1165976/1165976 fqs=1050
Oct 15 16:01:06 OpenWRT kernel: [87095.951267] #011(detected by 0, t=2102 jiffies, g=814833, c=814832, q=221)
Oct 15 16:01:06 OpenWRT kernel: [87095.960209] Sending NMI from CPU 0 to CPUs 1:
Oct 15 16:01:06 OpenWRT kernel: [87105.969246] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
Oct 15 16:01:06 OpenWRT kernel: [87105.969292] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 2, skipped old beacon
Oct 15 16:01:06 OpenWRT kernel: [87105.975373] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Oct 15 16:01:06 OpenWRT kernel: [87105.982754] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
Oct 15 16:01:06 OpenWRT kernel: [87105.990031] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 2, skipped old beacon
Oct 15 16:01:06 OpenWRT kernel: [87105.997310] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Oct 15 16:01:06 OpenWRT kernel: [87106.004540] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
Oct 15 16:01:06 OpenWRT kernel: [87106.011899] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 2, skipped old beacon
Oct 15 16:01:06 OpenWRT kernel: [87106.019182] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Oct 15 16:01:06 OpenWRT kernel: [87106.026415] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
Oct 15 16:01:06 OpenWRT kernel: [87106.033768] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 2, skipped old beacon
Oct 15 16:01:06 OpenWRT kernel: [87106.041078] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[...trimmed many repeating similar messages...]
Oct 15 16:01:07 OpenWRT kernel: [87107.294455] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
Oct 15 16:01:07 OpenWRT kernel: [87107.301829] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 2, skipped old beacon
Oct 15 16:01:07 OpenWRT kernel: [87107.309114] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
Oct 15 16:01:07 OpenWRT kernel: [87107.316334] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
Oct 15 16:01:07 OpenWRT kernel: [87107.327832] ath10k_pci 0001:01:00.0: SWBA overrun on vdev 1, skipped old beacon
Oct 15 16:01:07 OpenWRT kernel: [87107.330995] ath10k_pci 0001:01:00.0: SWBA overrun on vdev 1, skipped old beacon
Oct 15 16:01:07 OpenWRT kernel: [87107.338440] ath10k_pci 0001:01:00.0: SWBA overrun on vdev 1, skipped old beacon
Oct 15 16:01:07 OpenWRT kernel: [87107.345502] ath10k_pci 0001:01:00.0: SWBA overrun on vdev 2, skipped old beacon
Oct 15 16:01:07 OpenWRT kernel: [87107.354378] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon


If I should file a new issue instead, share more details, etc, just let me know! I cannot yet reliably reproduce this issue, but it seems to happen every 2-14 days.

@openwrt-bot
Copy link
Author

pauljb:

So I disabled the 5gz radio on my Zyxel Amor Z2 and the crashing has stopped. I am uptime for 10 days which has never happened before. Just putting this out there for anyone else running to this issue. There is clearly an issue with OpenWRT and the 5gz radio on this device.

@openwrt-bot
Copy link
Author

digitalcircuit:

I am still monitoring the situation, but I have noticed that the crash seemingly occurs //slightly// less often ever since a relative's Pixel 3a (non-XL, SoC Qualcomm Snapdragon 670) is no longer connecting to this ZyXEL NBG6817. Meanwhile, the second NBG6817 I set up for this relative at a different place has appeared to continue to crash at the same rate.

Unfortunately, this remains non-deterministic (e.g. right now the relative's NBG6817 has 13 days uptime, with only 10 days uptime here) and I don't have a good way to reliably reproduce the issue. I suspect different wireless clients play a major part in how this bug gets triggered.

2.4 GHz and 5 GHz are enabled here, 3 WiFi SSIDs per band (main, guest, openwireless.org), though the relative's NBG6817 only has 2 per band (main, guest).

@openwrt-bot
Copy link
Author

pauljb:

I'm still crashing on 19.07.6 with my 5gz radio disabled, just seems to be less frequent, but it could just be coincidence. Is anyone trying out the dev snapshot and can say if that solves the issue?

@openwrt-bot
Copy link
Author

digitalcircuit:

Good news: OpenWRT 21.02-rc1 appears to have possibly addressed the WiFi driver crash for ZyXEL NBG6817! I've gotten around 2 weeks of uptime without the same crash.

Bad news: I haven't managed to test for longer because [[https://bugs.openwrt.org/index.php?do=details&task_id=3099|Linux kernel 5.4 appears to have a new crashing bug related to the CPU governor frequency scaling]]; I appear to be getting this bug every time I'm backing up in Deja Dup over OpenSSH chroot'd SFTP (frequently alternating between high CPU and low CPU on the router). I'm testing the "performance" governor workaround now and I'll either add to that bug report (if related) or file a new one (if this workaround doesn't work).

@aparcar aparcar added the kernel pull request/issue with Linux kernel related changes label Feb 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flyspray kernel pull request/issue with Linux kernel related changes
Projects
None yet
Development

No branches or pull requests

2 participants