Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#3215 - Multiple kernel oops in airtime fairness (potentially) - mt7621 chipset #8043

Open
openwrt-bot opened this issue Jul 2, 2020 · 2 comments
Labels

Comments

@openwrt-bot
Copy link

atiensivu:

Device: Netgear R6260
Build: r13611-3f27a6e640 - downloaded from openwrt site
Target: ramips/mt7621
Arch: mipsel_24kc

Seems to happen during normal use of router - simple configuration of 1 2.4ghz network and 1 5ghz network, connected over ethernet to AT&T "really-not-DSL-but-kind-of" service. (400mbit bidirectional)

I had 2 oops within 24 hours of operation. I am unsure of steps to reproduce yet.
I have not rebooted, but I suspect the second oops was a side effect of the first one.

SQM configured but specifically disabled since I wanted to test hardware offloading capabilities.

[ 15.183669] mt7621-pci 1e140000.pcie: bus=2 slot=1 irq=24 [ 15.194488] pci 0000:00:01.0: enabling device (0004 -> 0007) [ 15.205781] mt7603e 0000:02:00.0: enabling device (0000 -> 0002) [ 15.217920] mt7603e 0000:02:00.0: ASIC revision: 76030010 [ 15.847734] mt7603e 0000:02:00.0: Firmware Version: ap_pcie [ 15.858870] mt7603e 0000:02:00.0: Build Time: 20160107100755 [ 15.894419] mt7603e 0000:02:00.0: firmware init done [ 16.080391] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht' [ 16.091938] mt7621-pci 1e140000.pcie: bus=1 slot=0 irq=23 [ 16.102807] pci 0000:00:00.0: enabling device (0004 -> 0007) [ 16.114166] mt7615e 0000:01:00.0: enabling device (0000 -> 0002) [ 16.140685] ieee80211 phy1: Selected rate control algorithm 'minstrel_ht' [ 16.238338] mt7615e 0000:01:00.0: HW/SW Version: 0x8a108a10, Build Time: 20180518100604a [ 16.568442] mt7615e 0000:01:00.0: N9 Firmware Version: 2.0, Build Time: 20200131181812 [ 16.594490] mt7615e 0000:01:00.0: CR4 Firmware Version: _reserved_, Build Time: 20190121161307 [ 3850.067170] ------------[ cut here ]------------ [ 3850.076587] WARNING: CPU: 0 PID: 0 at backports-5.7-rc3-1/net/mac80211/sta_info.c:1929 ieee80211_sta_update_pending_airtime+0x22c/0x234 [mac80211] [ 3850.102772] STA f8:e9:4e:4e:ff:f1 AC 2 txq pending airtime underflow: 4294966536, 760 [ 3850.102778] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack mt7615e mt7615_common mt7603e mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY wireguard slhc sch_cake nlmon nfnetlink_queue nfnetlink_log nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic [ 3850.118561] act_skbedit act_mirred dwc3 ohci_platform ohci_hcd ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ahci libahci libata fsl_mph_dr_of ehci_platform ehci_fsl sd_mod scsi_mod ehci_hcd ifb ip6_udp_tunnel udp_tunnel netlink_diag leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common [ 3850.419155] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.4.48 #0 [ 3850.430934] Stack : 86f2de80 80077324 806c0000 806bc3bc 80720000 806bc384 806bb4d8 87c0dc44 [ 3850.447558] 80860000 80707248 80706d83 80651380 00000000 00000001 87c0dbe8 077483ba [ 3850.464170] 00000000 00000000 808a0000 00000000 00000030 0000016d 342e3520 2038342e [ 3850.480782] 00000000 000000bf 00000000 00066553 00000000 80720000 00000000 86408a84 [ 3850.497395] 00000009 00000000 00000000 86f2e000 00000002 8033d914 00000000 80860000 [ 3850.514009] ... [ 3850.518860] Call Trace: [ 3850.523742] [<8000b72c>] show_stack+0x30/0x100 [ 3850.532579] [<805945b4>] dump_stack+0xa4/0xdc [ 3850.541245] [<80028058>] __warn+0xc0/0x10c [ 3850.549383] [<80028130>] warn_slowpath_fmt+0x8c/0xac [ 3850.559389] [<86408a84>] ieee80211_sta_update_pending_airtime+0x22c/0x234 [mac80211] [ 3850.574838] [<8640205c>] ieee80211_report_low_ack+0xf0/0x43c [mac80211] [ 3850.588039] [<864023c0>] ieee80211_free_txskb+0x18/0x30 [mac80211] [ 3850.600375] [<86f25468>] mt76_tx_complete_skb+0xa8/0xbc [mt76] [ 3850.611991] [<86feaa20>] mt7615_txp_skb_unmap+0x1f4/0x230 [mt7615_common] [ 3850.625731] ---[ end trace 176fb10434b931c6 ]--- [22886.192908] ------------[ cut here ]------------ [22886.202177] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:448 dev_watchdog+0x2fc/0x304 [22886.218705] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out [22886.232589] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack mt7615e mt7615_common mt7603e mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY wireguard slhc sch_cake nlmon nfnetlink_queue nfnetlink_log nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic [22886.232777] act_skbedit act_mirred dwc3 ohci_platform ohci_hcd ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ahci libahci libata fsl_mph_dr_of ehci_platform ehci_fsl sd_mod scsi_mod ehci_hcd ifb ip6_udp_tunnel udp_tunnel netlink_diag leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common [22886.533261] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G W 5.4.48 #0 [22886.547802] Stack : ffffffff 80077324 806c0000 806bc3bc 80720000 806bc384 806bb4d8 87c11db4 [22886.564425] 80860000 87c3c728 80706d83 80651380 00000002 00000001 87c11d58 077483ba [22886.581050] 00000000 00000000 808a0000 00000000 00000030 000001b9 342e3520 2038342e [22886.597673] 00000000 00000472 00000000 0008230d 00000000 80720000 00000000 8045e458 [22886.614291] 00000009 00000002 00200000 00000122 00000002 8033d914 00000008 80860008 [22886.630914] ... [22886.635772] Call Trace: [22886.640663] [<8000b72c>] show_stack+0x30/0x100 [22886.649507] [<805945b4>] dump_stack+0xa4/0xdc [22886.658176] [<80028058>] __warn+0xc0/0x10c [22886.666317] [<80028130>] warn_slowpath_fmt+0x8c/0xac [22886.676230] [<8045e458>] dev_watchdog+0x2fc/0x304 [22886.685595] [<800904e4>] call_timer_fn.isra.34+0x20/0x90 [22886.696155] [<8009072c>] run_timer_softirq+0x1d8/0x230 [22886.706398] [<805b510c>] __do_softirq+0x16c/0x334 [22886.715788] [<8002c80c>] irq_exit+0x98/0xb0 [22886.724121] [<802dcec0>] plat_irq_dispatch+0x64/0x104 [22886.734175] [<80006de8>] except_vec_vi_end+0xb8/0xc4 [22886.744064] [<805b47c0>] r4k_wait_irqoff+0x1c/0x24 [22886.753828] ---[ end trace 176fb10434b931c7 ]---

ethtool configuration dump:

Features for eth0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]

I'm attaching the full 'dmesg' output with this submission.

I can provide package list/configuration files if it will help.

@openwrt-bot
Copy link
Author

Hauke:

The same bug is reported in [[https://bugs.openwrt.org/index.php?do=details&task_id=3204|#3204]] and here: openwrt/mt76#406

@openwrt-bot
Copy link
Author

atiensivu:

Same build, a few days later:

[199807.650761] WARNING: CPU: 3 PID: 25 at target-mipsel_24kc_musl/linux-ramips_mt7621/mt76-2020-05-30-85c51608/mt7603/mac.c:206 mt7603_filter_tx+0x16c/0x174 [mt7603e]
[199807.680081] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack mt7615e mt7615_common mt7603e mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY wireguard slhc sch_cake nlmon nfnetlink_queue nfnetlink_log nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic
[199807.680281] act_skbedit act_mirred dwc3 ohci_platform ohci_hcd ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ahci libahci libata fsl_mph_dr_of ehci_platform ehci_fsl sd_mod scsi_mod ehci_hcd ifb ip6_udp_tunnel udp_tunnel netlink_diag leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[199807.981216] CPU: 3 PID: 25 Comm: ksoftirqd/3 Tainted: G W 5.4.48 #0
[199807.996440] Stack : 00000001 80077324 806c0000 806bc3bc 80720000 806bc384 806bb4d8 87ca5bd4
[199808.013232] 80860000 87c89fc8 80706d83 80651380 00000003 00000001 87ca5b78 0cfa4347
[199808.030019] 00000000 00000000 808a0000 00000000 30230000 00000381 00000010 20202020
[199808.046807] 00000000 000026d5 00000000 20202020 00000000 80720000 00000000 86f1451c
[199808.063596] 00000009 00000004 00000004 86f20000 00000010 8033d914 0000000c 8086000c
[199808.080386] ...
[199808.085416] Call Trace:
[199808.090481] [<8000b72c>] show_stack+0x30/0x100
[199808.099495] [<805945b4>] dump_stack+0xa4/0xdc
[199808.108337] [<80028058>] __warn+0xc0/0x10c
[199808.116652] [<80028100>] warn_slowpath_fmt+0x5c/0xac
[199808.126719] [<86f1451c>] mt7603_filter_tx+0x16c/0x174 [mt7603e]
[199808.138670] [<86f146b8>] mt7603_wtbl_set_ps+0x130/0x138 [mt7603e]
[199808.150965] [<86f11a04>] mt7603_sta_ps+0x38/0x430 [mt7603e]
[199808.162236] [<864d424c>] mt76_rx_poll_complete+0x4bc/0x5b4 [mt76]
[199808.174533] [<864d161c>] mt76_dma_cleanup+0x860/0x1064 [mt76]
[199808.186431] ---[ end trace 56c7b962a1309524 ]---

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant