OpenWrt/LEDE Project

  • Status Unconfirmed
  • Percent Complete
    0%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Medium
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Aaron Tiensivu - 02.07.2020

FS#3215 - Multiple kernel oops in airtime fairness (potentially) - mt7621 chipset

Device: Netgear R6260
Build: r13611-3f27a6e640 - downloaded from openwrt site
Target: ramips/mt7621
Arch: mipsel_24kc

Seems to happen during normal use of router - simple configuration of 1 2.4ghz network and 1 5ghz network, connected over ethernet to AT&T “really-not-DSL-but-kind-of” service. (400mbit bidirectional)

I had 2 oops within 24 hours of operation. I am unsure of steps to reproduce yet.
I have not rebooted, but I suspect the second oops was a side effect of the first one.

SQM configured but specifically disabled since I wanted to test hardware offloading capabilities.

[   15.183669] mt7621-pci 1e140000.pcie: bus=2 slot=1 irq=24
[   15.194488] pci 0000:00:01.0: enabling device (0004 -> 0007)
[   15.205781] mt7603e 0000:02:00.0: enabling device (0000 -> 0002)
[   15.217920] mt7603e 0000:02:00.0: ASIC revision: 76030010
[   15.847734] mt7603e 0000:02:00.0: Firmware Version: ap_pcie
[   15.858870] mt7603e 0000:02:00.0: Build Time: 20160107100755
[   15.894419] mt7603e 0000:02:00.0: firmware init done
[   16.080391] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
[   16.091938] mt7621-pci 1e140000.pcie: bus=1 slot=0 irq=23
[   16.102807] pci 0000:00:00.0: enabling device (0004 -> 0007)
[   16.114166] mt7615e 0000:01:00.0: enabling device (0000 -> 0002)
[   16.140685] ieee80211 phy1: Selected rate control algorithm 'minstrel_ht'
[   16.238338] mt7615e 0000:01:00.0: HW/SW Version: 0x8a108a10, Build Time: 20180518100604a
[   16.568442] mt7615e 0000:01:00.0: N9 Firmware Version: 2.0, Build Time: 20200131181812
[   16.594490] mt7615e 0000:01:00.0: CR4 Firmware Version: _reserved_, Build Time: 20190121161307
[ 3850.067170] ------------[ cut here ]------------
[ 3850.076587] WARNING: CPU: 0 PID: 0 at backports-5.7-rc3-1/net/mac80211/sta_info.c:1929 ieee80211_sta_update_pending_airtime+0x22c/0x234 [mac80211]
[ 3850.102772] STA f8:e9:4e:4e:ff:f1 AC 2 txq pending airtime underflow: 4294966536, 760
[ 3850.102778] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack mt7615e mt7615_common mt7603e mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY wireguard slhc sch_cake nlmon nfnetlink_queue nfnetlink_log nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic
[ 3850.118561]  act_skbedit act_mirred dwc3 ohci_platform ohci_hcd ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ahci libahci libata fsl_mph_dr_of ehci_platform ehci_fsl sd_mod scsi_mod ehci_hcd ifb ip6_udp_tunnel udp_tunnel netlink_diag leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[ 3850.419155] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.4.48 #0
[ 3850.430934] Stack : 86f2de80 80077324 806c0000 806bc3bc 80720000 806bc384 806bb4d8 87c0dc44
[ 3850.447558]         80860000 80707248 80706d83 80651380 00000000 00000001 87c0dbe8 077483ba
[ 3850.464170]         00000000 00000000 808a0000 00000000 00000030 0000016d 342e3520 2038342e
[ 3850.480782]         00000000 000000bf 00000000 00066553 00000000 80720000 00000000 86408a84
[ 3850.497395]         00000009 00000000 00000000 86f2e000 00000002 8033d914 00000000 80860000
[ 3850.514009]         ...
[ 3850.518860] Call Trace:
[ 3850.523742] [<8000b72c>] show_stack+0x30/0x100
[ 3850.532579] [<805945b4>] dump_stack+0xa4/0xdc
[ 3850.541245] [<80028058>] __warn+0xc0/0x10c
[ 3850.549383] [<80028130>] warn_slowpath_fmt+0x8c/0xac
[ 3850.559389] [<86408a84>] ieee80211_sta_update_pending_airtime+0x22c/0x234 [mac80211]
[ 3850.574838] [<8640205c>] ieee80211_report_low_ack+0xf0/0x43c [mac80211]
[ 3850.588039] [<864023c0>] ieee80211_free_txskb+0x18/0x30 [mac80211]
[ 3850.600375] [<86f25468>] mt76_tx_complete_skb+0xa8/0xbc [mt76]
[ 3850.611991] [<86feaa20>] mt7615_txp_skb_unmap+0x1f4/0x230 [mt7615_common]
[ 3850.625731] ---[ end trace 176fb10434b931c6 ]---
[22886.192908] ------------[ cut here ]------------
[22886.202177] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:448 dev_watchdog+0x2fc/0x304
[22886.218705] NETDEV WATCHDOG: eth0 (mtk_soc_eth): transmit queue 0 timed out
[22886.232589] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack mt7615e mt7615_common mt7603e mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY wireguard slhc sch_cake nlmon nfnetlink_queue nfnetlink_log nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic
[22886.232777]  act_skbedit act_mirred dwc3 ohci_platform ohci_hcd ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ahci libahci libata fsl_mph_dr_of ehci_platform ehci_fsl sd_mod scsi_mod ehci_hcd ifb ip6_udp_tunnel udp_tunnel netlink_diag leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[22886.533261] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W         5.4.48 #0
[22886.547802] Stack : ffffffff 80077324 806c0000 806bc3bc 80720000 806bc384 806bb4d8 87c11db4
[22886.564425]         80860000 87c3c728 80706d83 80651380 00000002 00000001 87c11d58 077483ba
[22886.581050]         00000000 00000000 808a0000 00000000 00000030 000001b9 342e3520 2038342e
[22886.597673]         00000000 00000472 00000000 0008230d 00000000 80720000 00000000 8045e458
[22886.614291]         00000009 00000002 00200000 00000122 00000002 8033d914 00000008 80860008
[22886.630914]         ...
[22886.635772] Call Trace:
[22886.640663] [<8000b72c>] show_stack+0x30/0x100
[22886.649507] [<805945b4>] dump_stack+0xa4/0xdc
[22886.658176] [<80028058>] __warn+0xc0/0x10c
[22886.666317] [<80028130>] warn_slowpath_fmt+0x8c/0xac
[22886.676230] [<8045e458>] dev_watchdog+0x2fc/0x304
[22886.685595] [<800904e4>] call_timer_fn.isra.34+0x20/0x90
[22886.696155] [<8009072c>] run_timer_softirq+0x1d8/0x230
[22886.706398] [<805b510c>] __do_softirq+0x16c/0x334
[22886.715788] [<8002c80c>] irq_exit+0x98/0xb0
[22886.724121] [<802dcec0>] plat_irq_dispatch+0x64/0x104
[22886.734175] [<80006de8>] except_vec_vi_end+0xb8/0xc4
[22886.744064] [<805b47c0>] r4k_wait_irqoff+0x1c/0x24
[22886.753828] ---[ end trace 176fb10434b931c7 ]---

ethtool configuration dump:


Features for eth0:
rx-checksumming: on
tx-checksumming: on
	tx-checksum-ipv4: on
	tx-checksum-ip-generic: off [fixed]
	tx-checksum-ipv6: on
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp-mangleid-segmentation: on
	tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]

I’m attaching the full ‘dmesg’ output with this submission.

I can provide package list/configuration files if it will help.


Project Manager
Hauke Mehrtens commented on 04.07.2020 14:30

The same bug is reported in #3204 and here: https://github.com/openwrt/mt76/issues/406

Aaron Tiensivu commented on 05.07.2020 20:53

Same build, a few days later:

[199807.650761] WARNING: CPU: 3 PID: 25 at target-mipsel_24kc_musl/linux-ramips_mt7621/mt76-2020-05-30-85c51608/mt7603/mac.c:206 mt7603_filter_tx+0x16c/0x174 [mt7603e]
[199807.680081] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack mt7615e mt7615_common mt7603e mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY wireguard slhc sch_cake nlmon nfnetlink_queue nfnetlink_log nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic
[199807.680281] act_skbedit act_mirred dwc3 ohci_platform ohci_hcd ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ahci libahci libata fsl_mph_dr_of ehci_platform ehci_fsl sd_mod scsi_mod ehci_hcd ifb ip6_udp_tunnel udp_tunnel netlink_diag leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[199807.981216] CPU: 3 PID: 25 Comm: ksoftirqd/3 Tainted: G W 5.4.48 #0
[199807.996440] Stack : 00000001 80077324 806c0000 806bc3bc 80720000 806bc384 806bb4d8 87ca5bd4
[199808.013232] 80860000 87c89fc8 80706d83 80651380 00000003 00000001 87ca5b78 0cfa4347
[199808.030019] 00000000 00000000 808a0000 00000000 30230000 00000381 00000010 20202020
[199808.046807] 00000000 000026d5 00000000 20202020 00000000 80720000 00000000 86f1451c
[199808.063596] 00000009 00000004 00000004 86f20000 00000010 8033d914 0000000c 8086000c
[199808.080386] ...
[199808.085416] Call Trace:
[199808.090481] [<8000b72c>] show_stack+0x30/0x100
[199808.099495] [<805945b4>] dump_stack+0xa4/0xdc
[199808.108337] [<80028058>] __warn+0xc0/0x10c
[199808.116652] [<80028100>] warn_slowpath_fmt+0x5c/0xac
[199808.126719] [<86f1451c>] mt7603_filter_tx+0x16c/0x174 [mt7603e]
[199808.138670] [<86f146b8>] mt7603_wtbl_set_ps+0x130/0x138 [mt7603e]
[199808.150965] [<86f11a04>] mt7603_sta_ps+0x38/0x430 [mt7603e]
[199808.162236] [<864d424c>] mt76_rx_poll_complete+0x4bc/0x5b4 [mt76]
[199808.174533] [<864d161c>] mt76_dma_cleanup+0x860/0x1064 [mt76]
[199808.186431] —[ end trace 56c7b962a1309524 ]—

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing