Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#3487 - tp-link 841n v9: ath79_tiny NETDEV WATCHDOG: eth0 (ag71xx): transmit queue 0 timed out #8356

Open
openwrt-bot opened this issue Dec 3, 2020 · 1 comment
Labels
flyspray kernel pull request/issue with Linux kernel related changes release/19.07 pull request/issue targeted (also) for OpenWrt 19.07 release

Comments

@openwrt-bot
Copy link

glycoknob:

tp-link 841n v9 crashes reliable after ~20-30minutes with: ath79_tiny NETDEV WATCHDOG: eth0 (ag71xx): transmit queue 0 timed out

Seen with OpenWrt 19.07.3 and also with latest commit on 19.07-branch (f1525e7)

This does not happen with ar71xx (identical commit f1525e7)

maybe it's related to flow control? it looks like it's activated (see swconfig output at the end) and I saw some commits that disabled it for these devices on ar71xx but nothing on ath79.

Almost stock OpenWrt just a few (unrelated) disabled config flags: https://github.com/weimarnetz/firmware/tree/96b0208a4307d0c7adc723d7ec49660b4423ed42/patches

crashlog:
crashlog Joki_2_841v9-TP-Link TL-WR841N/ND v9
reboot2020-19.07-beta.1-55-gc32bd44+openwrt-r0+11246-f1525e785e
Linux Joki_2_841v9 4.14.209 #0 Tue Dec 1 20:59:30 2020 mips GNU/Linux
65:f0:e0:67:57:6b:da:86:e5:33:1c:0c:75:8a:6a:2a:ae:f9:b6:5a
TP-Link TL-WR841N/ND v9
Time: 1607021529.501735
Modules: ath9k@80860000+16df0 ath9k_common@80f2c000+2a60 ath9k_hw@80800000+51d60 ath@80ef8000+4140 mac80211@80f80000+62060 iptable_nat@80ef2000+2a0 ipt_REJECT@80e72000+360 ipt_MASQUERADE@80e6e000+280 cfg80211@80ec0000+31790 xt_time@80e67000+620 xt_tcpudp@80e70000+680 xt_tcpmss@80e6b000+3e0 xt_statistic@80e68000+300 xt_state@80e66000+2a0 xt_nat@80e69000+5e0 xt_multiport@80e62000+460 xt_mark@80e61000+280 xt_mac@80e5f000+240 xt_limit@80e58000+3c0 xt_length@80e60000+2a0 xt_hl@80e5e000+300 xt_ecn@80e5d000+520 xt_dscp@80e50000+3c0 xt_conntrack@80e5a000+880 xt_comment@80e56000+1c0 xt_TCPMSS@80e53000+a40 xt_REDIRECT@80e51000+2a0 xt_LOG@80e40000+2e0 xt_HL@80e4b000+500 xt_FLOWOFFLOAD@80e4c000+af0 xt_DSCP@80e4a000+560 xt_CT@80e54000+980 xt_CLASSIFY@80e4f000+220 nf_reject_ipv4@80e47000+7a0 nf_nat_redirect@80e43000+440 nf_nat_masquerade_ipv4@80e42000+5f0 nf_conntrack_ipv4@80e48000+10a0 nf_nat_ipv4@80da7000+bf0 nf_nat@80da4000+1e90 nf_log_ipv4@80dbf000+b80 nf_flow_table_hw@80da1000+7e0 nf_flow_table@80dac000+2950 nf_defrag_ipv4@80da0000+3e0 nf_conntrack_rtcache@80dbe000+960 nf_conntrack@80db0000+b9e0 iptable_mangle@80d9a000+360 iptable_filter@80d93000+260 ipt_ECN@80d94000+540 ip_tables@80d9c000+2470 compat@80dde000+1600 nf_log_ipv6@80d95000+be0 nf_log_common@80d8f000+900 ip6table_mangle@80d8d000+460 ip6table_filter@80e33000+260 ip6_tables@80d90000+2400 ip6t_REJECT@80dd6000+380 x_tables@80e34000+3030 nf_reject_ipv6@80dd7000+840 tun@80d88000+4c40 gpio_button_hotplug@80d4c000+19e0
<5>[ 0.000000] Linux version 4.14.209 (runner@fv-az32-91) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 r0+11246-f1525e785e)) #0 Tue Dec 1 20:59:30 2020
<6>[ 0.000000] bootconsole [early0] enabled
<6>[ 0.000000] CPU0 revision is: 00019374 (MIPS 24Kc)
<6>[ 0.000000] MIPS: machine is TP-Link TL-WR841N/ND v9
<6>[ 0.000000] SoC: Qualcomm Atheros QCA9533 ver 1 rev 1
<6>[ 0.000000] Determined physical RAM map:
<6>[ 0.000000] memory: 02000000 @ 00000000 (usable)
<6>[ 0.000000] Initrd not found or empty - disabling initrd
<4>[ 0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
<4>[ 0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
<6>[ 0.000000] Zone ranges:
<6>[ 0.000000] Normal [mem 0x0000000000000000-0x0000000001ffffff]
<6>[ 0.000000] Movable zone start for each node
<6>[ 0.000000] Early memory node ranges
<6>[ 0.000000] node 0: [mem 0x0000000000000000-0x0000000001ffffff]
<6>[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000001ffffff]
<7>[ 0.000000] On node 0 totalpages: 8192
<7>[ 0.000000] free_area_init_node: node 0, pgdat 804268c0, node_mem_map 81007540
<7>[ 0.000000] Normal zone: 64 pages used for memmap
<7>[ 0.000000] Normal zone: 0 pages reserved
<7>[ 0.000000] Normal zone: 8192 pages, LIFO batch:0
<5>[ 0.000000] random: get_random_bytes called from 0x804296ec with crng_init=0
<7>[ 0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=132768
<7>[ 0.000000] pcpu-alloc: [0] 0
<6>[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 8128
<5>[ 0.000000] Kernel command line: console=ttyS0,115200n8 rootfstype=squashfs,jffs2
<6>[ 0.000000] PID hash table entries: 128 (order: -3, 512 bytes)
<6>[ 0.000000] Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
<6>[ 0.000000] Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
<6>[ 0.000000] Writing ErrCtl register=00000000
<6>[ 0.000000] Readback ErrCtl register=00000000
<6>[ 0.000000] Memory: 26760K/32768K available (3346K kernel code, 134K rwdata, 388K rodata, 1180K init, 195K bss, 6008K reserved, 0K cma-reserved)
<6>[ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
<6>[ 0.000000] NR_IRQS: 51
<6>[ 0.000000] CPU clock: 550.000 MHz
<6>[ 0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6950037990 ns
<6>[ 0.000011] sched_clock: 32 bits at 275MHz, resolution 3ns, wraps every 7809031678ns
<6>[ 0.008266] Calibrating delay loop... 366.18 BogoMIPS (lpj=1830912)
<6>[ 0.074841] pid_max: default: 32768 minimum: 301
<6>[ 0.079941] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
<6>[ 0.086934] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
<6>[ 0.099451] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
<6>[ 0.109877] futex hash table entries: 256 (order: -1, 3072 bytes)
<6>[ 0.116413] pinctrl core: initialized pinctrl subsystem
<6>[ 0.123115] NET: Registered protocol family 16
<6>[ 0.152445] clocksource: Switched to clocksource MIPS
<6>[ 0.159069] NET: Registered protocol family 2
<6>[ 0.164740] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
<6>[ 0.172113] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
<6>[ 0.178895] TCP: Hash tables configured (established 1024 bind 1024)
<6>[ 0.185796] UDP hash table entries: 256 (order: 0, 4096 bytes)
<6>[ 0.191975] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
<6>[ 0.198939] NET: Registered protocol family 1
<7>[ 0.203625] PCI: CLS 0 bytes, default 32
<4>[ 0.209116] Crashlog allocated RAM at address 0x1f00000
<6>[ 0.216654] workingset: timestamp_bits=30 max_order=13 bucket_order=0
<6>[ 0.229890] squashfs: version 4.0 (2009/01/31) Phillip Lougher
<6>[ 0.236100] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
<6>[ 0.258107] io scheduler noop registered
<6>[ 0.262249] io scheduler deadline registered (default)
<6>[ 0.270773] pinctrl-single 1804002c.pinmux: 576 pins at pa b804002c size 72
<6>[ 0.279362] Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
<6>[ 0.287028] console [ttyS0] disabled
<6>[ 0.290856] 18020000.uart: ttyS0 at MMIO 0x18020000 (irq = 9, base_baud = 1562500) is a 16550A
<6>[ 0.300026] console [ttyS0] enabled
<6>[ 0.307588] bootconsole [early0] disabled
<6>[ 0.335074] m25p80 spi0.0: w25q32 (4096 Kbytes)
<5>[ 0.339819] 3 fixed-partitions partitions found on MTD device spi0.0
<5>[ 0.346411] Creating 3 MTD partitions on "spi0.0":
<5>[ 0.351378] 0x000000000000-0x000000020000 : "u-boot"
<5>[ 0.357577] 0x000000020000-0x0000003f0000 : "firmware"
<5>[ 0.364139] 2 tplink-fw partitions found on MTD device firmware
<5>[ 0.370261] Creating 2 MTD partitions on "firmware":
<5>[ 0.375472] 0x000000000000-0x0000001438f6 : "kernel"
<5>[ 0.381536] 0x0000001438f8-0x0000003d0000 : "rootfs"
<5>[ 0.387623] mtd: device 3 (rootfs) set to be root filesystem
<5>[ 0.393596] 1 squashfs-split partitions found on MTD device rootfs
<5>[ 0.399995] 0x000000370000-0x0000003d0000 : "rootfs_data"
<5>[ 0.406508] 0x0000003f0000-0x000000400000 : "art"
<6>[ 0.413186] libphy: Fixed MDIO Bus: probed
<3>[ 0.753580] ag71xx 19000000.eth: Could not connect to PHY device. Deferring probe.
<6>[ 1.432891] libphy: ag71xx_mdio: probed
<6>[ 1.438427] libphy: ar8xxx-mdio: probed
<6>[ 1.451985] switch0: Atheros AR8229 rev. 1 switch registered on mdio-bus.0
<6>[ 1.504376] ag71xx 1a000000.eth: connected to PHY at fixed-0:00 [uid=00000000, driver=Generic PHY]
<6>[ 1.514537] eth0: Atheros AG71xx at 0xba000000, irq 5, mode: gmii
<6>[ 1.524220] NET: Registered protocol family 10
<6>[ 1.536076] Segment Routing with IPv6
<6>[ 1.539991] NET: Registered protocol family 17
<6>[ 1.545477] 8021q: 802.1Q VLAN Support v1.8
<6>[ 1.884807] ag71xx 19000000.eth: connected to PHY at mdio-bus.0:1f:04 [uid=004dd042, driver=Generic PHY]
<6>[ 1.895813] eth1: Atheros AG71xx at 0xb9000000, irq 4, mode: mii
<6>[ 1.903143] hctosys: unable to open rtc device (rtc0)
<6>[ 1.914565] VFS: Mounted root (squashfs filesystem) readonly on device 31:3.
<6>[ 1.929876] Freeing unused kernel memory: 1180K
<4>[ 1.934589] This architecture does not have kernel memory protection.
<5>[ 2.542471] random: fast init done
<14>[ 3.981320] init: Console is alive
<14>[ 3.985252] init: - watchdog -
<14>[ 5.226245] kmodloader: loading kernel modules from /etc/modules-boot.d/

<14>[ 5.448312] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
<14>[ 5.466446] init: - preinit -
<5>[ 7.317162] random: jshn: uninitialized urandom read (4 bytes read)
<5>[ 7.640937] random: jshn: uninitialized urandom read (4 bytes read)
<5>[ 7.881979] random: jshn: uninitialized urandom read (4 bytes read)
<6>[ 9.350148] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
<6>[ 9.356368] IPv6: ADDRCONF(NETDEV_UP): eth0.1: link is not ready
<4>[ 9.457494] urandom_read: 4 callbacks suppressed
<5>[ 9.457505] random: procd: uninitialized urandom read (4 bytes read)
<6>[ 10.413685] eth0: link up (1000Mbps/Full duplex)
<6>[ 10.418503] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
<6>[ 10.425640] IPv6: ADDRCONF(NETDEV_CHANGE): eth0.1: link becomes ready
<5>[ 13.204259] jffs2: notice: (387) jffs2_build_xattr_subsystem: complete building xattr subsystem, 46 of xdatum (2 unchecked, 44 orphan) and 56 of xref (44 dead, 0 orphan) found.
<14>[ 13.222309] mount_root: switching to jffs2 overlay
<4>[ 13.256999] overlayfs: upper fs does not support tmpfile.
<12>[ 13.270584] urandom-seed: Seeding with /etc/urandom.seed
<6>[ 13.606629] eth0: link down
<14>[ 13.635329] procd: - early -
<14>[ 13.638443] procd: - watchdog -
<14>[ 14.415246] procd: - watchdog -
<14>[ 14.418916] procd: - ubus -
<5>[ 14.862014] random: ubusd: uninitialized urandom read (4 bytes read)
<5>[ 14.870949] random: ubusd: uninitialized urandom read (4 bytes read)
<14>[ 14.879269] procd: - init -
<14>[ 16.496317] kmodloader: loading kernel modules from /etc/modules.d/*
<6>[ 16.515274] tun: Universal TUN/TAP device driver, 1.6
<6>[ 16.539819] ip6_tables: (C) 2000-2006 Netfilter Core Team
<6>[ 16.570947] Loading modules backported from Linux version v4.19.137-0-gc076c79e03c6
<6>[ 16.578942] Backport generated by backports.git v4.19.137-1-0-g60c3a249
<6>[ 16.604123] ip_tables: (C) 2000-2006 Netfilter Core Team
<6>[ 16.632886] nf_conntrack version 0.5.0 (1024 buckets, 4096 max)
<6>[ 16.798269] xt_time: kernel timezone is -0000
<7>[ 17.003259] ath: EEPROM regdomain: 0x0
<7>[ 17.003272] ath: EEPROM indicates default country code should be used
<7>[ 17.003276] ath: doing EEPROM country->regdmn map search
<7>[ 17.003294] ath: country maps to regdmn code: 0x3a
<7>[ 17.003301] ath: Country alpha2 being used: US
<7>[ 17.003306] ath: Regpair used: 0x3a
<7>[ 17.017937] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
<6>[ 17.020235] ieee80211 phy0: Atheros AR9531 Rev:1 mem=0xb8100000, irq=12
<14>[ 17.372997] kmodloader: done loading kernel modules from /etc/modules.d/*
<14>[ 17.626827] urngd: v1.0.2 started.
<5>[ 18.145367] random: crng init done
<6>[ 38.208915] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
<6>[ 38.216318] eth0: link up (1000Mbps/Full duplex)
<6>[ 38.221130] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
<6>[ 38.268978] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
<6>[ 38.304912] IPv6: ADDRCONF(NETDEV_UP): br-vap: link is not ready
<6>[ 38.354025] IPv6: ADDRCONF(NETDEV_UP): br-roam: link is not ready
<7>[ 40.476601] ath: EEPROM regdomain: 0x8114
<7>[ 40.476615] ath: EEPROM indicates we should expect a country code
<7>[ 40.476621] ath: doing EEPROM country->regdmn map search
<7>[ 40.476627] ath: country maps to regdmn code: 0x37
<7>[ 40.476634] ath: Country alpha2 being used: DE
<7>[ 40.476639] ath: Regpair used: 0x37
<7>[ 40.476646] ath: regdomain 0x8114 dynamically updated by user
<6>[ 45.401625] IPv6: ADDRCONF(NETDEV_UP): wlan0-1: link is not ready
<6>[ 45.414072] br-vap: port 1(wlan0-1) entered blocking state
<6>[ 45.419792] br-vap: port 1(wlan0-1) entered disabled state
<6>[ 45.425965] device wlan0-1 entered promiscuous mode
<6>[ 45.747193] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0-1: link becomes ready
<6>[ 45.754274] br-vap: port 1(wlan0-1) entered blocking state
<6>[ 45.759954] br-vap: port 1(wlan0-1) entered forwarding state
<6>[ 45.778367] br-roam: port 1(wlan0-2) entered blocking state
<6>[ 45.784586] br-roam: port 1(wlan0-2) entered disabled state
<6>[ 45.790829] device wlan0-2 entered promiscuous mode
<6>[ 45.802728] IPv6: ADDRCONF(NETDEV_CHANGE): br-vap: link becomes ready
<6>[ 45.827453] IPv6: ADDRCONF(NETDEV_UP): wlan0-2: link is not ready
<6>[ 45.834186] br-roam: port 1(wlan0-2) entered blocking state
<6>[ 45.839980] br-roam: port 1(wlan0-2) entered forwarding state
<6>[ 45.847573] IPv6: ADDRCONF(NETDEV_CHANGE): br-roam: link becomes ready
<6>[ 45.908943] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0-2: link becomes ready
<6>[ 46.051074] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
<6>[ 46.469277] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
<4>[ 98.092507] ------------[ cut here ]------------
<4>[ 98.097311] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 0x802bf250
<6>[ 98.104640] NETDEV WATCHDOG: eth0 (ag71xx): transmit queue 0 timed out
<0>[ 98.111390] Kernel panic - not syncing: panic_on_warn set ...
<0>[ 98.111390]

# swconfig dev switch0 show
Global attributes:
enable_vlan: 1
ar8xxx_mib_poll_interval: 500
ar8xxx_mib_type: 0
enable_mirror_rx: 0
enable_mirror_tx: 0
mirror_monitor_port: 0
mirror_source_port: 0
arl_table: address resolution table
Port 0: MAC c4:6e:1f:31:64:fa

Port 0:
mib: MIB counters
RxGoodByte : 515088 (503.0 KiB)
TxByte : 0

pvid: 0
link: port:0 link:up speed:1000baseT full-duplex txflow rxflow
Port 1:
mib: No MIB data
pvid: 1
link: port:1 link:down
Port 2:
mib: No MIB data
pvid: 1
link: port:2 link:down
Port 3:
mib: No MIB data
pvid: 1
link: port:3 link:down
Port 4:
mib: No MIB data
pvid: 1
link: port:4 link:down
VLAN 1:
vid: 1
ports: 0t 1 2 3 4

@openwrt-bot
Copy link
Author

glycoknob:

some additional info: we have 3 other routers that are 841nv9 and these just run fine with ath79_tiny - I'm trying to reproduce this - maybe this only happens under certain circumstances...

@aparcar aparcar added release/19.07 pull request/issue targeted (also) for OpenWrt 19.07 release kernel pull request/issue with Linux kernel related changes labels Feb 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flyspray kernel pull request/issue with Linux kernel related changes release/19.07 pull request/issue targeted (also) for OpenWrt 19.07 release
Projects
None yet
Development

No branches or pull requests

2 participants