OpenWrt/LEDE Project

Attached to Project: OpenWrt/LEDE Project
Opened by Martin Tippmann - 03.12.2020

tp-link 841n v9 crashes reliable after ~20-30minutes with: ath79_tiny NETDEV WATCHDOG: eth0 (ag71xx): transmit queue 0 timed out

Seen with OpenWrt 19.07.3 and also with latest commit on 19.07-branch (f1525e785e9359bfc34b1362ccf37aca48968028)

This does not happen with ar71xx (identical commit f1525e785e9359bfc34b1362ccf37aca48968028)

maybe it’s related to flow control? it looks like it’s activated (see swconfig output at the end) and I saw some commits that disabled it for these devices on ar71xx but nothing on ath79.

Almost stock OpenWrt just a few (unrelated) disabled config flags: https://github.com/weimarnetz/firmware/tree/96b0208a4307d0c7adc723d7ec49660b4423ed42/patches

crashlog:

 crashlog Joki_2_841v9-TP-Link TL-WR841N/ND v9
reboot2020-19.07-beta.1-55-gc32bd44+openwrt-r0+11246-f1525e785e
Linux Joki_2_841v9 4.14.209 #0 Tue Dec 1 20:59:30 2020 mips GNU/Linux
65:f0:e0:67:57:6b:da:86:e5:33:1c:0c:75:8a:6a:2a:ae:f9:b6:5a
TP-Link TL-WR841N/ND v9
Time: 1607021529.501735
Modules:	ath9k@80860000+16df0	ath9k_common@80f2c000+2a60	ath9k_hw@80800000+51d60	ath@80ef8000+4140	mac80211@80f80000+62060	iptable_nat@80ef2000+2a0	ipt_REJECT@80e72000+360	ipt_MASQUERADE@80e6e000+280	cfg80211@80ec0000+31790	xt_time@80e67000+620	xt_tcpudp@80e70000+680	xt_tcpmss@80e6b000+3e0	xt_statistic@80e68000+300	xt_state@80e66000+2a0	xt_nat@80e69000+5e0	xt_multiport@80e62000+460	xt_mark@80e61000+280	xt_mac@80e5f000+240	xt_limit@80e58000+3c0	xt_length@80e60000+2a0	xt_hl@80e5e000+300	xt_ecn@80e5d000+520	xt_dscp@80e50000+3c0	xt_conntrack@80e5a000+880	xt_comment@80e56000+1c0	xt_TCPMSS@80e53000+a40	xt_REDIRECT@80e51000+2a0	xt_LOG@80e40000+2e0	xt_HL@80e4b000+500	xt_FLOWOFFLOAD@80e4c000+af0	xt_DSCP@80e4a000+560	xt_CT@80e54000+980	xt_CLASSIFY@80e4f000+220	nf_reject_ipv4@80e47000+7a0	nf_nat_redirect@80e43000+440	nf_nat_masquerade_ipv4@80e42000+5f0	nf_conntrack_ipv4@80e48000+10a0	nf_nat_ipv4@80da7000+bf0	nf_nat@80da4000+1e90	nf_log_ipv4@80dbf000+b80	nf_flow_table_hw@80da1000+7e0	nf_flow_table@80dac000+2950	nf_defrag_ipv4@80da0000+3e0	nf_conntrack_rtcache@80dbe000+960	nf_conntrack@80db0000+b9e0	iptable_mangle@80d9a000+360	iptable_filter@80d93000+260	ipt_ECN@80d94000+540	ip_tables@80d9c000+2470	compat@80dde000+1600	nf_log_ipv6@80d95000+be0	nf_log_common@80d8f000+900	ip6table_mangle@80d8d000+460	ip6table_filter@80e33000+260	ip6_tables@80d90000+2400	ip6t_REJECT@80dd6000+380	x_tables@80e34000+3030	nf_reject_ipv6@80dd7000+840	tun@80d88000+4c40	gpio_button_hotplug@80d4c000+19e0
<5>[    0.000000] Linux version 4.14.209 (runner@fv-az32-91) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 r0+11246-f1525e785e)) #0 Tue Dec 1 20:59:30 2020
<6>[    0.000000] bootconsole [early0] enabled
<6>[    0.000000] CPU0 revision is: 00019374 (MIPS 24Kc)
<6>[    0.000000] MIPS: machine is TP-Link TL-WR841N/ND v9
<6>[    0.000000] SoC: Qualcomm Atheros QCA9533 ver 1 rev 1
<6>[    0.000000] Determined physical RAM map:
<6>[    0.000000]  memory: 02000000 @ 00000000 (usable)
<6>[    0.000000] Initrd not found or empty - disabling initrd
<4>[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
<4>[    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
<6>[    0.000000] Zone ranges:
<6>[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000001ffffff]
<6>[    0.000000] Movable zone start for each node
<6>[    0.000000] Early memory node ranges
<6>[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000001ffffff]
<6>[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000001ffffff]
<7>[    0.000000] On node 0 totalpages: 8192
<7>[    0.000000] free_area_init_node: node 0, pgdat 804268c0, node_mem_map 81007540
<7>[    0.000000]   Normal zone: 64 pages used for memmap
<7>[    0.000000]   Normal zone: 0 pages reserved
<7>[    0.000000]   Normal zone: 8192 pages, LIFO batch:0
<5>[    0.000000] random: get_random_bytes called from 0x804296ec with crng_init=0
<7>[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
<7>[    0.000000] pcpu-alloc: [0] 0
<6>[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 8128
<5>[    0.000000] Kernel command line: console=ttyS0,115200n8 rootfstype=squashfs,jffs2
<6>[    0.000000] PID hash table entries: 128 (order: -3, 512 bytes)
<6>[    0.000000] Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
<6>[    0.000000] Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
<6>[    0.000000] Writing ErrCtl register=00000000
<6>[    0.000000] Readback ErrCtl register=00000000
<6>[    0.000000] Memory: 26760K/32768K available (3346K kernel code, 134K rwdata, 388K rodata, 1180K init, 195K bss, 6008K reserved, 0K cma-reserved)
<6>[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
<6>[    0.000000] NR_IRQS: 51
<6>[    0.000000] CPU clock: 550.000 MHz
<6>[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6950037990 ns
<6>[    0.000011] sched_clock: 32 bits at 275MHz, resolution 3ns, wraps every 7809031678ns
<6>[    0.008266] Calibrating delay loop... 366.18 BogoMIPS (lpj=1830912)
<6>[    0.074841] pid_max: default: 32768 minimum: 301
<6>[    0.079941] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
<6>[    0.086934] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
<6>[    0.099451] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
<6>[    0.109877] futex hash table entries: 256 (order: -1, 3072 bytes)
<6>[    0.116413] pinctrl core: initialized pinctrl subsystem
<6>[    0.123115] NET: Registered protocol family 16
<6>[    0.152445] clocksource: Switched to clocksource MIPS
<6>[    0.159069] NET: Registered protocol family 2
<6>[    0.164740] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
<6>[    0.172113] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
<6>[    0.178895] TCP: Hash tables configured (established 1024 bind 1024)
<6>[    0.185796] UDP hash table entries: 256 (order: 0, 4096 bytes)
<6>[    0.191975] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
<6>[    0.198939] NET: Registered protocol family 1
<7>[    0.203625] PCI: CLS 0 bytes, default 32
<4>[    0.209116] Crashlog allocated RAM at address 0x1f00000
<6>[    0.216654] workingset: timestamp_bits=30 max_order=13 bucket_order=0
<6>[    0.229890] squashfs: version 4.0 (2009/01/31) Phillip Lougher
<6>[    0.236100] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
<6>[    0.258107] io scheduler noop registered
<6>[    0.262249] io scheduler deadline registered (default)
<6>[    0.270773] pinctrl-single 1804002c.pinmux: 576 pins at pa b804002c size 72
<6>[    0.279362] Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
<6>[    0.287028] console [ttyS0] disabled
<6>[    0.290856] 18020000.uart: ttyS0 at MMIO 0x18020000 (irq = 9, base_baud = 1562500) is a 16550A
<6>[    0.300026] console [ttyS0] enabled
<6>[    0.307588] bootconsole [early0] disabled
<6>[    0.335074] m25p80 spi0.0: w25q32 (4096 Kbytes)
<5>[    0.339819] 3 fixed-partitions partitions found on MTD device spi0.0
<5>[    0.346411] Creating 3 MTD partitions on "spi0.0":
<5>[    0.351378] 0x000000000000-0x000000020000 : "u-boot"
<5>[    0.357577] 0x000000020000-0x0000003f0000 : "firmware"
<5>[    0.364139] 2 tplink-fw partitions found on MTD device firmware
<5>[    0.370261] Creating 2 MTD partitions on "firmware":
<5>[    0.375472] 0x000000000000-0x0000001438f6 : "kernel"
<5>[    0.381536] 0x0000001438f8-0x0000003d0000 : "rootfs"
<5>[    0.387623] mtd: device 3 (rootfs) set to be root filesystem
<5>[    0.393596] 1 squashfs-split partitions found on MTD device rootfs
<5>[    0.399995] 0x000000370000-0x0000003d0000 : "rootfs_data"
<5>[    0.406508] 0x0000003f0000-0x000000400000 : "art"
<6>[    0.413186] libphy: Fixed MDIO Bus: probed
<3>[    0.753580] ag71xx 19000000.eth: Could not connect to PHY device. Deferring probe.
<6>[    1.432891] libphy: ag71xx_mdio: probed
<6>[    1.438427] libphy: ar8xxx-mdio: probed
<6>[    1.451985] switch0: Atheros AR8229 rev. 1 switch registered on mdio-bus.0
<6>[    1.504376] ag71xx 1a000000.eth: connected to PHY at fixed-0:00 [uid=00000000, driver=Generic PHY]
<6>[    1.514537] eth0: Atheros AG71xx at 0xba000000, irq 5, mode: gmii
<6>[    1.524220] NET: Registered protocol family 10
<6>[    1.536076] Segment Routing with IPv6
<6>[    1.539991] NET: Registered protocol family 17
<6>[    1.545477] 8021q: 802.1Q VLAN Support v1.8
<6>[    1.884807] ag71xx 19000000.eth: connected to PHY at mdio-bus.0:1f:04 [uid=004dd042, driver=Generic PHY]
<6>[    1.895813] eth1: Atheros AG71xx at 0xb9000000, irq 4, mode: mii
<6>[    1.903143] hctosys: unable to open rtc device (rtc0)
<6>[    1.914565] VFS: Mounted root (squashfs filesystem) readonly on device 31:3.
<6>[    1.929876] Freeing unused kernel memory: 1180K
<4>[    1.934589] This architecture does not have kernel memory protection.
<5>[    2.542471] random: fast init done
<14>[    3.981320] init: Console is alive
<14>[    3.985252] init: - watchdog -
<14>[    5.226245] kmodloader: loading kernel modules from /etc/modules-boot.d/*
<14>[    5.448312] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
<14>[    5.466446] init: - preinit -
<5>[    7.317162] random: jshn: uninitialized urandom read (4 bytes read)
<5>[    7.640937] random: jshn: uninitialized urandom read (4 bytes read)
<5>[    7.881979] random: jshn: uninitialized urandom read (4 bytes read)
<6>[    9.350148] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
<6>[    9.356368] IPv6: ADDRCONF(NETDEV_UP): eth0.1: link is not ready
<4>[    9.457494] urandom_read: 4 callbacks suppressed
<5>[    9.457505] random: procd: uninitialized urandom read (4 bytes read)
<6>[   10.413685] eth0: link up (1000Mbps/Full duplex)
<6>[   10.418503] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
<6>[   10.425640] IPv6: ADDRCONF(NETDEV_CHANGE): eth0.1: link becomes ready
<5>[   13.204259] jffs2: notice: (387) jffs2_build_xattr_subsystem: complete building xattr subsystem, 46 of xdatum (2 unchecked, 44 orphan) and 56 of xref (44 dead, 0 orphan) found.
<14>[   13.222309] mount_root: switching to jffs2 overlay
<4>[   13.256999] overlayfs: upper fs does not support tmpfile.
<12>[   13.270584] urandom-seed: Seeding with /etc/urandom.seed
<6>[   13.606629] eth0: link down
<14>[   13.635329] procd: - early -
<14>[   13.638443] procd: - watchdog -
<14>[   14.415246] procd: - watchdog -
<14>[   14.418916] procd: - ubus -
<5>[   14.862014] random: ubusd: uninitialized urandom read (4 bytes read)
<5>[   14.870949] random: ubusd: uninitialized urandom read (4 bytes read)
<14>[   14.879269] procd: - init -
<14>[   16.496317] kmodloader: loading kernel modules from /etc/modules.d/*
<6>[   16.515274] tun: Universal TUN/TAP device driver, 1.6
<6>[   16.539819] ip6_tables: (C) 2000-2006 Netfilter Core Team
<6>[   16.570947] Loading modules backported from Linux version v4.19.137-0-gc076c79e03c6
<6>[   16.578942] Backport generated by backports.git v4.19.137-1-0-g60c3a249
<6>[   16.604123] ip_tables: (C) 2000-2006 Netfilter Core Team
<6>[   16.632886] nf_conntrack version 0.5.0 (1024 buckets, 4096 max)
<6>[   16.798269] xt_time: kernel timezone is -0000
<7>[   17.003259] ath: EEPROM regdomain: 0x0
<7>[   17.003272] ath: EEPROM indicates default country code should be used
<7>[   17.003276] ath: doing EEPROM country->regdmn map search
<7>[   17.003294] ath: country maps to regdmn code: 0x3a
<7>[   17.003301] ath: Country alpha2 being used: US
<7>[   17.003306] ath: Regpair used: 0x3a
<7>[   17.017937] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
<6>[   17.020235] ieee80211 phy0: Atheros AR9531 Rev:1 mem=0xb8100000, irq=12
<14>[   17.372997] kmodloader: done loading kernel modules from /etc/modules.d/*
<14>[   17.626827] urngd: v1.0.2 started.
<5>[   18.145367] random: crng init done
<6>[   38.208915] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
<6>[   38.216318] eth0: link up (1000Mbps/Full duplex)
<6>[   38.221130] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
<6>[   38.268978] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
<6>[   38.304912] IPv6: ADDRCONF(NETDEV_UP): br-vap: link is not ready
<6>[   38.354025] IPv6: ADDRCONF(NETDEV_UP): br-roam: link is not ready
<7>[   40.476601] ath: EEPROM regdomain: 0x8114
<7>[   40.476615] ath: EEPROM indicates we should expect a country code
<7>[   40.476621] ath: doing EEPROM country->regdmn map search
<7>[   40.476627] ath: country maps to regdmn code: 0x37
<7>[   40.476634] ath: Country alpha2 being used: DE
<7>[   40.476639] ath: Regpair used: 0x37
<7>[   40.476646] ath: regdomain 0x8114 dynamically updated by user
<6>[   45.401625] IPv6: ADDRCONF(NETDEV_UP): wlan0-1: link is not ready
<6>[   45.414072] br-vap: port 1(wlan0-1) entered blocking state
<6>[   45.419792] br-vap: port 1(wlan0-1) entered disabled state
<6>[   45.425965] device wlan0-1 entered promiscuous mode
<6>[   45.747193] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0-1: link becomes ready
<6>[   45.754274] br-vap: port 1(wlan0-1) entered blocking state
<6>[   45.759954] br-vap: port 1(wlan0-1) entered forwarding state
<6>[   45.778367] br-roam: port 1(wlan0-2) entered blocking state
<6>[   45.784586] br-roam: port 1(wlan0-2) entered disabled state
<6>[   45.790829] device wlan0-2 entered promiscuous mode
<6>[   45.802728] IPv6: ADDRCONF(NETDEV_CHANGE): br-vap: link becomes ready
<6>[   45.827453] IPv6: ADDRCONF(NETDEV_UP): wlan0-2: link is not ready
<6>[   45.834186] br-roam: port 1(wlan0-2) entered blocking state
<6>[   45.839980] br-roam: port 1(wlan0-2) entered forwarding state
<6>[   45.847573] IPv6: ADDRCONF(NETDEV_CHANGE): br-roam: link becomes ready
<6>[   45.908943] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0-2: link becomes ready
<6>[   46.051074] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
<6>[   46.469277] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
<4>[   98.092507] ------------[ cut here ]------------
<4>[   98.097311] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 0x802bf250
<6>[   98.104640] NETDEV WATCHDOG: eth0 (ag71xx): transmit queue 0 timed out
<0>[   98.111390] Kernel panic - not syncing: panic_on_warn set ...
<0>[   98.111390]

# swconfig dev switch0 show
Global attributes:
  enable_vlan: 1
  ar8xxx_mib_poll_interval: 500
  ar8xxx_mib_type: 0
  enable_mirror_rx: 0
  enable_mirror_tx: 0
  mirror_monitor_port: 0
  mirror_source_port: 0
  arl_table: address resolution table
Port 0: MAC c4:6e:1f:31:64:fa

Port 0:
  mib: MIB counters
RxGoodByte  : 515088 (503.0 KiB)
TxByte      : 0

  pvid: 0
  link: port:0 link:up speed:1000baseT full-duplex txflow rxflow 
Port 1:
  mib: No MIB data
  pvid: 1
  link: port:1 link:down
Port 2:
  mib: No MIB data
  pvid: 1
  link: port:2 link:down
Port 3:
  mib: No MIB data
  pvid: 1
  link: port:3 link:down
Port 4:
  mib: No MIB data
  pvid: 1
  link: port:4 link:down
VLAN 1:
  vid: 1
  ports: 0t 1 2 3 4