OpenWrt/LEDE Project

  • Status Closed
  • Percent Complete
    100%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Low
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Polynomdivision - 07.10.2021
Last edited by Ted Hess - 20.10.2021

FS#4067 - ipq40xx: Switch (ar40xx) freezes

The ar40xx on an ipq40xx device freezes from time to time. The issue is hard to reproduce since it was seen on multiple devices on a very different time scale. We run a mesh network with olsr (IPv4) and babeld (IPv6) as routing daemons. The affected devices are mainly Fritz!Box 4040 and Fritz!Box 7530. Typically, on setups with a huge load, the switches begin to freeze. As an example, a church that acts as a central point of our mesh network and connection to the internet experiences the freeze daily. We can easily reproduce and test solutions in that location. Devices with fewer clients and just one mesh connection, also crash but it needs some time (30 days). Some devices with almost no traffic or client do not crash.

As a workaround, we wrote the naywatch daemon, which checks for ipv6 link-local connectivity. We also allow collecting debug output from it. With that, we can show what a diff of the swconfig looks like when a freeze happens. So the switch is already in a frozen state. As you can see the CPU Port 0 is not sending any frames (TX is not visible) to the CPU anymore. The switch still receives frames.

 Port 0:
        mib: Port 0 MIB counters
-RxBroad     : 472793
+RxBroad     : 472882
 RxPause     : 0
-RxMulti     : 36772
+RxMulti     : 36854
 RxFcsErr    : 0
 RxAlignErr  : 0
 RxRunt      : 0
 RxFragment  : 0
-Rx64Byte    : 2075611
-Rx128Byte   : 20964917
-Rx256Byte   : 16459560
-Rx512Byte   : 623700
-Rx1024Byte  : 907303
-Rx1518Byte  : 48003389
+Rx64Byte    : 2075618
+Rx128Byte   : 20964980
+Rx256Byte   : 16459583
+Rx512Byte   : 623752
+Rx1024Byte  : 907344
+Rx1518Byte  : 48003397
 RxMaxByte   : 21422694
 RxTooLong   : 0
-RxGoodByte  : 107718798589
+RxGoodByte  : 107718869396
 RxBadByte   : 0
 RxOverFlow  : 0
 Filtered    : 0

What could be wrong:

  • switch incorrectly configured
  • switch broken?
  • ...

I already wrote with blocktrron and to my understanding, he experienced also a freeze. There is a not published driver DSA implementation of the ar40xx this also has some issues. Maybe, the implementation will help to solve this issue.

The rest of the diff:

root@emma-core:~# diff -u 1633480443-swconfig\ dev\ switch0\ show.log 1633480463-swconfig\ dev\ switch0\ show.log
--- "1633480443-swconfig dev switch0 show.log"  2021-10-06 02:34:03.000000000 +0200
+++ "1633480463-swconfig dev switch0 show.log"  2021-10-06 02:34:23.000000000 +0200
@@ -7,22 +7,22 @@
        linkdown: ???
 Port 0:
        mib: Port 0 MIB counters
-RxBroad     : 472793
+RxBroad     : 472882
 RxPause     : 0
-RxMulti     : 36772
+RxMulti     : 36854
 RxFcsErr    : 0
 RxAlignErr  : 0
 RxRunt      : 0
 RxFragment  : 0
-Rx64Byte    : 2075611
-Rx128Byte   : 20964917
-Rx256Byte   : 16459560
-Rx512Byte   : 623700
-Rx1024Byte  : 907303
-Rx1518Byte  : 48003389
+Rx64Byte    : 2075618
+Rx128Byte   : 20964980
+Rx256Byte   : 16459583
+Rx512Byte   : 623752
+Rx1024Byte  : 907344
+Rx1518Byte  : 48003397
 RxMaxByte   : 21422694
 RxTooLong   : 0
-RxGoodByte  : 107718798589
+RxGoodByte  : 107718869396
 RxBadByte   : 0
 RxOverFlow  : 0
 Filtered    : 0
@@ -51,38 +51,38 @@
        link: port:0 link:up speed:1000baseT full-duplex txflow rxflow
 Port 1:
        mib: Port 1 MIB counters
-RxBroad     : 107158
+RxBroad     : 107267
 RxPause     : 0
-RxMulti     : 1147
+RxMulti     : 1173
 RxFcsErr    : 0
 RxAlignErr  : 0
 RxRunt      : 0
 RxFragment  : 0
-Rx64Byte    : 1536
-Rx128Byte   : 68123
-Rx256Byte   : 20668
-Rx512Byte   : 4912
-Rx1024Byte  : 10327
-Rx1518Byte  : 90128
-RxMaxByte   : 9851
+Rx64Byte    : 1555
+Rx128Byte   : 68180
+Rx256Byte   : 20673
+Rx512Byte   : 4925
+Rx1024Byte  : 10332
+Rx1518Byte  : 90209
+RxMaxByte   : 9860
 RxTooLong   : 0
-RxGoodByte  : 166901240
+RxGoodByte  : 167051151
 RxBadByte   : 0
 RxOverFlow  : 0
-Filtered    : 118
-TxBroad     : 856851
-TxPause     : 1074
-TxMulti     : 89960
+Filtered    : 307
+TxBroad     : 856940
+TxPause     : 3442
+TxMulti     : 90042
 TxUnderRun  : 0
-Tx64Byte    : 38606
-Tx128Byte   : 206906
-Tx256Byte   : 56950
-Tx512Byte   : 27986
-Tx1024Byte  : 60718
-Tx1518Byte  : 677209
+Tx64Byte    : 40978
+Tx128Byte   : 206965
+Tx256Byte   : 56973
+Tx512Byte   : 28038
+Tx1024Byte  : 60759
+Tx1518Byte  : 677217
 TxMaxByte   : 74048
 TxOverSize  : 0
-TxByte      : 1186157213
+TxByte      : 1186378970
 TxCollision : 0
 TxAbortCol  : 0
 TxMultiCol  : 0
@@ -95,38 +95,38 @@
        link: port:1 link:up speed:1000baseT full-duplex txflow rxflow auto
 Port 2:
        mib: Port 2 MIB counters
-RxBroad     : 170588
+RxBroad     : 170832
 RxPause     : 0
-RxMulti     : 11717
+RxMulti     : 11767
 RxFcsErr    : 0
 RxAlignErr  : 0
 RxRunt      : 0
 RxFragment  : 0
-Rx64Byte    : 28452
-Rx128Byte   : 6337895
-Rx256Byte   : 408749
-Rx512Byte   : 54975
-Rx1024Byte  : 62053
-Rx1518Byte  : 343977
-RxMaxByte   : 130630
+Rx64Byte    : 28455
+Rx128Byte   : 6338150
+Rx256Byte   : 408825
+Rx512Byte   : 55001
+Rx1024Byte  : 62066
+Rx1518Byte  : 344149
+RxMaxByte   : 130646
 RxTooLong   : 0
-RxGoodByte  : 1345452080
+RxGoodByte  : 1345783813
 RxBadByte   : 0
 RxOverFlow  : 0
-Filtered    : 574
-TxBroad     : 793647
-TxPause     : 1151
-TxMulti     : 79418
+Filtered    : 1135
+TxBroad     : 793736
+TxPause     : 3519
+TxMulti     : 79500
 TxUnderRun  : 0
-Tx64Byte    : 97711
-Tx128Byte   : 603920
-Tx256Byte   : 173282
-Tx512Byte   : 76156
-Tx1024Byte  : 104686
-Tx1518Byte  : 3582480
+Tx64Byte    : 100079
+Tx128Byte   : 603969
+Tx256Byte   : 173305
+Tx512Byte   : 76208
+Tx1024Byte  : 104727
+Tx1518Byte  : 3582488
 TxMaxByte   : 7391371
 TxOverSize  : 0
-TxByte      : 16528781103
+TxByte      : 16529001614
 TxCollision : 0
 TxAbortCol  : 0
 TxMultiCol  : 0
@@ -139,38 +139,38 @@
        link: port:2 link:up speed:1000baseT full-duplex txflow rxflow auto
 Port 3:
        mib: Port 3 MIB counters
-RxBroad     : 159441
+RxBroad     : 159671
 RxPause     : 0
-RxMulti     : 18188
+RxMulti     : 18257
 RxFcsErr    : 0
 RxAlignErr  : 0
 RxRunt      : 0
 RxFragment  : 0
-Rx64Byte    : 9911
-Rx128Byte   : 3051611
-Rx256Byte   : 711251
-Rx512Byte   : 377985
-Rx1024Byte  : 459307
-Rx1518Byte  : 46523339
-RxMaxByte   : 21133903
+Rx64Byte    : 9932
+Rx128Byte   : 3052044
+Rx256Byte   : 711326
+Rx512Byte   : 378003
+Rx1024Byte  : 459361
+Rx1518Byte  : 46523507
+RxMaxByte   : 21133924
 RxTooLong   : 0
-RxGoodByte  : 100988636934
+RxGoodByte  : 100989010961
 RxBadByte   : 0
 RxOverFlow  : 0
-Filtered    : 36461
-TxBroad     : 804786
-TxPause     : 6016
-TxMulti     : 72948
+Filtered    : 37251
+TxBroad     : 804875
+TxPause     : 8384
+TxMulti     : 73030
 TxUnderRun  : 0
-Tx64Byte    : 1661972
-Tx128Byte   : 18816233
-Tx256Byte   : 15830078
-Tx512Byte   : 276436
-Tx1024Byte  : 524862
-Tx1518Byte  : 3198326
+Tx64Byte    : 1664343
+Tx128Byte   : 18816282
+Tx256Byte   : 15830101
+Tx512Byte   : 276488
+Tx1024Byte  : 524903
+Tx1518Byte  : 3198334
 TxMaxByte   : 426878
 TxOverSize  : 0
-TxByte      : 9485704082
+TxByte      : 9485924799
 TxCollision : 0
 TxAbortCol  : 0
 TxMultiCol  : 0
@@ -202,19 +202,19 @@
 RxBadByte   : 871047744
 RxOverFlow  : 0
 Filtered    : 99
-TxBroad     : 909305
-TxPause     : 4319
-TxMulti     : 67716
+TxBroad     : 909394
+TxPause     : 6687
+TxMulti     : 67798
 TxUnderRun  : 0
-Tx64Byte    : 335196
-Tx128Byte   : 1832541
-Tx256Byte   : 520952
-Tx512Byte   : 320853
-Tx1024Byte  : 412422
-Tx1518Byte  : 42645040
+Tx64Byte    : 337564
+Tx128Byte   : 1832588
+Tx256Byte   : 520975
+Tx512Byte   : 320905
+Tx1024Byte  : 412463
+Tx1518Byte  : 42645048
 TxMaxByte   : 13773617
 TxOverSize  : 0
-TxByte      : 84230298783
+TxByte      : 84230519096
 TxCollision : 0
 TxAbortCol  : 0
 TxMultiCol  : 0
Closed by  Ted Hess
20.10.2021 11:21
Reason for closing:  Duplicate
Polynomdivision commented on 07.10.2021 11:18

This is a duplicate of: https://bugs.openwrt.org/index.php?do=details&task_id=4066 Sorry I thought it was not saved succefully.

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing