Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2157 - Hardware/software offload (HNAT) problem on MT7621 #7123

Closed
openwrt-bot opened this issue Mar 1, 2019 · 2 comments
Closed

FS#2157 - Hardware/software offload (HNAT) problem on MT7621 #7123

openwrt-bot opened this issue Mar 1, 2019 · 2 comments
Labels

Comments

@openwrt-bot
Copy link

mmg:

Device : DLink DIR-860L B1
Firmware : OpenWrt SNAPSHOT r9482-b2bf3745ff / LuCI Master (git-19.059.75296-bdfb370)
The firmware is compiled from source. Nothing changed in kernel config. Only 4-5 apps are installed on this firmware.
This problem started before 2-3 months... Maybe...

The problem is :
Random kernel panic. I think this is software/hardware offload problem. This option is checked for me.

Errors:

[47740.687415] CPU 1 Unable to handle kernel paging request at virtual address 00486f08, epc == 868d8f94, ra == 868d92f4
[47740.708574] Oops[#1]:
[47740.713088] CPU: 1 PID: 164 Comm: kworker/1:1 Not tainted 4.14.103 #0
[47740.725914] Workqueue: events_power_efficient nf_flow_dnat_port [nf_flow_table]
[47740.740456] task: 87d8c4c0 task.stack: 87edc000
[47740.749459] $ 0 : 00000000 00000001 fffffff5 00000000
[47740.759868] $ 4 : 87edde10 0000000f 00000000 ffff00fe
[47740.767415] CPU 3 Unable to handle kernel paging request at virtual address 1a0067b0, epc == 868d8eb0, ra == 868d8ed8
[47740.770282] $ 8 : 87eddfe0 00007c00 00002b6b 0016aba7
[47740.801777] $12 : 00000000 00000000 ffffffff 00002636
[47740.812180] $16 : 86f1eae0 00486eda 81123a00 86f1ea78
[47740.822581] $20 : 00000000 00000000 80580000 8059f3c0
[47740.832983] $24 : 00000000 8000cf94
[47740.843383] $28 : 87edc000 87edddf8 868e0000 868d92f4
[47740.853783] Hi : 0000000a
[47740.859504] Lo : 66666669
[47740.865249] epc : 868d8f94 nf_flow_table_iterate+0x13c/0x7b0 [nf_flow_table]
[47740.879618] ra : 868d92f4 nf_flow_table_iterate+0x49c/0x7b0 [nf_flow_table]
[47740.893979] Status: 11007c03 KERNEL EXL IE
[47740.902304] Cause : 40800008 (ExcCode 02)
[47740.910271] BadVA : 00486f08
[47740.915987] PrId : 0001992f (MIPS 1004Kc)
[47740.924120] Modules linked in: pppoe ppp_async pppox ppp_generic mt76x2e mt76x2_common mt76x02_lib mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv4 nf_conntrack_rtcache iptable_raw iptable_mangle iptable_filter ipt_ECN ip6table_raw ip_tables crc_ccitt compat act_connmark cls_matchall act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw cls_basic
[47741.065550] sch_tbf sch_htb sch_hfsc sch_ingress ledtrig_usbport cryptodev ip6t_NPT ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables ifb tun sha512_generic sha256_generic seqiv jitterentropy_rng drbg hmac ghash_generic gf128mul gcm ctr authenc leds_gpio xhci_mtk xhci_plat_hcd xhci_pci xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[47741.155777] Process kworker/1:1 (pid: 164, threadinfo=87edc000, task=87d8c4c0, tls=00000000)
[47741.172568] Stack : 8057e0c4 00002b6b 805d9d80 8057e1e0 81123d80 80580000 86f1ea80 00486eda
[47741.189215] 0000001b 00000100 00000200 863d8f00 0000000f 00000001 86f1eae0 87e94300
[47741.205849] 81123a00 81126b00 00000000 00000000 80580000 fffffffe 80580000 868d97c0
[47741.222485] 81123a00 81126700 00000000 86f1eae0 86f1eae0 80049f70 81123ba0 81123a18
[47741.239119] 80580000 81123ba0 80580000 fffffffe 87e94300 81123a00 87e94318 81123a18
[47741.255753] ...
[47741.260640] Call Trace:
[47741.265518] [<868d8f94>] nf_flow_table_iterate+0x13c/0x7b0 [nf_flow_table]
[47741.279203] Code: 00000000 100000d7 00000000 <9222002e> 144000d4 00000000 8e220078 3043000c 146000dd
[47741.298611]
[47741.301571] Oops[#2]:
[47741.302112] ---[ end trace 0051ff771cfd3267 ]---
[47741.306111] CPU: 3 PID: 69 Comm: kworker/3:1 Tainted: G D 4.14.103 #0

OR

[29808.595819] CPU 3 Unable to handle kernel paging request at virtual address 002cf5a0, epc == 86f02658, ra == 86f01cf8
[29808.616975] Oops[#1]:
[29808.621504] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.14.103 #0
[29808.633631] task: 87c3cb00 task.stack: 87c66000
[29808.642640] $ 0 : 00000000 00000001 00000100 001fff00
[29808.653046] $ 4 : 002cf5a0 805e0000 00000001 00000001
[29808.663454] $ 8 : 870f705c 8045dd68 d194b1ac 88000000
[29808.673874] $12 : 87c15f68 00000008 3b9aca00 00000000
[29808.684315] $16 : 870f7000 87c15c7c 86f10000 fffffff4
[29808.694741] $20 : 00000741 86f10000 86f10000 e82d16b9
[29808.705156] $24 : 513b7555 86f025b4
[29808.715566] $28 : 87c66000 87c15bd0 80594dc0 86f01cf8
[29808.725975] Hi : 00000741
[29808.731697] Lo : 40bd3800
[29808.737480] epc : 86f02658 nf_conntrack_free+0x124/0x468 [nf_conntrack]
[29808.751001] ra : 86f01cf8 __nf_ct_refresh_acct+0x1d4/0x538 [nf_conntrack]
[29808.765019] Status: 11007c03 KERNEL EXL IE
[29808.773348] Cause : 40800008 (ExcCode 02)
[29808.781318] BadVA : 002cf5a0
[29808.787042] PrId : 0001992f (MIPS 1004Kc)
[29808.795179] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt compat ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 tun leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[29808.934643] Process swapper/3 (pid: 0, threadinfo=87c66000, task=87c3cb00, tls=00000000)
[29808.950745] Stack : 876f4000 805741e0 00000003 00000001 870f703c 86f01cf8 00000000 803426cc
[29808.967391] 00000310 86f10000 87c15c7c 80570000 805741e0 86f0b3c0 87c15c9c 00000000
[29808.984031] 861e43c0 00000014 00000001 e82d16b9 80594dc0 86f0d1e4 00000002 00000002
[29809.000685] 8689ef20 86f02c4c 80590000 86aeb914 fffffff0 01080020 00000011 80594dc0
[29809.017342] 87c15c7c 8689ef20 86f0d1e4 00000000 00000001 00000000 805741e0 80570000
[29809.033989] ...
[29809.038880] Call Trace:
[29809.043781] [<86f02658>] nf_conntrack_free+0x124/0x468 [nf_conntrack]
[29809.056627] [<86f01cf8>] __nf_ct_refresh_acct+0x1d4/0x538 [nf_conntrack]
[29809.069983] Code: 1080000e 8fbf0014 0000000f 2462ffff e0820000 1040fffc 00000000 0000000f
[29809.089401]
[29809.092692] ---[ end trace fb4fb8203658d1b6 ]---

OR

[46765.897271] CPU 3 Unable to handle kernel paging request at virtual address 1a001629, epc == 868c1dbc, ra == 868c2c4c
[46765.918516] Oops[#1]:
[46765.923062] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.14.103 #0
[46765.935204] task: 87c3cb00 task.stack: 87c66000
[46765.944220] $ 0 : 00000000 00000001 80570000 000000d2
[46765.954635] $ 4 : 0046c906 0046e6ca 8f37ab15 87c15c94
[46765.965060] $ 8 : 87c15c94 8045dd68 b83893d3 00007e55
[46765.975495] $12 : 0000256e 8048f620 00000001 00000000
[46765.985910] $16 : 1a0015fa 87c15c7c 868d0000 fffffff4
[46765.996323] $20 : 00000479 868d0000 868d0000 8f37ab15
[46766.006739] $24 : c0a801c8 868c9ed0
[46766.017163] $28 : 87c66000 87c15be8 80594dc0 868c2c4c
[46766.027581] Hi : 00000479
[46766.033310] Lo : bd58a800
[46766.039108] epc : 868c1dbc __nf_ct_refresh_acct+0x298/0x538 [nf_conntrack]
[46766.053149] ra : 868c2c4c nf_conntrack_in+0x2b0/0x5f8 [nf_conntrack]
[46766.066303] Status: 11007c03 KERNEL EXL IE
[46766.074627] Cause : 40800008 (ExcCode 02)
[46766.082586] BadVA : 1a001629
[46766.088333] PrId : 0001992f (MIPS 1004Kc)
[46766.096501] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_mangle iptable_filter ip_tables crc_ccitt compat ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 tun leds_gpio xhci_plat_hcd xhci_pci xhci_mtk xhci_hcd gpio_button_hotplug usbcore nls_base usb_common
[46766.236063] Process swapper/3 (pid: 0, threadinfo=87c66000, task=87c3cb00, tls=00000000)
[46766.252168] Stack : 00000000 803426cc 00000719 868d0000 87c15c7c 80570000 805741e0 868cb3c0
[46766.268829] 838096c0 00000000 83809180 00000014 00000001 8f37ab15 80594dc0 868cd1e4
[46766.285494] 00000002 00000002 86fbcf20 868c2c4c 80590000 86119574 fffffff0 01080020
[46766.302150] 00000011 80594dc0 87c15c7c 86fbcf20 868cd1e4 8689c300 00000001 00000000
[46766.318809] 805741e0 80570000 83809180 11570000 00000014 c801a8c0 00000000 00000000
[46766.335484] ...
[46766.340376] Call Trace:
[46766.345284] [<868c1dbc>] __nf_ct_refresh_acct+0x298/0x538 [nf_conntrack]
[46766.358679] Code: 32020001 1440000e 8fa20014 <9203002f> 00032040 00832021 00042100 02642023 02042021
[46766.378141]
[46766.381327] ---[ end trace 8a6592eb0b1450bd ]---
[46766.394015] Kernel panic - not syncing: Fatal exception in interrupt

How you can reproduce this problem? This is not easy. :)
My way is :
With many connections. How? Torrent client and torrent with 3000-4000 peers. You can start download and after 5 minutes you can stop. Repeat this 3-4 times. After this you must change the torrent and you need repeat 3-4 times the same "download" procedure. Between every "download" you can wait 15-20-25 minutes. After 1-2-3-4 hours you will have reboot of your router and you can see the kernel panic in your logs(I use external log server).

@openwrt-bot
Copy link
Author

mmg:

Same problem here : https://bugs.openwrt.org/index.php?do=details&task_id=1917

And here : https://forum.openwrt.org/t/mt7621-offloading-reboots/24043

But I don't test only SW offload.

@openwrt-bot
Copy link
Author

mmg:

Without SW and HW offload the router is stable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant