OpenWrt/LEDE Project

  • Status Closed
  • Percent Complete
    100%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Medium
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by SirToffski - 15.06.2019
Last edited by Petr Štetiar - 24.06.2019

FS#2321 - Kernel Panic after enabling hardware offloading - EdgeRouter X

  • Device: Ubiquiti EdgeRouter X (ramips mt7621)
  • Software: LuCI Master (f138fc93) / OpenWrt SNAPSHOT r10210-09c6885
  • Steps to reproduce: enable hardware flow offloading

Brief description:
Enabling hardware flow offloading causes kernel oops/panic, resulting in device rebooting. The device remains in a bootloop.
Hard powercycle does not resolve the kernel panic.

Steps taken to resolve the issue:
Powering the device off, disconnecting all rj45 ports then reconnecting a single rj45 cable allows to disable the hardware offload. As the hardware offload is no longer active, the issue stops from happening.

Crash logs:
https://bugs.openwrt.org/index.php?do=details&task_id=1837

Additional information:
The issue was not present prior to this snapshot. Packet capture on a device connected to eth1 port did not reveal anything strange. There was a broadcast message sent from the router to UPD port 4919 stating

Please press button now to enter failsafe

. Presumably this is a normal part of the boot-up process.


Closed by  Petr Štetiar
24.06.2019 16:38
Reason for closing:  Fixed
Additional comments about closing:  

Fixed in https://git.o penwrt.org/?p=openwrt/openwrt.git;a=comm it;h=d344591e72e5ca96a2bf70a2df389615531 85ce8

SirToffski commented on 15.06.2019 02:11

Apologies, the above post had a wrong link for crash logs.

The correct link is: https://gist.github.com/SirToffski/41aa69bf4baca463bb04b9c1f746fad3

Attaching the logs here as well.

Jun 14 05:20:40 _gateway kernel: [  246.730704] CPU 1 Unable to handle kernel paging request at virtual address 000000a4, epc == 803901c8, ra == 8efda024
Jun 14 05:20:40 _gateway kernel: [  246.751874] Oops[#1]:
Jun 14 05:20:40 _gateway kernel: [  246.756403] CPU: 1 PID: 137 Comm: kworker/1:1 Not tainted 4.14.125 #0
Jun 14 05:20:40 _gateway kernel: [  246.769298] Workqueue: events_power_efficient nf_flow_dnat_port [nf_flow_table]
Jun 14 05:20:40 _gateway kernel: [  246.783871] task: 8fe44e00 task.stack: 8feac000
Jun 14 05:20:40 _gateway kernel: [  246.792887] $ 0   : 00000000 00000001 00000200 00000003
Jun 14 05:20:40 _gateway kernel: [  246.803306] $ 4   : 00000000 00000003 8feadd98 00000001
Jun 14 05:20:40 _gateway kernel: [  246.813720] $ 8   : 00000000 00000005 0c269a71 00000002
Jun 14 05:20:40 _gateway kernel: [  246.824146] $12   : 00000000 00000253 7fafc8f4 7fafc8f4
Jun 14 05:20:40 _gateway kernel: [  246.834572] $16   : 00000000 8feadd98 8da62804 8e9bba98
Jun 14 05:20:40 _gateway kernel: [  246.844989] $20   : 8e9b9df8 8da6283c 80600000 00000000
Jun 14 05:20:40 _gateway kernel: [  246.855398] $24   : 77ec1f88 8000cf94
Jun 14 05:20:40 _gateway kernel: [  246.865807] $28   : 8feac000 8feadd68 8ef30000 8efda024
Jun 14 05:20:40 _gateway kernel: [  246.876216] Hi    : 00000099
Jun 14 05:20:40 _gateway kernel: [  246.881934] Lo    : 999999c0
Jun 14 05:20:40 _gateway kernel: [  246.887682] epc   : 803901c8 dev_get_by_index_rcu+0x4/0x5c
Jun 14 05:20:40 _gateway kernel: [  246.898664] ra    : 8efda024 0x8efda024
Jun 14 05:20:40 _gateway kernel: [  246.906298] Status: 11007c03	KERNEL EXL IE
Jun 14 05:20:40 _gateway kernel: [  246.914629] Cause : 40800008 (ExcCode 02)
Jun 14 05:20:40 _gateway kernel: [  246.922595] BadVA : 000000a4
Jun 14 05:20:40 _gateway kernel: [  246.928314] PrId  : 0001992f (MIPS 1004Kc)
Jun 14 05:20:40 _gateway kernel: [  246.936450] Modules linked in: pppoe ppp_async pppox ppp_generic nft_fib_inet nf_nat_pptp nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet nf_conntrack_pptp iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD xt_CT wireguard ts_fsm ts_bm slhc sch_mqprio nft_set_rbtree nft_set_hash nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject_bridge nft_reject nft_redir_ipv6 nft_redir_ipv4 nft_redir nft_quota nft_numgen nft_nat nft_meta_bridge nft_meta nft_masq_ipv6 nft_masq_ipv4 nft_masq nft_log nft_limit nft_fwd_netdev nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_exthdr nft_dup_netdev nft_ct nft_counter nft_chain_route_ipv6 nft_chain_route_ipv4 nft_chain_nat_ipv6
Jun 14 05:20:40 _gateway kernel: [  247.077963]  nft_chain_nat_ipv4 nf_tables_netdev nf_tables_ipv6 nf_tables_ipv4 nf_tables_inet nf_tables_bridge nf_tables_arp nf_tables nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv6 nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_dup_netdev nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conntrack libcrc32c iptable_raw iptable_mangle iptable_filter ip_tables crc_ccitt sch_tbf sch_ingress sch_htb sch_hfsc
Jun 14 05:20:40 _gateway kernel: [  247.219602]  em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ip6_udp_tunnel udp_tunnel leds_gpio gpio_button_hotplug crc32c_generic
Jun 14 05:20:40 _gateway kernel: [  247.323344] Process kworker/1:1 (pid: 137, threadinfo=8feac000, task=8fe44e00, tls=00000000)
Jun 14 05:20:40 _gateway kernel: [  247.340122] Stack : 81263d30 8fc24880 615180f4 c0a8019c 00000000 8da62800 8e99f400 8efda1a4
Jun 14 05:20:40 _gateway kernel: [  247.356769]         8fc04000 8004a714 6a8ac264 00000039 00000000 00000000 00000000 00000000
Jun 14 05:20:40 _gateway kernel: [  247.373413]         00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jun 14 05:20:40 _gateway kernel: [  247.390056]         00000000 00000000 80600000 0000001e 8da62800 8efda3a4 8da2793c 81263980
Jun 14 05:20:40 _gateway kernel: [  247.406701]         0000001e 8da62800 8e99f400 8ef31308 80600000 80600000 805fe0c4 00000009
Jun 14 05:20:40 _gateway kernel: [  247.423346]         ...
Jun 14 05:20:40 _gateway kernel: [  247.428208] Call Trace:
Jun 14 05:20:40 _gateway kernel: [  247.433085] [<803901c8>] dev_get_by_index_rcu+0x4/0x5c
Jun 14 05:20:40 _gateway kernel: [  247.443340] [<8efda024>] 0x8efda024
Jun 14 05:20:40 _gateway kernel: [  247.450278] Code: 03e00008  00000000  30a300ff <8c8200a4> 00031880  00431021  8c420000  1040000e  00000000
Jun 14 05:20:40 _gateway kernel: [  247.469702]
Jun 14 05:20:40 _gateway kernel: [  247.473148] ---[ end trace 6301c6566f43e61d ]---
Jun 14 05:20:40 _gateway kernel: [  247.485047] Kernel panic - not syncing: Fatal exception in interrupt
Jun 14 05:22:22 _gateway logread[783]: Logread connected to 192.168.1.156:514
Jun 14 05:22:22 _gateway uhttpd[1144]: luci: accepted login on / for root from 192.168.1.151
Jun 14 05:22:23 _gateway kernel: [   41.395361] CPU 3 Unable to handle kernel paging request at virtual address 000000a4, epc == 803901c8, ra == 8f0fc024
Jun 14 05:22:23 _gateway kernel: [   41.416531] Oops[#1]:
Jun 14 05:22:23 _gateway kernel: [   41.421061] CPU: 3 PID: 307 Comm: kworker/3:1 Not tainted 4.14.125 #0
Jun 14 05:22:23 _gateway kernel: [   41.433962] Workqueue: events_power_efficient nf_flow_dnat_port [nf_flow_table]
Jun 14 05:22:23 _gateway kernel: [   41.448550] task: 8fe96800 task.stack: 8ff80000
Jun 14 05:22:23 _gateway kernel: [   41.457569] $ 0   : 00000000 00000001 00000200 00000003
Jun 14 05:22:23 _gateway kernel: [   41.467989] $ 4   : 00000000 00000003 8ff81d98 00001102
Jun 14 05:22:23 _gateway kernel: [   41.478413] $ 8   : 00000000 00000005 7e22e3bd 00000002
Jun 14 05:22:23 _gateway kernel: [   41.488832] $12   : 00000000 0041a278 00000000 00000000
Jun 14 05:22:23 _gateway kernel: [   41.499247] $16   : 00000000 8ff81d98 8f7b0b04 8fd6ca98
Jun 14 05:22:23 _gateway kernel: [   41.509681] $20   : 8f7b01bc 8f7b0b3c 80600000 00000000
Jun 14 05:22:23 _gateway kernel: [   41.520101] $24   : 00000000 8000cf94
Jun 14 05:22:23 _gateway kernel: [   41.530525] $28   : 8ff80000 8ff81d68 8f170000 8f0fc024
Jun 14 05:22:23 _gateway kernel: [   41.540964] Hi    : 00000133
Jun 14 05:22:23 _gateway kernel: [   41.546715] Lo    : 33333380
Jun 14 05:22:23 _gateway kernel: [   41.552472] epc   : 803901c8 dev_get_by_index_rcu+0x4/0x5c
Jun 14 05:22:23 _gateway kernel: [   41.563451] ra    : 8f0fc024 0x8f0fc024
Jun 14 05:22:23 _gateway kernel: [   41.571098] Status: 11007c03	KERNEL EXL IE
Jun 14 05:22:23 _gateway kernel: [   41.579442] Cause : 40800008 (ExcCode 02)
Jun 14 05:22:23 _gateway kernel: [   41.587428] BadVA : 000000a4
Jun 14 05:22:23 _gateway kernel: [   41.593159] PrId  : 0001992f (MIPS 1004Kc)
Jun 14 05:22:23 _gateway kernel: [   41.601307] Modules linked in: pppoe ppp_async pppox ppp_generic nft_fib_inet nf_nat_pptp nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet nf_conntrack_pptp iptable_nat ipt_REJECT ipt_MASQUERADE xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_FLOWOFFLOAD xt_CT wireguard ts_fsm ts_bm slhc sch_mqprio nft_set_rbtree nft_set_hash nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject_bridge nft_reject nft_redir_ipv6 nft_redir_ipv4 nft_redir nft_quota nft_numgen nft_nat nft_meta_bridge nft_meta nft_masq_ipv6 nft_masq_ipv4 nft_masq nft_log nft_limit nft_fwd_netdev nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_exthdr nft_dup_netdev nft_ct nft_counter nft_chain_route_ipv6 nft_chain_route_ipv4 nft_chain_nat_ipv6
Jun 14 05:22:23 _gateway kernel: [   41.742857]  nft_chain_nat_ipv4 nf_tables_netdev nf_tables_ipv6 nf_tables_ipv4 nf_tables_inet nf_tables_bridge nf_tables_arp nf_tables nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv6 nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_dup_netdev nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conntrack libcrc32c iptable_raw iptable_mangle iptable_filter ip_tables crc_ccitt sch_tbf sch_ingress sch_htb sch_hfsc
Jun 14 05:22:23 _gateway kernel: [   41.884434]  em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ip6_udp_tunnel udp_tunnel leds_gpio gpio_button_hotplug crc32c_generic
Jun 14 05:22:23 _gateway kernel: [   41.988111] Process kworker/3:1 (pid: 307, threadinfo=8ff80000, task=8fe96800, tls=00000000)
Jun 14 05:22:23 _gateway kernel: [   42.004887] Stack : 8127fd30 8fc24c80 00000000 00000000 00000000 8f7b0b00 8e132400 8f0fc1a4
Admin
Petr Štetiar commented on 15.06.2019 05:50
The issue was not present prior to this snapshot.

Do you know last working version, git hash?

SirToffski commented on 15.06.2019 07:45

Petr,

I have just tested using SNAPSHOT r10168-b8a72dfd28/LuCI Master (f138fc93) Kernel 4.14.123 generated on June 6th 20:30 UTC. The issue is not present.

Enabling hw offloading and rebooting works as expected.

root@OpenWrt:/etc# cat openwrt_release 
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='SNAPSHOT'
DISTRIB_REVISION='r10168-b8a72dfd28'
DISTRIB_TARGET='ramips/mt7621'
DISTRIB_ARCH='mipsel_24kc'
DISTRIB_DESCRIPTION='OpenWrt SNAPSHOT r10168-b8a72dfd28'
DISTRIB_TAINTS=''


root@OpenWrt:~# cat /etc/config/firewall

config defaults
	option syn_flood '1'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option flow_offloading '1'
	option flow_offloading_hw '1'
SirToffski commented on 17.06.2019 07:46

I was able to replicate the issue with June 16th snapshot as well.

  • The sysupbrade image contained exactly the same packages as the last snapshot built known to not have an issue (r10168-b8a72dfd28)
  • The sysupgrade was performed without preserving the configuration. Configuration was re-created manually with enabling HW Offload done as the last step.
  • As soon as the HW Offload was enabled, the router went into a boot loop.

Appears to be a bug introduced in kernel 4.14.125.

Syslogs are saved on my server, I will be able to provide them soon.

Let me know if there is anything else I can do to assist in resolving the issue.

Thanks!

Admin
Petr Štetiar commented on 22.06.2019 04:39

If you're able to compile your own image, then https://github.com/openwrt/openwrt/pull/2153 could be worth a try.

Simon Tretter commented on 23.06.2019 10:30

Kernel panics started after the inclusion of KERNEL_NET_NS=y.

This patch fixes it.

Tested on mt7621 (ER-X):

DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='SNAPSHOT'
DISTRIB_REVISION='r10307-629e6538a1'
DISTRIB_TARGET='ramips/mt7621'
DISTRIB_ARCH='mipsel_24kc'
DISTRIB_DESCRIPTION='OpenWrt SNAPSHOT r10307-629e6538a1'

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing