OpenWrt/LEDE Project

  • Status Closed
  • Percent Complete
    100%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Low
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by camel - 03.07.2019
Last edited by Baptiste Jonglez - 20.01.2020

FS#2353 - CPU: 0 PID: 11550 Comm: kworker/u2:3 Not tainted 4.14.131 #0 / Workqueue: netns cleanup_net

Supply the following if possible:
- Device problem occurs on MT7620 and MT7621 ( zbt 3526 / zbt 826)
- Software version: trunk LuCI Master (f138fc93) / OpenWrt SNAPSHOT r10374-fdd0a8d491
- Steps to reproduce: is happening after a while, strange as it was everywhere happening .. maybe it was related to keepalaive ?

 

installed mwan3 & tried to restart, but funny as all of the routers got this crash message.

Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.490007] ------------[ cut here ]------------
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.494765] WARNING: CPU: 0 PID: 11550 at net/netfilter/core.c:393 __nf_unregister_net_hook+0x84/0x330
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.504344] Modules linked in: rt2800soc rt2800mmio rt2800lib pppoe ppp_async option usb_wwan rt2x00soc rt2x00mmio rt2x00lib pptp pppox ppp_mppe ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_NETMAP xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY usbserial ts_fsm ts_bm slhc nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_rtsp nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_amanda
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.600916]  nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtsp nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast ts_kmp nf_conntrack_amanda macvlan iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat br_netfilter sch_cake nf_conntrack sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.698070]  ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsv3 nfs ip_gre gre ifb ip_tunnel tun vfat fat lockd sunrpc grace dns_resolver dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax nls_utf8 nls_iso8859_1 nls_cp437 sha1_generic ecb usb_storage sd_mod scsi_mod ext4 mbcache jbd2 crc16 cryptomgr aead crypto_null crc32c_generic crypto_hash mmc_block mtk_sd mmc_core leds_gpio ohci_platform ohci_hcd ehci_platform ehci_hcd gpio_button_hotplug usbcore nls_base usb_common
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.490007] ------------[ cut here ]------------
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.494765] WARNING: CPU: 0 PID: 11550 at net/netfilter/core.c:393 __nf_unregister_net_hook+0x84/0x330
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.504344] Modules linked in: rt2800soc rt2800mmio rt2800lib pppoe ppp_async option usb_wwan rt2x00soc rt2x00mmio rt2x00lib pptp pppox ppp_mppe ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_NETMAP xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY usbserial ts_fsm ts_bm slhc nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_rtsp nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_amanda
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.600916]  nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtsp nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast ts_kmp nf_conntrack_amanda macvlan iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat br_netfilter sch_cake nf_conntrack sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.698070]  ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsv3 nfs ip_gre gre ifb ip_tunnel tun vfat fat lockd sunrpc grace dns_resolver dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax nls_utf8 nls_iso8859_1 nls_cp437 sha1_generic ecb usb_storage sd_mod scsi_mod ext4 mbcache jbd2 crc16 cryptomgr aead crypto_null crc32c_generic crypto_hash mmc_block mtk_sd mmc_core leds_gpio ohci_platform ohci_hcd ehci_platform ehci_hcd gpio_button_hotplug usbcore nls_base usb_common
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.775586] CPU: 0 PID: 11550 Comm: kworker/u2:3 Not tainted 4.14.131 #0
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.782475] Workqueue: netns cleanup_net
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.786480] Stack : 8030d148 00000189 00000002 8005c208 804a8c98 00000000 00000001 80f76994
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.795046]         804a4c9c 8639dcc4 80500000 8005d0f4 00000002 80f76995 8639dca0 1b26b88a
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.803624]         00000000 00000000 00000000 00011da0 00000000 00000ab2 00000008 00000000
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.812182]         00000000 80500000 000bf08b 00000000 00000000 00000009 00000000 804ccb2c
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.820740]         8030d148 00000189 00000002 802c5d00 00000003 80247508 00000000 80550000
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.829299]         ...
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.831797] Call Trace:
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.834315] [<8000f0b0>] show_stack+0x58/0x100
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.838876] [<8002ad90>] __warn+0xe4/0x13c
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.843869] [<8002ae78>] warn_slowpath_null+0x1c/0x34
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.849817] [<8030d148>] __nf_unregister_net_hook+0x84/0x330
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.855616] [<8030d488>] nf_unregister_net_hooks+0x38/0x5c
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.861283] [<802c4bf4>] ops_exit_list.isra.11+0x58/0xa8
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.866709] [<802c5d60>] cleanup_net+0x1ac/0x2d0
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.871461] [<800408b4>] process_one_work+0x26c/0x418
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.876618] [<80040d80>] worker_thread+0x320/0x56c
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.881537] [<80046c64>] kthread+0x134/0x148
Wed Jul  3 15:36:08 2019 kern.warn kernel: [46645.885899] [<80009a98>] ret_from_kernel_thread+0x14/0x1c


Closed by  Baptiste Jonglez
20.01.2020 21:32
Reason for closing:  Fixed
Additional comments about closing:  

Fixed in 19.07.0 and 18.06.6

camel commented on 12.09.2019 08:51

any news ?
this is still a showstopper

xewonder commented on 03.10.2019 07:13

agree!!

This problemis also on taoma W06 latest build.

xewonder commented on 03.10.2019 09:16

yo!

anyone helping with this or are we back the the old openwrt of supplying firmware that do not work??

I appreciate the work, but not when we get un-usable systems!

Tank you

Admin
Petr Štetiar commented on 03.10.2019 13:28
I appreciate the work, but not when we get un-usable systems!

This is development snapshot, some breakage is unavoidable and expected. So until this is fixed you've following options:

1. Use one of the stable releases
2. If you want something newer then try to use 19.07 pre-release snapshot images https://downloads.openwrt.org/releases/19.07-SNAPSHOT/targets/ 3. Try to disable config option `KERNEL_NET_NS` (Network namespace), compile new image and see if it helps

Project Manager
Yousong Zhou commented on 13.11.2019 07:36

Likely  FS#1472  and this one has the same cause, possibly also  FS#2426 

 FS#2426  is very likely caused by nf_conntrack_rtcache. It's a backport not yet in the upstream.

It's WARN_ONCE and the trigger could be as easy as adding then removing a net namespace. rmmod the module can cause panic.

ip netns add a
ip netns del a

Please see if you can confirm this by moving the module /lib/modules/xx/nf_conntrack.ko to another place, reboot, then see if you can still reproduce the issue.

EDIT: It's /lib/modules/xx/nf_conntrack_rtcache.ko

camel commented on 14.11.2019 12:50

well, if i do that, then it can't load other modules wich seems to be related, and then other services are not working (eg: 4g via ppp)

eg:
Thu Nov 14 06:48:18 2019 daemon.err modprobe: failed to find dependency nf_conntrack
Thu Nov 14 06:48:18 2019 daemon.err modprobe: 1 module could not be probed
Thu Nov 14 06:48:18 2019 daemon.err modprobe: - nf_conntrack_netlink
Thu Nov 14 06:48:18 2019 daemon.err nlbwmon[3484]: Unable to dump conntrack: Invalid argument

eg:
Thu Nov 14 06:43:56 2019 daemon.err modprobe: failed to find dependency nf_conntrack
Thu Nov 14 06:43:56 2019 daemon.err modprobe: 1 module could not be probed
Thu Nov 14 06:43:56 2019 daemon.err modprobe: - xt_conntrack

so, testing without nf_conntrack.ko, is not really possible ... as many other services are crying for it ...

Project Manager
Yousong Zhou commented on 14.11.2019 12:57

It's nf_conntrack_rtcache.ko, not nf_conntrack.ko

Project Manager
Yousong Zhou commented on 14.11.2019 13:32

note that the way i understand it, the warning should be mostly harmless. The part I don't yet explore is how can it be triggered by restarting mwan3.

camel commented on 17.11.2019 10:25

i will test ...

Project Manager
Yousong Zhou commented on 23.11.2019 10:57
camel commented on 25.11.2019 19:59

hmm, how can i find out that it is also commited and built in the newest trunks ?
or is that only a private version ?

Project Manager
Baptiste Jonglez commented on 02.01.2020 10:59

This should be fixed in 19.07.0-rc2 and in the upcoming 18.06.6. Can you test and report if it works?

camel commented on 20.01.2020 22:22

2 months later ?
I'm on current trunk.
(git-20.004.33955-4c4d6e2) / OpenWrt SNAPSHOT r11890-e98e046f06

well, i guess, it is defintive not happening that often as before ...
fix seems to be something was in a good direction.

i would recommend to close that ticket.
if it happens again, then i will open anew ticket.

thx cu

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing