Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2353 - CPU: 0 PID: 11550 Comm: kworker/u2:3 Not tainted 4.14.131 #0 / Workqueue: netns cleanup_net #7557

Closed
openwrt-bot opened this issue Jul 3, 2019 · 13 comments
Labels

Comments

@openwrt-bot
Copy link

camel:

Supply the following if possible:

  • Device problem occurs on MT7620 and MT7621 ( zbt 3526 / zbt 826)
  • Software version: trunk LuCI Master (f138fc93) / OpenWrt SNAPSHOT r10374-fdd0a8d491
  • Steps to reproduce: is happening after a while, strange as it was everywhere happening .. maybe it was related to keepalaive ?

installed mwan3 & tried to restart, but funny as all of the routers got this crash message.

Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.490007] ------------[ cut here ]------------
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.494765] WARNING: CPU: 0 PID: 11550 at net/netfilter/core.c:393 __nf_unregister_net_hook+0x84/0x330
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.504344] Modules linked in: rt2800soc rt2800mmio rt2800lib pppoe ppp_async option usb_wwan rt2x00soc rt2x00mmio rt2x00lib pptp pppox ppp_mppe ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_NETMAP xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY usbserial ts_fsm ts_bm slhc nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_rtsp nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_amanda
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.600916] nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtsp nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast ts_kmp nf_conntrack_amanda macvlan iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat br_netfilter sch_cake nf_conntrack sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.698070] ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsv3 nfs ip_gre gre ifb ip_tunnel tun vfat fat lockd sunrpc grace dns_resolver dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax nls_utf8 nls_iso8859_1 nls_cp437 sha1_generic ecb usb_storage sd_mod scsi_mod ext4 mbcache jbd2 crc16 cryptomgr aead crypto_null crc32c_generic crypto_hash mmc_block mtk_sd mmc_core leds_gpio ohci_platform ohci_hcd ehci_platform ehci_hcd gpio_button_hotplug usbcore nls_base usb_common
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.490007] ------------[ cut here ]------------
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.494765] WARNING: CPU: 0 PID: 11550 at net/netfilter/core.c:393 __nf_unregister_net_hook+0x84/0x330
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.504344] Modules linked in: rt2800soc rt2800mmio rt2800lib pppoe ppp_async option usb_wwan rt2x00soc rt2x00mmio rt2x00lib pptp pppox ppp_mppe ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 mt76x2e mt76x2_common mt76x02_lib mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_NETMAP xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY usbserial ts_fsm ts_bm slhc nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_rtsp nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_amanda
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.600916] nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtsp nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast ts_kmp nf_conntrack_amanda macvlan iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat br_netfilter sch_cake nf_conntrack sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.698070] ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsv3 nfs ip_gre gre ifb ip_tunnel tun vfat fat lockd sunrpc grace dns_resolver dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax nls_utf8 nls_iso8859_1 nls_cp437 sha1_generic ecb usb_storage sd_mod scsi_mod ext4 mbcache jbd2 crc16 cryptomgr aead crypto_null crc32c_generic crypto_hash mmc_block mtk_sd mmc_core leds_gpio ohci_platform ohci_hcd ehci_platform ehci_hcd gpio_button_hotplug usbcore nls_base usb_common
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.775586] CPU: 0 PID: 11550 Comm: kworker/u2:3 Not tainted 4.14.131 #0
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.782475] Workqueue: netns cleanup_net
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.786480] Stack : 8030d148 00000189 00000002 8005c208 804a8c98 00000000 00000001 80f76994
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.795046] 804a4c9c 8639dcc4 80500000 8005d0f4 00000002 80f76995 8639dca0 1b26b88a
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.803624] 00000000 00000000 00000000 00011da0 00000000 00000ab2 00000008 00000000
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.812182] 00000000 80500000 000bf08b 00000000 00000000 00000009 00000000 804ccb2c
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.820740] 8030d148 00000189 00000002 802c5d00 00000003 80247508 00000000 80550000
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.829299] ...
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.831797] Call Trace:
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.834315] [<8000f0b0>] show_stack+0x58/0x100
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.838876] [<8002ad90>] __warn+0xe4/0x13c
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.843869] [<8002ae78>] warn_slowpath_null+0x1c/0x34
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.849817] [<8030d148>] __nf_unregister_net_hook+0x84/0x330
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.855616] [<8030d488>] nf_unregister_net_hooks+0x38/0x5c
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.861283] [<802c4bf4>] ops_exit_list.isra.11+0x58/0xa8
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.866709] [<802c5d60>] cleanup_net+0x1ac/0x2d0
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.871461] [<800408b4>] process_one_work+0x26c/0x418
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.876618] [<80040d80>] worker_thread+0x320/0x56c
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.881537] [<80046c64>] kthread+0x134/0x148
Wed Jul 3 15:36:08 2019 kern.warn kernel: [46645.885899] [<80009a98>] ret_from_kernel_thread+0x14/0x1c

@openwrt-bot
Copy link
Author

camel:

any news ?
this is still a showstopper

@openwrt-bot
Copy link
Author

xewonder:

agree!!

This problemis also on taoma W06 latest build.

@openwrt-bot
Copy link
Author

xewonder:

yo!

anyone helping with this or are we back the the old openwrt of supplying firmware that do not work??

I appreciate the work, but not when we get un-usable systems!

Tank you

@openwrt-bot
Copy link
Author

ynezz:

I appreciate the work, but not when we get un-usable systems!

This is development snapshot, some breakage is unavoidable and expected. So until this is fixed you've following options:

  1. Use one of the stable releases
  2. If you want something newer then try to use 19.07 pre-release snapshot images https://downloads.openwrt.org/releases/19.07-SNAPSHOT/targets/
  3. Try to disable config option KERNEL_NET_NS (Network namespace), compile new image and see if it helps

@openwrt-bot
Copy link
Author

yousong:

Likely FS#1472 and this one has the same cause, possibly also FS#2426

FS#2426 is very likely caused by nf_conntrack_rtcache. It's a backport not yet in the upstream.

It's WARN_ONCE and the trigger could be as easy as adding then removing a net namespace. rmmod the module can cause panic.

ip netns add a ip netns del a

Please see if you can confirm this by moving the module /lib/modules/xx/nf_conntrack.ko to another place, reboot, then see if you can still reproduce the issue.

EDIT: It's /lib/modules/xx/nf_conntrack_rtcache.ko

@openwrt-bot
Copy link
Author

camel:

well, if i do that, then it can't load other modules wich seems to be related, and then other services are not working (eg: 4g via ppp)

eg:
Thu Nov 14 06:48:18 2019 daemon.err modprobe: failed to find dependency nf_conntrack
Thu Nov 14 06:48:18 2019 daemon.err modprobe: 1 module could not be probed
Thu Nov 14 06:48:18 2019 daemon.err modprobe: - nf_conntrack_netlink
Thu Nov 14 06:48:18 2019 daemon.err nlbwmon[3484]: Unable to dump conntrack: Invalid argument

eg:
Thu Nov 14 06:43:56 2019 daemon.err modprobe: failed to find dependency nf_conntrack
Thu Nov 14 06:43:56 2019 daemon.err modprobe: 1 module could not be probed
Thu Nov 14 06:43:56 2019 daemon.err modprobe: - xt_conntrack

so, testing without nf_conntrack.ko, is not really possible ... as many other services are crying for it ...

@openwrt-bot
Copy link
Author

yousong:

It's nf_conntrack_rtcache.ko, not nf_conntrack.ko

@openwrt-bot
Copy link
Author

yousong:

note that the way i understand it, the warning should be mostly harmless. The part I don't yet explore is how can it be triggered by restarting mwan3.

@openwrt-bot
Copy link
Author

camel:

i will test ...

@openwrt-bot
Copy link
Author

yousong:

Patch just applied: http://patchwork.ozlabs.org/patch/1194403/

@openwrt-bot
Copy link
Author

camel:

hmm, how can i find out that it is also commited and built in the newest trunks ?
or is that only a private version ?

@openwrt-bot
Copy link
Author

bjonglez:

This should be fixed in 19.07.0-rc2 and in the upcoming 18.06.6. Can you test and report if it works?

@openwrt-bot
Copy link
Author

camel:

2 months later ?
I'm on current trunk.
(git-20.004.33955-4c4d6e2) / OpenWrt SNAPSHOT r11890-e98e046f06

well, i guess, it is defintive not happening that often as before ...
fix seems to be something was in a good direction.

i would recommend to close that ticket.
if it happens again, then i will open anew ticket.

thx cu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant