OpenWrt/LEDE Project

  • Status Closed
  • Percent Complete
    100%
  • Task Type Bug Report
  • Category Kernel
  • Assigned To No-one
  • Operating System All
  • Severity High
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Tony Ambardar - 24.08.2019
Last edited by Petr Štetiar - 16.09.2019

FS#2461 - Kernel BUG/Oops on tc qdisc delete with tc filter action xt -j CONNMARK

Overview

The kernel module act_ipt enables traffic control filter actions to directly use iptables targets. This is very useful for packet mangling in general, and for QOS or SQM work specifically.

Several xtables targets work as expected with “tc filter ... action xt”, allowing setup, use and teardown: e.g. xt_DSCP and xt_CLASSIFY.

With xt_CONNMARK however, trying to delete the qdisc/filter results in a kernel oops or hang/crash on all platforms and kernel versions tested.

This bug is also being tracked upstream as:
https://bugzilla.kernel.org/show_bug.cgi?id=204681

Steps to Reproduce

tc qdisc add dev lo clsact
tc filter add dev lo egress protocol ip matchall \
  action xt -j CONNMARK --save-mark
tc qdisc del dev lo clsact
# Kernel Oops Here

Systems Tested

  • OpenWrt master, r10666-fc5d46dc62 (kernel 4.19.62, QEMU/mips32_be, iptables 1.8.3, iproute2 5.0.0)
  • OpenWrt 19.07, r10324-8bf8de95a2 (kernel 4.14.133, EA6350v3/armv7l, iptables 1.8.3, iproute2 5.0.0)
  • Ubuntu 18.04 LTS (mainline kernel 5.2.6/x86_64, iptables 1.6.1, iproute2 4.15)
  • Ubuntu 18.04 LTS (distro kernel 4.15/x86_64, iptables 1.6.1, iproute2 4.15)

Some Kernel Logs

Openwrt master, kernel 4.19.62:

[  337.217664] CPU 0 Unable to handle kernel paging request at virtual address 00000e1c, epc == 8ec28c7c, ra == 8f3622a8
[  337.219536] Oops[#1]:
[  337.220138] CPU: 0 PID: 4023 Comm: kworker/u2:0 Not tainted 4.19.62 #0
[  337.221831] Workqueue: tc_filter_workqueue 0x8f1fc9e4
[  337.222761] $ 0   : 00000000 00000001 00000008 8ed86054
[  337.223271] $ 4   : 00000000 00000002 00000000 00000000
[  337.223694] $ 8   : fffffffd 8000001f 00000114 00000001
[  337.224087] $12   : ffffffe0 00000040 8f02c380 77f322a0
[  337.224515] $16   : 8f2b7f00 00000002 00000000 00000001
[  337.224909] $20   : 8f02c180 00000000 00000040 fffffffe
[  337.225286] $24   : 00000000 8015f138
[  337.225777] $28   : 8f33c000 8f33dd40 807e0000 8f3622a8
[  337.226224] Hi    : 00000000
[  337.226468] Lo    : 0002a400
[  337.227140] epc   : 8ec28c7c nf_ct_l4proto_put+0x49c/0x714 [nf_conntrack]
[  337.227656] ra    : 8f3622a8 0x8f3622a8
[  337.227974] Status: 1000a403 KERNEL EXL IE
[  337.228452] Cause : 00800008 (ExcCode 02)
[  337.228834] BadVA : 00000e1c
[  337.229095] PrId  : 00019300 (MIPS 24Kc)
[  337.229414] Modules linked in: xt_recent xt_helper xt_connmark xt_connlimit xt_connbytes nf_conncount iptable_raw sch_teql sch_sfq sch_red sch_prio sch_pie sch_multiq sch_gred sch_fq sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp act_simple act_police act_pedit act_ipt act_gact act_csum libcrc32c sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred pppoe ppp_async iptable_nat ipt_MASQUERADE xt_state xt_nat xt_conntrack xt_REDIRECT xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat_ipv4 nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack mac80211_hwsim mac80211 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_multiport xt_mark xt_mac xt_limit xt_comment xt_TCPMSS xt_LOG slhc pcnet32 nf_reject_ipv4 nf_log_ipv4
[  337.233510]  nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_filter ip_tables crc_ccitt compat nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 mii
[  337.234740] Process kworker/u2:0 (pid: 4023, threadinfo=6a833f9b, task=cca42c7d, tls=00000000)
[  337.235232] Stack : 8f6adab0 00000000 00000001 8f0d5904 8f6adac8 8f2b7f00 8f0d5900 00000001
[  337.235757]         00000001 8f3622a8 8f02c180 00000000 00000040 8fbb9980 00000000 8ed865b8
[  337.236206]         8f02c320 02000000 8f02c180 8f3622e4 00000040 80688dac 8f2b1e00 00000000
[  337.236645]         8f2b7f00 80582a70 8f6adab0 8f6adac8 8f2b7f00 8f0d5900 8f2b7f00 80582b24
[  337.237101]         8f965b80 8fa5abf0 8124b900 807e0000 8f02c104 00000001 8f362dc8 80582ba0
[  337.237538]         ...
[  337.237913] Call Trace:
[  337.238099] [<8ec28c7c>] nf_ct_l4proto_put+0x49c/0x714 [nf_conntrack]
[  337.238956] Code: 00809025  8c4209d4  00a08825 <8c830e1c> 00021080  00431021  8c530000  3c108ec3  0c1a68c8
[  337.239632]
[  337.240093] ---[ end trace 3d0a314b99232ef5 ]---
[  337.242908] Kernel panic - not syncing: Fatal exception
[  337.244423] Rebooting in 3 seconds..

Ubuntu 18.04 LTS, kernel 5.2.6:

[129936.003975] u32 classifier
[129936.003976]     Actions configured
[130653.065153] BUG: unable to handle page fault for address: 00000000000013d0
[130653.065161] #PF: supervisor read access in kernel mode
[130653.065164] #PF: error_code(0x0000) - not-present page
[130653.065168] PGD 0 P4D 0
[130653.065175] Oops: 0000 [#1] SMP PTI
[130653.065182] CPU: 2 PID: 7545 Comm: kworker/u16:1 Not tainted 5.2.6-050206-generic #201908040832
[130653.065185] Hardware name: Gigabyte Technology Co., Ltd. P55M-UD2/P55M-UD2, BIOS F6 11/23/2009
[130653.065195] Workqueue: tc_filter_workqueue u32_delete_key_freepf_work [cls_u32]
[130653.065211] RIP: 0010:nf_ct_netns_do_put+0x19/0xa0 [nf_conntrack]
[130653.065215] Code: e8 cc cc ff ff 5d c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 8b 05 f8 f9 00 00 48 89 e5 41 55 41 54 49 89 fc 53 89 f3 <48> 8b 97 d0 13 00 00 4c 8b 2c c2 48 c7 c7 40 fe 89 c0 e8 50 76 9e
[130653.065218] RSP: 0018:ffffaea2c3393ce8 EFLAGS: 00010202
[130653.065221] RAX: 0000000000000007 RBX: 0000000000000002 RCX: 0000000000000000
[130653.065224] RDX: ffffffffc079a060 RSI: 0000000000000002 RDI: 0000000000000000
[130653.065227] RBP: ffffaea2c3393d00 R08: ffff957a848266c8 R09: ffff957a848266f8
[130653.065229] R10: ffffaea2c3393d60 R11: ffff957b932e9f68 R12: 0000000000000000
[130653.065231] R13: ffff957b932e9f40 R14: ffffffffc0785080 R15: ffff957b8f828600
[130653.065235] FS:  0000000000000000(0000) GS:ffff957b97a80000(0000) knlGS:0000000000000000
[130653.065238] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[130653.065240] CR2: 00000000000013d0 CR3: 00000000c320a000 CR4: 00000000000006e0
[130653.065243] Call Trace:
[130653.065258]  nf_ct_netns_put+0x1c/0x40 [nf_conntrack]
[130653.065265]  ? __radix_tree_delete+0x8f/0xf0
[130653.065270]  connmark_tg_destroy+0x15/0x20 [xt_connmark]
[130653.065275]  ipt_destroy_target+0x52/0x80 [act_ipt]
[130653.065280]  tcf_ipt_release+0x1e/0x40 [act_ipt]
[130653.065285]  tcf_action_cleanup+0x21/0xa0
[130653.065288]  __tcf_action_put+0x62/0x70
[130653.065292]  __tcf_idr_release+0x2f/0x40
[130653.065295]  tcf_action_destroy+0x56/0x80
[130653.065301]  tcf_exts_destroy+0x1b/0x30
[130653.065305]  u32_destroy_key+0x20/0x80 [cls_u32]
[130653.065310]  u32_delete_key_freepf_work+0x20/0x30 [cls_u32]
[130653.065315]  process_one_work+0x1db/0x3c0
[130653.065318]  worker_thread+0x4d/0x400
[130653.065324]  kthread+0x104/0x140
[130653.065328]  ? process_one_work+0x3c0/0x3c0
[130653.065332]  ? kthread_park+0x80/0x80
[130653.065337]  ret_from_fork+0x35/0x40
[130653.065340] Modules linked in: act_skbedit cls_u32 xt_connmark sch_ingress xt_DSCP xt_CLASSIFY act_ipt xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bpfilter snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio binfmt_misc snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer intel_powerclamp input_leds kvm_intel kvm snd irqbypass soundcore intel_cstate i7core_edac serio_raw lpc_ich mac_hid sch_fq_codel it87 hwmon_vid coretemp parport_pc ppdev lp parport ip_tables x_tables autofs4 hid_generic usbhid hid firewire_ohci ahci firewire_core r8169 libahci crc_itu_t realtek
[130653.065393] CR2: 00000000000013d0
[130653.065398] ---[ end trace 0d84eb498f4c3f9f ]---
 


Closed by  Petr ┼átetiar
16.09.2019 11:38
Reason for closing:  Fixed
Additional comments about closing:  

Fixed in https:/ /git.openwrt.org/7735cce0c5c306bd9eea20c a2805e4a492c02be9

Tony Ambardar commented on 24.08.2019 07:17

Note that some prerequisite packages are needed to reproduce the problem: i.e.

opkg install tc kmod-sched iptables-mod-conntrack-extra
tc qdisc add dev lo clsact
tc filter add dev lo egress protocol ip matchall \
  action xt -j CONNMARK --save-mark
tc qdisc del dev lo clsact
# Kernel Oops Here
Tony Ambardar commented on 26.08.2019 03:40

There is now an open PR #2356 based on a proposed NetDev patch which has been tested to fix the problem on OpenWrt kernels v4.14.x and v4.19.x.

This should get applied to both openwrt-19.07 and master branches.

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing