Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2364 - Recent change cause IPv6 UDP Routing Out Of Memory and Crash #7233

Closed
openwrt-bot opened this issue Jul 8, 2019 · 2 comments
Closed
Labels

Comments

@openwrt-bot
Copy link

gwlim:

Supply the following if possible:

  • Device problem occurs on
  • Software versions of OpenWrt/LEDE release, packages, etc.
  • Steps to reproduce

On WR1043NDv1 Trunk
-Pass IPv6 UDP packets through the router JPerf
-Router goes OOM, crash and reboot
It was fine until yesterday
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.438771] procd invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.449729] CPU: 0 PID: 1 Comm: procd Not tainted 4.19.56 #0
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.455431] Stack : 00200000 ffffffff 00000012 800b6b1c 00000000 00000000 00000000 00000000
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.463859] 00000000 00000000 00000000 00000000 00000000 00000001 83c17b98 96eaea08
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.472278] 83c17c30 00000000 00000000 00003088 00000038 80496d38 00000007 00000000
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.480704] 000000b6 fd7661c4 000000b5 00000000 80530000 83c17b78 80000000 00000003
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.489127] 80531b30 00000001 00200000 ffffffff 00000003 802bc1d4 00000000 80680000
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.497553] ...
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.500021] Call Trace:
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.500037] [<800b6b1c>] 0x800b6b1c
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.506089] [<80496d38>] 0x80496d38
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.509637] [<802bc1d4>] 0x802bc1d4
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.513187] [<8006aec4>] 0x8006aec4
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.516705] [<8006aecc>] 0x8006aecc
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.520261] [<8010ee00>] 0x8010ee00
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.523846] [<8010e1d8>] 0x8010e1d8
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.527366] [<8010e9ac>] 0x8010e9ac
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.530888] [<80113b6c>] 0x80113b6c
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.534514] [<8010b6b8>] 0x8010b6b8
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.538044] [<80137d48>] 0x80137d48
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.541565] [<800c57b8>] 0x800c57b8
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.545124] [<8049cae4>] 0x8049cae4
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.548705] [<80133d2c>] 0x80133d2c
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.552250] [<801383f8>] 0x801383f8
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.555808] [<800ab310>] 0x800ab310
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.559386] [<8007195c>] 0x8007195c
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.562912] [<800772bc>] 0x800772bc
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.566440]
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.567948] Mem-Info:
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.570253] active_anon:351 inactive_anon:5 isolated_anon:0
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.570253] active_file:104 inactive_file:132 isolated_file:0
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.570253] unevictable:0 dirty:0 writeback:0 unstable:0
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.570253] slab_reclaimable:184 slab_unreclaimable:1741
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.570253] mapped:2 shmem:16 pagetables:70 bounce:0
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.570253] free:2047 free_pcp:6 free_cma:0
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.601949] Node 0 active_anon:1404kB inactive_anon:20kB active_file:416kB inactive_file:528kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:8kB dirty:0kB writeback:0kB shmem:64kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.624050] Normal free:8188kB min:8192kB low:10240kB high:12288kB active_anon:1404kB inactive_anon:20kB active_file:416kB inactive_file:528kB unevictable:0kB writepending:0kB present:65536kB managed:59364kB mlocked:0kB kernel_stack:336kB pagetables:280kB bounce:0kB free_pcp:24kB local_pcp:24kB free_cma:0kB
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.651334] lowmem_reserve[]: 0 0
Mon Jul 8 10:09:08 2019 kern.warn kernel: [ 181.654676] Normal: 794kB (UME) 828kB (UME) 9916kB (UME) 3432kB (ME) 1964kB (ME) 10128kB (ME) 4256kB (ME) 0512kB 11024kB (E) 02048kB 0*4096kB = 8188kB

@openwrt-bot
Copy link
Author

gwlim:

[[https://openwrt.org/docs/guide-user/perf_and_log/benchmark.ipv6.routing|Guide used to perform IPv6 UDP test]]
Tested Trunk commit version
commit 653e05d

@openwrt-bot
Copy link
Author

gwlim:

This issue arise from UDP connections not being removed fast enough.
On Openwrt the value is
/package/kernel/linux/files/sysctl-nf-conntrack.conf
net.netfilter.nf_conntrack_udp_timeout=60
net.netfilter.nf_conntrack_udp_timeout_stream=180
On Linux Mainline the value is
nf_conntrack_udp_timeout - INTEGER (seconds)
default 30
nf_conntrack_udp_timeout_stream - INTEGER (seconds)
default 120

It makes no sense that upstream uses a low value than openwrt given that openwrt is meant to run on smaller devices
If users are worried UDP will timeout it is not a concern because UDP is stateless, if a UDP is a stream flow Linux should detect this and automatically convert the UDP session to a stream and hence timeout will increase automatically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant