OpenWrt/LEDE Project

  • Status Unconfirmed
  • Percent Complete
    0%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Medium
  • Priority Very Low
  • Reported Version openwrt-19.07
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by wrighrc - 30.09.2020

FS#3366 - router keeps running out of memory, have to power cycle it periodically, need help figuring out why.

Supply the following if possible:
- Device problem occurs on
netgear r6350
- Software versions of OpenWrt/LEDE release, packages, etc.
OpenWrt 19.07.4 r11208-ce6496d796 / LuCI openwrt-19.07 branch git-20.247.75781-0d0ab01
I installed ocserv too but I don’t think its the problem
- Steps to reproduce
Just let router run, maybe will stay up for a week and then at some point gui tells me can’t fork, I can’t login to router with ssh and I have to power cycle it.

I added swap space on a USB stick to try to keep it from locking up.
I’ve ssh’ed into and run the following to try to figure out what’s going on.

I expected to find a process growing but nothing looks obvious to me. I just see free memory keep going down, and I see a bit of swap space used.

Any suggestions on what else I could do to figure out what’s going on with the memory usage?

looking at meminfo it shows about ~100M used but the numbers don’t seem to add up to that.

while sleep 60
do

date
uptime
top -b -n 1 | head -n 10
ps w | awk '$3 > 4000'
free -m
cat /proc/meminfo
printf "\n"

done

Here’s what 10 days looks like:

Wed Sep 30 00:56:16 EDT 2020
00:56:16 up 10 days, 2:38, load average: 0.00, 0.02, 0.00
Mem: 100276K used, 23496K free, 1076K shrd, 1832K buff, 4600K cached
CPU: 0% usr 0% sys 0% nic 100% idle 0% io 0% irq 0% sirq
Load average: 0.01 0.02 0.00 2/71 27832

PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND

27831 7096 root R 1224 1% 0% top -b -n 1
2175 2174 root S 4912 4% 0% ocserv-sm
2174 1 root S 4740 4% 0% ocserv-main

931     1 root     S     2324   2%   0% /sbin/rpcd -s /var/run/ubus.sock -t 30

2001 1 root S 1784 1% 0% /usr/sbin/hostapd -s -P /var/run/wifi-phy0.pid -B /var/run/hostapd-phy0.conf
1077 1 root S 1748 1% 0% /sbin/netifd

PID USER       VSZ STAT COMMAND

2174 root 4740 S ocserv-main
2175 root 4912 S ocserv-sm

            total        used        free      shared  buff/cache   available

Mem: 123772 93796 23544 1076 6432 2604
Swap: 262140 1024 261116
MemTotal: 123772 kB
MemFree: 23500 kB
MemAvailable: 2560 kB
Buffers: 1832 kB
Cached: 4600 kB
SwapCached: 232 kB
Active: 5184 kB
Inactive: 3364 kB
Active(anon): 1120 kB
Inactive(anon): 2072 kB
Active(file): 4064 kB
Inactive(file): 1292 kB
Unevictable: 0 kB
Mlocked: 0 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 123772 kB
LowFree: 23500 kB
SwapTotal: 262140 kB
SwapFree: 261116 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 1960 kB
Mapped: 2136 kB
Shmem: 1076 kB
Slab: 11756 kB
SReclaimable: 1912 kB
SUnreclaim: 9844 kB
KernelStack: 624 kB
PageTables: 308 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 324024 kB
Committed_AS: 6680 kB
VmallocTotal: 1040376 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB


wrighrc commented on 30.09.2020 12:31

Sorry for the poor formatting. I wish I could fix that but it seems I can't edit my own post. Here's a comparison from yesterday and this morning.


From 9/29: 
root@OpenWrt:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         123772       90884       27236        1104        5652        6208
Swap:        262140         520      261620

From 9/30
root@OpenWrt:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         123772       94848       22584         844        6340        1740
Swap:        262140        1544      260596

Wed Sep 30 08:23:26 EDT 2020
 08:23:26 up 10 days, 10:05,  load average: 0.06, 0.01, 0.01
Mem: 101308K used, 22464K free, 844K shrd, 1980K buff, 4356K cached
CPU:   0% usr   0% sys   0% nic 100% idle   0% io   0% irq   0% sirq
Load average: 0.06 0.02 0.02 2/71 25149
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 2175  2174 root     S     4912   4%   0% ocserv-sm
 2174     1 root     S     4740   4%   0% ocserv-main
  931     1 root     S     2324   2%   0% /sbin/rpcd -s /var/run/ubus.sock -t 30
 2001     1 root     S     1784   1%   0% /usr/sbin/hostapd -s -P /var/run/wifi-phy0.pid -B /var/run/hostapd-phy0.conf
 1077     1 root     S     1748   1%   0% /sbin/netifd
    1     0 root     S     1572   1%   0% /sbin/procd
  PID USER       VSZ STAT COMMAND
 2174 root      4740 S    ocserv-main
 2175 root      4912 S    ocserv-sm
              total        used        free      shared  buff/cache   available
Mem:         123772       95004       22432         844        6336        1588
Swap:        262140        1544      260596
MemTotal:         123772 kB
MemFree:           22376 kB
MemAvailable:       1532 kB
Buffers:            1980 kB
Cached:             4356 kB
SwapCached:          360 kB
Active:             4904 kB
Inactive:           3432 kB
Active(anon):        908 kB
Inactive(anon):     1936 kB
Active(file):       3996 kB
Inactive(file):     1496 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:         123772 kB
LowFree:           22376 kB
SwapTotal:        262140 kB
SwapFree:         260596 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          1672 kB
Mapped:             2228 kB
Shmem:               844 kB
Slab:              11800 kB
SReclaimable:       1968 kB
SUnreclaim:         9832 kB
KernelStack:         624 kB
PageTables:          352 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      324024 kB
Committed_AS:       6680 kB
VmallocTotal:    1040376 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing