OpenWrt/LEDE Project

  • Status Unconfirmed
  • Percent Complete
    0%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Medium
  • Priority Very Low
  • Reported Version openwrt-19.07
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by wrighrc - 30.09.2020
Last edited by Baptiste Jonglez - 04.01.2021

FS#3366 - router keeps running out of memory, have to power cycle it periodically, need help figuring out why.

Supply the following if possible:
- Device problem occurs on
netgear r6350
- Software versions of OpenWrt/LEDE release, packages, etc.
OpenWrt 19.07.4 r11208-ce6496d796 / LuCI openwrt-19.07 branch git-20.247.75781-0d0ab01
I installed ocserv too but I don’t think its the problem
- Steps to reproduce
Just let router run, maybe will stay up for a week and then at some point gui tells me can’t fork, I can’t login to router with ssh and I have to power cycle it.

I added swap space on a USB stick to try to keep it from locking up.
I’ve ssh’ed into and run the following to try to figure out what’s going on.

I expected to find a process growing but nothing looks obvious to me. I just see free memory keep going down, and I see a bit of swap space used.

Any suggestions on what else I could do to figure out what’s going on with the memory usage?

looking at meminfo it shows about ~100M used but the numbers don’t seem to add up to that.

while sleep 60
do
  date
  uptime
  top -b -n 1 | head -n 10
  ps w | awk '$3 > 4000'
  free -m
  cat /proc/meminfo
  printf "\n"
done

Here’s what 10 days looks like:

Wed Sep 30 00:56:16 EDT 2020
 00:56:16 up 10 days,  2:38,  load average: 0.00, 0.02, 0.00
Mem: 100276K used, 23496K free, 1076K shrd, 1832K buff, 4600K cached
CPU:   0% usr   0% sys   0% nic 100% idle   0% io   0% irq   0% sirq
Load average: 0.01 0.02 0.00 2/71 27832
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
27831  7096 root     R     1224   1%   0% top -b -n 1
 2175  2174 root     S     4912   4%   0% ocserv-sm
 2174     1 root     S     4740   4%   0% ocserv-main
  931     1 root     S     2324   2%   0% /sbin/rpcd -s /var/run/ubus.sock -t 30
 2001     1 root     S     1784   1%   0% /usr/sbin/hostapd -s -P /var/run/wifi-phy0.pid -B /var/run/hostapd-phy0.conf
 1077     1 root     S     1748   1%   0% /sbin/netifd
  PID USER       VSZ STAT COMMAND
 2174 root      4740 S    ocserv-main
 2175 root      4912 S    ocserv-sm
              total        used        free      shared  buff/cache   available
Mem:         123772       93796       23544        1076        6432        2604
Swap:        262140        1024      261116
MemTotal:         123772 kB
MemFree:           23500 kB
MemAvailable:       2560 kB
Buffers:            1832 kB
Cached:             4600 kB
SwapCached:          232 kB
Active:             5184 kB
Inactive:           3364 kB
Active(anon):       1120 kB
Inactive(anon):     2072 kB
Active(file):       4064 kB
Inactive(file):     1292 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:         123772 kB
LowFree:           23500 kB
SwapTotal:        262140 kB
SwapFree:         261116 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          1960 kB
Mapped:             2136 kB
Shmem:              1076 kB
Slab:              11756 kB
SReclaimable:       1912 kB
SUnreclaim:         9844 kB
KernelStack:         624 kB
PageTables:          308 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      324024 kB
Committed_AS:       6680 kB
VmallocTotal:    1040376 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
wrighrc commented on 30.09.2020 12:31

Sorry for the poor formatting. I wish I could fix that but it seems I can't edit my own post. Here's a comparison from yesterday and this morning.


From 9/29: 
root@OpenWrt:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         123772       90884       27236        1104        5652        6208
Swap:        262140         520      261620

From 9/30
root@OpenWrt:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         123772       94848       22584         844        6340        1740
Swap:        262140        1544      260596

Wed Sep 30 08:23:26 EDT 2020
 08:23:26 up 10 days, 10:05,  load average: 0.06, 0.01, 0.01
Mem: 101308K used, 22464K free, 844K shrd, 1980K buff, 4356K cached
CPU:   0% usr   0% sys   0% nic 100% idle   0% io   0% irq   0% sirq
Load average: 0.06 0.02 0.02 2/71 25149
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 2175  2174 root     S     4912   4%   0% ocserv-sm
 2174     1 root     S     4740   4%   0% ocserv-main
  931     1 root     S     2324   2%   0% /sbin/rpcd -s /var/run/ubus.sock -t 30
 2001     1 root     S     1784   1%   0% /usr/sbin/hostapd -s -P /var/run/wifi-phy0.pid -B /var/run/hostapd-phy0.conf
 1077     1 root     S     1748   1%   0% /sbin/netifd
    1     0 root     S     1572   1%   0% /sbin/procd
  PID USER       VSZ STAT COMMAND
 2174 root      4740 S    ocserv-main
 2175 root      4912 S    ocserv-sm
              total        used        free      shared  buff/cache   available
Mem:         123772       95004       22432         844        6336        1588
Swap:        262140        1544      260596
MemTotal:         123772 kB
MemFree:           22376 kB
MemAvailable:       1532 kB
Buffers:            1980 kB
Cached:             4356 kB
SwapCached:          360 kB
Active:             4904 kB
Inactive:           3432 kB
Active(anon):        908 kB
Inactive(anon):     1936 kB
Active(file):       3996 kB
Inactive(file):     1496 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:         123772 kB
LowFree:           22376 kB
SwapTotal:        262140 kB
SwapFree:         260596 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          1672 kB
Mapped:             2228 kB
Shmem:               844 kB
Slab:              11800 kB
SReclaimable:       1968 kB
SUnreclaim:         9832 kB
KernelStack:         624 kB
PageTables:          352 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      324024 kB
Committed_AS:       6680 kB
VmallocTotal:    1040376 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Project Manager
Baptiste Jonglez commented on 04.01.2021 11:09

The commit you mention fixes something that had not been backported to 19.07. So, it seems surprising that it would fix your issue.

Can you give more details about this "fix"?

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing