Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#3366 - router keeps running out of memory, have to power cycle it periodically, need help figuring out why. #6675

Open
openwrt-bot opened this issue Sep 30, 2020 · 3 comments
Labels
flyspray release/19.07 pull request/issue targeted (also) for OpenWrt 19.07 release

Comments

@openwrt-bot
Copy link

wrighrc:

Supply the following if possible:

  • Device problem occurs on
    netgear r6350
  • Software versions of OpenWrt/LEDE release, packages, etc.
    OpenWrt 19.07.4 r11208-ce6496d796 / LuCI openwrt-19.07 branch git-20.247.75781-0d0ab01
    I installed ocserv too but I don't think its the problem
  • Steps to reproduce
    Just let router run, maybe will stay up for a week and then at some point gui tells me can't fork, I can't login to router with ssh and I have to power cycle it.

I added swap space on a USB stick to try to keep it from locking up.
I've ssh'ed into and run the following to try to figure out what's going on.

I expected to find a process growing but nothing looks obvious to me. I just see free memory keep going down, and I see a bit of swap space used.

Any suggestions on what else I could do to figure out what's going on with the memory usage?

looking at meminfo it shows about ~100M used but the numbers don't seem to add up to that.

while sleep 60 do date uptime top -b -n 1 | head -n 10 ps w | awk '$3 > 4000' free -m cat /proc/meminfo printf "\n" done

Here's what 10 days looks like:

Wed Sep 30 00:56:16 EDT 2020 00:56:16 up 10 days, 2:38, load average: 0.00, 0.02, 0.00 Mem: 100276K used, 23496K free, 1076K shrd, 1832K buff, 4600K cached CPU: 0% usr 0% sys 0% nic 100% idle 0% io 0% irq 0% sirq Load average: 0.01 0.02 0.00 2/71 27832 PID PPID USER STAT VSZ %VSZ %CPU COMMAND 27831 7096 root R 1224 1% 0% top -b -n 1 2175 2174 root S 4912 4% 0% ocserv-sm 2174 1 root S 4740 4% 0% ocserv-main 931 1 root S 2324 2% 0% /sbin/rpcd -s /var/run/ubus.sock -t 30 2001 1 root S 1784 1% 0% /usr/sbin/hostapd -s -P /var/run/wifi-phy0.pid -B /var/run/hostapd-phy0.conf 1077 1 root S 1748 1% 0% /sbin/netifd PID USER VSZ STAT COMMAND 2174 root 4740 S ocserv-main 2175 root 4912 S ocserv-sm total used free shared buff/cache available Mem: 123772 93796 23544 1076 6432 2604 Swap: 262140 1024 261116 MemTotal: 123772 kB MemFree: 23500 kB MemAvailable: 2560 kB Buffers: 1832 kB Cached: 4600 kB SwapCached: 232 kB Active: 5184 kB Inactive: 3364 kB Active(anon): 1120 kB Inactive(anon): 2072 kB Active(file): 4064 kB Inactive(file): 1292 kB Unevictable: 0 kB Mlocked: 0 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 123772 kB LowFree: 23500 kB SwapTotal: 262140 kB SwapFree: 261116 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 1960 kB Mapped: 2136 kB Shmem: 1076 kB Slab: 11756 kB SReclaimable: 1912 kB SUnreclaim: 9844 kB KernelStack: 624 kB PageTables: 308 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 324024 kB Committed_AS: 6680 kB VmallocTotal: 1040376 kB VmallocUsed: 0 kB VmallocChunk: 0 kB
@openwrt-bot
Copy link
Author

wrighrc:

Sorry for the poor formatting. I wish I could fix that but it seems I can't edit my own post. Here's a comparison from yesterday and this morning.

From 9/29:
root@OpenWrt:~# free -m
total used free shared buff/cache available
Mem: 123772 90884 27236 1104 5652 6208
Swap: 262140 520 261620

From 9/30
root@OpenWrt:~# free -m
total used free shared buff/cache available
Mem: 123772 94848 22584 844 6340 1740
Swap: 262140 1544 260596

Wed Sep 30 08:23:26 EDT 2020 08:23:26 up 10 days, 10:05, load average: 0.06, 0.01, 0.01 Mem: 101308K used, 22464K free, 844K shrd, 1980K buff, 4356K cached CPU: 0% usr 0% sys 0% nic 100% idle 0% io 0% irq 0% sirq Load average: 0.06 0.02 0.02 2/71 25149 PID PPID USER STAT VSZ %VSZ %CPU COMMAND 2175 2174 root S 4912 4% 0% ocserv-sm 2174 1 root S 4740 4% 0% ocserv-main 931 1 root S 2324 2% 0% /sbin/rpcd -s /var/run/ubus.sock -t 30 2001 1 root S 1784 1% 0% /usr/sbin/hostapd -s -P /var/run/wifi-phy0.pid -B /var/run/hostapd-phy0.conf 1077 1 root S 1748 1% 0% /sbin/netifd 1 0 root S 1572 1% 0% /sbin/procd PID USER VSZ STAT COMMAND 2174 root 4740 S ocserv-main 2175 root 4912 S ocserv-sm total used free shared buff/cache available Mem: 123772 95004 22432 844 6336 1588 Swap: 262140 1544 260596 MemTotal: 123772 kB MemFree: 22376 kB MemAvailable: 1532 kB Buffers: 1980 kB Cached: 4356 kB SwapCached: 360 kB Active: 4904 kB Inactive: 3432 kB Active(anon): 908 kB Inactive(anon): 1936 kB Active(file): 3996 kB Inactive(file): 1496 kB Unevictable: 0 kB Mlocked: 0 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 123772 kB LowFree: 22376 kB SwapTotal: 262140 kB SwapFree: 260596 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 1672 kB Mapped: 2228 kB Shmem: 844 kB Slab: 11800 kB SReclaimable: 1968 kB SUnreclaim: 9832 kB KernelStack: 624 kB PageTables: 352 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 324024 kB Committed_AS: 6680 kB VmallocTotal: 1040376 kB VmallocUsed: 0 kB VmallocChunk: 0 kB

@openwrt-bot
Copy link
Author

kofec:

In my case it was this:
08a42ef

@openwrt-bot
Copy link
Author

bjonglez:

The commit you mention fixes something that had not been backported to 19.07. So, it seems surprising that it would fix your issue.

Can you give more details about this "fix"?

@aparcar aparcar added the release/19.07 pull request/issue targeted (also) for OpenWrt 19.07 release label Feb 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flyspray release/19.07 pull request/issue targeted (also) for OpenWrt 19.07 release
Projects
None yet
Development

No branches or pull requests

2 participants