New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FS#2734 - Opkg update fails although router has enough memory #7541
Comments
jch: Strace indicates that it's
|
jch: Enabling memory overcommit (which I'm not recommending should be the default) successfully works around the issue:
|
jow-: I don't see a solution to this. Opkg was already heavily patched and modified to use less RAM, even introducing a rather slow multi-pass list parsing algorithm to avoid keeping the entire dependency graph in memory. The lists are just big and keep growing, so devices with 64MB RAM and lower are nearing the end of their usefulness. |
DonkZZZ: Same issue here with a x86-64 image in Virtualbox with 128MB Ram |
jch: Jo-Philipp, may I suggest that you reopen this report? Just because we don't see a solution yet doesn't mean the bug should be closed. There are two points I'd like to add. First of all, "opkg install" and "opkg list-installable" work while "opkg update" fail. Is opkg perhaps keeping two copies (the old and the new one) in memory at the same time? Second, the failure is not due to lack of memory -- it is due to fork failing due to the large amount of writeable pages. To me, this implies that either opkg should avoid forking (by linking with libz instead of running gzip -- see gzip_fdopen in libbb/gzip.c), or fork before it allocates the data structures (and keep the pipe around for later usage), or use a custom memory allocator that marks the pages as unwriteable using mprotect. |
jow-: Sure, will reopen it if you prefer. |
jch: There might be a memory leak. A first run of opkg update indicates all memory has been freed; running it a second time, though, indicates a leak of almost 4MB:
|
TmorKax: Have the same problem. Only with Asus RT-N11P. Only with 19.07.X. 18.06 not affected. |
fnarfbargle: Can confirm this on an WNDR3700v1 with 19.07.01
So 26M free. Not like it's out of memory. The contents of /var/opkg-lists decompressed totals some 3.6M :
|
dark-penguin: If you are running "opkg update" for the first time, or if you delete /tmp/opkg-* , then it runs fine with only 2 Mb (or even less - I did not try beyond that). If there are already lists downloaded, then apparently no amount of memory will be enough (10M is certainly not enough). (We could just make it ignore previously downloaded lists, and that will fix it...) "opkg list" should only list the names and not do anything more complex. It also runs fine when there are no downloaded lists; that could be attributed to having less text to display, but 10M being not enough to just display some text without any checking? That's unlikely. Confirmed on: |
NelsonK.: 19.07.3 not fix this problem on Asus RT-N11P. Bug still there. Still "Out of memory" when opkg list-upgradable. Please pay attention to the one who leads MT7620N. |
dwardo: 19.07.3 has brought progress on my TP-Link re450v1 : No more Out of memory with opkg update However as stated in the previous post, opkg list-upgradable fails with Out of memory
strace shows : whereas |
karkarus50: 19.07.4 and ASUS RT-N11P. |
bjonglez: To get an idea of memory usage, run opkg with "time -v". Then look at "Maximum resident set size". That being said, I don't know if it accounts for forked processes. Here is memory usage with 19.07.4 on a TL-WDR4300 v1 (ath79): First ''opkg update'' takes 3.76 MB Second ''opkg update'' takes 3.76 MB ''opkg list'' takes 14.6 MB ''opkg list-upgradable'' takes 56.3 MB ''opkg list-installed'' takes 3.76 MB So, jow's optimizations are quite good at reducing memory usage for ''update'' and ''list''. Since ''list-upgradable'' has not been changed recently, it's not surprising it still takes a large amount of memory. |
RipperSoS: It's already been nine months since 19.07.0 was released. You have not done ANYTHING to fix the problem. Moreover, you have already started to drop support for 18.06, where this problem does not exist. When you completely stop supporting 18.06 what do we do then? Throw out fully working routers or become part of some kind of botnet network, without security updates? You understand that most people use budget models of routers with OpenWRT? Then why are you better than those technology companies that produce these routers and drop their support in 1.5-2 years? In some countries, people have wages of $ 100-200 per month (sometimes less) and they do not have money for good routers. PS: Have you ever wondered why "free" operating systems have a market share within one percent? That's because of these things. Due to the large number of small and huge bugs in these OS. You have a huge list of supported devices. But how many routers on this list actually work without any bugs? |
rrblocker: On D-Link DIR-869 with 19.07.5 same "Out of memory" after opkg list-upgradable in terminal. Opkg update works fine. |
kousu: I'm seeing this too on on Archer C50 v4 (https://openwrt.org/toh/hwdata/tp-link/tp-link_archer_c50_v4), but only on list-upgradable. I can run update as many times as I want, but I actually getting updates done. Weirdly, the luci interface doesn't seem to have a problem listing upgradable packages. This router was manufactured just last year. Though it's not a 16/128 device like [[https://openwrt.org/toh/views/toh_available_16128|recommended]], neither is it a [[https://openwrt.org/supported_devices/openwrt_on_432_devices|banned 4/32]]. It's in the middle: a 8/64, and that's supposed to be powerful enough. I'm running LuCI openwrt-19.07 branch (git-20.272.48698-9883bb3) / OpenWrt 19.07.4 r11208-ce6496d796. I tried running Baptiste's experiment: Summary: opkg update: 6.7MB
Turning on sysctl -w vm.overcommit_memory=1; allows the process to complete. A side issue: why is opkg upgrade running list-upgradable if it's just going to stop when it notices I haven't told it what packages to upgrade?
root@router:~# sysctl -w vm.overcommit_memory=1
vm.overcommit_memory = 1
root@router:~# time -v opkg list-upgradable
luci-app-statistics - git-20.348.38488-caae7ad-1 - git-20.358.74878-e019fd2-1
luci-app-opkg - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
libuci20130104 - 2019-09-01-415f9e48-3 - 2019-09-01-415f9e48-4
luci-lib-ip - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-mod-system - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-theme-bootstrap - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-mod-status - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-lib-iptparser - git-20.348.38488-caae7ad-1 - git-20.358.74878-e019fd2-1
luci-app-firewall - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
tcpdump - 4.9.3-1 - 4.9.3-2
uci - 2019-09-01-415f9e48-3 - 2019-09-01-415f9e48-4
luci-compat - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
rpcd-mod-luci - 20191114 - 20201107
luci-proto-ppp - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-mod-admin-full - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-base - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-app-simple-adblock - git-20.348.38488-caae7ad-50 - git-20.358.74878-e019fd2-50
luci-proto-ipv6 - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
uhttpd - 2020-03-13-975dce23-1 - 2020-10-01-3abcc891-1
luci-lib-nixio - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-lib-jsonc - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
luci-mod-network - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
wireless-regdb - 2019.06.03-1 - 2020.11.20-1
luci-app-openvpn - git-20.272.48698-9883bb3-1 - git-20.358.74878-e019fd2-1
Command being timed: "opkg list-upgradable"
User time (seconds): 3.44
System time (seconds): 0.29
Percent of CPU this job got: 96%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 3.87s
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 54096
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 6994
Voluntary context switches: 729
Involuntary context switches: 1482
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
root@router:~# time -v opkg upgrade
opkg: the ``upgrade'' command requires at least one argument
usage: opkg [options...] sub-command [arguments...]
where sub-command is one of:
Use as the root directory for offline installation of packages.
Trying the "rm" solution doesn't help me because it comes back as soon as re-run opkg update.
root@router:~# sysctl -w vm.overcommit_memory=0
vm.overcommit_memory = 0
root@router:~# rm /var/opkg-lists/openwrt_*
root@router:~# time -v opkg list-upgradable # this suceeds, but vacuously: it just prints an empty list
Command being timed: "opkg list-upgradable"
User time (seconds): 0.09
System time (seconds): 0.06
Percent of CPU this job got: 94%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 0.17s
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3984
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 1
Minor (reclaiming a frame) page faults: 102
Voluntary context switches: 1
Involuntary context switches: 18
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
root@router:~# time -v opkg update
Downloading http://downloads.openwrt.org/releases/19.07.4/targets/ramips/mt76x8/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_core
Downloading http://downloads.openwrt.org/releases/19.07.4/targets/ramips/mt76x8/packages/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/base/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_base
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/base/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/luci/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_luci
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/luci/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_packages
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/packages/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/routing/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_routing
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/routing/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/telephony/Packages.gz
Updated list of available packages in /var/opkg-lists/openwrt_telephony
Downloading http://downloads.openwrt.org/releases/19.07.4/packages/mipsel_24kc/telephony/Packages.sig
Signature check passed.
Command being timed: "opkg update"
User time (seconds): 4.02
System time (seconds): 1.59
Percent of CPU this job got: 45%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 12.27s
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 7072
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 56
Minor (reclaiming a frame) page faults: 4349
Voluntary context switches: 1096
Involuntary context switches: 2190
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
root@router:~# time -v opkg list-upgradable
Collected errors:
* pkg_hash_add_from_file: Failed to open /var/opkg-lists/openwrt_telephony: Out of memory.
Command exited with non-zero status 255
Command being timed: "opkg list-upgradable"
User time (seconds): 2.82
System time (seconds): 0.28
Percent of CPU this job got: 94%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0m 3.28s
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 44320
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 5980
Voluntary context switches: 670
Involuntary context switches: 1315
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 255
I was finally able to get my system updated with
sysctl -w vm.overcommit_memory=1
time -v opkg upgrade $(opkg list-upgradable | cut -f 1 -d ' ')
I share your anxiety, Karlito. Dropping support for older devices leaves a lot of perfectly good hardware sitting wasted, and people working on a budget out in the cold. The same thing happens with smartphones all the time; when the most common messenger apps your friends use decide your phone is too old then you're forced to buy a new one, or be cut off if you can't afford it. I imagine this turning out to be critical for hackers trying to network their communities in developing countries, like
I don't have a good solution. We need to get open source funded properly I guess! I do hope we can extend the life of cheaper routers. We've identified list-upgradable as the problem here, so maybe we can pitch in by tracing the code and seeing if there's a way to make it process the data in more of a stream instead of a blob. |
kousu: And by "we" I mean "we", we're the ones having a problem, if we want it solved we can help out by getting into the code. |
jch:
19.07.0 running on WNDR3700v2, 32MB of memory. The machine is certainly not short on memory:
Running "opkg update" a first time works fine. However, if I run "opkg update" a second time, it reports:
After doing "rm /var/opkg-lists/openwrt_*", everything works fine again.
The text was updated successfully, but these errors were encountered: