New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FS#120 - opkg fails with Out of memory on a device with 32 MB of RAM #5795
Comments
glycoknob: See also https://github.com/lede-project/source/issues/237 A workaround that works for me is setting: OPKG_USE_VFORK=1 opkg install ... |
nbd: fix pushed in r1454 |
VittGam: The fix seems to work, thanks! But I fail to understand why it's needed now. It didn't fail without vfork on CC with ~5 MB of free RAM, so why does it fail now on LEDE trunk with ~12 MB of free ram?... |
jow-: We had to revert Felix' original commit due to some issues in opkgs vfork code but I pushed an alternative approach to my staging tree which lets opkg use the external gzip command. As a side effect the opkg binary size is reduced by ~4K. I would appreciate some testing feedback on http://git.lede-project.org/d692bfec1306a514936eca5805dff6b8fac0ee0e |
paulcarroty: The same with TL-741ND, r1657 Workaround:
cat /etc/opkg/distfeeds.conf
src/gz reboot_core http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages
src/gz reboot_base http://downloads.lede-project.org/snapshots/packages/mips_24kc/base
#src/gz reboot_telephony http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony
src/gz reboot_packages http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages
src/gz reboot_routing http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing
#src/gz reboot_luci http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci
|
bobafetthotmail: @Jo-Philipp Wich I just tested the two patches on a TL-WR1043ND v1 (32MiB ram, 8 MiB flash), and I'm available for more testing on this device. It is attached with serial, so I can recover anything. Now installing a single package works fine (before it always failed). Installing more than one package gives weird errors:
Manual uninstallation of packages followed, then another try:
|
bobafetthotmail: tried with swap (it has like 1.5GiB of swap now), installing things still fails hard.
another try, this time seems to double-crash.
|
mgondium: LEDE head 1949, TPLINK 1043ND v1. |
cypa: same problem on TP-LINK TL-WR741ND v5 r2244 unable to download
|
cypa: commenting reboot_luci and reboot_telephony does the thing! |
bobafetthotmail: @Jo-Philipp Wich I think the only way to solve this is to simply don't keep local package lists on "low mem" devices. Each time opkg needs to look at package lists it downloads, reads and deletes each package list singularly. It will have a pretty big latency (as commands like "opkg list" will have to download and parse all lists before giving an output), but would preserve functionality. Since opkg is a package manager, I think list reading performance can be sacrificed here. |
nbd: Please test the latest version |
bobafetthotmail1: Still fails when it really shouldn't. it seems wget fails to download a 4kib package while I have 8 MiB of free RAM + 12 MiB of caches. root@lede:/# opkg install luci-app-minidlna
root@lede:/# free root@lede:/# cat /proc/meminfo |
sreekanth: Doesn't work for me too.
On a second attempt, opkg update was successful.
root@lede:~# opkg update
Downloading http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/Packages.gz.
Updated list of available packages in /var/opkg-lists/reboot_core.
Downloading http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/Packages.sig.
Signature check passed.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/base/Packages.gz.
Updated list of available packages in /var/opkg-lists/reboot_base.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/base/Packages.sig.
Signature check passed.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/Packages.gz.
Updated list of available packages in /var/opkg-lists/reboot_luci.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/Packages.sig.
Signature check passed.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/Packages.gz.
Updated list of available packages in /var/opkg-lists/reboot_packages.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/Packages.sig.
Signature check passed.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing/Packages.gz.
Updated list of available packages in /var/opkg-lists/reboot_routing.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing/Packages.sig.
Signature check passed.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony/Packages.gz.
Updated list of available packages in /var/opkg-lists/reboot_telephony.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony/Packages.sig.
Signature check passed.
However, when I tried to install a package, then it failed again.
root@lede:~# opkg install luci-app-adblock
Installing luci-app-adblock (git-17.002.34912-a48c5d2-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-adblock_git-17.002.34912-a48c5d2-1_all.ipk.
Collected errors:
* xsystem: wget: vfork: Out of memory.
* opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-adblock_git-17.002.34912-a48c5d2-1_all.ipk, wget returned -1.
* opkg_install_pkg: Failed to download luci-app-adblock. Perhaps you need to run 'opkg update'?
* opkg_install_cmd: Cannot install package luci-app-adblock.
|
worgenrage: Hey there! Got this problem too with a ver 1.X (4MB / 32MB) tplink 1043ND router. On some forums there was a suggestion for opkg vfork* mem error. sysctl -w vm.min_free_kbytes=0 when i used this and run opkg update then Packages list updated. I can see a couple of packages via opkg list-upgradable . But. I can upgrade small packages, such as odhcpd or odhcpd6 or luci-proto-ppp * But when i want to run opkg upgrade luci-base or luci-theme-bootstrap* (eg. bigger than ~10kb) then it just stuck at downloading. root@OpenWrt:/tmp# opkg upgrade luci-mod-admin-full When i go to /tmp and do an ls -la then i see that there is an opkg-XYJkfds tmp dir for this package. I check that. I got the .ipk file already downloaded for full size and a temporary directory. Probably that should be where .ipk extracted. Nah thats empty. Router working fine i can terminate opkg only via killing the process. btw root@OpenWrt:/tmp# df -h What information i can provide to the team about this "bug"? :) (btw, with owrt 15.0X version i did not have this problem on this router, so wondering) |
eripek: @C. Hamar: there is no TL-WR1043NDv1 I'd know of, that has 4MB flash memory. They all have 8. I'd suggest using image builder and add zram and zram-swap, ... //unless dev team decides to activate that as a default option with uci set system.@System[0].zram_size_mb=24 (or higher) for this build target//. That is the most quick and not so dirty solution for this some memory issue at the cost of loosing some cpu cycles... The main problem of the memory issue is sysupgrade failing during flash. Be sure to have the most recent bootloader for easy (tftp) recovery of your device! Especially elder devices have a UBoot dated back to 2009. It makes sense to flash back an image of the original firmware (follow revert procedures for NOboot/stripped original firmware w/o the bootloader) and then flash the newest original firmware (with "boot" in it's filename) available all over again using the TP-Link orginal firmware webinterface. After that a Uboot version from 2014 supporting reset-button-triggered TFTP should be installed on the router. |
murix: Hi guys, I have a TL-WR1043NDv1 running Reboot (17.01.0-rc2, r3131-42f3c1f) root@LEDE:~# opkg install tcpdump
|
jow-: I pushed an opkg update to master with https://git.lede-project.org/b65dc04 which will make opkg install processes consume much less memory. |
Greybeard: Using the 17.01.0 built code on a tplink wr842n v1 (32M ram) and a wr842m v3 (64M ram) without the opkg update from Jo-Phillip above, ie unpatched code, opkg fails on v1 device 32M ram, works ok on 64M ram device. So the issue is still present in the 17.01 build. |
jow-: Yes, the opkg update didn't make it to 17.01 as it was a too severe change. |
Greybeard: pity as it pretty much makes 17.01 useless for anyone with a 32M ram device who wants to add any packages. May need a warning on the release notes about the constraint. |
jow-: I know. We'll plan to ship the opkg update with 17.01.1 which is due mid/end of March so not all is lost :) The opkg rework just happened too late for .0 |
jow-: It should be possible to just install opkg from master on a 17.01 system. |
Greybeard: Appreciate the time & effort spent in sorting this all out. |
jow-: E.g. on an ar71xx device it would be |
bobafetthotmail1: @jow opkg fails to do that because of same low ram issues, at least here in a test system still on older snapshot. Lol waiting for March is going to be a blood bath. |
Greybeard: Running that update then 'opkg update' has changed all the feeds to 'snapshot' which then raises all sorts of errors when trying to run opkg. |
lalamper: How about temporary shut down all services except what needed for opkg to run? |
Evgeneys: Final release 17.01 on Tl-841 v5 :(
|
bobafetthotmail1: See a few posts above, the fix for this will be in LEDE 17.01.1 which is due mid/end of March. |
Snotmann: Hey guys, i used zram to solve my problems with my four wr1043n v1. |
lalamper: @snotto: Could you please provide more details how to solve it with zram. |
Snotmann: Simply install both zram packages you could find with opkg. kmod-zram and zram-swap are the packages. But take care ... until now all my 32 MB devices get problems over time with these |
bluewavenet: I have been getting round this using imagebuilder. If not I will have to revert back to OpenWrt CC for our latest production rollout. |
bluewavenet: Further to my comment of 18.03.2017 11:58, I have downloaded the opkg version in today's snapshot (opkg_2017-03-15-cfe46c7d-1_mips_24kc.ipk) and installed in 17.01.0 and it works fine. |
Magician24: Also been affected by this with a 32MB RAM router, but been able to get around it by disabling some package lists. Any updates on the release of 17.01.1 with this fix? It was said earlier "mid/end" of this month but still nothing so far. |
bobafetthotmail1: LEDE 17.01.1 is being prepared and should be released by the Easter holidays, according to this announcement http://lists.infradead.org/pipermail/lede-dev/2017-April/007003.html |
murix: running opkg with LEDE 17.01.2 in WR1043v1 (32mb ram/8mb flash) is fine for me. I installed several packages without any errors. |
bluewavenet: This was fixed in 17.01.1. Glad to see it is still fixed in .2 :D |
muhaha: Not working 17.01.2 TP-Link TL-WR1043N/ND v1
|
bluewavenet: @muhaha |
muhaha: @rob White, Here is stacktrace
root@LEDE:~# opkg update
Downloading http://downloads.lede-project.org/releases/17.01.2/targets/ar71xx/generic/packages/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.2/targets/ar71xx/generic/packages/Packages.gz
EDIT: |
bluewavenet: @muhaha \ LE \ / ----------------------------------------------------------- root@LEDE: |
bluewavenet: @muhaha I then loaded up php7 and ran a script that used up loads of memory, taking free down to less than 1MB. Then I get your problem. There is obviously still a threshold where opkg runs out of memory. Try a reboot and see if it helps ;) |
bluewavenet: @cypa |
jow-: Well, if a system is out of memory, it is out of memory. No amount of opkg optimizations will change that. I lowered the resource usage of opkg by 60-70% percent, thats as far as it will get. If you still suffer from out of memory conditions then your environment simply cannot accomodate opkg anymore. Either try to reduce the amounts of feeds or consider precompiling images yourself. |
Markinus: Hi! I have the same problem on my TPLINK RE450. I'm restarting the system and the upgrade is not possible, console output:
It looks like there is enough memory availible, where is the problem? Can someone explain it? With a sysctl -w vm.min_free_kbytes=0 the upgrade is possible. |
bluewavenet: I thought the RE450 had 128MB ram? Free is only showing 60664KB with 15860 free. |
bobafetthotmail1: Since your device does not have 32MB ram, and this bugreport was closed/solved months ago it would be better to open another bug report to track your issue. |
Markinus: Yes, it works with set vm.min_free_kbytes=0. |
VittGam:
This happens on a freshly installed image from downloads.lede-project.org for an ar71xx device with 4 MB of flash and 32 MB of RAM (TP-LINK TL-MR13U).
This didn't happen with Chaos Calmer.
root@lede:~# cat /etc/banner _________ / /\ _ ___ ___ ___ / LE / \ | | | __| \| __| / DE / \ | |__| _|| |) | _| /________/ LE \ |____|___|___/|___| lede-project.org \ \ DE / \ LE \ / ----------------------------------------------------------- \ DE \ / Reboot (HEAD, r1439) \________\/ ------------------------------------------------------------ gz_open: fork: Out of memory.
- pkg_hash_add_from_file: Failed to open /var/opkg-lists/reboot_routing: Out of memory.
- gz_open: fork: Out of memory.
- pkg_hash_add_from_file: Failed to open /var/opkg-lists/reboot_routing: Out of memory.
root@lede:
# opkg update# opkg install foo[...] (it succeeds)
root@lede:
Collected errors:
root@lede:~# opkg update
Collected errors:
root@lede:
# free# dftotal used free shared buffers cached
Mem: 27828 15592 12236 668 1028 3352
-/+ buffers/cache: 11212 16616
Swap: 0 0 0
root@lede:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 2048 2048 0 100% /rom
tmpfs 13912 636 13276 5% /tmp
tmpfs 13912 32 13880 0% /tmp/root
tmpfs 512 0 512 0% /dev
/dev/mtdblock3 576 212 364 37% /overlay
overlayfs:/overlay 576 212 364 37% /
root@lede:~#
Cheers,
Vittorio
The text was updated successfully, but these errors were encountered: