Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#120 - opkg fails with Out of memory on a device with 32 MB of RAM #5795

Closed
openwrt-bot opened this issue Aug 25, 2016 · 51 comments
Closed
Labels

Comments

@openwrt-bot
Copy link

VittGam:

This happens on a freshly installed image from downloads.lede-project.org for an ar71xx device with 4 MB of flash and 32 MB of RAM (TP-LINK TL-MR13U).

This didn't happen with Chaos Calmer.

root@lede:~# cat /etc/banner _________ / /\ _ ___ ___ ___ / LE / \ | | | __| \| __| / DE / \ | |__| _|| |) | _| /________/ LE \ |____|___|___/|___| lede-project.org \ \ DE / \ LE \ / ----------------------------------------------------------- \ DE \ / Reboot (HEAD, r1439) \________\/ -----------------------------------------------------------

root@lede:# opkg update
[...] (it succeeds)
root@lede:
# opkg install foo
Collected errors:

  • gz_open: fork: Out of memory.
  • pkg_hash_add_from_file: Failed to open /var/opkg-lists/reboot_routing: Out of memory.
    root@lede:~# opkg update
    Collected errors:
  • gz_open: fork: Out of memory.
  • pkg_hash_add_from_file: Failed to open /var/opkg-lists/reboot_routing: Out of memory.
    root@lede:# free
    total used free shared buffers cached
    Mem: 27828 15592 12236 668 1028 3352
    -/+ buffers/cache: 11212 16616
    Swap: 0 0 0
    root@lede:
    # df
    Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/root 2048 2048 0 100% /rom
    tmpfs 13912 636 13276 5% /tmp
    tmpfs 13912 32 13880 0% /tmp/root
    tmpfs 512 0 512 0% /dev
    /dev/mtdblock3 576 212 364 37% /overlay
    overlayfs:/overlay 576 212 364 37% /
    root@lede:~#

Cheers,
Vittorio

@openwrt-bot
Copy link
Author

glycoknob:

See also https://github.com/lede-project/source/issues/237

A workaround that works for me is setting:

OPKG_USE_VFORK=1 opkg install ...

@openwrt-bot
Copy link
Author

nbd:

fix pushed in r1454

@openwrt-bot
Copy link
Author

VittGam:

The fix seems to work, thanks! But I fail to understand why it's needed now. It didn't fail without vfork on CC with ~5 MB of free RAM, so why does it fail now on LEDE trunk with ~12 MB of free ram?...

@openwrt-bot
Copy link
Author

jow-:

We had to revert Felix' original commit due to some issues in opkgs vfork code but I pushed an alternative approach to my staging tree which lets opkg use the external gzip command. As a side effect the opkg binary size is reduced by ~4K.

I would appreciate some testing feedback on http://git.lede-project.org/d692bfec1306a514936eca5805dff6b8fac0ee0e

@openwrt-bot
Copy link
Author

paulcarroty:

The same with TL-741ND, r1657

Workaround:

cat /etc/opkg/distfeeds.conf src/gz reboot_core http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages src/gz reboot_base http://downloads.lede-project.org/snapshots/packages/mips_24kc/base #src/gz reboot_telephony http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony src/gz reboot_packages http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages src/gz reboot_routing http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing #src/gz reboot_luci http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci

@openwrt-bot
Copy link
Author

bobafetthotmail:

@Jo-Philipp Wich

I just tested the two patches on a TL-WR1043ND v1 (32MiB ram, 8 MiB flash), and I'm available for more testing on this device. It is attached with serial, so I can recover anything.

Now installing a single package works fine (before it always failed).

Installing more than one package gives weird errors:

root@lede:/# opkg install luci-app-minidlna
Installing luci-app-minidlna (git-16.288.36935-1e1a706-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-minidlna_git-16.288.36935-1e1a706-1_all.ipk.
Installing minidlna (1.1.5-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/minidlna_1.1.5-1_mips_24kc.ipk.
Installing libexif (0.6.21-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/libexif_0.6.21-1_mips_24kc.ipk.
Installing libjpeg (9a-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/libjpeg_9a-1_mips_24kc.ipk.
Installing libsqlite3 (3130000-2) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/libsqlite3_3130000-2_mips_24kc.ipk.
Installing libffmpeg-mini (2.8.8-2) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/libffmpeg-mini_2.8.8-2_mips_24kc.ipk.
Configuring libjpeg.
Configuring libexif.
Configuring libsqlite3.
Configuring minidlna.
Collected errors:

  • opkg_install_pkg: Failed to verify the signature of /var/opkg-lists/reboot_packages.
  • opkg_install_cmd: Cannot install package luci-app-minidlna.
  • xsystem: sh: vfork: Out of memory.
  • pkg_run_script: package "libjpeg" postinst script returned status -1.
  • opkg_configure: libjpeg.postinst returned -1.
  • xsystem: sh: vfork: Out of memory.
  • pkg_run_script: package "libexif" postinst script returned status -1.
  • opkg_configure: libexif.postinst returned -1.
  • xsystem: sh: vfork: Out of memory.
  • pkg_run_script: package "libsqlite3" postinst script returned status -1.
  • opkg_configure: libsqlite3.postinst returned -1.
  • xsystem: sh: vfork: Out of memory.
  • pkg_run_script: package "minidlna" postinst script returned status -1.
  • opkg_configure: minidlna.postinst returned -1.

root@lede:/# free
total used free shared buffers cached
Mem: 27820 17964 9856 644 552 4340
-/+ buffers/cache: 13072 14748
Swap: 0 0 0

Manual uninstallation of packages followed, then another try:

Installing luci-app-minidlna (git-16.288.36935-1e1a706-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-minidlna_git-16.288.36935-1e1a706-1_all.ipk.
Installing minidlna (1.1.5-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/minidlna_1.1.5-1_mips_24kc.ipk.
Installing libexif (0.6.21-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/libexif_0.6.21-1_mips_24kc.ipk.
Installing libjpeg (9a-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/libjpeg_9a-1_mips_24kc.ipk.
Configuring libexif.
Configuring minidlna.
Error loading shared library libjpeg.so.9: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libid3tag.so.0: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libsqlite3.so.0: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libavformat.so.56: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libavutil.so.54: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libFLAC.so.8: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libogg.so.0: No such file or directory (needed by /usr/bin/minidlna)
Error loading shared library libvorbis.so.0: No such file or directory (needed by /usr/bin/minidlna)
Error relocating /usr/bin/minidlna: ogg_stream_packetpeek: symbol not found
Error relocating /usr/bin/minidlna: ogg_sync_clear: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_total_changes: symbol not found
Error relocating /usr/bin/minidlna: jpeg_resync_to_restart: symbol not found
Error relocating /usr/bin/minidlna: ogg_page_bos: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_busy_timeout: symbol not found
Error relocating /usr/bin/minidlna: ogg_stream_clear: symbol not found
Error relocating /usr/bin/minidlna: jpeg_read_scanlines: symbol not found
Error relocating /usr/bin/minidlna: id3_tag_findframe: symbol not found
Error relocating /usr/bin/minidlna: ogg_page_pageno: symbol not found
Error relocating /usr/bin/minidlna: jpeg_start_decompress: symbol not found
Error relocating /usr/bin/minidlna: jpeg_destroy_compress: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_exec: symbol not found
Error relocating /usr/bin/minidlna: vorbis_info_init: symbol not found
Error relocating /usr/bin/minidlna: id3_ucs4_latin1duplicate: symbol not found
Error relocating /usr/bin/minidlna: id3_field_getstrings: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_open: symbol not found
Error relocating /usr/bin/minidlna: jpeg_write_scanlines: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_libversion_number: symbol not found
Error relocating /usr/bin/minidlna: jpeg_std_error: symbol not found
Error relocating /usr/bin/minidlna: vorbis_comment_init: symbol not found
Error relocating /usr/bin/minidlna: id3_ucs4_utf8duplicate: symbol not found
Error relocating /usr/bin/minidlna: ogg_stream_packetout: symbol not found
Error relocating /usr/bin/minidlna: id3_file_close: symbol not found
Error relocating /usr/bin/minidlna: av_log_set_level: symbol not found
Error relocating /usr/bin/minidlna: ogg_stream_pagein: symbol not found
Error relocating /usr/bin/minidlna: jpeg_set_defaults: symbol not found
Error relocating /usr/bin/minidlna: id3_field_getnstrings: symbol not found
Error relocating /usr/bin/minidlna: id3_file_tag: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_simple_iterator_init: symbol not found
Error relocating /usr/bin/minidlna: ogg_sync_buffer: symbol not found
Error relocating /usr/bin/minidlna: id3_file_open: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_object_delete: symbol not found
Error relocating /usr/bin/minidlna: vorbis_comment_clear: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_simple_iterator_status: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_column_int: symbol not found
Error relocating /usr/bin/minidlna: ogg_sync_wrote: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_errmsg: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_simple_iterator_delete: symbol not found
Error relocating /usr/bin/minidlna: av_register_all: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_get_table: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_column_type: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_step: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_column_int64: symbol not found
Error relocating /usr/bin/minidlna: jpeg_CreateDecompress: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_close: symbol not found
Error relocating /usr/bin/minidlna: ogg_page_granulepos: symbol not found
Error relocating /usr/bin/minidlna: ogg_sync_pageout: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_free_table: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_release_memory: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_snprintf: symbol not found
Error relocating /usr/bin/minidlna: avformat_close_input: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_column_bytes: symbol not found
Error relocating /usr/bin/minidlna: id3_field_getbinarydata: symbol not found
Error relocating /usr/bin/minidlna: jpeg_destroy_decompress: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_column_text: symbol not found
Error relocating /usr/bin/minidlna: jpeg_stdio_src: symbol not found
Error relocating /usr/bin/minidlna: avformat_find_stream_info: symbol not found
Error relocating /usr/bin/minidlna: id3_field_getstring: symbol not found
Error relocating /usr/bin/minidlna: av_dict_get: symbol not found
Error relocating /usr/bin/minidlna: jpeg_read_header: symbol not found
Error relocating /usr/bin/minidlna: jpeg_finish_decompress: symbol not found
Error relocating /usr/bin/minidlna: jpeg_CreateCompress: symbol not found
Error relocating /usr/bin/minidlna: id3_field_getlatin1: symbol not found
Error relocating /usr/bin/minidlna: vorbis_synthesis_headerin: symbol not found
Error relocating /usr/bin/minidlna: av_reduce: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_vmprintf: symbol not found
Error relocating /usr/bin/minidlna: ogg_stream_init: symbol not found
Error relocating /usr/bin/minidlna: id3_field_getfullstring: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_simple_iterator_next: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_prepare: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_threadsafe: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_finalize: symbol not found
Error relocating /usr/bin/minidlna: ogg_page_serialno: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_last_insert_rowid: symbol not found
Error relocating /usr/bin/minidlna: vorbis_info_clear: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_free: symbol not found
Error relocating /usr/bin/minidlna: sqlite3_mprintf: symbol not found
Error relocating /usr/bin/minidlna: jpeg_start_compress: symbol not found
Error relocating /usr/bin/minidlna: ogg_sync_init: symbol not found
Error relocating /usr/bin/minidlna: avformat_open_input: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_simple_iterator_get_block: symbol not found
Error relocating /usr/bin/minidlna: ogg_page_eos: symbol not found
Error relocating /usr/bin/minidlna: jpeg_finish_compress: symbol not found
Error relocating /usr/bin/minidlna: FLAC__metadata_simple_iterator_new: symbol not found
Error relocating /usr/bin/minidlna: av_strerror: symbol not found
Error relocating /usr/bin/minidlna: jpeg_set_quality: symbol not found
Error relocating /usr/bin/minidlna: FLAC__Metadata_SimpleIteratorStatusString: symbol not found
Collected errors:

  • gz_open: fork: Out of memory.
  • opkg_install_pkg: Failed to unpack control files from /tmp/opkg-CeephH/libjpeg_9a-1_mips_24kc.ipk.
  • opkg_install_cmd: Cannot install package luci-app-minidlna.

root@lede:/# free
total used free shared buffers cached
Mem: 27820 18420 9400 972 1112 3940
-/+ buffers/cache: 13368 14452
Swap: 0 0 0

@openwrt-bot
Copy link
Author

bobafetthotmail:

tried with swap (it has like 1.5GiB of swap now), installing things still fails hard.
(I'm installing a local package now, as I'm also trying to test other stuff)

root@lede:/mnt# opkg install gdisk_1.0.1-1_mips_24kc.ipk
[ 5094.764231] opkg invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
[ 5094.771788] CPU: 0 PID: 2312 Comm: opkg Not tainted 4.4.24 #0
[ 5094.777618] Stack : 803e07a4 00000000 00000001 80430000 8198d180 8042ae63 803c1e58 00000908
8049379c 811c3d1c 00000000 00000003 0006538b 800a7228 00000006 803ca770
00000000 00000000 803c58c8 811c3bbc 80496542 800a51a4 8042a4d0 00000000
00000001 801f4700 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
...
[ 5094.813699] Call Trace:
[ 5094.816181] [<80071b70>] show_stack+0x50/0x84
[ 5094.820577] [<800d3d68>] dump_header.isra.4+0x48/0x130
[ 5094.825768] [<800d414c>] oom_kill_process+0x98/0x43c
[ 5094.830774] [<800d4854>] out_of_memory+0x2e0/0x324
[ 5094.835629] [<800d8154>] __alloc_pages_nodemask+0x684/0x6f0
[ 5094.841312] [<800d314c>] filemap_fault+0x258/0x458
[ 5094.846185] [<800edde0>] __do_fault+0x3c/0xa8
[ 5094.850604] [<800f1084>] handle_mm_fault+0x4e4/0xb84
[ 5094.855635] [<80076820>] __do_page_fault+0x134/0x470
[ 5094.860641] [<80060820>] ret_from_exception+0x0/0x10
[ 5094.865652]
[ 5094.867151] Mem-Info:
[ 5094.869469] active_anon:1019 inactive_anon:1064 isolated_anon:32
[ 5094.869469] active_file:0 inactive_file:11 isolated_file:0
[ 5094.869469] unevictable:0 dirty:0 writeback:1088 unstable:0
[ 5094.869469] slab_reclaimable:144 slab_unreclaimable:2683
[ 5094.869469] mapped:1 shmem:2 pagetables:53 bounce:0
[ 5094.869469] free:256 free_pcp:0 free_cma:0
[ 5094.901444] Normal free:1024kB min:1024kB low:1280kB high:1536kB active_anon:4076kB inactive_anon:4256kB active_file:0kB inactive_file:44kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:32768kB managed:27820kB mlocked:0kB dirty:0kB writeback:4352kB mapped:4kB shmem:8kB slab_reclaimable:576kB slab_unreclaimable:10732kB kernel_stack:376kB pagetables:212kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:51240 all_unreclaimable? yes
[ 5094.945352] lowmem_reserve[]: 0 0
[ 5094.948710] Normal: 404kB (U) 808kB (UM) 1416kB (UM) 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 1024kB
[ 5094.960562] 1117 total pagecache pages
[ 5094.964345] 1104 pages in swap cache
[ 5094.967942] Swap cache stats: add 4379, delete 3275, find 1170/1704
[ 5094.974256] Free swap = 1624364kB
[ 5094.977678] Total swap = 1630592kB
[ 5094.981097] 8192 pages RAM
[ 5094.983830] 0 pages HighMem/MovableOnly
[ 5094.987689] 1237 pages reserved
[ 5094.990846] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 5094.999455] [ 522] 0 522 295 1 3 0 15 0 ubusd
[ 5095.008139] [ 527] 0 527 297 1 3 0 10 0 ash
[ 5095.016644] [ 704] 0 704 307 1 4 0 34 0 logd
[ 5095.025237] [ 713] 0 713 361 1 4 0 22 0 rpcd
[ 5095.033835] [ 782] 0 782 338 5 3 0 18 0 odhcpd
[ 5095.042609] [ 823] 0 823 265 1 4 0 10 0 dropbear
[ 5095.051541] [ 840] 0 840 380 1 4 0 25 0 uhttpd
[ 5095.060376] [ 974] 0 974 297 1 3 0 8 0 ntpd
[ 5095.068971] [ 1859] 0 1859 409 7 4 0 28 0 netifd
[ 5095.077737] [ 1961] 0 1961 230 1 3 0 10 0 odhcp6c
[ 5095.086591] [ 1970] 0 1970 296 1 3 0 8 0 udhcpc
[ 5095.095366] [ 2031] 453 2031 263 1 4 0 16 0 dnsmasq
[ 5095.104226] [ 2306] 0 2306 1871 1001 4 0 633 0 opkg
[ 5095.112821] [ 2312] 0 2312 1668 517 4 0 905 0 opkg
[ 5095.121397] Out of memory: Kill process 2306 (opkg) score 3 or sacrifice child
[ 5095.128687] Killed process 2312 (opkg) total-vm:6672kB, anon-rss:2068kB, file-rss:0kB
Installing gdisk (1.0.1-1) to root...
Collected errors:

  • gz_close: Unzip process killed by signal 9.

another try, this time seems to double-crash.

root@lede:/mnt# opkg install gdisk_1.0.1-1_mips_24kc.ipk
[ 5283.015505] netifd invoked oom-killer: gfp_mask=0x2420848, order=0, oom_score_adj=0
[ 5283.023280] CPU: 0 PID: 1859 Comm: netifd Not tainted 4.4.24 #0
[ 5283.029232] Stack : 803e07a4 00000000 00000001 80430000 8198f980 8042ae63 803c1e58 00000743
8049379c 80d35a7c 00000000 00000003 0006538b 800a7228 00000006 803ca770
00000000 00000000 803c58c8 80d3591c 80496542 800a51a4 8042a4d0 00000000
00000001 801f4700 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
...
[ 5283.065296] Call Trace:
[ 5283.067778] [<80071b70>] show_stack+0x50/0x84
[ 5283.072175] [<800d3d68>] dump_header.isra.4+0x48/0x130
[ 5283.077366] [<800d414c>] oom_kill_process+0x98/0x43c
[ 5283.082372] [<800d4854>] out_of_memory+0x2e0/0x324
[ 5283.087229] [<800d8154>] __alloc_pages_nodemask+0x684/0x6f0
[ 5283.092888] [<800d12f8>] pagecache_get_page+0x154/0x278
[ 5283.098166] [<80132cbc>] __getblk_slow+0x15c/0x374
[ 5283.103021] [<8015a204>] squashfs_read_data+0x1c8/0x6e8
[ 5283.108290] [<8015a8b8>] squashfs_cache_get+0x194/0x2cc
[ 5283.113577] [<8015e288>] squashfs_readpage_block+0x1a0/0x4d8
[ 5283.119277] [<8015c090>] squashfs_readpage+0x5bc/0x6d0
[ 5283.124487] [<800db8f0>] __do_page_cache_readahead+0x1f8/0x264
[ 5283.130366] [<800d30a0>] filemap_fault+0x1ac/0x458
[ 5283.135231] [<800edde0>] __do_fault+0x3c/0xa8
[ 5283.139626] [<800f1084>] handle_mm_fault+0x4e4/0xb84
[ 5283.144650] [<80076820>] __do_page_fault+0x134/0x470
[ 5283.149657] [<80060820>] ret_from_exception+0x0/0x10
[ 5283.154664]
[ 5283.156166] Mem-Info:
[ 5283.158485] active_anon:970 inactive_anon:1044 isolated_anon:0
[ 5283.158485] active_file:30 inactive_file:79 isolated_file:0
[ 5283.158485] unevictable:0 dirty:0 writeback:1039 unstable:0
[ 5283.158485] slab_reclaimable:144 slab_unreclaimable:2683
[ 5283.158485] mapped:1 shmem:0 pagetables:55 bounce:0
[ 5283.158485] free:256 free_pcp:0 free_cma:0
[ 5283.190374] Normal free:1024kB min:1024kB low:1280kB high:1536kB active_anon:3880kB inactive_anon:4176kB active_file:120kB inactive_file:316kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:32768kB managed:27820kB mlocked:0kB dirty:0kB writeback:4156kB mapped:4kB shmem:0kB slab_reclaimable:576kB slab_unreclaimable:10732kB kernel_stack:376kB pagetables:220kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:51708 all_unreclaimable? yes
[ 5283.234370] lowmem_reserve[]: 0 0
[ 5283.237725] Normal: 724kB (UME) 748kB (UE) 916kB (U) 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 04096kB = 1024kB
[ 5283.249586] 1161 total pagecache pages
[ 5283.253369] 1052 pages in swap cache
[ 5283.256967] Swap cache stats: add 6510, delete 5458, find 1987/2922
[ 5283.263281] Free swap = 1624560kB
[ 5283.266702] Total swap = 1630592kB
[ 5283.270120] 8192 pages RAM
[ 5283.272855] 0 pages HighMem/MovableOnly
[ 5283.276713] 1237 pages reserved
[ 5283.279871] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 5283.288479] [ 522] 0 522 295 1 3 0 15 0 ubusd
[ 5283.297159] [ 527] 0 527 297 1 3 0 10 0 ash
[ 5283.305664] [ 704] 0 704 307 1 4 0 34 0 logd
[ 5283.314263] [ 713] 0 713 361 1 4 0 22 0 rpcd
[ 5283.322860] [ 782] 0 782 338 3 3 0 20 0 odhcpd
[ 5283.331619] [ 823] 0 823 265 1 4 0 10 0 dropbear
[ 5283.340568] [ 840] 0 840 380 1 4 0 25 0 uhttpd
[ 5283.349335] [ 974] 0 974 297 1 3 0 8 0 ntpd
[ 5283.357932] [ 1859] 0 1859 409 5 4 0 30 0 netifd
[ 5283.366701] [ 1961] 0 1961 230 1 3 0 10 0 odhcp6c
[ 5283.375555] [ 1970] 0 1970 296 7 3 0 2 0 udhcpc
[ 5283.384328] [ 2031] 453 2031 263 1 4 0 16 0 dnsmasq
[ 5283.393190] [ 2320] 0 2320 1851 968 5 0 646 0 opkg
[ 5283.401772] [ 2326] 0 2326 1668 492 5 0 930 0 opkg
[ 5283.410357] Out of memory: Kill process 2320 (opkg) score 3 or sacrifice child
[ 5283.417647] Killed process 2326 (opkg) total-vm:6672kB, anon-rss:1968kB, file-rss:0kB
[ 5285.135381] udhcpc invoked oom-killer: gfp_mask=0x2420848, order=0, oom_score_adj=0
[ 5285.143147] CPU: 0 PID: 1970 Comm: udhcpc Not tainted 4.4.24 #0
[ 5285.149098] Stack : 803e07a4 00000000 00000001 80430000 8198ea80 8042ae63 803c1e58 000007b2
8049379c 81d47ad4 00000000 00000003 0006538b 800a7228 00000006 803ca770
00000000 00000000 803c58c8 81d47974 80496542 800a51a4 8042a4d0 00000000
00000001 801f4700 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
...
[ 5285.185181] Call Trace:
[ 5285.187662] [<80071b70>] show_stack+0x50/0x84
[ 5285.192058] [<800d3d68>] dump_header.isra.4+0x48/0x130
[ 5285.197255] [<800d414c>] oom_kill_process+0x98/0x43c
[ 5285.202257] [<800d4854>] out_of_memory+0x2e0/0x324
[ 5285.207110] [<800d8154>] __alloc_pages_nodemask+0x684/0x6f0
[ 5285.212761] [<800d12f8>] pagecache_get_page+0x154/0x278
[ 5285.218032] [<80132cbc>] __getblk_slow+0x15c/0x374
[ 5285.222898] [<8015a204>] squashfs_read_data+0x1c8/0x6e8
[ 5285.228171] [<8015e414>] squashfs_readpage_block+0x32c/0x4d8
[ 5285.233896] [<8015c090>] squashfs_readpage+0x5bc/0x6d0
[ 5285.239083] [<800db8f0>] __do_page_cache_readahead+0x1f8/0x264
[ 5285.244982] [<800d30a0>] filemap_fault+0x1ac/0x458
[ 5285.249822] [<800edde0>] __do_fault+0x3c/0xa8
[ 5285.254233] [<800f1084>] handle_mm_fault+0x4e4/0xb84
[ 5285.259252] [<80076820>] __do_page_fault+0x134/0x470
[ 5285.264272] [<80060820>] ret_from_exception+0x0/0x10
[ 5285.269266]
[ 5285.270767] Mem-Info:
[ 5285.273100] active_anon:968 inactive_anon:1046 isolated_anon:0
[ 5285.273100] active_file:58 inactive_file:64 isolated_file:0
[ 5285.273100] unevictable:0 dirty:0 writeback:1031 unstable:0
[ 5285.273100] slab_reclaimable:144 slab_unreclaimable:2685
[ 5285.273100] mapped:4 shmem:0 pagetables:50 bounce:0
[ 5285.273100] free:256 free_pcp:0 free_cma:0
[ 5285.305023] Normal free:1024kB min:1024kB low:1280kB high:1536kB active_anon:3872kB inactive_anon:4184kB active_file:232kB inactive_file:256kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:32768kB managed:27820kB mlocked:0kB dirty:0kB writeback:4124kB mapped:16kB shmem:0kB slab_reclaimable:576kB slab_unreclaimable:10740kB kernel_stack:376kB pagetables:200kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:54780 all_unreclaimable? yes
[ 5285.349120] lowmem_reserve[]: 0 0
[ 5285.352473] Normal: 76
4kB (UE) 768kB (UME) 716kB (U) 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 04096kB = 1024kB
[ 5285.364334] 1177 total pagecache pages
[ 5285.368108] 1055 pages in swap cache
[ 5285.371706] Swap cache stats: add 6520, delete 5465, find 1988/2927
[ 5285.378019] Free swap = 1624772kB
[ 5285.381441] Total swap = 1630592kB
[ 5285.384869] 8192 pages RAM
[ 5285.387589] 0 pages HighMem/MovableOnly
[ 5285.391443] 1237 pages reserved
[ 5285.394615] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[ 5285.403216] [ 522] 0 522 295 1 3 0 15 0 ubusd
[ 5285.411888] [ 527] 0 527 297 1 3 0 10 0 ash
[ 5285.420395] [ 704] 0 704 307 1 4 0 34 0 logd
[ 5285.428991] [ 713] 0 713 361 1 4 0 22 0 rpcd
[ 5285.437577] [ 782] 0 782 338 4 3 0 19 0 odhcpd
[ 5285.446344] [ 823] 0 823 265 1 4 0 10 0 dropbear
[ 5285.455282] [ 840] 0 840 380 1 4 0 25 0 uhttpd
[ 5285.464046] [ 974] 0 974 297 1 3 0 8 0 ntpd
[ 5285.472624] [ 1859] 0 1859 409 5 4 0 30 0 netifd
[ 5285.481390] [ 1961] 0 1961 230 1 3 0 10 0 odhcp6c
[ 5285.490241] [ 1970] 0 1970 296 10 3 0 2 0 udhcpc
[ 5285.499014] [ 2031] 453 2031 263 1 4 0 16 0 dnsmasq
[ 5285.507875] [ 2320] 0 2320 1851 968 5 0 646 0 opkg
[ 5285.516459] Out of memory: Kill process 2320 (opkg) score 3 or sacrifice child
[ 5285.523748] Killed process 2320 (opkg) total-vm:7404kB, anon-rss:3868kB, file-rss:4kB
Killed

@openwrt-bot
Copy link
Author

mgondium:

LEDE head 1949, TPLINK 1043ND v1.
Looks better, now able to list and pipe to grep without going OOM.
Install successful for many packages, but (luci-app-)minidlna is not one of them.

@openwrt-bot
Copy link
Author

cypa:

same problem on TP-LINK TL-WR741ND v5 r2244

unable to download
root@k16:/tmp# opkg install libpthread
Installing libpthread (1.1.15-1) to root...
Downloading http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/libpthread_1.1.15-1_mips_24kc.ipk.
Collected errors:

  • xsystem: wget: vfork: Out of memory.
  • opkg_download: Failed to download http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/libpthread_1.1.15-1_mips_24kc.ipk, wget returned -1.
  • opkg_install_pkg: Failed to download libpthread. Perhaps you need to run 'opkg update'?
  • opkg_install_cmd: Cannot install package libpthread.
    root@k16:/tmp#

    and install any package, even smallest one
    root@k16:/tmp# OPKG_USE_VFORK=1 opkg install libpthread_1.1.15-1_mips_24kc.ipk
    Collected errors:
  • gz_open: fork: Out of memory.
  • pkg_init_from_file: Failed to extract control file from libpthread_1.1.15-1_mips_24kc.ipk.
    root@k16:/tmp#

@openwrt-bot
Copy link
Author

cypa:

commenting reboot_luci and reboot_telephony does the thing!
thanks Pavlo Rudyi

@openwrt-bot
Copy link
Author

bobafetthotmail:

@Jo-Philipp Wich
@cypa

I think the only way to solve this is to simply don't keep local package lists on "low mem" devices. Each time opkg needs to look at package lists it downloads, reads and deletes each package list singularly.

It will have a pretty big latency (as commands like "opkg list" will have to download and parse all lists before giving an output), but would preserve functionality.

Since opkg is a package manager, I think list reading performance can be sacrificed here.

@openwrt-bot
Copy link
Author

nbd:

Please test the latest version

@openwrt-bot
Copy link
Author

bobafetthotmail1:

Still fails when it really shouldn't. it seems wget fails to download a 4kib package while I have 8 MiB of free RAM + 12 MiB of caches.

root@lede:/# opkg install luci-app-minidlna
Installing luci-app-minidlna (git-16.365.41161-ca79611-1) to root...
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-minidlna_git-16.365.41161-ca79611-1_all.ipk.
Collected errors:

root@lede:/# free
total used free shared buffers cached
Mem: 27940 19156 8784 560 1044 3704
-/+ buffers/cache: 14408 13532
Swap: 0 0 0

root@lede:/# cat /proc/meminfo
MemTotal: 27940 kB
MemFree: 7996 kB
MemAvailable: 10660 kB
Buffers: 1120 kB
Cached: 3716 kB
SwapCached: 0 kB
Active: 3840 kB
Inactive: 2172 kB
Active(anon): 1704 kB
Inactive(anon): 32 kB
Active(file): 2136 kB
Inactive(file): 2140 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 1184 kB
Mapped: 1644 kB
Shmem: 560 kB
Slab: 7068 kB
SReclaimable: 1892 kB
SUnreclaim: 5176 kB
KernelStack: 328 kB
PageTables: 216 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 13968 kB
Committed_AS: 4312 kB
VmallocTotal: 1048372 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB

@openwrt-bot
Copy link
Author

sreekanth:

Doesn't work for me too.

BusyBox v1.25.1 () built-in shell (ash)

 _________
/        /\      _    ___ ___  ___

/ LE / \ | | | | | |
/ DE / \ | || _|| |) | _|
/
_/ LE \ |||/|| lede-project.org
\ \ DE /
\ LE \ / -----------------------------------------------------------
\ DE \ / Reboot (SNAPSHOT, r2709-b7677f0)
________/ -----------------------------------------------------------

root@lede:# free
total used free shared buffers cached
Mem: 27808 20792 7016 868 1408 5884
-/+ buffers/cache: 13500 14308
Swap: 0 0 0
root@lede:
# opkg update
Downloading http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/Packages.gz.
Downloading http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/Packages.sig.
Signature check failed.
Remove wrong Signature file.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/base/Packages.gz.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/base/Packages.sig.
Signature check failed.
Remove wrong Signature file.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/Packages.gz.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/Packages.sig.
Signature check failed.
Remove wrong Signature file.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/Packages.gz.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/Packages.sig.
Signature check failed.
Remove wrong Signature file.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing/Packages.gz.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing/Packages.sig.
Signature check failed.
Remove wrong Signature file.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony/Packages.gz.
Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony/Packages.sig.
Signature check failed.
Remove wrong Signature file.
Collected errors:

On a second attempt, opkg update was successful.

root@lede:~# opkg update Downloading http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/Packages.gz. Updated list of available packages in /var/opkg-lists/reboot_core. Downloading http://downloads.lede-project.org/snapshots/targets/ar71xx/generic/packages/Packages.sig. Signature check passed. Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/base/Packages.gz. Updated list of available packages in /var/opkg-lists/reboot_base. Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/base/Packages.sig. Signature check passed. Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/Packages.gz. Updated list of available packages in /var/opkg-lists/reboot_luci. Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/Packages.sig. Signature check passed. Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/Packages.gz. Updated list of available packages in /var/opkg-lists/reboot_packages. Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/packages/Packages.sig. Signature check passed. Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing/Packages.gz. Updated list of available packages in /var/opkg-lists/reboot_routing. Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/routing/Packages.sig. Signature check passed. Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony/Packages.gz. Updated list of available packages in /var/opkg-lists/reboot_telephony. Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/telephony/Packages.sig. Signature check passed.

However, when I tried to install a package, then it failed again.

root@lede:~# opkg install luci-app-adblock Installing luci-app-adblock (git-17.002.34912-a48c5d2-1) to root... Downloading http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-adblock_git-17.002.34912-a48c5d2-1_all.ipk. Collected errors: * xsystem: wget: vfork: Out of memory. * opkg_download: Failed to download http://downloads.lede-project.org/snapshots/packages/mips_24kc/luci/luci-app-adblock_git-17.002.34912-a48c5d2-1_all.ipk, wget returned -1. * opkg_install_pkg: Failed to download luci-app-adblock. Perhaps you need to run 'opkg update'? * opkg_install_cmd: Cannot install package luci-app-adblock.

@openwrt-bot
Copy link
Author

worgenrage:

Hey there! Got this problem too with a ver 1.X (4MB / 32MB) tplink 1043ND router.

On some forums there was a suggestion for opkg vfork* mem error.

sysctl -w vm.min_free_kbytes=0

when i used this and run opkg update then Packages list updated. I can see a couple of packages via opkg list-upgradable .

But. I can upgrade small packages, such as odhcpd or odhcpd6 or luci-proto-ppp *

But when i want to run opkg upgrade luci-base or luci-theme-bootstrap* (eg. bigger than ~10kb) then it just stuck at downloading.

root@OpenWrt:/tmp# opkg upgrade luci-mod-admin-full
Upgrading luci-mod-admin-full on root from git-17.025.85178-472dc4b-1 to git-17.033.24085-e306ee6-1...
Downloading http://downloads.lede-project.org/releases/17.01.0-rc1/packages/mips_24kc/luci/luci-mod-admin-full_git-17.033.24085-e306ee6-1_mips_24kc.ipk.

When i go to /tmp and do an ls -la then i see that there is an opkg-XYJkfds tmp dir for this package. I check that. I got the .ipk file already downloaded for full size and a temporary directory. Probably that should be where .ipk extracted.

Nah thats empty. Router working fine i can terminate opkg only via killing the process.

btw

root@OpenWrt:/tmp# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 2.3M 2.3M 0 100% /rom
tmpfs 13.8M 988.0K 12.8M 7% /tmp
/dev/mtdblock3 4.4M 1.5M 2.9M 34% /overlay
overlayfs:/overlay 4.4M 1.5M 2.9M 34% /
tmpfs 512.0K 0 512.0K 0% /dev

What information i can provide to the team about this "bug"? :)

(btw, with owrt 15.0X version i did not have this problem on this router, so wondering)

@openwrt-bot
Copy link
Author

eripek:

@C. Hamar: there is no TL-WR1043NDv1 I'd know of, that has 4MB flash memory. They all have 8.

I'd suggest using image builder and add zram and zram-swap, ...

//unless dev team decides to activate that as a default option with uci set system.@System[0].zram_size_mb=24 (or higher) for this build target//.

That is the most quick and not so dirty solution for this some memory issue at the cost of loosing some cpu cycles...

The main problem of the memory issue is sysupgrade failing during flash. Be sure to have the most recent bootloader for easy (tftp) recovery of your device! Especially elder devices have a UBoot dated back to 2009. It makes sense to flash back an image of the original firmware (follow revert procedures for NOboot/stripped original firmware w/o the bootloader) and then flash the newest original firmware (with "boot" in it's filename) available all over again using the TP-Link orginal firmware webinterface. After that a Uboot version from 2014 supporting reset-button-triggered TFTP should be installed on the router.

@openwrt-bot
Copy link
Author

murix:

Hi guys,

I have a TL-WR1043NDv1 running Reboot (17.01.0-rc2, r3131-42f3c1f)
Bizarre "sysctl -w vm.min_free_kbytes=0" help but it not work everytime.
With Openwrt 15.05.1 opkg run fine.

root@LEDE:~# opkg install tcpdump
Installing tcpdump (4.9.0-1) to root...
Downloading http://downloads.lede-project.org/releases/17.01.0-rc2/packages/mips_24kc/base/tcpdump_4.9.0-1_mips_24kc.ipk

@openwrt-bot
Copy link
Author

jow-:

I pushed an opkg update to master with https://git.lede-project.org/b65dc04 which will make opkg install processes consume much less memory.

@openwrt-bot
Copy link
Author

Greybeard:

Using the 17.01.0 built code on a tplink wr842n v1 (32M ram) and a wr842m v3 (64M ram)
LEDE Reboot 17.01.0 r3205-59508e3 / LuCI lede-17.01 branch (git-17.051.53299-a100738)

without the opkg update from Jo-Phillip above, ie unpatched code, opkg fails on v1 device 32M ram, works ok on 64M ram device. So the issue is still present in the 17.01 build.

@openwrt-bot
Copy link
Author

jow-:

Yes, the opkg update didn't make it to 17.01 as it was a too severe change.

@openwrt-bot
Copy link
Author

Greybeard:

pity as it pretty much makes 17.01 useless for anyone with a 32M ram device who wants to add any packages. May need a warning on the release notes about the constraint.
I'll look at trunk as an option.

@openwrt-bot
Copy link
Author

jow-:

I know. We'll plan to ship the opkg update with 17.01.1 which is due mid/end of March so not all is lost :)

The opkg rework just happened too late for .0

@openwrt-bot
Copy link
Author

jow-:

It should be possible to just install opkg from master on a 17.01 system.

@openwrt-bot
Copy link
Author

Greybeard:

Appreciate the time & effort spent in sorting this all out.
Happy to search but, how do I install opkg from master to an existing install?

@openwrt-bot
Copy link
Author

@openwrt-bot
Copy link
Author

bobafetthotmail1:

@jow opkg fails to do that because of same low ram issues, at least here in a test system still on older snapshot.

Lol waiting for March is going to be a blood bath.
I would really think it's worht it to make an exception for opkg and get it in release regardless.

@openwrt-bot
Copy link
Author

Greybeard:

Running that update then 'opkg update' has changed all the feeds to 'snapshot' which then raises all sorts of errors when trying to run opkg.
You need to manually alter the opkg feeds back to 17.01 from snapshot to get things running as expected.
It also seems to cause any software package view via luci to fail. ie no packages are shown in the 'available tab'.
It is possible to run opkg from the command line.

@openwrt-bot
Copy link
Author

lalamper:

How about temporary shut down all services except what needed for opkg to run?
I am so sad as I have some devices with 32MB of RAM.

@openwrt-bot
Copy link
Author

Evgeneys:

Final release 17.01 on Tl-841 v5 :(
collected errors:

@openwrt-bot
Copy link
Author

bobafetthotmail1:

See a few posts above, the fix for this will be in LEDE 17.01.1 which is due mid/end of March.

@openwrt-bot
Copy link
Author

Snotmann:

Hey guys,

i used zram to solve my problems with my four wr1043n v1.
Maybe you should try this ...

@openwrt-bot
Copy link
Author

lalamper:

@snotto: Could you please provide more details how to solve it with zram.

@openwrt-bot
Copy link
Author

Snotmann:

Simply install both zram packages you could find with opkg.
After you instlled it opkg update and install will be fine.

kmod-zram and zram-swap are the packages.

But take care ... until now all my 32 MB devices get problems over time with these
17.01. LEDE. I went back to CC 15.01. because its rock solid !

@openwrt-bot
Copy link
Author

bluewavenet:

I have been getting round this using imagebuilder.
As Alberto says above "See a few posts above, the fix for this will be in LEDE 17.01.1 which is due mid/end of March. "
Is this confirmed?

If not I will have to revert back to OpenWrt CC for our latest production rollout.

@openwrt-bot
Copy link
Author

bluewavenet:

Further to my comment of 18.03.2017 11:58, I have downloaded the opkg version in today's snapshot (opkg_2017-03-15-cfe46c7d-1_mips_24kc.ipk) and installed in 17.01.0 and it works fine.

@openwrt-bot
Copy link
Author

Magician24:

Also been affected by this with a 32MB RAM router, but been able to get around it by disabling some package lists.

Any updates on the release of 17.01.1 with this fix? It was said earlier "mid/end" of this month but still nothing so far.

@openwrt-bot
Copy link
Author

bobafetthotmail1:

LEDE 17.01.1 is being prepared and should be released by the Easter holidays, according to this announcement http://lists.infradead.org/pipermail/lede-dev/2017-April/007003.html

@openwrt-bot
Copy link
Author

murix:

running opkg with LEDE 17.01.2 in WR1043v1 (32mb ram/8mb flash) is fine for me. I installed several packages without any errors.

@openwrt-bot
Copy link
Author

bluewavenet:

This was fixed in 17.01.1. Glad to see it is still fixed in .2 :D

@openwrt-bot
Copy link
Author

muhaha:

Not working 17.01.2

TP-Link TL-WR1043N/ND v1
LEDE Reboot 17.01.2 r3435-65eec8bd5f / LuCI lede-17.01 branch (git-17.163.57307-c79bb96)

root@LEDE:~# free
total used free shared buffers cached
Mem: 28176 23472 4704 1324 1388 5668
-/+ buffers/cache: 16416 11760
Swap: 0 0 0

root@LEDE:~# df -h |grep overlay
/dev/sda1 3.7G 266.1M 3.4G 7% /overlay
overlayfs:/overlay 3.7G 266.1M 3.4G 7% /

@openwrt-bot
Copy link
Author

bluewavenet:

@muhaha
Can you be more specific with what is not working. I have had complete success on various hardware with less free memory than you are showing.

@openwrt-bot
Copy link
Author

muhaha:

@rob White,
still getting Out of memory

Here is stacktrace

Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.480381] sh invoked oom-killer: gfp_mask=0x2420848, order=0, oom_score_adj=0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.487839] CPU: 0 PID: 3520 Comm: sh Not tainted 4.4.71 #0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.493464] Stack : 80387554 00000000 00000001 803e0000 00000000 00000000 00000000 00000000
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.493464] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.493464] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.493464] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.493464] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.493464] ...
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.529579] Call Trace:[<80071a50>] 0x80071a50
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.534126] [<80071a50>] 0x80071a50
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.537641] [<800d24ac>] 0x800d24ac
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.541179] [<8009f02c>] 0x8009f02c
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.544712] [<800d2890>] 0x800d2890
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.548234] [<8009f974>] 0x8009f974
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.551766] [<8017ef9c>] 0x8017ef9c
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.555302] [<800d2f98>] 0x800d2f98
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.558817] [<800d6898>] 0x800d6898
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.562365] [<800cf9ec>] 0x800cf9ec
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.565890] [<80131484>] 0x80131484
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.569423] [<80131704>] 0x80131704
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.572996] [<80131954>] 0x80131954
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.576535] [<80158ba0>] 0x80158ba0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.580099] [<8015cdb0>] 0x8015cdb0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.583646] [<801598a4>] 0x801598a4
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.587187] [<8015aa2c>] 0x8015aa2c
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.590761] [<800da034>] 0x800da034
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.594295] [<800cebbc>] 0x800cebbc
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.597888] [<800d17e4>] 0x800d17e4
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.601417] [<800ef350>] 0x800ef350
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.604939] [<8011a4cc>] 0x8011a4cc
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.608448] [<801af9e0>] 0x801af9e0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.612010] [<800ec5e4>] 0x800ec5e4
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.615521] [<800ef350>] 0x800ef350
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.619041] [<800ef888>] 0x800ef888
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.622583] [<80082188>] 0x80082188
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.626108] [<80081b90>] 0x80081b90
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.629653] [<8007670c>] 0x8007670c
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.633217] [<8009f02c>] 0x8009f02c
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.636727] [<80083408>] 0x80083408
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.640247] [<8009f974>] 0x8009f974
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.643784] [<8009b3f8>] 0x8009b3f8
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.647302] [<80066c10>] 0x80066c10
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.650843] [<80060820>] 0x80060820
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.654362] [<80065060>] 0x80065060
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.657874]
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.659376] Mem-Info:
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.661710] active_anon:3065 inactive_anon:22 isolated_anon:0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.661710] active_file:165 inactive_file:172 isolated_file:0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.661710] unevictable:0 dirty:1 writeback:0 unstable:0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.661710] slab_reclaimable:248 slab_unreclaimable:1253
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.661710] mapped:33 shmem:300 pagetables:97 bounce:0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.661710] free:354 free_pcp:0 free_cma:0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.693684] Normal free:1416kB min:1024kB low:1280kB high:1536kB active_anon:12260kB inactive_anon:88kB active_file:660kB inactive_file:688kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:32768kB managed:28176kB mlocked:0kB dirty:4kB writeback:0kB mapped:132kB shmem:1200kB slab_reclaimable:992kB slab_unreclaimable:5012kB kernel_stack:448kB pagetables:388kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scMon Jul 10 08:41:43 2017 kern.warn kernel: [40390.737592] lowmem_reserve[]: 0 0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.740969] Normal: 2684kB (UME) 318kB (UM) 616kB (UM) 032kB 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 0*4096kB = 1416kB
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.753004] 639 total pagecache pages
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.756697] 0 pages in swap cache
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.760037] Swap cache stats: add 0, delete 0, find 0/0
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.765305] Free swap = 0kB
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.768202] Total swap = 0kB
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.771113] 8192 pages RAM
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.773835] 0 pages HighMem/MovableOnly
Mon Jul 10 08:41:43 2017 kern.warn kernel: [40390.777689] 1148 pages reserved
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.780863] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.789462] [ 504] 0 504 295 15 3 0 0 0 ubusd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.798150] [ 505] 0 505 224 10 3 0 0 0 askfirst
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.807090] [ 704] 0 704 313 41 3 0 0 0 logd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.815685] [ 713] 0 713 361 23 4 0 0 0 rpcd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.824283] [ 760] 0 760 426 41 4 0 0 0 netifd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.833055] [ 786] 0 786 355 37 5 0 0 0 odhcpd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.841828] [ 839] 0 839 265 11 3 0 0 0 dropbear
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.850774] [ 857] 0 857 988 130 4 0 0 0 uhttpd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.859532] [ 872] 65534 872 846 76 4 0 0 0 openvpn
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.868389] [ 1001] 0 1001 297 9 3 0 0 0 udhcpc
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.877155] [ 1003] 0 1003 255 13 3 0 0 0 odhcp6c
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.886012] [ 1128] 0 1128 298 10 4 0 0 0 ntpd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.894610] [ 1534] 0 1534 345 61 3 0 0 0 dynamic_dns_upd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.904168] [ 1650] 0 1650 422 31 3 0 0 0 hostapd
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.913027] [ 2226] 453 2226 562 316 4 0 0 0 dnsmasq
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.921886] [ 2406] 0 2406 298 10 3 0 0 0 sleep
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.930571] [ 2444] 0 2444 282 29 4 0 0 0 dropbear
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.939502] [ 3456] 0 3456 732 233 3 0 0 0 luci
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.948098] [ 3486] 0 3486 297 10 4 0 0 0 sh
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.956516] [ 3487] 0 3487 1569 1312 4 0 0 0 opkg
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.965111] [ 3490] 0 3490 649 165 4 0 0 0 luci
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.973710] [ 3491] 0 3491 649 164 4 0 0 0 luci
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.982307] [ 3509] 0 3509 297 10 3 0 0 0 sh
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.990731] [ 3510] 0 3510 791 47 4 0 0 0 wget
Mon Jul 10 08:41:43 2017 kern.info kernel: [40390.999313] [ 3511] 0 3511 297 10 3 0 0 0 grep
Mon Jul 10 08:41:43 2017 kern.info kernel: [40391.007910] [ 3520] 0 3520 298 11 4 0 0 0 sh
Mon Jul 10 08:41:43 2017 kern.err kernel: [40391.016318] Out of memory: Kill process 3487 (opkg) score 181 or sacrifice child
Mon Jul 10 08:41:43 2017 kern.err kernel: [40391.023786] Killed process 3487 (opkg) total-vm:6276kB, anon-rss:5224kB, file-rss:24kB
Mon Jul 10 08:41:44 2017 daemon.err uhttpd[857]: Killed

root@LEDE:~# opkg update Downloading http://downloads.lede-project.org/releases/17.01.2/targets/ar71xx/generic/packages/Packages.gz *** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.2/targets/ar71xx/generic/packages/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/base/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/base/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/luci/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/luci/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/packages/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/packages/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/routing/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.2/packages/mips_24kc/routing/Packages.gz

Collected errors:

root@LEDE:# free
total used free shared buffers cached
Mem: 28176 20732 7444 1200 560 3896
-/+ buffers/cache: 16276 11900
Swap: 0 0 0
root@LEDE:
# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 2.3M 2.3M 0 100% /rom
tmpfs 13.8M 1.2M 12.6M 9% /tmp
/dev/sda1 3.7G 266.2M 3.4G 7% /overlay
overlayfs:/overlay 3.7G 266.2M 3.4G 7% /
tmpfs 512.0K 0 512.0K 0% /dev

EDIT:
No luck with opkg_2017-05-03-04e279eb-1_mips_24kc.ipk

@openwrt-bot
Copy link
Author

bluewavenet:

@muhaha
I just downloaded the current release and re-flashed a brand new Ubiquiti airGateway. It works perfectly. As you can see from the following, I seem to have less free memory than you do and I have not installed anything or have anything extra running. I suggest you try a new re-flash to check nothing is corrupted.

\ LE \ / -----------------------------------------------------------
\ DE \ / Reboot (17.01.2, r3435-65eec8bd5f)
________/ -----------------------------------------------------------

root@LEDE:# free
total used free shared buffers cached
Mem: 28176 23748 4428 1116 2060 8284
-/+ buffers/cache: 13404 14772
Swap: 0 0 0
root@LEDE:
# df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 2.3M 2.3M 0 100% /rom
tmpfs 13.8M 1.0M 12.7M 8% /tmp
tmpfs 13.8M 48.0K 13.7M 0% /tmp/root
tmpfs 512.0K 0 512.0K 0% /dev
/dev/mtdblock5 3.9M 280.0K 3.6M 7% /overlay
overlayfs:/overlay 3.9M 280.0K 3.6M 7% /
root@LEDE:~#

@openwrt-bot
Copy link
Author

bluewavenet:

@muhaha
I think I see your problem. On booting up I get:
root@LEDE:~# opkg update
Downloading http://downloads.lede-project.org/releases/17.01.2/targets/ar71xx/generic/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/reboot_core
Downloading http://downloads.lede-project.org/releases/17.01.2/targets/ar71xx/generic/packages/Packages.sig
Signature check passed.
Downloading http://down.... etc

I then loaded up php7 and ran a script that used up loads of memory, taking free down to less than 1MB. Then I get your problem. There is obviously still a threshold where opkg runs out of memory. Try a reboot and see if it helps ;)

@openwrt-bot
Copy link
Author

bluewavenet:

@cypa
@Jo-Philipp Wich
Maybe there is still a small issue here, but I can work round it 100% by a reboot.
Perhaps getting opkg to check free first and if not enough suggest stopping some services and/or rebooting?...

@openwrt-bot
Copy link
Author

jow-:

Well, if a system is out of memory, it is out of memory. No amount of opkg optimizations will change that. I lowered the resource usage of opkg by 60-70% percent, thats as far as it will get.

If you still suffer from out of memory conditions then your environment simply cannot accomodate opkg anymore. Either try to reduce the amounts of feeds or consider precompiling images yourself.

@openwrt-bot
Copy link
Author

Markinus:

Hi!

I have the same problem on my TPLINK RE450. I'm restarting the system and the upgrade is not possible, console output:

BusyBox v1.25.1 () built-in shell (ash)

 _________
/        /\      _    ___ ___  ___

/ LE / \ | | | | | |
/ DE / \ | || _|| |) | _|
/
_/ LE \ |||/|| lede-project.org
\ \ DE /
\ LE \ / -----------------------------------------------------------
\ DE \ / Reboot (17.01.4, r3560-79f57e422d)
________/ -----------------------------------------------------------

root@LEDE:# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 2560 2560 0 100% /rom
tmpfs 30332 80 30252 0% /tmp
/dev/mtdblock3 2176 236 1940 11% /overlay
overlayfs:/overlay 2176 236 1940 11% /
tmpfs 512 0 512 0% /dev
root@LEDE:
# free
total used free shared buffers cached
Mem: 60664 43600 17064 80 2320 6048
-/+ buffers/cache: 35232 25432
Swap: 0 0 0
root@LEDE:# opkg update
Downloading http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/reboot_core
Downloading http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.sig
Signature check passed.
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/base/Packages.gz
Updated list of available packages in /var/opkg-lists/reboot_base
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/base/Packages.sig
Signature check passed.
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/luci/Packages.gz
Updated list of available packages in /var/opkg-lists/reboot_luci
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/luci/Packages.sig
Signature check passed.
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/reboot_packages
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/packages/Packages.sig
Signature check passed.
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/routing/Packages.gz
Updated list of available packages in /var/opkg-lists/reboot_routing
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/routing/Packages.sig
Signature check passed.
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/telephony/Packages.gz
Updated list of available packages in /var/opkg-lists/reboot_telephony
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/telephony/Packages.sig
Signature check passed.
root@LEDE:
# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 2560 2560 0 100% /rom
tmpfs 30332 568 29764 2% /tmp
/dev/mtdblock3 2176 236 1940 11% /overlay
overlayfs:/overlay 2176 236 1940 11% /
tmpfs 512 0 512 0% /dev
root@LEDE:# free
total used free shared buffers cached
Mem: 60664 44804 15860 568 2320 6756
-/+ buffers/cache: 35728 24936
Swap: 0 0 0
root@LEDE:
# opkg list upgradable
root@LEDE:# opkg list-upgradable
luci-lib-ip - git-17.290.79498-d3f0685-1 - git-17.298.45605-07e7ff0-1
luci-theme-bootstrap - git-17.290.79498-d3f0685-1 - git-17.298.45605-07e7ff0-1
dnsmasq - 2.78-1 - 2.78-2
luci-app-firewall - git-17.290.79498-d3f0685-1 - git-17.298.45605-07e7ff0-1
luci-proto-ppp - git-17.290.79498-d3f0685-1 - git-17.298.45605-07e7ff0-1
luci-mod-admin-full - git-17.290.79498-d3f0685-1 - git-17.298.45605-07e7ff0-1
luci-base - git-17.290.79498-d3f0685-1 - git-17.298.45605-07e7ff0-1
luci-proto-ipv6 - git-17.290.79498-d3f0685-1 - git-17.298.45605-07e7ff0-1
luci-lib-nixio - git-17.290.79498-d3f0685-1 - git-17.298.45605-07e7ff0-1
luci-lib-jsonc - git-17.290.79498-d3f0685-1 - git-17.298.45605-07e7ff0-1
luci - git-17.290.79498-d3f0685-1 - git-17.298.45605-07e7ff0-1
root@LEDE:
# opkg upgrade luci-base
Upgrading luci-base on root from git-17.290.79498-d3f0685-1 to git-17.298.45605-07e7ff0-1...
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/luci/luci-base_git-17.298.45605-07e7ff0-1_mips_24kc.ipk
Collected errors:

It looks like there is enough memory availible, where is the problem? Can someone explain it? With a sysctl -w vm.min_free_kbytes=0 the upgrade is possible.

@openwrt-bot
Copy link
Author

bluewavenet:

I thought the RE450 had 128MB ram? Free is only showing 60664KB with 15860 free.
I think vm.min_free_kbytes is set to 16384 by default leaving nothing left for allocation.
Is the upgrade successful when you set vm.min_free_kbytes=0 ?

@openwrt-bot
Copy link
Author

bobafetthotmail1:

Since your device does not have 32MB ram, and this bugreport was closed/solved months ago it would be better to open another bug report to track your issue.

@openwrt-bot
Copy link
Author

Markinus:

Yes, it works with set vm.min_free_kbytes=0.
I will open a new bug.

@openwrt-bot
Copy link
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant