OpenWrt/LEDE Project

  • Status Unconfirmed   Reopened
  • Percent Complete
    0%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Medium
  • Priority Very Low
  • Reported Version openwrt-19.07
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Juliusz Chroboczek - 14.01.2020
Last edited by Jo-Philipp Wich - 15.01.2020

FS#2734 - Opkg update fails although router has enough memory

19.07.0 running on WNDR3700v2, 32MB of memory. The machine is certainly not short on memory: 

                total        used        free      shared  buff/cache   available
  Mem:          59560       26912       24108         956        8540       16564
  Swap:             0           0           0

Running “opkg update” a first time works fine. However, if I run “opkg update” a second time, it reports: 

  Collected errors:
   * pkg_hash_add_from_file: Failed to open /var/opkg-lists/openwrt_routing: Out of memory.

After doing “rm /var/opkg-lists/openwrt_*”, everything works fine again.

Juliusz Chroboczek commented on 14.01.2020 14:38

Strace indicates that it's `fork` that fails: 

  pipe([4, 5])                            = 4
  fork()                                  = -1 ENOMEM (Out of memory)
Juliusz Chroboczek commented on 14.01.2020 14:41

Enabling memory overcommit (which I'm not recommending should be the default) successfully works around the issue: 

  sysctl -w vm.overcommit_memory=1
Admin
Jo-Philipp Wich commented on 14.01.2020 16:34

I don't see a solution to this. Opkg was already heavily patched and modified to use less RAM, even introducing a rather slow multi-pass list parsing algorithm to avoid keeping the entire dependency graph in memory. The lists are just big and keep growing, so devices with 64MB RAM and lower are nearing the end of their usefulness.

Donk commented on 14.01.2020 16:45

Same issue here with a x86-64 image in Virtualbox with 128MB Ram

Juliusz Chroboczek commented on 14.01.2020 17:48

Jo-Philipp, may I suggest that you reopen this report? Just because we don't see a solution yet doesn't mean the bug should be closed.

There are two points I'd like to add.

First of all, "opkg install" and "opkg list-installable" work while "opkg update" fail. Is opkg perhaps keeping two copies (the old and the new one) in memory at the same time?

Second, the failure is not due to lack of memory – it is due to fork failing due to the large amount of writeable pages. To me, this implies that either opkg should avoid forking (by linking with libz instead of running gzip – see gzip_fdopen in libbb/gzip.c), or fork before it allocates the data structures (and keep the pipe around for later usage), or use a custom memory allocator that marks the pages as unwriteable using mprotect.

Admin
Jo-Philipp Wich commented on 15.01.2020 06:58

Sure, will reopen it if you prefer.

Juliusz Chroboczek commented on 16.01.2020 13:36

There might be a memory leak.

A first run of opkg update indicates all memory has been freed; running it a second time, though, indicates a leak of almost 4MB: 

  ==6835==
  ==6835== HEAP SUMMARY:
  ==6835==     in use at exit: 3,674,125 bytes in 71,915 blocks
  ==6835==   total heap usage: 164,305 allocs, 92,390 frees, 27,717,132 bytes allo  cated
  ==6835==
  ==6835== 920 (144 direct, 776 indirect) bytes in 9 blocks are definitely lost in   loss record 5 of 19
  ==6835==    at 0x483577F: malloc (vg_replace_malloc.c:309)
  ==6835==    by 0x122C91: xmalloc (xfuncs.c:30)
  ==6835==    by 0x11C0BA: parse_alternatives (pkg_parse.c:185)
  ==6835==    by 0x11C0BA: pkg_parse_line (pkg_parse.c:212)
  ==6835==    by 0x11E4BE: parse_from_stream_nomalloc (parse_util.c:162)
  ==6835==    by 0x11B86C: pkg_hash_add_from_file (pkg_hash.c:126)
  ==6835==    by 0x11BAC9: pkg_hash_load_feeds (pkg_hash.c:196)
  ==6835==    by 0x10FDBF: main (opkg-cl.c:433)
  ==6835==
  ==6835== 1,144 bytes in 47 blocks are definitely lost in loss record 6 of 19
  ==6835==    at 0x4837D7B: realloc (vg_replace_malloc.c:836)
  ==6835==    by 0x119D7F: parse_providelist (pkg_depends.c:629)
  ==6835==    by 0x11C4CD: pkg_parse_line (pkg_parse.c:296)
  ==6835==    by 0x11E4BE: parse_from_stream_nomalloc (parse_util.c:162)
  ==6835==    by 0x11B86C: pkg_hash_add_from_file (pkg_hash.c:126)
  ==6835==    by 0x11BAC9: pkg_hash_load_feeds (pkg_hash.c:196)
  ==6835==    by 0x10FDBF: main (opkg-cl.c:433)
  ==6835==
  ==6835== 31,056 (15,264 direct, 15,792 indirect) bytes in 477 blocks are definitely lost in loss record 11 of 19
  ==6835==    at 0x48356AF: malloc (vg_replace_malloc.c:308)
  ==6835==    by 0x4837DE7: realloc (vg_replace_malloc.c:836)
  ==6835==    by 0x11A069: parse_deplist (pkg_depends.c:816)
  ==6835==    by 0x11C705: pkg_parse_line (pkg_parse.c:321)
  ==6835==    by 0x11E4BE: parse_from_stream_nomalloc (parse_util.c:162)
  ==6835==    by 0x11B86C: pkg_hash_add_from_file (pkg_hash.c:126)
  ==6835==    by 0x11BAC9: pkg_hash_load_feeds (pkg_hash.c:196)
  ==6835==    by 0x10FDBF: main (opkg-cl.c:433)
  ==6835==
  ==6835== 792,670 (306,096 direct, 486,574 indirect) bytes in 4,057 blocks are definitely lost in loss record 17 of 19
  ==6835==    at 0x4837D7B: realloc (vg_replace_malloc.c:836)
  ==6835==    by 0x11A069: parse_deplist (pkg_depends.c:816)
  ==6835==    by 0x11C705: pkg_parse_line (pkg_parse.c:321)
  ==6835==    by 0x11E4BE: parse_from_stream_nomalloc (parse_util.c:162)
  ==6835==    by 0x11B86C: pkg_hash_add_from_file (pkg_hash.c:126)
  ==6835==    by 0x11BAC9: pkg_hash_load_feeds (pkg_hash.c:196)
  ==6835==    by 0x10FDBF: main (opkg-cl.c:433)
  ==6835==
  ==6835== 2,848,303 (599,456 direct, 2,248,847 indirect) bytes in 6,812 blocks are definitely lost in loss record 19 of 19
  ==6835==    at 0x4837B65: calloc (vg_replace_malloc.c:762)
  ==6835==    by 0x122D20: xcalloc (xfuncs.c:46)
  ==6835==    by 0x116B77: pkg_new (pkg.c:95)
  ==6835==    by 0x11B81E: pkg_hash_add_from_file (pkg_hash.c:121)
  ==6835==    by 0x11BAC9: pkg_hash_load_feeds (pkg_hash.c:196)
  ==6835==    by 0x10FDBF: main (opkg-cl.c:433)
  ==6835==
  ==6835== LEAK SUMMARY:
  ==6835==    definitely lost: 922,104 bytes in 11,402 blocks
  ==6835==    indirectly lost: 2,751,989 bytes in 60,511 blocks
  ==6835==      possibly lost: 0 bytes in 0 blocks
  ==6835==    still reachable: 32 bytes in 2 blocks
  ==6835==         suppressed: 0 bytes in 0 blocks

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing