OpenWrt/LEDE Project

  • Status Unconfirmed
  • Percent Complete
    0%
  • Task Type Bug Report
  • Category Packages
  • Assigned To No-one
  • Operating System All
  • Severity Low
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Drei Eck - 03.06.2021

FS#3849 - opkg: Feature request: Add possibility to read packages from a file instead from the command line.

I suggest to enhance `opkg` so that with a command line option it can read the list of packages to operate on (e.g. install, remove) from a file or from standard input rather than from the command line.

This would enable the build system to be re-written so that the build of images with a lot amount of packages compiled in succeeds. Now it bails out with `/usr/bin/env: Argument list too long`, since at one point all the packages are to be installed at once (and this cannot be split up to enable proper dependency handling), which will inevitably fail for a lot of packages (even with high stack size set with `ulimit`) due to a too long command line.

There are several reports of this in the forum; I refer to this thread "`/usr/bin/env: Argument list too long`-error persisting – OpenWRT 21.02.0-rc1 package installation creates a huge command line although far far from most packages selected", where a case is reported of a 210800 characters long command and ideas to change the build system.

Admin
Jo-Philipp Wich commented on 04.06.2021 09:20

Why not just use "xargs -r opkg ... < file.txt" ?

Drei Eck commented on 04.06.2021 09:28
Why not just use "xargs -r opkg ... < file.txt" ?

This would also generate a long command line internally; at the end there will be a call like "opkg ... pkg1.ipk pkg2.ipk ... pkgN.ipk". As I understand the issue with maximum length of command line arguments, it is something within Linux, nothing shell-specific. So as I understand that would not help. – Am I wrong?

Admin
Jo-Philipp Wich commented on 04.06.2021 09:35
This would also generate a long command line internally

From "man 1 xargs":

The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits.
Drei Eck commented on 04.06.2021 09:47
From "man 1 xargs":
>
> > The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits.

OK, thanks!, but then this would mean that opkg does not see all packages at once and cannot do dependency handling correctly.

We also cannot install with –force-depends, since it turns out every now and then that the configuration (make menuconfig) by itself does not always make sure that all dependencies are met or that conflicting packages are excluded, sometimes this only get noticed in the install step.

So opkg would need to know at once about all packages to be installed.

(It seems even that when opkg misses some packages it could refuse to install other packages because of some spurious "incompatible with the architectures configured". Below (cf. this forum post) is an output of the install step of make -j1 V=sc with a change to the makefile that installs every package one by one:

IPKG_NO_SCRIPT=1 IPKG_INSTROOT=/home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/build_dir/target-i386_pentium-mmx_musl/root-x86 TMPDIR=/home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/build_dir/target-i386_pentium-mmx_musl/root-x86/tmp /home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/staging_dir/host/bin/opkg --offline-root /home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/build_dir/target-i386_pentium-mmx_musl/root-x86 --force-postinstall --add-dest root:/ --add-arch all:100 --add-arch i386_pentium-mmx:200 install /home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/bin/targets/x86/geode/packages/base-files_1-r16046-59980f7aaf_i386_pentium-mmx.ipk
Unknown package 'base-files'.
Collected errors:
 * pkg_hash_fetch_best_installation_candidate: Packages for base-files found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package base-files.
make[2]: *** [package/Makefile:79: package/install] Error 255

)

Using xargs then as you cited can also break up the command line and might lead to similar problems, since it is not guaranteed that all packages that belong to each other are grouped into one call to opkg.

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing