Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#3849 - opkg: Feature request: Add possibility to read packages from a file instead from the command line. #8856

Open
openwrt-bot opened this issue Jun 3, 2021 · 4 comments
Labels
core packages pull request/issue for core (in-tree) packages flyspray

Comments

@openwrt-bot
Copy link

dreieck:

I suggest to enhance opkg so that with a command line option it can read the list of packages to operate on (e.g. install, remove) from a file or from standard input rather than from the command line.

This would enable the build system to be re-written so that the build of images with a lot amount of packages compiled in succeeds. Now it bails out with /usr/bin/env: Argument list too long, since at one point all the packages are to be installed at once (and this cannot be split up to enable proper dependency handling), which will inevitably fail for a lot of packages (even with high stack size set with ulimit) due to a too long command line.

There are several reports of this in the forum; I refer to [[https://forum.openwrt.org/t/96942|this thread "/usr/bin/env: Argument list too long-error persisting – OpenWRT 21.02.0-rc1 package installation creates a huge command line although far far from most packages selected"]], where a case is reported of a 210800 characters long command and ideas to change the build system.

@openwrt-bot
Copy link
Author

jow-:

Why not just use "xargs -r opkg ... < file.txt" ?

@openwrt-bot
Copy link
Author

dreieck:

Why not just use "''xargs -r opkg ... < file.txt''" ?

This would also generate a long command line internally; at the end there will be a call like "''opkg ... pkg1.ipk pkg2.ipk ... pkgN.ipk''". As I understand the issue with maximum length of command line arguments, it is something within Linux, nothing shell-specific. So as I understand that would not help. -- Am I wrong?

@openwrt-bot
Copy link
Author

jow-:

This would also generate a long command line internally

From "man 1 xargs":

The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits.

@openwrt-bot
Copy link
Author

dreieck:

From "''man 1 xargs''":

The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits.

OK, thanks!, but then this would mean that ''opkg'' does not see all packages at once and cannot do dependency handling correctly.

We also cannot install with ''--force-depends'', since it turns out every now and then that the configuration (''make menuconfig'') by itself does not always make sure that all dependencies are met or that conflicting packages are excluded, sometimes this only get noticed in the install step.

So ''opkg'' would need to know at once about //all// packages to be installed.

(It seems even that when ''opkg'' misses some packages it could refuse to install other packages because of some spurious "incompatible with the architectures configured". Below (cf. [[https://forum.openwrt.org/t/96942/13|this forum post]]) is an output of the install step of ''make -j1 V=sc'' with a change to the makefile that installs every package one by one:

IPKG_NO_SCRIPT=1 IPKG_INSTROOT=/home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/build_dir/target-i386_pentium-mmx_musl/root-x86 TMPDIR=/home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/build_dir/target-i386_pentium-mmx_musl/root-x86/tmp /home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/staging_dir/host/bin/opkg --offline-root /home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/build_dir/target-i386_pentium-mmx_musl/root-x86 --force-postinstall --add-dest root:/ --add-arch all:100 --add-arch i386_pentium-mmx:200 install /home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/bin/targets/x86/geode/packages/base-files_1-r16046-59980f7aaf_i386_pentium-mmx.ipk Unknown package 'base-files'. Collected errors: * pkg_hash_fetch_best_installation_candidate: Packages for base-files found, but incompatible with the architectures configured * opkg_install_cmd: Cannot install package base-files. make[2]: *** [package/Makefile:79: package/install] Error 255

)

Using ''xargs'' then as you cited can also break up the command line and might lead to similar problems, since it is not guaranteed that all packages that belong to each other are grouped into one call to ''opkg''.

@aparcar aparcar added the core packages pull request/issue for core (in-tree) packages label Feb 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core packages pull request/issue for core (in-tree) packages flyspray
Projects
None yet
Development

No branches or pull requests

2 participants