New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FS#2495 - ucert rebuild within SDK segfaults on Debian 10 and Ubuntu 18.04 #7299
Comments
ynezz:
Can you provide something reproducible, like failing commands including the dummy key files and/or stacktrace?
That's very good ldd in Debian10, here it shows:
ldd staging_dir/host/bin/ucert
not a dynamic executable
|
cshoredaniel: Aargh! Lost the my writeup due to 'wrongtoken' error from FS. Let's do this again:
daniel@buildserver:~/Build/sdk-19.07-ucert$ file staging_dir/host/bin/ucert
staging_dir/host/bin/ucert: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=68de4d7aa97907c6d230b5a6ad9b87b55282417a, with debug_info, not stripped
To see the error in action:
rm -f /home/daniel/Build/sdk-19.07-ucert/build_dir/target-mips_24kc_musl/linux-ath79_generic/base-files/.configured_*
rm -f /home/daniel/Build/sdk-19.07-ucert/staging_dir/target-mips_24kc_musl/stamp/.base-files_installed
[ -s /home/daniel/Build/sdk-19.07-ucert/key-build -a -s /home/daniel/Build/sdk-19.07-ucert/key-build.pub ] || /home/daniel/Build/sdk-19.07-ucert/staging_dir/host/bin/usign -G -s /home/daniel/Build/sdk-19.07-ucert/key-build -p /home/daniel/Build/sdk-19.-c /home/daniel/Build/sdk-19.07-ucert/key-build.ucert -p /home/daniel/Build/sdk-19.07-ucert/key-build.pub -s /home/daniel/Build/sdk-19.07-ucert/key-build
make[3]: *** [Makefile:221: /home/daniel/Build/sdk-19.07-ucert/build_dir/target-mips_24kc_musl/linux-ath79_generic/base-files/.configured_b19dd3aae2bcca394b7e6453144e9502_8e081b74cf069e1e6800a5bbcbb282f0] Segmentation fault
make[3]: Leaving directory '/home/daniel/Build/sdk-19.07-ucert/feeds/base/package/base-files'
time: package/feeds/base/base-files/compile#0.20#0.07#0.24
|
cshoredaniel: Note that on initial extraction your observation of a script holds true, but after compilation it no longer is. |
cshoredaniel: Also note that make V=sc package/index sometimes succeeds so this may not be completely deterministic. |
ynezz: I was able to reproduce it even with the snapshot SDK, so it's not related to just 19.07 SDK. The problem is the rebuild of ucert under SDK, which then uses dynamic linker from the host which probably doesn't work properly with the libc from the SDK (ABI version 2.6.32), host libc has ABI version 3.2.0.
==1425== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==1425== Bad permissions for mapped region at address 0x0
==1425== at 0x0: ???
==1425== by 0x40040DB: dl_main (rtld.c:2199)
==1425== by 0x401864F: _dl_sysdep_start (dl-sysdep.c:253)
==1425== by 0x4002117: _dl_start_final (rtld.c:415)
==1425== by 0x4002117: _dl_start (rtld.c:522)
==1425== by 0x4001097: ??? (in /lib/x86_64-linux-gnu/ld-2.28.so)
==1425== Jump to the invalid address stated on the next line
==1425== at 0x1036: ???
==1425== by 0x4F83AE4: ??? (in /build/openwrt-sdk-ath79-generic_gcc-7.4.0_musl.Linux-x86_64/staging_dir/host/lib/libc.so.6)
==1425== by 0x4F0C866: time (in /build/openwrt-sdk-ath79-generic_gcc-7.4.0_musl.Linux-x86_64/staging_dir/host/lib/libc.so.6)
==1425== by 0x400BAD7: elf_machine_lazy_rel (dl-machine.h:576)
==1425== by 0x400BAD7: elf_dynamic_do_Rela (do-rel.h:77)
==1425== by 0x400BAD7: _dl_relocate_object (dl-reloc.c:258)
==1425== by 0x40040DB: dl_main (rtld.c:2199)
==1425== by 0x401864F: _dl_sysdep_start (dl-sysdep.c:253)
==1425== by 0x4002117: _dl_start_final (rtld.c:415)
==1425== by 0x4002117: _dl_start (rtld.c:522)
==1425== by 0x4001097: ??? (in /lib/x86_64-linux-gnu/ld-2.28.so)
==1425== Address 0x1036 is not stack'd, malloc'd or (recently) free'd
|
jow-: We should inhibit the building of host utilities in the SDK. The SDK is meant to provide a precompiled feature-fixed toolchain environment. Compilation of host utilities (not package host builds) or toolchain components within the SDK is undefined due to the use of a bundled and shipped libc. I think the best course forward here would be to prevent enabling signed package lists on a non-ucert equipped SDK. |
stangri: On Ubuntu 18.04.3 and 19.07.0 I'm also getting a segmentation fault on ucert (while trying to build package base-files): |
ynezz:
Nope, someone has to fix it. Meanwhile disable signing of the packages //CONFIG_SIGNED_PACKAGES// under SDK or if you want signed packages, you've to build from scratch, not within SDK. |
recycler: Same here for Ubuntu 20.04.2 Seems that downloaded ucert is relying on certain library versions.
A solution could be a static binary for download. |
tmn505: This will affect all distro versions which weren't used for building particular SDK. On Arch Linux it doesn't segfault and gives a little hint what's wrong: |
This problem is still happening on Ubuntu 20.04 when using the OpenWRT SDK v19.07.10. Running (side-note, for me, it looks like WorkaroundUnfortunately, it looks like @tmn505's workaround mentioned in #7299 (comment) was lost when migrating to GitHub Issues. It's probably easier to just disable # wait until OpenWRT build creates the broken ucert bin and stampfiles
make package/ucert/compile
# copies back the original working ucert bin from the SDK .tar.xz archive
tar xf "../<PATH_TO_YOUR_SDK__DOWNLOAD_HERE>.tar.xz" --strip-components 1 --directory "." --wildcards '*/staging_dir/host/bin/*'
# continues make
make |
This is the workaround: https://paste.debian.net/1241483 |
Hi @tmn505, you're a rock-star for still having that patch, even from a year ago! Would you be okay with me submitting your modified patch as a PR? I'd be happy to add your name/email as a I've adapted your patch so it works on the current OpenWRT It looks like this issue is still affecting the |
Feel free to submit. |
Currently, when building packages with the OpenWRT SDK, the pre-packaged binaries in `./staging_dir/host/bin` may be overwritten with newly created binaries. However, if the SDK was built with an older GLIBC version, the new binary will not work with the SDK, as it's built for the host's GLIBC version. This would result in a Segmentation Fault on older OSes. Newer OSes print the better following error: > ``` > ./staging_dir/host/bin/.ucert.bin: > ./staging_dir/host/bin/../lib/libc.so.6: > version `GLIBC_2.33' not found > (required by ./staging_dir/host/bin/../lib/libblobmsg_json.so) > ``` To avoid this, this commit adds the `./staging_dir/host/stamp/*` files to the SDK, and modifies `include/host-build.mk` to avoid recompiling and reinstalling host tools if CONFIG_IN_SDK==y and the HOST_STAMP_INSTALLED stamp file already exists. Fixes: openwrt#7299 Co-authored-by: Tomasz Maciej Nowak <tmn505@gmail.com> Signed-off-by: Alois Klink <alois@aloisklink.com>
cshoredaniel:
Using the 19.07-SNAPSHOT SDK build Sept 12 (last successful build by the buildbots it looks like), and current openwrt-19.07 branch on the feeds, if doing signed build, the build fails due to ucert segfaulting (this is on Debian 10).
ldd staging_dir/host/bin/ucert shows:
I'm wondering if the problem is due to using host instead SDK libraries.
The text was updated successfully, but these errors were encountered: