Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2495 - ucert rebuild within SDK segfaults on Debian 10 and Ubuntu 18.04 #7299

Open
openwrt-bot opened this issue Sep 15, 2019 · 14 comments · May be fixed by #10255
Open

FS#2495 - ucert rebuild within SDK segfaults on Debian 10 and Ubuntu 18.04 #7299

openwrt-bot opened this issue Sep 15, 2019 · 14 comments · May be fixed by #10255
Labels

Comments

@openwrt-bot
Copy link

cshoredaniel:

Using the 19.07-SNAPSHOT SDK build Sept 12 (last successful build by the buildbots it looks like), and current openwrt-19.07 branch on the feeds, if doing signed build, the build fails due to ucert segfaulting (this is on Debian 10).

ldd staging_dir/host/bin/ucert shows:

    linux-vdso.so.1 (0x00007ffd03d63000)
    libubox.so => /home/daniel/Build/sdk-19.07-ath79/staging_dir/host/lib/libubox.so (0x00007ff0db3b0000)
    libblobmsg_json.so => /home/daniel/Build/sdk-19.07-ath79/staging_dir/host/lib/libblobmsg_json.so (0x00007ff0db3a8000)
    libjson-c.so.2 => /home/daniel/Build/sdk-19.07-ath79/staging_dir/host/lib/libjson-c.so.2 (0x00007ff0db398000)
    libc.so.6 => /home/daniel/Build/sdk-19.07-ath79/staging_dir/host/lib/libc.so.6 (0x00007ff0daff8000)
    /lib64/ld-linux-x86-64.so.2 (0x00007ff0db3c8000)

I'm wondering if the problem is due to using host instead SDK libraries.

@openwrt-bot
Copy link
Author

ynezz:

build fails due to ucert segfaulting (this is on Debian 10).

Can you provide something reproducible, like failing commands including the dummy key files and/or stacktrace?

ldd staging_dir/host/bin/ucert shows

That's very good ldd in Debian10, here it shows:

ldd staging_dir/host/bin/ucert not a dynamic executable

file staging_dir/host/bin/ucert
ucert: Bourne-Again shell script, ASCII text executable

@openwrt-bot
Copy link
Author

cshoredaniel:

Aargh! Lost the my writeup due to 'wrongtoken' error from FS. Let's do this again:

daniel@buildserver:~/Build/sdk-19.07-ucert$ file staging_dir/host/bin/ucert staging_dir/host/bin/ucert: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=68de4d7aa97907c6d230b5a6ad9b87b55282417a, with debug_info, not stripped

To see the error in action:

  • Extract the ath79 19.07-SNAPSHOT SDK
  • cd to the sdk dir
  • ./scripts/feeds update && ./scripts/feeds install -a
  • make menuconfig
  • ensure CONFIG_SIGNED_PACKAGES=y
  • recommend CONFIG_BUILD_LOG
  • recommend CONFIG_ALL is unset
  • make -j9 (or whatever suits your build)
  • You'll get a mysterious, unhelpful error
  • make V=sc
rm -f /home/daniel/Build/sdk-19.07-ucert/build_dir/target-mips_24kc_musl/linux-ath79_generic/base-files/.configured_* rm -f /home/daniel/Build/sdk-19.07-ucert/staging_dir/target-mips_24kc_musl/stamp/.base-files_installed [ -s /home/daniel/Build/sdk-19.07-ucert/key-build -a -s /home/daniel/Build/sdk-19.07-ucert/key-build.pub ] || /home/daniel/Build/sdk-19.07-ucert/staging_dir/host/bin/usign -G -s /home/daniel/Build/sdk-19.07-ucert/key-build -p /home/daniel/Build/sdk-19.-c /home/daniel/Build/sdk-19.07-ucert/key-build.ucert -p /home/daniel/Build/sdk-19.07-ucert/key-build.pub -s /home/daniel/Build/sdk-19.07-ucert/key-build make[3]: *** [Makefile:221: /home/daniel/Build/sdk-19.07-ucert/build_dir/target-mips_24kc_musl/linux-ath79_generic/base-files/.configured_b19dd3aae2bcca394b7e6453144e9502_8e081b74cf069e1e6800a5bbcbb282f0] Segmentation fault make[3]: Leaving directory '/home/daniel/Build/sdk-19.07-ucert/feeds/base/package/base-files' time: package/feeds/base/base-files/compile#0.20#0.07#0.24

@openwrt-bot
Copy link
Author

cshoredaniel:

Note that on initial extraction your observation of a script holds true, but after compilation it no longer is.

@openwrt-bot
Copy link
Author

cshoredaniel:

Also note that make V=sc package/index sometimes succeeds so this may not be completely deterministic.

@openwrt-bot
Copy link
Author

ynezz:

I was able to reproduce it even with the snapshot SDK, so it's not related to just 19.07 SDK. The problem is the rebuild of ucert under SDK, which then uses dynamic linker from the host which probably doesn't work properly with the libc from the SDK (ABI version 2.6.32), host libc has ABI version 3.2.0.

==1425== Process terminating with default action of signal 11 (SIGSEGV): dumping core ==1425== Bad permissions for mapped region at address 0x0 ==1425== at 0x0: ??? ==1425== by 0x40040DB: dl_main (rtld.c:2199) ==1425== by 0x401864F: _dl_sysdep_start (dl-sysdep.c:253) ==1425== by 0x4002117: _dl_start_final (rtld.c:415) ==1425== by 0x4002117: _dl_start (rtld.c:522) ==1425== by 0x4001097: ??? (in /lib/x86_64-linux-gnu/ld-2.28.so) ==1425== Jump to the invalid address stated on the next line ==1425== at 0x1036: ??? ==1425== by 0x4F83AE4: ??? (in /build/openwrt-sdk-ath79-generic_gcc-7.4.0_musl.Linux-x86_64/staging_dir/host/lib/libc.so.6) ==1425== by 0x4F0C866: time (in /build/openwrt-sdk-ath79-generic_gcc-7.4.0_musl.Linux-x86_64/staging_dir/host/lib/libc.so.6) ==1425== by 0x400BAD7: elf_machine_lazy_rel (dl-machine.h:576) ==1425== by 0x400BAD7: elf_dynamic_do_Rela (do-rel.h:77) ==1425== by 0x400BAD7: _dl_relocate_object (dl-reloc.c:258) ==1425== by 0x40040DB: dl_main (rtld.c:2199) ==1425== by 0x401864F: _dl_sysdep_start (dl-sysdep.c:253) ==1425== by 0x4002117: _dl_start_final (rtld.c:415) ==1425== by 0x4002117: _dl_start (rtld.c:522) ==1425== by 0x4001097: ??? (in /lib/x86_64-linux-gnu/ld-2.28.so) ==1425== Address 0x1036 is not stack'd, malloc'd or (recently) free'd

@openwrt-bot
Copy link
Author

jow-:

We should inhibit the building of host utilities in the SDK. The SDK is meant to provide a precompiled feature-fixed toolchain environment. Compilation of host utilities (not package host builds) or toolchain components within the SDK is undefined due to the use of a bundled and shipped libc.

I think the best course forward here would be to prevent enabling signed package lists on a non-ucert equipped SDK.

@openwrt-bot
Copy link
Author

stangri:

On Ubuntu 18.04.3 and 19.07.0 I'm also getting a segmentation fault on ucert (while trying to build package base-files):

SDK_mvebu.19.07.0/staging_dir/host/bin/ucert -I -c SDK_mvebu.19.07.0/key-build.ucert -p SDK_mvebu.19.07.0/key-build.pub -s SDK_mvebu.19.07.0/key-build
Segmentation fault (core dumped)

jow -- any updates on this?

@openwrt-bot
Copy link
Author

ynezz:

any updates on this

Nope, someone has to fix it. Meanwhile disable signing of the packages //CONFIG_SIGNED_PACKAGES// under SDK or if you want signed packages, you've to build from scratch, not within SDK.

@openwrt-bot
Copy link
Author

recycler:

Same here for Ubuntu 20.04.2

Seems that downloaded ucert is relying on certain library versions.

ldd ./staging_dir/host/bin/ucert
linux-vdso.so.1 (0x00007ffd69dae000)
libubox.so => /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/libubox.so (0x00007f743222b000)
libblobmsg_json.so => /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/libblobmsg_json.so (0x00007f7432226000)
libc.so.6 => /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/libc.so.6 (0x00007f7432065000)
libjson-c.so.5 => /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/libjson-c.so.5 (0x00007f7432052000)
/lib64/ld-linux-x86-64.so.2 (0x00007f7432245000)

(gdb) run -I -c /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build.ucert -p /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build.pub -s /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build
Starting program: /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/bin/ucert -I -c /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build.ucert -p /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build.pub -s /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7e18133 in __libc_start_main () from /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/libc.so.6
(gdb) backtrace
#0 0x00007ffff7e18133 in __libc_start_main () from /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/libc.so.6
#1 0x0000555555556a7e in _start () at /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/build_dir/hostpkg/ucert-2020-05-24-00b921d8/ucert.c:625

A solution could be a static binary for download.

@openwrt-bot
Copy link
Author

tmn505:

This will affect all distro versions which weren't used for building particular SDK. On Arch Linux it doesn't segfault and gives a little hint what's wrong:
/home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/bin/ucert: /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/lib/libc.so.6: version GLIBC_2.33' not found (required by /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/bin/ucert) /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/bin/ucert: /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/lib/libc.so.6: version GLIBC_2.33' not found (required by /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/lib/libjson-c.so.5)
that's on ramips SDK.
I created a workaround (see attachment) partly following Jo's advice. The workaround copies install stamps when creating SDK. Then if SDK is used with package which has a host part, the host part build is omitted if install stamp is present.

@aloisklink
Copy link

aloisklink commented May 20, 2022

This problem is still happening on Ubuntu 20.04 when using the OpenWRT SDK v19.07.10.

Running make overwrites the working ucert binary from the SDK with a new one, which is broken and causes a segmentation fault when creating base-files.

(side-note, for me, it looks like fwtool, usign, and jshn are also compiled and overwrite the SDK bins, but only jshn and ucert have segmentation faults when running jshn --help)

Workaround

Unfortunately, it looks like @tmn505's workaround mentioned in #7299 (comment) was lost when migrating to GitHub Issues.

It's probably easier to just disable CONFIG_SIGNED_PACKAGES so you don't even need to use ucert, but if you do want to sign packages, my work around is:

# wait until OpenWRT build creates the broken ucert bin and stampfiles
make package/ucert/compile
# copies back the original working ucert bin from the SDK .tar.xz archive
tar xf "../<PATH_TO_YOUR_SDK__DOWNLOAD_HERE>.tar.xz" --strip-components 1 --directory "." --wildcards '*/staging_dir/host/bin/*'
# continues make
make

@tmn505
Copy link
Contributor

tmn505 commented May 20, 2022

This is the workaround: https://paste.debian.net/1241483
It needs to be applied before SDK is created, otherwise the stamps are not there and the workaround won't work. You could try to patch the existing SDK but You'll have copy/create stamps Yourself.

@aloisklink
Copy link

Hi @tmn505, you're a rock-star for still having that patch, even from a year ago!

Would you be okay with me submitting your modified patch as a PR? I'd be happy to add your name/email as a Co-authored-by: ... and/or Suggested-by: ....

I've adapted your patch so it works on the current OpenWRT master branch and it seems to work in my testing aloisklink@36c4ede

It looks like this issue is still affecting the master branch: I tested building an SDK using Ubuntu 20.04, then tried using the SDK to build base-files on the newer Ubuntu 22.04, which failed as ucert returned a version `GLIBC_2.33' not found error.

@tmn505
Copy link
Contributor

tmn505 commented Jul 11, 2022

Feel free to submit.

@aloisklink aloisklink linked a pull request Jul 12, 2022 that will close this issue
aloisklink added a commit to aloisklink/openwrt that referenced this issue Jul 19, 2022
Currently, when building packages with the OpenWRT SDK,
the pre-packaged binaries in `./staging_dir/host/bin` may be
overwritten with newly created binaries.

However, if the SDK was built with an older GLIBC version,
the new binary will not work with the SDK, as it's built for the host's
GLIBC version. This would result in a Segmentation Fault on older OSes.
Newer OSes print the better following error:

> ```
> ./staging_dir/host/bin/.ucert.bin:
>      ./staging_dir/host/bin/../lib/libc.so.6:
>      version `GLIBC_2.33' not found
>      (required by ./staging_dir/host/bin/../lib/libblobmsg_json.so)
> ```

To avoid this, this commit adds the `./staging_dir/host/stamp/*` files
to the SDK, and modifies `include/host-build.mk` to avoid recompiling
and reinstalling host tools if CONFIG_IN_SDK==y and the
HOST_STAMP_INSTALLED stamp file already exists.

Fixes: openwrt#7299
Co-authored-by: Tomasz Maciej Nowak <tmn505@gmail.com>
Signed-off-by: Alois Klink <alois@aloisklink.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants