OpenWrt/LEDE Project

  • Status New
  • Percent Complete
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Low
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Daniel Dickinson - 15.09.2019
Last edited by Petr Štetiar - 17.09.2019

FS#2495 - ucert rebuild within SDK segfaults on Debian 10 and Ubuntu 18.04

Using the 19.07-SNAPSHOT SDK build Sept 12 (last successful build by the buildbots it looks like), and current openwrt-19.07 branch on the feeds, if doing signed build, the build fails due to ucert segfaulting (this is on Debian 10).

ldd staging_dir/host/bin/ucert shows: (0x00007ffd03d63000) => /home/daniel/Build/sdk-19.07-ath79/staging_dir/host/lib/ (0x00007ff0db3b0000) => /home/daniel/Build/sdk-19.07-ath79/staging_dir/host/lib/ (0x00007ff0db3a8000) => /home/daniel/Build/sdk-19.07-ath79/staging_dir/host/lib/ (0x00007ff0db398000) => /home/daniel/Build/sdk-19.07-ath79/staging_dir/host/lib/ (0x00007ff0daff8000)
      /lib64/ (0x00007ff0db3c8000)

I’m wondering if the problem is due to using host instead SDK libraries.

Petr Štetiar commented on 16.09.2019 21:26
build fails due to ucert segfaulting (this is on Debian 10).

Can you provide something reproducible, like failing commands including the dummy key files and/or stacktrace?

ldd staging_dir/host/bin/ucert shows

That's very good ldd in Debian10, here it shows:

ldd staging_dir/host/bin/ucert
not a dynamic executable

file staging_dir/host/bin/ucert
ucert: Bourne-Again shell script, ASCII text executable
Daniel Dickinson commented on 16.09.2019 23:58

Aargh! Lost the my writeup due to 'wrongtoken' error from FS. Let's do this again:

daniel@buildserver:~/Build/sdk-19.07-ucert$ file staging_dir/host/bin/ucert 
staging_dir/host/bin/ucert: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/, for GNU/Linux 3.2.0, BuildID[sha1]=68de4d7aa97907c6d230b5a6ad9b87b55282417a, with debug_info, not stripped

To see the error in action:

  1. Extract the ath79 19.07-SNAPSHOT SDK
  2. cd to the sdk dir
  3. ./scripts/feeds update && ./scripts/feeds install -a
  4. make menuconfig
  6. recommend CONFIG_BUILD_LOG
  7. recommend CONFIG_ALL is unset
  8. make -j9 (or whatever suits your build)
  9. You'll get a mysterious, unhelpful error
  10. make V=sc
rm -f /home/daniel/Build/sdk-19.07-ucert/build_dir/target-mips_24kc_musl/linux-ath79_generic/base-files/.configured_*
rm -f /home/daniel/Build/sdk-19.07-ucert/staging_dir/target-mips_24kc_musl/stamp/.base-files_installed
[ -s /home/daniel/Build/sdk-19.07-ucert/key-build -a -s /home/daniel/Build/sdk-19.07-ucert/ ] || /home/daniel/Build/sdk-19.07-ucert/staging_dir/host/bin/usign -G -s /home/daniel/Build/sdk-19.07-ucert/key-build -p /home/daniel/Build/sdk-19.-c /home/daniel/Build/sdk-19.07-ucert/key-build.ucert -p /home/daniel/Build/sdk-19.07-ucert/ -s /home/daniel/Build/sdk-19.07-ucert/key-build
make[3]: *** [Makefile:221: /home/daniel/Build/sdk-19.07-ucert/build_dir/target-mips_24kc_musl/linux-ath79_generic/base-files/.configured_b19dd3aae2bcca394b7e6453144e9502_8e081b74cf069e1e6800a5bbcbb282f0] Segmentation fault                            
make[3]: Leaving directory '/home/daniel/Build/sdk-19.07-ucert/feeds/base/package/base-files'                                
time: package/feeds/base/base-files/compile#0.20#0.07#0.24
Daniel Dickinson commented on 16.09.2019 23:59

Note that on initial extraction your observation of a script holds true, but after compilation it no longer is.

Daniel Dickinson commented on 17.09.2019 00:01

Also note that make V=sc package/index sometimes succeeds so this may not be completely deterministic.

Petr Štetiar commented on 17.09.2019 21:29

I was able to reproduce it even with the snapshot SDK, so it's not related to just 19.07 SDK. The problem is the rebuild of ucert under SDK, which then uses dynamic linker from the host which probably doesn't work properly with the libc from the SDK (ABI version 2.6.32), host libc has ABI version 3.2.0.

==1425== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==1425==  Bad permissions for mapped region at address 0x0
==1425==    at 0x0: ???
==1425==    by 0x40040DB: dl_main (rtld.c:2199)
==1425==    by 0x401864F: _dl_sysdep_start (dl-sysdep.c:253)
==1425==    by 0x4002117: _dl_start_final (rtld.c:415)
==1425==    by 0x4002117: _dl_start (rtld.c:522)
==1425==    by 0x4001097: ??? (in /lib/x86_64-linux-gnu/
==1425== Jump to the invalid address stated on the next line
==1425==    at 0x1036: ???
==1425==    by 0x4F83AE4: ??? (in /build/openwrt-sdk-ath79-generic_gcc-7.4.0_musl.Linux-x86_64/staging_dir/host/lib/
==1425==    by 0x4F0C866: time (in /build/openwrt-sdk-ath79-generic_gcc-7.4.0_musl.Linux-x86_64/staging_dir/host/lib/
==1425==    by 0x400BAD7: elf_machine_lazy_rel (dl-machine.h:576)
==1425==    by 0x400BAD7: elf_dynamic_do_Rela (do-rel.h:77)
==1425==    by 0x400BAD7: _dl_relocate_object (dl-reloc.c:258)
==1425==    by 0x40040DB: dl_main (rtld.c:2199)
==1425==    by 0x401864F: _dl_sysdep_start (dl-sysdep.c:253)
==1425==    by 0x4002117: _dl_start_final (rtld.c:415)
==1425==    by 0x4002117: _dl_start (rtld.c:522)
==1425==    by 0x4001097: ??? (in /lib/x86_64-linux-gnu/
==1425==  Address 0x1036 is not stack'd, malloc'd or (recently) free'd
Jo-Philipp Wich commented on 18.09.2019 07:49

We should inhibit the building of host utilities in the SDK. The SDK is meant to provide a precompiled feature-fixed toolchain environment. Compilation of host utilities (not package host builds) or toolchain components within the SDK is undefined due to the use of a bundled and shipped libc.

I think the best course forward here would be to prevent enabling signed package lists on a non-ucert equipped SDK.

Stan commented on 10.01.2020 18:21

On Ubuntu 18.04.3 and 19.07.0 I'm also getting a segmentation fault on ucert (while trying to build package base-files):

SDK_mvebu.19.07.0/staging_dir/host/bin/ucert -I -c SDK_mvebu.19.07.0/key-build.ucert -p SDK_mvebu.19.07.0/ -s SDK_mvebu.19.07.0/key-build
Segmentation fault (core dumped)

jow – any updates on this?

Petr Štetiar commented on 10.01.2020 19:57
any updates on this

Nope, someone has to fix it. Meanwhile disable signing of the packages CONFIG_SIGNED_PACKAGES under SDK or if you want signed packages, you've to build from scratch, not within SDK.

recycler commented on 24.05.2021 22:48

Same here for Ubuntu 20.04.2

Seems that downloaded ucert is relying on certain library versions.

ldd ./staging_dir/host/bin/ucert (0x00007ffd69dae000) => /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/ (0x00007f743222b000) => /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/ (0x00007f7432226000) => /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/ (0x00007f7432065000) => /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/ (0x00007f7432052000)
        /lib64/ (0x00007f7432245000)
(gdb) run -I -c /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build.ucert -p /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/ -s /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build
Starting program: /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/bin/ucert -I -c /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build.ucert -p /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/ -s /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/key-build

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7e18133 in __libc_start_main () from /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/
(gdb) backtrace
#0  0x00007ffff7e18133 in __libc_start_main () from /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/staging_dir/host/lib/
#1  0x0000555555556a7e in _start () at /home/recycler/proj/openwrt/openwrt-sdk-ramips-mt7620_gcc-8.4.0_musl.Linux-x86_64/build_dir/hostpkg/ucert-2020-05-24-00b921d8/ucert.c:625

A solution could be a static binary for download.

Tomasz Maciej Nowak commented on 10.09.2021 10:39

This will affect all distro versions which weren't used for building particular SDK. On Arch Linux it doesn't segfault and gives a little hint what's wrong:

/home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/bin/ucert: /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/lib/ version `GLIBC_2.33' not found (required by /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/bin/ucert)
/home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/bin/ucert: /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/lib/ version `GLIBC_2.33' not found (required by /home/tomek/Devel/openwrt/onion/sdk/staging_dir/host/lib/

that's on ramips SDK.
I created a workaround (see attachment) partly following Jo's advice. The workaround copies install stamps when creating SDK. Then if SDK is used with package which has a host part, the host part build is omitted if install stamp is present.


Available keyboard shortcuts


Task Details

Task Editing