guidosarducci:

I've confirmed this problem also occurs on malta/mipsel64, once the binutils build failure on that platform is fixed by [[https://github.com//pull/3288|PR#3288]]

Reproducing the issue is trivial using QEMU (image built from defaults):./scripts/qemustart malta be64 -cpu MIPS64R2-generic

This bug could use input from a developer familiar with procd init.

guidosarducci:

Still seeing the same error after updating to current openwrt commit b59a98b. Enabling debug of procd init shows a little more information.

[ 1.264652] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100
[ 1.276184] scsi 1:0:0:0: CD-ROM QEMU QEMU DVD-ROM 2.5+ PQ: 0 ANSI: 5
[ 1.311226] Freeing unused kernel memory: 21628K
[ 1.311469] This architecture does not have kernel memory protection.
[ 1.311787] Run /init as init process
[ 1.418543] init: Console is alive
[ 1.436103] init: Ping
[ 1.447138] init: Ping
[ 1.457733] init: Ping
[ 1.468352] init: Ping
[ 1.478950] init: Ping
[ 1.489642] init: Ping
[ 1.500256] init: Ping
[ 1.506097] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[ 1.511427] init: Ping
[ 1.516345] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[ 1.522048] init: Ping
[ 1.525605] init: - preinit -
[ 1.529807] init: Launched preinit instance, pid=505
[ 1.535329] do_page_fault(): sending SIGSEGV to init for invalid read access from 0000000000000360
[ 1.535755] epc = 0000000000000360 in init[aaab9d8000+4000]
[ 1.536552] ra = 000000fffc2f05f0 in
[ 1.538997] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[ 1.540118] Rebooting in 1 seconds..


Looking at the [[https://lxr.openwrt.org/source/procd/initd/init.c|procd code]], the main init seems to start OK, then forks kmodloader which also completes without error. The subsequent code does:
uloop_init();
preinit();
uloop_run();

Looking at the [[https://lxr.openwrt.org/source/procd/initd/preinit.c#L125|preinit() code]], it appears to fork plugd:
/sbin/procd -h /etc/hotplug-preinit.json
and then fork a preinit instance:
/bin/sh /etc/preinit
before printing a DEBUG message and returning. The above seems to have worked.

The kernel panic is immediately after, and the logged "exitcode=0x0000000b" is:
EAGAIN 11 Resource temporarily unavailable

Perhaps there's an issue with one of the forked process, or during uloop_run().

yousong:

Likely the 0xb exitcode is for SIGSEGV. See /bin/kill -L

guidosarducci:

As another experiment I configured init as /bin/sh to test some basic functions:

qemu-system-mips64 -M malta -kernel bpf-openwrt-malta-be64-vmlinux.elf
-drive file=bpf-openwrt-malta-be64-rootfs-ext4.img,index=0,media=disk
-nographic -m 512 -append "root=/dev/sda rootfstype=ext4 init=/bin/sh"

/bin/sh: can't access tty; job control turned off
[ 22.326975] random: fast init done
/ #
/ # echo *
bin dev etc init lib lib64 lost+found mnt overlay proc rom root sbin sys tmp usr var www
/ #
/ # ls
[ 35.314543] do_page_fault(): sending SIGSEGV to ls for invalid read access from 0000000000000360
[ 35.314858] epc = 0000000000000360 in busybox[120000000+82000]
[ 35.315434] ra = 000000fffeb105f0 in
Segmentation fault
/ #

While shell builtins appear to work, commands that require fork/exec are yielding the same SIGSEGV fault. Looks like something very wrong with this system.

guidosarducci:

BTW, I realized the "exitcode" refers to a status returned by wait() which, in the case of signal termination, encodes the signal in the low byte. So yes, 0x0b is SIGSEGV.

I also keep coming back to the fact the invalid read access address and the epc are the same. It seems like there's a crazy jump being made into inaccessible memory, from somewhere well away (ra=000000fffc2f05f0) from the init code.

While booting with "init=/bin/sh" a few times, I could manually mount /proc and look at the self memory map. The previous ra seems very close to the [vvar]/[vdso] regions, which could make sense given a long jump, and gives me some ideas/clues to follow up on.