Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#1071 - Sysupgrade fails when kernel console is disabled #6005

Closed
openwrt-bot opened this issue Oct 17, 2017 · 8 comments · May be fixed by openwrt/procd#9
Closed

FS#1071 - Sysupgrade fails when kernel console is disabled #6005

openwrt-bot opened this issue Oct 17, 2017 · 8 comments · May be fixed by openwrt/procd#9
Labels
bug issue report with a confirmed bug flyspray target/imx pull request/issue for imx (imx6) target

Comments

@openwrt-bot
Copy link

xback:

Hardware used:

  • Gateworks Ventana imx6 board (GW5200)

Synopsis:

Running sysupgrade with a valid image fails if the kernel console is disabled in u-boot (setenv console ; saveenv; reset)
As soon as sysupgrade starts, the board reboots immediately.

The detected reboot reason displayed is exactly the same in both cases.

Reset cause: WDOG
WDOG1 Reset cause: SFTW
WDOG2 Reset cause: POR

Basically:

  • Kernel console enabled (console = ttymxc1) --> sysupgrade runs normally and succeeds
  • Kernel console disabled (console = ) --> The board reboots within 1 second as soon as sysupgrade starts

Rationale for disabling kernel console:

  • This board exposes 2 UART ports for user applications (ttymxc0, ttymxc1)
  • ttymxc1 is shared with the kernel console and needs to be disabled so it does not interfere with the peripheral communication
  • Both ports are used in my case (ttymxc0: GPS received ; ttymxc1: Iridium modem)
@openwrt-bot
Copy link
Author

xback:

More details as I'm currently digging into this issue:

/sbin/sysupgrade finally makes following call:

ubus call system sysupgrade '{
"prefix": "/tmp/root",
"path": "/tmp/upgrade/5.0.0.17_imx6.ubi",
"command": ". /lib/functions.sh; include /lib/upgrade; do_upgrade_stage2"

When this call is issued and there is no kernel console available, the board reboots immediately.

More details later on .. hopefully ..

@openwrt-bot
Copy link
Author

xback:

Issue identified:

The wiki page [1] states that the ventana kernel prints can be disabled using following u-boot command:

setenv console
saveenv

In this case, I would expect the following line checking the kernel boot using dmesg:
[ 0.000000] Kernel command line: ubi0:ubi ubi.mtd=2 rootfstype=squashfs,ubifs

But I'm seeing this:
[ 0.000000] Kernel command line: console=,115200 ubi0:ubi ubi.mtd=2 rootfstype=squashfs,ubifs

I've also tried this:

setenv console null
saveenv

Which results in this one:
[ 0.000000] Kernel command line: console=null,115200 ubi0:ubi ubi.mtd=2 rootfstype=squashfs,ubifs

Practically, these faulty strings cause sysupgrade to reboot instantly as soon as the upgrade starts. (100% repro rate)
When removing the faulty console= line from the kernel bootargs, sysupgrade works perfectly again.

I've reported this back to Gateworks Corp.

[1]
http://trac.gateworks.com/wiki/silenceconsole#DisableKernelserialconsole

@openwrt-bot
Copy link
Author

TB:

I have the same problem, is there any solution?

@openwrt-bot
Copy link
Author

dpprog:

The problem with this code is:
procd/upgraded/upgraded.c:

static void sysupgrade(char *path, char *command)
{
char *args[] = { "/lib/upgrade/stage2", path, command, NULL };

upgrade_proc.cb = upgrade_proc_cb;
upgrade_proc.pid = fork();
if (upgrade_proc.pid < 0) {
	fprintf(stderr, "Failed to fork sysupgrade\n");
	return;
}

if (!upgrade_proc.pid) {
	/* Child */
	execvp(args[0], args);
	fprintf(stderr, "Failed to exec sysupgrade\n");
	_exit(-1);
}

uloop_process_add(&upgrade_proc);
uloop_run();

}
execvp() returned -1. errno: 2

@openwrt-bot
Copy link
Author

dpprog:

With the console turned off, the process /sbin/upgraded will start up with a working directory /dev/

@openwrt-bot
Copy link
Author

TB:

Dmitry, thank you for the information.
If we do not have physical TTYs, we need to enable virtual.
In my case, configuration for the kernel;

CONFIG_CMDLINE="rootfstype=squashfs,jffs2 noinitrd console=tty0 console=null"
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_HW_CONSOLE_BINDING=n
CONFIG_VGA_CONSOLE=n

After this procedure, the update proceeds correctly.

@openwrt-bot
Copy link
Author

abatyiev:

Hello everyone,

I've got hit by same bug on ath79. Sysupgrade almost immediately goes into reboot without changing firmware.

Apparently, it is a bug in procd.

There is state.c file with following function:

static void set_stdio(const char* tty)
{
if (chdir("/dev") ||
!freopen(tty, "r", stdin) ||
!freopen(tty, "w", stdout) ||
!freopen(tty, "w", stderr) ||
chdir("/"))
ERROR("failed to set stdio: %m\n");
else
fcntl(STDERR_FILENO, F_SETFL, fcntl(STDERR_FILENO, F_GETFL) | O_NONBLOCK);
}

As you can see, if there is an error, the chdir("/") function is not called. That means, that during sysupgrade the cwd of procd process would be "/dev" (you can check that by "ls -l /proc/1/cwd").

Such arragement breaks /sbin/upgraded process (that is exec'ed into from procd), when it tries to chroot("."), because instead of chrooting into "/", it instead chroots into "/dev" and fails to find/exec "/lib/upgrade/stage2".

Steps to reproduce:

  1. /dev/console should be broken (access to it should yield "No such device" error)
  2. In logs, there should be "user.err kernel: [ 8.256052] procd: failed to set stdio: No such device"
  3. "ls -l /proc/1/cwd" should point to "/dev"
  4. "sysupgrade new.bin" should immediately reboot the device without changing the firmware

Quick and dirty fix (i.e. "I want to upgrade my firmware somehow"):

mv /sbin/udevtrigger /sbin/udevtrigger.old
cat > /sbin/udevtrigger <<<EOF
#!/bin/sh

/sbin/udevtrigger.old $@
rm /dev/console
ln -s /dev/null /dev/console

EOF

Proper fix should insert "chdir("/");" into sysupgrade.c file of procd just before chroot.

@ynezz
Copy link
Member

ynezz commented Mar 8, 2022

Too old.

@ynezz ynezz closed this as completed Mar 8, 2022
@ynezz ynezz added target/imx pull request/issue for imx (imx6) target bug issue report with a confirmed bug labels Mar 8, 2022
Rondom added a commit to Rondom/procd that referenced this issue Apr 15, 2024
set_stdio chdirs to /dev/ to facilitate easy freopen of the console
device name given by the tty parameter. Make sure to chdir back to / in
all cases, even in the error path. This keeps the function free from
unintended side effects.

Before this commit, in case of an error, the working directory would
remain /dev/ which would break sysupgrade because the rest of the code
would rely on the current working directory to be unchanged, which is
not an unreasonable expectation to make.

Fixing this fixes an issue where sysupgrade would fail, when
/dev/console does not exist or cannot be opened, which can happen for
example when setting console= on kernel cmdline.

Closes: openwrt/openwrt#6005
Fixes: 91da63d ("properly handle return codes")
Signed-off-by: Andreas Gnau <andreas.gnau@iopsys.eu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug issue report with a confirmed bug flyspray target/imx pull request/issue for imx (imx6) target
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants