Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FS#2666 - procd: reboot inside lxc container causes shutdown #7568

Closed
openwrt-bot opened this issue Dec 9, 2019 · 9 comments
Closed

FS#2666 - procd: reboot inside lxc container causes shutdown #7568

openwrt-bot opened this issue Dec 9, 2019 · 9 comments
Labels
flyspray

Comments

@openwrt-bot
Copy link

pgwipeout:

  • Device problem occurs on
    armvirt64
  • Software versions of OpenWrt/LEDE release, packages, etc.
    openwrt-19.07-rc2
  • Steps to reproduce

Create a lxc package from the openwrt-19.07-rc2 rootfs package.
Start a lxc container using the openwrt lxc package.
Initiate a reboot command from inside the container.

Expected result: Container reboots.
Actual result: Container halts.

Current Version openwrt-19.07-rc2.
While running in a lxc container, initiating a reboot from openwrt results in a shutdown of the container instead of rebooting the container.

This appears to have been caused by commit <832369078d818d19ab64051fdc8da9e06c90ad88> state: fix shutdown when running in a container (FS#2425).
Instead of triggering reboot(reboot_event);, it detects that it is inside a container and exits pid 1, resulting in the container halting.

There is a patch at [0] to resolve this, but would likely break the original intention of the fix above.
@openwrt-bot
Copy link
Author

pgwipeout:

Add a link button has a maximum character limit which truncated the reference link.

Reference Link:
[0] https://github.com/mikma/lxd-openwrt/blob/master/patches/procd-master/0001-state-support-reboot-of-containers.patch

@openwrt-bot
Copy link
Author

ynezz:

Hi,

I've just tried following:

docker run --rm -it openwrtorg/rootfs:x86-64-19.07.0 reboot

and it works as expected, it doesnt hang, so probably lxd/lxc does it differently. I don't have any prior experience with lxc/lxd, can you provide exact commands one has to run in order to reproduce it?

@openwrt-bot
Copy link
Author

ynezz:

In other words, following commands:

docker run --env DBGLVL=5 --rm -it openwrtorg/rootfs:x86-64-19.07.0 root@4c37be61eef4:/# logread -f &

provides following debugging log output:

Mon Jan 20 07:28:50 2020 daemon.notice procd: Triggering reboot Mon Jan 20 07:28:50 2020 daemon.notice procd: Shutting down system with event 1234567 Mon Jan 20 07:28:50 2020 daemon.info procd: - shutdown - Mon Jan 20 07:28:50 2020 daemon.notice procd: running /etc/rc.d/K* shutdown Mon Jan 20 07:28:50 2020 daemon.notice procd: start /etc/rc.d/K10gpio_switch shutdown Mon Jan 20 07:28:50 2020 daemon.notice procd: stop /etc/rc.d/K10gpio_switch shutdown - took 0.030491852s Mon Jan 20 07:28:50 2020 daemon.notice procd: start /etc/rc.d/K50dropbear shutdown Mon Jan 20 07:28:50 2020 daemon.notice procd: stop /etc/rc.d/K50dropbear shutdown - took 0.021862346s Mon Jan 20 07:28:50 2020 authpriv.info dropbear[389]: Early exit: Terminated by signal Mon Jan 20 07:28:50 2020 daemon.notice procd: start /etc/rc.d/K85odhcpd shutdown Mon Jan 20 07:28:50 2020 daemon.notice procd: Instance dropbear::instance1 exit with error code 256 after 10 seconds Mon Jan 20 07:28:50 2020 daemon.notice procd: Stop instance odhcpd::instance1 Mon Jan 20 07:28:50 2020 daemon.notice procd: Instance odhcpd::instance1 exit with error code 0 after 10 seconds Mon Jan 20 07:28:50 2020 daemon.notice procd: stop /etc/rc.d/K85odhcpd shutdown - took 0.030896977s Mon Jan 20 07:28:50 2020 daemon.notice procd: start /etc/rc.d/K89log shutdown Mon Jan 20 07:28:50 2020 daemon.notice procd: Stop instance log::instance1

I would like to get the same logs from LXC container running under LXD.

@openwrt-bot
Copy link
Author

bjonglez:

Are you running an OpenWrt LXC container on an OpenWrt host? Or an OpenWrt container on a host running another distribution? (Debian, Fedora...)

I can test OpenWrt LXC container on a x86_64 host running Debian, but I want to make sure it's really the same setup as you :)

Also, how do you prepare your image? Usually I just dump openwrt-x86-64-rootfs-ext4.img to a LVM volume.

@openwrt-bot
Copy link
Author

bjonglez:

To create a container:

cd /tmp/ wget http://downloads.openwrt.org/releases/19.07.0/targets/x86/64/openwrt-19.07.0-x86-64-generic-rootfs.tar.gz

mkdir -p /var/lib/lxc/openwrt/rootfs
touch /var/lib/lxc/openwrt/config
cd /var/lib/lxc/openwrt/rootfs
tar xvf /tmp/openwrt-19.07.0-x86-64-generic-rootfs.tar.gz

Edit /var/lib/lxc/openwrt/config:

# LXC 2: use "lxc.rootfs" instead lxc.rootfs.path = /var/lib/lxc/openwrt/rootfs

Common configuration

lxc.include = /usr/share/lxc/config/common.conf

LXC 2: use "lxc.utsname" instead

lxc.uts.name = openwrt
lxc.arch = amd64

Start container:

lxc-start -n openwrt

To debug if it doesn't work, add -F.

To connect to the container:

lxc-attach -n openwrt --clear-env /bin/ash

To look at the current container state:

lxc-ls -f

When testing here, on a x86_64 host running Arch Linux and LXC 3.2.1, indeed a reboot causes the container to shut down.

@openwrt-bot
Copy link
Author

bjonglez:

Hmm, couldn't get any useful log:

# lxc-attach -n openwrt --clear-env --set-var DBGLVL=5 /bin/ash

BusyBox v1.30.1 () built-in shell (ash)

~ # logread -f &
~ # reboot
~ # Mon Jan 20 09:51:32 2020 daemon.info procd: - shutdown -
Mon Jan 20 09:51:32 2020 authpriv.info dropbear[287]: Early exit: Terminated by signal
Failed to find log object: Not found
Failed to find log object: Not found
Failed to find log object: Connection failed
Failed to find log object: Invalid argument

@openwrt-bot
Copy link
Author

bjonglez:

Ok, the DBGLVL=5 should go in /var/lib/lxc/openwrt/config:

... lxc.environment = DBGLVL=5

Then I get some debug:

# lxc-attach -n openwrt --clear-env /bin/ash ... ~ # logread -f & ~ # reboot ~ # Mon Jan 20 09:56:00 2020 daemon.notice procd: Triggering reboot Mon Jan 20 09:56:00 2020 daemon.notice procd: Shutting down system with event 1234567 Mon Jan 20 09:56:00 2020 daemon.info procd: - shutdown - Mon Jan 20 09:56:00 2020 daemon.notice procd: running /etc/rc.d/K* shutdown Mon Jan 20 09:56:00 2020 daemon.notice procd: start /etc/rc.d/K10gpio_switch shutdown Mon Jan 20 09:56:00 2020 daemon.notice procd: stop /etc/rc.d/K10gpio_switch shutdown - took 0.024061621s Mon Jan 20 09:56:00 2020 daemon.notice procd: start /etc/rc.d/K50dropbear shutdown Mon Jan 20 09:56:00 2020 authpriv.info dropbear[287]: Early exit: Terminated by signal Mon Jan 20 09:56:00 2020 daemon.notice procd: stop /etc/rc.d/K50dropbear shutdown - took 0.015703745s Mon Jan 20 09:56:00 2020 daemon.notice procd: Instance dropbear::instance1 exit with error code 256 after 10 seconds Mon Jan 20 09:56:00 2020 daemon.notice procd: start /etc/rc.d/K85odhcpd shutdown Mon Jan 20 09:56:00 2020 daemon.notice procd: Stop instance odhcpd::instance1 Mon Jan 20 09:56:00 2020 daemon.notice procd: Instance odhcpd::instance1 exit with error code 0 after 10 seconds Mon Jan 20 09:56:00 2020 daemon.notice procd: stop /etc/rc.d/K85odhcpd shutdown - took 0.022317393s Mon Jan 20 09:56:00 2020 daemon.notice procd: start /etc/rc.d/K89log shutdown Mon Jan 20 09:56:00 2020 daemon.notice procd: Stop instance log::instance1 Failed to find log object: Not found Failed to find log object: Not found Failed to find log object: Not found Failed to find log object: Invalid argument

@openwrt-bot
Copy link
Author

bjonglez:

Petr, your patch https://gitlab.com/ynezz/openwrt-procd/commit/aa8689ccdff0124f3d477f86433496aeb2c49d24 fixes the issue, you can add my Tested-By.

I've tested with both LXC 2.0.7 on Debian and LXC 3.2.1 on Arch Linux.

@openwrt-bot
Copy link
Author

pgwipeout:

Confirmed your patch https://gitlab.com/ynezz/openwrt-procd/commit/aa8689ccdff0124f3d477f86433496aeb2c49d24 fixes the issue on Ubuntu 19.10 LXD 3.18.

Tested-by: Peter Geis pgwipeout@gmail.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
flyspray
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@openwrt-bot