OpenWrt/LEDE Project

  • Status Unconfirmed
  • Percent Complete
    0%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Low
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Ken - 09.04.2019
Last edited by Yousong Zhou - 30.10.2019

FS#2231 - fstools: PREINIT calling of block extroot doesn't acknowledge non-MTD rootfs overlays

Extroot overlay mount fails on my ZyXEL NBG6817.

I can confirm this bug on both the latest stable 18.06.2, and the snapshot from the 4th of April.

 

When PREINIT calls ‘block extroot’, block fails to load the custom fstab from the eMMC ext4 overlay mounted at /tmp/overlay:

[...]
Thu Apr  4 12:01:51 2019 user.info kernel: [    3.508150] init: - preinit -
Thu Apr  4 12:01:51 2019 kern.info kernel: [    8.592387] EXT4-fs (loop0): recovery complete
Thu Apr  4 12:01:51 2019 kern.info kernel: [    8.595484] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Thu Apr  4 12:01:51 2019 user.info kernel: [    8.598875] mount_root: loading kmods from internal overlay
Thu Apr  4 12:01:51 2019 user.info kernel: [    8.649260] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Thu Apr  4 12:01:51 2019 user.info kernel: [    8.663223] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.119972] block: attempting to load /etc/config/fstab
Thu Apr  4 12:01:51 2019 user.err kernel: [   10.120211] block: unable to load configuration (fstab: Entry not found)
Thu Apr  4 12:01:51 2019 user.err kernel: [   10.124238] block: no usable configuration
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.132103] mount_root: switching to ext4 overlay
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.292110] procd: - early -
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.292264] procd: - watchdog -
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.988093] procd: - watchdog -
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.988475] procd: - ubus -
Thu Apr  4 12:01:51 2019 user.info kernel: [   11.046160] procd: - init -
[...]

Unrelated dmesg entries omitted, full log here:
https://gist.github.com/knuddelknoedel/2985ce7777a0263fbc22a02f8ef5307c

Custom modules are loaded with the correct overlay /tmp/overlay/upper prefix by libfstools, however the forked ‘block extroot’ process behaves differently when searching for /etc/config/fstab configuration.

Steps to reproduce:

  1. Flash any current openwrt sysupgrade image on a device where the rootfs+overlay don’t reside on MTD storage
  2. Configure an appropriate /overlay uci fstab extroot entry as specified in the respective wiki documentation
  3. Reboot

Further notes: adding /etc/config/fstab with the desired /overlay entry to the sysupgrade squashfs image before flashing allows block to successfully find the uci fstab config, however the mounting of the therein configured /overlay mount still fails.

Ken commented on 09.04.2019 16:28

I wrote a small patch that allows proper rootfs and fstab discovery on devices where the rootfs+overlay reside on a block device partition.

Here the dmesg of a current openwrt HEAD build with the patch:

Mon Apr  8 22:25:35 2019 user.info kernel: [    3.335289] init: - preinit -
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.035073] mount_root: loading kmods from internal overlay
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.046998] kmodloader: loading kernel modules from //etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.048462] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.828623] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.828798] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.834600] block: attempting to load /tmp/overlay/etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.841281] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.847486] block: attempting to load /etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.854319] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.859098] block: no usable configuration
Mon Apr  8 22:25:35 2019 kern.info kernel: [    8.985003] EXT4-fs (loop0): recovery complete
Mon Apr  8 22:25:35 2019 kern.info kernel: [    8.986207] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.988754] mount_root: loading kmods from internal overlay
Mon Apr  8 22:25:35 2019 user.info kernel: [    9.013356] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    9.013492] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.011824] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Mon Apr  8 22:25:35 2019 kern.info kernel: [   10.017872] EXT4-fs (mmcblk0p10): mounted filesystem with ordered data mode. Opts:
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.162997] mount_root: switched to extroot
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.331230] procd: - early -
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.331412] procd: - watchdog -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.009306] procd: - watchdog -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.009775] procd: - ubus -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.064866] procd: - init -

Unrelated dmesg entries omitted, full log here: https://gist.github.com/knuddelknoedel/46db572959056d2e69fd861d59ad0daf

Val Kulkov commented on 14.05.2019 20:51

The proposed patch works beautifully to enable extroot on my x86_64 device. With it, I can finally make use of the available space on my /dev/sda without worrying about losing data on it upon a sysupgrade.

The sysupgrade process on a x86_64 system that I use to avoid losing data is not yet a straightforward one. First, I burn openwrt-x86-64-combined-squashfs.img on a USB drive. Then, I copy first two partitions from the USB drive to the corresponding partitions on my /dev/sda. Finally, I copy the MBR bootstrap only, without the partition table: "dd if=/dev/sdc of=/dev/sda bs=446 count=1". This procedure retains sda3, my root overlay partition, and sda4, a swap partition.

utrumo commented on 04.11.2019 21:17

Now on Zyxel Armor Z2 (NBG6817) extroot doesn't work without this patch.
Is there a developer of openwrt? Please add this patch to master! :)

To make work extroot on this router we need compile own firmware and apply this patch:
1. clone src and change dir to it: git clone https://github.com/openwrt/openwrt.git ~/source && cd ~/source
2. make directory for patch: mkdir ./package/system/fstools/patches
3. write patch content from https://patchwork.ozlabs.org/patch/1082599/ to ./package/system/fstools/patches/001-add_propper_rootfs_and_fstab_discovery_on_a_block_device_partitions.patch
4. build own image:
./scripts/feeds update -a
./scripts/feeds install -a
export MAKEFLAGS=-j5
make menuconfig

system ⇒ Qualcomm Atheros IPQ806X y
Target Profile ⇒ Zyxel NBG6817 y
Base system ⇒
⇒ block-mount y (for extroot on mmcblk0p10)
⇒ blockd y (for extroot on mmcblk0p10)
Kernel modules ⇒ USB Support ⇒ kmod-usb-storage y (for usb-hdd support)
LuCI ⇒
⇒ Collections ⇒ luci y
⇒ Applications ⇒ luci-app-advanced-reboot y

make defconfig
make download
ionice -c 3 nice -n19 make

results you can find in:
./bin/targets/ipq806x/generic/

5. flash this firmware to router
scp ./bin/targets/ipq806x/generic/openwrt-ipq806x-zyxel_nbg6817-squashfs-sysupgrade.bin root@openwrt:/tmp
ssh root@openwrt
sysupgrade -v /tmp/openwrt-ipq806x-zyxel_nbg6817-squashfs-sysupgrade.bin

6. set extroot:
ssh root@openwrt
DEVICE="$(awk -e '/\s\/overlay\s/{print $1}' /etc/mtab)"
uci -q delete fstab.rwm
uci set fstab.rwm="mount"
uci set fstab.rwm.device="${DEVICE}"
uci set fstab.rwm.target="/rwm"
uci set fstab.rwm.enabled="1"
uci commit fstab

DEVICE="/dev/mmcblk0p10" // or if you want to use usb-hdd: DEVICE="/dev/sda1"
mkfs.ext4 "${DEVICE}"
eval $(block info "${DEVICE}" | grep -o -e "UUID=\S*")
uci -q delete fstab.overlay
uci set fstab.overlay="mount"
uci set fstab.overlay.uuid="${UUID}"
uci set fstab.overlay.target="/overlay"
uci set fstab.overlay.enabled="1"
uci commit fstab

mount "${DEVICE}" /mnt
cp -a -f /overlay/. /mnt
umount /mnt

reboot

Project Manager
Rafał Miłecki commented on 05.05.2020 07:29

That fstools patch cannot be applied as is (I reviewed & commented on it).

I pushed three related fstools changes:

That moves us closer to fixing this issue.

Someone now needs to improve main_extroot() and make it:

  1. Find standard overlay device (e.g. /dev/sda3)
  2. Mount it using f2fs
  3. Call mount_extroot() for it

It's basically the same logic as what we already have for JFFS2 and UBIFS in that main_extroot() function.

leifliddy commented on 05.05.2020 18:41

If we're going to use f2fs as the filesystem for the extroot overlay device (which is a great idea), then we should probably include these package as part of the base build for the zyxel_nbg6817:
f2fsck
f2fs-tools
kmod-fs-f2fs
mkf2fs

Actually, this brings up another question, why aren't we using the f2fs filesystem for the default overlay device?

/dev/loop0 on /overlay type ext4 (rw,noatime,data=ordered)

Project Manager
Rafał Miłecki commented on 05.05.2020 18:46

I just noticed that orignal report was about ext4. Recently I was chatting with m4t who was using f2fs.

So this task seems more generic: we need to check /dev/loop0 for /etc/config/fstab with overlay setup. That /dev/loop0 may be ext4 or f2fs (or other?).

leifliddy commented on 05.05.2020 19:34

I'm not sure if this is the correct forum for this or not, but here goes.
If you look at partition schedule for the nbg6817

https://openwrt.org/toh/zyxel/nbg6817

You'll see that mmcblk0p5 is divided up into two parts

mmcblk0p5 rootfs 64 MiB

/rom (squashfs) 4MB
/overlay (ext4) 60MB (/dev/loop0)

So, if you've configured mmcblk0p10 to act as the /overlay,
then the 4MB squashfs filesystem from mmcblk0p5 will continue to act as the 4Mb read-only filesystem?

Is there a reason we can't just re-partition this entire device, I mean there's no reason for openwrt to need 10 partitions.
Or is there some sort of unspoken rule that you need to operate within the confines of the OEM flash layout?

If we need to maintain the OEM partitioning scheme, then we should make mmcblk0p1 the 4MB /rom partition
...and make mmcblk0p10 the default /overlay partition.

That's makes the most sense...but I'll save this discussion for another day. Need to do a bit more research on how openwrt is designed....

leifliddy commented on 05.05.2020 19:40

@Ken would you be able to create a new patch based on the changes Rafał made to fstools?

tofurky commented on 17.05.2020 21:06

leifliddy, this works for me with f2fs on 19.07.3. note that the change to check_filesystem() isn't strictly necessary, but i found that newer f2fs-tools gave some issues without it.

diff --git a/block.c b/block.c
index 569bf56..b88e208 100644
--- a/block.c
+++ b/block.c
@@ -747,7 +747,7 @@ static void check_filesystem(struct probe_info *pr)
        pid = fork();
        if (!pid) {
                if(!strncmp(pr->type, "f2fs", 4)) {
-                       execl(ckfs, ckfs, "-f", pr->dev, NULL);
+                       execl(ckfs, ckfs, "-p", "2", "-f", pr->dev, NULL);
                        exit(EXIT_FAILURE);
                } else if(!strncmp(pr->type, "btrfs", 5)) {
                        execl(ckfs, ckfs, "--repair", pr->dev, NULL);
@@ -1591,7 +1591,7 @@ static int main_extroot(int argc, char **argv)
 #endif
 
        /* As a last resort look for /etc/config/fstab on "rootfs" partition */
-       return mount_extroot(NULL);
+       return mount_extroot("/tmp/overlay");
 }
 
 static int main_mount(int argc, char **argv)

i spent a bit of time trying to figure out the proper way to do it per rafal's suggestion, but was running into segfaults with find_block(NULL, NULL, "loop0", NULL) so i just took the easy way out ;) not sure if i was even close to taking the right approach with that, though.

Stefan commented on 20.06.2020 16:51

Hi,

I guess this fix wasn't included in the 19.07.3 release yet, since I am struggling to set up overlayfs on my new NBG6817 with usb hard disk and ext4.

Are there any patched images to download/test? Any chance this will be fixed in one of the next releases?

I am not that experienced to build/figure out this myself, so any links/instructions are much appreciated. On the other hand, my zyxel isn't productive yet, so I am willing to test...

Thanks,
kat

Project Manager
Rafał Miłecki commented on 20.06.2020 17:46

Noone cared to develop a proper patch, people keep posting some hacks that are unacceptable due to breaking other setups.

Stefan commented on 21.06.2020 10:52

Thanks for the quick, if also a bit unsatisfying, answer. I got the zyxel because it seemed to be properly supported by openwrt (and having a bit more oomph than my ageing tp-link). I don't really understand where the problem is: ext4 of f2fs, hard disk or flash drive, or the drives not being ready at boot time, or just some configuration hiccup.

Can you suggest what I could do? I don't mind playing around a bit.

Project Manager
Rafał Miłecki commented on 21.06.2020 11:13
  Someone now needs to improve main_extroot() and make it:
  
  1. Find standard overlay device (e.g. /dev/sda3)
  2. Mount it using f2fs
  3. Call mount_extroot() for it
  
  It's basically the same logic as what we already have for JFFS2 and UBIFS in that main_extroot() function.
  I just noticed that orignal report was about ext4. Recently I was chatting with m4t who was using f2fs.
  
  So this task seems more generic: we need to check /dev/loop0 for /etc/config/fstab with overlay setup. That /dev/loop0 may be ext4 or f2fs (or other?).

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing