Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2231 - fstools: PREINIT calling of block extroot doesn't acknowledge non-MTD rootfs overlays #7352

Open
openwrt-bot opened this issue Apr 9, 2019 · 88 comments
Labels

Comments

@openwrt-bot
Copy link

openwrt-bot commented Apr 9, 2019

knodel:

Extroot overlay mount fails on my ZyXEL NBG6817.

I can confirm this bug on both the latest stable 18.06.2, and the snapshot from the 4th of April.

When PREINIT calls 'block extroot', block fails to load the custom fstab from the eMMC ext4 overlay mounted at /tmp/overlay:

[...]
Thu Apr  4 12:01:51 2019 user.info kernel: [    3.508150] init: - preinit -
Thu Apr  4 12:01:51 2019 kern.info kernel: [    8.592387] EXT4-fs (loop0): recovery complete
Thu Apr  4 12:01:51 2019 kern.info kernel: [    8.595484] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Thu Apr  4 12:01:51 2019 user.info kernel: [    8.598875] mount_root: loading kmods from internal overlay
Thu Apr  4 12:01:51 2019 user.info kernel: [    8.649260] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Thu Apr  4 12:01:51 2019 user.info kernel: [    8.663223] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.119972] block: attempting to load /etc/config/fstab
Thu Apr  4 12:01:51 2019 user.err kernel: [   10.120211] block: unable to load configuration (fstab: Entry not found)
Thu Apr  4 12:01:51 2019 user.err kernel: [   10.124238] block: no usable configuration
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.132103] mount_root: switching to ext4 overlay
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.292110] procd: - early -
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.292264] procd: - watchdog -
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.988093] procd: - watchdog -
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.988475] procd: - ubus -
Thu Apr  4 12:01:51 2019 user.info kernel: [   11.046160] procd: - init -
[...]

Unrelated dmesg entries omitted, full log here:
https://gist.github.com/knuddelknoedel/2985ce7777a0263fbc22a02f8ef5307c

Custom modules are loaded with the correct overlay /tmp/overlay/upper prefix by libfstools, however the forked 'block extroot' process behaves differently when searching for /etc/config/fstab configuration.

Steps to reproduce:

  • Flash any current openwrt sysupgrade image on a device where the rootfs+overlay don't reside on MTD storage
  • Configure an appropriate /overlay uci fstab extroot entry as specified in the respective wiki documentation
  • Reboot

Further notes: adding /etc/config/fstab with the desired /overlay entry to the sysupgrade squashfs image before flashing allows block to successfully find the uci fstab config, however the mounting of the therein configured /overlay mount still fails.

@openwrt-bot
Copy link
Author

openwrt-bot commented Apr 9, 2019

knodel:

I wrote a small patch that allows proper rootfs and fstab discovery on devices where the rootfs+overlay reside on a block device partition.

Here the dmesg of a current openwrt HEAD build with the patch:

Mon Apr  8 22:25:35 2019 user.info kernel: [    3.335289] init: - preinit -
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.035073] mount_root: loading kmods from internal overlay
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.046998] kmodloader: loading kernel modules from //etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.048462] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.828623] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.828798] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.834600] block: attempting to load /tmp/overlay/etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.841281] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.847486] block: attempting to load /etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.854319] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.859098] block: no usable configuration
Mon Apr  8 22:25:35 2019 kern.info kernel: [    8.985003] EXT4-fs (loop0): recovery complete
Mon Apr  8 22:25:35 2019 kern.info kernel: [    8.986207] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.988754] mount_root: loading kmods from internal overlay
Mon Apr  8 22:25:35 2019 user.info kernel: [    9.013356] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    9.013492] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.011824] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Mon Apr  8 22:25:35 2019 kern.info kernel: [   10.017872] EXT4-fs (mmcblk0p10): mounted filesystem with ordered data mode. Opts:
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.162997] mount_root: switched to extroot
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.331230] procd: - early -
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.331412] procd: - watchdog -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.009306] procd: - watchdog -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.009775] procd: - ubus -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.064866] procd: - init -

Unrelated dmesg entries omitted, full log here: https://gist.github.com/knuddelknoedel/46db572959056d2e69fd861d59ad0daf

@openwrt-bot
Copy link
Author

val-kulkov:

The proposed patch works beautifully to enable extroot on my x86_64 device. With it, I can finally make use of the available space on my /dev/sda without worrying about losing data on it upon a sysupgrade.

The sysupgrade process on a x86_64 system that I use to avoid losing data is not yet a straightforward one. First, I burn openwrt-x86-64-combined-squashfs.img on a USB drive. Then, I copy first two partitions from the USB drive to the corresponding partitions on my /dev/sda. Finally, I copy the MBR bootstrap only, without the partition table: "dd if=/dev/sdc of=/dev/sda bs=446 count=1". This procedure retains sda3, my root overlay partition, and sda4, a swap partition.

@openwrt-bot
Copy link
Author

utrumo:

Now on Zyxel Armor Z2 (NBG6817) extroot doesn't work without this patch.
Is there a developer of openwrt? Please add this patch to master! :)

To make work extroot on this router we need compile own firmware and apply this patch:

  1. clone src and change dir to it: git clone https://github.com/openwrt/openwrt.git ~/source && cd ~/source
  2. make directory for patch: mkdir ./package/system/fstools/patches
  3. write patch content from https://patchwork.ozlabs.org/patch/1082599/ to ./package/system/fstools/patches/001-add_propper_rootfs_and_fstab_discovery_on_a_block_device_partitions.patch
  4. build own image:
    ./scripts/feeds update -a
    ./scripts/feeds install -a
    export MAKEFLAGS=-j5
    make menuconfig
    ===================================================
    system => Qualcomm Atheros IPQ806X y
    Target Profile => Zyxel NBG6817 y
    Base system =>
    => block-mount y (for extroot on mmcblk0p10)
    => blockd y (for extroot on mmcblk0p10)
    Kernel modules => USB Support => kmod-usb-storage y (for usb-hdd support)
    LuCI =>
    => Collections => luci y
    => Applications => luci-app-advanced-reboot y
    ===================================================
    make defconfig
    make download
    ionice -c 3 nice -n19 make

results you can find in:
./bin/targets/ipq806x/generic/

  1. flash this firmware to router
    scp ./bin/targets/ipq806x/generic/openwrt-ipq806x-zyxel_nbg6817-squashfs-sysupgrade.bin root@openwrt:/tmp
    ssh root@openwrt
    sysupgrade -v /tmp/openwrt-ipq806x-zyxel_nbg6817-squashfs-sysupgrade.bin

  2. set extroot:
    ssh root@openwrt
    DEVICE="$(awk -e '/\s/overlay\s/{print $1}' /etc/mtab)"
    uci -q delete fstab.rwm
    uci set fstab.rwm="mount"
    uci set fstab.rwm.device="${DEVICE}"
    uci set fstab.rwm.target="/rwm"
    uci set fstab.rwm.enabled="1"
    uci commit fstab

DEVICE="/dev/mmcblk0p10" // or if you want to use usb-hdd: DEVICE="/dev/sda1"
mkfs.ext4 "${DEVICE}"
eval $(block info "${DEVICE}" | grep -o -e "UUID=\S*")
uci -q delete fstab.overlay
uci set fstab.overlay="mount"
uci set fstab.overlay.uuid="${UUID}"
uci set fstab.overlay.target="/overlay"
uci set fstab.overlay.enabled="1"
uci commit fstab

mount "${DEVICE}" /mnt
cp -a -f /overlay/. /mnt
umount /mnt

reboot

@openwrt-bot
Copy link
Author

rmilecki:

That fstools patch cannot be applied as is (I reviewed & commented on it).

I pushed three related fstools changes:

  • [[https://git.openwrt.org/?p=project/fstools.git;a=commitdiff;h=d70774de24413f91087e9c3c873cf7a4ec3df813|d70774de2441 ("block: add some basic extroot documentation")]]
  • [[https://git.openwrt.org/?p=project/fstools.git;a=commitdiff;h=37c9148a3ae17665e75f99b459b6ff2b461d6739|37c9148a3ae1 ("block: simplify check_extroot() a bit")]]
  • [[https://git.openwrt.org/?p=project/fstools.git;a=commitdiff;h=8b9e601160712f74ff46313579d0225a7f90ca3c|8b9e60116071 ("block: always use st_dev (device ID) of / when looking for root")]]

That moves us closer to fixing this issue.

Someone now needs to improve main_extroot() and make it:

  • Find standard overlay device (e.g. /dev/sda3)
  • Mount it using f2fs
  • Call mount_extroot() for it

It's basically the same logic as what we already have for JFFS2 and UBIFS in that main_extroot() function.

@openwrt-bot
Copy link
Author

leifliddy:

If we're going to use f2fs as the filesystem for the extroot overlay device (which is a great idea), then we should probably include these package as part of the base build for the zyxel_nbg6817:
f2fsck
f2fs-tools
kmod-fs-f2fs
mkf2fs

Actually, this brings up another question, why aren't we using the f2fs filesystem for the default overlay device?

/dev/loop0 on /overlay type ext4 (rw,noatime,data=ordered)

@openwrt-bot
Copy link
Author

rmilecki:

I just noticed that orignal report was about ext4. Recently I was chatting with m4t who was using f2fs.

So this task seems more generic: we need to check /dev/loop0 for /etc/config/fstab with overlay setup. That /dev/loop0 may be ext4 or f2fs (or other?).

@openwrt-bot
Copy link
Author

leifliddy:

I'm not sure if this is the correct forum for this or not, but here goes.
If you look at partition schedule for the nbg6817

https://openwrt.org/toh/zyxel/nbg6817

You'll see that mmcblk0p5 is divided up into two parts

mmcblk0p5 rootfs 64 MiB

/rom (squashfs) 4MB
/overlay (ext4) 60MB (/dev/loop0)

So, if you've configured mmcblk0p10 to act as the /overlay,
then the 4MB squashfs filesystem from mmcblk0p5 will continue to act as the 4Mb read-only filesystem?

Is there a reason we can't just re-partition this entire device, I mean there's no reason for openwrt to need 10 partitions.
Or is there some sort of unspoken rule that you need to operate within the confines of the OEM flash layout?

If we need to maintain the OEM partitioning scheme, then we should make mmcblk0p1 the 4MB /rom partition
...and make mmcblk0p10 the default /overlay partition.

That's makes the most sense...but I'll save this discussion for another day. Need to do a bit more research on how openwrt is designed....

@openwrt-bot
Copy link
Author

leifliddy:

@ken would you be able to create a new patch based on the changes Rafał made to fstools?

@openwrt-bot
Copy link
Author

openwrt-bot commented May 17, 2020

tofurky:

leifliddy, this works for me with f2fs on 19.07.3. note that the change to check_filesystem() isn't strictly necessary, but i found that newer f2fs-tools gave some issues without it.

diff --git a/block.c b/block.c
index 569bf56..b88e208 100644
--- a/block.c
+++ b/block.c
@@ -747,7 +747,7 @@ static void check_filesystem(struct probe_info *pr)
        pid = fork();
        if (!pid) {
                if(!strncmp(pr->type, "f2fs", 4)) {
-                       execl(ckfs, ckfs, "-f", pr->dev, NULL);
+                       execl(ckfs, ckfs, "-p", "2", "-f", pr->dev, NULL);
                        exit(EXIT_FAILURE);
                } else if(!strncmp(pr->type, "btrfs", 5)) {
                        execl(ckfs, ckfs, "--repair", pr->dev, NULL);
@@ -1591,7 +1591,7 @@ static int main_extroot(int argc, char **argv)
 #endif
 
        /* As a last resort look for /etc/config/fstab on "rootfs" partition */
-       return mount_extroot(NULL);
+       return mount_extroot("/tmp/overlay");
 }
 
 static int main_mount(int argc, char **argv)

i spent a bit of time trying to figure out the proper way to do it per rafal's suggestion, but was running into segfaults with find_block(NULL, NULL, "loop0", NULL) so i just took the easy way out ;) not sure if i was even close to taking the right approach with that, though.

@openwrt-bot
Copy link
Author

katrat:

Hi,

I guess this fix wasn't included in the 19.07.3 release yet, since I am struggling to set up overlayfs on my new NBG6817 with usb hard disk and ext4.

Are there any patched images to download/test? Any chance this will be fixed in one of the next releases?

I am not that experienced to build/figure out this myself, so any links/instructions are much appreciated. On the other hand, my zyxel isn't productive yet, so I am willing to test...

Thanks,
kat

@openwrt-bot
Copy link
Author

rmilecki:

Noone cared to develop a proper patch, people keep posting some hacks that are unacceptable due to breaking other setups.

@openwrt-bot
Copy link
Author

katrat:

Thanks for the quick, if also a bit unsatisfying, answer. I got the zyxel because it seemed to be properly supported by openwrt (and having a bit more oomph than my ageing tp-link). I don't really understand where the problem is: ext4 of f2fs, hard disk or flash drive, or the drives not being ready at boot time, or just some configuration hiccup.

Can you suggest what I could do? I don't mind playing around a bit.

@openwrt-bot
Copy link
Author

rmilecki:

Someone now needs to improve main_extroot() and make it:

1. Find standard overlay device (e.g. /dev/sda3)
2. Mount it using f2fs
3. Call mount_extroot() for it

It's basically the same logic as what we already have for JFFS2 and UBIFS in that main_extroot() function.

I just noticed that orignal report was about ext4. Recently I was chatting with m4t who was using f2fs.

So this task seems more generic: we need to check /dev/loop0 for /etc/config/fstab with overlay setup. That /dev/loop0 may be ext4 or f2fs (or other?).

@openwrt-bot
Copy link
Author

visualage:

Is it reasonable to use libfstools to search for rootfs_data volume and mount it, just like what mount_root.c does?

@openwrt-bot
Copy link
Author

visualage:

I made a patch to openwrt to support extroot on non-MTD rootfs_data volumes. It uses libfstools in block.c to support other squashfs volumes, such as /dev/sd* or /dev/mmc*.

The patch itself contains a patch to fstools project that can be applied to that codebase directly.

I tested the build locally on a x86-64 virtualbox environment as well as rockchip friendlyarm nanopi r2s. It should work on any other environments as long as mount_root works since they use the same underlying library.

@openwrt-bot
Copy link
Author

visualage:

This is the fstools patch. The openwrt patch is just to make sure block-mount depends on libfstools when packaging.
Rafal, would you please take a look?

@openwrt-bot
Copy link
Author

kofec:

Hi,
Aby update about this issue ?

@openwrt-bot
Copy link
Author

uutrumo:

Qi Liu, for wich version of openwrt your patch?
I try it on v19.07.7 and have got an error:
Package block-mount is missing dependencies for the following libraries:
libfstools.so

@openwrt-bot
Copy link
Author

openwrt-bot commented May 10, 2021

uutrumo:

tofurky, your hack works, thank you.
Some patch for v19.07.7 and v21.02.0-rc1:

--- a/block.c
+++ b/block.c
@@ -747,7 +747,7 @@ static void check_filesystem(struct prob
 	pid = fork();
 	if (!pid) {
 		if(!strncmp(pr->type, "f2fs", 4)) {
-			execl(ckfs, ckfs, "-f", pr->dev, NULL);
+			execl(ckfs, ckfs, "-p", "2", "-f", pr->dev, NULL);
 			exit(EXIT_FAILURE);
 		} else if(!strncmp(pr->type, "btrfs", 5)) {
 			execl(ckfs, ckfs, "--repair", pr->dev, NULL);
@@ -1591,7 +1591,7 @@ static int main_extroot(int argc, char *
 #endif
 
 	/* As a last resort look for /etc/config/fstab on "rootfs" partition */
-	return mount_extroot(NULL);
+	return mount_extroot("/tmp/overlay");
 }
 
 static int main_mount(int argc, char **argv)

@openwrt-bot
Copy link
Author

luce.nera:

How do I apply these patches and which one to use?

@openwrt-bot
Copy link
Author

flowercow:

i met this bug in 21.02.0-rc4 installed in kvm.

Is there any other method to solve this problem?

For example, do something in /etc/preinit?

@openwrt-bot
Copy link
Author

Gandalf:

Still got the issue in 21.02.0 !!!
Is there any work in this patch ?

@openwrt-bot
Copy link
Author

flowercow:

i have tried to recompile firmware applying patch as mentioned by utrumo above.
after upgrade to that firmware and install block-mount package that recompiled, this issue was not fixed.

@openwrt-bot
Copy link
Author

Op3n:

The same bug occurs on the latest stable release 20.02, Is there any way to avoid this bug other than by compiling it myself? I prefer use official pre-compiled version on downloads.openwrt.com

@openwrt-bot
Copy link
Author

katrat:

Well, any solutions apart from patching and compiling openwrt ourselves? Does anyone have a precompiled working solution for 21.02?

@openwrt-bot
Copy link
Author

siicosmos:

Need some solution for 21.02 as well. Either patches or a pre-compiled image is fine...

@openwrt-bot
Copy link
Author

siicosmos:

Updates:
Actually, the patch tofurky provided actually works in 21.02. You need to apply the patch using Quilt or edit directly on block.c file.

@VA1DER
Copy link

VA1DER commented Feb 28, 2022

Is there a timeline for this being officially fixed?

@MaxSKash
Copy link

MaxSKash commented Mar 7, 2022

There is an alternative way for Zyxel NBG6817: change the uboot-env variable so that rootfs is loaded from /dev/mmсblk0p10

Example:
Boot into an alternative configuration (which is 01, not FF)
We copy the sysupgrade.bin to the router in the folder /tmp
Connect to it via ssh
Manual flash firmvare:
(based on a standard script: /lib/upgrade/zyxel.sh)

view at the contents of the firmware:

root@OpenWrt:~# cd /tmp
root@OpenWrt:/tmp# tar tf openwrt-ipq806x-generic-zyxel_nbg6817-squashfs-sysupgrade.bin
tar: sysupgrade-nbg6817/CONTROL: time stamp 2022-03-07 10:14:29 is 42124622.3872312 s in the future
tar: sysupgrade-nbg6817/kernel: time stamp 2022-03-07 10:14:29 is 42124622.35407792 s in the future
tar: sysupgrade-nbg6817/root: time stamp 2022-03-07 10:14:29 is 42124622.32335552 s in the future
tar: sysupgrade-nbg6817: time stamp 2022-03-07 10:14:29 is 42124622.32289632 s in the future

flash: (tar xf <<filename.bin>> <<contentname>> -O > <<path>> )

root@OpenWrt:/tmp# tar xf openwrt-ipq806x-generic-zyxel_nbg6817-squashfs-sysupgrade.bin sysupgrade-nbg6817/kernel -O > /dev/mmcblk0p4
root@OpenWrt:/tmp# tar xf openwrt-ipq806x-generic-zyxel_nbg6817-squashfs-sysupgrade.bin sysupgrade-nbg6817/root -O > /dev/mmcblk0p10

make an overlay:
find out the offset:

root@OpenWrt:/tmp# tar xf openwrt-ipq806x-generic-zyxel_nbg6817-squashfs-sysupgrade.bin sysupgrade-nbg6817/root -O | wc -c
6553600

find free loop:

root@OpenWrt:/tmp# losetup -f
/dev/loop1

Create and format it:

root@OpenWrt:/tmp# losetup -o 6553600 /dev/loop1 /dev/mmcblk0p10
root@OpenWrt:/tmp# mkfs.ext4 -F -L rootfs_data /dev/loop1
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 849847 4k blocks and 212576 inodes
Filesystem UUID: ae0d6b77-d336-4bdc-a899-12436c82e0f9
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

Now change uboot-env:
get the current value:

root@OpenWrt:/tmp# fw_printenv flashargs flashargs_1
flashargs=setenv bootargs board=NBG6817 root=/dev/mmcblk0p5 rootwait ${bootmode} ${zld_ver}
flashargs_1=setenv bootargs board=NBG6817 root=/dev/mmcblk0p8 rootwait ${bootmode} ${zld_ver}

The first variable is responsible for booting from the main partition (FF), the second for booting from the alternate partition (01)
change the first variable:

root@OpenWrt:/tmp# fw_setenv flashargs 'setenv bootargs board=NBG6817 root=/dev/mmcblk0p10 rootwait ${bootmode} ${zld_ver}'

checking:

root@OpenWrt:/tmp# fw_printenv flashargs flashargs_1
flashargs=setenv bootargs board=NBG6817 root=/dev/mmcblk0p10 rootwait ${bootmode} ${zld_ver}
flashargs_1=setenv bootargs board=NBG6817 root=/dev/mmcblk0p8 rootwait ${bootmode} ${zld_ver}

Everything. Now you can reboot into the main partition and check the result:

root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 6.5M      6.5M         0 100% /rom
tmpfs                   232.6M    264.0K    232.4M   0% /tmp
/dev/loop0                3.1G    104.0K      2.9G   0% /overlay
overlayfs:/overlay        3.1G    104.0K      2.9G   0% /
tmpfs                   512.0K         0    512.0K   0% /dev

And no block-mount, blockd are no longer needed ;)
Sorry for my english, google translate.

pwned-pixel added a commit to pwned-pixel/openwrt that referenced this issue Oct 26, 2023
pwned-pixel added a commit to pwned-pixel/openwrt that referenced this issue Oct 26, 2023
pwned-pixel added a commit to pwned-pixel/openwrt that referenced this issue Oct 26, 2023
pwned-pixel added a commit to pwned-pixel/openwrt that referenced this issue Oct 26, 2023
pwned-pixel added a commit to pwned-pixel/openwrt that referenced this issue Oct 28, 2023
pwned-pixel added a commit to pwned-pixel/openwrt that referenced this issue Oct 28, 2023
pwned-pixel added a commit to pwned-pixel/openwrt that referenced this issue Oct 29, 2023
pwned-pixel added a commit to pwned-pixel/openwrt that referenced this issue Oct 29, 2023
pwned-pixel added a commit to pwned-pixel/openwrt that referenced this issue Oct 29, 2023
@projectgus
Copy link

projectgus commented Nov 9, 2023

I've been following this bug for a few years and wanted to check if there's anything I can do to help. This is my understanding of the current situation, please correct me if I'm wrong:

  • @lu-zero's patches to fstools have been merged by @dangowrt. This issue is fixed on the fstools master branch.
  • Openwrt master branch (and current stable v23.05.0) are packaging the fstools commit from before these fixes (2023-02-28, commit bfe882d5).

Is there anything a motivated bystander can do to help bump the version of fstools in openwrt master? Is more testing needed?

(I assume submitting a patch to bump the version in package/system/fstools/Makefile is the preserve of the maintainers, otherwise I'm also happy to do that!)

@matysekg
Copy link

Hi,
For rationale of this fix - I have BPI-R3, and apparently emmc doesn't work. I have a NVME disk, but cannot use it as extroot. I'm stuck with NAND, and it gets filled with packages pretty quickly..

@VA1DER
Copy link

VA1DER commented Nov 23, 2023

I have BPI-R3

This fix is important in its own right, but if you'll email me or contact me on the forum I'm pretty sure I can help you get your device working with either its internal NVME booting from an sdcard.

@taylor-cz
Copy link

Is there anything new?
How can we finally make the fix available in openwrt distributions?

@xrh0905
Copy link

xrh0905 commented Feb 17, 2024

It's already 2024 but there aren't any solution to this.
Looking forward to any direct patch on precompile x86 image.

@dangowrt
Copy link
Member

Afaik @lu-zero addressed this with commits merged by now:
openwrt/fstools@c1a8d95
openwrt/fstools@9cd09d4
openwrt/fstools@56a9b4e
openwrt/fstools@3a07943

@taylor-cz
Copy link

Is it now part of official openwrt images?

@xrh0905
Copy link

xrh0905 commented Feb 18, 2024

At least I tried with latest stable 23.05.2 combined-squashfs-efi no luck at all.
It just stuck with the hidden f2fs.

@dangowrt
Copy link
Member

You can try this in snapshot images, in 23.05.x this is not yet included (but we may include it in the next 23.05.x point release once users confirm that it fixes the issues previously affecting extroot use)

@xrh0905
Copy link

xrh0905 commented Feb 18, 2024

I'll go test that out and report back.
Thanks for the hint.

@xrh0905
Copy link

xrh0905 commented Feb 18, 2024

Nope it doesn't seem to work.

@tkapin
Copy link

tkapin commented Feb 23, 2024

Finally got to this thread after spending decent hours on figuring out where the problem with configuring an overlay is on my Turris Omnia after I've flashed OpenWrt to it. It'd be great to see this finally fixed, thank you to everyone it this thread for allowing me to understand where the problem is.

@taylor-cz
Copy link

There are more Turris Omnia owners who are awaiting eagerly any new info about this issue.

@tkapin
Copy link

tkapin commented Feb 23, 2024

Just to confirm, I was able to resolve the problem by flashing the latest snapshot on my Turris Omnia. Steps I've performed, for anyone who's in a similar situation.

  1. Figured out the right snapshot by running
    ubus call system board. This yielded "target": "mvebu/cortexa9".
  2. Navigated to the snapshots downloads at https://downloads.openwrt.org/snapshots/targets/mvebu/cortexa9/ (based on the target above)
  3. Downloaded the latest snapshot for my Turris Omnia (cznic_turris-omnia-squashfs-sysupgrade.img.gz) from that page
  4. Flashed the FW via LuCI
  5. Inserted USB flash to the Turris
  6. Followed the steps at https://openwrt.org/docs/guide-user/additional-software/extroot_configuration
  7. Logged in and checked the USB is mounted properly as the overlay

Content of the /etc/config/fstab shown via uci show fstab, for reference:

root@turris:~# uci show fstab
fstab.@global[0]=global
fstab.@global[0].anon_swap='0'
fstab.@global[0].anon_mount='0'
fstab.@global[0].auto_swap='1'
fstab.@global[0].auto_mount='0'
fstab.@global[0].delay_root='20'
fstab.@global[0].check_fs='0'
fstab.rwm=mount
fstab.rwm.device='/dev/loop0'
fstab.rwm.target='/rwm'
fstab.rwm.enable='1'
fstab.extroot=mount
fstab.extroot.uuid='b43d06f8-65c1-45cc-aa49-905d77f3a779'
fstab.extroot.target='/overlay'
root@turris:~# block info
/dev/mmcblk0p1: UUID="65D7-0315" VERSION="FAT16" TYPE="vfat"
/dev/mmcblk0p2: UUID="7ef9c660-31553901-5a19a5ee-6cd2ee75" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/sdb1: UUID="b43d06f8-65c1-45cc-aa49-905d77f3a779" LABEL="extroot" VERSION="1.0" MOUNT="/overlay" TYPE="ext4"

Validation of the overlay FS mounted properly from the USB partition:

root@turris:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.0M      4.0M         0 100% /rom
tmpfs                  1008.1M    252.0K   1007.8M   0% /tmp
/dev/sdb1                14.1G      3.7M     13.4G   0% /overlay
overlayfs:/overlay       14.1G      3.7M     13.4G   0% /
tmpfs                   512.0K         0    512.0K   0% /dev

Thanks to everyone in this thread for helping me to resolve the issue, I hope this will help others to resolve the issue. It'd be great to see this included in the main release and the issue closed.

@taylor-cz
Copy link

Awesome, thanks for detailed description. I will try later too.
I wonder why it didn't work for @xrh0905 .

@xrh0905
Copy link

xrh0905 commented Feb 24, 2024

I'm wondering too, I'll try again with the instruction of tkapin mentioned.
Prehaps I missed something?

@xrh0905
Copy link

xrh0905 commented Feb 24, 2024

Follow the same steps and no luck.
I grab the Analyze the preinit stage of the boot log with the command mention in the same page of instructions

root@OpenWrt:~# block info; uci show fstab; logread | sed -n -e "/- preinit -/,/- init -/p"
/dev/loop0: UUID="0948cdf0-a9ad-40ac-9b9d-42792c0166ca" LABEL="rootfs_data" VERSION="1.16" MOUNT="/rom/overlay" TYPE="f2fs"
/dev/sda1: UUID="1234-ABCD" LABEL="kernel" VERSION="FAT16" MOUNT="/boot" TYPE="vfat"
/dev/sda2: UUID="a8c8b112-78eb0163-3d8550be-0d816278" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/sdb1: UUID="ed1e19a8-bd60-458b-9caf-49165ba82c47" LABEL="extroot" VERSION="1.0" MOUNT="/rom/overlay" TYPE="ext4"
fstab.@global[0]=global
fstab.@global[0].anon_swap='0'
fstab.@global[0].anon_mount='0'
fstab.@global[0].auto_swap='1'
fstab.@global[0].auto_mount='1'
fstab.@global[0].delay_root='5'
fstab.@global[0].check_fs='0'
fstab.@mount[0]=mount
fstab.@mount[0].target='/overlay'
fstab.@mount[0].uuid='0948cdf0-a9ad-40ac-9b9d-42792c0166ca'
fstab.@mount[0].enabled='0'
fstab.@mount[1]=mount
fstab.@mount[1].target='/boot'
fstab.@mount[1].uuid='1234-ABCD'
fstab.@mount[1].enabled='0'
fstab.@mount[2]=mount
fstab.@mount[2].target='/rom'
fstab.@mount[2].uuid='a8c8b112-78eb0163-3d8550be-0d816278'
fstab.@mount[2].enabled='0'
fstab.extroot=mount
fstab.extroot.uuid='ed1e19a8-bd60-458b-9caf-49165ba82c47'
fstab.extroot.target='/rom/overlay'
fstab.rwm=mount
fstab.rwm.device='/dev/loop0'
fstab.rwm.target='/rwm'
Sat Feb 24 04:09:53 2024 user.info kernel: [    6.455213] init: - preinit -
Sat Feb 24 04:09:53 2024 kern.info kernel: [    7.357885] 8021q: adding VLAN 0 to HW filter on device eth0
Sat Feb 24 04:09:53 2024 kern.info kernel: [    9.537156] loop0: detected capacity change from 0 to 212992
Sat Feb 24 04:09:53 2024 kern.info kernel: [    9.599513] loop0: detected capacity change from 212992 to 204928
Sat Feb 24 04:09:53 2024 kern.notice kernel: [    9.917037] F2FS-fs (loop0): Mounted with checkpoint version = 44a7eff9
Sat Feb 24 04:09:53 2024 user.info kernel: [    9.970504] mount_root: loading kmods from internal overlay
Sat Feb 24 04:09:53 2024 user.info kernel: [   10.015785] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Sat Feb 24 04:09:53 2024 user.info kernel: [   10.034342] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Sat Feb 24 04:09:53 2024 user.info kernel: [   10.325152] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Sat Feb 24 04:09:53 2024 user.info kernel: [   10.356573] block: extroot: not configured
Sat Feb 24 04:09:53 2024 user.info kernel: [   10.360717] block: attempting to load /etc/config/fstab
Sat Feb 24 04:09:53 2024 user.err kernel: [   10.370260] block: unable to load configuration (fstab: Entry not found)
Sat Feb 24 04:09:53 2024 user.err kernel: [   10.372395] block: no usable configuration
Sat Feb 24 04:09:53 2024 user.info kernel: [   10.379497] mount_root: switching to f2fs overlay
Sat Feb 24 04:09:53 2024 kern.warn kernel: [   10.389118] overlayfs: null uuid detected in lower fs '/', falling back to xino=off,index=off,nfs_export=off.
Sat Feb 24 04:09:53 2024 user.warn kernel: [   10.988727] urandom-seed: Seeding with /etc/urandom.seed
Sat Feb 24 04:09:53 2024 user.info kernel: [   11.127429] procd: - early -
Sat Feb 24 04:09:53 2024 user.info kernel: [   11.915918] procd: - ubus -
Sat Feb 24 04:09:53 2024 user.info kernel: [   12.175554] procd: - init -

Did I missed anything?
Using x86_64 efi combined squashfs

@tkapin
Copy link

tkapin commented Feb 24, 2024

@xrh0905: I think the /overlay in the fstab config should point directly to the /dev/sdb1 device rather to the /dev/loop0. In your case, it still points to the /dev/loop0 device (the one with UUID starting with "0948").

Can you try to update the /etc/config/fstab (either directly or via uci set commands) so that the original /overlay mount points to your /dev/sdb1 partition, i.e., in your case:

config mount 'extroot'
    option target '/overlay'
    option uuid 'ed1e19a8-bd60-458b-9caf-49165ba82c47'

Eventually, you could try specifying option device '/dev/sdb1' instead of the uuid option.

Also, as described in the https://openwrt.org/docs/guide-user/additional-software/extroot_configuration#troubleshooting, you might need to set the config global / option delay_root to some higher value (e.g., 15 to 20) to give the partition enough time to mount during the boot.

@tkapin
Copy link

tkapin commented Feb 25, 2024

Just verified that the setup I've described above works both with external USB and the internal mSATA SSD disks on my Turris Omnia.

@xrh0905
Copy link

xrh0905 commented Mar 1, 2024

@tkapin Sorry for the late reply, been busy with something else.
I tried to increase the delay_root and it seems to working now, except it got mount under /rom/overlay instead of /overlay
Opkg seems fine, and I'll go check whether it's fully functional.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests