knodel:

I wrote a small patch that allows proper rootfs and fstab discovery on devices where the rootfs+overlay reside on a block device partition.

Here the dmesg of a current openwrt HEAD build with the patch:

Mon Apr  8 22:25:35 2019 user.info kernel: [    3.335289] init: - preinit -
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.035073] mount_root: loading kmods from internal overlay
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.046998] kmodloader: loading kernel modules from //etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.048462] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.828623] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.828798] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.834600] block: attempting to load /tmp/overlay/etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.841281] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.847486] block: attempting to load /etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.854319] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.859098] block: no usable configuration
Mon Apr  8 22:25:35 2019 kern.info kernel: [    8.985003] EXT4-fs (loop0): recovery complete
Mon Apr  8 22:25:35 2019 kern.info kernel: [    8.986207] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.988754] mount_root: loading kmods from internal overlay
Mon Apr  8 22:25:35 2019 user.info kernel: [    9.013356] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    9.013492] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.011824] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Mon Apr  8 22:25:35 2019 kern.info kernel: [   10.017872] EXT4-fs (mmcblk0p10): mounted filesystem with ordered data mode. Opts:
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.162997] mount_root: switched to extroot
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.331230] procd: - early -
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.331412] procd: - watchdog -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.009306] procd: - watchdog -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.009775] procd: - ubus -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.064866] procd: - init -

Unrelated dmesg entries omitted, full log here: https://gist.github.com/knuddelknoedel/46db572959056d2e69fd861d59ad0daf

val-kulkov:

The proposed patch works beautifully to enable extroot on my x86_64 device. With it, I can finally make use of the available space on my /dev/sda without worrying about losing data on it upon a sysupgrade.

The sysupgrade process on a x86_64 system that I use to avoid losing data is not yet a straightforward one. First, I burn openwrt-x86-64-combined-squashfs.img on a USB drive. Then, I copy first two partitions from the USB drive to the corresponding partitions on my /dev/sda. Finally, I copy the MBR bootstrap only, without the partition table: "dd if=/dev/sdc of=/dev/sda bs=446 count=1". This procedure retains sda3, my root overlay partition, and sda4, a swap partition.

utrumo:

Now on Zyxel Armor Z2 (NBG6817) extroot doesn't work without this patch.
Is there a developer of openwrt? Please add this patch to master! :)

To make work extroot on this router we need compile own firmware and apply this patch:

1. clone src and change dir to it: git clone https://github.com/openwrt/openwrt.git ~/source && cd ~/source
2. make directory for patch: mkdir ./package/system/fstools/patches
3. write patch content from https://patchwork.ozlabs.org/patch/1082599/ to ./package/system/fstools/patches/001-add_propper_rootfs_and_fstab_discovery_on_a_block_device_partitions.patch
4. build own image:
   ./scripts/feeds update -a
   ./scripts/feeds install -a
   export MAKEFLAGS=-j5
   make menuconfig
   ===================================================
   system => Qualcomm Atheros IPQ806X y
   Target Profile => Zyxel NBG6817 y
   Base system =>
   => block-mount y (for extroot on mmcblk0p10)
   => blockd y (for extroot on mmcblk0p10)
   Kernel modules => USB Support => kmod-usb-storage y (for usb-hdd support)
   LuCI =>
   => Collections => luci y
   => Applications => luci-app-advanced-reboot y
   ===================================================
   make defconfig
   make download
   ionice -c 3 nice -n19 make

results you can find in:
./bin/targets/ipq806x/generic/

5. flash this firmware to router
   scp ./bin/targets/ipq806x/generic/openwrt-ipq806x-zyxel_nbg6817-squashfs-sysupgrade.bin root@openwrt:/tmp
   ssh root@openwrt
   sysupgrade -v /tmp/openwrt-ipq806x-zyxel_nbg6817-squashfs-sysupgrade.bin

6. set extroot:
   ssh root@openwrt
   DEVICE="$(awk -e '/\s/overlay\s/{print $1}' /etc/mtab)"
   uci -q delete fstab.rwm
   uci set fstab.rwm="mount"
   uci set fstab.rwm.device="${DEVICE}"
   uci set fstab.rwm.target="/rwm"
   uci set fstab.rwm.enabled="1"
   uci commit fstab

DEVICE="/dev/mmcblk0p10" // or if you want to use usb-hdd: DEVICE="/dev/sda1"
mkfs.ext4 "${DEVICE}"
eval $(blockinfo"${DEVICE}" | grep -o -e "UUID=\S*")
uci -q delete fstab.overlay
uci set fstab.overlay="mount"
uci set fstab.overlay.uuid="${UUID}"
uci set fstab.overlay.target="/overlay"
uci set fstab.overlay.enabled="1"
uci commit fstab

mount "${DEVICE}" /mnt
cp -a -f /overlay/. /mnt
umount /mnt

reboot

rmilecki:

That fstools patch cannot be applied as is (I reviewed & commented on it).

I pushed three related fstools changes:

• [[https://git.openwrt.org/?p=project/fstools.git;a=commitdiff;h=d70774de24413f91087e9c3c873cf7a4ec3df813|d70774de2441 ("block: add some basic extroot documentation")]]
• [[https://git.openwrt.org/?p=project/fstools.git;a=commitdiff;h=37c9148a3ae17665e75f99b459b6ff2b461d6739|37c9148a3ae1 ("block: simplify check_extroot() a bit")]]
• [[https://git.openwrt.org/?p=project/fstools.git;a=commitdiff;h=8b9e601160712f74ff46313579d0225a7f90ca3c|8b9e60116071 ("block: always use st_dev (device ID) of / when looking for root")]]

That moves us closer to fixing this issue.

Someone now needs to improve main_extroot() and make it:

• Find standard overlay device (e.g. /dev/sda3)
• Mount it using f2fs
• Call mount_extroot() for it

It's basically the same logic as what we already have for JFFS2 and UBIFS in that main_extroot() function.

leifliddy:

If we're going to use f2fs as the filesystem for the extroot overlay device (which is a great idea), then we should probably include these package as part of the base build for the zyxel_nbg6817:
f2fsck
f2fs-tools
kmod-fs-f2fs
mkf2fs

Actually, this brings up another question, why aren't we using the f2fs filesystem for the default overlay device?

/dev/loop0 on /overlay type ext4 (rw,noatime,data=ordered)

rmilecki:

I just noticed that orignal report was about ext4. Recently I was chatting with m4t who was using f2fs.

So this task seems more generic: we need to check /dev/loop0 for /etc/config/fstab with overlay setup. That /dev/loop0 may be ext4 or f2fs (or other?).

leifliddy:

I'm not sure if this is the correct forum for this or not, but here goes.
If you look at partition schedule for the nbg6817

https://openwrt.org/toh/zyxel/nbg6817

You'll see that mmcblk0p5 is divided up into two parts

mmcblk0p5 rootfs 64 MiB

/rom (squashfs) 4MB
/overlay (ext4) 60MB (/dev/loop0)

So, if you've configured mmcblk0p10 to act as the /overlay,
then the 4MB squashfs filesystem from mmcblk0p5 will continue to act as the 4Mb read-only filesystem?

Is there a reason we can't just re-partition this entire device, I mean there's no reason for openwrt to need 10 partitions.
Or is there some sort of unspoken rule that you need to operate within the confines of the OEM flash layout?

If we need to maintain the OEM partitioning scheme, then we should make mmcblk0p1 the 4MB /rom partition
...and make mmcblk0p10 the default /overlay partition.

That's makes the most sense...but I'll save this discussion for another day. Need to do a bit more research on how openwrt is designed....

leifliddy:

@ken would you be able to create a new patch based on the changes Rafał made to fstools?

tofurky:

leifliddy, this works for me with f2fs on 19.07.3. note that the change to check_filesystem() isn't strictly necessary, but i found that newer f2fs-tools gave some issues without it.

diff --git a/block.c b/block.c
index 569bf56..b88e208 100644
--- a/block.c
+++ b/block.c
@@ -747,7 +747,7 @@ static void check_filesystem(struct probe_info *pr)
        pid = fork();
        if (!pid) {
                if(!strncmp(pr->type, "f2fs", 4)) {
-                       execl(ckfs, ckfs, "-f", pr->dev, NULL);
+                       execl(ckfs, ckfs, "-p", "2", "-f", pr->dev, NULL);
                        exit(EXIT_FAILURE);
                } else if(!strncmp(pr->type, "btrfs", 5)) {
                        execl(ckfs, ckfs, "--repair", pr->dev, NULL);
@@ -1591,7 +1591,7 @@ static int main_extroot(int argc, char **argv)
 #endif
 
        /* As a last resort look for /etc/config/fstab on "rootfs" partition */
-       return mount_extroot(NULL);
+       return mount_extroot("/tmp/overlay");
 }
 
 static int main_mount(int argc, char **argv)

i spent a bit of time trying to figure out the proper way to do it per rafal's suggestion, but was running into segfaults with find_block(NULL, NULL, "loop0", NULL) so i just took the easy way out ;) not sure if i was even close to taking the right approach with that, though.

katrat:

Hi,

I guess this fix wasn't included in the 19.07.3 release yet, since I am struggling to set up overlayfs on my new NBG6817 with usb hard disk and ext4.

Are there any patched images to download/test? Any chance this will be fixed in one of the next releases?

I am not that experienced to build/figure out this myself, so any links/instructions are much appreciated. On the other hand, my zyxel isn't productive yet, so I am willing to test...

Thanks,
kat

rmilecki:

Noone cared to develop a proper patch, people keep posting some hacks that are unacceptable due to breaking other setups.

katrat:

Thanks for the quick, if also a bit unsatisfying, answer. I got the zyxel because it seemed to be properly supported by openwrt (and having a bit more oomph than my ageing tp-link). I don't really understand where the problem is: ext4 of f2fs, hard disk or flash drive, or the drives not being ready at boot time, or just some configuration hiccup.

Can you suggest what I could do? I don't mind playing around a bit.

rmilecki:

Someone now needs to improve main_extroot() and make it:

1. Find standard overlay device (e.g. /dev/sda3)
2. Mount it using f2fs
3. Call mount_extroot() for it

It's basically the same logic as what we already have for JFFS2 and UBIFS in that main_extroot() function.

I just noticed that orignal report was about ext4. Recently I was chatting with m4t who was using f2fs.

So this task seems more generic: we need to check /dev/loop0 for /etc/config/fstab with overlay setup. That /dev/loop0 may be ext4 or f2fs (or other?).

visualage:

Is it reasonable to use libfstools to search for rootfs_data volume and mount it, just like what mount_root.c does?

visualage:

I made a patch to openwrt to support extroot on non-MTD rootfs_data volumes. It uses libfstools in block.c to support other squashfs volumes, such as /dev/sd* or /dev/mmc*.

The patch itself contains a patch to fstools project that can be applied to that codebase directly.

I tested the build locally on a x86-64 virtualbox environment as well as rockchip friendlyarm nanopi r2s. It should work on any other environments as long as mount_root works since they use the same underlying library.

visualage:

This is the fstools patch. The openwrt patch is just to make sure block-mount depends on libfstools when packaging.
Rafal, would you please take a look?

kofec:

Hi,
Aby update about this issue ?

uutrumo:

Qi Liu, for wich version of openwrt your patch?
I try it on v19.07.7 and have got an error:
Package block-mount is missing dependencies for the following libraries:
libfstools.so

uutrumo:

tofurky, your hack works, thank you.
Some patch for v19.07.7 and v21.02.0-rc1:

--- a/block.c
+++ b/block.c
@@ -747,7 +747,7 @@ static void check_filesystem(struct prob
 	pid = fork();
 	if (!pid) {
 		if(!strncmp(pr->type, "f2fs", 4)) {
-			execl(ckfs, ckfs, "-f", pr->dev, NULL);
+			execl(ckfs, ckfs, "-p", "2", "-f", pr->dev, NULL);
 			exit(EXIT_FAILURE);
 		} else if(!strncmp(pr->type, "btrfs", 5)) {
 			execl(ckfs, ckfs, "--repair", pr->dev, NULL);
@@ -1591,7 +1591,7 @@ static int main_extroot(int argc, char *
 #endif
 
 	/* As a last resort look for /etc/config/fstab on "rootfs" partition */
-	return mount_extroot(NULL);
+	return mount_extroot("/tmp/overlay");
 }
 
 static int main_mount(int argc, char **argv)

luce.nera:

How do I apply these patches and which one to use?

flowercow:

i met this bug in 21.02.0-rc4 installed in kvm.

Is there any other method to solve this problem?

For example, do something in /etc/preinit?

Gandalf:

Still got the issue in 21.02.0 !!!
Is there any work in this patch ?

flowercow:

i have tried to recompile firmware applying patch as mentioned by utrumo above.
after upgrade to that firmware and install block-mount package that recompiled, this issue was not fixed.

Op3n:

The same bug occurs on the latest stable release 20.02, Is there any way to avoid this bug other than by compiling it myself? I prefer use official pre-compiled version on downloads.openwrt.com

katrat:

Well, any solutions apart from patching and compiling openwrt ourselves? Does anyone have a precompiled working solution for 21.02?

siicosmos:

Need some solution for 21.02 as well. Either patches or a pre-compiled image is fine...

siicosmos:

Updates:
Actually, the patch tofurky provided actually works in 21.02. You need to apply the patch using Quilt or edit directly on block.c file.

Is there a timeline for this being officially fixed?

There is an alternative way for Zyxel NBG6817: change the uboot-env variable so that rootfs is loaded from /dev/mmсblk0p10

Example:
Boot into an alternative configuration (which is 01, not FF)
We copy the sysupgrade.bin to the router in the folder /tmp
Connect to it via ssh
Manual flash firmvare:
(based on a standard script: /lib/upgrade/zyxel.sh)

view at the contents of the firmware:

root@OpenWrt:~# cd /tmp
root@OpenWrt:/tmp# tar tf openwrt-ipq806x-generic-zyxel_nbg6817-squashfs-sysupgrade.bin
tar: sysupgrade-nbg6817/CONTROL: time stamp 2022-03-07 10:14:29 is 42124622.3872312 s in the future
tar: sysupgrade-nbg6817/kernel: time stamp 2022-03-07 10:14:29 is 42124622.35407792 s in the future
tar: sysupgrade-nbg6817/root: time stamp 2022-03-07 10:14:29 is 42124622.32335552 s in the future
tar: sysupgrade-nbg6817: time stamp 2022-03-07 10:14:29 is 42124622.32289632 s in the future

flash: (tar xf <<filename.bin>> <<contentname>> -O > <<path>> )

root@OpenWrt:/tmp# tar xf openwrt-ipq806x-generic-zyxel_nbg6817-squashfs-sysupgrade.bin sysupgrade-nbg6817/kernel -O > /dev/mmcblk0p4
root@OpenWrt:/tmp# tar xf openwrt-ipq806x-generic-zyxel_nbg6817-squashfs-sysupgrade.bin sysupgrade-nbg6817/root -O > /dev/mmcblk0p10

make an overlay:
find out the offset:

root@OpenWrt:/tmp# tar xf openwrt-ipq806x-generic-zyxel_nbg6817-squashfs-sysupgrade.bin sysupgrade-nbg6817/root -O | wc -c
6553600

find free loop:

root@OpenWrt:/tmp# losetup -f
/dev/loop1

Create and format it:

root@OpenWrt:/tmp# losetup -o 6553600 /dev/loop1 /dev/mmcblk0p10
root@OpenWrt:/tmp# mkfs.ext4 -F -L rootfs_data /dev/loop1
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 849847 4k blocks and 212576 inodes
Filesystem UUID: ae0d6b77-d336-4bdc-a899-12436c82e0f9
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

Now change uboot-env:
get the current value:

root@OpenWrt:/tmp# fw_printenv flashargs flashargs_1
flashargs=setenv bootargs board=NBG6817 root=/dev/mmcblk0p5 rootwait ${bootmode} ${zld_ver}
flashargs_1=setenv bootargs board=NBG6817 root=/dev/mmcblk0p8 rootwait ${bootmode} ${zld_ver}

The first variable is responsible for booting from the main partition (FF), the second for booting from the alternate partition (01)
change the first variable:

root@OpenWrt:/tmp# fw_setenv flashargs 'setenv bootargs board=NBG6817 root=/dev/mmcblk0p10 rootwait ${bootmode} ${zld_ver}'

checking:

root@OpenWrt:/tmp# fw_printenv flashargs flashargs_1
flashargs=setenv bootargs board=NBG6817 root=/dev/mmcblk0p10 rootwait ${bootmode} ${zld_ver}
flashargs_1=setenv bootargs board=NBG6817 root=/dev/mmcblk0p8 rootwait ${bootmode} ${zld_ver}

Everything. Now you can reboot into the main partition and check the result:

root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 6.5M      6.5M         0 100% /rom
tmpfs                   232.6M    264.0K    232.4M   0% /tmp
/dev/loop0                3.1G    104.0K      2.9G   0% /overlay
overlayfs:/overlay        3.1G    104.0K      2.9G   0% /
tmpfs                   512.0K         0    512.0K   0% /dev

And no block-mount, blockd are no longer needed ;)
Sorry for my english, google translate.

I've been following this bug for a few years and wanted to check if there's anything I can do to help. This is my understanding of the current situation, please correct me if I'm wrong:

• @lu-zero's patches to fstools have been merged by @dangowrt. This issue is fixed on the fstools master branch.
• Openwrt master branch (and current stable v23.05.0) are packaging the fstools commit from before these fixes (2023-02-28, commit bfe882d5).

Is there anything a motivated bystander can do to help bump the version of fstools in openwrt master? Is more testing needed?

(I assume submitting a patch to bump the version in package/system/fstools/Makefile is the preserve of the maintainers, otherwise I'm also happy to do that!)

Hi,
For rationale of this fix - I have BPI-R3, and apparently emmc doesn't work. I have a NVME disk, but cannot use it as extroot. I'm stuck with NAND, and it gets filled with packages pretty quickly..

I have BPI-R3

This fix is important in its own right, but if you'll email me or contact me on the forum I'm pretty sure I can help you get your device working with either its internal NVME booting from an sdcard.

Is there anything new?
How can we finally make the fix available in openwrt distributions?

It's already 2024 but there aren't any solution to this.
Looking forward to any direct patch on precompile x86 image.

Afaik @lu-zero addressed this with commits merged by now:
openwrt/fstools@c1a8d95
openwrt/fstools@9cd09d4
openwrt/fstools@56a9b4e
openwrt/fstools@3a07943

Is it now part of official openwrt images?

At least I tried with latest stable 23.05.2 combined-squashfs-efi no luck at all.
It just stuck with the hidden f2fs.

You can try this in snapshot images, in 23.05.x this is not yet included (but we may include it in the next 23.05.x point release once users confirm that it fixes the issues previously affecting extroot use)

I'll go test that out and report back.
Thanks for the hint.

Nope it doesn't seem to work.

Finally got to this thread after spending decent hours on figuring out where the problem with configuring an overlay is on my Turris Omnia after I've flashed OpenWrt to it. It'd be great to see this finally fixed, thank you to everyone it this thread for allowing me to understand where the problem is.

There are more Turris Omnia owners who are awaiting eagerly any new info about this issue.

Just to confirm, I was able to resolve the problem by flashing the latest snapshot on my Turris Omnia. Steps I've performed, for anyone who's in a similar situation.

1. Figured out the right snapshot by running
   ubus call system board. This yielded "target": "mvebu/cortexa9".
2. Navigated to the snapshots downloads at https://downloads.openwrt.org/snapshots/targets/mvebu/cortexa9/ (based on the target above)
3. Downloaded the latest snapshot for my Turris Omnia (cznic_turris-omnia-squashfs-sysupgrade.img.gz) from that page
4. Flashed the FW via LuCI
5. Inserted USB flash to the Turris
6. Followed the steps at https://openwrt.org/docs/guide-user/additional-software/extroot_configuration
7. Logged in and checked the USB is mounted properly as the overlay

Content of the /etc/config/fstab shown via uci show fstab, for reference:

root@turris:~# uci show fstab
fstab.@global[0]=global
fstab.@global[0].anon_swap='0'
fstab.@global[0].anon_mount='0'
fstab.@global[0].auto_swap='1'
fstab.@global[0].auto_mount='0'
fstab.@global[0].delay_root='20'
fstab.@global[0].check_fs='0'
fstab.rwm=mount
fstab.rwm.device='/dev/loop0'
fstab.rwm.target='/rwm'
fstab.rwm.enable='1'
fstab.extroot=mount
fstab.extroot.uuid='b43d06f8-65c1-45cc-aa49-905d77f3a779'
fstab.extroot.target='/overlay'

root@turris:~# block info
/dev/mmcblk0p1: UUID="65D7-0315" VERSION="FAT16" TYPE="vfat"
/dev/mmcblk0p2: UUID="7ef9c660-31553901-5a19a5ee-6cd2ee75" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/sdb1: UUID="b43d06f8-65c1-45cc-aa49-905d77f3a779" LABEL="extroot" VERSION="1.0" MOUNT="/overlay" TYPE="ext4"

Validation of the overlay FS mounted properly from the USB partition:

root@turris:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.0M      4.0M         0 100% /rom
tmpfs                  1008.1M    252.0K   1007.8M   0% /tmp
/dev/sdb1                14.1G      3.7M     13.4G   0% /overlay
overlayfs:/overlay       14.1G      3.7M     13.4G   0% /
tmpfs                   512.0K         0    512.0K   0% /dev

Thanks to everyone in this thread for helping me to resolve the issue, I hope this will help others to resolve the issue. It'd be great to see this included in the main release and the issue closed.

Awesome, thanks for detailed description. I will try later too.
I wonder why it didn't work for @xrh0905 .

I'm wondering too, I'll try again with the instruction of tkapin mentioned.
Prehaps I missed something?

Follow the same steps and no luck.
I grab the Analyze the preinit stage of the boot log with the command mention in the same page of instructions

root@OpenWrt:~# block info; uci show fstab; logread | sed -n -e "/- preinit -/,/- init -/p"
/dev/loop0: UUID="0948cdf0-a9ad-40ac-9b9d-42792c0166ca" LABEL="rootfs_data" VERSION="1.16" MOUNT="/rom/overlay" TYPE="f2fs"
/dev/sda1: UUID="1234-ABCD" LABEL="kernel" VERSION="FAT16" MOUNT="/boot" TYPE="vfat"
/dev/sda2: UUID="a8c8b112-78eb0163-3d8550be-0d816278" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/sdb1: UUID="ed1e19a8-bd60-458b-9caf-49165ba82c47" LABEL="extroot" VERSION="1.0" MOUNT="/rom/overlay" TYPE="ext4"
fstab.@global[0]=global
fstab.@global[0].anon_swap='0'
fstab.@global[0].anon_mount='0'
fstab.@global[0].auto_swap='1'
fstab.@global[0].auto_mount='1'
fstab.@global[0].delay_root='5'
fstab.@global[0].check_fs='0'
fstab.@mount[0]=mount
fstab.@mount[0].target='/overlay'
fstab.@mount[0].uuid='0948cdf0-a9ad-40ac-9b9d-42792c0166ca'
fstab.@mount[0].enabled='0'
fstab.@mount[1]=mount
fstab.@mount[1].target='/boot'
fstab.@mount[1].uuid='1234-ABCD'
fstab.@mount[1].enabled='0'
fstab.@mount[2]=mount
fstab.@mount[2].target='/rom'
fstab.@mount[2].uuid='a8c8b112-78eb0163-3d8550be-0d816278'
fstab.@mount[2].enabled='0'
fstab.extroot=mount
fstab.extroot.uuid='ed1e19a8-bd60-458b-9caf-49165ba82c47'
fstab.extroot.target='/rom/overlay'
fstab.rwm=mount
fstab.rwm.device='/dev/loop0'
fstab.rwm.target='/rwm'
Sat Feb 24 04:09:53 2024 user.info kernel: [    6.455213] init: - preinit -
Sat Feb 24 04:09:53 2024 kern.info kernel: [    7.357885] 8021q: adding VLAN 0 to HW filter on device eth0
Sat Feb 24 04:09:53 2024 kern.info kernel: [    9.537156] loop0: detected capacity change from 0 to 212992
Sat Feb 24 04:09:53 2024 kern.info kernel: [    9.599513] loop0: detected capacity change from 212992 to 204928
Sat Feb 24 04:09:53 2024 kern.notice kernel: [    9.917037] F2FS-fs (loop0): Mounted with checkpoint version = 44a7eff9
Sat Feb 24 04:09:53 2024 user.info kernel: [    9.970504] mount_root: loading kmods from internal overlay
Sat Feb 24 04:09:53 2024 user.info kernel: [   10.015785] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Sat Feb 24 04:09:53 2024 user.info kernel: [   10.034342] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Sat Feb 24 04:09:53 2024 user.info kernel: [   10.325152] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Sat Feb 24 04:09:53 2024 user.info kernel: [   10.356573] block: extroot: not configured
Sat Feb 24 04:09:53 2024 user.info kernel: [   10.360717] block: attempting to load /etc/config/fstab
Sat Feb 24 04:09:53 2024 user.err kernel: [   10.370260] block: unable to load configuration (fstab: Entry not found)
Sat Feb 24 04:09:53 2024 user.err kernel: [   10.372395] block: no usable configuration
Sat Feb 24 04:09:53 2024 user.info kernel: [   10.379497] mount_root: switching to f2fs overlay
Sat Feb 24 04:09:53 2024 kern.warn kernel: [   10.389118] overlayfs: null uuid detected in lower fs '/', falling back to xino=off,index=off,nfs_export=off.
Sat Feb 24 04:09:53 2024 user.warn kernel: [   10.988727] urandom-seed: Seeding with /etc/urandom.seed
Sat Feb 24 04:09:53 2024 user.info kernel: [   11.127429] procd: - early -
Sat Feb 24 04:09:53 2024 user.info kernel: [   11.915918] procd: - ubus -
Sat Feb 24 04:09:53 2024 user.info kernel: [   12.175554] procd: - init -

Did I missed anything?
Using x86_64 efi combined squashfs

@xrh0905: I think the /overlay in the fstab config should point directly to the /dev/sdb1 device rather to the /dev/loop0. In your case, it still points to the /dev/loop0 device (the one with UUID starting with "0948").

Can you try to update the /etc/config/fstab (either directly or via uci set commands) so that the original /overlay mount points to your /dev/sdb1 partition, i.e., in your case:

config mount 'extroot'
    option target '/overlay'
    option uuid 'ed1e19a8-bd60-458b-9caf-49165ba82c47'

Eventually, you could try specifying option device '/dev/sdb1' instead of the uuid option.

Also, as described in the https://openwrt.org/docs/guide-user/additional-software/extroot_configuration#troubleshooting, you might need to set the config global / option delay_root to some higher value (e.g., 15 to 20) to give the partition enough time to mount during the boot.

Just verified that the setup I've described above works both with external USB and the internal mSATA SSD disks on my Turris Omnia.

@tkapin Sorry for the late reply, been busy with something else.
I tried to increase the delay_root and it seems to working now, except it got mount under /rom/overlay instead of /overlay
Opkg seems fine, and I'll go check whether it's fully functional.

I'm curious if this fix will be intergrated into branch 23.05.

In my case, my ipq60xx device with emmc can't boot from switching to /overlay extroot in /etc/config/fstab
but after applying this fix
(only modifies the Makefile PKG_MIRROR_HASH,PKG_SOURCE_DATA,PKG_SOURCE_VERSION to b66a340 2024-04-22 commit),
it works.

Before:

Sat Jun 22 02:46:40 2024 user.info kernel: [   14.456553] block: attempting to load /etc/config/fstab
Sat Jun 22 02:46:40 2024 user.err kernel: [   14.459131] block: unable to load configuration (fstab: Entry not found)
Sat Jun 22 02:46:40 2024 user.err kernel: [   14.460614] block: no usable configuration
Sat Jun 22 02:46:40 2024 kern.info kernel: [   14.468230] loop0: detected capacity change from 0 to 122880
Sat Jun 22 02:46:40 2024 kern.info kernel: [   14.535127] loop0: detected capacity change from 122880 to 8832
Sat Jun 22 02:46:40 2024 kern.info kernel: [   14.555858] EXT4-fs (loop0): recovery complete
Sat Jun 22 02:46:40 2024 kern.info kernel: [   14.556414] EXT4-fs (loop0): mounted filesystem with ordered data mode. Quota mode: disabled.
Sat Jun 22 02:46:40 2024 user.info kernel: [   14.561917] mount_root: loading kmods from internal overlay
Sat Jun 22 02:46:40 2024 user.info kernel: [   14.592924] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Sat Jun 22 02:46:40 2024 user.info kernel: [   14.593703] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Sat Jun 22 02:46:40 2024 user.info kernel: [   14.717259] block: attempting to load /etc/config/fstab
Sat Jun 22 02:46:40 2024 user.err kernel: [   14.717345] block: unable to load configuration (fstab: Entry not found)
Sat Jun 22 02:46:40 2024 user.err kernel: [   14.721317] block: no usable configuration
Sat Jun 22 02:46:40 2024 user.info kernel: [   14.728948] mount_root: switching to ext4 overlay
Sat Jun 22 02:46:40 2024 kern.warn kernel: [   14.735463] overlayfs: null uuid detected in lower fs '/', falling back to xino=off,index=off,nfs_export=off.

After:

Mon Jun 24 03:44:49 2024 user.info kernel: [   14.100648] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Mon Jun 24 03:44:49 2024 user.err kernel: [   14.100760] block: unable to load configuration (fstab: Entry not found)
Mon Jun 24 03:44:49 2024 user.info kernel: [   14.106593] block: attempting to load /tmp/overlay/etc/config/fstab
Mon Jun 24 03:44:49 2024 user.err kernel: [   14.113239] block: unable to load configuration (fstab: Entry not found)
Mon Jun 24 03:44:49 2024 user.info kernel: [   14.119220] block: attempting to load /etc/config/fstab
Mon Jun 24 03:44:49 2024 user.err kernel: [   14.126253] block: unable to load configuration (fstab: Entry not found)
Mon Jun 24 03:44:49 2024 user.err kernel: [   14.131098] block: no usable configuration
Mon Jun 24 03:44:49 2024 user.info kernel: [   14.138177] block: attempting to load /etc/config/fstab
Mon Jun 24 03:44:49 2024 user.err kernel: [   14.141958] block: unable to load configuration (fstab: Entry not found)
Mon Jun 24 03:44:49 2024 user.err kernel: [   14.147075] block: no usable configuration
Mon Jun 24 03:44:49 2024 kern.info kernel: [   14.154560] loop0: detected capacity change from 0 to 122880
Mon Jun 24 03:44:49 2024 kern.info kernel: [   14.214251] loop0: detected capacity change from 122880 to 94848
Mon Jun 24 03:44:49 2024 kern.info kernel: [   14.256599] EXT4-fs (loop0): recovery complete
Mon Jun 24 03:44:49 2024 kern.info kernel: [   14.257155] EXT4-fs (loop0): mounted filesystem with ordered data mode. Quota mode: disabled.
Mon Jun 24 03:44:49 2024 user.info kernel: [   14.260307] mount_root: loading kmods from internal overlay
Mon Jun 24 03:44:49 2024 user.info kernel: [   14.286819] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Mon Jun 24 03:44:49 2024 user.info kernel: [   14.287063] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Mon Jun 24 03:44:49 2024 user.info kernel: [   14.413687] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Mon Jun 24 03:44:49 2024 kern.info kernel: [   14.506900] EXT4-fs (mmcblk0p27): mounted filesystem with ordered data mode. Quota mode: disabled.
Mon Jun 24 03:44:49 2024 kern.warn kernel: [   14.511366] overlayfs: null uuid detected in lower fs '/', falling back to xino=off,index=off,nfs_export=off.
Mon Jun 24 03:44:49 2024 kern.info kernel: [   14.517383] EXT4-fs (loop0): unmounting filesystem.
Mon Jun 24 03:44:49 2024 user.info kernel: [   14.527882] mount_root: switched to extroot

You can try this in snapshot images, in 23.05.x this is not yet included (but we may include it in the next 23.05.x point release once users confirm that it fixes the issues previously affecting extroot use)

Since 23.05.4 is out, would this patch be included in this release?

23.05.5 still doesn't work with Omnia.

The 24.10.0-rc5 works as expected, great thanks.

But I have problem to mount the overlay with the 'discard' option, the /etc/config/fstab:

config global
	option anon_swap '0'
	option anon_mount '0'
	option auto_swap '1'
	option auto_mount '1'
	option delay_root '20'
	option check_fs '0'

config mount 'rwm'
	option device '/dev/loop0'
	option target '/rwm'
	option enabled '1'

config mount 'extroot'
	option target '/overlay'
	option device '/dev/sda1'
	option options 'discard'
	option enabled '1'

config mount
	option target '/mnt/data'
	option device '/dev/sda2'
	option options 'discard'
	option enabled '1'

The mounts:

# mount
/dev/root on /rom type squashfs (ro,relatime,errors=continue)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/sda1 on /overlay type ext4 (rw,relatime)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmpfs on /dev type tmpfs (rw,nosuid,noexec,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,noatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,noatime)
bpffs on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,noatime,mode=700)
/dev/sda2 on /mnt/data type ext4 (rw,relatime,discard)

The /mnt/data is correctly mounted with the discard, but the overlay isn't.

With 24.10-rc7 extroot on USB became worked.

But non-extroot wifi radio0(5g) and radio1(2g) swapped comparing to extroot radio0(2g) and radio1(5g) default config and MAC-addresses not assigned.
So, with extroot activated I have similar wifi issue described here https://github.com/openwrt/openwrt/issues/17818