Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#3798 - flashing the install image leaves overlay filesystem in undefined state #8797

Open
openwrt-bot opened this issue May 11, 2021 · 4 comments
Labels
flyspray release/21.02 pull request/issue targeted (also) for OpenWrt 21.02 release

Comments

@openwrt-bot
Copy link

slingamn:

I'm using the image builder for 21.02.0-rc1, specifically the one targeting bcm2709.

When I flash an install image, the resulting system image references an overlay filesystem (an ext4), the superblock of which is beyond the end of the install image itself. Therefore, flashing an install image may leave files from a previous overlay filesystem in place. This is counterintuitive, because I expected that flashing the install image would leave the OS in a well-defined state.

As a workaround, I've been appending 64KB of zeroes to the end of the install image, which seems to fix the problem.

Reproduction steps:

  1. Build an image, flash it, boot it
  2. Create files in the overlay filesystem
  3. Remove card and reflash; files in the overlay filesystem remain

Thanks for your time!

@openwrt-bot
Copy link
Author

bjonglez:

Can you specify what you mean by "the superblock of which is beyond the end of the install image itself"? How do you see that, and can you reproduce with the ext4 images from https://downloads.openwrt.org/releases/21.02.0-rc1/targets/bcm27xx/bcm2709/ ?

Does the squashfs image work?

@openwrt-bot
Copy link
Author

slingamn:

Thanks! I was able to reproduce this with the squashfs factory image:

using the following steps:

  1. Download and flash the image to a SD card
  2. Boot the device from the SD card, set a root password, ssh in
  3. Create files on the overlayfs, e.g.
root@OpenWrt:~# df -Th Filesystem Type Size Used Available Use% Mounted on /dev/root squashfs 5.3M 5.3M 0 100% /rom tmpfs tmpfs 431.4M 212.0K 431.2M 0% /tmp /dev/loop0 ext4 91.7M 1.5M 83.2M 2% /overlay overlayfs:/overlay overlay 91.7M 1.5M 83.2M 2% / /dev/mmcblk0p1 vfat 63.9M 26.1M 37.7M 41% /boot tmpfs tmpfs 512.0K 0 512.0K 0% /dev root@OpenWrt:~# touch /etc/openwrt_issue_fs_3798 root@OpenWrt:~# touch /usr/bin/openwrt_issue_fs_3798 root@OpenWrt:~# find /overlay/upper/ | grep 3798 /overlay/upper/usr/bin/openwrt_issue_fs_3798 /overlay/upper/etc/openwrt_issue_fs_3798 root@OpenWrt:~# find /overlay/upper/ | grep 3798 | xargs ls -lai 49 -rw-r--r-- 1 root root 0 Apr 18 10:14 /overlay/upper/etc/openwrt_issue_fs_3798 52 -rw-r--r-- 1 root root 0 Apr 18 10:14 /overlay/upper/usr/bin/openwrt_issue_fs_3798
  1. poweroff the device
  2. Reflash the SD card with the factory image, reinsert, boot up again
  3. Root password remains set; files remain in place:
root@OpenWrt:~# find /overlay/upper/ | grep 3798 | xargs ls -lai 49 -rw-r--r-- 1 root root 0 Apr 18 10:14 /overlay/upper/etc/openwrt_issue_fs_3798 52 -rw-r--r-- 1 root root 0 Apr 18 10:14 /overlay/upper/usr/bin/openwrt_issue_fs_3798

What I said about the superblock is at best informed speculation :-) What I'm seeing is that the squashfs image (decompressed) is only 80881906 bytes long, but the total length of the partition table is 360447 sectors, i.e., 184548864 bytes. So there's data referenced in the partitions that is not overwritten by a factory flash. My belief is that the ext4 superblock of /overlay/upper is in this area.

@aparcar aparcar added the release/21.02 pull request/issue targeted (also) for OpenWrt 21.02 release label Feb 22, 2022
@plan44
Copy link

plan44 commented Jun 1, 2023

Hi @aparcar,

Did you find a solution for this in the meantime? It seems to me this bug is still unfixed.

I just stumbled over your post while trying to figure out a bcm27xx sysupgrade issue and getting help with in the openwrt-devel mailing list and in the forum.

It seems to me that both have the same origin: the bcm27xx squashfs based setup seems to simply rely on the data following the squashfs to be "not a filesystem" for mount_root, but without actually making sure that's really the case by padding the image.

  • in your case, just reflashing the same image did not change any data occupied by the overlay, so the overlay did persist.
  • in my case, where the new image was slightly larger, it overwrote a bit of the overlay (f2fs) but just not enough to make mount_root detect "not a filesystem" - instead, it did detect a f2fs but then found it broken, which made the startup after upgrade fail.
  • both cases are probably rare - usually the new image is larger and thus kills an existing overlay simply by overwriting enough of it. Maybe because of that, it has not been really recognized as a bug, altough it can lead to a sysupgrade bricking a box.

Unfortunately, I got very little response in the channels I tried (openwrt-devel, forum, maybe there are better places?) so far.

@plan44
Copy link

plan44 commented Jun 1, 2023

Just found this suggestion which addresses the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flyspray release/21.02 pull request/issue targeted (also) for OpenWrt 21.02 release
Projects
None yet
Development

No branches or pull requests

3 participants