OpenWrt/LEDE Project

  • Status Closed
  • Percent Complete
    100%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Critical
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Hannu Nyman - 01.03.2017
Last edited by Felix Fietkau - 06.03.2017

FS#579 - Rename failure when file both in rom and overlay in mvebu / wrt3200acm, due to kernel 4.9 or ubi?

LEDE master r3623 mvebu WRT3200ACM

Router fails to rename files that are both in /rom and in /overlay, i.e. they were part of the original flash and have been modified later or were included in sysupgrade.

Rename fails.
Deleting the same file succeeds.
Copying is also ok.

(Deleting the file and copying a new one into its place does not enable rename, which still fails.)

I noticed this in mvebu / WRT3200ACM, but the bug may also concern all mvebu devices, all kernel 4.9 users and/or all ubi filesystem users.

I did originally build my settings with 17.01 with 4.4 and/or master with k4.4 and noticed no problems at that stage, so this may have surfaced at the bump to 4.9, or at the last bump to 4.9.13. (But I have only used mvebu for a few days, and haven’t yet verified the 4.9.10 or lede-17.01 behaviour)

I originally noticed this when debugging the new opkg, but then realised that this is a wider file-system level problem. I verified the problem by flashing a pure buildbot snapshot image without any additional settings.

Rename fails

  • in console
    root@LEDE:~# mv /etc/dropbear/dropbear_rsa_host_key /etc/dropbear/dropbear_rsa_host_key-x
    mv: can't rename '/etc/dropbear/dropbear_rsa_host_key': Invalid argument
    
  • in script:
    root@LEDE:/etc/config# /etc/init.d/luci_statistics restart
    mv: can't rename '/etc/collectd.conf': Invalid argument
    
  • “rename” function call in a C program (strace from opkg):
    rename("/etc/config/bcp38", "/etc/config/bcp38-opkg") = -1 EINVAL (Invalid argument)
    

No such problem in ar71xx device with the same r3623 build but with kernel 4.4 (and without NAND & ubi).

Steps to reproduce:

  1. Try to rename a file that is both in firmware and in overlay. E.g. a config file related to a package included in the build

I used /etc/dropbear/dropbear_rsa_host_key in the next example, as that is a file included in the clean snapshot firmware.

Reboot (SNAPSHOT, r3623-339de82)

root@LEDE:~# mv /etc/dropbear/dropbear_rsa_host_key /etc/dropbear/dropbear_rsa_host_key-x
mv: can't rename '/etc/dropbear/dropbear_rsa_host_key': Invalid argument

root@LEDE:~# ls -l /rom/etc/dropbear/dropbear_rsa_host_key
-rw-r--r--    1 root     root             0 Mar  1 06:10 /rom/etc/dropbear/dropbear_rsa_host_key

root@LEDE:~# ls -l /overlay/upper/etc/dropbear/dropbear_rsa_host_key
-rw-------    1 root     root           805 Mar  1 06:10 /overlay/upper/etc/dropbear/dropbear_rsa_host_key

root@LEDE:~# rm  /etc/dropbear/dropbear_rsa_host_key
root@LEDE:~# ls -l /overlay/upper/etc/dropbear/dropbear_rsa_host_key
c---------    1 root     root        0,   0 Mar  1 14:34 /overlay/upper/etc/dropbear/dropbear_rsa_host_key

root@LEDE:~# cp /etc/dropbear/dropbear_rsa_host_key-x /etc/dropbear/dropbear_rsa_host_key
root@LEDE:~# mv /etc/dropbear/dropbear_rsa_host_key /etc/dropbear/dropbear_rsa_host_key-x
mv: can't rename '/etc/dropbear/dropbear_rsa_host_key': Invalid argument

Rename also fails for newly created files in /overlay. E.g. a file does not yet exist in /overlay, but is created there due to editing:

root@LEDE:/etc# ls /overlay/upper/etc/
board.json                dropbear                  firewall.user             switch_boot_partition.sh  urandom.seed
config                    ethers                    fw_env.config             uci-defaults
root@LEDE:/etc# nano /etc/opkg.conf
root@LEDE:/etc# mv /etc/opkg.conf /etc/opkg.conf2
mv: can't rename '/etc/opkg.conf': Invalid argument
Closed by  Felix Fietkau
06.03.2017 09:13
Reason for closing:  Fixed
Hannu Nyman commented on 01.03.2017 15:32

Renaming works ok in 17.01.0

I flashed the official 17.01.0 build without settings, and tested renaming. Works ok.

So, this is something new. Likely related to the 4.9 bump.

Reboot (17.01.0, r3205-59508e3)

root@LEDE:/etc# ls -l /overlay/upper/etc/dropbear/
-rw-------    1 root     root           804 Feb 20 17:13 dropbear_rsa_host_key
root@LEDE:/etc# ls -l /rom/etc/dropbear/dropbear_rsa_host_key
-rw-r--r--    1 root     root             0 Feb 20 17:13 /rom/etc/dropbear/dropbear_rsa_host_key

root@LEDE:/etc# mv /etc/dropbear/dropbear_rsa_host_key /etc/dropbear/dropbear_rsa_host_key-x
root@LEDE:/etc#
Hannu Nyman commented on 01.03.2017 18:23

r3509-2374549916 kernel 4.4 : rename works ok.
r3510-2bf9ea6a31 kernel 4.9 : fails to rename

I compiled the last commit where mvebu is with kernel 4.4 and rename works normally. The next commit bumping mvebu to 4.9 makes rename to fail. So this bug came with kernel 4.9.

I have no idea if this is restricted to mvebu or if this concerns all 4.9 targets (using ubifs?).

Supposedly kernel 4.9 supports ubifs rename "built-in", so the patches for that (052-...) were dropped from LEDE's generic patches-4.9

Linux kernel's ubifs rename support commits have been committed on 2016-10-02:
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/log/fs/ubifs?h=linux-4.9.y

Hannu Nyman commented on 02.03.2017 09:25

I think that I have identified the culprit commit causing the rename bug for ubifs with overlayfs, but I am not sure about the fix. Deleting the offending two lines?

I compared the respective files in 4.4 and 4.9. I took the files that the kernel 4.4 patches 052-... handled, meaning fs/ubifs/ubifs.h, journal.c and dir.c.

  • 4.4 patched journal.c equals 4.9 journal.c
  • 4.4 patched ubifs.h has only trivial differences to 4.9
  • dir.c has otherwise only trivial changes (e.g. mutex lock changes), but there is one significant change in 4.9, where "return -EINVAL" has been inserted.

https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/ubifs/dir.c?h=linux-4.9.y#n1091

	struct timespec time;
	unsigned int uninitialized_var(saved_nlink);

	if (flags & ~RENAME_NOREPLACE)
		return -EINVAL;

	/*

The change is here:
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/fs/ubifs/dir.c?h=linux-4.9.y&id=f03b8ad8d38634d13e802165cc15917481b47835

Full commit:
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-4.9.y&id=f03b8ad8d38634d13e802165cc15917481b47835

fs: support RENAME_NOREPLACE for local filesystems

This is trivial to do:

 - add flags argument to foo_rename()
 - check if flags doesn't have any other than RENAME_NOREPLACE
 - assign foo_rename() to .rename2 instead of .rename

Filesystems converted:

affs, bfs, exofs, ext2, hfs, hfsplus, jffs2, jfs, logfs, minix, msdos,
nilfs2, omfs, reiserfs, sysvfs, ubifs, udf, ufs, vfat.

(overlayfs is not mentioned in the list.)

Hannu Nyman commented on 02.03.2017 10:23

Fix proposal here. Any comments from ubifs / overlayfs specialists?

I patched my system with this small change that reverts things more or less to the state they were for ubifs with 4.4:

target/linux/generic/patches-4.9/552-ubifs-overlayfs-fix-rename.patch

--- a/fs/ubifs/dir.c
+++ b/fs/ubifs/dir.c
@@ -1088,8 +1088,10 @@
 	struct timespec time;
 	unsigned int uninitialized_var(saved_nlink);
 
+	/*
	if (flags & ~RENAME_NOREPLACE)
		return -EINVAL;
+	*/
 
 	/*
 	 * Budget request settings: deletion direntry, new direntry, removing

At the first glance it looks ok:
Renaming a problematic file works again, and both the deletion record and the newly named file are visible in /overlay. And /etc looks normal:

root@LEDE:~# mv /etc/dropbear/dropbear_rsa_host_key  /etc/dropbear/dropbear_rsa_host_key-x
root@LEDE:~# ls -l /rom/etc/dropbear/
-rw-r--r--    1 root     root             0 Mar  2 10:09 dropbear_rsa_host_key
root@LEDE:~# ls -l /overlay/upper/etc/dropbear/
-rwxr--r--    1 root     root           959 Feb 18 18:09 authorized_keys
c---------    1 root     root        0,   0 Mar  2 10:16 dropbear_rsa_host_key
-rw-------    1 root     root           804 Feb 20 17:13 dropbear_rsa_host_key-x
root@LEDE:~# ls -l /etc/dropbear/
-rwxr--r--    1 root     root           959 Feb 18 18:09 authorized_keys
-rw-------    1 root     root           804 Feb 20 17:13 dropbear_rsa_host_key-x
Project Manager
Felix Fietkau commented on 06.03.2017 09:13

Fixed in r3651-697ff33771

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing