hnyman:

I tested sysupgrading wrt3200acm from r4283 to r4291 and failed with similar symptoms as diizzyy. After a manual power-off the router rebooted to the old firmware, so no actual flashing had been done.

But I sysupgraded ok from r4289 to r4291 with ipq806x R7800 that is also a NAND device, so sysupgrade still works for some targets.

So, the bug may be related to just mvebu, not the general NAND sysupgrade.

I noticed that the large sysupgrade patch set two days ago made target-specific changes to e.g. x86, sunxi and ramips, but did not touch mvebu. Maybe some target-specific change needs to be done also for mvebu.

dedeckeh:

lede-project/source@21f25bc is just about passing the firewall zone of the PPP master interface to the dynamic created DHCPv6 interface. This logic will only be triggered if you use a wan PPP interface; don't think this is the case in the described issue.

hnyman:

Yep, it is not related to that PPP commit. Like diizzyy says, the issue has been noticed already with r4263-c87aa0d and that PPP commit is much later.

The problem is likely related to a large bunch of sysupgrade related commits just before r4263 (so far the earliest build found to fail).

commit log leading to r4263:
https://github.com/lede-project/source/commits/c87aa0d7ca8f2dcd3cf559a4933fef3cb44f7a21

diizzyy:

@ Hans:
I provided both commits as data points as my r4263 build wasn't a vanilla/stock build.

hnyman:

I have narrowed down the regression range:

r4249-393817df5d sysupgrade from this ok
r4252-218af1957e sysupgrade from this fails

So, one of these commits makes mvebu wrt3200acm to fail in sysupgrade:

218af19 fstools: snapshot: handle jffs2 conversion using upgraded
6bbb3a2 fstools: clean up trailing whitespace in snapshot script
30f61a3 base-files: always use staged sysupgrade

Of these, 30f61a3 looks most likely at the first glance.

I reverted the changes done by 30f61a3 manually.
That enabled again the sysupgrade from my r4252-218af1957e build.

So, it looks like the reason for the broken mvebu wrt3200acm sysupgrade is
30f61a3 base-files: always use staged sysupgrade

NeoRaider:

Note that my sysupgrade patchset is not bisectable, either all or none of the patches should be applied for sysupgrade to work. I had intended to keep bisectability, but I didn't notice that the procd master was broken before I pushed my patches, so there was no nice way to achieve that...

NeoRaider:

In partiticular, reverting 30f61a3 will not lead to a working sysupgrade on NAND systems unless the procd upgrades are reverted as well.

A serial console output with the following patch applied would be very interesting:

if [ -n "$FAILSAFE" ]; then
printf '%s\x00%s\x00%s' "$RAM_ROOT" "$IMAGE" "$COMMAND" >/tmp/sysupgrade


hnyman:

Note that my sysupgrade patchset is not bisectable,

Too bad. So, my bisecting exercise was futile, right?

Well, at least sysupgrading works until the middle of your patchset.

NeoRaider:

I have pushed a fix for an issue I found while reading the code to my staging tree at https://git.lede-project.org/?p=lede/neoraider/staging.git;a=summary . As I don't have a mvebu device, it's completely untested, but it might solve this problem...

hnyman:

Thanks, I will test it.

One quick comment:
I noticed that you also removed the part that saves settings in mvebu sysupgrade:

I haven't checked, but is there now some other place where the settings are copied into /tmp/syscfg ?

/tmp/syscfg is a flash partition mounted there and to my knowledge it is used for saving settings in sysupgrade. mvebu restores settings after syupgrade from there with /lib/preinit/81_linksys_syscfg:

https://git.lede-project.org/?p=source.git;a=blob;f=target/linux/mvebu/base-files/lib/preinit/81_linksys_syscfg;h=3edd82ebcdd124d7db9fd2867a36accabbb70a8a;hb=HEAD

NeoRaider:

Thanks for noticing, will fix.

NeoRaider:

New version pushed.

I wonder though why /tmp/syscfg is mounted all the time, and not just shortly during boot and sysupgrade like it is done with the /boot partition on many targets.

jow-:

Yes and even if it is supposed to be persistently mounted, then /tmp/syscfg as mountpoint is an odd choice... Maybe its worth moving it to /var/lib/syscfg or something.

NeoRaider:

I don't think we have a /var symlink in the sysupgrade ramfs at the moment, so maybe /tmp/lib/syscfg. But I'll leave that change to someone who can actually test on mvebu :)

hnyman:

I wonder though why /tmp/syscfg is mounted all the time

Well, it is a large partition that offers some 30-70 MB storage space (depending on Linksys router model) that is accessible from both kernel1 and kernel2 partitions, so it can be used for various storage purposes for stuff that should be accessible from both alternative firmwares.

(Originally it contains various stuff logs etc. for Linksys OEM firmware)

hnyman:

New version pushed.

Looks good.

I compiled r4293-7f9143168f plus your patch from https://git.lede-project.org/?p=lede/neoraider/staging.git;a=commitdiff;h=570e4c347bb60d17cca36df91fdd1eecadbd6de2;hp=7f9143168faef445effceead08d77ecc7c3e9dc6

I first flashed it from r4252 (with 30f61a3 reverted...) and then
tested the patch by re-flashing r4293 using r4293.

Flash went ok and also the settings got transferred ok.

So, your patch looks ok to me.

NeoRaider:

Great, will push to master.