adrianschmutzler:

sysupgrade on these devices is broken, so the images have been removed in 0cc87b3

wwortel:

Thanks. By now the kernel that gets compiled is 5.4.63
Is this kernel problem still present?
The situation poses a big difficulty to me as several Routerstation systems are not easily physically accessible. It turned out that only on an old compilation of some years ago the now experimentally produced sysupgrade.bin worked. Once upgraded to today's OpenWrt, the sysupgrade process commences but stops and the system reboots in the old setting.
Is there a manual way to produce a working sysupgrade binary for Routerstation boards?
Today's factory.bin works without tweaks. But you cannot apply it remotely.

adrianschmutzler:

You can try to contact Tomasz (the patch author) about that.

Personally, I only know what I read in the commit message. From that perspective it looks to me like something nobody is interested in and which will probably stay that way until somebody feels enough pain to fix it.

wwortel:

experimented under kernel 5 with the previous sysupgrade code and indeed ended up bricking the Routerstation.
Revived it using Jtag; had copies of the mtd 0-5. Restored mtd0 (uboot) and mtd4 (fis) and mtd5 (uboot data), the latter two I combined into one image.
Interestingly also the Openocd jtag programme provides for erasing flash per sector, and then writing it. But mtd4 and mtd5 together occupy the last sector of the 256 sectors the Routerstation's flash has. So when changing something in one of mtd4 or 5, both need to be rewritten as the flash erase that Openocd offers is not limited to just a part of the sector size.

wwortel:

Further experimentation and contact with the patch' author has confirmed that a fix has already been proposed but has not been introduced yet. Experimintally have verified that it works fine. The sysupgrade works on the Routerstation under kernel 5 when pending kernel patch #411
(411-mtd-partial_eraseblock_write.patch) in target/linux/generic/pending-5.4/ is patched using https://patchwork.ozlabs.org/project/openwrt/patch/20200805211354.3922-1-git@johnthomson.fastmail.com.au/
Revert the commits
https://git.openwrt.org/cc5256a8bfa0
https://git.openwrt.org/0cc87b3bacee
and compile. The produced image has the ability to do sysupgrade on systems with partitions that are smaller than the erase block size of its flash, such as the Ubiquiti Routerstation and RS-Pro. When applying sysupgrade to provide Routerstations with new firmware do verify that the existing firmware does not have a kernel 4 or 5. Sysupgrade then will brick the board and require Jtag to revive it. If in doubt, use tftp and a factory image to upgrade the first time. From then on sysupgrade can be used again using compilations with the fix included.

acoul:

Greetings,

is it safe to use [[https://patchwork.ozlabs.org/project/openwrt/patch/20200805211354.3922-1-git@johnthomson.fastmail.com.au/|this patch]] on current trunk ?

is this going to properly work on both RouterStations & RouterStations Pro ?

remote sysupgrade feature is quite valuable, but I am afraid to test this as I don't have access to Jtag tools for unbricking the devices

acoul:

it looks like this BUG has been resolved in commit [[https://git.openwrt.org/?p=openwrt/openwrt.git;a=commitdiff;h=760952ad022561931846b02ddfe69c21afead0fe|760952ad022561931846b02ddfe69c21afead0fe]] and this ticket can now close

wwortel:

Careful!
Compiling a fresh OpenWrt for RS(pro) with inside a re-enabled and safe sysupgrade and also a sysupgrade image binary being produced as compile result is one thing. This is what the ticket was about.

Another thing is to know whether the old firmware currently on your RS(pro), who's sysupgrade function you are going to use to install the fresh OpenWrt, has the mtd kernel bug or not.
If it does you'll brick the board.

acoul:

Hello william,

thank you for the warning.

in my understanding, this is a kernel BUG that appeared in kernel >=4.19 & is still in kernel 5.4.x thereof, any device with an openwrt snapshot older than let's say October 2020 ([[https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=bf2870c1d9e1|to be safe]]) is vulnerable using sysupgrade.

upgrading sysupgrade itself will not do, as this is a kernel BUG.

openwrt-19.x (with kernel 4.14.x) or openwrt-shapshot >=October 2020, should be safe to use sysupgrade

am I correct on this ?

wwortel:

Cannot confirm or deny the exact start moment of the problem. Did not look into it as I went the trial and error way, before starting to understand what the underlying issues were.
Ever since OpenWrts with a sysupgrade that is aware of the problem but still with the problem in the kernel, it will reject the upgrade. You can safely test using the -T option; it will not write anything.

Much older firmwares will be OK. So the risk is in those of in-between when the problem existed but had not been discovered yet. To go a safe way I recommend the one-time factory upgrade.
The flash writing is then done by the bootloader. In case the RS(pro) is difficult to physically access but the power supply can be accessed there is a trick. A pin hole opening in the power supply and pressing a miniature switch behind it with a pin and connecting power will bring the RS in tftp upload mode.

acoul:

Hello William,

//In case the RS(pro) is difficult to physically access but the power supply can be accessed there is a trick. A pin hole opening in the power supply and pressing a miniature switch behind it with a pin and connecting power will bring the RS in tftp upload mode//

so this is the way to enable tftp in case you do not have serial access to the devices right ? I am attaching pictures of the RouterStation & RouterStation Pro with those mini switches circled in red.

but if I understand correctly, if you brick your box, these switches are useless. You will need a jtag to restore a backup bootloader on the device, correct ?

thank you for the feedback

wwortel:

"so this is the way to enable tftp in case you do not have serial access to the devices right ?" No. Please read again; I write physically access, i.e. touch, and refer to the power supply, the rectangular black 24V supply. These power supplies from some date onwards have on one side this pin hole. The side opposite the LED, near the RJ45 connectors. So e.g. the device sits in a box in a mast, and the supply is accessible at the ground.
Powering up pressing that switch is detected by the bootloader and it starts waiting for a tftp connection to receive a factory image.

"but if I understand correctly, if you brick your box, these switches are useless. You will need a jtag to restore a backup bootloader on the device, correct ?"
Yes. If the bootloader is damaged, or the data it needs, nothing will start and nothing will be seen. Jtag then still can access the memory and the CPU because of special jtag circuitry in the CPU. The CPU is brought into sleep mode via JTAG and the flash can be unprotected, erased, written, and reprotected per block or range of blocks.

I think in this case it is the bootloader configuration that mtd damages, but am not sure.
0x000000ff0000-0x000000fff000 : "FIS directory"
0x000000fff000-0x000001000000 : "RedBoot config"
The erase block size is 0x10000. So this fff000 is halfway. Sysupgrade needs to write into FIS to put the new size of the kernel and rootfs there but if mtd, doing the writing, does not stop at fff000 or writes somewhere else ... Did not investigate where exactly the damage is done. Just flashed again the bootloader mtd0 and its configuration mtd5, using jtag. And then could use tftp again.

acoul:

//No. Please read again; I write physically access, i.e. touch, and refer to the power supply, the rectangular black 24V supply. These power supplies from some date onwards have on one side this pin hole. The side opposite the LED, near the RJ45 connectors. So e.g. the device sits in a box in a mast, and the supply is accessible at the ground.
Powering up pressing that switch is detected by the bootloader and it starts waiting for a tftp connection to receive a factory image. //

amazing. I was totally unaware of this remote reset capability through PoE on ubiquiti devices

I guess ubiquiti has a similar to the following circuitry [[https://forum.mikrotik.com/viewtopic.php?t=124781|discussed on this thread]] build-in their devices:

//Add a photocoupler, like 4N35, and a resistor. Connected near the reset button to command it remotely with two wire by a voltage, for example the power voltage.//

great input William, thank you

wwortel:

well, not sure that all ubiquiti devices support this, but the RS(pro) do.
And not all power supplies. The early ones did not have this. The later ones do, with the text RESET.
Indeed this is not documented on the OpenWrt wiki pages about RS and RSpro.
Of course these boards now date from some 10 years back and do not receive much attention anymore.
At their time they were advanced and still today do OpenWrt nicely.

Technically it works not as you think. Clever use is made of the fact that the two data pairs are terminated on the RS, as generally the case with ethernet, in a transformer.
So called balanced transmission lines. To signal whether RESET is pressed at the power supply the DC level is controlled. A voltage is on it when the switch is pressed.
At the RS this can be simply sensed at the center tap of the transformer input winding without affecting the data transmission in any way.