OpenWrt/LEDE Project

  • Status Unconfirmed
  • Percent Complete
    0%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Low
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by R. Diez - 02.02.2019

FS#2096 - A second 'make' always rebuilds something

After a full OpenWrt Git HEAD build finishes, running “make” repeatedly again always rebuilds something. Another user is reporting that a subsequent build takes 5 minutes on his Fedora 27 desktop, which I am also seeing on my oldish Ubuntu 16.04.5 laptop.

Some more information is in the original mailing list discussion. This is the first message in the relevant thread:

http://lists.infradead.org/pipermail/openwrt-devel/2019-January/015648.html

Please find attached a build script to reproduce this problem, and a full build log file from my laptop.

Look for “Running the second make...” in the log file. Afterwards, things like these are being rebuilt, which probably should not:

make[6]: Entering directory ‘/home/rdiez/rdiez/freifunk/openwrt/git-repo/build_dir/target-x86_64_musl/uci-2018-08-11-4c8b4d6e’ [ 5%] Building C object CMakeFiles/uci.dir/libuci.c.o
[ 10%] Building C object CMakeFiles/uci.dir/file.c.o

Project Manager
Petr Štetiar commented on 09.02.2019 23:10

I'm not able to reproduce it on fresh VPS(Scaleway,Vultr) running Fedora 28/29, Ubuntu 14.04/16.04/18.04. I see times under 1m for the 2nd make run. Since I'm not able to reproduce it on 5 different hosts, I think, that it would be really helpful if you could provide Dockerfile which could show this 2nd make problem.

R. Diez commented on 10.02.2019 08:15

I have not learnt Docker yet, and I am an OpenWrt newbie. Before I invest more time in this matter:

1) Did you try the exact script I uploaded to this bug?

2) Does the log file in this bug provide no clues at all about what could be triggering the rebuild?

3) A build time under 1 minute could be considered very long if nothing has changed and the system has loads of RAM and an SSD disk. Normally, it takes almost no time to re-run GNU Make on a set of files that has not changed at all. Do you have an idea why it does not take just 2-3 seconds on your VPS systems?

Project Manager
Petr Štetiar commented on 10.02.2019 13:37

You don't need to use Docker, but as you can see, I'm not able to reproduce the issue with provided build script on fresh Ubuntu 16.04 VPS, which probably means, that this 5 minutes 2nd build issue is only related to your build host.

I'm aware about similar issue on Fedora 29 http://palmtree.beeroclock.net/~karlp/bugs/owrt-rebuilds/ which I'm not able to reproduce on fresh Fedora 29 VPS either.

In other words, if we can't reproduce it, the problem doesn't exist. I just think, that Docker could provide a good reproducible (from scratch) build environment, and is quite easy to setup.

1) Yes, but I've removed the download step (waste of expensive CPU/RAM resources) and I've added timing to the build steps
2) No, I'm sorry but I don't have time to inspect build logs.
3) Feel free to improve it. I think, that once you understand how complex the build system is, you'll probably understand, that 2-3 seconds is a nonsense.

R. Diez commented on 10.02.2019 14:49

I'll take a closer look.

In the meantime, could you provide a build log? This way, I can compare the build log outputs and find out what my machine is rebuilding that your VMs are not.

How long you do you think that rebuilding nothing should take? In the Freifunk community, we are often building firmwares for all routers and all cities. After every little change, such long build times add up. This is killing developer productivity. We are all volunteers, so this is very much wasting our personal time.

Project Manager
Petr Štetiar commented on 11.02.2019 07:45

I don't have the build log, I've deleted the VPS after the build has finished. It's quite easy to spin up new VPS or KVM machine with clean Ubuntu 16.04 system for testing.

For example https://www.hetzner.de/cloud offers VPS with 8 vCPU, 32GB RAM, NVMe SSD disk which can do build from scratch relatively fast, in around 30 minutes (basic ar71xx config). This VPS costs 0.05 EUR per hour, which I find relatively cheap and fast option for testing.

BTW we are all volunteers, so feel free to send patches which will improve the (re)build times for all of us.

You've probably missed it, but here http://palmtree.beeroclock.net/~karlp/bugs/owrt-rebuilds/vm.html you can find 2nd build times for Ubuntu 16.04 running in the VM.

R. Diez commented on 09.03.2019 17:58

I am finding it hard to understand the OpenWrt build system internals. Is there any documentation about it?

In any case, I have rewritten the timestamp.pl script to help me see what is going on. My version is here:

https://github.com/rdiez/Tools/tree/master/Timestamp

You may want to replace the OpenWrt script with mine. The current one is tricky, to say the least.

Lucian CRISTIAN commented on 16.04.2019 06:25

I think that util-linux is always recompiling, I saw doing that almost every time

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing