OpenWrt/LEDE Project

  • Status Unconfirmed
  • Percent Complete
  • Task Type Build Failure
  • Category Packages
  • Assigned To No-one
  • Operating System All
  • Severity Critical
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Rob White - 11.05.2018

FS#1543 - Many packages in snapshot failing on one build, ok on next but no code changes.

Occurs on all architectures apparently at random with faillogs.
After next build some that failed will be ok and some that previously built will fail.
eg submit a PR on Travis shows fail on dependencies. After next automated build, Travis errors will be different, or if lucky will pass, only to fail again after the next build.
It seems the build system is chasing its tail somehow.

Jo-Philipp Wich commented on 11.05.2018 13:49

Not sure what to make out of this ticket. Do you have any *specific* examples I can investigate? I need at least *some* pointers to correlate things...

Rob White commented on 11.05.2018 15:10

A long story, so I will try to keep it short.
I noticed this after one of my custom image builds began having a problem with lighttpd after 25th April (LEDE 17.01.4)
The version in snapshot should have the fix but a line in Makefile was missed out.
I noticed lighttpd was missing in snapshot/packages so looked at faillogs and found many errors.
However later the faillog was different and in fact seemed to change every time the build system re-ran.
I looked at other packages and found the same pattern of apparently random things happening.
The other day lighttpd re-appeared in snapshots, but later failed again (but no changes in the code).
This morning it was back again.
I tried resubmitting my previous PR and Travis failed again (with different errors yet again).
Looking at faillogs for many other packages I can see similar things going on.

I might be misunderstanding how this all works, but it seems to me to be a package will fail to build if a dependency has also failed, which seems reasonable, but this seems to be leading to the build system chasing its tail, or somehow there was a "storm of errors".
Here is my last Travis log:

My past experience in IT support made me imagine maintainers tearing their hair out trying to fix something causing this, hence my initial sketchy ticket details. If I am doing something wrong, or there is something wrong with the package, then I can look in depth there, but the same pattern seems to appear in the faillogs for other packages...

Rob White commented on 11.05.2018 15:13
Jo-Philipp Wich commented on 11.05.2018 15:13

The travis build job is totally unrelated to the snapshot mirrors though, it uses the SDK to build the package and all its dependencies locally. It does not utilize the opkg repositories at all.

Rob White commented on 11.05.2018 15:42

I guessed that, but the errors seemed to be the same or similar, although both change without the code changing.
Looking at it at this moment, it does seem to have stabilised, whereas couple of days ago, some architectures had no packages at all.
I still can't get Travis to pass my PR though. Feeling very frustrated by it all!

Jo-Philipp Wich commented on 11.05.2018 15:46

I can understand that... will see if I can take a deeper look at it later. Note that the travis integration is not directly administered by OpenWrt/LEDE, it is local to the Github package repository. The toplevel travis scripts in the repo should probably get adjusted to actually show the compile logs (or to build the package with V=s in the first place).

Rob White commented on 11.05.2018 16:40

Much appreciated, along with all your other efforts!

Rob White commented on 11.05.2018 16:51

The 25th of April seems to be a common date.
The version of lighttpd in lede 17.01.4 jumped from 1.4.45-3 (working with SSL) to 1.4.28-1 (broken for SSL) on that date.

I thought package version jumps did not push into the stable release but stayed only available in snapshots, or have I got that wrong?


Available keyboard shortcuts


Task Details

Task Editing