OpenWrt/LEDE Project

  • Status Unconfirmed
  • Percent Complete
    0%
  • Task Type Bug Report
  • Category Base system
  • Assigned To No-one
  • Operating System All
  • Severity Low
  • Priority Very Low
  • Reported Version openwrt-19.07
  • Due in Version Undecided
  • Due Date Undecided
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Nick - 08.04.2021

FS#3730 - resolve_conffiles reports spurious errors

When I run upgrades

<code>
root@router:~# opkg update && opkg list-upgradable | awk '{print $1}' | xargs opkg upgrade

I usually end up with some messages like

Collected errors:
 * resolve_conffiles: Existing conffile /etc/config/luci_statistics is different from the conffile in the new package. The new conffile will be placed at /etc/config/luci_statistics-opkg.
 * resolve_conffiles: Existing conffile /etc/config/luci is different from the conffile in the new package. The new conffile will be placed at /etc/config/luci-opkg.
 * resolve_conffiles: Existing conffile /etc/config/ucitrack is different from the conffile in the new package. The new conffile will be placed at /etc/config/ucitrack-opkg.

Sometimes the conflicts are genuine – because I’ve configured the router – but most of them are because uci doesn’t force a canonical output format. For example

root@router:~# diff -u /etc/config/luci-opkg /etc/config/luci
--- /etc/config/luci-opkg	2021-03-27 02:44:43.000000000 -0400
+++ /etc/config/luci	2021-01-30 05:15:01.000000000 -0500
@@ -1,31 +1,39 @@
-config core main
-	option lang auto
-	option mediaurlbase /luci-static/bootstrap
-	option resourcebase /luci-static/resources
-	option ubuspath /ubus/
-	
-config extern flash_keep
-	option uci 		"/etc/config/"
-	option dropbear "/etc/dropbear/"
-	option openvpn	"/etc/openvpn/"
-	option passwd	"/etc/passwd"
-	option opkg		"/etc/opkg.conf"
-	option firewall	"/etc/firewall.user"
-	option uploads	"/lib/uci/upload/"
-	
-config internal languages
-	
-config internal sauth
-	option sessionpath "/tmp/luci-sessions"
-	option sessiontime 3600
-	
-config internal ccache
-	option enable 1
-		
-config internal themes
-
-config internal apply
-	option rollback 90
-	option holdoff 4
-	option timeout 5
-	option display 1.5
+
+config core 'main'
+	option lang 'auto'
+	option mediaurlbase '/luci-static/bootstrap'
+	option resourcebase '/luci-static/resources'
+	option ubuspath '/ubus/'
+
+config extern 'flash_keep'
+	option uci '/etc/config/'
+	option dropbear '/etc/dropbear/'
+	option openvpn '/etc/openvpn/'
+	option passwd '/etc/passwd'
+	option opkg '/etc/opkg.conf'
+	option firewall '/etc/firewall.user'
+	option uploads '/lib/uci/upload/'
+
+config internal 'languages'
+
+config internal 'sauth'
+	option sessionpath '/tmp/luci-sessions'
+	option sessiontime '3600'
+
+config internal 'ccache'
+	option enable '1'
+
+config internal 'themes'
+	option Bootstrap '/luci-static/bootstrap'
+
+config internal 'apply'
+	option rollback '90'
+	option holdoff '4'
+	option timeout '5'
+	option display '1.5'
+
+config internal 'diag'
+	option dns 'openwrt.org'
+	option ping 'openwrt.org'
+	option route 'openwrt.org'
+

Except for `luci.diag` and `luci.themes.Bootstrap`, these are identical config files, and I don’t think even configured luci.diag so that’s probably just from a previous change.

It can be hard to separate out what I’ve configured from what the packagers have changed, but it is much more tedious when I also need to separate out what’s spurious changes from the quotes and indentation changing.

It would save users a lot of time if either:

* `resolve_conffiles` called `uci` to check differences instead of comparing entire files, *or*
* `uci` had a canonical format that both it and `luci` agreed to stick to, strictly so that running `diff` (or whatever `resolve_conffiles` does) would only report genuine conflicts

Don’t get me wrong, I do like openwrt a lot. It’s way better than using an unfixable vendor firmware. The fact that I can upgrade packages on it at all is marvelous. Thank you for your work.


Model: TP-Link Archer C50 v4 https://openwrt.org/toh/tp-link/archer-c50

Build: OpenWrt 19.07.7 r11306-c4a6851c72 / LuCI openwrt-19.07 branch git-21.096.73306-254083c

Kernel: 4.14.221

Packages:

root@router:~# opkg list-installed
base-files - 204.2-r11306-c4a6851c72
busybox - 1.30.1-6
cgi-io - 19
collectd - 5.12.0-1
collectd-mod-conntrack - 5.12.0-1
collectd-mod-cpu - 5.12.0-1
collectd-mod-interface - 5.12.0-1
collectd-mod-iwinfo - 5.12.0-1
collectd-mod-load - 5.12.0-1
collectd-mod-memory - 5.12.0-1
collectd-mod-network - 5.12.0-1
collectd-mod-ping - 5.12.0-1
collectd-mod-rrdtool - 5.12.0-1
collectd-mod-uptime - 5.12.0-1
collectd-mod-wireless - 5.12.0-1
conntrack - 2018-05-01-88610abe-1
diffutils - 3.7-2
dnsmasq - 2.80-16.3
dropbear - 2019.78-2
firewall - 2019-11-22-8174814a-3
fstools - 2020-05-12-84269037-1
fwtool - 2
getrandom - 2019-06-16-4df34a4d-3
hostapd-common - 2019-08-08-ca8c2bd2-7
ip6tables - 1.8.3-1
iptables - 1.8.3-1
iptables-mod-conntrack-extra - 1.8.3-1
iptables-mod-ipopt - 1.8.3-1
iw - 5.0.1-1
iwinfo - 2019-10-16-07315b6f-1
jshn - 2020-05-25-66195aee-1
jsonfilter - 2018-02-04-c7e938d6-1
kernel - 4.14.221-1-d92769dc5268e102503ae83fe968a56c
kmod-cfg80211 - 4.14.221+4.19.161-1-1
kmod-gpio-button-hotplug - 4.14.221-3
kmod-ifb - 4.14.221-1
kmod-ip6tables - 4.14.221-1
kmod-ipt-conntrack - 4.14.221-1
kmod-ipt-conntrack-extra - 4.14.221-1
kmod-ipt-core - 4.14.221-1
kmod-ipt-ipopt - 4.14.221-1
kmod-ipt-nat - 4.14.221-1
kmod-ipt-offload - 4.14.221-1
kmod-ipt-raw - 4.14.221-1
kmod-leds-gpio - 4.14.221-1
kmod-lib-crc-ccitt - 4.14.221-1
kmod-mac80211 - 4.14.221+4.19.161-1-1
kmod-mt76-core - 4.14.221+2021-02-15-5c768dec-1
kmod-mt7603 - 4.14.221+2021-02-15-5c768dec-1
kmod-mt76x02-common - 4.14.221+2021-02-15-5c768dec-1
kmod-mt76x2 - 4.14.221+2021-02-15-5c768dec-1
kmod-mt76x2-common - 4.14.221+2021-02-15-5c768dec-1
kmod-nf-conntrack - 4.14.221-1
kmod-nf-conntrack-netlink - 4.14.221-1
kmod-nf-conntrack6 - 4.14.221-1
kmod-nf-flow - 4.14.221-1
kmod-nf-ipt - 4.14.221-1
kmod-nf-ipt6 - 4.14.221-1
kmod-nf-nat - 4.14.221-1
kmod-nf-reject - 4.14.221-1
kmod-nf-reject6 - 4.14.221-1
kmod-nfnetlink - 4.14.221-1
kmod-ppp - 4.14.221-1
kmod-pppoe - 4.14.221-1
kmod-pppox - 4.14.221-1
kmod-sched-cake - 4.14.221+2019-03-12-057c7388-1
kmod-sched-core - 4.14.221-1
kmod-slhc - 4.14.221-1
libblobmsg-json - 2020-05-25-66195aee-1
libc - 1.1.24-2
libcap - 2.27-1
libelf1 - 0.177-1
libgcc1 - 7.5.0-2
libip4tc2 - 1.8.3-1
libip6tc2 - 1.8.3-1
libiwinfo-lua - 2019-10-16-07315b6f-1
libiwinfo20181126 - 2019-10-16-07315b6f-1
libjson-c2 - 0.12.1-3.1
libjson-script - 2020-05-25-66195aee-1
libltdl7 - 2.4.6-2
liblua5.1.5 - 5.1.5-3
liblucihttp-lua - 2019-07-05-a34a17d5-1
liblucihttp0 - 2019-07-05-a34a17d5-1
libmbedtls12 - 2.16.10-1
libmnl0 - 1.0.4-2
libnetfilter-conntrack3 - 2018-05-01-3ccae9f5-2
libnetfilter-cthelper0 - 1.0.0-2
libnetfilter-cttimeout1 - 1.0.0-2
libnetfilter-queue1 - 2017-06-27-601abd1c-2
libnfnetlink0 - 1.0.1-3
libnl-tiny - 0.1-5
liboping - 1.10.0-2
libpthread - 1.1.24-2
librrd1 - 1.0.50-3
libubox20191228 - 2020-05-25-66195aee-1
libubus-lua - 2019-12-27-041c9d1c-1
libubus20191227 - 2019-12-27-041c9d1c-1
libuci20130104 - 2019-09-01-415f9e48-4
libuclient20160123 - 2020-06-17-51e16ebf-1
libustream-mbedtls20150806 - 2020-03-13-40b563b1-1
libxtables12 - 1.8.3-1
logd - 2019-06-16-4df34a4d-3
lua - 5.1.5-3
luci - git-21.096.73306-254083c-1
luci-app-firewall - git-21.096.73306-254083c-1
luci-app-openvpn - git-21.096.73306-254083c-1
luci-app-opkg - git-21.096.73306-254083c-1
luci-app-simple-adblock - git-21.050.37945-c33df8f-1
luci-app-sqm - 1.4.0-2
luci-app-statistics - git-21.096.73306-254083c-1
luci-base - git-21.096.73306-254083c-1
luci-compat - git-21.096.73306-254083c-1
luci-lib-ip - git-21.096.73306-254083c-1
luci-lib-iptparser - git-21.096.73306-254083c-1
luci-lib-jsonc - git-21.096.73306-254083c-1
luci-lib-nixio - git-21.096.73306-254083c-1
luci-mod-admin-full - git-21.096.73306-254083c-1
luci-mod-network - git-21.096.73306-254083c-1
luci-mod-status - git-21.096.73306-254083c-1
luci-mod-system - git-21.096.73306-254083c-1
luci-proto-ipv6 - git-21.096.73306-254083c-1
luci-proto-ppp - git-21.096.73306-254083c-1
luci-theme-bootstrap - git-21.096.73306-254083c-1
mtd - 24
netifd - 2021-01-09-753c351b-1
odhcp6c - 2021-01-09-64e1b4e7-16
odhcpd-ipv6only - 2020-05-03-49e4949c-3
openwrt-keyring - 2019-07-25-8080ef34-1
opkg - 2021-01-31-c5dccea9-1
ppp - 2.4.7.git-2019-05-25-3
ppp-mod-pppoe - 2.4.7.git-2019-05-25-3
procd - 2020-03-07-09b9bd82-1
rpcd - 2020-05-26-67c8a3fd-1
rpcd-mod-file - 2020-05-26-67c8a3fd-1
rpcd-mod-iwinfo - 2020-05-26-67c8a3fd-1
rpcd-mod-luci - 20201107
rpcd-mod-rrdns - 20170710
rrdtool1 - 1.0.50-3
simple-adblock - 1.8.4-10
sqm-scripts - 1.4.0-2
swconfig - 12
tc - 5.0.0-2.1
ubox - 2019-06-16-4df34a4d-3
ubus - 2019-12-27-041c9d1c-1
ubusd - 2019-12-27-041c9d1c-1
uci - 2019-09-01-415f9e48-4
uclient-fetch - 2020-06-17-51e16ebf-1
uhttpd - 2020-10-01-3abcc891-1
urandom-seed - 1.0-1
urngd - 2020-01-21-c7f7b6b6-1
usign - 2020-05-23-f1f65026-1
wireless-regdb - 2020.11.20-1
wpad-basic - 2019-08-08-ca8c2bd2-7
zlib - 1.2.11-3
Nick commented on 08.04.2021 15:10

Here's a prototype of my first suggestion; given the reported conflicts above, I translated them into `uci`'s cli syntax, which implicitly imposes a canonical format:

root@router:~# TMP=$(mktemp -d)
root@router:~# uci show luci > $TMP/luci
root@router:~# uci show luci-opkg | sed 's/^luci-opkg/luci/' > $TMP/luci-opkg
root@router:~# diff -u $TMP/luci-opkg $TMP/luci
--- /tmp/tmp.LNmpnm/luci-opkg	2021-04-08 11:07:26.422562972 -0400
+++ /tmp/tmp.LNmpnm/luci	2021-04-08 11:07:14.892396639 -0400
@@ -18,8 +18,13 @@
 luci.ccache=internal
 luci.ccache.enable='1'
 luci.themes=internal
+luci.themes.Bootstrap='/luci-static/bootstrap'
 luci.apply=internal
 luci.apply.rollback='90'
 luci.apply.holdoff='4'
 luci.apply.timeout='5'
 luci.apply.display='1.5'
+luci.diag=internal
+luci.diag.dns='openwrt.org'
+luci.diag.ping='openwrt.org'
+luci.diag.route='openwrt.org'

That diff is much easier to read and merge manually.

Here's a prototype of my second idea, by getting `uci` to regenerate the file:

root@router:~# cp /etc/config/luci-opkg /etc/config/luci-opkg-canonical
root@router:~# uci show luci-opkg-canonical | xargs -n 1 uci set && uci commit
root@router:~# diff -u /etc/config/luci-opkg-canonical /etc/config/luci-mine
--- /etc/config/luci-opkg-canonical	2021-04-08 11:01:48.000000000 -0400
+++ /etc/config/luci-mine	2021-01-30 05:15:01.000000000 -0500
@@ -24,6 +24,7 @@
 	option enable '1'
 
 config internal 'themes'
+	option Bootstrap '/luci-static/bootstrap'
 
 config internal 'apply'
 	option rollback '90'
@@ -31,3 +32,8 @@
 	option timeout '5'
 	option display '1.5'
 
+config internal 'diag'
+	option dns 'openwrt.org'
+	option ping 'openwrt.org'
+	option route 'openwrt.org'
+

You can see that both of these are the same diff, just one is in the cli format and the other is in the config file format. uci took care of replacing tabs and other extra whitespace with single spaces, and double quotes with single quotes, so that the files are directly comparable.

What would you say to making `resolve_conffiles` run `uci show $CONFFILE-opkg | xargs -n 1 uci set && uci commit` as a first step?

Nick commented on 08.04.2021 15:11

Another strategy is to adopt what desktop linux distros do: have /etc/$something.d/ directories, whose contents are merged virtually by uci. That way you can isolate changes made by the user without having to edit the files installed from packages.

Nick commented on 09.04.2021 14:09

On ucitrack, the first prototype works directly,

root@router:~# uci show ucitrack > /tmp/b
root@router:~# uci show ucitrack-opkg | sed 's/-opkg//' > /tmp/a
root@router:~# diff -u /tmp/a /tmp/b
--- /tmp/a	2021-04-08 11:32:41.281049289 -0400
+++ /tmp/b	2021-04-08 11:32:03.460601813 -0400
@@ -1,11 +1,11 @@
 ucitrack.@network[0]=network
 ucitrack.@network[0].init='network'
-ucitrack.@network[0].affects='dhcp'
+ucitrack.@network[0].affects='dhcp' 'radvd'
 ucitrack.@wireless[0]=wireless
 ucitrack.@wireless[0].affects='network'
 ucitrack.@firewall[0]=firewall
 ucitrack.@firewall[0].init='firewall'
-ucitrack.@firewall[0].affects='luci-splash'
+ucitrack.@firewall[0].affects='luci-splash' 'qos' 'miniupnpd' 'sqm'
 ucitrack.@olsr[0]=olsr
 ucitrack.@olsr[0].init='olsrd'
 ucitrack.@dhcp[0]=dhcp
@@ -24,9 +24,9 @@
 ucitrack.@system[0]=system
 ucitrack.@system[0].init='led'
 ucitrack.@system[0].exec='/etc/init.d/log reload'
-ucitrack.@system[0].affects='luci_statistics'
+ucitrack.@system[0].affects='luci_statistics' 'dhcp'
 ucitrack.@luci_splash[0]=luci_splash
-ucitrack.@luci_splash[0].init=''\''luci_splash'\'''
+ucitrack.@luci_splash[0].init='luci_splash'
 ucitrack.@upnpd[0]=upnpd
 ucitrack.@upnpd[0].init='miniupnpd'
 ucitrack.@ntpclient[0]=ntpclient
@@ -35,3 +35,7 @@
 ucitrack.@samba[0].init='samba'
 ucitrack.@tinyproxy[0]=tinyproxy
 ucitrack.@tinyproxy[0].init='tinyproxy'
+ucitrack.@sqm[0]=sqm
+ucitrack.@sqm[0].init='sqm'
+ucitrack.@luci_statistics[0]=luci_statistics
+ucitrack.@luci_statistics[0].init='luci_statistics'

but I is harder to apply back to the actual system because it doesn't give me mergeable config files directly. Unfortunately my second prototype didn't work too well on ucitrack:

root@router:~# uci show ucitrack-opkg | xargs -n 1 uci set && uci commit
uci: Entry not found
uci: Entry not found
uci: Entry not found
uci: Invalid argument

I think this is probably because this file uses this syntax which can be misinterpreted by the shell:

root@router:~# uci show ucitrack-opkg
ucitrack-opkg.@network[0]=network
ucitrack-opkg.@network[0].init='network'
ucitrack-opkg.@network[0].affects='dhcp'
ucitrack-opkg.@wireless[0]=wireless
[...]
ucitrack-opkg.@samba[0]=samba
ucitrack-opkg.@samba[0].init='samba'
ucitrack-opkg.@tinyproxy[0]=tinyproxy
ucitrack-opkg.@tinyproxy[0].init='tinyproxy'

So here's a second revision of my second prototype using `uci batch`:

root@router:~# uci show ucitrack-opkg | sed 's/^/set /' | uci batch
root@router:~# uci changes
ucitrack-opkg.cfg0102af='network'
ucitrack-opkg.cfg02e233='wireless'
ucitrack-opkg.cfg03b57b='firewall'
ucitrack-opkg.cfg04c845='olsr'
ucitrack-opkg.cfg05ad04='dhcp'
ucitrack-opkg.cfg06f057='odhcpd'
ucitrack-opkg.cfg074dd4='dropbear'
ucitrack-opkg.cfg088cc9='httpd'
ucitrack-opkg.cfg09cd75='fstab'
ucitrack-opkg.cfg0aa258='qos'
ucitrack-opkg.cfg0be48a='system'
ucitrack-opkg.cfg0c003c='luci_splash'
ucitrack-opkg.cfg0d804c='upnpd'
ucitrack-opkg.cfg0e8036='ntpclient'
ucitrack-opkg.cfg0f0f89='samba'
ucitrack-opkg.cfg10822b='tinyproxy'
root@router:~# uci commit
root@router:~# diff -u /etc/config/ucitrack-opkg /etc/config/ucitrack
--- /etc/config/ucitrack-opkg	2021-04-08 11:39:20.000000000 -0400
+++ /etc/config/ucitrack	2021-04-08 10:18:05.000000000 -0400
@@ -1,21 +1,25 @@
 
 config network
 	option init 'network'
-	option affects 'dhcp'
+	list affects 'dhcp'
+	list affects 'radvd'
 
 config wireless
-	option affects 'network'
+	list affects 'network'
 
 config firewall
 	option init 'firewall'
-	option affects 'luci-splash'
+	list affects 'luci-splash'
+	list affects 'qos'
+	list affects 'miniupnpd'
+	list affects 'sqm'
 
 config olsr
 	option init 'olsrd'
 
 config dhcp
 	option init 'dnsmasq'
-	option affects 'odhcpd'
+	list affects 'odhcpd'
 
 config odhcpd
 	option init 'odhcpd'
@@ -35,10 +39,11 @@
 config system
 	option init 'led'
 	option exec '/etc/init.d/log reload'
-	option affects 'luci_statistics'
+	list affects 'luci_statistics'
+	list affects 'dhcp'
 
 config luci_splash
-	option init ''\''luci_splash'\'''
+	option init 'luci_splash'
 
 config upnpd
 	option init 'miniupnpd'
@@ -52,3 +57,9 @@
 config tinyproxy
 	option init 'tinyproxy'
 
+config sqm
+	option init 'sqm'
+
+config luci_statistics
+	option init 'luci_statistics'
+

I now aspire the patch to be adding

uci show $PKG-opkg | sed 's/^/set /' | uci batch && uci commit

into resolve_conffiles.

Nick commented on 16.04.2021 15:41

Just answering my own question a bit over here, I saw this thread and the related wiki pages and which explains why this code path hasn't bothered others.

It's because the standard upgrade method is `sysupgrade`, with optional package list saving and reinstalling via this bonus script.

This also explains to me why I keep having to do `sysctl -w vm.overcommit_memory=1` just to run updates.

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing