Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[22.03] jsonfilter yields unreliable results #8703

Closed
openwrt-bot opened this issue Mar 15, 2021 · 9 comments
Closed

[22.03] jsonfilter yields unreliable results #8703

openwrt-bot opened this issue Mar 15, 2021 · 9 comments
Labels
flyspray release/22.03 pull request/issue targeted (also) for OpenWrt 22.03 release

Comments

@openwrt-bot
Copy link

arnysch:

  • Device is an Arcadyan VGV7510KW22 (o2 Box 6431)
  • Box software version is OpenWrt 19.07-SNAPSHOT, r11318-c64742a96e
  • Problem shows up in package wireguard_tool which provides script wireguard_watchdog" which uses jsonfilter.
  • Short steps to reproduce:
  • Use attached file jinp1 as input and run this command:
    jsonfilter -a -e '@[@.proto="wireguard"].interface' <jout1
    => nothing is output (empty output), which is incorrect

  • Now use attached file jinp2, which is jinp1 with lines slighty reordered:
    jsonfilter -a -e '@[@.proto="wireguard"].interface' <jout2
    => "wgvpn" is output, which is correct, this time.

  • Background: Script wireguard_watchdog tries to identify the WireGuard network interfaces.
    To this end it runs these commands:
    ubus -S call network.interface dump | jsonfilter -e '@.interface[@.up=true]' | jsonfilter -a -e '@[@.proto="wireguard"].interface'
    The second jsonfilter call in this line erroneously does not show a result.
    The attached files jinp1, jinp2 contain example input for the 2nd jsonfilter call
@openwrt-bot
Copy link
Author

mikma:

The problem is caused by a bug in the array parsing. It can be solved with the following patch.

https://patchwork.ozlabs.org/project/openwrt/list/?series=244999

@aparcar aparcar added the release/19.07 pull request/issue targeted (also) for OpenWrt 19.07 release label Feb 22, 2022
@ynezz
Copy link
Member

ynezz commented May 24, 2022

OpenWrt 19.07 release is EOL, try to reproduce the issue with latest supported release and feel free to ask for issue reopening if the problem is still present, thanks.

@ynezz ynezz closed this as completed May 24, 2022
@riobard
Copy link

riobard commented Sep 9, 2022

This problem still exists in 22.03. Please reopen. Below is another example that would trigger the bug:

{ "interface": "lan", "up": true, "pending": false, "available": true, "autostart": true, "dynamic": false, "uptime": 455693, "l3_device": "br-lan", "proto": "dhcp", "device": "br-lan", "metric": 0, "dns_metric": 0, "delegation": true, "ipv4-address": [ { "address": "192.168.123.58", "mask": 24 } ], "ipv6-address": [ ], "ipv6-prefix": [ ], "ipv6-prefix-assignment": [ ], "route": [ { "target": "0.0.0.0", "mask": 0, "nexthop": "192.168.123.1", "source": "192.168.123.58\/32" } ], "dns-server": [ "192.168.123.1" ], "dns-search": [ ], "neighbors": [ ], "inactive": { "ipv4-address": [ ], "ipv6-address": [ ], "route": [ ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ] }, "data": { "dhcpserver": "192.168.123.1", "hostname": "foo", "leasetime": 43200 } }
{ "interface": "lan6", "up": true, "pending": false, "available": true, "autostart": true, "dynamic": false, "uptime": 455689, "l3_device": "br-lan", "proto": "dhcpv6", "device": "br-lan", "metric": 0, "dns_metric": 0, "delegation": false, "ipv4-address": [ ], "ipv6-address": [ { "address": "1234:123:1234:1234:1234:123:1234:1234", "mask": 64, "preferred": 172773, "valid": 172773 } ], "ipv6-prefix": [ ], "ipv6-prefix-assignment": [ ], "route": [ { "target": "1234:123:1234:1234::", "mask": 64, "nexthop": "::", "metric": 256, "valid": 172773, "source": "::\/0" }, { "target": "::", "mask": 0, "nexthop": "fe80::1234:1234:1234:1234", "metric": 512, "valid": 1173, "source": "1234:123:1234:1234:1234:123:1234:1234\/64" } ], "dns-server": [ "1234:123:1234:1234::1", "1234:123:1234:1234::1" ], "dns-search": [ ], "neighbors": [ ], "inactive": { "ipv4-address": [ ], "ipv6-address": [ ], "route": [ ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ] }, "data": { "passthru": "0000000000000000000000000000000000000001" } }
{ "interface": "wg0", "up": true, "pending": false, "available": true, "autostart": true, "dynamic": false, "uptime": 1079353, "l3_device": "wg0", "proto": "wireguard", "updated": [ "addresses" ], "metric": 0, "dns_metric": 0, "delegation": false, "ipv4-address": [ { "address": "100.100.100.2", "mask": 24 } ], "ipv6-address": [ ], "ipv6-prefix": [ ], "ipv6-prefix-assignment": [ ], "route": [ ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ], "inactive": { "ipv4-address": [ ], "ipv6-address": [ ], "route": [ ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ] }, "data": { } }

Save the above into a file named test.json and run

<test.json jsonfilter -a -e '@[@.proto="wireguard"].interface'

would produce nothing, where the expected result would be wg0.

@riobard
Copy link

riobard commented Sep 15, 2022

@ynezz ping

@ynezz ynezz reopened this Sep 15, 2022
@ynezz ynezz added release/22.03 pull request/issue targeted (also) for OpenWrt 22.03 release and removed release/19.07 pull request/issue targeted (also) for OpenWrt 19.07 release labels Sep 15, 2022
@ynezz ynezz changed the title FS#3683 - jsonfilter yields unreliable results [22.03] jsonfilter yields unreliable results Sep 15, 2022
@0x5e
Copy link

0x5e commented Oct 2, 2022

Same issue at ImmortalWrt 21.02.1 r19441+1-f8624db86c, jsonfilter - 2018-02-04-c7e938d6-1
same version with master branch

PKG_NAME:=jsonfilter
PKG_RELEASE:=1
PKG_SOURCE_PROTO:=git
PKG_SOURCE_URL=$(PROJECT_GIT)/project/jsonpath.git
PKG_SOURCE_DATE:=2018-02-04
PKG_SOURCE_VERSION:=c7e938d6582a436dddc938539e72dd1320625c54
PKG_MIRROR_HASH:=0601b4d7aa5ee096e99388a57cb0701673ab58fccd6ed2984a2abbd4f846e045
CMAKE_INSTALL:=1

Here's my sample data:

{ "interface": "lan", "up": true, "pending": false, "available": true, "autostart": true, "dynamic": false, "uptime": 362849, "l3_device": "br-lan", "proto": "static", "device": "br-lan", "updated": [ "addresses" ], "metric": 0, "dns_metric": 0, "delegation": false, "ipv4-address": [ { "address": "192.168.50.1", "mask": 24 } ], "ipv6-address": [ ], "ipv6-prefix": [ ], "ipv6-prefix-assignment": [ { "address": "240e:000:000:0000::", "mask": 60, "preferred": 135300, "valid": 221700, "local-address": { "address": "240e:000:000:0000::1", "mask": 60 } } ], "route": [ ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ], "inactive": { "ipv4-address": [ ], "ipv6-address": [ ], "route": [ ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ] }, "data": { } }
{ "interface": "loopback", "up": true, "pending": false, "available": true, "autostart": true, "dynamic": false, "uptime": 362848, "l3_device": "lo", "proto": "static", "device": "lo", "updated": [ "addresses" ], "metric": 0, "dns_metric": 0, "delegation": true, "ipv4-address": [ { "address": "127.0.0.1", "mask": 8 } ], "ipv6-address": [ ], "ipv6-prefix": [ ], "ipv6-prefix-assignment": [ ], "route": [ ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ], "inactive": { "ipv4-address": [ ], "ipv6-address": [ ], "route": [ ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ] }, "data": { } }
{ "interface": "modem", "up": true, "pending": false, "available": true, "autostart": true, "dynamic": false, "uptime": 362848, "l3_device": "eth0", "proto": "static", "device": "eth0", "updated": [ "addresses", "routes" ], "metric": 0, "dns_metric": 0, "delegation": true, "ipv4-address": [ { "address": "192.168.1.2", "mask": 24 } ], "ipv6-address": [ ], "ipv6-prefix": [ ], "ipv6-prefix-assignment": [ ], "route": [ ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ], "inactive": { "ipv4-address": [ ], "ipv6-address": [ ], "route": [ { "target": "0.0.0.0", "mask": 0, "nexthop": "192.168.1.1", "source": "0.0.0.0\/0" } ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ] }, "data": { } }
{ "interface": "wan", "up": true, "pending": false, "available": true, "autostart": true, "dynamic": false, "uptime": 296705, "l3_device": "pppoe-wan", "proto": "pppoe", "device": "eth0", "updated": [ "addresses", "routes" ], "metric": 0, "dns_metric": 0, "delegation": false, "ipv4-address": [ { "address": "111.111.111.111", "mask": 32, "ptpaddress": "222.222.222.222" } ], "ipv6-address": [ { "address": "fe80::0000:0000:0000:0000", "mask": 128 } ], "ipv6-prefix": [ ], "ipv6-prefix-assignment": [ ], "route": [ { "target": "0.0.0.0", "mask": 0, "nexthop": "111.111.111.111", "source": "0.0.0.0\/0" } ], "dns-server": [ "111.111.111.111", "111.111.111.111" ], "dns-search": [ ], "neighbors": [ ], "inactive": { "ipv4-address": [ ], "ipv6-address": [ ], "route": [ ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ] }, "data": { } }
{ "interface": "wan_6", "up": true, "pending": false, "available": true, "autostart": true, "dynamic": true, "uptime": 296702, "l3_device": "pppoe-wan", "proto": "dhcpv6", "device": "pppoe-wan", "metric": 0, "dns_metric": 0, "delegation": true, "ipv4-address": [ ], "ipv6-address": [ { "address": "240e:000:000:0000:0000:0000:0000:0000", "mask": 64, "preferred": 172518, "valid": 258918 } ], "ipv6-prefix": [ { "address": "240e:000:000:0000::", "mask": 60, "preferred": 135300, "valid": 221700, "class": "wan_6", "assigned": { "lan": { "address": "240e:000:000:0000::", "mask": 60 } } } ], "ipv6-prefix-assignment": [ ], "route": [ { "target": "::", "mask": 0, "nexthop": "fe80::000:0000:0000:0000", "metric": 512, "valid": 1518, "source": "240e:000:000:000::\/60" }, { "target": "::", "mask": 0, "nexthop": "fe80::000:0000:0000:0000", "metric": 512, "valid": 1518, "source": "240e:000:000:000:0000:0000:0000:0000\/64" } ], "dns-server": [ "240e:00:000::1", "240e:00:000::2" ], "dns-search": [ ], "neighbors": [ ], "inactive": { "ipv4-address": [ ], "ipv6-address": [ ], "route": [ ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ] }, "data": { "passthru": "000000000000000000000000000000000000000000000000000000000000000000000000", "zone": "wan" } }
{ "interface": "wg0", "up": true, "pending": false, "available": true, "autostart": true, "dynamic": false, "uptime": 296701, "l3_device": "wg0", "proto": "wireguard", "updated": [ "addresses", "routes" ], "metric": 0, "dns_metric": 0, "delegation": true, "ipv4-address": [ { "address": "192.168.253.2", "mask": 32 } ], "ipv6-address": [ ], "ipv6-prefix": [ ], "ipv6-prefix-assignment": [ ], "route": [ { "target": "192.168.253.1", "mask": 32, "nexthop": "0.0.0.0", "source": "0.0.0.0\/0" }, { "target": "192.168.253.5", "mask": 32, "nexthop": "0.0.0.0", "source": "0.0.0.0\/0" }, { "target": "192.168.253.10", "mask": 32, "nexthop": "0.0.0.0", "source": "0.0.0.0\/0" }, { "target": "192.168.40.0", "mask": 24, "nexthop": "0.0.0.0", "source": "0.0.0.0\/0" }, { "target": "192.168.60.0", "mask": 24, "nexthop": "0.0.0.0", "source": "0.0.0.0\/0" } ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ], "inactive": { "ipv4-address": [ ], "ipv6-address": [ ], "route": [ ], "dns-server": [ ], "dns-search": [ ], "neighbors": [ ] }, "data": { } }

My temp workaround: add grep wireguard between two jsonfilter call :)

# /usr/bin/wireguard_watchdog

...

# query ubus for all active wireguard interfaces
# wg_ifaces=$(ubus -S call network.interface dump | jsonfilter -e '@.interface[@.up=true]' | jsonfilter -a -e '@[@.proto="wireguard"].interface' | tr "\n" " ")
wg_ifaces=$(ubus -S call network.interface dump | jsonfilter -e '@.interface[@.up=true]' | grep wireguard | jsonfilter -a -e '@[@.proto="wireguard"].interface' | tr "\n" " ")

...

@pputerla
Copy link

one can filter with one call to mitigate the bug in jsonfilter:

'@.interface[@.up=true && @.proto="wireguard"].interface'

@pputerla
Copy link

@ynezz I didn't quite understand the label release/22.03

I can confirm fully that the issue exists on 21.02.3 but does not on 22.03.2.

riobard added a commit to riobard/openwrt that referenced this issue Apr 7, 2023
Eliminate the extra call to jsonfilter to enumerate active WireGuard
interfaces to avoid potential bugs and be more efficient.

This change was originally proposed by @pputerla in openwrt#11649 as a
workaround for openwrt#8703. Since 22.03.2 the issue seems to be fixed so
@pputerla closed the pull request, but I think it's still very worth it as
it is more efficient and robust.

Signed-off-by: Rio <riobard@users.noreply.github.com>
@gralpli
Copy link

gralpli commented Jan 22, 2024

This issue still exists in 23.05.2, today I ran into it. I fixed it by installing the “jq” package and replacing

wg_ifaces=$(ubus -S call network.interface dump | jsonfilter -e '@.interface[@.up=true]' | jsonfilter -a -e '@[@.proto="wireguard"].interface' | tr "\n" " ")

with

wg_ifaces=$(ubus -S call network.interface dump | jq --join-output '[.interface[] | select(.proto=="wireguard" and .up==true) | .interface] | join(" ")')


I can also confirm that #12344 fixes my problem by using the following replacement:

wg_ifaces=$(ubus -S call network.interface dump | jsonfilter -e '@.interface[@.up=true && @.proto="wireguard"].interface' | tr "\n" " ")


I attached some example data that triggers the error: example_data.json

  • cat example_data.json | jsonfilter -e '@.interface[@.up=true]' | jsonfilter -a -e '@[@.proto="wireguard"].interface' | tr "\n" " " does not work (empty output).

  • cat example_data.json | jq --join-output '[.interface[] | select(.proto=="wireguard" and .up==true) | .interface] | join(" ")' (my version) does work (output: “wg0 wg1 wg2”).

  • cat example_data.json | jsonfilter -e '@.interface[@.up=true && @.proto="wireguard"].interface' | tr "\n" " " (the version from wireguard-tools: call jsonfilter just once #12344) does also work (output: “wg0 wg1 wg2 ”).

openwrt-bot pushed a commit that referenced this issue Jan 23, 2024
013b75ab0598 jsonfilter: drop legacy json-c support
594cfa86469c main: fix spurious premature parse aborts in array mode

Fixes: https://bugs.openwrt.org/?task_id=3683
Fixes: #8703
Fixes: #11649
Fixes: #12344
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
(cherry picked from commit 33f15dd)
openwrt-bot pushed a commit that referenced this issue Jan 23, 2024
013b75ab0598 jsonfilter: drop legacy json-c support
594cfa86469c main: fix spurious premature parse aborts in array mode

Fixes: https://bugs.openwrt.org/?task_id=3683
Fixes: #8703
Fixes: #11649
Fixes: #12344
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
(cherry picked from commit 33f15dd)
HiGarfield pushed a commit to HiGarfield/lede-17.01.4-Mod that referenced this issue Jan 23, 2024
013b75ab0598 jsonfilter: drop legacy json-c support
594cfa86469c main: fix spurious premature parse aborts in array mode

Fixes: https://bugs.openwrt.org/?task_id=3683
Fixes: openwrt/openwrt#8703
Fixes: openwrt/openwrt#11649
Fixes: openwrt/openwrt#12344
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
@efahl
Copy link

efahl commented Jan 24, 2024

This bug appears to be fixed in today's snapshot build of jsonfilter 2024-01-23-594cfa86-1. Tested with a handful of previously problematic files and the test cases all work now.

Below is my worst test, as snort produces randomly ordered output with this command and previously jsonfilter would produce somewhere on the order of 10-40 output lines with each invocation.

$ snort -c /var/snort.d/snort_conf.lua --dump-rule-meta | wc -l
4230
$ snort -c /var/snort.d/snort_conf.lua --dump-rule-meta | jsonfilter -a -e '$[*].sid' | wc -l
4230

Vladdrako pushed a commit to Vladdrako/openwrt that referenced this issue Jan 25, 2024
013b75ab0598 jsonfilter: drop legacy json-c support
594cfa86469c main: fix spurious premature parse aborts in array mode

Fixes: https://bugs.openwrt.org/?task_id=3683
Fixes: openwrt#8703
Fixes: openwrt#11649
Fixes: openwrt#12344
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
db260179 pushed a commit to db260179/openwrt that referenced this issue Jan 31, 2024
013b75ab0598 jsonfilter: drop legacy json-c support
594cfa86469c main: fix spurious premature parse aborts in array mode

Fixes: https://bugs.openwrt.org/?task_id=3683
Fixes: openwrt#8703
Fixes: openwrt#11649
Fixes: openwrt#12344
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
sbeach92 pushed a commit to sbeach92/openwrt that referenced this issue Feb 16, 2024
013b75ab0598 jsonfilter: drop legacy json-c support
594cfa86469c main: fix spurious premature parse aborts in array mode

Fixes: https://bugs.openwrt.org/?task_id=3683
Fixes: openwrt#8703
Fixes: openwrt#11649
Fixes: openwrt#12344
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
rondoval pushed a commit to rondoval/openwrt that referenced this issue Feb 25, 2024
013b75ab0598 jsonfilter: drop legacy json-c support
594cfa86469c main: fix spurious premature parse aborts in array mode

Fixes: https://bugs.openwrt.org/?task_id=3683
Fixes: openwrt#8703
Fixes: openwrt#11649
Fixes: openwrt#12344
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
(cherry picked from commit 33f15dd)
Zack0617 pushed a commit to Zack0617/openwrt that referenced this issue Mar 25, 2024
013b75ab0598 jsonfilter: drop legacy json-c support
594cfa86469c main: fix spurious premature parse aborts in array mode

Fixes: https://bugs.openwrt.org/?task_id=3683
Fixes: openwrt/openwrt#8703
Fixes: openwrt/openwrt#11649
Fixes: openwrt/openwrt#12344
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
(cherry picked from commit 33f15dd6d41873b02eb8895b8886763659f1390c)
Zack0617 pushed a commit to Zack0617/openwrt that referenced this issue Mar 25, 2024
013b75ab0598 jsonfilter: drop legacy json-c support
594cfa86469c main: fix spurious premature parse aborts in array mode

Fixes: https://bugs.openwrt.org/?task_id=3683
Fixes: openwrt/openwrt#8703
Fixes: openwrt/openwrt#11649
Fixes: openwrt/openwrt#12344
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
(cherry picked from commit 33f15dd6d41873b02eb8895b8886763659f1390c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flyspray release/22.03 pull request/issue targeted (also) for OpenWrt 22.03 release
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants