Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#3499 - [netifd] no way to monitor the real state of a connection via protocol handlers #8368

Closed
openwrt-bot opened this issue Dec 9, 2020 · 8 comments
Labels

Comments

@openwrt-bot
Copy link

aleksander0m:

The current ModemManager integration in netifd is not fully operational right now, and usually requires an additional management script on top of it to properly reconnect the modems under certain scenarios.

The simplest and most common use case is the following:

  • The network interface configured for "modemmanager" protocol is ifup-ed.
  • netifd runs the connection start procedure in the protocol handler and the modem gets connected.
  • Some time passes, and the network operator decides to disconnect the connection (network-initiated disconnection).
  • ModemManager detects the disconnection, reports the bearer objects a "disconnected" and the modem back to only "registered".
  • netifd still thinks the interface is up, and reports it as up everywhere.

See e.g. openwrt/packages#14096

What would be the best way to solve this issue?

Is there currently any way to extend the protocol handler in a way that allows some polling inside netifd to ensure the interface is still up? Would that be a big change? Any other solution suggested?

@openwrt-bot
Copy link
Author

johnth:

Hi,
Thank you for you work with ModemManager!
For a long time, I have used a USB NCM modem (without knowing of MM),
and have wanted a way for it to detect disconnect and automatically reconnect.

I do not know ModemManager, or ubus / netifd.

I do not know if it works, but it looks like at one stage, ppp used netifd for reconnects:
openwrt-mirror/openwrt@3411334

Could we use MM's dbus state change to make a ubus call to netifd?
Even if it is an external daemon monitoring dbus?
https://www.freedesktop.org/software/ModemManager/api/latest/ModemManager-Flags-and-Enumerations.html#MMModemState

Example:
From the ppp proto, the proto_run_command uses the ip-down-script /lib/netifd/ppp-down, which sends a message to netifd via ubus through the shell library.
https://github.com/openwrt/openwrt/blob/master/package/network/services/ppp/files/lib/netifd/ppp-down

ubus monitor -> 97d0daf0 #97d0daf0 hello: {} <- 97d0daf0 #00000000 lookup: {"objpath":"network.interface"} -> 97d0daf0 #00000000 data: {"objpath":"network.interface","objid":140326987,"objtype":-806725414,"signature":{"up":{},"down":{},"renew":{},"status":{},"prepare":{},"dump":{},"add_device":{"name":3,"link-ext":7,"vlan":1},"remove_device":{"name":3,"link-ext":7,"vlan":1},"notify_proto":{},"remove":{},"set_data":{}}} -> 97d0daf0 #00000000 status: {"status":0} <- 97d0daf0 #085d384b invoke: {"objid":140326987,"method":"notify_proto","data":{"action":0,"ifname":"ifname_example","link-up":false,"keep":false,"interface":"interface_example"}} -> bacdf6e3 #97d0daf0 invoke: {"objid":140326987,"method":"notify_proto","data":{"action":0,"ifname":"ifname_example","link-up":false,"keep":false,"interface":"interface_example"},"user":"root","group":"root"}

. /lib/netifd/netifd-proto.sh
proto_init_update "ifname_example" 0
proto_send_update "interface_example"

echo "$(json_dump)"
{ "action": 0, "ifname": "ifname_example", "link-up": false, "keep": false, "interface": "interface_example" }
ubus call network.interface notify_proto "$(json_dump)"

Netifd shell functions:
/lib/netifd/netifd-proto.sh
https://git.openwrt.org/?p=project/netifd.git;a=blob;f=scripts/netifd-proto.sh

json shell functions:
/usr/share/libubox/jshn.sh
https://git.openwrt.org/?p=project/libubox.git;a=blob;f=sh/jshn.sh

If something like this //might// work, I can keep digging?

Cheers

@openwrt-bot
Copy link
Author

aleksander0m:

I haven't seen your reply until now; sorry!

We recently discussed about how to best handle this in the MM mailing list, and a solution like the one you suggest, using proto_init_update $ifname 0; proto_send_update, makes sense.

The way forward I'm suggesting is to allow ModemManager to run "post-event" dispatcher scripts; e.g. we could allow the user to have custom scripts that are run by the MM daemon when a given connection is connected or disconnected. Then, the openwrt packaging could provide a script to call on network-triggered disconnections, which would do the proto_init_update $ifname 0; proto_send_update call to notify netifd that the connection is really down.

See https://lists.freedesktop.org/archives/modemmanager-devel/2022-January/009075.html

What do you think of this? It would not require a separate daemon to monitor the connection, it would all be managed by MM and the openwrt-installed dispatcher script.

@aleksander0m
Copy link

A fix for this is explained here: https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775
Comments welcome

@wicadmin
Copy link

This looks great and definitely a welcomed enhancement.

aleksander0m added a commit to aleksander0m/openwrt-packages that referenced this issue May 4, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
aleksander0m added a commit to aleksander0m/openwrt-packages that referenced this issue May 4, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
aleksander0m added a commit to aleksander0m/openwrt-packages that referenced this issue May 4, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
aleksander0m added a commit to aleksander0m/openwrt-packages that referenced this issue May 4, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
1715173329 pushed a commit to immortalwrt/packages that referenced this issue May 17, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt/packages#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
(cherry picked from commit bc754f3)
Beginner-Go pushed a commit to coolsnowwolf/packages that referenced this issue Jun 16, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt/packages#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
Beginner-Go pushed a commit to coolsnowwolf/packages that referenced this issue Jun 16, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt/packages#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
Beginner-Go added a commit to coolsnowwolf/packages that referenced this issue Jun 16, 2022
* modemmanager: bump to 1.16.10

Signed-off-by: Nicholas Smith <nicholas@nbembedded.com>

* modemmanager: switch to autorelease

Signed-off-by: Nicholas Smith <nicholas@nbembedded.com>

* modemmanager: bump to 1.18.2

Signed-off-by: Kuan-Yi Li <kyli@abysm.org>

* modemmanager: add support for wwan subsystem in hotplug

WWAN devices may now be exposed in the new 'wwan' subsystem in the
kernel (since 5.13), initially applicable to devices exposed in PCIe
(no USB), but at some point may also apply to USB devices that until
now were exposed via other subsystems (e.g. usbmisc, tty).

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>

* modemmanager: fix physdev sysfs path detection in PCIe modems

The PCIe physdev path lookup relies on the 'vendor' and 'device'
attribute files, instead of the 'idVendor' and 'idProduct' ones, which
are USB specific.

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>

* ModemManager: refactoring procd init script

The way the init script is written now, we get a bad output when calling
the ubus service backend.

ubus call service list "{'verbose':true,'name':'modemmanager'}"
>{
>        "modemmanager": {
>                "instances": {
>                        "instance1": {
>                                "running": true,
>                                "pid": 20511,
>                                "command": [
>                                        "sh",
>                                        "-c",
>                                        ".
>/usr/share/ModemManager/modemmanager.common;    \t
>mkdir -m 0755 -p /var/run/modemmanager;          \t
>mm_cleanup_interfaces;                            \t
>( mm_report_events_from_cache ) >/dev/null 2>&1 & \t
>/usr/sbin/ModemManager"
>                                ],
>                                "term_timeout": 5,
>                                "respawn": {
>                                        "threshold": 3600,
>                                        "timeout": 5,
>                                        "retry": 5
>                                },
>                                "pidfile":"/var/run/modemmanager/modemmanager.pid"
>                        }
>                }
>        }
>}"

I also get the output in the log that the PID file cannot be created.

> daemon.err procd: Failed to remove pidfile: :No such file or directory

The changes in this commit fixes this issues, by moving startup into a
wrapper script.

Signed-off-by: Florian Eckert <fe@dev.tdt.de>

* ModemManager: add service options

Signed-off-by: Florian Eckert <fe@dev.tdt.de>

* modemmanager: refactoring hotplug debug logging

The output of the hotplug is very chatty and floods the log with
messages that are not necessary in functioning operation.

So that the log can be filtered. A log level was added to each message
as the first opiton on mm_log function call.

In addition, the facility of the hotplug script has been set to daemon,
which in my view fits better than user.

Signed-off-by: Florian Eckert <fe@dev.tdt.de>

* modemmanager: remove 2s timeout before reporting cached events

When ModemManager is started on boot we may end up with hotplug events
reported directly to the daemon, plus some others already cached in
the cache file before the daemon was started.

If the cached events correspond to the same device that is still
notifying ports directly, we may end up with a modem object created
before the cached events have been emitted, so the modem may not
handle all control/data ports it should.

E.g.:

  - modem detected
  - hotplug event for wwan0 port, cached as MM not running
  - hotplug event for cdc-wdm0 port, cached as MM not running
  - hotplug event for ttyUSB0, cached as MM not running
  - MM starts
  - hotplug event for ttyUSB1, directly processed as MM is running
  - hotplug event for ttyUSB2, directly processed as MM is running
  - modem object created with ttyUSB1 and ttyUSB2
  - 2s after MM starts, cached events for wwan0, cdc-wdm0 and ttyUSB0
    happen, but are ignored because the modem object has already been
    created

MM expects that ports of the same device are reported with less than
1500ms in between ports. In other words, if ports are reported more
than 1500ms after the last reported port, they may get ignored.

If we remove the 2s timeout, the report of the cached events will
happen as soon as MM starts, which makes it much more likely to happen
in the timeslot that MM expects for ports of the same device reported.
The logic is still not perfect, and we may also need to increase that
1500ms timeout inside MM, but removing the 2s timeout right away here
makes sense.

This 2s timeout was introduced along with the new wrapper launcher for
the daemon, it didn't exist before.

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>

* modemmanager: bump to 1.18.4 with patch

The included patch corrects an issue preventing udev rules from
being parsed correctly. [1][2]

This patch should not be needed after the next release. [3]

More information:
[1] https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/commit/e9f254ba3f923248efffe4fd458dc5e5469e0f73
[2] https://lists.freedesktop.org/archives/modemmanager-devel/2021-December/009024.html
[3] https://lists.freedesktop.org/archives/modemmanager-devel/2021-December/009022.html

Signed-off-by: Nicholas Smith <nicholas@nbembedded.com>

* modemmanager: bump to 1.18.6

Remove upstreamed patch.

Signed-off-by: Kuan-Yi Li <kyli@abysm.org>

* modemmanager: switch to meson build tools

Using https://gitlab.freedesktop.org/mobile-broadband/ModemManager.git to download the source code.
Added compile option to compile qrtr support.
Enabled lto and additional gcc flags for perfomance and less size.
Modified to use meson as upstream has abandoned autotools.
Removed BUILD_PARALLEL options. These are default with ninja/meson.

Signed-off-by: Maxim Anisimov <maxim.anisimov.ua@gmail.com>

* modemmanager: bump to 1.18.8

Also removed two patches that are already included in the upstream
release.

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>

* modemmanager: install available FCC unlock scripts

The FCC unlock scripts are installed but not enabled by default.

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>

* modemmanager: report network initiated disconnections to netifd

The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt/packages#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>

Co-authored-by: Nicholas Smith <nicholas@nbembedded.com>
Co-authored-by: Kuan-Yi Li <kyli@abysm.org>
Co-authored-by: Aleksander Morgado <aleksander@aleksander.es>
Co-authored-by: Florian Eckert <fe@dev.tdt.de>
Co-authored-by: Maxim Anisimov <maxim.anisimov.ua@gmail.com>
AZUSAHMR pushed a commit to AZUSAHMR/packages that referenced this issue Aug 11, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
AZUSAHMR pushed a commit to AZUSAHMR/packages that referenced this issue Aug 11, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
garikello3d pushed a commit to garikello3d/packages that referenced this issue Aug 15, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
nemesifier pushed a commit to nemesifier/packages that referenced this issue Oct 17, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
(cherry picked from commit bc754f3)
nemesifier pushed a commit to nemesifier/packages that referenced this issue Oct 17, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
(cherry picked from commit bc754f3)
nemesifier pushed a commit to nemesifier/packages that referenced this issue Oct 17, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
(cherry picked from commit bc754f3)
nemesifier pushed a commit to nemesifier/packages that referenced this issue Oct 20, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
(cherry picked from commit bc754f3)
nemesifier pushed a commit to nemesifier/packages that referenced this issue Oct 27, 2022
The new connection dispatcher scripts support integrated in
ModemManager 1.18.8 allows us to provide a openwrt-specific dispatcher
script used to report netifd that the underlying network connection is
down.

See also https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/775

Fixes openwrt/openwrt#8368
Fixes openwrt#14096

Signed-off-by: Aleksander Morgado <aleksander@aleksander.es>
(cherry picked from commit bc754f3)
@lynxthecat
Copy link

lynxthecat commented Nov 1, 2022

I am confused - this important fix does not seem to be present in 22.03.2?

Also, this fix addresses pushing disconnected state to netifd, but what about automatic reconnection upon such disconnection events? @aleksander0m how should that be addressed? In my case I see:

Mon Oct 31 17:37:46 2022 daemon.info [2719]: <info>  [modem0/bearer1]
verbose call end reason (6,36): [3gpp] regular-deactivation

And output of ifstatus is wrong, and I have lost internet connectivity since there is no automatic reconnection. This default behaviour seems somewhat broken - I mean I installed ModemManager in the first place in the hope it would retain internet connectivity through these 48 hour disconnects that my 4G ISP effects.

@aleksander0m
Copy link

And output of ifstatus is wrong,

What is your output of ifstatus after the regular-deactivation event is received in MM?

@lynxthecat
Copy link

lynxthecat commented Nov 1, 2022

@aleksander0m I can't get this unless I wait another 48 hours. Can I simulate with:

root@OpenWrt:~# mmcli -m a -K | grep bearers
modem.generic.bearers.length                    : 1
modem.generic.bearers.value[1]                  : /org/freedesktop/ModemManager1/Bearer/15

root@OpenWrt:~# mmcli -b 15 --disconnect

or an alternative?

I did capture the output from both 'logread' and also 'mmcli -m 0' if that's of any use. The connection state in the latter is reported as 'registered' (i.e. not 'connected').

@lynxthecat
Copy link

@aleksander0m OK I got it as another disconnect occurred in the meantime:

Wed Nov  2 04:20:11 2022 daemon.info [2719]: <info>  [modem3/bearer9] verbose call end reason (6,36): [3gpp] regular-deactivation
Wed Nov  2 04:20:11 2022 daemon.info [2719]: <info>  [modem3] state changed (connected -> registered)
Wed Nov  2 04:20:11 2022 daemon.info [2719]: <info>  [modem3/bearer9] connection #1 finished: duration 64629s, tx: 940108442 bytes, rx: 9966819431 bytes

root@OpenWrt:~# ifstatus wan
{
        "up": true,
        "pending": false,
        "available": true,
        "autostart": true,
        "dynamic": false,
        "uptime": 76945,
        "l3_device": "wwan0",
        "proto": "modemmanager",
        "updated": [
                "addresses",
                "routes"
        ],
        "metric": 0,
        "dns_metric": 0,
        "delegation": true,
        "ipv4-address": [
                {
                        "address": "10.90.XX.XX",
                        "mask": 30
                }
        ],
        "ipv6-address": [

        ],
        "ipv6-prefix": [

        ],
        "ipv6-prefix-assignment": [

        ],
        "route": [
                {
                        "target": "0.0.0.0",
                        "mask": 0,
                        "nexthop": "10.90.YY.ZZ",
                        "source": "10.90.XX.XX/32"
                }
        ],
        "dns-server": [
                "192.168.XX.1"
        ],
        "dns-search": [

        ],
        "neighbors": [

        ],
        "inactive": {
                "ipv4-address": [

                ],
                "ipv6-address": [

                ],
                "route": [

                ],
                "dns-server": [

                ],
                "dns-search": [

                ],
                "neighbors": [

                ]
        },
        "data": {

        }
}
root@OpenWrt:~# ip addr show dev wan
ip: can't find device 'wan'
root@OpenWrt:~# ip addr show dev wwan0
9: wwan0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN qlen 1000
    link/[65534]
    inet 10.90.XX.XX/30 brd 10.90.ZZ.ZZ scope global wwan0
       valid_lft forever preferred_lft forever
    inet6 fe80::92c2:79f2:d9a4:24e3/64 scope link flags 800
       valid_lft forever preferred_lft forever

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants