OpenWrt/LEDE Project

  • Status Waiting on reporter
  • Percent Complete
    0%
  • Task Type Feature Request
  • Category Packages
  • Assigned To No-one
  • Operating System All
  • Severity Low
  • Priority Very Low
  • Reported Version Trunk
  • Due in Version Undecided
  • Due Date Undecided
  • Votes 1
  • Private
Attached to Project: OpenWrt/LEDE Project
Opened by Nicolás Pace - 21.04.2019
Last edited by Jo-Philipp Wich - 22.04.2019

FS#2248 - uhttpd: implement ubus event listener via Server-Sent Event protocol

Usecase:
As a web developer I would benefit from having the possibility to send events from the server to the browser so the browser can react to it dynamically.

The proposed implementation is the leanest possible.

A SSE (Server-Sent Event) is a simple protocol on top of basic http polling to send messages from the http server to the browser for the purpose of instant notifications.
It comes implemented in all major browsers, including mobile.

If paired with `ubus listen` or `ubus monitor`, a seamless integration with the ubus pipeline could be implemented.

We tried to implement it by adding an option in this line of code but got stuck in our lack of corutine mechanisms and how uhttpd works.

The difference with the call and list options is that this one should be long-lived and progressive, preventing the dreaded polling from the web (also, better us of resources, more elegant implementation, etc).

As an example, a very basic implementation of a SSE server in php:

<?php
header('Content-Type: text/event-stream');
header('Cache-Control: no-cache');

echo "data: \"hello world\"" . PHP_EOL;
echo PHP_EOL;
echo "data: \"goodbye world\"" . PHP_EOL;
echo PHP_EOL;
ob_flush();
flush();

An example of a very basic implementation of a SSE client in HTML:

<html>
	<body></body>
	<script>
	var source = new EventSource("demo_sse.php");
	source.onmessage = function(event) {
		document.getElementById("result").innerHTML += event.data + "<br>";
	}; 
	</script>
</html>
Admin
Jo-Philipp Wich commented on 22.04.2019 20:30

Just implement a long living cgi process and adjust timeout and concurrency values accordingly. I do not understand why modifications to uhttpd itself are required.

Please elaborate.

Nicolás Pace commented on 23.04.2019 01:03

Hi Jo,
thanks for your response.
As uhttpd is single threaded (assumption here), i felt that was not possible to keep a long living thread subscribing to the ubus bus.

So, would something like this be possible?

#!/usr/bin/env lua

require "ubus"
require "uloop"

uloop.init()

local conn = ubus.connect()
if not conn then
	error("Failed to connect to ubus")
end

local sub = {
	notify = function( msg, name )
		print("data:{name:\"", name, "\", count:", msg["count"], "\n\n")
	end,
}

conn:subscribe( "test", sub )

uloop.run()

with a publisher like this?
https://gitlab.labs.nic.cz/turris/ubus/blob/master/lua/publisher.lua

if so, will document the usecase so others can benefit.

Still, it feels that it could be something that is integrated in the ubus api, as it would complete it by providing the listen and monitor apis that are not available right now.

Thanks!

Admin
Jo-Philipp Wich commented on 23.04.2019 06:21

The problem I see with listen/monitor APIs is that they can easily use up the entire available connection / request pool in the server. Since there is no builtin timeout by design, a few clients keeping such listen connections open can essentially DoS the entire server.

Nicolás Pace commented on 23.04.2019 10:11

Having someone connected to the router shouldn't take up much ram/cpu. no buffers need to be kept (each event that comes has to be spitted to the client right away, so no need to buffer it).
How many connections would make the DoS kill the device? a few tenths would be enough for many implementations? even one would be enough for some uses.

And as ubus listen/monitor api uses uloop to wait for it, it wouldn't take to much space.

Still... if the functionality is available, would be possible to explore this.
In a use of this functionality, if I am worried about doing a DoS, I would put it behind authentication so only a few users could use it.

Admin
Jo-Philipp Wich commented on 23.04.2019 15:12

Well for starters you would need to untangle the max_request, max_connections and timeout variables because right now they apply to both ubus and http (cgi) requests.

Furthermore we need to treat ubus monitor and/or listen requests differently than list or call requests because right now ubus calls are strictly limited to the global timeout set through the `-t` option.

The SSE protocol looks simple enough so I might consider implementing it but I can't promise any time frame for this at all.

Nicolás Pace commented on 23.04.2019 17:08

Because the way SSE works, if the session times out, the browser will resume it.
If you give the messages ids, the browser will tell you which was the last id it received.

so, no timeout/max_connections untanglement needed (at least at the beginning), neither change of global timeout.

an mvp would be pretty simple, and then we can walk together once it starts being used.

Nicolás Pace commented on 24.04.2019 19:45

Hi Jo,

I did a small example, put these two files in /www/cgi-bin;

ping:

#!/usr/bin/lua

require "ubus"
require "uloop"

uloop.init()

local conn = ubus.connect()
if not conn then
        error("Failed to connect to ubus")
end

ubus_objects = {test = {}}

conn:add(ubus_objects)

conn:notify(ubus_objects.test.__ubusobj, "ping", {})

io.stdout:write("\rContent-Type: application/text\n\n")
io.stdout:write("ping")

pong:

#!/usr/bin/lua

require "ubus"
require "uloop"

io.stdout:write("\rContent-Type: text/eventstream\n\n")
io.stdout:flush()

--uloop.init()

local conn = ubus.connect()
if not conn then
        error("Failed to connect to ubus")
end

local my_event = {
        ping = function(msg)
                io.stdout:write("data:'long pong'\n\n")
                io.stdout:flush()
        end,
}

conn:listen(my_event)

uloop.run()

when i head to router/cgi-bin/ping, it certainly shows 'ping'.
when i head to router/cgi-bin/pong, it stays waiting, with no output.
even if i go again to /ping while leaving pong open, nothing happens on pong.

i think the uloop loop is not working properly there... any suggestions?

ps: no errors in logread either...

Nicolás Pace commented on 25.04.2019 11:43

Ok, did another test with os.popen and it works.
though it is a hack, it is pretty consistent:

#!/usr/bin/lua

require "ubus"
require "uloop"

io.stdout:write("\rContent-Type: text/eventstream\n\n")
io.stdout:write("data:'ping'\n\n")
io.stdout:flush()

ubuslisten = io.popen("ubus listen")

for line in ubuslisten:lines() do
	io.stdout:write("data:'ping'\n\n")
	io.stdout:flush()
end
Nicolás Pace commented on 25.04.2019 13:17

So, it worked!

This is what goes on /www/cgi-bin/ubuslisten

#!/usr/bin/lua

io.stdout:write("Content-Type: text/event-stream\n\n")
io.stdout:flush()

ubuslisten = io.popen("ubus listen")

for line in ubuslisten:lines() do
	io.stdout:flush()
	io.stdout:write("data: ")
	io.stdout:write(line)
	io.stdout:write("\n\n")
	io.stdout:flush()
end

and this on /www/ubuslisten.html

<html>
	<meta charset="UTF-8">
	<body></body>
	<script>
	var source = new EventSource("/cgi-bin/ubuslisten");
		source.addEventListener('message', function(e) {
		  console.log(e.data);
	}, false);
	</script>
</html>

two drawbacks of this implementation:

  1. you loose messages between reconnections: can be fixed by having a small daemon that has a 5second buffer of what has been happening, and each time a new connection happens, if it has a timestamp for the last message (done with id: parameter on SSE if we want), you can ask all the messages that came since that one
  2. the resources usage: this one has a long session, and also uses an extra ubus listen process for each one.

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing