New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FS#3759 - Idle ssh Connection exits with: client_loop: send disconnect: Broken pipe #8776
Comments
farmergreg: Update: it turns out that my simple script above (watch ‘sleep 3000;date’) also exits on 19.07.7 with the "client_loop: send disconnect: Broken pipe" error. I have another script ( a long running backup job) that doesn't exit with the error when running 19.07.7, but does exit with the error when running the rc version. That script runs a backup (typically 2 - 3 minutes), then sleeps until the next hour starts and then runs again. So I'm not completely sure this is an openwrt bug, but there is certainly a change in behavior between the rc and the 19.07.x releases. |
farmergreg: Update: SSH without wireguard is working fine for me when using the RC candidate. SSH over a wireguard tunnel is where I start seeing the "client_loop: send disconnect: Broken pipe " error messages. Network Diagram: machine1 <-- wireguard --> OpenWRT A <--> internet <--> OpenWRT B <---> wireguard <---> machine2 The wireguard vpn is a site to site link. Both routers involved are running openwrt. Both have ports open so that either router can initiate the wireguard tunnel. I am able to quickly and easily switch between 19.07.7 and 21.02.0-rc1 on OpenWRT router "A". 19.07.7 works great and ssh connections appear to stay alive indefinitely. 21.02.0-rc1 has the "client_loop: send disconnect: Broken pipe" problem and ssh connections typically exit early/unexpectedly. |
farmergreg: This is present in all rc releases thus far. I installed 21.02.0-rc3 today and an idle ssh connection made over wireguard exits after a little while with "client_loop: send disconnect: Broken pipe" this does not happen in 19.07.7 |
farmergreg: I can confirm that this is still an issue in rc4. The wireguard tunnel in question is has ipv4 and ipv6 addresses locally, however the ssh connection is made using ipv4 to a vm on the remote side that only has ipv4 enabled. Turning off software flow offloading is the workaround for now. |
patrakov: This is the same as https://forum.openwrt.org/t/software-flow-offloading-and-conntrack-timeouts/74588 and effectively confirmed as a WONTFIX in that thread. |
buraktamturk: I am having the same issue with no wireguard setup (just pppoe connection), disabling software flow offloading as a workaround fixed the issue. |
Kernel issue, not an OpenWRT issue. Caused by kernel commit e97d940. It's fixed in Master (kernel 5.15) with kernel commit 4592ee7f. To fix it for 22.03 (kernel 5.10), you can directly backport 4592ee7f into kernel 5.10. For 21.02 (kernel 5.4), see the attached patch for a fix. Note that the attached patch for kernel 5.4 is not a revert or backport, but a hack. Changes in the kernel code before 5.4 prevent e97d940 (which caused the issue) from being reverted directly, while features added later than kernel 5.4 prevent 4592ee7f (which fixed the issue for kernel 5.15) from being backported directly. So this patch essentially picks all necessary changes from 4592ee7f, while skipping / working around stuff that kernel 5.4 doesn't have yet (ie. sysctl-configurable offload timeout) - to restore the behavior from before e97d940. |
@Ansuel Hope that mentioning you here triggers a notification to you in GH. I don't want to clutter the 22.03 release thread. More detail (I hope that I can recall everything properly after all those months): When conntrack drops a flow from offloaded back to non-offloaded, pre-e97d940 kept the flow lingering until the actual TCP/UDP connection timeout (then hardcoded, iirc). This was, and still is, the proper behavior. However, the author of e97d940 decided that arbitrarily using timeouts of 2 min (TCP) resp. 30 sec (UDP) was a much better idea and hardcoded these into his commit. The effect was that, once a flow is given back from offloaded to non-offloaded due to inactivity, and it stayed idle for more than the arbitrary 2m resp. 30s, it was dropped completely, causing this very issue. Later, these "offload pickup timeouts" were made sysctl-configurable, and finally, with 4592ee7f, luckily someone decided that it's pointless to have different TCP/UDP timeouts for flows released from offloading than for non-offloaded flows. Therefore, "flow offload pickup" timeouts were dropped completely, making normal TCP/UDP timeouts apply again. By then, the timeout for giving back offloaded flows was configurable, so 4592ee7f can't be backported to 5.4 without changes. I wasn't (knowingly) affected by this issue, but when someone insisted in this being the cause for #8239, I decided to test it out. Of course, wasn't related... Then I almost forgot that I created this 5.4-hack. But at some point - or rather, when I switched from 19.07 to 21.02 - I noticed that I'm experiencing exactly this issue: unstable SSH connections when they're idle. Since then, I'm using my 5.4-hack in every 21.02 build (everyday use), and the 5.15-backport for every 22.03 build
Edit: Should've re-read the commit messages again, before writing this text off-hand. This is about offloaded vs. non-offloaded, not hw vs. sw offload. Text above fixed. Sorry. |
I think that I saw this backport as part of a "backport some missing flowtable fixes" commit somewhere in @hauke's staging tree. So perhaps it's already pending for 22.03 release? As for 5.4, anybody can take my patch and do with it whatever they want - use it, change it, PR it, etc. - I don't need (or want) to be credited. |
Ping? |
@Shine- thank you very much for investigating this issue and also for you patience! @Ansuel if you have the time please backport this to 22.03 as soon as possible, I think we have more time until the next 21.03 point release. @hauke does this ring a bell? I didn't see these patches in your latest work but may overseen something |
@aparcar ok will backport this... just to make sure for both 5.10 and 5.15 right? |
Based on the comments I assume it's fixed in 5.15 |
Thanks for your quick replies! P.S.: found Hauke's staging commit, it's backport 613-02 from here: https://git.openwrt.org/?p=openwrt/staging/hauke.git;a=commitdiff;h=ac21c648ad5b16359bea99f2af3b512639b10e3c |
You can take the patch from my branch. I do not have a real test setup here now. |
This backports some patches from kernel 5.15 to fix issues with flowtable offloading in kernel 5.10. OpenWrt backports most of the patches related to flowtable offloading from kernel 5.15 already, but we are missing some of the extra fixes. This fixes some connection tracking problems when a flow gets removed from the offload and added to the normal SW path again. The patch 614-v5.18-netfilter-flowtable-fix-TCP-flow-teardown.patch was extended manually with the nf_conntrack_tcp_established() function. All changes are already included in kernel 5.15. Fixes: openwrt#8776 Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
This backports some patches from kernel 5.15 to fix issues with flowtable offloading in kernel 5.10. OpenWrt backports most of the patches related to flowtable offloading from kernel 5.15 already, but we are missing some of the extra fixes. This fixes some connection tracking problems when a flow gets removed from the offload and added to the normal SW path again. The patch 614-v5.18-netfilter-flowtable-fix-TCP-flow-teardown.patch was extended manually with the nf_conntrack_tcp_established() function. All changes are already included in kernel 5.15. Fixes: openwrt#8776 Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
This backports some patches from kernel 5.15 to fix issues with flowtable offloading in kernel 5.10. OpenWrt backports most of the patches related to flowtable offloading from kernel 5.15 already, but we are missing some of the extra fixes. This fixes some connection tracking problems when a flow gets removed from the offload and added to the normal SW path again. The patch 614-v5.18-netfilter-flowtable-fix-TCP-flow-teardown.patch was extended manually with the nf_conntrack_tcp_established() function. All changes are already included in kernel 5.15. Fixes: openwrt#8776 Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
This backports some patches from kernel 5.15 to fix issues with flowtable offloading in kernel 5.10. OpenWrt backports most of the patches related to flowtable offloading from kernel 5.15 already, but we are missing some of the extra fixes. This fixes some connection tracking problems when a flow gets removed from the offload and added to the normal SW path again. The patch 614-v5.18-netfilter-flowtable-fix-TCP-flow-teardown.patch was extended manually with the nf_conntrack_tcp_established() function. All changes are already included in kernel 5.15. Fixes: #8776 Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de> (cherry picked from commit 96ef2da)
This backports some patches from kernel 5.15 to fix issues with flowtable offloading in kernel 5.10. OpenWrt backports most of the patches related to flowtable offloading from kernel 5.15 already, but we are missing some of the extra fixes. This fixes some connection tracking problems when a flow gets removed from the offload and added to the normal SW path again. The patch 614-v5.18-netfilter-flowtable-fix-TCP-flow-teardown.patch was extended manually with the nf_conntrack_tcp_established() function. All changes are already included in kernel 5.15. Fixes: openwrt#8776 Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
farmergreg:
=====Summary=====
=====Steps to reproduce=====
SSH from one computer to another machine that is on the internet
machine 1 < ---- > openwrt 21.02.0-rc1 <---- internet ----> machine 2
Run this command on machine 1: watch 'sleep 3000;date'
After some time has passed (wait about 30 minutes, maybe a little less):
The text was updated successfully, but these errors were encountered: