Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2186 - DSA driver performance issue #7023

Open
openwrt-bot opened this issue Mar 14, 2019 · 10 comments
Open

FS#2186 - DSA driver performance issue #7023

openwrt-bot opened this issue Mar 14, 2019 · 10 comments
Labels
flyspray kernel pull request/issue with Linux kernel related changes

Comments

@openwrt-bot
Copy link

npcomplete:

Device: Linksys EA9500
Trunk: Kernel v4.19.25

I've compiled my own kernel and used DSA driver instead of swconfig
one for this router. Note: Kernel v4.19 enables broadcom tags for
B53 family of switches in this DSA driver

Here are some of my observations:

  1. Lan to Lan speed is indeed 1Gbps (file copy)
  2. However, Lan to Wan port is capped at ~500Mbps
    i. My internet is limited to 200Mbps, please see by test
    setup below
    ii. One of the users who has 1GBps internet had initially
    reported this issue

I've tried all the lan ports but the result is same. Note that this
router has 8 lan ports and 1 wan port. Where, switch bcm53012
covers lan4, lan7, lan8 and wan ports. While bcm53125 covers lan1,
lan2, lan3, lan5 and lan6 ports. bcm53125's cpu port(8) hangs of
bcm53012 at port(0). bcm53012 is connected to GMAC.

When I load factory firmware, I do get 1Gbps from lan to wan port

Test setup:

Router <=> pfsense box <=> Modem
||
SSD

File copy is limited to ~500Mbps

Thanks,

Vivek

@openwrt-bot
Copy link
Author

ffainelli:

Keep in mind that the stock firmware makes use of the NATP accelerator, whereas the upstream kernel or OpenWrt does not have that capability. This is somewhere on my TODO of things to play with, but I have not had the time yet. Maybe Rafal has something though I suspect he is in the same boat.

If you can get a "perf record" of your iperf session on the router while doing forwarding that would be helpful to see if there are simple hotspots that could be optimized.

@openwrt-bot
Copy link
Author

npcomplete:

Sure, I'll try to get the perf record from my machine. As well as from the user that has 1Gbps internet.

btw, on the side note I modified the code to force enable tagging for bcm53125. Over all tagging is working great.

Thanks Florian!

@openwrt-bot
Copy link
Author

LGA1150:

As Florian stated above, this is not related to b53 driver.
Nonetheless you can still enable 'software flow-offloading' in firewall settings to reach line speed.

@openwrt-bot
Copy link
Author

npcomplete:

Florian,

Here you go:

root@OpenWrt:~# perf report -k /tmp/vmlinux # To display the perf.data header info, please use --header/--header-only options. # Failed to open /lib/libc.so, continuing without symbols Failed to open /usr/bin/iperf3, continuing without symbols Failed to open /lib/libgcc_s.so.1, continuing without symbols # # Total Lost Samples: 0 # # Samples: 39K of event 'cycles:ppp' # Event count (approx.): 13631283929 # # Overhead Command Shared Object Symbol # ........ ....... ...................... ...................................... # 18.59% iperf3 [kernel.kallsyms] [k] csum_partial 14.75% iperf3 [kernel.kallsyms] [k] __copy_to_user_std 8.05% iperf3 [kernel.kallsyms] [k] v7_dma_inv_range 7.07% iperf3 [kernel.kallsyms] [k] __irqentry_text_end 5.42% iperf3 [kernel.kallsyms] [k] l2c210_inv_range 2.43% iperf3 [kernel.kallsyms] [k] bcma_host_soc_read32 1.83% iperf3 [kernel.kallsyms] [k] tcp_rcv_established 1.72% iperf3 [kernel.kallsyms] [k] bgmac_poll 1.38% iperf3 [kernel.kallsyms] [k] copy_page_to_iter 1.38% iperf3 [kernel.kallsyms] [k] page_address 1.31% iperf3 [kernel.kallsyms] [k] skb_try_coalesce 1.21% iperf3 [kernel.kallsyms] [k] net_rx_action 1.07% iperf3 [kernel.kallsyms] [k] netdev_alloc_frag 1.05% iperf3 [ip_tables] [k] ipt_do_table 0.91% iperf3 [kernel.kallsyms] [k] tcp_v4_do_rcv 0.90% iperf3 [kernel.kallsyms] [k] enqueue_to_backlog 0.80% iperf3 [kernel.kallsyms] [k] dev_gro_receive 0.70% iperf3 [kernel.kallsyms] [k] build_skb 0.68% iperf3 [kernel.kallsyms] [k] dma_cache_maint_page 0.61% iperf3 [kernel.kallsyms] [k] __free_pages_ok 0.59% iperf3 [kernel.kallsyms] [k] __skb_get_hash 0.54% iperf3 [kernel.kallsyms] [k] ___bpf_prog_run 0.51% iperf3 [kernel.kallsyms] [k] mmioset 0.49% iperf3 [kernel.kallsyms] [k] __skb_flow_dissect 0.48% iperf3 [kernel.kallsyms] [k] kmem_cache_alloc 0.47% iperf3 [kernel.kallsyms] [k] __slab_free.constprop.2 0.43% iperf3 [kernel.kallsyms] [k] ktime_get_with_offset 0.40% iperf3 [kernel.kallsyms] [k] copyout 0.40% iperf3 [kernel.kallsyms] [k] tcp_recvmsg 0.39% iperf3 [kernel.kallsyms] [k] generic_exec_single 0.38% iperf3 [kernel.kallsyms] [k] sched_clock 0.38% iperf3 [kernel.kallsyms] [k] bcma_host_soc_write32 0.38% iperf3 [kernel.kallsyms] [k] sch_direct_xmit 0.36% iperf3 [kernel.kallsyms] [k] bgmac_dma_rx_skb_for_slot 0.36% iperf3 [kernel.kallsyms] [k] skb_release_head_state 0.36% iperf3 [kernel.kallsyms] [k] skb_copy_datagram_iter 0.35% iperf3 [kernel.kallsyms] [k] kmem_cache_free 0.35% iperf3 [kernel.kallsyms] [k] __netif_receive_skb_core 0.34% iperf3 [kernel.kallsyms] [k] __tcp_ack_snd_check 0.33% iperf3 [kernel.kallsyms] [k] netif_receive_skb_internal 0.33% iperf3 [kernel.kallsyms] [k] arm_dma_unmap_page 0.31% iperf3 [kernel.kallsyms] [k] skb_release_data 0.31% iperf3 [kernel.kallsyms] [k] tcp_event_data_recv 0.31% iperf3 [kernel.kallsyms] [k] skb_put 0.30% iperf3 [kernel.kallsyms] [k] eth_type_trans 0.30% iperf3 [kernel.kallsyms] [k] __release_sock 0.29% iperf3 [kernel.kallsyms] [k] get_rps_cpu 0.29% iperf3 [kernel.kallsyms] [k] tcp_v4_rcv 0.28% iperf3 [nf_conntrack] [k] _429 0.27% iperf3 [kernel.kallsyms] [k] __build_skb 0.27% iperf3 [kernel.kallsyms] [k] tcp_try_coalesce.part.9 0.27% iperf3 [kernel.kallsyms] [k] __dev_queue_xmit 0.25% iperf3 [kernel.kallsyms] [k] __skb_checksum 0.25% iperf3 [xt_conntrack] [k] _15 0.24% iperf3 [nf_conntrack] [k] _254 0.24% iperf3 [kernel.kallsyms] [k] __skb_checksum_complete 0.23% iperf3 [nf_conntrack] [k] nf_conntrack_in 0.23% iperf3 [kernel.kallsyms] [k] gt_sched_clock_read 0.22% iperf3 [kernel.kallsyms] [k] napi_gro_receive 0.22% iperf3 [kernel.kallsyms] [k] arm_dma_map_page 0.22% iperf3 [kernel.kallsyms] [k] tcp_queue_rcv 0.22% iperf3 [kernel.kallsyms] [k] tcp_send_delayed_ack 0.22% iperf3 [kernel.kallsyms] [k] bcma_bgmac_read 0.22% iperf3 [kernel.kallsyms] [k] nf_hook_slow 0.22% iperf3 [kernel.kallsyms] [k] __tcp_select_window 0.21% iperf3 [kernel.kallsyms] [k] __dma_page_dev_to_cpu 0.21% iperf3 [kernel.kallsyms] [k] gic_raise_softirq 0.21% iperf3 [kernel.kallsyms] [k] ipv4_dst_check 0.20% iperf3 [kernel.kallsyms] [k] __local_bh_enable_ip 0.20% iperf3 [kernel.kallsyms] [k] __dma_page_cpu_to_dev 0.19% iperf3 [kernel.kallsyms] [k] smp_cross_call 0.18% iperf3 [kernel.kallsyms] [k] bcma_bgmac_write 0.18% iperf3 [kernel.kallsyms] [k] process_backlog 0.18% iperf3 [kernel.kallsyms] [k] smp_call_function_single_async 0.18% iperf3 [kernel.kallsyms] [k] __kunmap_atomic 0.18% iperf3 [kernel.kallsyms] [k] _copy_to_iter 0.16% iperf3 [kernel.kallsyms] [k] dsa_switch_rcv 0.15% iperf3 [kernel.kallsyms] [k] arm_heavy_mb 0.15% iperf3 [kernel.kallsyms] [k] tcp_rcv_space_adjust 0.15% iperf3 [kernel.kallsyms] [k] tcp_poll 0.14% iperf3 [kernel.kallsyms] [k] do_select 0.13% iperf3 [nf_nat] [k] nf_nat_inet_fn 0.13% iperf3 [kernel.kallsyms] [k] __tcp_transmit_skb 0.13% iperf3 [kernel.kallsyms] [k] skb_gro_reset_offset 0.13% iperf3 [kernel.kallsyms] [k] net_rps_send_ipi 0.13% iperf3 [kernel.kallsyms] [k] skb_checksum 0.12% iperf3 [kernel.kallsyms] [k] __kfree_skb_flush 0.12% iperf3 [kernel.kallsyms] [k] kmap_atomic 0.12% iperf3 [kernel.kallsyms] [k] napi_complete_done 0.12% iperf3 [kernel.kallsyms] [k] sock_def_readable 0.12% iperf3 [kernel.kallsyms] [k] csum_partial_ext 0.12% iperf3 [kernel.kallsyms] [k] skb_pull 0.12% iperf3 [kernel.kallsyms] [k] sock_poll 0.12% iperf3 [kernel.kallsyms] [k] tcp_grow_window 0.11% iperf3 [kernel.kallsyms] [k] arch_send_call_function_single_ipi 0.11% iperf3 [nf_conntrack] [k] _422 0.11% iperf3 [kernel.kallsyms] [k] mod_timer 0.10% iperf3 [kernel.kallsyms] [k] rcu_bh_qs 0.10% iperf3 [kernel.kallsyms] [k] _cond_resched 0.10% iperf3 [kernel.kallsyms] [k] gt_clocksource_read 0.10% iperf3 [kernel.kallsyms] [k] __vfs_read 0.10% iperf3 [kernel.kallsyms] [k] cpumask_next 0.10% iperf3 [iptable_mangle] [k] _10 0.09% iperf3 [kernel.kallsyms] [k] rcu_all_qs 0.09% iperf3 [nf_nat_ipv4] [k] _39 0.09% iperf3 [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore 0.09% iperf3 [nf_defrag_ipv4] [k] _13 0.09% iperf3 [kernel.kallsyms] [k] ktime_get_ts64 0.09% iperf3 [kernel.kallsyms] [k] l2c210_sync 0.09% iperf3 [kernel.kallsyms] [k] arm_dma_mapping_error 0.09% iperf3 [nf_conntrack] [k] _389 0.09% iperf3 [kernel.kallsyms] [k] bgmac_start_xmit 0.09% iperf3 [kernel.kallsyms] [k] __kmalloc_track_caller 0.09% iperf3 [kernel.kallsyms] [k] dev_hard_start_xmit 0.09% iperf3 [kernel.kallsyms] [k] validate_xmit_skb.constprop.28 0.08% iperf3 [kernel.kallsyms] [k] ip_rcv_core.constprop.2 0.08% iperf3 [nf_conntrack] [k] _308 0.08% iperf3 [kernel.kallsyms] [k] dst_release 0.08% iperf3 [kernel.kallsyms] [k] ip_finish_output2 0.08% iperf3 [kernel.kallsyms] [k] ip_rcv_finish_core.constprop.3 0.08% iperf3 [kernel.kallsyms] [k] __inet_lookup_established 0.08% iperf3 [kernel.kallsyms] [k] tcp_v4_early_demux 0.08% iperf3 [kernel.kallsyms] [k] _raw_spin_lock_bh 0.08% iperf3 [kernel.kallsyms] [k] core_sys_select 0.08% iperf3 [nf_conntrack] [k] __nf_ct_refresh_acct 0.08% iperf3 [kernel.kallsyms] [k] page_frag_free 0.07% iperf3 [nf_conntrack] [k] _337 0.07% iperf3 [kernel.kallsyms] [k] __ip_queue_xmit 0.07% iperf3 [kernel.kallsyms] [k] should_failslab 0.07% iperf3 [xt_conntrack] [k] _23 0.07% iperf3 [kernel.kallsyms] [k] vector_swi 0.07% iperf3 [kernel.kallsyms] [k] put_cpu_partial 0.07% iperf3 [kernel.kallsyms] [k] _raw_spin_lock 0.07% iperf3 [nf_conntrack] [k] _306 0.07% iperf3 [kernel.kallsyms] [k] __qdisc_run 0.07% iperf3 [nf_conntrack] [k] nf_ct_deliver_cached_events 0.07% iperf3 [kernel.kallsyms] [k] __put_user_1 0.07% iperf3 [kernel.kallsyms] [k] netif_skb_features 0.07% iperf3 [kernel.kallsyms] [k] ip_local_deliver_finish 0.07% iperf3 [kernel.kallsyms] [k] __do_div64 0.06% iperf3 [kernel.kallsyms] [k] __usecs_to_jiffies 0.06% iperf3 [kernel.kallsyms] [k] brcm_tag_rcv_ll.constprop.0 0.06% iperf3 [kernel.kallsyms] [k] refcount_inc_not_zero_checked 0.06% iperf3 [kernel.kallsyms] [k] vfs_read 0.06% iperf3 [kernel.kallsyms] [k] ip_local_deliver 0.06% iperf3 [kernel.kallsyms] [k] kfree_skb_partial 0.06% iperf3 [kernel.kallsyms] [k] fq_codel_dequeue 0.06% iperf3 [kernel.kallsyms] [k] _raw_spin_unlock_bh 0.06% iperf3 [kernel.kallsyms] [k] memmove 0.06% iperf3 [kernel.kallsyms] [k] poll_select_copy_remaining 0.06% iperf3 [kernel.kallsyms] [k] __fget_light 0.06% iperf3 [kernel.kallsyms] [k] fsnotify 0.05% iperf3 [kernel.kallsyms] [k] refcount_sub_and_test_checked 0.05% iperf3 [nf_nat] [k] nf_nat_packet 0.05% iperf3 [kernel.kallsyms] [k] llist_add_batch 0.05% iperf3 [kernel.kallsyms] [k] __bpf_prog_run32 0.05% iperf3 [kernel.kallsyms] [k] do_softirq.part.2 0.05% iperf3 [kernel.kallsyms] [k] div_s64_rem 0.05% iperf3 [kernel.kallsyms] [k] ns_to_timespec 0.05% iperf3 [nf_nat_ipv4] [k] _40 0.05% iperf3 [xt_comment] [k] _9 0.05% iperf3 [kernel.kallsyms] [k] _find_next_bit_le 0.05% iperf3 libc.so [.] 0x00056c70 0.05% iperf3 [kernel.kallsyms] [k] mmiocpy 0.05% iperf3 [kernel.kallsyms] [k] inet_ehashfn.constprop.4 0.05% iperf3 [kernel.kallsyms] [k] tcp_v4_fill_cb 0.05% iperf3 [kernel.kallsyms] [k] __netif_receive_skb 0.05% iperf3 [kernel.kallsyms] [k] ip_finish_output 0.05% iperf3 [kernel.kallsyms] [k] b15_dma_unmap_area 0.05% iperf3 [kernel.kallsyms] [k] fput 0.05% iperf3 [kernel.kallsyms] [k] tcp_cleanup_rbuf 0.05% iperf3 [kernel.kallsyms] [k] skb_network_protocol 0.05% iperf3 [kernel.kallsyms] [k] skb_checksum_help 0.04% iperf3 [kernel.kallsyms] [k] brcm_tag_rcv 0.04% iperf3 [iptable_filter] [k] _15 0.04% iperf3 [nf_conntrack] [k] _341 0.04% iperf3 [kernel.kallsyms] [k] _find_first_bit_le 0.04% iperf3 [kernel.kallsyms] [k] kfree 0.04% iperf3 [kernel.kallsyms] [k] sk_reset_timer 0.04% iperf3 [kernel.kallsyms] [k] raw_local_deliver 0.04% iperf3 [kernel.kallsyms] [k] tcp_add_backlog 0.04% iperf3 [kernel.kallsyms] [k] ip_rcv 0.04% iperf3 [kernel.kallsyms] [k] 0x004ef5c0 0.04% iperf3 [kernel.kallsyms] [k] __fsnotify_parent 0.04% iperf3 [kernel.kallsyms] [k] skb_push 0.04% iperf3 [kernel.kallsyms] [k] __tcp_send_ack.part.6 0.04% iperf3 [kernel.kallsyms] [k] skb_pull_rcsum 0.04% iperf3 [kernel.kallsyms] [k] put_timespec64 0.04% iperf3 [kernel.kallsyms] [k] sys_clock_gettime 0.04% iperf3 libc.so [.] 0x0005c194 0.04% iperf3 [kernel.kallsyms] [k] ktime_get_real_ts64 0.04% iperf3 [kernel.kallsyms] [k] inet_recvmsg 0.04% iperf3 [kernel.kallsyms] [k] tcp_options_write 0.04% iperf3 libc.so [.] 0x00057850 0.04% iperf3 [kernel.kallsyms] [k] tcp_filter 0.04% iperf3 [kernel.kallsyms] [k] __alloc_skb 0.04% iperf3 [kernel.kallsyms] [k] skb_free_head 0.04% iperf3 [kernel.kallsyms] [k] arm_copy_from_user 0.04% iperf3 [iptable_raw] [k] _14 0.04% iperf3 [kernel.kallsyms] [k] ipv4_mtu 0.04% iperf3 [kernel.kallsyms] [k] ksys_read 0.03% iperf3 [kernel.kallsyms] [k] __netif_receive_skb_one_core 0.03% iperf3 [kernel.kallsyms] [k] netif_receive_skb 0.03% iperf3 [kernel.kallsyms] [k] rw_verify_area 0.03% iperf3 [kernel.kallsyms] [k] release_sock 0.03% iperf3 [kernel.kallsyms] [k] ksize 0.03% iperf3 [kernel.kallsyms] [k] __pollwait 0.03% iperf3 [kernel.kallsyms] [k] dql_completed 0.03% iperf3 iperf3 [.] 0x00009c30 0.03% iperf3 [kernel.kallsyms] [k] _test_and_set_bit 0.03% iperf3 [kernel.kallsyms] [k] sock_rfree 0.03% iperf3 [kernel.kallsyms] [k] posix_clock_realtime_get 0.03% iperf3 [nf_conntrack] [k] _339 0.03% iperf3 [nf_conntrack] [k] __nf_ct_l4proto_find 0.03% iperf3 [kernel.kallsyms] [k] b15_dma_map_area 0.03% iperf3 [kernel.kallsyms] [k] timespec64_add_safe 0.03% iperf3 [kernel.kallsyms] [k] __sk_dst_check 0.03% iperf3 [kernel.kallsyms] [k] sk_filter_trim_cap 0.03% iperf3 [kernel.kallsyms] [k] ip_copy_addrs 0.03% iperf3 [kernel.kallsyms] [k] dsa_slave_xmit 0.03% iperf3 libc.so [.] 0x0005e87c 0.03% iperf3 [kernel.kallsyms] [k] __kfree_skb 0.03% iperf3 [kernel.kallsyms] [k] kfree_skbmem 0.03% iperf3 [kernel.kallsyms] [k] lock_sock_nested 0.03% iperf3 libc.so [.] 0x00052f68 0.03% iperf3 [kernel.kallsyms] [k] sys_select 0.02% iperf3 [kernel.kallsyms] [k] ns_to_timespec.part.0 0.02% iperf3 [kernel.kallsyms] [k] refcount_dec_and_test_checked 0.02% iperf3 [nf_conntrack_rtcache] [k] _21 0.02% iperf3 [kernel.kallsyms] [k] ktime_get_seconds 0.02% iperf3 [kernel.kallsyms] [k] skb_release_all 0.02% iperf3 libc.so [.] 0x0001d0c8 0.02% iperf3 [kernel.kallsyms] [k] sock_read_iter 0.02% iperf3 [kernel.kallsyms] [k] ip_rcv_finish 0.02% iperf3 [kernel.kallsyms] [k] ptp_classify_raw 0.02% iperf3 [kernel.kallsyms] [k] add_wait_queue 0.02% iperf3 [kernel.kallsyms] [k] ip_local_out 0.02% iperf3 [kernel.kallsyms] [k] iov_iter_init 0.02% iperf3 [kernel.kallsyms] [k] ip_output 0.02% iperf3 [kernel.kallsyms] [k] finish_task_switch 0.02% iperf3 [kernel.kallsyms] [k] l2c210_clean_range 0.02% iperf3 [kernel.kallsyms] [k] skb_condense 0.02% iperf3 [nf_nat_ipv4] [k] _42 0.02% iperf3 [kernel.kallsyms] [k] consume_skb 0.02% iperf3 libgcc_s.so.1 [.] 0x00006164 0.02% iperf3 [kernel.kallsyms] [k] skb_csum_hwoffload_help 0.02% iperf3 iperf3 [.] 0x0000de9c 0.02% iperf3 libc.so [.] 0x0005bef4 0.02% iperf3 [kernel.kallsyms] [k] __aeabi_idiv 0.02% iperf3 [kernel.kallsyms] [k] brcm_tag_xmit_ll 0.02% iperf3 libgcc_s.so.1 [.] 0x00008658 0.02% iperf3 [kernel.kallsyms] [k] v7_dma_clean_range 0.02% iperf3 [kernel.kallsyms] [k] 0x004f0294 0.02% iperf3 libc.so [.] 0x0005beb8 0.02% iperf3 [kernel.kallsyms] [k] _raw_spin_lock_irqsave 0.02% iperf3 [kernel.kallsyms] [k] netdev_pick_tx 0.02% iperf3 iperf3 [.] 0x0000e04c 0.02% iperf3 iperf3 [.] 0x00009bb8 0.02% iperf3 [kernel.kallsyms] [k] select_estimate_accuracy 0.02% iperf3 [kernel.kallsyms] [k] tcp_v4_send_check 0.02% iperf3 [kernel.kallsyms] [k] kern_select 0.02% iperf3 iperf3 [.] 0x00003f8c 0.01% iperf3 [kernel.kallsyms] [k] set_normalized_timespec64 0.01% iperf3 [kernel.kallsyms] [k] __entry_text_start 0.01% iperf3 [kernel.kallsyms] [k] refcount_add_checked 0.01% iperf3 [nf_conntrack] [k] _338 0.01% iperf3 [kernel.kallsyms] [k] run_timer_softirq 0.01% iperf3 [kernel.kallsyms] [k] _test_and_clear_bit 0.01% iperf3 libc.so [.] 0x0005bf04 0.01% iperf3 [kernel.kallsyms] [k] 0x004ef5dc 0.01% iperf3 libc.so [.] 0x00056c48 0.01% iperf3 [kernel.kallsyms] [k] brcm_tag_xmit 0.01% iperf3 [kernel.kallsyms] [k] __page_cache_release 0.01% iperf3 libc.so [.] 0x00052ffc 0.01% iperf3 iperf3 [.] 0x0000ad1c 0.01% iperf3 [kernel.kallsyms] [k] 0x004ef5e0 0.01% iperf3 libc.so [.] 0x00052ff8 0.01% iperf3 [kernel.kallsyms] [k] __kmalloc_reserve.constprop.22 0.01% iperf3 libc.so [.] 0x00057824 0.01% iperf3 [kernel.kallsyms] [k] __ip_local_out 0.01% iperf3 [kernel.kallsyms] [k] 0x004ef5e8 0.01% iperf3 libc.so [.] 0x00056864 0.01% iperf3 [kernel.kallsyms] [k] ip_queue_xmit 0.01% iperf3 iperf3 [.] 0x00001aec 0.01% iperf3 [kernel.kallsyms] [k] 0x004f02a0 0.01% iperf3 [kernel.kallsyms] [k] ip_send_check 0.01% iperf3 libc.so [.] 0x0005e8a4 0.01% iperf3 [kernel.kallsyms] [k] __fdget_pos 0.01% iperf3 iperf3 [.] 0x00001d50 0.01% iperf3 [kernel.kallsyms] [k] poll_select_set_timeout 0.01% iperf3 [kernel.kallsyms] [k] __fdget 0.01% iperf3 [kernel.kallsyms] [k] tcp_release_cb 0.01% iperf3 libc.so [.] 0x00049980 0.01% iperf3 [kernel.kallsyms] [k] tcp_sendmsg_locked 0.01% iperf3 [kernel.kallsyms] [k] dev_queue_xmit 0.01% iperf3 [kernel.kallsyms] [k] kmalloc_slab 0.01% iperf3 [kernel.kallsyms] [k] 0x004ef5f0 0.01% iperf3 iperf3 [.] 0x0000ad40 0.01% iperf3 [kernel.kallsyms] [k] refcount_add_not_zero_checked 0.01% iperf3 iperf3 [.] 0x00009bf4 0.01% iperf3 libgcc_s.so.1 [.] 0x000043c4 0.01% iperf3 iperf3 [.] 0x0000df48 0.01% iperf3 iperf3 [.] 0x0000e070 0.01% iperf3 iperf3 [.] 0x00003fd4 0.01% iperf3 iperf3 [.] 0x0000ad60 0.01% iperf3 libgcc_s.so.1 [.] 0x00008660 0.01% iperf3 [kernel.kallsyms] [k] sys_read 0.01% iperf3 [kernel.kallsyms] [k] poll_freewait 0.01% iperf3 iperf3 [.] 0x00009b58 0.01% iperf3 iperf3 [.] 0x00003f54 0.01% iperf3 [kernel.kallsyms] [k] tcp_stream_memory_free 0.01% iperf3 [kernel.kallsyms] [k] find_busiest_group 0.01% iperf3 [kernel.kallsyms] [k] validate_xmit_skb_list 0.01% iperf3 libc.so [.] 0x0005bec8 0.01% iperf3 libc.so [.] 0x00053004 0.01% iperf3 [kernel.kallsyms] [k] free_slab 0.01% iperf3 [kernel.kallsyms] [k] free_unref_page 0.01% iperf3 [kernel.kallsyms] [k] 0x004f02b0 0.01% iperf3 libc.so [.] 0x000577e4 0.01% iperf3 libc.so [.] 0x00053000 0.01% iperf3 iperf3 [.] 0x00009acc 0.01% iperf3 libc.so [.] 0x00057800 0.01% iperf3 [kernel.kallsyms] [k] free_compound_page 0.01% iperf3 [kernel.kallsyms] [k] 0x004ef5f8 0.01% iperf3 libgcc_s.so.1 [.] 0x0000876c 0.01% iperf3 libc.so [.] 0x00053008 0.01% iperf3 iperf3 [.] 0x00009a20 0.01% iperf3 iperf3 [.] 0x0000ad58 0.01% iperf3 iperf3 [.] 0x0000d9b0 0.01% iperf3 iperf3 [.] 0x0000ad20 0.00% iperf3 [kernel.kallsyms] [k] __put_page 0.00% iperf3 iperf3 [.] 0x00009a10 0.00% iperf3 libc.so [.] 0x0005782c 0.00% iperf3 iperf3 [.] 0x00009a58 0.00% iperf3 iperf3 [.] 0x00009bc4 0.00% iperf3 [kernel.kallsyms] [k] tcp_send_ack 0.00% iperf3 libc.so [.] 0x0003c69c 0.00% iperf3 [nf_nat_ipv4] [k] _46 0.00% iperf3 libc.so [.] 0x0003bcec 0.00% iperf3 libc.so [.] 0x00052fb0 0.00% iperf3 libc.so [.] 0x0005bec0 0.00% iperf3 iperf3 [.] 0x0000d99c 0.00% iperf3 iperf3 [.] 0x00009bd8 0.00% iperf3 iperf3 [.] 0x00003f60 0.00% iperf3 iperf3 [.] 0x00003f3c 0.00% iperf3 [kernel.kallsyms] [k] load_balance 0.00% iperf3 [kernel.kallsyms] [k] mod_node_page_state 0.00% iperf3 iperf3 [.] 0x0000acfc 0.00% iperf3 iperf3 [.] 0x0000e068 0.00% iperf3 iperf3 [.] 0x00009ab0 0.00% iperf3 iperf3 [.] 0x0000df80 0.00% iperf3 iperf3 [.] 0x0000dea0 0.00% iperf3 libc.so [.] 0x00052f80 0.00% iperf3 [kernel.kallsyms] [k] lock_timer_base 0.00% iperf3 iperf3 [.] 0x0000d9f4 0.00% iperf3 iperf3 [.] 0x00009c38 0.00% iperf3 [kernel.kallsyms] [k] __free_slab 0.00% iperf3 libc.so [.] 0x00057808 0.00% iperf3 iperf3 [.] 0x00001cb4 0.00% iperf3 libc.so [.] 0x0005bf00 0.00% iperf3 libc.so [.] 0x00052fe0 0.00% iperf3 [kernel.kallsyms] [k] free_pcp_prepare 0.00% iperf3 [kernel.kallsyms] [k] tcp_ack 0.00% iperf3 libc.so [.] 0x000499a8 0.00% iperf3 [kernel.kallsyms] [k] tcp_try_rmem_schedule 0.00% iperf3 [kernel.kallsyms] [k] preempt_schedule_common 0.00% iperf3 libc.so [.] 0x00056c64 0.00% iperf3 [kernel.kallsyms] [k] cpu_v7_bpiall_set_pte_ext 0.00% iperf3 iperf3 [.] 0x00009974 0.00% iperf3 iperf3 [.] 0x0000e080 0.00% iperf3 [xt_tcpudp] [k] _13 0.00% iperf3 iperf3 [.] 0x0000ad50 0.00% iperf3 iperf3 [.] 0x0000ebc4 0.00% iperf3 libc.so [.] 0x00021094 0.00% iperf3 iperf3 [.] 0x00009be0 0.00% iperf3 iperf3 [.] 0x00009af4 0.00% iperf3 [kernel.kallsyms] [k] do_tcp_getsockopt.constprop.9 0.00% iperf3 libc.so [.] 0x0005c1ac 0.00% iperf3 iperf3 [.] 0x00001b78 0.00% iperf3 [kernel.kallsyms] [k] __sock_wfree 0.00% iperf3 iperf3 [.] 0x00001b58 0.00% iperf3 iperf3 [.] 0x00003f90 0.00% iperf3 [kernel.kallsyms] [k] percpu_counter_add_batch 0.00% perf [kernel.kallsyms] [k] perf_event_exec

@openwrt-bot
Copy link
Author

ffainelli:

Yes so this comes down to bgmac not having receive checksum offload enabled, which means you spend a lot of time computing the packet's checksum in SW. I believe that FA/CTF might be able to help there since it should have checksum capabilities, how to support it properly is still a bit of mystery for me (it is a separate intermediate network device? an add-on to the bgmac driver?)

@openwrt-bot
Copy link
Author

npcomplete:

Florian,

Thanks for the explanation. Will this issue persist even when we have a NATP accelerator implementation in place? Or - are the FA/CTF/NATP accelerator all one and the same?

@LGA1150 I'll try the software offload option.

Thanks,

Vivek

@openwrt-bot
Copy link
Author

ffainelli:

AFAIR, the FA/NAPT should solve the RX/TX checksum problem and would speed up NAT in HW anyway.

@openwrt-bot
Copy link
Author

npcomplete:

Thank you for clarifying.

@openwrt-bot
Copy link
Author

npcomplete:

Florian,

After reading through the GPL source, it seems that FA is not enabled for this router (i.e it maybe used in bypass mode) when GMAC3 is enabled.

I found this declaration:

#if defined(BCM_GMAC3)
//...
#if defined(ETFA)
#error "GMAC3 based forwarding not compatible with ETFA - AuX port"
#endif /* ETFA /
//..
#endif /
BCM_GMAC3 */

The decompressed factory firmware only contains ctf.ko but no FA related modules. Also, the ctf_fa_mode nvram parameter is missing for this router.

Thanks,

@openwrt-bot
Copy link
Author

a_guy:

Hello team.

Any updates on this? This bug affects all Broadcom-based routers from bcm53xx target (at least) and makes OpenWrt unusable for 500Mbps+ connections.

Example: https://forum.openwrt.org/t/nighthawk-r8000-throughput-speed-issues/66740/3

I'm ready to perform necessary tests if needed.
Thanks.

@aparcar aparcar added the kernel pull request/issue with Linux kernel related changes label Feb 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flyspray kernel pull request/issue with Linux kernel related changes
Projects
None yet
Development

No branches or pull requests

2 participants