[ovs-dev] 8021q module is not loaded when adding vlan through ovs (ovs-vsctl)

Michael Shteinbok michaelsh86 at gmail.com
Thu Oct 25 11:34:04 UTC 2018


I'v used ftrace to see which paths we take in the kernel.
I remind you that the packet (in my scenario) looks like the following:
ETH | IP | UDP | VXLAN | L2 | VLAN | IP | TCP | payload

When 8021q is not loaded:
dev_gro_receive <-napi_gro_receive
inet_gro_receive <-dev_gro_receive
udp4_gro_receive <-inet_gro_receive
udp_gro_receive <-udp4_gro_receive
udp4_lib_lookup_skb <-udp_gro_receive
__udp4_lib_lookup <-udp4_lib_lookup_skb
compute_score <-__udp4_lib_lookup
vxlan_gro_receive <-udp_gro_receive
__pskb_pull_tail <-vxlan_gro_receive
skb_copy_bits <-__pskb_pull_tail
eth_gro_receive <-vxlan_gro_receive
__pskb_pull_tail <-eth_gro_receive
skb_copy_bits <-__pskb_pull_tail
gro_find_receive_by_type <-eth_gro_receive
netif_receive_skb_internal <-napi_gro_receive
ktime_get_real <-netif_receive_skb_internal
getnstimeofday64 <-ktime_get_real
__getnstimeofday64 <-getnstimeofday64
skb_defer_rx_timestamp <-netif_receive_skb_internal
classify <-skb_defer_rx_timestamp
__netif_receive_skb <-netif_receive_skb_internal
__netif_receive_skb_core <-__netif_receive_skb
ip_rcv <-__netif_receive_skb_core
nf_hook_slow <-ip_rcv
nf_iterate <-nf_hook_slow
ipv4_conntrack_defrag <-nf_iterate
ipv4_conntrack_in <-nf_iterate

When 8021q was loaded manually:


inet_gro_receive <-dev_gro_receive
classify <-skb_clone_tx_timestamp
udp4_gro_receive <-inet_gro_receive
_raw_spin_lock <-sch_direct_xmit
udp_gro_receive <-udp4_gro_receive
udp4_lib_lookup_skb <-udp_gro_receive
local_bh_enable <-__dev_queue_xmit
__udp4_lib_lookup <-udp4_lib_lookup_skb
__local_bh_enable_ip <-local_bh_enable
compute_score <-__udp4_lib_lookup
local_bh_enable <-ip_finish_output
vxlan_gro_receive <-udp_gro_receive
__local_bh_enable_ip <-local_bh_enable
__pskb_pull_tail <-vxlan_gro_receive
skb_copy_bits <-__pskb_pull_tail
local_bh_enable <-__dev_queue_xmit
eth_gro_receive <-vxlan_gro_receive
__local_bh_enable_ip <-local_bh_enable
__pskb_pull_tail <-eth_gro_receive
skb_copy_bits <-__pskb_pull_tail
gro_find_receive_by_type <-eth_gro_receive
local_bh_enable <-__dev_queue_xmit
__local_bh_enable_ip <-local_bh_enable
vlan_gro_receive <-eth_gro_receive
__pskb_pull_tail <-vlan_gro_receive
local_bh_enable <-ip_finish_output
skb_copy_bits <-__pskb_pull_tail
__local_bh_enable_ip <-local_bh_enable
gro_find_receive_by_type <-vlan_gro_receive
inet_gro_receive <-vlan_gro_receive
kfree_skb_partial <-tcp_rcv_established
__pskb_pull_tail <-inet_gro_receive
skb_release_all <-kfree_skb_partial
skb_copy_bits <-__pskb_pull_tail
skb_release_head_state <-skb_release_all
skb_release_data <-skb_release_all
tcp4_gro_receive <-inet_gro_receive
tcp_gro_receive <-tcp4_gro_receive
put_page <-skb_release_data

You can see that eth_gro_receive calls vlan_gro_recieve when 8021q module
is loaded and then performs ding gro_receive for the next layers as well
(inet, tcp), whereas on default scenario where 8021q is not loaded the vlan
device ndo cannot be used.

‫בתאריך יום ה׳, 25 באוק׳ 2018 ב-9:40 מאת ‪Jiri Pirko‬‏ <‪jiri at resnulli.us
‬‏>:‬

> Wed, Oct 24, 2018 at 10:12:54PM CEST, michaelsh86 at gmail.com wrote:
> >Hi,
> >I noticed that there is a performance issue when running traffic on a vlan
> >interface that was created by OVS.
> >If we create a bridge with a vlan interface, the 8021q module is not
> loaded.
> >Then when packets with a 8021q tag arrives, the linux stack can't use the
> >vlan ndos (such as gro_recieve) since there is no such vlan device.
> >If I perform the same test after loading the 8021q module, I get 2x-5x
> >better performance.
>
> Could you please describe why exacly do you think you see this increase?
>
>
> >I personally test that using tunnels (vxlan) but I think the issue exists
> >even without the tunnel.
> >
> >*Creating a bridge and vlan interface:*
> >ovs-vsctl --no-wait add-br br_0
> >ip link set br_0 up
> >ovs-vsctl --no-wait add-port br_0 ens5f0
> >ovs-vsctl --no-wait add-port br_0 vlan10 tag=10 -- set interface vlan10
> >type=internal
> >ip link set vlan10 up
> >ip addr add <ip> dev vlan10
> >
> >run traffic (netperf on the vlan10 interface) and see the result.
> >Then do 'modprobe 8021q' and perform the same test -> you will see much
> >better numbers.
> >
> >I am suggesting to load 80021q module once vlan interface is added through
> >ovs.
> >similarly as a vlan interface would be created using 'ip link'.
> >
> >Michael
>


More information about the dev mailing list