[ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow

Shahaf Shuler shahafs at mellanox.com
Mon May 21 14:25:56 UTC 2018


Hi Flavio, thanks for reporting this issue.

Saturday, May 19, 2018 1:47 AM, Flavio Leitner:
> Subject: Re: [ovs-dev] [PATCH v9 0/7] OVS-DPDK flow offload with rte_flow
> 
> 
> 
> Hello,
> 
> I looked at the patchset (v9) and I found no obvious problems, but I miss
> some instrumentation to understand what is going on. For example, how
> many flows are offloaded, or how many per second, etc... We can definitely
> work on that as a follow up.
> 
> I have a MLX5 (16.20.1010) which is connected to a traffic generator.
> The results are unexpected and I don't know why yet. I will continue on it
> next week.
> 
> The flow is pretty simple, just echo the packet back:
> 
>    ovs-ofctl add-flow ovsbr0 in_port=10,action=output:in_port
> 
> This is the result without HW offloading enabled:
> Partial:  14619675.00 pps  7485273570.00 bps
> Partial:  14652351.00 pps  7502003940.00 bps
> Partial:  14655548.00 pps  7503640570.00 bps
> Partial:  14679556.00 pps  7515932630.00 bps
> Partial:  14681188.00 pps  7516768670.00 bps
> Partial:  14597427.00 pps  7473882390.00 bps
> Partial:  14712617.00 pps  7532860090.00 bps
> 
> pmd thread numa_id 0 core_id 2:
>         packets received: 53859055
>         packet recirculations: 0
>         avg. datapath passes per packet: 1.00
>         emc hits: 53859055
>         megaflow hits: 0
>         avg. subtable lookups per megaflow hit: 0.00
>         miss with success upcall: 0
>         miss with failed upcall: 0
>         avg. packets per output batch: 28.20
>         idle cycles: 0 (0.00%)
>         processing cycles: 12499399115 (100.00%)
>         avg cycles per packet: 232.08 (12499399115/53859055)
>         avg processing cycles per packet: 232.08 (12499399115/53859055)
> 
> 
> Based on the stats, it seems 14.7Mpps is the maximum a core can do it.
> 
> This is the result with HW Offloading enabled:
> 
> Partial:  10713500.00 pps  5485312330.00 bps
> Partial:  10672185.00 pps  5464158240.00 bps
> Partial:  10747756.00 pps  5502850960.00 bps
> Partial:  10713404.00 pps  5485267400.00 bps
> 
> 
> pmd thread numa_id 0 core_id 2:
>         packets received: 25902718
>         packet recirculations: 0
>         avg. datapath passes per packet: 1.00
>         emc hits: 25902697
>         megaflow hits: 0
>         avg. subtable lookups per megaflow hit: 0.00
>         miss with success upcall: 0
>         miss with failed upcall: 0
>         avg. packets per output batch: 28.11
>         idle cycles: 0 (0.00%)
>         processing cycles: 12138284463 (100.00%)
>         avg cycles per packet: 468.61 (12138284463/25902718)
>         avg processing cycles per packet: 468.61 (12138284463/25902718)
> 
> 2018-05-
> 18T22:34:57.865Z|00001|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> id for ufid caaf720e-5dfe-4879-adb9-155bd92f9b    40 was not found
> 
> 2018-05-
> 18T22:35:02.920Z|00002|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> id for ufid c75ae6c5-1d14-40ce-b4c7-6d5001a458    4c was not found
> 

Up till now makes sense. The first packet should trigger the flow insertion to HW.

> 2018-05-18T22:35:05.160Z|00105|memory|INFO|109700 kB peak resident set
> size after 10.4 seconds
> 
> 2018-05-18T22:35:05.160Z|00106|memory|INFO|handlers:1 ports:3
> revalidators:1 rules:5 udpif keys:2
> 
> 2018-05-
> 18T22:35:21.910Z|00003|dpif_netdev(dp_netdev_flow_8)|WARN|Mark
> id for ufid 73bdddc9-b12f-4007-9f12-b66b4bc189    3e was not found
> 
> 2018-05-18T22:35:21.924Z|00004|netdev_dpdk(dp_netdev_flow_8)|ERR|rte
> flow creat error: 2 : message : flow rule creati    on failure

This error is unexpected. 
I was able to reproduce the issue with latest 17.11 stable. Looks like either modify flow is broken on the DPDK side or OVS-DPDK issue which popped up due to some commit on stable DPDK. 
I will investigate more on it and provide a fix. 

For the sake of testing, can you try with 17.11 instead of stable? 
With it I found it work correctly:

2018-05-21T14:20:32.954Z|00120|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.9.90                                         │
2018-05-21T14:20:33.906Z|00121|connmgr|INFO|br0<->unix#3: 1 flow_mods in the last 0 s (1 adds)                        │
2018-05-21T14:20:33.923Z|00002|dpif_netdev(dp_netdev_flow_4)|DBG|succeed to modify netdev flow      

BTW - if you want to avoid the modify flow, try to insert the rule before you start the traffic on the traffic gen. 
The below output is the expected (in both LTS and 17.11 versions):


2018-05-21T13:19:18.055Z|00120|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.9.90                                         
2018-05-21T13:19:23.560Z|00121|memory|INFO|294204 kB peak resident set size after 10.4 seconds                        
2018-05-21T13:19:23.560Z|00122|memory|INFO|handlers:17 ports:2 revalidators:7 rules:5                                 
2018-05-21T13:19:25.726Z|00123|connmgr|INFO|br0<->unix#3: 1 flow_mods in the last 0 s (1 adds)                        
2018-05-21T13:19:38.378Z|00001|dpif_netdev(pmd30)|DBG|ovs-netdev: miss upcall:                                        
skb_priority(0),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),recirc_id(0),dp_hash(0),in_port(1),packet_ty
pe(ns=0,id=0),eth(src=24:8a:07:91:8d:68,dst=02:00:00:00:00:00),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.0.2,p
roto=17,tos=0,ttl=64,frag=no),udp(src=1024,dst=1024)                                                                  
udp,vlan_tci=0x0000,dl_src=24:8a:07:91:8d:68,dl_dst=02:00:00:00:00:00,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,n
w_ecn=0,nw_ttl=64,tp_src=1024,tp_dst=1024 udp_csum:0                                                                  
2018-05-21T13:19:38.379Z|00002|dpif_netdev(pmd30)|DBG|Creating dpcls 0x7f29fc005310 for in_port 1                     
2018-05-21T13:19:38.379Z|00003|dpif_netdev(pmd30)|DBG|Creating 1. subtable 0x7f29fc006150 for in_port 1               
2018-05-21T13:19:38.379Z|00001|dpif_netdev(dp_netdev_flow_4)|WARN|Mark id for ufid 3bad389c-507e-4b02-a1cf-c0e084703d5
f was not found                                                                                                       
2018-05-21T13:19:38.379Z|00004|dpif_netdev(pmd30)|DBG|flow_add: ufid:dd9f6d14-f085-4706-a1ac-19d458109a79 recirc_id(0)
,in_port(1),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), actions:1                                          
2018-05-21T13:19:38.379Z|00005|dpif_netdev(pmd30)|DBG|flow match: recirc_id=0,eth,ip,vlan_tci=0x0000,nw_frag=no       
2018-05-21T13:19:38.387Z|00002|dpif_netdev(dp_netdev_flow_4)|DBG|Associated dp_netdev flow 0x7f29fc005a90 with mark 0 
2018-05-21T13:19:38.387Z|00003|dpif_netdev(dp_netdev_flow_4)|DBG|succeed to add netdev flow


> 
> 
> Looks like offloading didn't work and now the tput rate is lower, which is not
> expected either.
> 
> I plan to continue on this and I appreciate if you have any idea of what is
> going on.
> 
> Thanks,
> fbl



More information about the dev mailing list