[ovs-discuss] Test with vxlan using ovs-dpdk (2.4)

Praveen MANKARA RADHAKRISHNAN praveen.mankara at 6wind.com
Mon Nov 30 13:35:47 UTC 2015


Hi Sugesh,

I made this scenario work with 2 physical machines.

I am not seeing packets distributed across cores when i send multiple flows
from ixia.
it seems all the packets from a port is handled by only a single core. its
not spreading across the cores.
is there any specific configuration is needed for enabling the packet
spreading across different cores.

given below is the description of my test set up.

2 servers are connected via 1 10Gb interface.
each server 1 ixia port is connected.

from the ixia vlan traffic is coming to dpdk port 0 of server 1. it strips
valn go to a vxlan tunnel through dpdk port 1 to the next server. there
vxla is stripped and through the next going to the ixia port2.

same for reverse direction also.

when i tried the test i have the following observations.
packets are not distributing across the cores. because of this performance
is not scaling.
is there any configuration i should specifically do to have the scaling.

Please find the details about my test.

Configuration:
-----------------------

Server 1
------------

   ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
   ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
   ovs-vsctl add-port br0 vxlan0 -- set interface vxlan0 type=vxlan
options:remote_ip=10.200.0.6

   ovs-vsctl add-br br1 -- set bridge br1 datapath_type=netdev
   ovs-vsctl add-port br1 dpdk1 -- set Interface dpdk1 type=dpdk

   ip link set br0 up
   ip link set br1 up
   ip a a 10.200.0.5/24 dev br1

   ovs-ofctl add-flow br0 in_port=1,actions=strip_vlan,output:2
   ovs-ofctl add-flow br0 in_port=2,actions=output:1

Server 2
------------

   ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
   ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
   ovs-vsctl add-port br0 vxlan0 -- set interface vxlan0 type=vxlan
options:remote_ip=10.200.0.5

   ovs-vsctl add-br br1 -- set bridge br1 datapath_type=netdev
   ovs-vsctl add-port br1 dpdk1 -- set Interface dpdk1 type=dpdk

   ip link set br0 up
   ip link set br1 up
   ip a a 10.200.0.6/24 dev br1

   ovs-ofctl add-flow br0 in_port=1,actions=strip_vlan,output:2
   ovs-ofctl add-flow br0 in_port=2,actions=output:1

1C_2T ~~~~~~~~ ovs-vsctl set Open_vSwitch .
other_config:pmd-cpu-mask=2000002

[root at controller ~]# ovs-appctl dpif-netdev/pmd-stats-show main thread: emc
hits:0 megaflow hits:0 miss:0 lost:0 polling cycles:844386 (100.00%)
processing cycles:0 (0.00%) pmd thread numa_id 1 core_id 1: emc
hits:140438952 megaflow hits:126 miss:2 lost:0 polling cycles:11260105290
(21.84%) processing cycles:40302774189 (78.16%) avg cycles per packet:
367.15 (51562879479/140439080) avg processing cycles per packet: 286.98
(40302774189/140439080) pmd thread numa_id 1 core_id 25: emc hits:148663170
megaflow hits:126 miss:2 lost:0 polling cycles:11918926350 (23.16%)
processing cycles:39554751984 (76.84%) avg cycles per packet: 346.24
(51473678334/148663298) avg processing cycles per packet: 266.07
(39554751984/148663298)

Here traffic is coming from both ixia ports so bothcores are getting
packets.

Changed the threads to 2C_4T :

[root at redhat7 ~]# ovs-vsctl set Open_vSwitch .
other_config:pmd-cpu-mask=a00000a

[root at redhat7 ~]#  ovs-appctl dpif-netdev/pmd-stats-show
main thread:
        emc hits:0
        megaflow hits:0
        miss:0
        lost:0
        polling cycles:1105692 (100.00%)
        processing cycles:0 (0.00%)
pmd thread numa_id 1 core_id 1:
        emc hits:174248214
        megaflow hits:126
        miss:2
        lost:0
        polling cycles:41551018158 (49.30%)
        processing cycles:42726992931 (50.70%)
        avg cycles per packet: 483.67 (84278011089/174248342)
        avg processing cycles per packet: 245.21 (42726992931/174248342)
pmd thread numa_id 1 core_id 3:
        emc hits:182375144
        megaflow hits:126
        miss:2
        lost:0
        polling cycles:39728724567 (46.68%)
        processing cycles:45381262518 (53.32%)
        avg cycles per packet: 466.68 (85109987085/182375272)
        avg processing cycles per packet: 248.83 (45381262518/182375272)
pmd thread numa_id 1 core_id 27:
        emc hits:0
        megaflow hits:0
        miss:0
        lost:0
pmd thread numa_id 1 core_id 25:
        emc hits:0
        megaflow hits:0
        miss:0
        lost:0
[root at redhat7 ~]#

As you can see still only 2 cores were used.



The same is for the other server also.
from the ixia i have sending multiple flows more than 50 flows by changing
the source ip.
the problem is its as if 1 port is bind to the 1 core. its not distributing
the packets.


Thanks
Praveen

On Tue, Nov 3, 2015 at 11:33 AM, Praveen MANKARA RADHAKRISHNAN <
praveen.mankara at 6wind.com> wrote:

> Hi Sugesh,
>
> Thanks for the information.
> by adding the flow in reverse direction OVS learns the arp and the
> scenario works correctly.
>
> Thanks
> Praveen
>
>
> On Wed, Oct 28, 2015 at 5:37 PM, Chandran, Sugesh <
> sugesh.chandran at intel.com> wrote:
>
>> Hi Praveen,
>>
>> Thank you sharing your setup details!
>>
>> It looks like the issue here is ovs doesn’t have the arp entry for remote
>> end point i.e 10.200.0.6.
>>
>> Can you let OVS to learn the mac address for 10.200.0.6?? You can send a
>> arp reply from Ixia port-2 to make it learn this value. Alternatively you
>> can do the same with tnl/arp/set as well, but it looks like to me that the
>> command not yet available on your OVS version.
>>
>> And also please remove the static flow rule that configured lately.
>>
>>
>>
>> You can verify the rules being used in the datapath with
>>
>> ovs-appctl dpctl/dump-flows
>>
>>
>>
>> There must be rules for `tnl_push`  for pushing the vxlan header and also
>> for forward the packet to physical interface.
>>
>>
>>
>> *Regards*
>>
>> *_Sugesh*
>>
>>
>>
>> *From:* Praveen MANKARA RADHAKRISHNAN [mailto:praveen.mankara at 6wind.com]
>> *Sent:* Wednesday, October 28, 2015 11:17 AM
>>
>> *To:* Chandran, Sugesh
>> *Cc:* bugs at openvswitch.org; Daniel Virassamy
>> *Subject:* Re: [ovs-discuss] Test with vxlan using ovs-dpdk (2.4)
>>
>>
>>
>> Hi Sugesh,
>>
>>
>>
>> Thanks. but its still the same.It seems the packets are not getting
>> forwarded from the vxlan interface.
>>
>> Please find the logs attached.
>>
>>
>>
>> root at ubuntu1404:~# ovs-vsctl show
>>
>> e2127cda-4b01-4020-8a6c-b5ff6be8f2fa
>>
>>     Bridge "br0"
>>
>>         Port "dpdk1"
>>
>>             Interface "dpdk1"
>>
>>                 type: dpdk
>>
>>         Port "vxlan0"
>>
>>             Interface "vxlan0"
>>
>>                 type: vxlan
>>
>>                 options: {remote_ip="10.200.0.6"}
>>
>>         Port "br0"
>>
>>             Interface "br0"
>>
>>                 type: internal
>>
>>     Bridge "br1"
>>
>>         Port "dpdk0"
>>
>>             Interface "dpdk0"
>>
>>                 type: dpdk
>>
>>         Port "br1"
>>
>>             Interface "br1"
>>
>>                 type: internal
>>
>>
>>
>>
>>
>> root at ubuntu1404:~# ovs-ofctl dump-flows br0
>>
>> NXST_FLOW reply (xid=0x4):
>>
>>  cookie=0x0, duration=341.284s, table=0, n_packets=1039, n_bytes=68574,
>> idle_age=0, dl_vlan=1,dl_dst=02:11:22:44:55:22 actions=strip_vlan,output:2
>>
>>  cookie=0x0, duration=427.090s, table=0, n_packets=8, n_bytes=648,
>> idle_age=324, priority=0 actions=NORMAL
>>
>>
>>
>>
>>
>> root at ubuntu1404:~# ovs-ofctl dump-flows br1
>>
>> NXST_FLOW reply (xid=0x4):
>>
>>  cookie=0x0, duration=66.576s, table=0, n_packets=0, n_bytes=0,
>> idle_age=66, in_port=LOCAL actions=output:1
>>
>>  cookie=0x0, duration=401.153s, table=0, n_packets=8, n_bytes=648,
>> idle_age=325, priority=0 actions=NORMAL
>>
>>
>>
>> root at ubuntu1404:~# ovs-appctl tnl/arp/show
>>
>> IP               MAC                 Bridge
>>
>> =============================================
>>
>> 10.200.0.5       b8:ca:3a:6f:16:c8   br1
>>
>>
>>
>> root at ubuntu1404:~# ovs-ofctl show br0
>>
>> OFPT_FEATURES_REPLY (xid=0x2): dpid:0000b8ca3a6f16ca
>>
>> n_tables:254, n_buffers:256
>>
>> capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
>>
>> actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src
>> mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
>>
>>  1(dpdk1): addr:b8:ca:3a:6f:16:ca
>>
>>      config:     0
>>
>>      state:      0
>>
>>      current:    10GB-FD
>>
>>      supported:  1GB-FD FIBER AUTO_PAUSE
>>
>>      speed: 10000 Mbps now, 1000 Mbps max
>>
>>  2(vxlan0): addr:46:6f:4a:fd:94:75
>>
>>      config:     0
>>
>>      state:      0
>>
>>      speed: 0 Mbps now, 0 Mbps max
>>
>>  LOCAL(br0): addr:b8:ca:3a:6f:16:ca
>>
>>      config:     0
>>
>>      state:      0
>>
>>      current:    10MB-FD COPPER
>>
>>      speed: 10 Mbps now, 0 Mbps max
>>
>> OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
>>
>>
>>
>>
>>
>> As you can see in the br0 packets are reaching the vxlan interface and
>> its not forwarding form there it seems dropped at vxlan interface.
>>
>>
>>
>> The same configurations is working without the dpdk.
>>
>> please refer  the attached diagram for my set up.
>>
>>
>>
>> Thanks
>>
>> Praveen
>>
>>
>>
>>
>>
>> On Wed, Oct 28, 2015 at 11:39 AM, Chandran, Sugesh <
>> sugesh.chandran at intel.com> wrote:
>>
>> Hi Praveen,
>>
>>
>>
>> The arp entries are learned properly, if the packets are reaching at br1.
>>
>> Can you please confirm what flow rules are populated on br1 using ,
>>
>>                 ovs-ofctl dump-flows br1
>>
>>
>>
>> If there is no rules present, please add a rule to forward packet to the
>> physical port as follows,
>>
>> ovs-ofctl add-flow br1 idle_timeout=0,in_port=LOCAL,action=output:1
>>
>>
>>
>>
>>
>> *Regards*
>>
>> *_Sugesh*
>>
>>
>>
>> *From:* Praveen MANKARA RADHAKRISHNAN [mailto:praveen.mankara at 6wind.com]
>> *Sent:* Wednesday, October 28, 2015 10:05 AM
>> *To:* Chandran, Sugesh
>> *Cc:* bugs at openvswitch.org; Daniel Virassamy
>> *Subject:* Re: [ovs-discuss] Test with vxlan using ovs-dpdk (2.4)
>>
>>
>>
>> Hi Sugesh,
>>
>>
>>
>> Thanks for the reply.
>>
>> arp entries are show in vs-appctl tnl/arp/show command. but it seems o
>> packet is coming to the bridge with physical port.
>>
>>
>>
>> I have already tried to set using tnl/arp/set command but it seems set is
>> not present in openvswitch 2.4
>>
>> it returns the following error.
>>
>>
>>
>> "
>>
>> ovs-appctl tnl/arp/set br0 10.200.0.6 02:11:22:44:55:11
>>
>> "tnl/arp/set" is not a valid command
>>
>> ovs-appctl: ovs-vswitchd: server returned an error "
>>
>>
>>
>> One important thing is to note that i was using dpdk-2.0.0. does that
>> make any difference?
>>
>>
>>
>> Thanks
>>
>> Praveen
>>
>>
>>
>> On Tue, Oct 27, 2015 at 6:28 PM, Chandran, Sugesh <
>> sugesh.chandran at intel.com> wrote:
>>
>>
>>
>>
>>
>> *Regards*
>>
>> *_Sugesh*
>>
>>
>>
>> *From:* discuss [mailto:discuss-bounces at openvswitch.org] *On Behalf Of *Praveen
>> MANKARA RADHAKRISHNAN
>> *Sent:* Monday, October 26, 2015 5:23 PM
>> *To:* bugs at openvswitch.org
>> *Cc:* Daniel Virassamy
>> *Subject:* [ovs-discuss] Test with vxlan using ovs-dpdk (2.4)
>>
>>
>>
>> HI,
>>
>>
>>
>> I am Praveen,
>>
>>
>>
>> I am trying a specific test scenario using ovs-dpdk.
>>
>>
>>
>> Test set up
>>
>> -------------------
>>
>>
>>
>> from the traffgen vlan packets are coming to dpdk interface its stripped
>> using ovs command.
>>
>> and sending to a vxlan interface which is comnnected in theo other dpdk
>> interface and send back to traffgen.
>>
>>
>>
>> i have done the vxlan set up similar to the sample test case.
>>
>>
>>
>> https://github.com/openvswitch/ovs/blob/master/README-native-tunneling.md
>>
>>
>>
>> the issue is that from the sumpflows i can see that packets are send to
>> vxlan but it is not going to the other bridge.
>>
>>
>>
>> the same scenario is working without dpdk.
>>
>>
>>
>> I am using openvswitch-2.4.0.
>>
>>
>>
>> Linux (without dpdk)
>>
>> ---------------------------------
>>
>>
>>
>> ovs-vsctl add-br br0
>>
>> ovs-vsctl add-port br0 eth0
>>
>> ovs-vsctl add-port br0 vxlan0 -- set interface vxlan0 type=vxlan
>> options:remote_ip=10.200.0.6
>>
>> ovs-ofctl add-flow br0
>> dl_vlan=1,dl_dst=02:11:22:44:55:22,actions=strip_vlan,output:2
>>
>>
>>
>> ovs-vsctl add-br br1
>>
>> ovs-vsctl add-port br1 eth1
>>
>>
>>
>>  ip a a 10.200.0.5/24 dev br1
>>
>> arp -s 10.200.0.6 02:11:22:44:55:11 -i eth2
>>
>>
>>
>> and from the traffgen i am sending traffic with dst mac 02:11:22:44:55:22
>> and vlan id 1.
>>
>> it is correctly working.
>>
>>
>>
>> ovs-dpdk
>>
>> ------------------
>>
>>
>>
>> started ovs-dpdk ::
>>
>> ovs-vswitchd --dpdk -c 0x1 -n 4 -- unix:$DB_SOCK --pidfile --detach
>>
>>
>>
>> ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
>>
>> ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
>>
>>
>>
>> ovs-vsctl add-br br1 -- set bridge br1 datapath_type=netdev
>>
>> ovs-vsctl add-port br1 dpdk1 -- set Interface dpdk1 type=dpdk
>>
>>
>>
>> ovs-vsctl add-port br0 vxlan0 -- set interface vxlan0 type=vxlan
>> options:remote_ip=10.200.0.6
>>
>> ovs-ofctl add-flow br0
>> dl_vlan=1,dl_dst=02:11:22:44:55:22,actions=strip_vlan,output:2
>>
>>
>>
>>  ip a a 10.200.0.5/24 dev br1
>>
>> arp -s 10.200.0.6 02:11:22:44:55:11 -i eth2
>>
>>
>>
>>
>>
>> here i have started the ovs-dpddk and do the same set up
>>
>> but packets are not seen in bridge1 (br1) and in traffgen.
>>
>>
>>
>> is there any specif configuration i need to do in ovs-dpdk
>>
>> could it be because of the arp?
>>
>> Is there a way to set static arp to the ovs-dpdk?
>>
>> >>>> Does the arp entries shown properly in “ovs-appctl tnl/arp/show”???
>>
>> >>>>> You can set the arp entries using ovs-appctl tnl/arp/set <bridge>
>> <ip> <mac>. In OVS-DPDK the DPDK interfaces are not managed by kernel so
>> assigning static arp to physical interface may not work.
>>
>> Could you help in identifying the problem?
>>
>>
>>
>> Thanking you in advance
>>
>> Praveen
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20151130/a32e17bf/attachment-0002.html>


More information about the discuss mailing list