[ovs-discuss] Starting OVS w/ DPDK ...

[Flavio Leitner]

Hi,

On Mon, Oct 08, 2018 at 11:52:24AM +0200, Andrzej Ostruszka wrote:
> First of all thank you Flavio for your reply and sorry for late reply
> but I had to switch to something else.  Now I resume my ovs ramp up and
> have some additional problems.
> 
> My goal is actually to measure performance gain resulting from
> offloading of flow matching to HW.  I started with Marvell (even though
> it does not support at the moment RSS+MARK action in DPDK flow API)
> since I thought that if I get it up and running that would be a good
> starting point for adding support for flow matching offload to Marvell
> PMD.  However I do have problems even with what seems to be already
> supported.
> 
> For the moment I suspend the starting of OVS/DPDK on Marvell PMD and I'm
> trying to bring up the OVS with Mellanox MLX5 PMD.  That should be
> supported according to the documentation however I fail to bring it up.
> Maybe there is something very basic that I get wrong so I'd like to
> solicit some help to first start OVS/DPDK on this card (MCX516A-CDAT).
> Once I get this up and running I'll move to Marvell PMDs.
> 
> Let me shortly describe my attempt - maybe you'll spot something wrong.
> 1. I compile and install DPDK and test if testpmd has no problems
> (output at the end [1] in order to not clutter this e-mail too much).
> 
> 2. Then I configure ovs
> 
> ./configure --with-dpdk=$HOME/dpdk-install/usr/local
> --with-linux=/lib/modules/$(uname -r)/build LIBS="-libverbs -lmlx5"
> 
> and install it.
> 
> 3. I tell OVS to initialize DPDK
> 
> # ovs-vsctl get Open_vSwitch . other_config
> {dpdk-init="true"}
> 
> 4. I start it (without any bridge)
> 
> # ovs-ctl --delete-bridges start
> 
>  * Starting ovsdb-server
>  * system ID not configured, please use --system-id
>  * Configuring Open vSwitch system IDs
> Zone 0: name:<rte_eth_dev_data>, IO:0x5e9b6000, len:0x34900,
> virt:0x7f8dd29b6000, socket_id:0, flags:0
>  * Starting ovs-vswitchd
>  * Enabling remote OVSDB managers
> 
> 5. Then I attach to ovs-vswitchd in gdb and set breakpoint in
> rte_eal_hotplug_add
> 
> 6. Then I add bridge and first DPDK port
> 
> # ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
> # ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk
> 	options:dpdk-devargs=08:00.0
> 
> 7. In the debugger I step through until I get to pci_find_device() and
> in there the comparison function used is 'cmp_detached_dev_name()' which
> checks for dev->drive to be NULL and in my case it is not as it points
> to mlx5 driver:
> 
> (gdb) p name
> $23 = 0x262c250 "08:00.0"
> (gdb) p *dev
> $24 = {next = {tqe_next = 0x0, tqe_prev = 0x0}, name = 0x25fa5c0 "08:00.0",
>   driver = 0xc96b30 <mlx5_driver+16>, numa_node = 0, devargs = 0x25fa5a0}
> (gdb) n
> 55                      return 1;
> 
> and as a result it fails to add port:
> 
> ovs-vsctl: Error detected while setting up 'dpdk-p0': Error attaching
> device '08:00.0' to DPDK.  See ovs-vswitchd log for details.
> ovs-vsctl: The default log directory is "/usr/local/var/log/openvswitch".
> 
> and in there the confirmation from DPDK:
> 
> 2018-10-08T09:38:28.234Z|00058|dpdk|ERR|EAL: Cannot find unplugged
> device (08:00.0)
> 2018-10-08T09:38:28.234Z|00059|netdev_dpdk|WARN|Error attaching device
> '08:00.0' to DPDK
> 2018-10-08T09:38:28.234Z|00060|netdev|WARN|dpdk-p0: could not set
> configuration (Invalid argument)
> 
> So basically port seems to be attached (to correct driver) but somehow
> detached port is searched for.  Seems very much like some config problem
> but I don't know OVS enough to figure it out.
> 
> So I'd still appreciate some help in bringing OVS w/ DPDK up.

I've not tried Mellanox myself in a while now, but I found a piece of
documentation that you might have missed:

Documentation/howto/dpdk.rst:
[...]
Some NICs (i.e. Mellanox ConnectX-3) have only one PCI address associated with
multiple ports. Using a PCI device like above won't work. Instead, below usage
is suggested::   
                 
    $ ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0
type=dpdk \
        options:dpdk-devargs="class=eth,mac=00:11:22:33:44:55"
    $ ovs-vsctl add-port br0 dpdk-p1 -- set Interface dpdk-p1
type=dpdk \
        options:dpdk-devargs="class=eth,mac=00:11:22:33:44:56"
[...]


> > BTW the bridge 'br0' also needs to be in the userspace DP.
> 
> Ohh, that seems to be in contrast to what this doc:
> http://docs.openvswitch.org/en/latest/howto/dpdk/
> seems to suggest.  It tells that the bridge has to be netdev (or maybe I
> don't understand what the 'netdev' datapath type is).

That is correct.

-- 
Flavio