[ovs-dev] OVS DPDK VF port representors not working

Ravi Kerur rkerur at gmail.com
Mon Apr 20 20:04:11 UTC 2020


On Mon, Apr 20, 2020 at 10:37 AM Ravi Kerur <rkerur at gmail.com> wrote:

>
>
> On Mon, Apr 20, 2020 at 3:14 AM Ilya Maximets <i.maximets at ovn.org> wrote:
>
>> On 4/19/20 6:08 PM, Ravi Kerur wrote:
>> >
>> >
>> > On Sun, Apr 19, 2020 at 6:14 AM Ilya Maximets <i.maximets at ovn.org
>> <mailto:i.maximets at ovn.org>> wrote:
>> >
>> >     On 4/18/20 2:33 PM, Ravi Kerur wrote:
>> >     >
>> >     >
>> >     > On Fri, Apr 17, 2020 at 9:57 AM Ilya Maximets <i.maximets at ovn.org
>> <mailto:i.maximets at ovn.org> <mailto:i.maximets at ovn.org <mailto:
>> i.maximets at ovn.org>>> wrote:
>> >     >
>> >     >     On 4/16/20 11:00 PM, Ravi Kerur wrote:
>> >     >     > Hello OvS-DPDK team,
>> >     >     >
>> >     >     > We are expanding our usage of OvS-DPDK and want to use VF
>> port representors
>> >     >     > for one use-case. Going through the following document it
>> is unclear how
>> >     >     > this functionality can be achieved
>> >     >     >
>> >     >     > http://docs.openvswitch.org/en/latest/topics/dpdk/phy/
>> >     >     >
>> >     >     > I believe following things need to be configured before
>> configuring OvS, I
>> >     >     > am using Intel NICs for evaluation
>> >     >     >
>> >     >     > (1) Create VFs on Intel NICs(82599). This can only happen
>> when Intel NIC is
>> >     >     > bound to its respective kernel driver (ixgbe) via
>> >     >     >
>> >     >     > echo 2 > /sys/devices/<pci-BDF>/sriov_numvfs
>> >     >     >
>> >     >     >
>> >     >     > It creates 2 VF interfaces bound to ixgbevf driver
>> >     >     >
>> >     >     >
>> >     >     > (2) Using driverctl, bind VF and PF interfaces to vfio-pci.
>> Moment PF
>> >     >     > device is bound to vfio-pci, VF interfaces vanish
>> >     >     >
>> >     >     >
>> >     >     > Question based on the OvS document shown above
>> >     >     >
>> >     >     >
>> >     >     > (1) Binding PF interface?
>> >     >     >
>> >     >     >
>> >     >     > ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf
>> type=dpdk
>> >     >     > options:dpdk-devargs=0000:08:00.0
>> >     >     >
>> >     >     >
>> >     >     > What driver interface <0000:08:00.0> is bound to? igb_uio,
>> vfio-pci or
>> >     >     > kernel driver?
>> >     >
>> >     >     I believe that it was tested with igb_uio driver, but I never
>> tried it myself.
>> >     >
>> >     >     In order to have sr-iov configuration support with vfio-pci
>> driver you need
>> >     >     a brand new v5.7-rc1 kernel and DPDK changes that are not
>> merged yet:
>> >     >     https://patchwork.dpdk.org/project/dpdk/list/?series=9356
>> >     >     Not sure if it will work smoothly with OVS out of the box,
>> though.
>> >     >
>> >     >
>> >     > Thanks Ilya for the link. Will look into it.
>> >     >
>> >     > Unfortunately igb_uio doesn't work either. Basics of creating
>> VF's out of PF and assigning them to DPDK driver itself is not working.
>> >     > My understanding is
>> >     > (1) In order to create VF's NIC should be bound to its kernel
>> driver. Cannot create VFs when NIC is bound to igb_uio or vfio-pci
>> >     > (2) No mechanism exists to bind both PF and VF to igb_uio to use
>> this feature
>> >     >
>> >     > Output below shows what I am observing on 0000:18:00.0
>> >     >
>> >     > (1) PCI device has PF and 1 VF, both assigned to kernel driver
>> >     >
>> >     > driverctl list-devices network
>> >     >
>> >     > 0000:04:00.0 igb
>> >     >
>> >     > *0000:18:00.0 ixgbe*
>> >     >
>> >     > 0000:18:00.1 ixgbe
>> >     >
>> >     > *0000:18:10.0 ixgbevf*
>> >     >
>> >     > 0000:b3:00.0 vfio-pci [*]
>> >     >
>> >     > 0000:b3:00.1 ixgbe
>> >     >
>> >     >
>> >     > (2) Bind 0000:18:00.0 to igb_uio, VF device is gone
>> >     > # driverctl set-override 0000:18:00.0 igb_uio
>> >     >
>> >     > #driverctl list-devices network
>> >     >
>> >     > 0000:04:00.0 igb
>> >     >
>> >     > *0000:18:00.0 igb_uio [*]*
>> >     >
>> >     > 0000:18:00.1 ixgbe
>> >     >
>> >     > 0000:b3:00.0 vfio-pci [*]
>> >     >
>> >     > 0000:b3:00.1 ixgbe
>> >     >
>> >     >
>> >     > (3) Try to create VF when bound to igb_uio
>> >     >
>> >     >
>> >     > echo 1 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
>> >     >
>> >     > bash: echo: write error: No such file or directory
>> >
>> >     OVS and DPDK docs both says that knob should be "max_vfs", not
>> "sriov_numvfs".
>> >     I didn't test, but looking at the code of igb_uio driver it
>> actually creates
>> >     this sysfs entry.
>> >
>> >
>> > Thanks for your kind help. My mistake. On Ubuntu max_vfs is created in
>> >
>> > /sys/devices/pci0000:16/0000:16:02.0/0000:18:00.0/max_vfs instead
>> of usual
>> >
>> > /sys/bus/pci/devices/0000\:18\:00.0/
>> >
>> >  I am able to create VF's when PF is attached to igb_uio.
>> > When adding dpdk-pf port to bridge I get following errors (OvS 2.13.90,
>> DPDK 19.11). Please note I have started with empty conf.db in OvS.
>> >
>> > driverctl list-devices network
>> >
>> > 0000:04:00.0 igb
>> >
>> > 0000:18:00.0 igb_uio [*]
>> >
>> > 0000:18:00.1 ixgbe
>> >
>> > 0000:18:10.0 vfio-pci [*]
>> >
>> > 0000:18:10.2 vfio-pci [*]
>> >
>> > 0000:b3:00.0 vfio-pci [*]
>> >
>> > 0000:b3:00.1 ixgbe
>> >
>> >
>> > ovs-vsctl add-br br-phy -- set bridge br-phy datapath_type=netdev
>> >
>> > ovs-vsctl add-port br-phy dpdk-pf -- set Interface dpdk-pf type=dpdk
>> options:dpdk-devargs=0000:18:00.0
>> >
>> > ovs-vsctl: Error detected while setting up 'dpdk-pf': could not add
>> network device dpdk-pf to ofproto (Invalid argument).  See ovs-vswitchd log
>> for details.
>> >
>> > ovs-vsctl: The default log directory is
>> "/usr/local/var/log/openvswitch".
>> >
>> >
>> > Snippets below
>> >
>> >
>> >
>> > 2020-04-19T15:50:54.279Z|00170|dpif_netdev|INFO|PMD thread on numa_id:
>> 0, core id:  9 created.
>> >
>> > 2020-04-19T15:50:54.448Z|00171|dpif_netdev|INFO|PMD thread on numa_id:
>> 0, core id:  6 created.
>> >
>> > 2020-04-19T15:50:54.617Z|00172|dpif_netdev|INFO|PMD thread on numa_id:
>> 0, core id:  8 created.
>> >
>> > 2020-04-19T15:50:54.786Z|00173|dpif_netdev|INFO|PMD thread on numa_id:
>> 0, core id:  7 created.
>> >
>> > 2020-04-19T15:50:54.786Z|00174|dpif_netdev|INFO|There are 4 pmd threads
>> on numa node 0
>> >
>> > 2020-04-19T15:50:54.786Z|00175|dpdk|INFO|Device with port_id=0 already
>> stopped
>> >
>> > 2020-04-19T15:50:54.909Z|00176|dpdk|ERR|ixgbe_check_mq_mode(): SRIOV is
>> active, nb_rx_q=1 nb_tx_q=5 queue number must be less than or equal to 2.
>> >
>> > 2020-04-19T15:50:54.909Z|00177|dpdk|ERR|ixgbe_dev_configure():
>> ixgbe_check_mq_mode fails with -22.
>> >
>> > 2020-04-19T15:50:54.909Z|00178|dpdk|ERR|Port0 dev_configure = -22
>> >
>> > 2020-04-19T15:50:54.909Z|00179|netdev_dpdk|WARN|Interface dpdk-pf
>> eth_dev setup error Invalid argument
>> >
>> > 2020-04-19T15:50:54.910Z|00180|netdev_dpdk|ERR|Interface dpdk-pf(rxq:1
>> txq:5 lsc interrupt mode:false) configure error: Invalid argument
>> >
>> > 2020-04-19T15:50:54.910Z|00181|dpif_netdev|ERR|Failed to set interface
>> dpdk-pf new configuration
>> >
>> > 2020-04-19T15:50:54.910Z|00182|dpif|WARN|netdev at ovs-netdev: failed to
>> add dpdk-pf as port: Invalid argument
>> >
>> > 2020-04-19T15:50:54.910Z|00183|bridge|WARN|could not add network device
>> dpdk-pf to ofproto (Invalid argument)
>> >
>> > 2020-04-19T15:50:54.910Z|00184|dpdk|INFO|Device with port_id=0 already
>> stopped
>>
>>
>> Oh. That's nasty.  It's hard to find out what went wrong on configure()
>> failure,
>> since DPDK doesn't provide any additional information beside EINVAL.  It
>> only
>> prints some logs.  But tx queues are fully controlled by OVS and not by
>> the user.
>>
>> There are 2 things you can try:
>>
>> 1. Configure only one PMD thread, i.e. keep only one bit set in
>> pmd-cpu-mask.
>>    In this case OVS will try to configure only 2 tx queues which is your
>> limit.
>>
>> 2. Try out following not that brilliant solution:
>> ---
>> diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
>> index 44ebf96da..3eab4e549 100644
>> --- a/lib/netdev-dpdk.c
>> +++ b/lib/netdev-dpdk.c
>> @@ -1011,7 +1011,18 @@ dpdk_eth_dev_port_config(struct netdev_dpdk *dev,
>> int n_rxq, int n_txq)
>>          if (diag) {
>>              VLOG_WARN("Interface %s eth_dev setup error %s\n",
>>                        dev->up.name, rte_strerror(-diag));
>> -            break;
>> +            if (n_rxq == 1 && n_txq == 1) {
>> +                /* Can't keep guessing anymore. */
>> +                break;
>> +            }
>> +            /* Trying to guess if it was an issue with number of queues.
>> */
>> +            if (n_rxq > 1) {
>> +                n_rxq--;
>> +            }
>> +            if (n_txq > 1) {
>> +                n_txq--;
>> +            }
>> +            continue;
>>          }
>>
>>          diag = rte_eth_dev_set_mtu(dev->port_id, dev->mtu);
>> ---
>>
>>
>> Looks like the issue is only seen when VF's are enabled via igb_uio
> max_vfs. As shown below, with no VFs for 0000:18:00.0, able to add port
> successfully.
>
> driverctl list-devices network
>
> 0000:04:00.0 igb
>
> 0000:18:00.0 igb_uio [*]
>
> 0000:18:00.1 ixgbe
>
> 0000:b3:00.0 igb_uio [*]
>
> 0000:b3:00.1 ixgbe
>
> #ovs-vsctl add-br br-phy -- set bridge br-phy datapath_type=netdev
> # ovs-vsctl add-port br-phy dpdk-pf -- set Interface dpdk-pf type=dpdk
> options:dpdk-devargs=0000:18:00.0
>
> ovs-vsctl show
>
> 2eb77bfa-ff68-4abe-adcd-ae9dccde8c51
>
>     Bridge br-phy
>
>         datapath_type: netdev
>
>         Port br-phy
>
>             Interface br-phy
>
>                 type: internal
>
>         Port dpdk-pf
>
>             Interface dpdk-pf
>
>                 type: dpdk
>
>                 options: {dpdk-devargs="0000:18:00.0"}
>
>     ovs_version: "2.13.90"
>
>
> I will try the same configuration with your changes and update. I have
> couple of questions though
>
>
> (1) Number of txqueues auto determined based on no. of rxqueues?
>
> (2) In function netdev_dpdk_set_config there is a call to
> dpdk_set_rxq_config, shouldn't there be a similar for txqueue as well?
>
>
> netdev_dpdk_set_config(struct netdev *netdev, const struct smap *args,
>
>                        char **errp)
>
> {
>
>
>
>     ovs_mutex_lock(&dpdk_mutex);
>
>     ovs_mutex_lock(&dev->mutex);
>
>
>     dpdk_set_rxq_config(dev, args);
>
> ...
>
> }
>
>
I tested your changes and it fixes adding PF interface issue. However, I
have stumbled into adding representor ports as it ends up getting errors.
I have tested with VF devices bound to either igb_uio or vfio-pci devices,
both get same errors. I tried with representor[1], [2], [3], same errors

driverctl -v list-devices network

0000:04:00.0 igb (I210 Gigabit Network Connection)

0000:18:00.0 igb_uio [*] (82599ES 10-Gigabit SFI/SFP+ Network Connection
(Ethernet Server Adapter X520-2))

0000:18:00.1 ixgbe (82599ES 10-Gigabit SFI/SFP+ Network Connection
(Ethernet Server Adapter X520-2))

0000:18:10.0 igb_uio [*] (82599 Ethernet Controller Virtual Function)

0000:18:10.2 igb_uio [*] (82599 Ethernet Controller Virtual Function)

0000:b3:00.0 igb_uio [*] (82599ES 10-Gigabit SFI/SFP+ Network Connection
(Ethernet OCP Server Adapter X520-2))

0000:b3:00.1 ixgbe (82599ES 10-Gigabit SFI/SFP+ Network Connection
(Ethernet OCP Server Adapter X520-2))


ovs-vsctl add-br br-phy -- set bridge br-phy datapath_type=netdev

# ovs-vsctl add-port br-phy dpdk-pf -- set Interface dpdk-pf type=dpdk
options:dpdk-devargs=0000:18:00.0

# ovs-vsctl add-port br-phy dpdk-rep0 -- set Interface dpdk-rep0 type=dpdk
options:dpdk-devargs=0000:18:00.0,representor=[0]

ovs-vsctl: Error detected while setting up 'dpdk-rep0': Error attaching
device '0000:18:00.0,representor=[0]' to DPDK.  See ovs-vswitchd log for
details.

ovs-vsctl: The default log directory is "/usr/local/var/log/openvswitch".


Snippets...

2020-04-20T19:54:18.793Z|00190|dpdk|INFO|EAL: PCI device 0000:18:00.0 on
NUMA socket 0

2020-04-20T19:54:18.793Z|00191|dpdk|ERR|EAL: Failed to attach device on
primary process

2020-04-20T19:54:18.793Z|00192|netdev_dpdk|WARN|Error attaching device
'0000:18:00.0,representor=[0]' to DPDK

2020-04-20T19:54:18.793Z|00193|netdev|WARN|dpdk-rep0: could not set
configuration (Invalid argument)

2020-04-20T19:54:18.793Z|00194|dpdk|ERR|Invalid port_id=32

ovs-vsctl show

c53ebb42-b4c1-4a83-9047-a2b012703679

    Bridge br-phy

        datapath_type: netdev

        Port dpdk-rep0

            Interface dpdk-rep0

                type: dpdk

                options: {dpdk-devargs="0000:18:00.0,representor=[0]"}

                error: "Error attaching device
'0000:18:00.0,representor=[0]' to DPDK"

        Port br-phy

            Interface br-phy

                type: internal

        Port dpdk-pf

            Interface dpdk-pf

                type: dpdk

                options: {dpdk-devargs="0000:18:00.0"}

    ovs_version: "2.13.90"

>
>> >
>> >     >
>> >     >
>> >     > (4) Unset 0000:18:00.0 to kernel driver, can create VF
>> >     >
>> >     >
>> >     > # driverctl unset-override 0000:18:00.0
>> >     >
>> >     > #echo 1 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
>> >     >
>> >     > #driverctl list-devices network
>> >     >
>> >     > 0000:04:00.0 igb
>> >     >
>> >     > *0000:18:00.0 ixgbe*
>> >     >
>> >     > 0000:18:00.1 ixgbe
>> >     >
>> >     > *0000:18:10.0 ixgbevf*
>> >     >
>> >     > 0000:b3:00.0 vfio-pci [*]
>> >     >
>> >     > 0000:b3:00.1 ixgbe
>> >     >
>> >     >
>> >     > Without binding both PF and VF to userspace drivers to use DPDK
>> PMD it is not clear to me how this functionality will work?
>> >     >
>> >     >
>> >     >     >
>> >     >     >
>> >     >     > (2) VF interface to representor connection, does
>> representor=[3] represent
>> >     >     > VF device 3?
>> >     >     >
>> >     >     >
>> >     >     > ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3
>> type=dpdk
>> >     >     > options:dpdk-devargs=0000:08:00.0,representor=[3]
>> >     >     >
>> >     >     > ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep5
>> type=dpdk
>> >     >     > options:dpdk-devargs=0000:08:00.0,representor=[5]
>> >     >     >
>> >     >     >
>> >     >     > Looking into linked DPDK document within OvS is confusing
>> as well.
>> >     >     >
>> >     >     >
>> >     >     > Kindly let me know what I need to do for the setup?
>> >     >     >
>> >     >     >
>> >     >     > Thanks,
>> >     >     >
>> >     >     > Ravi
>> >     >     > _______________________________________________
>> >     >     > dev mailing list
>> >     >     > dev at openvswitch.org <mailto:dev at openvswitch.org> <mailto:
>> dev at openvswitch.org <mailto:dev at openvswitch.org>>
>> >     >     > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>> >     >     >
>> >     >
>> >
>>
>>


More information about the dev mailing list