[ovs-dev] OVS DPDK VF port representors not working
Ravi Kerur
rkerur at gmail.com
Mon Apr 20 20:04:11 UTC 2020
On Mon, Apr 20, 2020 at 10:37 AM Ravi Kerur <rkerur at gmail.com> wrote:
>
>
> On Mon, Apr 20, 2020 at 3:14 AM Ilya Maximets <i.maximets at ovn.org> wrote:
>
>> On 4/19/20 6:08 PM, Ravi Kerur wrote:
>> >
>> >
>> > On Sun, Apr 19, 2020 at 6:14 AM Ilya Maximets <i.maximets at ovn.org
>> <mailto:i.maximets at ovn.org>> wrote:
>> >
>> > On 4/18/20 2:33 PM, Ravi Kerur wrote:
>> > >
>> > >
>> > > On Fri, Apr 17, 2020 at 9:57 AM Ilya Maximets <i.maximets at ovn.org
>> <mailto:i.maximets at ovn.org> <mailto:i.maximets at ovn.org <mailto:
>> i.maximets at ovn.org>>> wrote:
>> > >
>> > > On 4/16/20 11:00 PM, Ravi Kerur wrote:
>> > > > Hello OvS-DPDK team,
>> > > >
>> > > > We are expanding our usage of OvS-DPDK and want to use VF
>> port representors
>> > > > for one use-case. Going through the following document it
>> is unclear how
>> > > > this functionality can be achieved
>> > > >
>> > > > http://docs.openvswitch.org/en/latest/topics/dpdk/phy/
>> > > >
>> > > > I believe following things need to be configured before
>> configuring OvS, I
>> > > > am using Intel NICs for evaluation
>> > > >
>> > > > (1) Create VFs on Intel NICs(82599). This can only happen
>> when Intel NIC is
>> > > > bound to its respective kernel driver (ixgbe) via
>> > > >
>> > > > echo 2 > /sys/devices/<pci-BDF>/sriov_numvfs
>> > > >
>> > > >
>> > > > It creates 2 VF interfaces bound to ixgbevf driver
>> > > >
>> > > >
>> > > > (2) Using driverctl, bind VF and PF interfaces to vfio-pci.
>> Moment PF
>> > > > device is bound to vfio-pci, VF interfaces vanish
>> > > >
>> > > >
>> > > > Question based on the OvS document shown above
>> > > >
>> > > >
>> > > > (1) Binding PF interface?
>> > > >
>> > > >
>> > > > ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf
>> type=dpdk
>> > > > options:dpdk-devargs=0000:08:00.0
>> > > >
>> > > >
>> > > > What driver interface <0000:08:00.0> is bound to? igb_uio,
>> vfio-pci or
>> > > > kernel driver?
>> > >
>> > > I believe that it was tested with igb_uio driver, but I never
>> tried it myself.
>> > >
>> > > In order to have sr-iov configuration support with vfio-pci
>> driver you need
>> > > a brand new v5.7-rc1 kernel and DPDK changes that are not
>> merged yet:
>> > > https://patchwork.dpdk.org/project/dpdk/list/?series=9356
>> > > Not sure if it will work smoothly with OVS out of the box,
>> though.
>> > >
>> > >
>> > > Thanks Ilya for the link. Will look into it.
>> > >
>> > > Unfortunately igb_uio doesn't work either. Basics of creating
>> VF's out of PF and assigning them to DPDK driver itself is not working.
>> > > My understanding is
>> > > (1) In order to create VF's NIC should be bound to its kernel
>> driver. Cannot create VFs when NIC is bound to igb_uio or vfio-pci
>> > > (2) No mechanism exists to bind both PF and VF to igb_uio to use
>> this feature
>> > >
>> > > Output below shows what I am observing on 0000:18:00.0
>> > >
>> > > (1) PCI device has PF and 1 VF, both assigned to kernel driver
>> > >
>> > > driverctl list-devices network
>> > >
>> > > 0000:04:00.0 igb
>> > >
>> > > *0000:18:00.0 ixgbe*
>> > >
>> > > 0000:18:00.1 ixgbe
>> > >
>> > > *0000:18:10.0 ixgbevf*
>> > >
>> > > 0000:b3:00.0 vfio-pci [*]
>> > >
>> > > 0000:b3:00.1 ixgbe
>> > >
>> > >
>> > > (2) Bind 0000:18:00.0 to igb_uio, VF device is gone
>> > > # driverctl set-override 0000:18:00.0 igb_uio
>> > >
>> > > #driverctl list-devices network
>> > >
>> > > 0000:04:00.0 igb
>> > >
>> > > *0000:18:00.0 igb_uio [*]*
>> > >
>> > > 0000:18:00.1 ixgbe
>> > >
>> > > 0000:b3:00.0 vfio-pci [*]
>> > >
>> > > 0000:b3:00.1 ixgbe
>> > >
>> > >
>> > > (3) Try to create VF when bound to igb_uio
>> > >
>> > >
>> > > echo 1 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
>> > >
>> > > bash: echo: write error: No such file or directory
>> >
>> > OVS and DPDK docs both says that knob should be "max_vfs", not
>> "sriov_numvfs".
>> > I didn't test, but looking at the code of igb_uio driver it
>> actually creates
>> > this sysfs entry.
>> >
>> >
>> > Thanks for your kind help. My mistake. On Ubuntu max_vfs is created in
>> >
>> > /sys/devices/pci0000:16/0000:16:02.0/0000:18:00.0/max_vfs instead
>> of usual
>> >
>> > /sys/bus/pci/devices/0000\:18\:00.0/
>> >
>> > I am able to create VF's when PF is attached to igb_uio.
>> > When adding dpdk-pf port to bridge I get following errors (OvS 2.13.90,
>> DPDK 19.11). Please note I have started with empty conf.db in OvS.
>> >
>> > driverctl list-devices network
>> >
>> > 0000:04:00.0 igb
>> >
>> > 0000:18:00.0 igb_uio [*]
>> >
>> > 0000:18:00.1 ixgbe
>> >
>> > 0000:18:10.0 vfio-pci [*]
>> >
>> > 0000:18:10.2 vfio-pci [*]
>> >
>> > 0000:b3:00.0 vfio-pci [*]
>> >
>> > 0000:b3:00.1 ixgbe
>> >
>> >
>> > ovs-vsctl add-br br-phy -- set bridge br-phy datapath_type=netdev
>> >
>> > ovs-vsctl add-port br-phy dpdk-pf -- set Interface dpdk-pf type=dpdk
>> options:dpdk-devargs=0000:18:00.0
>> >
>> > ovs-vsctl: Error detected while setting up 'dpdk-pf': could not add
>> network device dpdk-pf to ofproto (Invalid argument). See ovs-vswitchd log
>> for details.
>> >
>> > ovs-vsctl: The default log directory is
>> "/usr/local/var/log/openvswitch".
>> >
>> >
>> > Snippets below
>> >
>> >
>> >
>> > 2020-04-19T15:50:54.279Z|00170|dpif_netdev|INFO|PMD thread on numa_id:
>> 0, core id: 9 created.
>> >
>> > 2020-04-19T15:50:54.448Z|00171|dpif_netdev|INFO|PMD thread on numa_id:
>> 0, core id: 6 created.
>> >
>> > 2020-04-19T15:50:54.617Z|00172|dpif_netdev|INFO|PMD thread on numa_id:
>> 0, core id: 8 created.
>> >
>> > 2020-04-19T15:50:54.786Z|00173|dpif_netdev|INFO|PMD thread on numa_id:
>> 0, core id: 7 created.
>> >
>> > 2020-04-19T15:50:54.786Z|00174|dpif_netdev|INFO|There are 4 pmd threads
>> on numa node 0
>> >
>> > 2020-04-19T15:50:54.786Z|00175|dpdk|INFO|Device with port_id=0 already
>> stopped
>> >
>> > 2020-04-19T15:50:54.909Z|00176|dpdk|ERR|ixgbe_check_mq_mode(): SRIOV is
>> active, nb_rx_q=1 nb_tx_q=5 queue number must be less than or equal to 2.
>> >
>> > 2020-04-19T15:50:54.909Z|00177|dpdk|ERR|ixgbe_dev_configure():
>> ixgbe_check_mq_mode fails with -22.
>> >
>> > 2020-04-19T15:50:54.909Z|00178|dpdk|ERR|Port0 dev_configure = -22
>> >
>> > 2020-04-19T15:50:54.909Z|00179|netdev_dpdk|WARN|Interface dpdk-pf
>> eth_dev setup error Invalid argument
>> >
>> > 2020-04-19T15:50:54.910Z|00180|netdev_dpdk|ERR|Interface dpdk-pf(rxq:1
>> txq:5 lsc interrupt mode:false) configure error: Invalid argument
>> >
>> > 2020-04-19T15:50:54.910Z|00181|dpif_netdev|ERR|Failed to set interface
>> dpdk-pf new configuration
>> >
>> > 2020-04-19T15:50:54.910Z|00182|dpif|WARN|netdev at ovs-netdev: failed to
>> add dpdk-pf as port: Invalid argument
>> >
>> > 2020-04-19T15:50:54.910Z|00183|bridge|WARN|could not add network device
>> dpdk-pf to ofproto (Invalid argument)
>> >
>> > 2020-04-19T15:50:54.910Z|00184|dpdk|INFO|Device with port_id=0 already
>> stopped
>>
>>
>> Oh. That's nasty. It's hard to find out what went wrong on configure()
>> failure,
>> since DPDK doesn't provide any additional information beside EINVAL. It
>> only
>> prints some logs. But tx queues are fully controlled by OVS and not by
>> the user.
>>
>> There are 2 things you can try:
>>
>> 1. Configure only one PMD thread, i.e. keep only one bit set in
>> pmd-cpu-mask.
>> In this case OVS will try to configure only 2 tx queues which is your
>> limit.
>>
>> 2. Try out following not that brilliant solution:
>> ---
>> diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
>> index 44ebf96da..3eab4e549 100644
>> --- a/lib/netdev-dpdk.c
>> +++ b/lib/netdev-dpdk.c
>> @@ -1011,7 +1011,18 @@ dpdk_eth_dev_port_config(struct netdev_dpdk *dev,
>> int n_rxq, int n_txq)
>> if (diag) {
>> VLOG_WARN("Interface %s eth_dev setup error %s\n",
>> dev->up.name, rte_strerror(-diag));
>> - break;
>> + if (n_rxq == 1 && n_txq == 1) {
>> + /* Can't keep guessing anymore. */
>> + break;
>> + }
>> + /* Trying to guess if it was an issue with number of queues.
>> */
>> + if (n_rxq > 1) {
>> + n_rxq--;
>> + }
>> + if (n_txq > 1) {
>> + n_txq--;
>> + }
>> + continue;
>> }
>>
>> diag = rte_eth_dev_set_mtu(dev->port_id, dev->mtu);
>> ---
>>
>>
>> Looks like the issue is only seen when VF's are enabled via igb_uio
> max_vfs. As shown below, with no VFs for 0000:18:00.0, able to add port
> successfully.
>
> driverctl list-devices network
>
> 0000:04:00.0 igb
>
> 0000:18:00.0 igb_uio [*]
>
> 0000:18:00.1 ixgbe
>
> 0000:b3:00.0 igb_uio [*]
>
> 0000:b3:00.1 ixgbe
>
> #ovs-vsctl add-br br-phy -- set bridge br-phy datapath_type=netdev
> # ovs-vsctl add-port br-phy dpdk-pf -- set Interface dpdk-pf type=dpdk
> options:dpdk-devargs=0000:18:00.0
>
> ovs-vsctl show
>
> 2eb77bfa-ff68-4abe-adcd-ae9dccde8c51
>
> Bridge br-phy
>
> datapath_type: netdev
>
> Port br-phy
>
> Interface br-phy
>
> type: internal
>
> Port dpdk-pf
>
> Interface dpdk-pf
>
> type: dpdk
>
> options: {dpdk-devargs="0000:18:00.0"}
>
> ovs_version: "2.13.90"
>
>
> I will try the same configuration with your changes and update. I have
> couple of questions though
>
>
> (1) Number of txqueues auto determined based on no. of rxqueues?
>
> (2) In function netdev_dpdk_set_config there is a call to
> dpdk_set_rxq_config, shouldn't there be a similar for txqueue as well?
>
>
> netdev_dpdk_set_config(struct netdev *netdev, const struct smap *args,
>
> char **errp)
>
> {
>
>
>
> ovs_mutex_lock(&dpdk_mutex);
>
> ovs_mutex_lock(&dev->mutex);
>
>
> dpdk_set_rxq_config(dev, args);
>
> ...
>
> }
>
>
I tested your changes and it fixes adding PF interface issue. However, I
have stumbled into adding representor ports as it ends up getting errors.
I have tested with VF devices bound to either igb_uio or vfio-pci devices,
both get same errors. I tried with representor[1], [2], [3], same errors
driverctl -v list-devices network
0000:04:00.0 igb (I210 Gigabit Network Connection)
0000:18:00.0 igb_uio [*] (82599ES 10-Gigabit SFI/SFP+ Network Connection
(Ethernet Server Adapter X520-2))
0000:18:00.1 ixgbe (82599ES 10-Gigabit SFI/SFP+ Network Connection
(Ethernet Server Adapter X520-2))
0000:18:10.0 igb_uio [*] (82599 Ethernet Controller Virtual Function)
0000:18:10.2 igb_uio [*] (82599 Ethernet Controller Virtual Function)
0000:b3:00.0 igb_uio [*] (82599ES 10-Gigabit SFI/SFP+ Network Connection
(Ethernet OCP Server Adapter X520-2))
0000:b3:00.1 ixgbe (82599ES 10-Gigabit SFI/SFP+ Network Connection
(Ethernet OCP Server Adapter X520-2))
ovs-vsctl add-br br-phy -- set bridge br-phy datapath_type=netdev
# ovs-vsctl add-port br-phy dpdk-pf -- set Interface dpdk-pf type=dpdk
options:dpdk-devargs=0000:18:00.0
# ovs-vsctl add-port br-phy dpdk-rep0 -- set Interface dpdk-rep0 type=dpdk
options:dpdk-devargs=0000:18:00.0,representor=[0]
ovs-vsctl: Error detected while setting up 'dpdk-rep0': Error attaching
device '0000:18:00.0,representor=[0]' to DPDK. See ovs-vswitchd log for
details.
ovs-vsctl: The default log directory is "/usr/local/var/log/openvswitch".
Snippets...
2020-04-20T19:54:18.793Z|00190|dpdk|INFO|EAL: PCI device 0000:18:00.0 on
NUMA socket 0
2020-04-20T19:54:18.793Z|00191|dpdk|ERR|EAL: Failed to attach device on
primary process
2020-04-20T19:54:18.793Z|00192|netdev_dpdk|WARN|Error attaching device
'0000:18:00.0,representor=[0]' to DPDK
2020-04-20T19:54:18.793Z|00193|netdev|WARN|dpdk-rep0: could not set
configuration (Invalid argument)
2020-04-20T19:54:18.793Z|00194|dpdk|ERR|Invalid port_id=32
ovs-vsctl show
c53ebb42-b4c1-4a83-9047-a2b012703679
Bridge br-phy
datapath_type: netdev
Port dpdk-rep0
Interface dpdk-rep0
type: dpdk
options: {dpdk-devargs="0000:18:00.0,representor=[0]"}
error: "Error attaching device
'0000:18:00.0,representor=[0]' to DPDK"
Port br-phy
Interface br-phy
type: internal
Port dpdk-pf
Interface dpdk-pf
type: dpdk
options: {dpdk-devargs="0000:18:00.0"}
ovs_version: "2.13.90"
>
>> >
>> > >
>> > >
>> > > (4) Unset 0000:18:00.0 to kernel driver, can create VF
>> > >
>> > >
>> > > # driverctl unset-override 0000:18:00.0
>> > >
>> > > #echo 1 > /sys/bus/pci/devices/0000\:18\:00.0/sriov_numvfs
>> > >
>> > > #driverctl list-devices network
>> > >
>> > > 0000:04:00.0 igb
>> > >
>> > > *0000:18:00.0 ixgbe*
>> > >
>> > > 0000:18:00.1 ixgbe
>> > >
>> > > *0000:18:10.0 ixgbevf*
>> > >
>> > > 0000:b3:00.0 vfio-pci [*]
>> > >
>> > > 0000:b3:00.1 ixgbe
>> > >
>> > >
>> > > Without binding both PF and VF to userspace drivers to use DPDK
>> PMD it is not clear to me how this functionality will work?
>> > >
>> > >
>> > > >
>> > > >
>> > > > (2) VF interface to representor connection, does
>> representor=[3] represent
>> > > > VF device 3?
>> > > >
>> > > >
>> > > > ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3
>> type=dpdk
>> > > > options:dpdk-devargs=0000:08:00.0,representor=[3]
>> > > >
>> > > > ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep5
>> type=dpdk
>> > > > options:dpdk-devargs=0000:08:00.0,representor=[5]
>> > > >
>> > > >
>> > > > Looking into linked DPDK document within OvS is confusing
>> as well.
>> > > >
>> > > >
>> > > > Kindly let me know what I need to do for the setup?
>> > > >
>> > > >
>> > > > Thanks,
>> > > >
>> > > > Ravi
>> > > > _______________________________________________
>> > > > dev mailing list
>> > > > dev at openvswitch.org <mailto:dev at openvswitch.org> <mailto:
>> dev at openvswitch.org <mailto:dev at openvswitch.org>>
>> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>> > > >
>> > >
>> >
>>
>>
More information about the dev
mailing list