[ovs-discuss] Issue with Intel 82599 PF and OVS

Dmitry Nikishov dnikishov at mirantis.com
Wed Nov 26 18:16:38 UTC 2014


Yes, it is still visible.

root at node-6:~# ovs-vsctl list interface eth0 #without VF
_uuid               : 16d4d75a-baf2-4613-baa5-c9a77d8720e2
admin_state         : up
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : full
external_ids        : {}
ifindex             : 2
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 16
link_speed          : 10000000000
link_state          : up
mac                 : []
mac_in_use          : "38:ea:a7:35:dd:60"
mtu                 : 1500
name                : "eth0"
ofport              : 1
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {collisions=0, rx_bytes=1160109, rx_crc_err=0,
rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=7238,
tx_bytes=1799066, tx_dropped=0, tx_errors=0, tx_packets=9634}
status              : {driver_name=ixgbe, driver_version="3.19.1-k",
firmware_version="0x80000684"}
type                : ""


root at node-6:~# ovs-vsctl list interface eth0 #with 1 VF enabled
_uuid               : 16d4d75a-baf2-4613-baa5-c9a77d8720e2
admin_state         : up
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : full
external_ids        : {}
ifindex             : 2
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 16
link_speed          : 10000000000
link_state          : up
mac                 : []
mac_in_use          : "38:ea:a7:35:dd:60"
mtu                 : 1500
name                : "eth0"
ofport              : 1
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {collisions=0, rx_bytes=1729, rx_crc_err=0,
rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=9,
tx_bytes=2115, tx_dropped=0, tx_errors=0, tx_packets=25}
status              : {driver_name=ixgbe, driver_version="3.19.1-k",
firmware_version="0x80000684"}
type                : ""


ovs-vsctl show (eth0/mgmt related part):
      Bridge "br-eth0"
        Port "br-eth0--br-mgmt"
            tag: 102
            Interface "br-eth0--br-mgmt"
                type: patch
                options: {peer="br-mgmt--br-eth0"}
        Port "br-eth0"
            Interface "br-eth0"
                type: internal
        Port "br-eth0--br-prv"
            Interface "br-eth0--br-prv"
                type: patch
                options: {peer="br-prv--br-eth0"}
        Port "br-eth0--br-ex"
            tag: 104
            Interface "br-eth0--br-ex"
                type: patch
                options: {peer="br-ex--br-eth0"}
        Port "br-eth0--br-storage"
            tag: 105
            Interface "br-eth0--br-storage"
                type: patch
                options: {peer="br-storage--br-eth0"}
        Port "eth0"
            Interface "eth0"
    Bridge br-mgmt
        Port br-mgmt
            Interface br-mgmt
                type: internal
        Port "br-mgmt--br-eth0"
            Interface "br-mgmt--br-eth0"
                type: patch
                options: {peer="br-eth0--br-mgmt"}

Not sure what you mean by inserting a static flow to a desired port, but
I've tried to do this earlier, though without any success:
ovs-ofctl add-flow br-eth0 in_port=4,actions=output:6 (4 is br-mgmt, 6 is
eth0)

tcpdumps, while trying to ping other host with VF enabled:

root at node-6:~# tcpdump -i br-mgmt
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-mgmt, link-type EN10MB (Ethernet), capture size 65535 bytes
18:04:06.545409 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:04:06.549458 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:04:07.545411 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:04:07.549437 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:04:08.567610 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:04:09.545788 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:04:09.565425 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:04:10.545406 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:04:10.565428 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:04:11.545405 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:04:11.582662 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:04:12.545460 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:04:12.581454 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:04:13.545440 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
(and so on)


root at node-6:~# tcpdump -i br-eth0
tcpdump: WARNING: br-eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:06:29.301447 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:30.319659 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:31.317444 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:32.317441 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:33.335671 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:34.333447 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:35.333444 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:36.350617 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:36.598210 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:06:37.349453 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:37.597440 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:06:38.349456 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:38.597411 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:06:39.366663 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:39.597453 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:06:40.365439 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:40.597447 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:06:41.365445 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:06:41.597410 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:06:42.382659 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
(and so on)

root at node-6:~# tcpdump -i eth0
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:07:45.637408 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:07:45.719618 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:07:46.637412 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:07:46.717445 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:07:47.098968 LACPv1, length 110
18:07:47.637450 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:07:47.717449 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:07:47.830263 LLDP, name HP, length 313
18:07:48.637411 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:07:48.735607 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:07:49.637406 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
18:07:49.733454 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:07:50.733445 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:07:51.751608 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:07:52.749456 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:07:53.749457 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:07:54.767539 ARP, Request who-has 192.168.0.2 tell node-6.domain.tld,
length 28
18:07:55.614723 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 28
(and so on)

meanwhile on node-1 (that has 192.168.0.2 and 192.168.0.3):
root at node-1:~# tcpdump -i br-mgmt
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-mgmt, link-type EN10MB (Ethernet), capture size 65535 bytes
18:11:57.479249 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 46
18:11:57.479285 ARP, Reply node-1.domain.tld is-at 96:e1:e9:7d:53:45 (oui
Unknown), length 28
18:11:58.479204 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 46
18:11:58.479245 ARP, Reply node-1.domain.tld is-at 96:e1:e9:7d:53:45 (oui
Unknown), length 28
18:11:59.497440 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 46
18:11:59.497473 ARP, Reply node-1.domain.tld is-at 96:e1:e9:7d:53:45 (oui
Unknown), length 28
18:12:00.495222 ARP, Request who-has node-1.domain.tld tell
node-6.domain.tld, length 46
18:12:00.495257 ARP, Reply node-1.domain.tld is-at 96:e1:e9:7d:53:45 (oui
Unknown), length 28

Seems like the reply doesn't make it to eth0 on node-6. I've tried running
tcpdump on eth6 (VF), but no packages are being captured.

On Wed, Nov 26, 2014 at 8:04 PM, Andrey Korolyov <andrey at xdel.ru> wrote:

> On Wed, Nov 26, 2014 at 6:54 PM, Dmitry Nikishov <dnikishov at mirantis.com>
> wrote:
> > I've just tried installing ovs-switch 2.3.0 and ovs-common 2.3.0, so
> > all the ovs packages versions are aligned. That didn't help: as long
> > as VFs are disabled, it works fine, but not when they are enabled.
> >
> > On Wed, Nov 26, 2014 at 4:27 PM, Andrey Korolyov <andrey at xdel.ru> wrote:
> >> On Wed, Nov 26, 2014 at 3:17 PM, Dmitry Nikishov <
> dnikishov at mirantis.com> wrote:
> >>> Hello all.
> >>>
> >>> I'm trying to get OVS working with Intel 82599 and SR-IOV.
> >>>
> >>> Basically what I currently have is an Ubuntu 12.04 with custom kernel
> >>> (3.14.23) and Intel 82599 NIC (eth0) and OVS 1.10 installed. There is
> >>> also openvswitch-datapath-dkms 2.3.0 which was built for 3.14.23.
> >>>
> >>> There is a br-eth0 bridge in OVS, which is connected to eth0
> >>> interface. Bridge br-mgmt is connected to br-eth0 and is using VLAN
> >>> tag 102.
> >>>
> >>> Network connectivity via br-mgmt -> br-eth0 -> eth0 chain work fine as
> >>> long as 82599 Virtual Functions (VFs) are disabled. Once I enable at
> >>> least 1 VF (echo 1 > /sys/bus/pci/devices/0000:04:00.0/sriov_numvfs),
> >>> server looses connectivity via this chain. If I try to set up VLAN 102
> >>> via ip link directly on eth0, it works.
> >>>
> >>> With the help of tcpdump I've found out that packets don't go any
> >>> further than br-eth0.
> >>>
> >>> Software versions:
> >>> Ubuntu 12.04 with custom kernel 3.14.23
> >>> openvswitch-switch 1.10.1+git20130823-0ubuntu3~cloud0
> >>> openvswitch-datapath-dkms 2.3.0
> >>>
> >>> On Ubuntu 12.04 with 3.11.8-generic, same openvswitch-switch and
> >>> openvswitch-datapath-dkms-lts-saucy there is no such issue.
> >>>
> >>> Any suggestions? Thanks.
> >>> _______________________________________________
> >>> discuss mailing list
> >>> discuss at openvswitch.org
> >>> http://openvswitch.org/mailman/listinfo/discuss
> >>
> >> May be it is better to align dp and vswitch versions first, or try
> >> against -saucy dp with same 3.14, if it is new enough to be built
> >> against it.
>
> As a VF initialization resulting in an effective interface
> reappearance with probably different udev name, did you checked that
> the interface with same name is still visible via netlink, sorry for a
> silly question? If so, try to re-add the interface first after VF
> initialization, if it will not help, please post 'ovs-vsctl list
> Interface eth0' output. Any hints for traffic flow, like inserting
> static flow with desired port# as a destination and looking into
> appropriate tcpdump output, if arp propagation is not likely working,
> are appreciated as well.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20141126/12d7f598/attachment-0002.html>


More information about the discuss mailing list