[ovs-discuss] STP / VXLAN mesh: ETH0, Connection lost to OVS 2.3.1 server

Thu Feb 5 23:23:51 UTC 2015

Folks,

I have a small testbed with 4 ubuntu servers, fully meshed with VXLAN. To avoid loops I use Spanning-Tree.

The 4 servers run Ubuntu 14.10 fully updated.

I upgraded manually to openvswitch 2.3.1 with the vivid packages from ubuntu.

3 Servers run kvm and consume via rbd/ceph images. CEPH runs over openvswitch to sync data.

The 3 KVM servers share via VXLAN flat networks trunked with VLAN's.

These physical servers are single homed and therefore I need to bridge eth0 also to the outside world.

Randomly these servers loose connectivity to the outside world (eth0). Troubleshooting shows, that it is often related to STP events (VM's restarting/shutdown/hosts booting, etc).

After very long searches I think I found the culprit: A lock file ".conf.db.~lock~" is present in /etc/openvswitch directory. If I delete that file and reboot the host connectivity reappears.

When the host looses connectivity, no ipv4 traffic is forwarded anymore. No real culprits are visible in ovs-vswitch logs, except that a STP topology change happened.

local ovsbr0p1 bridged to eth0 interface can be ipv4 pinged. Gateway cannot be pinged anymore (sometimes ipv6 connectivity still works)

Any tips how to troubleshoot this and get rid of this sudden loss of connectivity ? Same problem happened on OVS 2.0 release.

Second question: How can I get rid of SPT events when VM's boot or similar ? On HW switches there is a concept of Access-Ports for end devices, where no STP event will be flood to the fabric.

thanks
marco

Below some outputs that could help in troubleshooting:

root at nuv-vir-kvm-server-3 ~ # ovs-vsctl show
5733eb7c-f4ac-446c-bfd1-c0121c0337b7
    Bridge "ovsbr0"
        Port "eth0"
            Interface "eth0"
        Port "ovsbr0p1"
            Interface "ovsbr0p1"
                type: internal
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
    Bridge "ovsbr1"
        Port "ovsbr1.8p1"
            tag: 8
            Interface "ovsbr1.8p1"
                type: internal
        Port "ovsbr1.12p1"
            tag: 12
            Interface "ovsbr1.12p1"
                type: internal
        Port "ovsbr1.3p1"
            tag: 3
            Interface "ovsbr1.3p1"
                type: internal
        Port "ovsbr1.4"
            tag: 4
            Interface "ovsbr1.4"
                type: internal
        Port "ovsbr1.101p1"
            tag: 101
            Interface "ovsbr1.101p1"
                type: internal
        Port "ovsvx5"
            trunks: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 20, 30, 101]
            Interface "ovsvx5"
                type: vxlan
                options: {key=flow, remote_ip="46.4.100.54"}
        Port "ovsbr1.1"
            tag: 1
            Interface "ovsbr1.1"
                type: internal
        Port "ovsbr1.9"
            tag: 9
            Interface "ovsbr1.9"
                type: internal
        Port "ovsbr1.2"
            tag: 2
            Interface "ovsbr1.2"
                type: internal
        Port "ovsbr1.20"
            tag: 20
            Interface "ovsbr1.20"
                type: internal
        Port "ovsbr1.30p1"
            tag: 30
            Interface "ovsbr1.30p1"
                type: internal
        Port "ovsbr1.101"
            tag: 101
            Interface "ovsbr1.101"
                type: internal
        Port "ovsbr1.5"
            tag: 5
            Interface "ovsbr1.5"
                type: internal
        Port "ovsbr1.7"
            tag: 7
            Interface "ovsbr1.7"
                type: internal
        Port "ovsbr1.1p1"
            tag: 1
            Interface "ovsbr1.1p1"
                type: internal
        Port "ovsbr1.6"
            tag: 6
            Interface "ovsbr1.6"
                type: internal
        Port "ovsbr1.11p1"
            tag: 11
            Interface "ovsbr1.11p1"
                type: internal
        Port "ovsbr1.4p1"
            tag: 4
            Interface "ovsbr1.4p1"
                type: internal
        Port "ovsbr1"
            Interface "ovsbr1"
                type: internal
        Port "ovsbr1.8"
            tag: 8
            Interface "ovsbr1.8"
                type: internal
        Port "ovsbr1.12"
            tag: 12
            Interface "ovsbr1.12"
                type: internal
        Port "ovsbr1.11"
            tag: 11
            Interface "ovsbr1.11"
                type: internal
        Port "ovsbr1.10p1"
            tag: 10
            Interface "ovsbr1.10p1"
                type: internal
        Port "ovsbr1.7p1"
            tag: 7
            Interface "ovsbr1.7p1"
                type: internal
        Port "ovsbr1.2p1"
            tag: 2
            Interface "ovsbr1.2p1"
                type: internal
        Port "ovsbr1.9p1"
            tag: 9
            Interface "ovsbr1.9p1"
                type: internal
        Port "ovsbr1.30"
            tag: 30
            Interface "ovsbr1.30"
                type: internal
        Port "ovsvxnuv-vir-aci-ubuntu-1"
            trunks: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 20, 30, 101]
            Interface "ovsvxnuv-vir-aci-ubuntu-1"
                type: vxlan
                options: {key=flow, remote_ip="46.4.17.109"}
        Port "ovsbr1.10"
            tag: 10
            Interface "ovsbr1.10"
                type: internal
        Port "ovsvx7"
            trunks: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 20, 30, 101]
            Interface "ovsvx7"
                type: vxlan
                options: {key=flow, remote_ip="46.4.17.67"}
        Port "ovsbr1.5p1"
            tag: 5
            Interface "ovsbr1.5p1"
                type: internal
        Port "ovsvx6"
            trunks: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 20, 30, 101]
            Interface "ovsvx6"
                type: vxlan
                options: {key=flow, remote_ip="46.4.109.146"}
        Port "ovsbr1.6p1"
            tag: 6
            Interface "ovsbr1.6p1"
                type: internal
        Port "ovsbr1.3"
            tag: 3
            Interface "ovsbr1.3"
                type: internal
        Port "ovsbr1.20p1"
            tag: 20
            Interface "ovsbr1.20p1"
                type: internal
    ovs_version: "2.3.1"

_uuid               : c960a8da-aa1a-4c8a-a3f3-1b7863424313
bond_active_slave   : []
bond_downdelay      : 0
bond_fake_iface     : false
bond_mode           : []
bond_updelay        : 0
external_ids        : {}
fake_bridge         : false
interfaces          : [e8ec16b3-1a83-44a0-b3f8-f3dccfdde91d]
lacp                : []
mac                 : []
name                : "eth0"
other_config        : {}
qos                 : []
statistics          : {}
status              : {}
tag                 : []
trunks              : []
vlan_mode           : []

root at nuv-vir-kvm-server-5 ~ # ovs-vsctl list port
_uuid               : 6b14b501-a85b-4fc0-8cd3-1302830ff35c
bond_active_slave   : []
bond_downdelay      : 0
bond_fake_iface     : false
bond_mode           : []
bond_updelay        : 0
external_ids        : {}
fake_bridge         : false
interfaces          : [7dfc6aa1-23bf-4784-98c3-583ea20884c6]
lacp                : []
mac                 : []
name                : "ovsbr1.1p1"
other_config        : {}
qos                 : []
statistics          : {}
status              : {}
tag                 : 1
trunks              : []
vlan_mode           : []

_uuid               : eec4c406-3cf5-4507-9ca3-251741631e77
bond_active_slave   : []
bond_downdelay      : 0
bond_fake_iface     : false
bond_mode           : []
bond_updelay        : 0
external_ids        : {}
fake_bridge         : false
interfaces          : [8e88f0c9-31e0-4c44-81b8-53511ab8d8bd]
lacp                : []
mac                 : []
name                : "ovsbr1"
other_config        : {}
qos                 : []
statistics          : {}
status              : {}
tag                 : []
trunks              : []
vlan_mode           : []

_uuid               : a191cc12-89d3-4a79-bdaf-55502c5b213d
bond_active_slave   : []
bond_downdelay      : 0
bond_fake_iface     : false
bond_mode           : []
bond_updelay        : 0
external_ids        : {}
fake_bridge         : false
interfaces          : [4281866e-11b4-4c04-8a6c-066aea7af246]
lacp                : []
mac                 : []
name                : "ovsbr0p1"
other_config        : {}
qos                 : []
statistics          : {}
status              : {}
tag                 : []
trunks              : []
vlan_mode           : []

_uuid               : 22ee48dc-7fec-4742-b577-a5293f7e3bd9
admin_state         : up
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
external_ids        : {}
ifindex             : 5
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 1
link_speed          : []
link_state          : up
mac                 : []
mac_in_use          : "c8:60:00:5f:f0:3f"
mtu                 : 1500
name                : "ovsbr0"
ofport              : 65534
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {collisions=0, rx_bytes=360, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=4, tx_bytes=44058, tx_dropped=0, tx_errors=0, tx_packets=328}
status              : {driver_name=openvswitch}
type                : internal

_uuid               : e8ec16b3-1a83-44a0-b3f8-f3dccfdde91d
admin_state         : up
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : full
external_ids        : {}
ifindex             : 2
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 1
link_speed          : 1000000000
link_state          : up
mac                 : []
mac_in_use          : "c8:60:00:5f:f0:3f"
mtu                 : 1500
name                : "eth0"
ofport              : 2
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {collisions=0, rx_bytes=1381410080, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=6910253, tx_bytes=1055169046, tx_dropped=0, tx_errors=0, tx_packets=2504716}
status              : {driver_name="r8169", driver_version="2.3LK-NAPI", firmware_version="rtl_nic/rtl8168e-2.fw"}
type                : ""

_uuid               : 4281866e-11b4-4c04-8a6c-066aea7af246
admin_state         : up
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
external_ids        : {}
ifindex             : 4
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 1
link_speed          : []
link_state          : up
mac                 : []
mac_in_use          : "62:ee:a5:83:b5:7e"
mtu                 : 1500
name                : "ovsbr0p1"
ofport              : 1
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {collisions=0, rx_bytes=1053520420, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=2474191, tx_bytes=9971342322, tx_dropped=0, tx_errors=0, tx_packets=6910215}
status              : {driver_name=openvswitch}
type                : internal

_uuid               : 7dfc6aa1-23bf-4784-98c3-583ea20884c6
admin_state         : up
bfd                 : {}
bfd_status          : {}
cfm_fault           : []
cfm_fault_status    : []
cfm_flap_count      : []
cfm_health          : []
cfm_mpid            : []
cfm_remote_mpids    : []
cfm_remote_opstate  : []
duplex              : []
external_ids        : {}
ifindex             : 36
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current        : []
link_resets         : 1
link_speed          : []
link_state          : up
mac                 : []
mac_in_use          : "6e:e8:ec:44:f9:1a"
mtu                 : 1400
name                : "ovsbr1.1p1"
ofport              : 2
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {collisions=0, rx_bytes=800790181, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=2044691, tx_bytes=9595662921, tx_dropped=0, tx_errors=0, tx_packets=6887123}
status              : {driver_name=openvswitch}
type                : internal

_uuid               : 8c602077-b2b1-4f2c-9201-93304008a814
bond_active_slave   : []
bond_downdelay      : 0
bond_fake_iface     : false
bond_mode           : []
bond_updelay        : 0
external_ids        : {}
fake_bridge         : false
interfaces          : [18cb6a6c-1f66-4eb5-bc78-304e8b404ac0]
lacp                : []
mac                 : []
name                : "ovsvx3"
other_config        : {}
qos                 : []
statistics          : {stp_error_count=0, stp_rx_count=311, stp_tx_count=4}
status              : {stp_port_id="8002", stp_role=root, stp_sec_in_state="592", stp_state=forwarding}
tag                 : []
trunks              : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 20, 30, 101]
vlan_mode           : trunk