[ovs-discuss] Openvswitch first packets losts ?

Sébastien Riccio sr at swisscenter.com
Tue Jul 5 19:04:57 UTC 2011


On 05.07.2011 20:46, Justin Pettit wrote:
> Most of the packet processing intelligence happens in user-space, and the kernel module maintains a cache of the active flows.  Flows are expired from the cache after five seconds of inactivity.  My guess is that there is some sort of delay in the user-space processing.  I have not heard of such a long delay.  Are you running an off-box controller through OpenFlow?  Is there a lot of load that is preventing ovs-vswitchd from running?  What is the output of "ovs-vsctl list bridge" and "ovs-ofctl dump-flows<br>" for the various bridges?
>
> --Justin
>
>
> On Jul 5, 2011, at 12:18 AM, Sébastien Riccio wrote:
>
>> Hi,
>>
>> Being new to openvswitch I'm still trying to get my setup working like a charm.
>> I'm close to reach my goal but I've run into an annoying problem.
>>
>> When there is no activity on a bridge openvswitch miss the first packets
>> that are sent from or to the host, and I have no idea why :(
>>
>> Distrib: Debian 6
>> Kernel:  2.6.32-35
>>
>> hardware:
>> eth0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709S Gigabit Ethernet (rev 20)
>> eth1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709S Gigabit Ethernet (rev 20)
>> eth2 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709S Gigabit Ethernet (rev 20)
>> eth3 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709S Gigabit Ethernet (rev 20)
>> eth4 Ethernet controller: Broadcom Corporation NetXtreme II BCM57711 10-Gigabit PCIe
>> eth5 Ethernet controller: Broadcom Corporation NetXtreme II BCM57711 10-Gigabit PCIe
>>
>>
>> PIFs:
>>
>> eth0 trunk port
>> eth1 trunk port
>> eth2 iscsi path2
>> eth3 iscsi path1
>> eth4 unused yet yet
>> eth5 unused yet
>>
>>
>> What I did to setup networking:
>>
>> ifconfig eth0 up
>> ifconfig eth1 up
>> ifconfig eth2 up
>> ifconfig eth3 up
>>
>> ovs-vsctl add-br trunk0
>> ovs-vsctl add-bond trunk0 bond0 eth0 eth1
>> ovs-vsctl add-port trunk0 mgmt0 tag=85
>> ovs-vsctl set internface mgmt0 type=internal
>>
>> ovs-vsctl add-br stor0
>> ovs-vsctl add-port stor0 eth3
>>
>> ovs-vsctl add-br stor1
>> ovs-vsctl add-port stor1 eth2
>>
>> ifconfig mgmt0 10.111.10.116 netmask 255.255.0.0
>> ifconfig stor0 10.50.50.16 netmask 255.255.255.0
>> ifconfig stor1 10.50.60.16 netmask 255.255.255.0
>>
>>
>>
>> Testing bonded trunk:
>>
>> root at xen-blade16:~# ping 10.111.0.1
>> PING 10.111.0.1 (10.111.0.1) 56(84) bytes of data.
>> 64 bytes from 10.111.0.1: icmp_req=3 ttl=255 time=0.718 ms
>> 64 bytes from 10.111.0.1: icmp_req=4 ttl=255 time=0.426 ms
>> 64 bytes from 10.111.0.1: icmp_req=5 ttl=255 time=0.300 ms
>> 64 bytes from 10.111.0.1: icmp_req=6 ttl=255 time=0.356 ms
>> 64 bytes from 10.111.0.1: icmp_req=7 ttl=255 time=0.390 ms
>> 64 bytes from 10.111.0.1: icmp_req=8 ttl=255 time=0.346 ms
>> ^C
>> --- 10.111.0.1 ping statistics ---
>> 8 packets transmitted, 6 received, 25% packet loss, time 7014ms
>> rtt min/avg/max/mdev = 0.300/0.422/0.718/0.139 ms
>>
>>
>> Testing iscsi path1
>>
>> root at xen-blade16:~# ping 10.50.50.15
>> PING 10.50.50.15 (10.50.50.15) 56(84) bytes of data.
>> 64 bytes from 10.50.50.15: icmp_req=4 ttl=64 time=0.179 ms
>> 64 bytes from 10.50.50.15: icmp_req=5 ttl=64 time=0.181 ms
>> 64 bytes from 10.50.50.15: icmp_req=6 ttl=64 time=0.150 ms
>> 64 bytes from 10.50.50.15: icmp_req=7 ttl=64 time=0.164 ms
>> 64 bytes from 10.50.50.15: icmp_req=8 ttl=64 time=0.177 ms
>> ^C
>> --- 10.50.50.15 ping statistics ---
>> 8 packets transmitted, 5 received, 37% packet loss, time 7022ms
>> rtt min/avg/max/mdev = 0.150/0.170/0.181/0.014 ms
>>
>>
>> Testing iscsi path2
>>
>> root at xen-blade16:~# ping 10.50.60.15
>> PING 10.50.60.15 (10.50.60.15) 56(84) bytes of data.
>> 64 bytes from 10.50.60.15: icmp_req=4 ttl=64 time=0.163 ms
>> 64 bytes from 10.50.60.15: icmp_req=5 ttl=64 time=0.161 ms
>> 64 bytes from 10.50.60.15: icmp_req=6 ttl=64 time=0.165 ms
>> 64 bytes from 10.50.60.15: icmp_req=7 ttl=64 time=0.168 ms
>> 64 bytes from 10.50.60.15: icmp_req=8 ttl=64 time=0.154 ms
>> ^C
>> --- 10.50.60.15 ping statistics ---
>> 8 packets transmitted, 5 received, 37% packet loss, time 7026ms
>>
>>
>>
>> As you can see the first packets are dropped, either on the bonded
>> interface and on the "normal" interfaces. So doesn't seems to be
>> an issue with the bonding.
>>
>> Also if I ping, ctrl-c and then directly ping again, it doesn't miss
>> the first packets.
>>
>> Seems that after a few seconds of inactivity it "forgets" the paths.
>>
>>
>>
>> more infos:
>>
>> root at xen-blade16:~# ovs-dpctl show
>> system at trunk0:
>>         lookups: frags:0, hit:1141399, missed:377043, lost:0
>>         port 0: trunk0 (internal)
>>         port 1: eth1
>>         port 2: mgmt0 (internal)
>>         port 3: eth0
>> system at stor0:
>>         lookups: frags:0, hit:515, missed:68, lost:0
>>         port 0: stor0 (internal)
>>         port 1: eth3
>> system at stor1:
>>         lookups: frags:0, hit:501, missed:62, lost:0
>>         port 0: stor1 (internal)
>>         port 1: eth2
>>
>>
>> root at xen-blade16:~# ovs-appctl bond/show bond0
>> bond_mode: balance-slb
>> lacp: off
>> bond-hash-algorithm: balance-slb
>> bond-detect-mode: carrier
>> updelay: 0 ms
>> downdelay: 0 ms
>> next rebalance: 6578 ms
>>
>> slave eth0: enabled
>>         active slave
>>         hash 95: 0 kB load
>>                 00:23:20:d4:c7:7b
>>
>> slave eth1: enabled
>>
>>
>> root at xen-blade16:~# ovs-vsctl list port
>> _uuid               : 7c18f4b0-e644-45f2-81ba-cb588e8bdaf8
>> bond_downdelay      : 0
>> bond_fake_iface     : false
>> bond_mode           : []
>> bond_updelay        : 0
>> external_ids        : {}
>> fake_bridge         : false
>> interfaces          : [c956aabf-da4c-4d2b-970e-65338c842006]
>> lacp                : []
>> mac                 : []
>> name                : "trunk0"
>> other_config        : {}
>> qos                 : []
>> tag                 : []
>> trunks              : []
>>
>> _uuid               : df40b36c-c55b-406b-8613-cff1e59b35d6
>> bond_downdelay      : 0
>> bond_fake_iface     : false
>> bond_mode           : []
>> bond_updelay        : 0
>> external_ids        : {}
>> fake_bridge         : false
>> interfaces          : [e7f2fee3-93a3-4703-a818-500d91bff8b2]
>> lacp                : []
>> mac                 : []
>> name                : "mgmt0"
>> other_config        : {}
>> qos                 : []
>> tag                 : 85
>> trunks              : []
>>
>> _uuid               : 8ecce71c-7ff6-436f-a968-e6974d6ee100
>> bond_downdelay      : 0
>> bond_fake_iface     : false
>> bond_mode           : []
>> bond_updelay        : 0
>> external_ids        : {}
>> fake_bridge         : false
>> interfaces          : [34b58c6b-b97b-47c8-a57d-e8cef8ad918b]
>> lacp                : []
>> mac                 : []
>> name                : "stor0"
>> other_config        : {}
>> qos                 : []
>> tag                 : []
>> trunks              : []
>>
>> _uuid               : d435be0d-f371-4adf-ba50-3ff3989f7c18
>> bond_downdelay      : 0
>> bond_fake_iface     : false
>> bond_mode           : []
>> bond_updelay        : 0
>> external_ids        : {}
>> fake_bridge         : false
>> interfaces          : [485b7425-8ae5-4658-a65a-705b3f565934, 8f589dae-4d42-4cf0-b9e4-be7ab6693c4d]
>> lacp                : []
>> mac                 : []
>> name                : "bond0"
>> other_config        : {}
>> qos                 : []
>> tag                 : []
>> trunks              : []
>>
>> _uuid               : c3c55f92-04f9-4bdd-a119-9d851273e158
>> bond_downdelay      : 0
>> bond_fake_iface     : false
>> bond_mode           : []
>> bond_updelay        : 0
>> external_ids        : {}
>> fake_bridge         : false
>> interfaces          : [d55b71e6-e470-4921-8c44-f3154683ea7b]
>> lacp                : []
>> mac                 : []
>> name                : "eth3"
>> other_config        : {}
>> qos                 : []
>> tag                 : []
>> trunks              : []
>>
>> _uuid               : ec958047-ddbf-425f-97cc-dad30c5914c0
>> bond_downdelay      : 0
>> bond_fake_iface     : false
>> bond_mode           : []
>> bond_updelay        : 0
>> external_ids        : {}
>> fake_bridge         : false
>> interfaces          : [3cc9f429-5306-45ed-bfff-cd4cf9205748]
>> lacp                : []
>> mac                 : []
>> name                : "stor1"
>> other_config        : {}
>> qos                 : []
>> tag                 : []
>> trunks              : []
>>
>> _uuid               : 2d7b24b7-7049-47df-9d5f-60e6ff96ce10
>> bond_downdelay      : 0
>> bond_fake_iface     : false
>> bond_mode           : []
>> bond_updelay        : 0
>> external_ids        : {}
>> fake_bridge         : false
>> interfaces          : [fee9ee9c-d427-4c09-8182-1ea8492851c5]
>> lacp                : []
>> mac                 : []
>> name                : "eth2"
>> other_config        : {}
>> qos                 : []
>> tag                 : []
>> trunks              : []
>>
>>
>> Any ideas ? :)
>>
>> Thanks a lot for your help.
>>
>> Sébastien
>> _______________________________________________
>> discuss mailing list
>> discuss at openvswitch.org
>> http://openvswitch.org/mailman/listinfo/discuss
>

Hi Justin,

Thanks for your reply.

I'm not (yet) running an off-box controller. There is 0 load on the 
machine. The only process
that appears sometime in the top of "top" is the switchd daemon.
Is it a normal TIME+ count for the daemon, on a box up for one day 
without activity?

root at xen-blade16:~# uptime
  21:01:42 up 1 day,  5:12, 20 users,  load average: 0.01, 0.00, 0.00

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
2070 root      20   0 22608 2492 1144 S    0  0.0   3:13.89 ovs-vswitchd

root at xen-blade16:~# ovs-vsctl list bridge
_uuid               : 125a159c-b20f-4dfa-a7b4-d607db94f480
controller          : []
datapath_id         : "0000bc305b3883aa"
datapath_type       : ""
external_ids        : {}
fail_mode           : []
flood_vlans         : []
mirrors             : []
name                : "stor1"
netflow             : []
other_config        : {}
ports               : [2d7b24b7-7049-47df-9d5f-60e6ff96ce10, 
ec958047-ddbf-425f-97cc-dad30c5914c0]
sflow               : []

_uuid               : 9d01feb1-695c-413d-b333-e4532173b9f4
controller          : []
datapath_id         : "0000bc305b3883ac"
datapath_type       : ""
external_ids        : {}
fail_mode           : []
flood_vlans         : []
mirrors             : []
name                : "stor0"
netflow             : []
other_config        : {}
ports               : [8ecce71c-7ff6-436f-a968-e6974d6ee100, 
c3c55f92-04f9-4bdd-a119-9d851273e158]
sflow               : []

_uuid               : 353c2bb1-4ce0-4164-a1cb-3edb0553c809
controller          : []
datapath_id         : "0000bc305b3883a6"
datapath_type       : ""
external_ids        : {}
fail_mode           : []
flood_vlans         : []
mirrors             : []
name                : "trunk0"
netflow             : []
other_config        : {}
ports               : [7c18f4b0-e644-45f2-81ba-cb588e8bdaf8, 
d435be0d-f371-4adf-ba50-3ff3989f7c18, df40b36c-c55b-406b-8613-cff1e59b35d6]
sflow               : []

root at xen-blade16:~# ovs-ofctl dump-flows trunk0
NXST_FLOW reply (xid=0x4):
  cookie=0x0, duration=43370.807s, table_id=0, n_packets=1883934, 
n_bytes=143638556, priority=0 actions=NORMAL

root at xen-blade16:~# ovs-ofctl dump-flows stor0
NXST_FLOW reply (xid=0x4):
  cookie=0x0, duration=43389.711s, table_id=0, n_packets=91742, 
n_bytes=5651456, priority=0 actions=NORMAL

root at xen-blade16:~# ovs-ofctl dump-flows stor1
NXST_FLOW reply (xid=0x4):
  cookie=0x0, duration=43393.091s, table_id=0, n_packets=91796, 
n_bytes=5653888, priority=0 actions=NORMAL

Best regards,
Sébastien




More information about the discuss mailing list