[ovs-dev] OpenFlow rule deletion during port destroy

[Zoltan Kiss]

To elaborate a bit more: now I figured out that normally the controller 
should take care of removing OpenFlow rules when the port is deleted. I 
thought ovs-vswitchd checks the tables for any reference to the deleted 
port, but it seems it doesn't.
My actual problem is that an important rule gets deleted:

cookie=0x0, duration=1581.083s, table=0, n_packets=52804, 
n_bytes=88968151, idle_age=0, priority=0,in_port=ANY actions=NORMAL

This is important for the outgoing traffic of the LOCAL port, as flows 
from this rule end up there:

cookie=0x0, duration=1581.083s, table=0, n_packets=34466, 
n_bytes=83841901, idle_age=0, priority=32766,in_port=LOCAL 
actions=resubmit:ANY

I've printed out "ovs-dpctl dump-flows <bridge>|grep ANY" in every 
second, intertwined with the snoop of the OpenFlow traffic, and I can 
see when it happens:

Wed Apr 16 13:53:35 UTC 2014
  cookie=0x0, duration=1019.071s, table=0, n_packets=3642, 
n_bytes=3851860, idle_age=15, 
priority=65529,ip,in_port=4,dl_src=3e:e6:cf:83:83:57,nw_src=10.81.27.225 
actions=resubmit:ANY
  cookie=0x0, duration=1019.071s, table=0, n_packets=0, n_bytes=0, 
idle_age=1019, 
priority=65528,ip,in_port=4,dl_src=3e:e6:cf:83:83:57,nw_src=10.81.27.225 
actions=resubmit:ANY
  cookie=0x0, duration=1580.063s, table=0, n_packets=52791, 
n_bytes=88962010, idle_age=0, priority=0,in_port=ANY actions=NORMAL
  cookie=0x0, duration=1580.063s, table=0, n_packets=34453, 
n_bytes=83835760, idle_age=0, priority=32766,in_port=LOCAL 
actions=resubmit:ANY
  cookie=0x0, duration=1019.071s, table=0, n_packets=56, n_bytes=4718, 
idle_age=15, 
priority=65530,udp,in_port=4,dl_src=3e:e6:cf:83:83:57,nw_src=10.81.27.225,tp_dst=53 
actions=resubmit:ANY
...
OFPST_TABLE request (xid=0x0):
OFPST_TABLE reply (xid=0x0): 254 tables
...
OFPT_PORT_STATUS (xid=0x0): MOD: 4(vif3.0): addr:fe:ff:ff:ff:ff:ff
      config:     0
      state:      LINK_DOWN
      speed: 0 Mbps now, 0 Mbps max
OFPT_PORT_STATUS (xid=0x0): DEL: 4(vif3.0): addr:fe:ff:ff:ff:ff:ff
      config:     0
      state:      LINK_DOWN
      speed: 0 Mbps now, 0 Mbps max
OFPT_FLOW_MOD (xid=0x0): DEL dl_dst=3e:e6:cf:83:83:57 actions=drop
OFPT_FLOW_MOD (xid=0x0): DEL dl_src=3e:e6:cf:83:83:57 actions=drop
OFPT_FLOW_MOD (xid=0x0): DEL in_port=4 actions=drop
Wed Apr 16 13:53:36 UTC 2014
  cookie=0x0, duration=1581.083s, table=0, n_packets=52804, 
n_bytes=88968151, idle_age=0, priority=0,in_port=ANY actions=NORMAL
  cookie=0x0, duration=1581.083s, table=0, n_packets=34466, 
n_bytes=83841901, idle_age=0, priority=32766,in_port=LOCAL 
actions=resubmit:ANY
Wed Apr 16 13:53:37 UTC 2014
  cookie=0x0, duration=1582.099s, table=0, n_packets=34492, 
n_bytes=83852746, idle_age=0, priority=32766,in_port=LOCAL 
actions=resubmit:ANY
Wed Apr 16 13:53:38 UTC 2014
  cookie=0x0, duration=1583.115s, table=0, n_packets=34504, 
n_bytes=83857060, idle_age=0, priority=32766,in_port=LOCAL 
actions=resubmit:ANY

And I set the ovs-vswitchd loglevel to DBG, but the only thing happens 
around there is the removal of a VM's ports during shutdown:

Apr 16 13:53:36 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as 
/usr/bin/ovs-vsctl --timeout=30 -- --if-exists del-port vif3.1
Apr 16 13:53:36 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as 
/usr/bin/ovs-vsctl --timeout=30 -- --if-exists del-port vif3.0

There is no sign the controller did anything about deleting those rules, 
but somehow it still happened. Does anyone knows
Unfortunately it is hard to reproduce the problem, it is only 
intermittent in one of our testcases.

Regards,

Zoli


On 16/04/14 00:59, Zoltan Kiss wrote:
> I came accross a problem recently, where the destroying of a VM's port
> (with ovs-vsctl del-port) seems to blow away other OpenFlow rules from
> the userspace. It seems to me logical for ovs-vswitchd to delete the
> associated OpenFlow rules when the port is deleted, but I couldn't find
> in the code where it happens. And as far as I can see in the OpenFlow
> connection's monitoring, the controller doesn't do it either.
> So my question is, who does delete the rules associated to a port during
> teardown? I might have overlooked something in the ovs-vswitchd code,
> and it happens deferred somewhere else, but can someone point me there?