[ovs-discuss] ARP Behavior in XenServer Host

[David Erickson]

On 9/9/2011 9:54 AM, Ben Pfaff wrote:
> On Fri, Sep 09, 2011 at 09:36:25AM -0700, David Erickson wrote:
>> On 9/9/2011 9:16 AM, Ben Pfaff wrote:
>>> On Thu, Sep 08, 2011 at 06:11:39PM -0700, David Erickson wrote:
>>>> Hi All-
>>>> I'm seeing some confusing behavior happening relating to ARPs and
>>>> OVS 1.2.1 (also on 1.1.1 and probably 1.0.1). This is a XS machine
>>>> with 1 ethernet port, so in-band connection from OVS to the
>>>> controller. Specifically there are two cases where I would expect
>>>> packets to be sent up to the connected controller, but they aren't:
>>> By "sent up", do you mean "forwarded like an Ethernet switch" or "sent
>>> as OpenFlow packet-out messages"?
>> Sent to the controller as packet-ins.
>>
>>>> 1- VLAN Tagged ARP replies coming in eth0, attached as tagged_arp.pcap
>>> I think that these match in-band rule (b).  (The in-band control rules
>>> that OVS sets up wildcard the VLAN field.  We've never really thought
>>> that through, though.)
>> I think they do as well, however the problem is the local machine
>> completely ignores them because in my network they are vlan tagged,
>> so it is important that I am able to strip the vlan tag via a
>> controller-set rule before delivering them locally.
> For this particular case, I think that you can add an "internal" port
> with a VLAN tag and put your IP address on that:
>
> 	ovs-vsctl add-port br0 int0 tag=1234 -- set Interface int0 type=internal
> 	ifconfig int0 1.2.3.4

I'm not sure I understand how this would work, I want traffic to/from 
the XS host to be untagged, but the mesh uses VLAN tags to select the 
path, thus OVS needs to tag/untag packets traversing the mesh before 
delivering them to the XS host or guests.  The one exception being 
traffic to/from the gateway is fully untagged.  Is your suggestion to 
add an additional interface and another new ip for that specific tag?

> But the in-band control rules are necessarily designed to work with
> the most common ways that networks work.  You can't reasonably expect
> them to work for every case.

Completely agree.  I'm trying to understand the base assumptions to see 
what/how to change things to work for my use case.

>>>> 2- Gratuitous ARPs sent from the XS host, eg: /sbin/arping -U -c 1
>>>> -I xenbr0 -s 10.0.2.5 255.255.255.255, attached as
>>>> gratuitous_arp.pcap
>>> I think that these match in-band rule (c) described in DESIGN, so I'm
>>> surprised that these don't get forwarded.
>> I believe these do get forwarded, but my actual intent with sending
>> them was forced visibility to the controller, which is subverted by
>> the in-band rules that send these directly out eth0 rather than as
>> packet-ins to the controller.
>>
>> In the document it looks to me like rules b/c are used to facilitate
>> arp request/replies from the local host requesting from the remotes.
>> However they seem un-necessarily broad, ultimately encompassing all
>> arp request/replies coming from/to the XS Host.  Couldn't these
>> rules be replaced with much more narrow rules such as: ARP requests
>> containing remote's IP address as a target, ARP replies containing
>> the remote's IP address as a source?
> You might be right, but in-band control is remarkably subtle and it is
> easy to get it wrong.  Whenever we change anything in the design, it
> almost always causes problems for someone.  It looks to me like we
> haven't changed anything in the design in over 2 years now.  I'm not
> eager to make changes that are not well justified (at a minimum,
> explaining why they won't adversely affect any of the supported
> scenarios listed in DESIGN).

I'll call my two suggested rules:

(b*) ARP replies containing the remote's IP address as a source
(c*) ARP requests containing remote's IP address as a target

Where remote is defined as the controller's IP address if on the same subnet or the next hop towards the controller if not on the same subnet.


    - Locally Connected.  The switch and remote are on the same
      subnet.  This uses rules (a), (b), (c), (h), and (i).

---
Here the host will optionally send a dhcp request (a), followed by an 
arp request to the controller (c), receive an arp reply from the 
controller (b), and setup a tcp connection to the controller (h,i). 
Periodically the controller may issue an arp request to the switch (g), 
and the host will send a reply (f).  Note (c) will contain the 
controller's IP as target, and can be replaced by (c*), and (b) will 
contain the controller's IP as source and can be replaced by (b*)


    - Reached through Gateway.  The switch and remote are on
      different subnets and must go through a gateway.  This uses
      rules (a), (b), (c), (h), and (i).
---
Here the host will optionally send a dhcp request (a), followed by an 
arp request to the next hop for the controller (c), receive an arp reply 
from the next hop  (b), and setup a tcp connection to the controller 
(h,i). Periodically the gateway may issue an arp request to the switch 
(g), and the host will send a reply (f). Note (c) will contain the next 
hop's IP as target, and can be replaced by (c*), and (b) will contain 
the next hop's IP as source and can be replaced by (b*)


    - Between Switch and Remote.  This switch is between another
      switch and the remote, and we want to allow the other
      switch's traffic through.  This uses rules (d), (e), (h), and
      (i).  It uses (b) and (c) indirectly in order to know the MAC
      address for rules (d) and (e).  Note that DHCP for the other
      switch will not work unless an OpenFlow controller explicitly lets 
this
      switch pass the traffic.

---
This intermediate switch (s1) will follow the logic in "Locally 
Connected" above to determine the MAC address of the remote. At this 
point the second switch (s2) will send an arp request containing the 
remote's IP as target (I actually don't see which rule this matches in 
the above design description although my c* rule would match this). An 
arp reply containing the remote's IP as source will return (similar, b* 
would match this). At this point s2 will establish a tcp connection to 
the remote (h,i). Periodically the controller may issue an arp request 
to the switch (g), and the host will send a reply (f).


    - Between Switch and Gateway.  This switch is between another
      switch and the gateway, and we want to allow the other switch's
      traffic through.  This uses the same rules and logic as the
      "Between Switch and Remote" configuration described earlier.
---
Similar to above

    - Remote on Local VM.  The remote is a guest VM on the
      system running in-band control.  This uses rules (a), (b), (c),
      (h), and (i).
---
Same as "Locally Connected"

    - Remote on Local VM with Different Networks.  The remote
      is a guest VM on the system running in-band control, but the
      local port is not used to connect to the remote.  For
      example, an IP address is configured on eth0 of the switch.  The
      remote's VM is connected through eth1 of the switch, but an
      IP address has not been configured for that port on the switch.
      As such, the switch will use eth0 to connect to the remote,
      and eth1's rules about the local port will not work.  In the
      example, the switch attached to eth0 would use rules (a), (b),
      (c), (h), and (i) on eth0.  The switch attached to eth1 would use
      rules (f), (g), (h), and (i).
---
For eth0 this is the same as "Reached through Gateway" case above, the 
rules on eth1 are unmodified.

>> Also I understand these rules are for safety in attempting to
>> prevent anyone from interupting the connection to the controller,
>> but it would be nice if there was a way to override the default
>> priority space of these rules (other than recompiling the source of
>> course) so the controller could insert rules at higher priority if
>> it desired.
> You will probably get your wish.  We are talking about moving these
> rules out of the default flow table with id 0 into a separate flow
> table (e.g. table 254), at which point they will become visible to and
> modifiable over OpenFlow.
>
> I'd also be open to some kind of template-based configuration for
> rules, but I'm not planning to implement it myself.

Great!

Thanks,
David