[ovs-dev] [PATCH] DESIGN: Move in-band control design discussion here.

Justin Pettit jpettit at nicira.com
Wed May 4 22:27:26 UTC 2011


Good idea.  Looks fine to me.

--Justin


On May 4, 2011, at 2:04 PM, Ben Pfaff wrote:

> It seems more likely that interested users and administrators will be able
> to find it here.
> ---
> DESIGN            |  163 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> ofproto/in-band.c |  160 ----------------------------------------------------
> 2 files changed, 163 insertions(+), 160 deletions(-)
> 
> diff --git a/DESIGN b/DESIGN
> index 6e25f01..2e3fced 100644
> --- a/DESIGN
> +++ b/DESIGN
> @@ -71,6 +71,169 @@ nodes that do not connect to link with such large MTUs.  Currently, Open
> vSwitch doesn't process jumbograms.
> 
> 
> +In-Band Control
> +===============
> +
> +In-band control allows a single network to be used for OpenFlow traffic and
> +other data traffic.  See ovs-vswitchd.conf.db(5) for a description of
> +configuring in-band control.
> +
> +This comment is an attempt to describe how in-band control works at a
> +wire- and implementation-level.  Correctly implementing in-band
> +control has proven difficult due to its many subtleties, and has thus
> +gone through many iterations.  Please read through and understand the
> +reasoning behind the chosen rules before making modifications.
> +
> +In Open vSwitch, in-band control is implemented as "hidden" flows (in that
> +they are not visible through OpenFlow) and at a higher priority than
> +wildcarded flows can be set up by through OpenFlow.  This is done so that
> +the OpenFlow controller cannot interfere with them and possibly break
> +connectivity with its switches.  It is possible to see all flows, including
> +in-band ones, with the ovs-appctl "bridge/dump-flows" command.
> +
> +The Open vSwitch implementation of in-band control can hide traffic to
> +arbitrary "remotes", where each remote is one TCP port on one IP address.
> +Currently the remotes are automatically configured as the in-band OpenFlow
> +controllers plus the OVSDB managers, if any.  (The latter is a requirement
> +because OVSDB managers are responsible for configuring OpenFlow controllers,
> +so if the manager cannot be reached then OpenFlow cannot be reconfigured.)
> +
> +The following rules (with the OFPP_NORMAL action) are set up on any bridge
> +that has any remotes:
> +
> +   (a) DHCP requests sent from the local port.
> +   (b) ARP replies to the local port's MAC address.
> +   (c) ARP requests from the local port's MAC address.
> +
> +In-band also sets up the following rules for each unique next-hop MAC
> +address for the remotes' IPs (the "next hop" is either the remote
> +itself, if it is on a local subnet, or the gateway to reach the remote):
> +
> +   (d) ARP replies to the next hop's MAC address.
> +   (e) ARP requests from the next hop's MAC address.
> +
> +In-band also sets up the following rules for each unique remote IP address:
> +
> +   (f) ARP replies containing the remote's IP address as a target.
> +   (g) ARP requests containing the remote's IP address as a source.
> +
> +In-band also sets up the following rules for each unique remote (IP,port)
> +pair:
> +
> +   (h) TCP traffic to the remote's IP and port.
> +   (i) TCP traffic from the remote's IP and port.
> +
> +The goal of these rules is to be as narrow as possible to allow a
> +switch to join a network and be able to communicate with the
> +remotes.  As mentioned earlier, these rules have higher priority
> +than the controller's rules, so if they are too broad, they may
> +prevent the controller from implementing its policy.  As such,
> +in-band actively monitors some aspects of flow and packet processing
> +so that the rules can be made more precise.
> +
> +In-band control monitors attempts to add flows into the datapath that
> +could interfere with its duties.  The datapath only allows exact
> +match entries, so in-band control is able to be very precise about
> +the flows it prevents.  Flows that miss in the datapath are sent to
> +userspace to be processed, so preventing these flows from being
> +cached in the "fast path" does not affect correctness.  The only type
> +of flow that is currently prevented is one that would prevent DHCP
> +replies from being seen by the local port.  For example, a rule that
> +forwarded all DHCP traffic to the controller would not be allowed,
> +but one that forwarded to all ports (including the local port) would.
> +
> +As mentioned earlier, packets that miss in the datapath are sent to
> +the userspace for processing.  The userspace has its own flow table,
> +the "classifier", so in-band checks whether any special processing
> +is needed before the classifier is consulted.  If a packet is a DHCP
> +response to a request from the local port, the packet is forwarded to
> +the local port, regardless of the flow table.  Note that this requires
> +L7 processing of DHCP replies to determine whether the 'chaddr' field
> +matches the MAC address of the local port.
> +
> +It is interesting to note that for an L3-based in-band control
> +mechanism, the majority of rules are devoted to ARP traffic.  At first
> +glance, some of these rules appear redundant.  However, each serves an
> +important role.  First, in order to determine the MAC address of the
> +remote side (controller or gateway) for other ARP rules, we must allow
> +ARP traffic for our local port with rules (b) and (c).  If we are
> +between a switch and its connection to the remote, we have to
> +allow the other switch's ARP traffic to through.  This is done with
> +rules (d) and (e), since we do not know the addresses of the other
> +switches a priori, but do know the remote's or gateway's.  Finally,
> +if the remote is running in a local guest VM that is not reached
> +through the local port, the switch that is connected to the VM must
> +allow ARP traffic based on the remote's IP address, since it will
> +not know the MAC address of the local port that is sending the traffic
> +or the MAC address of the remote in the guest VM.
> +
> +With a few notable exceptions below, in-band should work in most
> +network setups.  The following are considered "supported' in the
> +current implementation:
> +
> +   - Locally Connected.  The switch and remote are on the same
> +     subnet.  This uses rules (a), (b), (c), (h), and (i).
> +
> +   - Reached through Gateway.  The switch and remote are on
> +     different subnets and must go through a gateway.  This uses
> +     rules (a), (b), (c), (h), and (i).
> +
> +   - Between Switch and Remote.  This switch is between another
> +     switch and the remote, and we want to allow the other
> +     switch's traffic through.  This uses rules (d), (e), (h), and
> +     (i).  It uses (b) and (c) indirectly in order to know the MAC
> +     address for rules (d) and (e).  Note that DHCP for the other
> +     switch will not work unless an OpenFlow controller explicitly lets this
> +     switch pass the traffic.
> +
> +   - Between Switch and Gateway.  This switch is between another
> +     switch and the gateway, and we want to allow the other switch's
> +     traffic through.  This uses the same rules and logic as the
> +     "Between Switch and Remote" configuration described earlier.
> +
> +   - Remote on Local VM.  The remote is a guest VM on the
> +     system running in-band control.  This uses rules (a), (b), (c),
> +     (h), and (i).
> +
> +   - Remote on Local VM with Different Networks.  The remote
> +     is a guest VM on the system running in-band control, but the
> +     local port is not used to connect to the remote.  For
> +     example, an IP address is configured on eth0 of the switch.  The
> +     remote's VM is connected through eth1 of the switch, but an
> +     IP address has not been configured for that port on the switch.
> +     As such, the switch will use eth0 to connect to the remote,
> +     and eth1's rules about the local port will not work.  In the
> +     example, the switch attached to eth0 would use rules (a), (b),
> +     (c), (h), and (i) on eth0.  The switch attached to eth1 would use
> +     rules (f), (g), (h), and (i).
> +
> +The following are explicitly *not* supported by in-band control:
> +
> +   - Specify Remote by Name.  Currently, the remote must be
> +     identified by IP address.  A naive approach would be to permit
> +     all DNS traffic.  Unfortunately, this would prevent the
> +     controller from defining any policy over DNS.  Since switches
> +     that are located behind us need to connect to the remote,
> +     in-band cannot simply add a rule that allows DNS traffic from
> +     the local port.  The "correct" way to support this is to parse
> +     DNS requests to allow all traffic related to a request for the
> +     remote's name through.  Due to the potential security
> +     problems and amount of processing, we decided to hold off for
> +     the time-being.
> +
> +   - Differing Remotes for Switches.  All switches must know
> +     the L3 addresses for all the remotes that other switches
> +     may use, since rules need to be set up to allow traffic related
> +     to those remotes through.  See rules (f), (g), (h), and (i).
> +
> +   - Differing Routes for Switches.  In order for the switch to
> +     allow other switches to connect to a remote through a
> +     gateway, it allows the gateway's traffic through with rules (d)
> +     and (e).  If the routes to the remote differ for the two
> +     switches, we will not know the MAC address of the alternate
> +     gateway.
> +
> +
> Suggestions
> ===========
> 
> diff --git a/ofproto/in-band.c b/ofproto/in-band.c
> index e75d19e..9aa03af 100644
> --- a/ofproto/in-band.c
> +++ b/ofproto/in-band.c
> @@ -40,166 +40,6 @@
> 
> VLOG_DEFINE_THIS_MODULE(in_band);
> 
> -/* In-band control allows a single network to be used for OpenFlow traffic and
> - * other data traffic.  See ovs-vswitchd.conf.db(5) for a description of
> - * configuring in-band control.
> - *
> - * This comment is an attempt to describe how in-band control works at a
> - * wire- and implementation-level.  Correctly implementing in-band
> - * control has proven difficult due to its many subtleties, and has thus
> - * gone through many iterations.  Please read through and understand the
> - * reasoning behind the chosen rules before making modifications.
> - *
> - * In Open vSwitch, in-band control is implemented as "hidden" flows (in that
> - * they are not visible through OpenFlow) and at a higher priority than
> - * wildcarded flows can be set up by through OpenFlow.  This is done so that
> - * the OpenFlow controller cannot interfere with them and possibly break
> - * connectivity with its switches.  It is possible to see all flows, including
> - * in-band ones, with the ovs-appctl "bridge/dump-flows" command.
> - *
> - * The Open vSwitch implementation of in-band control can hide traffic to
> - * arbitrary "remotes", where each remote is one TCP port on one IP address.
> - * Currently the remotes are automatically configured as the in-band OpenFlow
> - * controllers plus the OVSDB managers, if any.  (The latter is a requirement
> - * because OVSDB managers are responsible for configuring OpenFlow controllers,
> - * so if the manager cannot be reached then OpenFlow cannot be reconfigured.)
> - *
> - * The following rules (with the OFPP_NORMAL action) are set up on any bridge
> - * that has any remotes:
> - *
> - *    (a) DHCP requests sent from the local port.
> - *    (b) ARP replies to the local port's MAC address.
> - *    (c) ARP requests from the local port's MAC address.
> - *
> - * In-band also sets up the following rules for each unique next-hop MAC
> - * address for the remotes' IPs (the "next hop" is either the remote
> - * itself, if it is on a local subnet, or the gateway to reach the remote):
> - *
> - *    (d) ARP replies to the next hop's MAC address.
> - *    (e) ARP requests from the next hop's MAC address.
> - *
> - * In-band also sets up the following rules for each unique remote IP address:
> - *
> - *    (f) ARP replies containing the remote's IP address as a target.
> - *    (g) ARP requests containing the remote's IP address as a source.
> - *
> - * In-band also sets up the following rules for each unique remote (IP,port)
> - * pair:
> - *
> - *    (h) TCP traffic to the remote's IP and port.
> - *    (i) TCP traffic from the remote's IP and port.
> - *
> - * The goal of these rules is to be as narrow as possible to allow a
> - * switch to join a network and be able to communicate with the
> - * remotes.  As mentioned earlier, these rules have higher priority
> - * than the controller's rules, so if they are too broad, they may
> - * prevent the controller from implementing its policy.  As such,
> - * in-band actively monitors some aspects of flow and packet processing
> - * so that the rules can be made more precise.
> - *
> - * In-band control monitors attempts to add flows into the datapath that
> - * could interfere with its duties.  The datapath only allows exact
> - * match entries, so in-band control is able to be very precise about
> - * the flows it prevents.  Flows that miss in the datapath are sent to
> - * userspace to be processed, so preventing these flows from being
> - * cached in the "fast path" does not affect correctness.  The only type
> - * of flow that is currently prevented is one that would prevent DHCP
> - * replies from being seen by the local port.  For example, a rule that
> - * forwarded all DHCP traffic to the controller would not be allowed,
> - * but one that forwarded to all ports (including the local port) would.
> - *
> - * As mentioned earlier, packets that miss in the datapath are sent to
> - * the userspace for processing.  The userspace has its own flow table,
> - * the "classifier", so in-band checks whether any special processing
> - * is needed before the classifier is consulted.  If a packet is a DHCP
> - * response to a request from the local port, the packet is forwarded to
> - * the local port, regardless of the flow table.  Note that this requires
> - * L7 processing of DHCP replies to determine whether the 'chaddr' field
> - * matches the MAC address of the local port.
> - *
> - * It is interesting to note that for an L3-based in-band control
> - * mechanism, the majority of rules are devoted to ARP traffic.  At first
> - * glance, some of these rules appear redundant.  However, each serves an
> - * important role.  First, in order to determine the MAC address of the
> - * remote side (controller or gateway) for other ARP rules, we must allow
> - * ARP traffic for our local port with rules (b) and (c).  If we are
> - * between a switch and its connection to the remote, we have to
> - * allow the other switch's ARP traffic to through.  This is done with
> - * rules (d) and (e), since we do not know the addresses of the other
> - * switches a priori, but do know the remote's or gateway's.  Finally,
> - * if the remote is running in a local guest VM that is not reached
> - * through the local port, the switch that is connected to the VM must
> - * allow ARP traffic based on the remote's IP address, since it will
> - * not know the MAC address of the local port that is sending the traffic
> - * or the MAC address of the remote in the guest VM.
> - *
> - * With a few notable exceptions below, in-band should work in most
> - * network setups.  The following are considered "supported' in the
> - * current implementation:
> - *
> - *    - Locally Connected.  The switch and remote are on the same
> - *      subnet.  This uses rules (a), (b), (c), (h), and (i).
> - *
> - *    - Reached through Gateway.  The switch and remote are on
> - *      different subnets and must go through a gateway.  This uses
> - *      rules (a), (b), (c), (h), and (i).
> - *
> - *    - Between Switch and Remote.  This switch is between another
> - *      switch and the remote, and we want to allow the other
> - *      switch's traffic through.  This uses rules (d), (e), (h), and
> - *      (i).  It uses (b) and (c) indirectly in order to know the MAC
> - *      address for rules (d) and (e).  Note that DHCP for the other
> - *      switch will not work unless an OpenFlow controller explicitly lets this
> - *      switch pass the traffic.
> - *
> - *    - Between Switch and Gateway.  This switch is between another
> - *      switch and the gateway, and we want to allow the other switch's
> - *      traffic through.  This uses the same rules and logic as the
> - *      "Between Switch and Remote" configuration described earlier.
> - *
> - *    - Remote on Local VM.  The remote is a guest VM on the
> - *      system running in-band control.  This uses rules (a), (b), (c),
> - *      (h), and (i).
> - *
> - *    - Remote on Local VM with Different Networks.  The remote
> - *      is a guest VM on the system running in-band control, but the
> - *      local port is not used to connect to the remote.  For
> - *      example, an IP address is configured on eth0 of the switch.  The
> - *      remote's VM is connected through eth1 of the switch, but an
> - *      IP address has not been configured for that port on the switch.
> - *      As such, the switch will use eth0 to connect to the remote,
> - *      and eth1's rules about the local port will not work.  In the
> - *      example, the switch attached to eth0 would use rules (a), (b),
> - *      (c), (h), and (i) on eth0.  The switch attached to eth1 would use
> - *      rules (f), (g), (h), and (i).
> - *
> - * The following are explicitly *not* supported by in-band control:
> - *
> - *    - Specify Remote by Name.  Currently, the remote must be
> - *      identified by IP address.  A naive approach would be to permit
> - *      all DNS traffic.  Unfortunately, this would prevent the
> - *      controller from defining any policy over DNS.  Since switches
> - *      that are located behind us need to connect to the remote,
> - *      in-band cannot simply add a rule that allows DNS traffic from
> - *      the local port.  The "correct" way to support this is to parse
> - *      DNS requests to allow all traffic related to a request for the
> - *      remote's name through.  Due to the potential security
> - *      problems and amount of processing, we decided to hold off for
> - *      the time-being.
> - *
> - *    - Differing Remotes for Switches.  All switches must know
> - *      the L3 addresses for all the remotes that other switches
> - *      may use, since rules need to be set up to allow traffic related
> - *      to those remotes through.  See rules (f), (g), (h), and (i).
> - *
> - *    - Differing Routes for Switches.  In order for the switch to
> - *      allow other switches to connect to a remote through a
> - *      gateway, it allows the gateway's traffic through with rules (d)
> - *      and (e).  If the routes to the remote differ for the two
> - *      switches, we will not know the MAC address of the alternate
> - *      gateway.
> - */
> -
> /* Priorities used in classifier for in-band rules.  These values are higher
>  * than any that may be set with OpenFlow, and "18" kind of looks like "IB".
>  * The ordering of priorities is not important because all of the rules set up
> -- 
> 1.7.4.4
> 
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> http://openvswitch.org/mailman/listinfo/dev




More information about the dev mailing list