[ovs-dev] Way forward: L3 tunneling, NSH, Packet type-aware pipeline and Generic encap/decap

Jan Scheurich jan.scheurich at ericsson.com
Fri Nov 4 13:02:19 UTC 2016


Hi,

With this mail we would like to re-focus the long-pending discussions on how to best support new tunnel stacks such as MPLS in GRE and new tunneling protocols like NSH in OVS. 

Contributions addressing individual aspects of the problem complex (L3 tunneling, NSH) have been in the make for more than a year now and yet there is no acceptable overall solution in sight even for the known use cases.

Ericsson would like to engage more actively to help define and implement a comprehensive solution in OVS for a flexible and extensible support of tunneling protocols implemented either as tunnel ports, in OpenFlow, or both.

We would like to discuss our proposal on the OVS Conference next week and hope that we can reach agreement on the way forward.

Best regards, Jan


Situation:
----------
1. OpenFlow up to version 1.4 only supports L2 (Ethernet) packets. At any point in time a packet in the OF pipeline must have an Ethernet header. Particular push/pop actions were defined to insert/delete certain "tunnel" headers within the packet (VLAN, MPLS, PBB).

2. OpenFlow 1.5 standardized the "Packet type-aware pipeline" (PTAP, EXT-112) [1], which allows an OpenFlow switch to handle non-Ethernet packets. It introduces the new match field packet_type encoding the type of packet and new pre-requisites for existing match fields. PTAP is not yet implemented in OVS. Jean Tourrilhes did a prototype implementation in OVS at the time of standardization [8,9].

3. OF extension "EXT-382: Generic tunnel Encap and Decap" [2] was proposed in ONF as means for an OF controller to push/pop individual tunnel headers at the front of a packet to handle arbitrary stacks of tunnel encapsulations within OpenFlow. As the encap/decap actions would change the packet_type of the packet in the pipeline, EXT-382 builds on PTAP. EXT-382 is candidate for inclusion in upcoming OF 1.6.
 
4. Traditionally OVS implements tunnels as logical OF ports. Supported tunnel types include GRE, VXLAN, GENEVE, STT. As the datapath and OF pipeline in OVS are Ethernet-only, all tunnel ports are L2 tunnels, even though some of the tunnel protocols would support other payload (GRE, GENEVE or VXLAN-GPE). The concept of tunnel ports only works for IP encapsulation as OVS relies on IP routing to select the outgoing interface for encapsulated packets.
 
5. The need in OVS for non-L2 tunnels such as MPLS in GRE or LISP to interwork with external network equipment has driven the development of a series of patches known as "L3 Tunneling support" by Lorand Jakub, Thomas Morin, Simon Horman and Jiri Benc. For the latest version see [3,4]. This work has been going on with interruptions for a long time.

In essence these patches make OVS pop the Ethernet header when sending a packet to an "L3" port and to push a dummy Ethernet header with the correct Ethertype when receiving a packet from an "L3" port. 

However, the actual implementation carries the non-Ethernet packets received on an L3 tunnel all the way through the datapath and the ofproto-dpif pipeline, but without any visibility to the OF controller. The missing Ethernet header is only pushed implicitly at transmission to a normal Ethernet port. Conversely, an Ethernet header is stripped off just before sending to an L3 tunnel. ofproto-dpif inserts internal push/pop_eth actions into the datapath when needed. 

This adds a lot of complexity to these patches without reaping the benefit of supporting the packet type-aware pipeline. Without PTAP, however, it would be sufficient to let the L3 tunnel vport push and pop Ethernet headers so that datapath and ofproto-dpif would see Ethernet packets only.

The current patches also have conceptual problems. The controller can, for example, match on and set MAC address fields of the non-existing Ethernet header in an L3 packet, as these are cached in the packet's struct flow, but it is prevented from pushing and popping VLAN headers and setting VLAN IDs.

Another issue with the patches is that a given tunnel port instance can either carry Ethernet packets or non-Ethernet packets, but not both, even if the tunnel protocol would allow that. This is a real limitation as it requires the setup of two parallel tunnel meshes if both L2 and L3 payload is needed.

6. Since autumn 2015 Yi Yang and Johnson Li have been working on a set of patches for OVS to support NSH encapsulation for SFC [5]. NSH is a service tunnel header that can encapsulate different packet types, typically Ethernet or IP. For transport it requires an outer Ethernet header or a "L3" transport tunnel such as VXLAN-GPE or GRE that can carry NSH frames. A complication with NSH is that in MD2 format the header includes optional TLV metadata fields, similar but not quite identical to GENEVE.

The NSH work is stuck mainly because of the difficulty to handle the variable encapsulations. For VXLAN-GPE transport it depends on the L3-Tunneling support. Furthermore the OpenFlow pipeline must be able to push/pop an NSH header both with and without Ethernet header. This calls for separate actions to push/pop NSH and push/pop Ethernet headers, along the lines proposed in EXT-382, which in turn requires the PTAP.

The alternative, integrated actions to push/pop NSH and Ethernet headers together to not leave the Ethernet-only pipeline, was rejected by the OVS maintainers [7].


Technical proposal:
-------------------
It seems that the only acceptable way forward to support NSH in OVS is to implement PTAP and EXT-382 to push/pop Ethernet and NSH headers separately. This would anyway be a good investment because PTAP is a major feature of OF 1.5 and EXT-382 is the key to implementing arbitrary encapsulation stacks in OpenFlow. Together with the upcoming support for P4-specified protocols and match fields, this would make OVS fit for whole new class of applications.

With PTAP in place it would be natural to let "versatile" tunnel ports (tunnels which support multiple payloads, e.g. GRE, GENEVE, VXLAN-GPE) directly exchange their payload packets with the PTAP, mapping the tunnel's protocol field to packet_type and vice versa and dropping packets where no such mapping is defined. A single versatile tunnel port could thus carry Ethernet and non-Ethernet packets and there would be no need for an "L3" option to specify the behavior.

It appears desirable to keep the legacy Ethernet-only mode of OVS bridges as an option to maintain backward-compatibility with controllers that have not yet implemented PTAP and EXT-382. The question is whether such L2-only bridges must be limited to L2 tunnels or could support versatile tunnel ports as well. On ingress from a versatile tunnel port, the vport could push a dummy Ethernet header with an Ethertype derived from the tunnel's protocol field. On egress to the tunnel, however, we'd need an "L3" option on the vport to determine whether to pop the L2 header (including any VLAN tags) or not.


Suggested way forward:
----------------------
A) Implement PTAP based on Jean's prototype implementation [9]. PTAP introduces the packet_type match field in struct flow to explicitly track the type of packet. This is cleaner and more robust than the current implicit derivation from base_layer, Ethertype and layer offsets.

B) Reduce the scope of the current L3 tunneling patches to the vport implementation. Do the mapping  between tunnel protocol field and packet_type when connected to PTAP-enabled bridge. For non-PTAP bridges and "L3" tunnels let the vport push/pop Ethernet/L2 headers as needed.

C) Implement VXLAN-GPE as a new "versatile" tunnel port.

D) Implement the generic parts of EXT-382 to handle generic encap/decap actions.

E) As part of NSH patches implement the new packet_type NSH and generic encap and decap actions for NSH and ETH in the datapath. Use them to realize all relevant NSH encapsulations. 

F) OpenFlow support
PTAP is standard in OF 1.5, EXT-382 is not. It would be beneficial if both PTAP and the generic encap/decap actions of EXT-382 would be made available in OF 1.3, 1.4 and 1.5 (as Nicira extensions, where necessary).


References:
-----------
[1] EXT-112 Packet Type-aware Pipeline
    https://rs.opennetworking.org/bugs/browse/EXT-112
[2] EXT-382 Generic tunnel Encap and Decap
	https://rs.opennetworking.org/bugs/browse/EXT-382
[3] [PATCH net-next v12 0/9] openvswitch: support for layer 3 encapsulated packets
	http://openvswitch.org/pipermail/dev/2016-October/080697.html
[4] [PATCH v12 rebased 0/3] userspace: Support for layer 3 encapsulated packets
	http://openvswitch.org/pipermail/dev/2016-October/080828.html
[5] [RFC PATCH v2 00/13] Add Network Service Header Support
	http://openvswitch.org/pipermail/dev/2016-July/074922.html
[7] Jesse Gross last mail regarding push/pop_nsh_eth
	http://openvswitch.org/pipermail/dev/2016-September/079824.html
[8] Jean's post with link to his PTAP prototype
	http://openvswitch.org/pipermail/dev/2016-February/066842.html
[9] Jean's latest PTAP prototype
	https://github.com/jean2/openvswitch/commits/jean/ext-112-l3



More information about the dev mailing list