[ovs-dev] OVN implementation of the NAT/FIP architecture

Fri Feb 12 10:54:25 UTC 2016

Hi,

I would also like to express my concerns.

I agree that FIP and in general NAT traversal on a single node are bottlenecks 
and single points of failure. In general even supporting HA using VRRP save from
the single point of failure but doesn’t really solve the bottleneck issue.

We should also think about bigger providers and in general multi-datacenter 
deployments like my case for example where the validity of a publicly routable ip 
address is restricted to a subset of the compute nodes available in specific to a 
target availability zone.

That said a deployment where vms that need public access are attached to a 
provider network seems more reasonable. And in general more productive because
the network topology can be maintained valid even if the underlaying datacenter is 
different.

It is also possible to provide remote access to vms by hooking a port in the network
in atarget node that can run simply ovs+ovn-controller and this is replicable in more 
than onenode making effectively no single point of failure and in general no bottlenecks.

As I can see right now the only real missing part is a way to provide external access 
to vms without attaching a provider network interface but this can be achieved in my 
personal opinion using NAT capability of the linux stack holding the hypervisor as 
described:

VM (private_net, 10.0.0.10/24) -> ROUTER(10.0.0.1, 172.16.0.1) -> COMPUTE(172.16.0.2)

Where 172.16.0.2 is an internal port directly attached to an ovs bridge and on the
compute an iptables policy like

“iptables -t nat -A POSTROUTING -s 172.16.0.0/24 -o eth0 -j MASQUERADE"

This solution will not permit implementation of a DNAT in general because ip addresses
in the range of 172.16.0.0/24 are only known internally in every compute node even routing 
between this ip addresses on different computes is impossible (actually not easy but possible)

The only missing point in OVN right now is the ability to route private network traffic to
provider networks but the implementation of routing already make possible to have a 
virtual ip, 10.0.0.1 in the previous example, distributed on every compute node.

In this way practically vms can access the external world using the network stack of the
compute node and can be accessed by specific nodes (with hooked ports)

This nodes in general can provide any king of service in my specific case I use this nodes
to proxy and load balance traffic with a bunch of haproxy installations and a DNS round robin.

For vms that need instead direct access the solution with a port in a provider network is just
perfect. In this case a provider network can be extended over more physical locations and 
the FIP (in a publicly routable range) can be dynamically assigned by automatically configuring
a DNAT on the network equipments. 

> On 12 Feb 2016, at 01:41, Matt Kassawara <mkassawara at gmail.com> wrote:
> 
> Hi,
> 
> I see interest in OVN implementing the conventional NAT/FIP architecture
> from the OpenStack Networking service (neutron) and think it warrants
> further discussion to make sure the project takes an optimal approach to it.
> 
> For those not familiar with neutron, routers between provider and private
> networks primarily use NAT rather than actual routing. Each router contains
> at least two interfaces, one on a provider network typically using publicly
> routable IP space and one on each private network typically using RFC1918
> IP space. Instances on private networks access the provider network and
> other external networks via SNAT on the router. Instances that require
> access from external networks can use a FIP address, essentially DNAT on
> the router. If you disable NAT on a router, you must create a static route
> for each private network or an aggregate of them pointing to the router
> interface on the provider network... definitely not practical or scalable.
> Finally, the NAT/FIP architecture does not work with IPv6.
> 
> Before you think I'm just crazy, I contribute significantly to the
> OpenStack installation and networking guides and also provide support for a
> variety of operators. In my opinion, many operators implemented the NAT/FIP
> architecture because it was the only neutron architecture with some level
> of documentation for a couple of years after the initial release of
> neutron. The NAT/FIP architecture quickly leads to performance, scaling,
> and reliability problems because all network traffic between private and
> provider networks and among private networks must traverse a single node
> (network node) that contains the neutron routers. Furthermore, the
> architecture also runs DHCP and metadata services on the network node. For
> many new and existing operators, the lack of documentation and plenty of
> horror stories gave neutron the appearance of a complex and fragile "black
> box"... leading to skepticism about it that continues today. Attempting to
> rectify these issues with DVR only made the situation worse, but I'll spare
> you the details for now.
> 
> A couple of years ago, some OpenStack documentation contributors with
> networking experience including myself decided that neutron would benefit
> from a separate guide [1]. Most importantly, the guide would include a
> variety of practical deployment architectures or scenarios with operational
> details and functional example configurations that operators could use as a
> starting point. Looking at all of the scenarios [2], the classic (NAT/FIP)
> scenarios offer the least performance, scaling, and reliability. The
> provider network scenarios offer the highest performance, scaling, and
> reliability... mostly due to simplicity. However, these scenarios
> effectively disable "self-service" networking... a cornerstone of cloud
> computing.
> 
> So, what do we do? Suggest a hybrid architecture with the following aspects:
> 
> 1) Instances that require access from external networks contain a port on
> the provider network.
> 2) Instances that only require occasional access to external networks
> contain a port on a private network and use SNAT on a router. If an
> instance requires access from external networks at a later point, add a
> port on the provider network.
> 3) Instances optionally contain a port on one or more private networks to
> facilitate communication among instances.
> 4) Use a combination of multiple network nodes, clever network scheduling,
> and L3 HA (neutron implementation of VRRP) to increase performance,
> scalability, and reliability of routing among private networks and SNAT.
> 5) Distribute DHCP and metadata services across multiple controller or
> compute nodes.
> 
> What's missing? Floating IP addresses!
> 
> I think OVN should support the NAT/FIP architecture, but only as a
> temporary measure to increase adoption by operators using it. In fact, we
> should use the words "classic" or "legacy" around it in any documentation.
> For new deployments, suggest a hybrid architecture and provide ample
> documentation for it similar to the scenarios in the networking guide.
> Meanwhile, development should focus on supporting actual routing such as
> advertisement of private networks via one or more dynamic routing
> protocols. After release of this feature, deprecate and eventually remove
> NAT/FIP support.
> 
> P.S. - If OVN decides to support the NAT/FIP architecture, I strongly
> recommend also offering some form of HA such as VRRP in the initial release
> to increase appeal to operators, most of whom became wary of neutron's
> inability to eliminate single points of failure for so many releases.
> I don't want OVN to repeat mistakes of neutron, even for a little bit.
> 
> Matt
> 
> [1] http://docs.openstack.org/liberty/networking-guide/
> [2] http://docs.openstack.org/liberty/networking-guide/deploy.html
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> http://openvswitch.org/mailman/listinfo/dev

Hi,