[ovs-dev] Possible OVN enhancement: implementing OpenShift's "unidling" behavior
Lorenzo Bianconi
lorenzo.bianconi at redhat.com
Wed Nov 28 14:56:51 UTC 2018
> > > Hi Mark, thanks for writing up the interesting topic. I agree with Guru
> > > that holding packets and waiting for control plane change to get ready
> > > doesn't seem to work/scale.
> > > I am curious about the problem and I want to make sure I understand it
> > > correctly. It sounds like some FaaS scenario, is it? Is it reasonable in
> > > such scenario to keep at least one pod for the VIP even if it is idle?
> > > If not, is there any example system that actually works well by spinning
> > > up pods after requests came?
Hi all,
>
> I think there are two requirements here:
>
> 1. Trigger the pod spin up when packet comes.
> In theory this is doable as the options you described. However, I think it
> depends on how long the client can wait before it times out. What's the
> requirement for the e2e time? As Guru mentioned, it would be at least
> "seconds". How fast the pod spin-up finishes e2e, even without considering
> OVN? From OVN perspective, for the pods' network to be working, it depends
> on the scale and whether incremental processing is in place. It could be
> tens of seconds in large scale environment with current upstream code base.
>
I think point one is the basic requirement from OpenShift folks.
As already described by Mark I guess the best option to establish a
communication between ovn-controll and the CMS plugin is though a new table in
the SB/NB DBs. A possible table implementation could be:
- column 'event type' (e.g. type=unidling)
- column data (generic info, e.g. L2/L3 addresses)
- column done, used by the CMS to indicate that the event has been correctly
served
I think NB and SB tables can be mirrored.
Are these info enough or something is missing?
> 2. Hold the packet and reinject.
> I wonder if this requirement is really necessary. L4 protocols usually
> should handle the first packet lost properly (just like how the first
> packet lost before ARP resolving is handled). For example, TCP SYN should
> retry and whether the first packets are buffered doesn't matter at all. Is
> there anything I am missing?
>
I guess this point is a 'nice to have' since the L4 timeout here will probably fix
the issue
Regards,
Lorenzo
> Thanks,
> Han
>
> > >
> > > Regards,
> > > Han
> > > On Fri, Nov 16, 2018 at 11:28 AM Guru Shetty <guru at ovn.org
> > > <mailto:guru at ovn.org>> wrote:
> > >
> > > I don't have specific comments. But I do believe that "holding"
> > > packet may
> > > not work well as the time taken to create new pods, assign IPs to
> those
> > > pod, create new load-balancer rules for the new pods, will be in
> > > "seconds".
> > >
> > >
> > > On Fri, 9 Nov 2018 at 12:26, Mark Michelson <mmichels at redhat.com
> > > <mailto:mmichels at redhat.com>> wrote:
> > >
> > > > Yesterday during the OVN IRC meeting, I brought up a subject that
> > > > requires a bit more discussion.
> > > >
> > > > The issue: OpenShift has a use case where it uses a load
> > > balancer. The
> > > > VIP of the load balancer is backed with k8s pods. During times of
> > > > inactivity, these pods may be deleted due to idleness.
> > > Eventually, all
> > > > backing pods may be removed due to idleness. What OpenShift
> wants is
> > > > when a packet arrives for the load balancer VIP in this scenario,
> > > they
> > > > want to be able to react by creating new backing pods, updating
> load
> > > > balancer configuration, and reprocessing the packet so that it
> will
> > > > reach a destination pod.
> > > >
> > > > From an OVS/OVN perspective, what needs to happen is that based
> > > on an
> > > > incoming packet, an outside party needs to be notified to take an
> > > > action. The interesting part is at what layer the third party is
> > > > notified and how the incoming packet is further handled. We've
> > > discussed
> > > > this within Red Hat and have three ideas for how to do this.
> > > >
> > > > 1) The database approach
> > > > We add a new table to the southbound database, perhaps called
> > > > CMS_Events. In the situation where OpenShift wants unidling
> > > behavior, it
> > > > sets an option that results in ovn-controller installing a flow
> > > in OVS
> > > > to send the incoming packet to ovn-controller. ovn-controller,
> upon
> > > > receiving the incoming packet, buffers the packet and puts a new
> > > row in
> > > > the CMS_Events table. The CMS, upon seeing the event, takes its
> > > > necessary action. In this particular case, the CMS action is to
> > > spin up
> > > > pods and alter load balancer configuration. The CMS then sets a
> > > value in
> > > > row to indicate it has handled the event. ovn-controller, seeing
> the
> > > > event is handled, deletes the event and continues processing the
> > > packet,
> > > > allowing it to reach its destination.
> > > >
> > > > 2) The message passing approach
> > > > In the situation where OpenShift wants unidling behavior, it
> sets an
> > > > option that results in ovn-controller installing a flow in OVS to
> > > send
> > > > the incoming packet to ovn-controller. ovn-controller, upon
> receiving
> > > > the packet, buffers the packet and sends some sort of message to
> the
> > > > CMS. The contents of this message and the method by which the
> > > message is
> > > > sent would likely be dependent on the CMS in use. The CMS would
> > > act on
> > > > this message, spinning up new pods and altering load balancer
> > > > configuration. The CMS would then send a message back to
> > > ovn-controller,
> > > > where ovn-controller would continue processing the packet,
> > > allowing it
> > > > to reach its destination.
> > > >
> > > > 3) The second controller approach
> > > > In the situation where OpenShift wants unidling behavior, it
> sets an
> > > > option that results in ovn-controller installing a flow in OVS to
> > > send
> > > > the incoming packet to an external controller. The external
> > > controller,
> > > > upon receiving the packet, spins up new pods and alters load
> balancer
> > > > configuration. The external controller then (somehow) sends the
> > > packet
> > > > back into OVS so it can reach its destination.
> > > >
> > > > I presented the database approach during the meeting yesterday,
> > > but it's
> > > > worth presenting the other ideas we've discussed. Does any one of
> > > these
> > > > three ideas jump out as being significantly better than the
> > > others? Do
> > > > you have other ideas for how an external service can be contacted
> > > based
> > > > on the reception of a packet destined for a certain IP address?
> > > >
> > > > Thanks,
> > > > Mark Michelson
> > > > _______________________________________________
> > > > dev mailing list
> > > > dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > >
> > > _______________________________________________
> > > dev mailing list
> > > dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > >
> >
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
More information about the dev
mailing list