[ovs-dev] [ovs-dev,v2,2/4] ovn-controller: add quiet mode

[Ryan Moats]

Ben Pfaff <blp at ovn.org> wrote on 10/05/2016 12:37:26 PM:

> From: Ben Pfaff <blp at ovn.org>
> To: Ryan Moats/Omaha/IBM at IBMUS
> Cc: dev at openvswitch.org
> Date: 10/05/2016 12:37 PM
> Subject: Re: [ovs-dev,v2,2/4] ovn-controller: add quiet mode
>
> On Tue, Oct 04, 2016 at 05:11:37PM -0500, Ryan Moats wrote:
> > Ben Pfaff <blp at ovn.org> wrote on 10/04/2016 12:14:32 PM:
> >
> > > From: Ben Pfaff <blp at ovn.org>
> > > To: Ryan Moats/Omaha/IBM at IBMUS
> > > Cc: dev at openvswitch.org
> > > Date: 10/04/2016 12:14 PM
> > > Subject: Re: [ovs-dev,v2,2/4] ovn-controller: add quiet mode
> > >
> > > On Wed, Aug 31, 2016 at 03:22:44PM +0000, Ryan Moats wrote:
> > > > As discussed in [1], what the incremental processing code
> > > > actually accomplished was that the ovn-controller would
> > > > be "quiet" and not burn CPU when things weren't changing.
> > > > This patch set recreates this state by calculating whether
> > > > changes have occured that would require a full calculation
> > > > to be performed.  It does this by persisting a copy of
> > > > the localvif_to_ofport and tunnel information in the
> > > > controller module, rather than in the physical.c module
> > > > as was the case with previous commits.
> > > >
> > > > [1] http://openvswitch.org/pipermail/dev/2016-August/078272.html
> > > >
> > > > Signed-off-by: Ryan Moats <rmoats at us.ibm.com>
> > >
> > > Hi Ryan.
> > >
> > > I like the idea behind this patch.  However, it no longer applies to
> > > master, so it needs a rebase.
> >
> > So done, but before submitting a new patch....
> >
> > >
> > > It also seems like this TODO should be addressed:
> > > +        /* TODO (regXboi): this next line is needed for the 3 HVs, 3
LS,
> > > +         * 3 lports/LS, 1 LR test case, but has the potential side
> > effect
> > > +         * of defeating quiet mode once a logical router leads to
> > creating
> > > +         * patch ports. Need to understand the failure mode better
and
> > > +         * what is needed to remove this. */
> > > +        force_full_process();
> >
> > I've been looking at what happens here and I'm seeing some signatures
> > that concern me.  The test case that fails is no longer the cited one
above
> > but is "2 HVs, 4 lports/HV, localnet ports" ...
> >
> > What I'm seeing when I peer in is that the information populating the
> > local_datapath structures doesn't appear to be consistent on each pass
> > through binding_run: Looking at the ovn-controller process under hv2
> > for the above test (when it passes), I'll see signatures that look
> > like the following:
> >
> > 2016-10-04T21:57:58.257Z|00020|physical|INFO|looking at binding record
> > 3736404b-c69d-4878-8d45-81ad9be06f5f
> > 2016-10-04T21:57:58.257Z|00021|physical|INFO|dp_key is 1
> > 2016-10-04T21:57:58.257Z|00022|physical|INFO|looking for ofport of lp11
> > (LP)
> > 2016-10-04T21:57:58.257Z|00023|physical|INFO|looking for ofport of ln1
> > (localnet)
> > ...
> > 2016-10-04T21:57:58.259Z|00034|physical|INFO|looking at binding record
> > 3736404b-c69d-4878-8d45-81ad9be06f5f
> > 2016-10-04T21:57:58.259Z|00035|physical|INFO|dp_key is 1
> > 2016-10-04T21:57:58.259Z|00036|physical|INFO|looking for ofport of lp11
> > (LP)
> > 2016-10-04T21:57:58.259Z|00037|physical|INFO|looking for tunnel to hv1
> > ...
>
> This sort of oscillation seems super-weird to me.  I'd also like to
> learn more.
>

I found the problem and fixed it in the v3 patch set - some of the methods
(patch_run for example) now include a test of ovs_idl_txn before they run
and some don't (physical_run for example), so you can see guess what was
happening...

(If we had a txn to work with everything is fine, but without one,
physical_run was trying to calculate flows based on local datapaths
with no localnet ports being defined, which leads to the above
oscillation.)

I added a txn check to the logic for whether to call things or not in
ovn-controller.c and I added a note in the email about thinking that a
follow-on patch to clean up all of the ovs_idl_txn gates might make sense
(so that nobody trips on this in the future)

Ryan