[ovs-dev] Possible Regression due to "ossfuzz: Break flow test target into two targets to speed up fuzzing."

[Ben Pfaff]

On Wed, Oct 17, 2018 at 02:54:17PM +0200, Lorenzo Bianconi wrote:
> >
> > On Mon, Oct 15, 2018 at 11:31:55PM +0200, Lorenzo Bianconi wrote:
> > > >
> > > > On Fri, Oct 12, 2018 at 08:54:49AM -0700, Ben Pfaff wrote:
> > > > > On Fri, Oct 12, 2018 at 03:44:51PM +0300, Ilya Maximets wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > it seems that travis-ci is failing due to a testsuite regression introduced
> > > > > > > by 1adcbcee8f4c ("ossfuzz: Break flow test target into two targets to speed
> > > > > > > up fuzzing.").
> > > > > > >
> > > > > > > https://travis-ci.org/openvswitch/ovs/jobs/439811394
> > > > > >
> > > > > > Hi Simon,
> > > > > >
> > > > > > As it was already said, this is not the patch that produces the issue.
> > > > > > I'm able to reproduce '2649: ovn -- 3 HVs, 3 LS, 3 lports/LS, 1 LR'
> > > > > > test failure locally by restricting to use only one CPU core with taskset.
> > > > > > So, I bisected the issue and found that the fist patch that fails is:
> > > > > >
> > > > > >     2e5cdb4b1392 ("OVN: add buffering support for ip packets").
> > > > > >
> > > > > > Issue happens only if single core assigned:
> > > > > >
> > > > > >     taskset -c 2   make check TESTSUITEFLAGS='2649'  #FAILS
> > > > > >     taskset -c 2-3 make check TESTSUITEFLAGS='2649'  #OK
> > > > > >
> > > > > > As Travis has no much CPU resources it fails all the time.
> > > > > >
> > > > > > I'm not much familiar with OVN code/tests, hope above information will
> > > > > > be useful for your investigation.
> > > > >
> > > > > I can reproduce this too, with your hint (although it's test 3309, not
> > > > > 2649).
> > > > >
> > > > > The failure is because a lot of expected packets don't show up:
> > > > >
> > > > >     ...
> > > > >     rcv_n=212 exp_n=233
> > > > >     ovn.at:12: wait failed after 10 seconds
> > > > >
> > > > > It needs more looking into.
> > > >
> > > > I don't have time to look into it, but we need the tests passing, so I
> > > > sent a revert: https://patchwork.ozlabs.org/patch/984336/
> > >
> > > Hi all,
> > >
> > > I looked into it and it is probably a timing issue. In particular the
> > > ARP replies generated during test 8
> > > do not reach the router before the ip buffering gc time-out so we will
> > > end up having less packets than
> > > expected.
> > > A trivial solution is to invert test 8 and 4,5,6 or maybe we can add a
> > > unixctl command to disable ip
> > > buffering.
> > > Could you please try following patch? it fixes in my tests
> >
> > Could you try it out on travis, please?
> 
> Hi Ben,
> 
> these are travis tests applying 'OVN ip buffering' patch on top of
> current master branch:
> - https://travis-ci.org/LorenzoBianconi/ovs/builds/442605149

OK, same patch as before, same failure as before.

> these are travis tests applying the patch I posted to fix the issue on
> top of previous branch:
> - https://travis-ci.org/LorenzoBianconi/ovs/builds/442642315

OK, great, thank you.  Please post a squashed patch to apply to master.