[ovs-dev] Failing Test 2214 on i686

[Ben Pfaff]

Hmm.  I'm truly puzzled.  I run the tests on i686 all the time and don't
get any consistent failures.  Is there anything unusual about your
setup, or anything that might assist me in reproducing it?

We have checked in a few other commits more recently than the commit
0a0f39d that you tested that should act to make some of the OVN tests
more reliable.  Maybe one of them will have some positive effect.  This
one in particular comes to mind.

commit 207d29e13d103cdae911ecf3863c7493d89a5a48
Author: Ben Pfaff <blp at ovn.org>
Date:   Tue Jul 26 23:55:25 2016 -0700

    ovsdb-idl: Wake up ovsdb_idl_loop when a transaction commits.
    
    There is a fair amount of code that defers modifying the database when a
    transaction cannot be created (because there is already one outstanding).
    This code tends to assume that the main loop will wake up again when it
    becomes possible again to modify the database, but the actual ovsdb_id_loop
    implementation only did this if the database had changed.  This is too
    conservative a policy and may account for some failures I've seen in tests.
    
    Signed-off-by: Ben Pfaff <blp at ovn.org>
    Acked-by: Ryan Moats <rmoats at us.ibm.com>

On Fri, Aug 12, 2016 at 07:35:14AM +0200, Christian Ehrhardt wrote:
> Hi Ben,
> we ran last week with openvswitch-2.6.0~git20160804.0a0f39d
> The patch you asked for was already in at the time.
> 
> Maybe the attachements got dropped somewhere - my sent mail still had them.
> Anyway lets make sure you get the content via pastebin at least:
> I had logs when I build-the-debian-way [1] and one to be sure when I
> build-"normal" [2], both running into the same issue.
> And finally I also attached the logs also to the LP bug at [3].
> 
> [1]: http://paste.ubuntu.com/23048271/
> [2]: http://paste.ubuntu.com/23048272/
> [3]: https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1609846
> 
> Kind Regards,
> Christian
> 
> 
> On Thu, Aug 11, 2016 at 7:29 PM, Ben Pfaff <blp at ovn.org> wrote:
> 
> > On Thu, Aug 11, 2016 at 10:08:12AM -0700, Ben Pfaff wrote:
> > > On Fri, Aug 05, 2016 at 03:25:47PM +0200, Christian Ehrhardt wrote:
> > > > Hi,
> > > > while checking latest master we found that there were several tests
> > that
> > > > failed sometimes.
> > > >   15: bfd - bfd decay
> > > > 1149: ofproto-dpif - in place modification (vlan)
> > > > But one of them seems to reliably fail in i686 mode while working just
> > fine
> > > > in a similar amd64 build.
> > > >
> > > > Steps to reproduce:
> > > > make check TESTSUITEFLAGS='2214'
> > > >
> > > > I usually use the Debian build using the debian/ subdir packaged in
> > > > openvswitch:
> > > > dh build --with autoreconf,python2,python3,systemd --parallel
> > > >
> > > > But to make sure I also ran a "normal" clean build with:
> > > > ./boot.sh && ./configure && make
> > > > Built that way it fails just as well.
> > > >
> > > > It seems that the packets were expected twice but only received once:
> > > > checking packets in hv1/vif1-tx.pcap against 1.expected:
> > > > expected 6 packets, only received 3
> > > > ../../tests/ovn.at:1326: sort $rcv_text
> > > > --- expout 2016-08-05 12:13:09.661688314 +0000
> > > > +++ /<<PKGBUILDDIR>>/_dpdk/tests/testsuite.dir/at-groups/2214/stdout
> > > > 2016-08-05 12:13:09.661688314 +0000
> > > > @@ -1,6 +1,3 @@
> > > >  010000000000f0000000000202ff
> > > > -010000000000f0000000000303ff
> > > >  f00000000001f000000000020021
> > > > -f00000000001f000000000030031
> > > >  fffffffffffff0000000000202ff
> > > > -fffffffffffff0000000000303ff
> > > >
> > > > Full log(s) attached
> > >
> > > I'd like to take a look at this but the logs don't seem to have been
> > > attached.
> > >
> > > (As a data point, I develop on with 32-bit userspace but I don't see
> > > this consistent failure.)
> >
> > Did the "master" that you tested include the following commit?  It fixed
> > a number of races in practice.
> >
> > commit 49d7c759b82e78acd03288c123b1bc0c8904a52b
> > Author: Ben Pfaff <blp at ovn.org>
> > Date:   Tue Jul 26 23:50:06 2016 -0700
> >
> >     tests: Remove most packet-forwarding related "sleep"s from OVN tests.
> >
> >     These arbitrary sleeps are often longer than necessary and can be too
> > short
> >     in corner cases.  This commit replaces them by a common macro that
> > waits
> >     only as long as necessary.
> >
> >     Signed-off-by: Ben Pfaff <blp at ovn.org>
> >     Acked-by: Ryan Moats <rmoats at us.ibm.com>
> >     Acked-by: Flavio Fernandes <flavio at flaviof.com>
> >
> 
> 
> 
> -- 
> Christian Ehrhardt
> Software Engineer, Ubuntu Server
> Canonical Ltd