[ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex autovalidator

[Amber, Kumar]

Hi Flavio,

Comments Inline.

> -----Original Message-----
> From: Flavio Leitner <fbl at sysclose.org>
> Sent: Tuesday, June 29, 2021 11:49 PM
> To: Amber, Kumar <kumar.amber at intel.com>
> Cc: Eelco Chaudron <echaudro at redhat.com>; dev at openvswitch.org;
> i.maximets at ovn.org
> Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for mfex
> autovalidator
> 
> On Tue, Jun 29, 2021 at 05:11:00PM +0000, Amber, Kumar wrote:
> > Hi Eelco, Flavio,
> >
> > Pls find my replies Inline
> >
> > > -----Original Message-----
> > > From: Flavio Leitner <fbl at sysclose.org>
> > > Sent: Tuesday, June 29, 2021 7:51 PM
> > > To: Eelco Chaudron <echaudro at redhat.com>
> > > Cc: Amber, Kumar <kumar.amber at intel.com>; Van Haaren, Harry
> > > <harry.van.haaren at intel.com>; dev at openvswitch.org;
> > > i.maximets at ovn.org
> > > Subject: Re: [ovs-dev] [v4 07/12] test/sytem-dpdk: Add unit test for
> > > mfex autovalidator
> > >
> > > On Tue, Jun 29, 2021 at 03:50:22PM +0200, Eelco Chaudron wrote:
> > > >
> > > >
> > > > On 28 Jun 2021, at 4:57, Flavio Leitner wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > On Thu, Jun 17, 2021 at 09:57:49PM +0530, Kumar Amber wrote:
> > > > >> Tests:
> > > > >>   6: OVS-DPDK - MFEX Autovalidator
> > > > >>   7: OVS-DPDK - MFEX Autovalidator Fuzzy
> > > > >>
> > > > >> Added a new directory to store the PCAP file used in the tests
> > > > >> and a script to generate the fuzzy traffic type pcap to be used
> > > > >> in fuzzy unit test.
> > > > >
> > > > >
> > > > > I haven't tried this yet but am I right that these tests are
> > > > > going to pass a pcap to send traffic in a busy loop for 5
> > > > > seconds in the first case and 20 seconds in the second case?
> > > > >
> > > > > I see that when autovalidator is set OVS will crash if one
> > > > > implementation returns a different value, so I wonder why we
> > > > > need to run for that long.
> > > >
> > > > I think we should remove the assert (already suggested by Harry),
> > > > so it will not crass by accident if someone selects autovalidator
> > > > in the field (and runs into an issue).
> > > > Failure will then be detected by the ERROR log entries on shutdown.
> > >
> > > That's true for the testsuite, but not in production as there is
> > > nothing to disable that.
> > >
> > > Perhaps if autovalidator detects an issue, it should log an ERROR
> > > level log to report to testsuite, disable the failing mode and make
> > > sure OVS is either in default or in another functional mode.
> >
> > So I have put the following :
> > 	Removed the assert
> > 	Allow the Auto-validator to run for all implementation and for a full
> batch
> > 	Document error via Vlog_Error
> > 	Set the auto-validator to default {Scalar} when returning out in case
> of failure.
> 
> Sounds like a plan to me.
> Is that okay with you Eelco?
> 
> 
> > > > I’m wondering if there is another way than a simple delay, as
> > > > these tend to
> > > cause issues later on. Can we check packets processed or something?
> > >
> > > Yeah, maybe we can pass all packets like 5x at least.
> >
> > Sure I will try to find something to do it more nicely.
> > But just a thought keeping it 20sec allows for a full-stabilization and also
> thorough testing of stability as well.
> > So keeping it may not be just a bad idea.
> 
> The issue is that if every test decides to delay seconds, the testsuite
> becomes impractical. We have removed 'sleep' over time. Instead, we have
> functions to wait for a certain cmdline output, or some event.
> Yes, there are still some left to be fixed.
> 
> Back to the point, maybe there is a signal of some sort we can get that
> indicates the stability you're looking for.
> 

I agree to the point and I am looking for a singal but currently due to assert removal we don’t have any marker.
To Minimize the time, I did analysis of the time taken in each test-case :

1) for the simple test-case we don’t need the 5sec wait time as PCAP only contains one traffic or each type.
2) for fuzzy we do need at least 5sec for all 10k packets to be sent at-least 2x and also stability.

Will the reductions suffice for now until I find a way to remove it completely from Fuzzy ?

Regards
Amber
> --
> fbl