[ovs-dev] [PATCH v13 07/11] test/sytem-dpdk: Add unit test for mfex autovalidator

Eelco Chaudron echaudro at redhat.com
Thu Jul 15 15:23:40 UTC 2021



On 15 Jul 2021, at 17:06, Stokes, Ian wrote:

>> Added some missing negative test cases, see below.
>>
>
> Thanks for these Eelco, would you like me to add you as a co-author for this patch as you have code that is part of the next revison?

Whatever is easiest for you, I’m fine either way.

> Regards
> Ian
>> On 15 Jul 2021, at 14:42, kumar Amber wrote:
>>
>>> From: Kumar Amber <kumar.amber at intel.com>
>>>
>>> Tests:
>>>   6: OVS-DPDK - MFEX Autovalidator
>>>   7: OVS-DPDK - MFEX Autovalidator Fuzzy
>>>   8: OVS-DPDK - MFEX Configuration
>>>
>>> Added a new directory to store the PCAP file used
>>> in the tests and a script to generate the fuzzy traffic
>>> type pcap to be used in fuzzy unit test.
>>>
>>> Signed-off-by: Kumar Amber <kumar.amber at intel.com>
>>> Acked-by: Flavio Leitner <fbl at sysclose.org>
>>>
>>> ---
>>> v13:
>>> - fix -v in the command
>>> - added the configuration test case and supporting doc update
>>> v12:
>>> - change skip paramter for unit test
>>> v11:
>>> - fix comments from Eelco
>>> v7:
>>> - fix review comments(Eelco)
>>> v5:
>>> - fix review comments(Ian, Flavio, Eelco)
>>> - remove sleep from first test and added minor 5 sec sleep to fuzzy
>>> ---
>>> ---
>>>  Documentation/topics/dpdk/bridge.rst |  56 ++++++++++++
>>>  tests/.gitignore                     |   1 +
>>>  tests/automake.mk                    |   6 ++
>>>  tests/mfex_fuzzy.py                  |  33 +++++++
>>>  tests/pcap/mfex_test.pcap            | Bin 0 -> 416 bytes
>>>  tests/system-dpdk.at                 | 129 +++++++++++++++++++++++++++
>>>  6 files changed, 225 insertions(+)
>>>  create mode 100755 tests/mfex_fuzzy.py
>>>  create mode 100644 tests/pcap/mfex_test.pcap
>>>
>>> diff --git a/Documentation/topics/dpdk/bridge.rst
>> b/Documentation/topics/dpdk/bridge.rst
>>> index 8c500c504..913b3e6f6 100644
>>> --- a/Documentation/topics/dpdk/bridge.rst
>>> +++ b/Documentation/topics/dpdk/bridge.rst
>>> @@ -346,3 +346,59 @@ A compile time option is available in order to test it
>> with the OVS unit
>>>  test suite. Use the following configure option ::
>>>
>>>      $ ./configure --enable-mfex-default-autovalidator
>>> +
>>> +Unit Test Miniflow Extract
>>> +++++++++++++++++++++++++++
>>> +
>>> +Unit test can also be used to test the workflow mentioned above by running
>>> +the following test-case in tests/system-dpdk.at ::
>>> +
>>> +    make check-dpdk TESTSUITEFLAGS='-k MFEX'
>>> +    OVS-DPDK - MFEX Autovalidator
>>> +
>>> +The unit test uses mulitple traffic type to test the correctness of the
>>> +implementaions.
>>> +
>>> +The MFEX commands can also be tested for negative and positive cases to
>>> +verify that the MFEX set command does not allow for incorrect parameters.
>>> +A user can directly run the following configuration test case in
>>> +tests/system-dpdk.at ::
>>> +
>>> +    make check-dpdk TESTSUITEFLAGS='-k MFEX'
>>> +    OVS-DPDK - MFEX Configuration
>>> +
>>> +Running Fuzzy test with Autovalidator
>>> ++++++++++++++++++++++++++++++++++++++
>>> +
>>> +Fuzzy tests can also be done on miniflow extract with the help of
>>> +auto-validator and Scapy. The steps below describes the steps to
>>> +reproduce the setup with IP being fuzzed to generate packets.
>>> +
>>> +Scapy is used to create fuzzy IP packets and save them into a PCAP ::
>>> +
>>> +    pkt = fuzz(Ether()/IP()/TCP())
>>> +
>>> +Set the miniflow extract to autovalidator using ::
>>> +
>>> +    $ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator
>>> +
>>> +OVS is configured to receive the generated packets ::
>>> +
>>> +    $ ovs-vsctl add-port br0 pcap0 -- \
>>> +        set Interface pcap0 type=dpdk options:dpdk-devargs=net_pcap0
>>> +        "rx_pcap=fuzzy.pcap"
>>> +
>>> +With this workflow, the autovalidator will ensure that all MFEX
>>> +implementations are classifying each packet in exactly the same way.
>>> +If an optimized MFEX implementation causes a different miniflow to be
>>> +generated, the autovalidator has ovs_assert and logging statements that
>>> +will inform about the issue.
>>> +
>>> +Unit Fuzzy test with Autovalidator
>>> ++++++++++++++++++++++++++++++++++++++
>>> +
>>> +Unit test can also be used to test the workflow mentioned above by running
>>> +the following test-case in tests/system-dpdk.at ::
>>> +
>>> +    make check-dpdk TESTSUITEFLAGS='-k MFEX'
>>> +    OVS-DPDK - MFEX Autovalidator Fuzzy
>>> diff --git a/tests/.gitignore b/tests/.gitignore
>>> index 45b4f67b2..a3d927e5d 100644
>>> --- a/tests/.gitignore
>>> +++ b/tests/.gitignore
>>> @@ -11,6 +11,7 @@
>>>  /ovsdb-cluster-testsuite
>>>  /ovsdb-cluster-testsuite.dir/
>>>  /ovsdb-cluster-testsuite.log
>>> +/pcap/
>>>  /pki/
>>>  /system-afxdp-testsuite
>>>  /system-afxdp-testsuite.dir/
>>> diff --git a/tests/automake.mk b/tests/automake.mk
>>> index f45f8d76c..a6c15ba55 100644
>>> --- a/tests/automake.mk
>>> +++ b/tests/automake.mk
>>> @@ -143,6 +143,11 @@ $(srcdir)/tests/fuzz-regression-list.at:
>> tests/automake.mk
>>>  	    echo "TEST_FUZZ_REGRESSION([$$basename])"; \
>>>  	done > $@.tmp && mv $@.tmp $@
>>>
>>> +EXTRA_DIST += $(MFEX_AUTOVALIDATOR_TESTS)
>>> +MFEX_AUTOVALIDATOR_TESTS = \
>>> +	tests/pcap/mfex_test.pcap \
>>> +	tests/mfex_fuzzy.py
>>> +
>>>  OVSDB_CLUSTER_TESTSUITE_AT = \
>>>  	tests/ovsdb-cluster-testsuite.at \
>>>  	tests/ovsdb-execution.at \
>>> @@ -512,6 +517,7 @@ tests_test_type_props_SOURCES = tests/test-type-
>> props.c
>>>  CHECK_PYFILES = \
>>>  	tests/appctl.py \
>>>  	tests/flowgen.py \
>>> +	tests/mfex_fuzzy.py \
>>>  	tests/ovsdb-monitor-sort.py \
>>>  	tests/test-daemon.py \
>>>  	tests/test-json.py \
>>> diff --git a/tests/mfex_fuzzy.py b/tests/mfex_fuzzy.py
>>> new file mode 100755
>>> index 000000000..5b056bb48
>>> --- /dev/null
>>> +++ b/tests/mfex_fuzzy.py
>>> @@ -0,0 +1,33 @@
>>> +#!/usr/bin/python3
>>> +try:
>>> +    from scapy.all import RandMAC, RandIP, PcapWriter, RandIP6, RandShort,
>> fuzz
>>> +    from scapy.all import IPv6, Dot1Q, IP, Ether, UDP, TCP
>>> +except ModuleNotFoundError as err:
>>> +    print(err + ": Scapy")
>>> +import sys
>>> +
>>> +path = str(sys.argv[1]) + "/pcap/fuzzy.pcap"
>>> +pktdump = PcapWriter(path, append=False, sync=True)
>>> +
>>> +for i in range(0, 2000):
>>> +
>>> +    # Generate random protocol bases, use a fuzz() over the combined packet
>>> +    # for full fuzzing.
>>> +    eth = Ether(src=RandMAC(), dst=RandMAC())
>>> +    vlan = Dot1Q()
>>> +    ipv4 = IP(src=RandIP(), dst=RandIP())
>>> +    ipv6 = IPv6(src=RandIP6(), dst=RandIP6())
>>> +    udp = UDP(dport=RandShort(), sport=RandShort())
>>> +    tcp = TCP(dport=RandShort(), sport=RandShort())
>>> +
>>> +    # IPv4 packets with fuzzing
>>> +    pktdump.write(fuzz(eth / ipv4 / udp))
>>> +    pktdump.write(fuzz(eth / ipv4 / tcp))
>>> +    pktdump.write(fuzz(eth / vlan / ipv4 / udp))
>>> +    pktdump.write(fuzz(eth / vlan / ipv4 / tcp))
>>> +
>>> +    # IPv6 packets with fuzzing
>>> +    pktdump.write(fuzz(eth / ipv6 / udp))
>>> +    pktdump.write(fuzz(eth / ipv6 / tcp))
>>> +    pktdump.write(fuzz(eth / vlan / ipv6 / udp))
>>> +    pktdump.write(fuzz(eth / vlan / ipv6 / tcp))
>>> diff --git a/tests/pcap/mfex_test.pcap b/tests/pcap/mfex_test.pcap
>>> new file mode 100644
>>> index
>> 0000000000000000000000000000000000000000..1aac67b8d643ecb016c758cb
>> a4cc32212a80f52a
>>> GIT binary patch
>>> literal 416
>>>
>> zcmca|c+)~A1{MYw`2U}Qff2}Q<eHVR>K`M68ITRa|G at yFii5$Gfk6YL%z>@uY&}o
>> |
>>>
>> z2s4N<1VH2&7y^V87$)XGOtD~MV$cFgfG~zBGGJ2#YtF$<F=a4i;9x8Q*<ZrSM6Uf
>> z
>>> xK>KST_NTIwYriok6N4Vm)gX-
>> Q@<yO<!C`>c^{cp<7_5LgK^UuU{2>VS0RZ!RQ+EIW
>>>
>>> literal 0
>>> HcmV?d00001
>>>
>>> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
>>> index 802895488..a764c69ca 100644
>>> --- a/tests/system-dpdk.at
>>> +++ b/tests/system-dpdk.at
>>> @@ -232,3 +232,132 @@ OVS_VSWITCHD_STOP(["\@does not exist. The
>> Open vSwitch kernel module is probably
>>>  \@EAL: No free hugepages reported in hugepages-1048576kB at d"])
>>>  AT_CLEANUP
>>>  dnl --------------------------------------------------------------------------
>>> +
>>> +dnl --------------------------------------------------------------------------
>>> +dnl Add standard DPDK PHY port
>>> +AT_SETUP([OVS-DPDK - MFEX Autovalidator])
>>> +AT_KEYWORDS([dpdk])
>>> +
>>> +OVS_DPDK_START()
>>> +
>>> +dnl Add userspace bridge and attach it to OVS
>>> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
>>> +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk
>> options:dpdk-
>> devargs=net_pcap1,rx_pcap=$srcdir/pcap/mfex_test.pcap,infinite_rx=1], [],
>> [stdout], [stderr])
>>> +AT_CHECK([ovs-vsctl show], [], [stdout])
>>> +
>>> +AT_SKIP_IF([! ovs-appctl dpif-netdev/miniflow-parser-get | sed 1,4d | grep
>> "True"], [], [dnl
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0],
>> [dnl
>>> +Miniflow extract implementation set to autovalidator.
>>> +])
>>> +
>>> +OVS_WAIT_UNTIL([test `ovs-vsctl get interface p1 statistics | grep -oP
>> 'rx_packets=\s*\K\d+'` -ge 1000])
>>> +
>>> +dnl Clean up
>>> +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
>>> +AT_CLEANUP
>>> +dnl --------------------------------------------------------------------------
>>> +
>>> +dnl --------------------------------------------------------------------------
>>> +dnl Add standard DPDK PHY port
>>> +AT_SETUP([OVS-DPDK - MFEX Autovalidator Fuzzy])
>>> +AT_KEYWORDS([dpdk])
>>> +AT_SKIP_IF([! pip3 list | grep scapy], [], [])
>>> +AT_CHECK([$PYTHON3 $srcdir/mfex_fuzzy.py $srcdir], [], [stdout])
>>> +OVS_DPDK_START()
>>> +
>>> +dnl Add userspace bridge and attach it to OVS
>>> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
>>> +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk
>> options:dpdk-
>> devargs=net_pcap1,rx_pcap=$srcdir/pcap/fuzzy.pcap,infinite_rx=1], [], [stdout],
>> [stderr])
>>> +AT_CHECK([ovs-vsctl show], [], [stdout])
>>> +
>>> +AT_SKIP_IF([! ovs-appctl dpif-netdev/miniflow-parser-get | sed 1,4d | grep
>> "True"], [], [dnl
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set autovalidator], [0],
>> [dnl
>>> +Miniflow extract implementation set to autovalidator.
>>> +])
>>> +
>>> +OVS_WAIT_UNTIL([test `ovs-vsctl get interface p1 statistics | grep -oP
>> 'rx_packets=\s*\K\d+'` -ge 100000])
>>> +
>>> +dnl Clean up
>>> +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
>>> +AT_CLEANUP
>>> +dnl --------------------------------------------------------------------------
>>> +
>>> +dnl --------------------------------------------------------------------------
>>> +AT_SETUP([OVS-DPDK - MFEX Configuration])
>>> +AT_KEYWORDS([dpdk])
>>> +OVS_DPDK_START()
>>> +dnl Add userspace bridge and attach it to OVS
>>> +AT_CHECK([ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev])
>>> +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dpdk
>> options:dpdk-
>> devargs=net_pcap1,rx_pcap=$srcdir/pcap/mfex_test.pcap,infinite_rx=1], [],
>> [stdout], [stderr])
>>> +AT_CHECK([ovs-vsctl show], [], [stdout])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set scalar 1], [2],
>>> +[], [dnl
>>> +Error: unknown argument 1.
>>> +ovs-appctl: ovs-vswitchd: server returned an error
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set -pmd 3 study 300
>> xyz], [2],
>>> +[], [dnl
>>> +Error: Invalid study_pkt_cnt value: xyz.
>>> +ovs-appctl: ovs-vswitchd: server returned an error
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set scalar abcd], [2],
>>> +[], [dnl
>>> +Error: unknown argument abcd.
>>> +ovs-appctl: ovs-vswitchd: server returned an error
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set -pmd 0 scalar abcd],
>> [2],
>>> +[], [dnl
>>> +Error: unknown argument abcd.
>>> +ovs-appctl: ovs-vswitchd: server returned an error
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set -pmd], [2],
>>> +[], [dnl
>>> +Error: -pmd option requires a thread id argument.
>>> +ovs-appctl: ovs-vswitchd: server returned an error
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set tudy abcd], [2],
>>> +[], [dnl
>>> +Error: unknown argument abcd.
>>> +ovs-appctl: ovs-vswitchd: server returned an error
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set -pmd 3 study abcd],
>> [2],
>>> +[], [dnl
>>> +Error: Invalid study_pkt_cnt value: abcd.
>>> +ovs-appctl: ovs-vswitchd: server returned an error
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set -pmd 21 study], [0],
>> [dnl
>>> +Miniflow extract implementation set to study, on pmd thread 21, studying
>> 128 packets.
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set -pmd 21 study 512],
>> [0], [dnl
>>> +Miniflow extract implementation set to study, on pmd thread 21, studying
>> 512 packets.
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set study 512], [0], [dnl
>>> +Miniflow extract implementation set to study, studying 512 packets.
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set study], [0], [dnl
>>> +Miniflow extract implementation set to study, studying 128 packets.
>>> +])
>>> +
>>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set -pmd 21
>> autovalidator], [0], [dnl
>>> +Miniflow extract implementation set to autovalidator, on pmd thread 21.
>>> +])
>>> +
>>> +dnl Clean up
>>> +AT_CHECK([ovs-vsctl del-port br0 p1], [], [stdout], [stderr])
>>> +AT_CLEANUP dnl
>>> +dnl --------------------------------------------------------------------------
>>> --
>>
>> It’s missing some of the negative tests cases, which I have added based on the
>> comment on the v6 patch on name changes.
>> Here is the diff:
>>
>> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
>> index a764c69ca..929bbf0a9 100644
>> --- a/tests/system-dpdk.at
>> +++ b/tests/system-dpdk.at
>> @@ -337,6 +337,30 @@ Error: Invalid study_pkt_cnt value: abcd.
>>  ovs-appctl: ovs-vswitchd: server returned an error
>>  ])
>>
>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set -pmd zero study], [2],
>> +[], [dnl
>> +Error: Miniflow extract parser not changed, PMD thread argument passed is
>> not valid: 'zero'. Pass a valid pmd thread ID.
>> +ovs-appctl: ovs-vswitchd: server returned an error
>> +])
>> +
>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set -pmd 1], [2],
>> +[], [dnl
>> +Error: no miniflow extract name provided. Output of miniflow-parser-get
>> shows implementation list.
>> +ovs-appctl: ovs-vswitchd: server returned an error
>> +])
>> +
>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set -pmd 1 superstudy],
>> [2],
>> +[], [dnl
>> +Error: unknown miniflow extract implementation superstudy.
>> +ovs-appctl: ovs-vswitchd: server returned an error
>> +])
>> +
>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set superstudy], [2],
>> +[], [dnl
>> +Error: unknown miniflow extract implementation superstudy.
>> +ovs-appctl: ovs-vswitchd: server returned an error
>> +])
>> +
>> +AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set -pmd 1 study -pmd],
>> [2],
>> +[], [dnl
>> +Error: Invalid study_pkt_cnt value: -pmd.
>> +ovs-appctl: ovs-vswitchd: server returned an error
>> +])
>> +
>>  AT_CHECK([ovs-appctl dpif-netdev/miniflow-parser-set -pmd 21 study], [0], [dnl
>>  Miniflow extract implementation set to study, on pmd thread 21, studying 128
>> packets.
>>  ])



More information about the dev mailing list