[ovs-dev] [PATCH v4 0/7] Packet type aware pipeline

Darrell Ball dball at vmware.com
Tue Jul 11 16:27:46 UTC 2017


dball at ubuntu:~/ovs$ sudo make check-system-userspace TESTSUITEFLAGS='90' -C _gcc
make: Entering directory `/home/dball/ovs/_gcc'
make  all-recursive
make[1]: Entering directory `/home/dball/ovs/_gcc'
Making all in datapath
make[2]: Entering directory `/home/dball/ovs/_gcc/datapath'
Making all in linux
make[3]: Entering directory `/home/dball/ovs/_gcc/datapath/linux'
make -C /lib/modules/4.2.0-42-generic/build M=/home/dball/ovs/_gcc/datapath/linux modules
make[4]: Entering directory `/usr/src/linux-headers-4.2.0-42-generic'
  Building modules, stage 2.
  MODPOST 6 modules
make[4]: Leaving directory `/usr/src/linux-headers-4.2.0-42-generic'
make[3]: Leaving directory `/home/dball/ovs/_gcc/datapath/linux'
make[3]: Entering directory `/home/dball/ovs/_gcc/datapath'
make[3]: Leaving directory `/home/dball/ovs/_gcc/datapath'
make[2]: Leaving directory `/home/dball/ovs/_gcc/datapath'
make[2]: Entering directory `/home/dball/ovs/_gcc'
make[3]: Entering directory `/home/dball/ovs/_gcc/datapath'
make[3]: `distfiles' is up to date.
make[3]: Leaving directory `/home/dball/ovs/_gcc/datapath'
make[2]: Leaving directory `/home/dball/ovs/_gcc'
make[1]: Leaving directory `/home/dball/ovs/_gcc'
set /bin/bash '../tests/system-userspace-testsuite' -C tests  AUTOTEST_PATH='utilities:vswitchd:ovsdb:vtep:tests::ovn/controller-vtep:ovn/northd:ovn/utilities:ovn/controller' 90 -j1; \
	"$@" || (test X'' = Xyes && "$@" --recheck)
## ------------------------------ ##
## openvswitch 2.7.90 test suite. ##
## ------------------------------ ##
 90: ptap - triangle bridge setup with L2 and L3 GRE tunnels ok

## ------------- ##
## Test results. ##
## ------------- ##

1 test was successful.
make: Leaving directory `/home/dball/ovs/_gcc'


On 7/11/17, 9:09 AM, "Zoltán Balogh" <zoltan.balogh at ericsson.com> wrote:

    Hi, 
    
    
    
    I've created a new patch, that does check the OpenFlow statistics of integration bridges instead of the statistics of datapath flows. 
    
    I sent the patch to the dev list:
    
    https://urldefense.proofpoint.com/v2/url?u=https-3A__patchwork.ozlabs.org_patch_786669_&d=DwIGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=dGZmbKhBG9tJHY4odedsGA&m=J0GZfRvnrtFm9-jR_Xrx7-OSPXmzKTJP5oR9Ytha-bY&s=ZKoZ-eh3U1mBQyDz5uyP85-bgyYKD_cXwCcSKZj1RNU&e= 
    
    
    
    Best regards,
    
    Zoltan
    
    
    
    
    
    > -----Original Message-----
    
    > From: Jan Scheurich
    
    > Sent: Tuesday, July 11, 2017 10:26 AM
    
    > To: Zoltán Balogh <zoltan.balogh at ericsson.com>; Darrell Ball <dball at vmware.com>; Yang, Yi Y <yi.y.yang at intel.com>;
    
    > Eric Garver <e at erig.me>
    
    > Cc: 'dev at openvswitch.org' <dev at openvswitch.org>; simon.horman at netronome.com; Jiri Benc <jbenc at redhat.com>
    
    > Subject: RE: [ovs-dev] [PATCH v4 0/7] Packet type aware pipeline
    
    > 
    
    > I think the dropped packets are indeed noise generated by the host kernel when bringing up interfaces and are
    
    > dependent on system configuration and timing. Dumping datapath flows seems pretty vulnerable to such noise in system
    
    > tests and it may be hard to create filters that work reliably on all systems.
    
    > 
    
    > Since we are checking DP flows in the normal PTAP unit tests already, perhaps we can remove the check here and
    
    > instead check on something more predictable, e.g. tunnel port or stats or OpenFlow stats.
    
    > 
    
    > BR, Jan
    
    > 
    
    > > -----Original Message-----
    
    > > From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-bounces at openvswitch.org] On Behalf Of Zoltán Balogh
    
    > > Sent: Monday, 10 July, 2017 16:37
    
    > > To: Darrell Ball <dball at vmware.com>; Yang, Yi Y <yi.y.yang at intel.com>; Eric Garver <e at erig.me>
    
    > > Cc: 'dev at openvswitch.org' <dev at openvswitch.org>; simon.horman at netronome.com; Jiri Benc <jbenc at redhat.com>
    
    > > Subject: Re: [ovs-dev] [PATCH v4 0/7] Packet type aware pipeline
    
    > >
    
    > > Hi Darrell,
    
    > >
    
    > > Thank you for the log and the investigation. Unfortunately, I wasn't able to reproduce the fault.
    
    > > I used two systems:
    
    > >  - Ubuntu 17.04
    
    > >    Linux host1 4.10.0-26-generic #30-Ubuntu SMP Tue Jun 27 09:30:12 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
    
    > >
    
    > >  - Ubuntu 14.04.5 LTS
    
    > >    Linux host2 3.16.0-70-generic #90~14.04.1-Ubuntu SMP Wed Apr 6 22:56:34 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
    
    > >
    
    > > The dp ports 8-13 below are veth ports. Maybe, there is a system configuration, that affects behavior of the veth
    
    > ports.
    
    > >
    
    > > +recirc_id(0),in_port(10),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(dst=255.255.255.255/128.0.0.0,frag=no),
    
    > packets:2, bytes:684,
    
    > > used:0.0s, actions:drop
    
    > > +recirc_id(0),in_port(11),packet_type(ns=0,id=0),eth(dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
    
    > packets:2, bytes:684, used:0.0s,
    
    > > actions:drop
    
    > > +recirc_id(0),in_port(12),packet_type(ns=0,id=0),eth(dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
    
    > packets:2, bytes:684, used:0.0s,
    
    > > actions:drop
    
    > > +recirc_id(0),in_port(13),packet_type(ns=0,id=0),eth(dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
    
    > packets:2, bytes:684, used:0.0s,
    
    > > actions:drop
    
    > > +recirc_id(0),in_port(8),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(dst=255.255.255.255/128.0.0.0,frag=no),
    
    > packets:2, bytes:684,
    
    > > used:0.0s, actions:drop
    
    > > +recirc_id(0),in_port(9),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(dst=255.255.255.255/128.0.0.0,frag=no),
    
    > packets:3, bytes:1026,
    
    > > used:0.0s, actions:drop
    
    > >
    
    > >
    
    > > However there are other veth ports (dp ports: 15-17) used in the unit test setup, they connect namespaces to
    
    > bridges. If it's about a system
    
    > > level configuration that causes veth ports 8-13 to behave differently, I guess we should see broadcast packets
    
    > from the ports connecting
    
    > > namespaces and bridges too. But we do not. Those ports connect a namespace to a bridge, and there is an m4 macro
    
    > in the system-
    
    > > common-macros.at file to do this:
    
    > >
    
    > > # ADD_VETH([port], [namespace], [ovs-br], [ip_addr] [mac_addr [gateway]])
    
    > > #
    
    > > # Add a pair of veth ports. 'port' will be added to name space 'namespace',
    
    > > # and "ovs-'port'" will be added to ovs bridge 'ovs-br'.
    
    > > #
    
    > > # The 'port' in 'namespace' will be brought up with static IP address
    
    > > # with 'ip_addr' in CIDR notation.
    
    > > #
    
    > > # Optionally, one can specify the 'mac_addr' for 'port' and the default
    
    > > # 'gateway'.
    
    > > #
    
    > > # The existing 'port' or 'ovs-port' will be removed before new ones are added.
    
    > > #
    
    > > m4_define([ADD_VETH],
    
    > >     [ AT_CHECK([ip link add $1 type veth peer name ovs-$1 || return 77])
    
    > >       CONFIGURE_VETH_OFFLOADS([$1])
    
    > >       AT_CHECK([ip link set $1 netns $2])
    
    > >       AT_CHECK([ip link set dev ovs-$1 up])
    
    > >       AT_CHECK([ovs-vsctl add-port $3 ovs-$1 -- \
    
    > >                 set interface ovs-$1 external-ids:iface-id="$1"])
    
    > >       NS_CHECK_EXEC([$2], [ip addr add $4 dev $1])
    
    > >       NS_CHECK_EXEC([$2], [ip link set dev $1 up])
    
    > >       if test -n "$5"; then
    
    > >         NS_CHECK_EXEC([$2], [ip link set dev $1 address $5])
    
    > >       fi
    
    > >       if test -n "$6"; then
    
    > >         NS_CHECK_EXEC([$2], [ip route add default via $6])
    
    > >       fi
    
    > >       on_exit 'ip link del ovs-$1'
    
    > >     ]
    
    > > )
    
    > >
    
    > > I have not found any m4 macro to create veth port pairs for the unit test, so I used individual commands to create
    
    > the dp ports 8-13, for
    
    > > instance:
    
    > >
    
    > > AT_CHECK([ip link add p1-0 type veth peer name p0-1])
    
    > > AT_CHECK([ip link set p1-0 up])
    
    > > AT_CHECK([ip link set p0-1 up])
    
    > > AT_CHECK([ip link set dev p1-0 mtu 3300])
    
    > > AT_CHECK([ip link set dev p0-1 mtu 3300])
    
    > > on_exit 'ip link del p0-1'
    
    > >
    
    > > Well, there is a difference compared to the VETH_ADD macro. The ADD_VETH macro disables Tx offloading for the veth
    
    > port in the
    
    > > namespace. I don't know if this really matters. We could try to diable Tx offloading for ports 8-13.
    
    > >
    
    > > diff --git a/tests/system-userspace-packet-type-aware.at b/tests/system-userspace-packet-type-aware.at
    
    > > index 65d81ce8e..1ec6fa7f6 100644
    
    > > --- a/tests/system-userspace-packet-type-aware.at
    
    > > +++ b/tests/system-userspace-packet-type-aware.at
    
    > > @@ -51,6 +51,8 @@ AT_CHECK([ip link set p1-0 up])
    
    > >  AT_CHECK([ip link set p0-1 up])
    
    > >  AT_CHECK([ip link set dev p1-0 mtu 3300])
    
    > >  AT_CHECK([ip link set dev p0-1 mtu 3300])
    
    > > +CONFIGURE_VETH_OFFLOADS([p1-0])
    
    > > +CONFIGURE_VETH_OFFLOADS([p0-1])
    
    > >  on_exit 'ip link del p0-1'
    
    > >
    
    > >  AT_CHECK([ip link add p2-0 type veth peer name p0-2])
    
    > > @@ -58,6 +60,8 @@ AT_CHECK([ip link set p2-0 up])
    
    > >  AT_CHECK([ip link set p0-2 up])
    
    > >  AT_CHECK([ip link set dev p2-0 mtu 3300])
    
    > >  AT_CHECK([ip link set dev p0-2 mtu 3300])
    
    > > +CONFIGURE_VETH_OFFLOADS([p2-0])
    
    > > +CONFIGURE_VETH_OFFLOADS([p0-2])
    
    > >  on_exit 'ip link del p0-2'
    
    > >
    
    > >  AT_CHECK([ip link add p3-0 type veth peer name p0-3])
    
    > > @@ -65,6 +69,8 @@ AT_CHECK([ip link set p3-0 up])
    
    > >  AT_CHECK([ip link set p0-3 up])
    
    > >  AT_CHECK([ip link set dev p3-0 mtu 3300])
    
    > >  AT_CHECK([ip link set dev p0-3 mtu 3300])
    
    > > +CONFIGURE_VETH_OFFLOADS([p3-0])
    
    > > +CONFIGURE_VETH_OFFLOADS([p0-3])
    
    > >  on_exit 'ip link del p0-3'
    
    > >
    
    > >  # Setup bridge infrastructure
    
    > > ---
    
    > >
    
    > > If this does not solve the problem, we can still filter dp flow entries with 'drop' action out.
    
    > >
    
    > > diff --git a/tests/system-userspace-packet-type-aware.at b/tests/system-userspace-packet-type-aware.at
    
    > > index 1ec6fa7f6..9b0ace0fd 100644
    
    > > --- a/tests/system-userspace-packet-type-aware.at
    
    > > +++ b/tests/system-userspace-packet-type-aware.at
    
    > > @@ -348,7 +348,7 @@ NS_CHECK_EXEC([ns1], [ping -q -c 3 -i 0.3 -w 2 $N3_IP | FORMAT_PING], [0], [dnl
    
    > >  sleep 1
    
    > >
    
    > >  AT_CHECK([
    
    > > -    ovs-appctl dpctl/dump-flows | strip_used | grep -v ipv6 | grep -v arp |sort
    
    > > +    ovs-appctl dpctl/dump-flows | strip_used | grep -v ipv6 | grep -v arp | grep -v drop| sort
    
    > >  ], [0], [flow-dump from non-dpdk interfaces:
    
    > >
    
    > >
    
    > recirc_id(0),in_port(10),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:03),eth_type(0x0800),ipv4(src=10.0.0.1,dst=10
    
    > .0.0.3,proto=47,frag
    
    > > =no), packets:2, bytes:272, used:0.0s, actions:set(ipv4(src=30.0.0.1,dst=30.0.0.3)),tnl_pop(14)
    
    > >  recirc_id(0),in_port(11),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:03),eth_type(0x0800),ipv4(frag=no),
    
    > packets:2, bytes:272,
    
    > > used:0.0s, actions:13
    
    > > @@ -377,7 +377,7 @@ NS_CHECK_EXEC([ns1], [ping -q -c 3 -i 0.3 -w 2 $N2_IP | FORMAT_PING], [0], [dnl
    
    > >  sleep 1
    
    > >
    
    > >  AT_CHECK([
    
    > > -    ovs-appctl dpctl/dump-flows | strip_used | grep -v ipv6 | grep -v arp | sort
    
    > > +    ovs-appctl dpctl/dump-flows | strip_used | grep -v ipv6 | grep -v arp | grep -v drop | sort
    
    > >  ], [0], [flow-dump from non-dpdk interfaces:
    
    > >  recirc_id(0),in_port(11),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:02),eth_type(0x0800),ipv4(frag=no),
    
    > packets:2, bytes:272,
    
    > > used:0.0s, actions:12
    
    > >  recirc_id(0),in_port(12),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:01),eth_type(0x0800),ipv4(frag=no),
    
    > packets:2, bytes:272,
    
    > > used:0.0s, actions:11
    
    > > @@ -402,7 +402,7 @@ NS_CHECK_EXEC([ns3], [ping -q -c 3 -i 0.3 -w 2 $N1_IP | FORMAT_PING], [0], [dnl
    
    > >  sleep 1
    
    > >
    
    > >  AT_CHECK([
    
    > > -    ovs-appctl dpctl/dump-flows | strip_used | grep -v ipv6 | grep -v arp | sort
    
    > > +    ovs-appctl dpctl/dump-flows | strip_used | grep -v ipv6 | grep -v arp | grep -v drop | sort
    
    > >  ], [0], [flow-dump from non-dpdk interfaces:
    
    > >
    
    > >
    
    > recirc_id(0),in_port(10),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:03),eth_type(0x0800),ipv4(src=10.0.0.1,dst=10
    
    > .0.0.3,proto=47,frag
    
    > > =no), packets:2, bytes:272, used:0.0s, actions:set(ipv4(src=30.0.0.1,dst=30.0.0.3)),tnl_pop(14)
    
    > >  recirc_id(0),in_port(11),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:03),eth_type(0x0800),ipv4(frag=no),
    
    > packets:2, bytes:272,
    
    > > used:0.0s, actions:13
    
    > > ---
    
    > >
    
    > > Could you verify if any of the patches solves the problem?
    
    > >
    
    > > Otherwise we could simply remove the verification of dp flows.
    
    > >
    
    > > Best regards,
    
    > > Zoltan
    
    > >
    
    > >
    
    > >
    
    > > > -----Original Message-----
    
    > > > From: Darrell Ball [mailto:dball at vmware.com]
    
    > > > Sent: Friday, July 07, 2017 6:56 PM
    
    > > > To: Zoltán Balogh <zoltan.balogh at ericsson.com>; Yang, Yi Y <yi.y.yang at intel.com>; Eric Garver <e at erig.me>
    
    > > > Cc: 'dev at openvswitch.org' <dev at openvswitch.org>; simon.horman at netronome.com; Jiri Benc <jbenc at redhat.com>
    
    > > > Subject: Re: [ovs-dev] [PATCH v4 0/7] Packet type aware pipeline
    
    > > >
    
    > > > A few points worth noting:
    
    > > >
    
    > > > 1/ This test only uses the ping application and there are no advanced options, so my first guess is
    
    > > >      distro should not matter, but I could be wrong.
    
    > > >
    
    > > > 2/ This will not be fixed by increasing the sleep time before the check below; there are extra
    
    > > >      drops for broadcast packets.
    
    > > > There are 6 extra lines of them for this failure – see the below o/p.
    
    > > > The trailing checks in the test may fail of course.
    
    > > >              If these are noise, then they can be filtered. But an explanation would be needed.
    
    > > >
    
    > > >             I assume you added sleep because you want to make sure datapath flows are established and stats are
    
    > as
    
    > > > expected
    
    > > >             Again, adding more sleep for this part will not help here.
    
    > > >
    
    > > >            3/ Your group needs to be able to reproduce the issue in house.
    
    > > >                 We used standard Ubuntu 14.04 and 16.04 and also Ubuntu with upgraded kernel.
    
    > > >
    
    > > >            4/ The system tests should pass the far majority of the time.
    
    > > >
    
    > > >
    
    > > > Portion of test:
    
    > > >
    
    > > >  # Ping between N1 and N3, via the L2 GRE tunnel between br-in1 and br-in3
    
    > > > NS_CHECK_EXEC([ns1], [ping -q -c 3 -i 0.3 -w 2 $N3_IP | FORMAT_PING], [0], [dnl
    
    > > > 3 packets transmitted, 3 received, 0% packet loss, time 0ms
    
    > > > ])
    
    > > >
    
    > > > sleep 1
    
    > > >
    
    > > > AT_CHECK([
    
    > > >     ovs-appctl dpctl/dump-flows | strip_used | grep -v ipv6 | grep -v arp |sort
    
    > > > ], [0], [flow-dump from non-dpdk interfaces:
    
    > > >
    
    > recirc_id(0),in_port(10),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:03),eth_type(0x0800),ipv4(src=10.0.0.1,dst=10
    
    > > > .0.0.3,proto=47,frag=no), packets:2, bytes:272, used:0.0s,
    
    > actions:set(ipv4(src=30.0.0.1,dst=30.0.0.3)),tnl_pop(14)
    
    > > > recirc_id(0),in_port(11),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:03),eth_type(0x0800),ipv4(frag=no),
    
    > > > packets:2, bytes:272, used:0.0s, actions:13
    
    > > > recirc_id(0),in_port(12),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:01),eth_type(0x0800),ipv4(frag=no),
    
    > > > packets:2, bytes:244, used:0.0s, actions:11
    
    > > > recirc_id(0),in_port(13),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:02),eth_type(0x0800),ipv4(frag=no),
    
    > > > packets:2, bytes:244, used:0.0s, actions:12
    
    > > >
    
    > > >
    
    > > > relevant error output lines
    
    > > >
    
    > > >
    
    > recirc_id(0),in_port(10),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:03),eth_type(0x0800),ipv4(src=10.0.0.1,dst=10
    
    > > > .0.0.3,proto=47,frag=no), packets:2, bytes:272, used:0.0s,
    
    > actions:set(ipv4(src=30.0.0.1,dst=30.0.0.3)),tnl_pop(14)
    
    > > > +recirc_id(0),in_port(10),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(dst=255.255.255.255/128.0.0.0,frag=no),
    
    > > > packets:2, bytes:684, used:0.0s, actions:drop
    
    > > >  recirc_id(0),in_port(11),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:03),eth_type(0x0800),ipv4(frag=no),
    
    > > > packets:2, bytes:272, used:0.0s, actions:13
    
    > > > +recirc_id(0),in_port(11),packet_type(ns=0,id=0),eth(dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
    
    > > > packets:2, bytes:684, used:0.0s, actions:drop
    
    > > >  recirc_id(0),in_port(12),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:01),eth_type(0x0800),ipv4(frag=no),
    
    > > > packets:2, bytes:244, used:0.0s, actions:11
    
    > > > +recirc_id(0),in_port(12),packet_type(ns=0,id=0),eth(dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
    
    > > > packets:2, bytes:684, used:0.0s, actions:drop
    
    > > >  recirc_id(0),in_port(13),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:02),eth_type(0x0800),ipv4(frag=no),
    
    > > > packets:2, bytes:244, used:0.0s, actions:12
    
    > > > +recirc_id(0),in_port(13),packet_type(ns=0,id=0),eth(dst=ff:ff:ff:ff:ff:ff),eth_type(0x0800),ipv4(frag=no),
    
    > > > packets:2, bytes:684, used:0.0s, actions:drop
    
    > > >  recirc_id(0),in_port(15),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(dst=192.168.10.30,tos=0/0x3,frag=no),
    
    > > > packets:2, bytes:196, used:0.0s,
    
    > > >
    
    > actions:tnl_push(tnl_port(14),header(size=38,type=3,eth(dst=aa:55:00:00:00:03,src=aa:55:00:00:00:01,dl_type=0x0800),
    
    > > > ipv4(src=10.0.0.1,dst=10.0.0.3,proto=47,tos=0,ttl=64,frag=0x4000),gre((flags=0x0,proto=0x6558))),out_port(5))
    
    > > >  recirc_id(0),in_port(17),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(dst=192.168.10.10,tos=0/0x3,frag=no),
    
    > > > packets:2, bytes:196, used:0.0s,
    
    > > >
    
    > actions:pop_eth,tnl_push(tnl_port(14),header(size=38,type=3,eth(dst=aa:55:00:00:00:02,src=aa:55:00:00:00:03,dl_type=
    
    > > >
    
    > 0x0800),ipv4(src=30.0.0.3,dst=30.0.0.2,proto=47,tos=0,ttl=64,frag=0x4000),gre((flags=0x0,proto=0x800))),out_port(7))
    
    > > >  recirc_id(0),in_port(5),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:2, bytes:272, used:0.0s,
    
    > > > actions:8
    
    > > >  recirc_id(0),in_port(6),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:2, bytes:244, used:0.0s,
    
    > > > actions:9
    
    > > >  recirc_id(0),in_port(7),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(frag=no), packets:2, bytes:244, used:0.0s,
    
    > > > actions:10
    
    > > >
    
    > > >
    
    > recirc_id(0),in_port(8),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:01),eth_type(0x0800),ipv4(src=20.0.0.2,dst=20.
    
    > > > 0.0.1,proto=47,frag=no), packets:2, bytes:244, used:0.0s,
    
    > actions:set(ipv4(src=10.0.0.2,dst=10.0.0.1)),tnl_pop(14)
    
    > > > +recirc_id(0),in_port(8),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(dst=255.255.255.255/128.0.0.0,frag=no),
    
    > > > packets:2, bytes:684, used:0.0s, actions:drop
    
    > > >
    
    > > >
    
    > recirc_id(0),in_port(9),packet_type(ns=0,id=0),eth(dst=aa:55:00:00:00:02),eth_type(0x0800),ipv4(src=30.0.0.3,dst=30.
    
    > > > 0.0.2,proto=47,frag=no), packets:2, bytes:244, used:0.0s,
    
    > actions:set(ipv4(src=20.0.0.3,dst=20.0.0.2)),tnl_pop(14)
    
    > > > +recirc_id(0),in_port(9),packet_type(ns=0,id=0),eth_type(0x0800),ipv4(dst=255.255.255.255/128.0.0.0,frag=no),
    
    > > > packets:3, bytes:1026, used:0.0s, actions:drop
    
    > > >  tunnel(src=10.0.0.2,dst=10.0.0.1,flags(-df-
    
    > > > csum)),recirc_id(0),in_port(14),packet_type(ns=1,id=0x800),ipv4(dst=192.168.10.10,frag=no), packets:2,
    
    > bytes:168,
    
    > > > used:0.0s, actions:push_eth(src=00:00:00:00:00:00,dst=aa:55:aa:55:00:01),15
    
    > > >  tunnel(src=20.0.0.3,dst=20.0.0.2,flags(-df-
    
    > > > csum)),recirc_id(0),in_port(14),packet_type(ns=1,id=0x800),ipv4(dst=192.168.10.10,tos=0/0x3,frag=no), packets:2,
    
    > > > bytes:168, used:0.0s,
    
    > > >
    
    > actions:tnl_push(tnl_port(14),header(size=38,type=3,eth(dst=aa:55:00:00:00:01,src=aa:55:00:00:00:02,dl_type=0x0800),
    
    > > > ipv4(src=20.0.0.2,dst=20.0.0.1,proto=47,tos=0,ttl=64,frag=0x4000),gre((flags=0x0,proto=0x800))),out_port(6))
    
    > > >  tunnel(src=30.0.0.1,dst=30.0.0.3,flags(-df-
    
    > > >
    
    > csum)),recirc_id(0),in_port(14),packet_type(ns=0,id=0),eth(dst=aa:55:aa:55:00:03),eth_type(0x0800),ipv4(dst=192.168.
    
    > > > 10.30,frag=no), packets:2, bytes:196, used:0.0s, actions:17
    
    > > >
    
    > > >
    
    > > > On 7/7/17, 12:23 AM, "Zoltán Balogh" <zoltan.balogh at ericsson.com> wrote:
    
    > > >
    
    > > >     Hi,
    
    > > >
    
    > > >
    
    > > >
    
    > > >     The script seems to be stopped at line 344 in system-userspace-packet-type-aware.at. Checking root privilege
    
    > > > happens before that. There is a "sleep 1" command before line 344. Maybe it's too short on some systems.
    
    > > >
    
    > > >
    
    > > >
    
    > > >     Darell, could you share the log file of this test case, please? It should be :
    
    > > >
    
    > > >     tests/system-userspace-testsuite.dir/94/system-userspace-testsuite.log
    
    > > >
    
    > > >
    
    > > >
    
    > > >     If the log file too large, then you could try to grep for the relevant part:
    
    > > >
    
    > > >      grep -A 40 "\--- " tests/system-userspace-testsuite.dir/94/system-userspace-testsuite.log
    
    > > >
    
    > > >
    
    > > >
    
    > > >
    
    > > >
    
    > > >
    
    > > >     Thank you,
    
    > > >
    
    > > >     Zoltan
    
    > > >
    
    > > >
    
    > > >
    
    > > >     > -----Original Message-----
    
    > > >
    
    > > >     > From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-bounces at openvswitch.org] On Behalf Of Yang, Yi Y
    
    > > >
    
    > > >     > Sent: Friday, July 07, 2017 3:36 AM
    
    > > >
    
    > > >     > To: Darrell Ball <dball at vmware.com>; Eric Garver <e at erig.me>
    
    > > >
    
    > > >     > Cc: 'dev at openvswitch.org' <dev at openvswitch.org>; simon.horman at netronome.com; Jiri Benc <jbenc at redhat.com>
    
    > > >
    
    > > >     > Subject: Re: [ovs-dev] [PATCH v4 0/7] Packet type aware pipeline
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     > I think it needs root privilege to run.
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     > -----Original Message-----
    
    > > >
    
    > > >     > From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-bounces at openvswitch.org] On Behalf Of Darrell Ball
    
    > > >
    
    > > >     > Sent: Friday, July 7, 2017 5:06 AM
    
    > > >
    
    > > >     > To: Eric Garver <e at erig.me>
    
    > > >
    
    > > >     > Cc: 'dev at openvswitch.org' <dev at openvswitch.org>; simon.horman at netronome.com; Jiri Benc <jbenc at redhat.com>
    
    > > >
    
    > > >     > Subject: Re: [ovs-dev] [PATCH v4 0/7] Packet type aware pipeline
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     > btw, I asked Andy to check on his Trusty system and he observes the same failure.
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     > On 7/6/17, 1:39 PM, "Darrell Ball" <dball at vmware.com> wrote:
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >     On 7/6/17, 1:22 PM, "Eric Garver" <e at erig.me> wrote:
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >         On Thu, Jul 06, 2017 at 03:59:51PM +0000, Darrell Ball wrote:
    
    > > >
    
    > > >     >         > Patch 4 added unit tests
    
    > > >
    
    > > >     >         > One test was added to system userspace.
    
    > > >
    
    > > >     >         > I ran it many times and it has never passed.
    
    > > >
    
    > > >     >         > Is something missing in the test or supporting code ?
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >         Works for me. Maybe there is a timing problem?
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >     Thanks
    
    > > >
    
    > > >     >     There are races in the system tests, but those tests will pass on a second attempt.
    
    > > >
    
    > > >     >     This is 100% failure.
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >         OVS: current master (fb16fec66498)
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >     Tip of master - same for me
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >         Kernel: 4.12.0-rc7+
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >     I tried 2 kernels 4.2.0 and 3.19 – always fails in both.
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >         OS: RHEL 7.4-ish
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >     I really hope the distro is not relevant here
    
    > > >
    
    > > >     >     If you have reason to think so, pls speak up
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         > ## ------------------------------ ##
    
    > > >
    
    > > >     >         > ## openvswitch 2.7.90 test suite. ##
    
    > > >
    
    > > >     >         > ## ------------------------------ ##
    
    > > >
    
    > > >     >         >  94: ptap - triangle bridge setup with L2 and L3 GRE tunnels FAILED (system-userspace-packet-
    
    > type-
    
    > > >
    
    > > >     > aware.at:344)
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         > ## ------------- ##
    
    > > >
    
    > > >     >         > ## Test results. ##
    
    > > >
    
    > > >     >         > ## ------------- ##
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         > ERROR: 1 test was run,
    
    > > >
    
    > > >     >         > 1 failed unexpectedly.
    
    > > >
    
    > > >     >         > ## ------------------------------------------- ##
    
    > > >
    
    > > >     >         > ## system-userspace-testsuite.log was created. ##
    
    > > >
    
    > > >     >         > ## ------------------------------------------- ##
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         > Please send `tests/system-userspace-testsuite.log' and all information you think might help:
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         >    To: <bugs at openvswitch.org>
    
    > > >
    
    > > >     >         >    Subject: [openvswitch 2.7.90] system-userspace-testsuite: 94 failed
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         > You may investigate any problem if you feel able to do so, in which
    
    > > >
    
    > > >     >         > case the test suite provides a good starting point.  Its output may
    
    > > >
    
    > > >     >         > be found below `tests/system-userspace-testsuite.dir'.
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         > make: *** [check-system-userspace] Error 1
    
    > > >
    
    > > >     >         > make: Leaving directory `/home/dball/openvswitch/ovs/_gcc'
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         > On 6/27/17, 2:29 PM, "ovs-dev-bounces at openvswitch.org on behalf of Ben Pfaff" <ovs-dev-
    
    > > >
    
    > > >     > bounces at openvswitch.org on behalf of blp at ovn.org> wrote:
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         >     On Fri, Jun 23, 2017 at 04:47:48PM +0000, Zoltán Balogh wrote:
    
    > > >
    
    > > >     >         >     >
    
    > > >
    
    > > >     >         >     > This series was started by Ben Pfaff, v3 can be found here:
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_778070_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=01fMZLg-sq_Rf_2X9TcxmtDN0Vx2hS2v4BABA77en0w&e=
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_778071_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=5fpQZh33OQ_0AGjRuJNQqvS-1E6yhFoD5TxUoLayzwM&e=
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_778076_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=rqnhEoIuJ9rECQaXZafZGrAW9us8y184GCexqK8VUxQ&e=
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_778072_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=p5Fnh2SQ8fUrNF0V8ChNRQyQLmbjiuvKRd1UUwE93s8&e=
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_778074_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=02CPoSKkiGGsFX4neL8TpGa75VUPaaeNYWz_2WbqBsQ&e=
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_778073_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=eMl7m4NpJSD07nXTN_oOdQFvysmJnclSRw39GgGg9BQ&e=
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_778075_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=71zSrD_kcUjZwuKvjfeqam_rdCpeXuxtm4TW_SsZ_3w&e=
    
    > > >
    
    > > >     >         >     >
    
    > > >
    
    > > >     >         >     > Ben's series is based on this one:
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_770490_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=W1AcbVTr4qKOBEfQuMSmVH-Illfhs0R3TYJs1s9HtfU&e=
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_770487_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=EHhVEHj-Pvdl0I609BkK_Z44CV-641MVWlLy7rZH6Qo&e=
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_770495_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=izZUroix8O9Bcg5XNCmvsEseCctZfho3ovxvZRz0rwI&e=
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_770498_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=VFf2BDMVyA1OU5h0hCBJFboZtp_U2NlqmPYkLDGnWnE&e=
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_770488_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=3_rtktraZwXdr6VkYz8ujpi_eV9MDKc8EwCDk43nnLg&e=
    
    > > >
    
    > > >     >         >     > https://urldefense.proofpoint.com/v2/url?u=https-
    
    > > >
    
    > > >     > 3A__patchwork.ozlabs.org_patch_770489_&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=1OhDZfiGpq3GF4Vie9KKwKudHEAHpG8nj8qTMqlmKV8&e=
    
    > > >
    
    > > >     >         >     >
    
    > > >
    
    > > >     >         >     > v1->v2:
    
    > > >
    
    > > >     >         >     >   - Squash fixup patches.
    
    > > >
    
    > > >     >         >     >   - Apply changes agreed with Jan.
    
    > > >
    
    > > >     >         >     >   - Not yet done: Figure out whether to really show packet_type in (some)
    
    > > >
    
    > > >     >         >     >     match_format() output.
    
    > > >
    
    > > >     >         >     >   - New patch at the end unsuccessfully tries to re-enable packet-aware
    
    > > >
    
    > > >     >         >     >     test.  Either I don't have enough insight yet, or it just reveals a
    
    > > >
    
    > > >     >         >     >     bug or two.
    
    > > >
    
    > > >     >         >     >   - 4 new patches at beginning.  First one is trivial.  Next 3 are intended
    
    > > >
    
    > > >     >         >     >     to make it easier to debug the packet aware test that is still failing.
    
    > > >
    
    > > >     >         >     >     Jan, you don't have to feel obligated to review these if you feel they
    
    > > >
    
    > > >     >         >     >     are off-topic; I will get separate reviews.
    
    > > >
    
    > > >     >         >     >
    
    > > >
    
    > > >     >         >     > v2->v3:
    
    > > >
    
    > > >     >         >     >   - Drop first two patches, which have been applied to master.
    
    > > >
    
    > > >     >         >     >   - Drop patches 3 and 4, which have been moved to a new series.
    
    > > >
    
    > > >     >         >     >   - Drop last patch, which was incomplete.  I'll hope that Jan or Zoltan
    
    > > >
    
    > > >     >         >     >     can pick it back up.
    
    > > >
    
    > > >     >         >     >   - Apply Jan's and Zoltan's comments on a few other patches.
    
    > > >
    
    > > >     >         >     >
    
    > > >
    
    > > >     >         >     > v3->v4:
    
    > > >
    
    > > >     >         >     >   - Rebase to recent origin/master (0722f3410).
    
    > > >
    
    > > >     >         >     >   - Re-introduce packet-type aware unit tests.
    
    > > >
    
    > > >     >         >     >   - Unwildcard dl_type only for packet_type=PT_ETH.
    
    > > >
    
    > > >     >         >     >   - Apply comments.
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         >     Thanks.  I applied this series to master.
    
    > > >
    
    > > >     >         >     _______________________________________________
    
    > > >
    
    > > >     >         >     dev mailing list
    
    > > >
    
    > > >     >         >     dev at openvswitch.org
    
    > > >
    
    > > >     >         >     https://urldefense.proofpoint.com/v2/url?u=https-
    
    > 3A__mail.openvswitch.org_mailman_listinfo_ovs-
    
    > > >
    
    > > >     > 2Ddev&d=DwIFAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=YZOYZFR6QorkvkAZXXYPq7Z9SWuK0lZg9EH6ND2me04&s=eFnoeVgPHfExQ3X4kGuiRhqxnn-iw7rq2RoY9fvwj4I&e=
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         >
    
    > > >
    
    > > >     >         > _______________________________________________
    
    > > >
    
    > > >     >         > dev mailing list
    
    > > >
    
    > > >     >         > dev at openvswitch.org
    
    > > >
    
    > > >     >         > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.openvswitch.org_mailman_listinfo_ovs-
    
    > > >
    
    > > >     > 2Ddev&d=DwIDAw&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-
    
    > > >
    
    > > >     > uZnsw&m=msEftCAccuIB4BWrvoXOXgDDsp7LUK5CHuIscL6JtsY&s=wbZ8iJo4O-gC9UmULTSz6NABL3Dmz-9d-Pg_28JNlmE&e=
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     >
    
    > > >
    
    > > >     > _______________________________________________
    
    > > >
    
    > > >     > dev mailing list
    
    > > >
    
    > > >     > dev at openvswitch.org
    
    > > >
    
    > > >     > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.openvswitch.org_mailman_listinfo_ovs-
    
    > > > 2Ddev&d=DwIGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
    
    > > >
    
    > > uEs&r=dGZmbKhBG9tJHY4odedsGA&m=L7nAsK74d2EKUrXtQVIVX1ExwBLsoe6W3XwyyU6lvi0&s=lEqDjzoYGuE8UKddAndoqJC_x_HwlzECAD
    
    > > Vds-
    
    > > > 2P0uU&e=
    
    > > >
    
    > > >     > _______________________________________________
    
    > > >
    
    > > >     > dev mailing list
    
    > > >
    
    > > >     > dev at openvswitch.org
    
    > > >
    
    > > >     > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.openvswitch.org_mailman_listinfo_ovs-
    
    > > > 2Ddev&d=DwIGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-
    
    > > >
    
    > > uEs&r=dGZmbKhBG9tJHY4odedsGA&m=L7nAsK74d2EKUrXtQVIVX1ExwBLsoe6W3XwyyU6lvi0&s=lEqDjzoYGuE8UKddAndoqJC_x_HwlzECAD
    
    > > Vds-
    
    > > > 2P0uU&e=
    
    > > >
    
    > > >
    
    > >
    
    > > _______________________________________________
    
    > > dev mailing list
    
    > > dev at openvswitch.org
    
    > > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.openvswitch.org_mailman_listinfo_ovs-2Ddev&d=DwIGaQ&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=dGZmbKhBG9tJHY4odedsGA&m=J0GZfRvnrtFm9-jR_Xrx7-OSPXmzKTJP5oR9Ytha-bY&s=PZHZ3aYSWhWl66NiQb8vUTrqT7tkWxkRwZr70uKf3GQ&e= 
    
    
    
    



More information about the dev mailing list