[ovs-dev] [PATCH v2 5/5] dpif-lookup: add avx512 gather implementation

Tue May 26 14:52:04 UTC 2020

> -----Original Message-----
> From: William Tu <u9012063 at gmail.com>
> Sent: Thursday, May 21, 2020 11:30 PM
> To: Van Haaren, Harry <harry.van.haaren at intel.com>
> Cc: ovs-dev at openvswitch.org; i.maximets at ovn.org
> Subject: Re: [ovs-dev] [PATCH v2 5/5] dpif-lookup: add avx512 gather
> implementation
> 
> > > ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev
> > > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x3
> > > ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
> > > ovs-ofctl add-flow br0 'actions=drop'
> > > ovs-appctl dpif-netdev/subtable-lookup-set avx512_gather 5
> > > ovs-vsctl add-port br0 tg0 -- set int tg0 type=dpdk \
> > >   options:dpdk-
> > > devargs=vdev:net_pcap0,rx_pcap=/root/ovs/p0.pcap,infinite_rx=1
> >
> > I use Ether/VLAN/IPv4 to achieve a subtable with (4,1), is that the same as
> you?
> > Just trying to remove variables between our setups.
> >
> btw I have only one OpenFlow rule, 'actions=drop'
> The pcap file as input is a random one I just capture in my machine's interface
> root at instance-3:~/ovs# tcpdump -n -r p0.pcap  | wc -l
> reading from file p0.pcap, link-type EN10MB (Ethernet)
> 22
> root at instance-3:~/ovs# tcpdump -n -r p0.pcap

Hi William,

Thanks for the info - I was away on a longer weekend, back now!

<snip lots of pcap & details>

> > > 2020-05-20T13:49:26.648Z|00057|netdev_dpdk|INFO|Port 0:
> 02:70:63:61:70:00
> > > 2020-05-20T13:49:26.648Z|00058|dpif_netdev|INFO|Core 0 on numa node 0
> > > assigned port 'tg0' rx queue 0 (measured processing cycles 0).
> > > 2020-05-20T13:49:26.648Z|00059|bridge|INFO|bridge br0: added interface
> > > tg0 on port 1
> > > 2020-05-20T13:49:26.648Z|00001|ofproto_dpif_upcall(pmd-
> > > c00/id:9)|WARN|upcall_cb
> > > failure: ukey installation fails
> > > 2020-05-20T13:49:27.562Z|00002|dpif_netdev(pmd-
> c00/id:9)|INFO|reprobing
> > > sub func, 4 1
> >
> > Aha! This shows somethings going wrong - there should not be any ukey-
> install fails!
> >
> > This also explains why your logs (as per follow-up email in thread) have a high
> upcall count/sec,
> > the installed flow isn't being hit when matched. I'm not sure what the root
> cause of these
> > ukey-installation fails are - but this is what we need to investigate :)
> >
> > Understanding the traffic, and attempting to reproduce here would a good
> step forward.
> >
> > Would you describe the traffic contained in the pcap?
> > Is it a single packet, or something that should hit a single DPCLS wildcarded
> flow?
> >
> describe in comment above.

Thanks - the details of the pcap should be enough for me to debug from here.

<snip>

> > > > > 5) pmd-stat-show
> > > > > root at instance-3:~/ovs# ovs-appctl dpif-netdev/pmd-stats-show
> > > > > pmd thread numa_id 0 core_id 0:
> > > > >   packets received: 19838528
> > > > >   packet recirculations: 0
> > > > >   avg. datapath passes per packet: 1.00
> > > > >   emc hits: 0
> > > > >   smc hits: 0
> > > > >   megaflow hits: 0
> > > > >   avg. subtable lookups per megaflow hit: 0.00  (---> this doesn't
> > > > > look right ....)
> > > > >   miss with success upcall: 78
> > > > >   miss with failed upcall: 19838418
> > > > >   avg. packets per output batch: 2.00
> > > > >   idle cycles: 0 (0.00%)
> > > > >   processing cycles: 103431787838 (100.00%)
> > > > >   avg cycles per packet: 5213.68 (103431787838/19838528)
> > > > >   avg processing cycles per packet: 5213.68 (103431787838/19838528)
> > > >
> > > > Would you try the pmd-stats-show command before setting the AVX512
> > > lookup?
> Yes.
> before setting avx512:
> root at instance-3:~/ovs# ovs-appctl dpif-netdev/pmd-stats-show
> pmd thread numa_id 0 core_id 0:
>   packets received: 70630720
>   packet recirculations: 0
>   avg. datapath passes per packet: 1.00
>   emc hits: 64206054
>   smc hits: 0
>   megaflow hits: 6424309
>   avg. subtable lookups per megaflow hit: 1.00
>   miss with success upcall: 1
>   miss with failed upcall: 324
>   avg. packets per output batch: 0.00
>   idle cycles: 1668002 (0.01%)
>   processing cycles: 17710219822 (99.99%)
>   avg cycles per packet: 250.77 (17711887824/70630720)
>   avg processing cycles per packet: 250.74 (17710219822/70630720)

Thanks - this looks good - as expected.

> > > > If the issue is still present it would indicate its not related to the exact
> lookup
> > > > implementation.
> > >
> > > Before setting AVX512
> > > ### Scalar Lookup
> > > pmd thread numa_id 0 core_id 0:
> > >   packets received: 77470176
> > >   packet recirculations: 0
> > >   avg. datapath passes per packet: 1.00
> > >   emc hits: 70423947
> > >   smc hits: 0
> > >   megaflow hits: 7045897
> > >   avg. subtable lookups per megaflow hit: 1.00
> > >   miss with success upcall: 1
> > >   miss with failed upcall: 331
> > >   avg. packets per output batch: 0.00
> > >   idle cycles: 0 (0.00%)
> > >   processing cycles: 19596627706 (100.00%)
> > >   avg cycles per packet: 252.96 (19596627706/77470176)
> > >   avg processing cycles per packet: 252.96 (19596627706/77470176)
> > >
> > > ### AVX512 Lookup (restart ovs-vswitchd with additional command
> > > "dpif-netdev/subtable-lookup-set avx512_gather 5"
> > > pmd thread numa_id 0 core_id 0:
> > >   packets received: 1178784
> > >   packet recirculations: 0
> > >   avg. datapath passes per packet: 1.00
> > >   emc hits: 0
> > >   smc hits: 0
> > >   megaflow hits: 0
> > >   avg. subtable lookups per megaflow hit: 0.00
> > >   miss with success upcall: 13
> > >   miss with failed upcall: 1178739     ---> this looks not right
> > >   avg. packets per output batch: 0.00
> > >   idle cycles: 0 (0.00%)
> > >   processing cycles: 5408870500 (100.00%)
> > >   avg cycles per packet: 4588.52 (5408870500/1178784)
> > >   avg processing cycles per packet: 4588.52 (5408870500/1178784)
> >
> > The statistics seem accurate (but indeed the upcall count is unexpected and
> too high).
> > This aligns with a ukey-install fail as noted in the logs above.
> >
> > This seems to indicate that with the AVX512 lookup the ukey install fails.
> > I'd like to reproduce to investigate - above questions about traffic/rules
> > is hopefully enough to identify.
> 
> Why ukey is related here? Does you avx512 patch make any change to ukey?

No AVX512 doesn't make any ukey changes - but issues in the hashing of the
miniflow data blocks cause ukeys to be installed in different locations than where
they are looked up - hence "ukey install fail" == "issue in miniflow iteration" in this
context.

> > There is an alternative - set the "autovalidator" DPCLS implementation to
> > the highest priority, and it should ovs_assert() if the scalar/AVX512
> implementations
> > mismatch. Then a dump of the OVS miniflow will give what's needed to verify
> root cause.
> >
> that's a cool feature.
> When setting
> ovs-appctl dpif-netdev/subtable-lookup-set autovalidator 100
> log shows
> 2020-05-21T22:28:55.964Z|77007|dpif_lookup_autovalidator(pmd-
> c00/id:9)|ERR|matches_good
> 7 != matches_test 0 for func avx512_gather

Brilliant - this is exactly why the autovalidator is there. It has correctly flagged
an issue here - I've reproduced using a pcap and your commands above. I will
investigate a fix and include in the v3.

Thanks for the details - will keep you all posted on progress. -Harry