[ovs-dev] Possible Regression due to "ossfuzz: Break flow test target into two targets to speed up fuzzing."

Mark Michelson mmichels at redhat.com
Tue Oct 16 14:22:24 UTC 2018


On 10/15/2018 05:31 PM, Lorenzo Bianconi wrote:
>>
>> On Fri, Oct 12, 2018 at 08:54:49AM -0700, Ben Pfaff wrote:
>>> On Fri, Oct 12, 2018 at 03:44:51PM +0300, Ilya Maximets wrote:
>>>>> Hi,
>>>>>
>>>>> it seems that travis-ci is failing due to a testsuite regression introduced
>>>>> by 1adcbcee8f4c ("ossfuzz: Break flow test target into two targets to speed
>>>>> up fuzzing.").
>>>>>
>>>>> https://travis-ci.org/openvswitch/ovs/jobs/439811394
>>>>
>>>> Hi Simon,
>>>>
>>>> As it was already said, this is not the patch that produces the issue.
>>>> I'm able to reproduce '2649: ovn -- 3 HVs, 3 LS, 3 lports/LS, 1 LR'
>>>> test failure locally by restricting to use only one CPU core with taskset.
>>>> So, I bisected the issue and found that the fist patch that fails is:
>>>>
>>>>      2e5cdb4b1392 ("OVN: add buffering support for ip packets").
>>>>
>>>> Issue happens only if single core assigned:
>>>>
>>>>      taskset -c 2   make check TESTSUITEFLAGS='2649'  #FAILS
>>>>      taskset -c 2-3 make check TESTSUITEFLAGS='2649'  #OK
>>>>
>>>> As Travis has no much CPU resources it fails all the time.
>>>>
>>>> I'm not much familiar with OVN code/tests, hope above information will
>>>> be useful for your investigation.
>>>
>>> I can reproduce this too, with your hint (although it's test 3309, not
>>> 2649).
>>>
>>> The failure is because a lot of expected packets don't show up:
>>>
>>>      ...
>>>      rcv_n=212 exp_n=233
>>>      ovn.at:12: wait failed after 10 seconds
>>>
>>> It needs more looking into.
>>
>> I don't have time to look into it, but we need the tests passing, so I
>> sent a revert: https://patchwork.ozlabs.org/patch/984336/
> 
> Hi all,
> 
> I looked into it and it is probably a timing issue. In particular the
> ARP replies generated during test 8
> do not reach the router before the ip buffering gc time-out so we will
> end up having less packets than
> expected.
> A trivial solution is to invert test 8 and 4,5,6 or maybe we can add a
> unixctl command to disable ip
> buffering.
> Could you please try following patch? it fixes in my tests
> 
> Regards,
> Lorenzo

Chiming in to confirm that with the patch below applied, running 
`taskset -c 2 make check TESTSUITEFLAGS=2649` passes. Without the patch, 
the test fails for me.

> 
> diff --git a/tests/ovn.at b/tests/ovn.at
> index 002ef0dfb..562d96533 100644
> --- a/tests/ovn.at
> +++ b/tests/ovn.at
> @@ -2673,17 +2673,62 @@ test_arp() {
>       fi
>   }
> 
> +: > mac_bindings.expected
> +# 4. Generate an ARP reply for each of the IP addresses ARPed for
> +#    earlier as #3.
> +#
> +#    Here, the $s is the VIF that originated the ARP request and $d is
> +#    the VIF that sends the ARP reply, which is somewhat backward but
> +#    it means that $s and $d are the same as #3.
> +for is in 1 2 3; do
> +  for js in 1 2 3; do
> +    for ks in 1 2 3; do
> +      s=$is$js$ks
> +      for id in 1 2 3; do
> +        for jd in 1 2 3; do
> +          if test $is$js = $id$jd; then
> +            continue
> +          fi
> +
> +          kd=1
> +          d=$id$jd$kd
> +
> +          o4=`expr $is '*' 9 + $js '*' 3 + $ks + 10`
> +          host_ip=`ip_to_hex 192 168 $id$jd $o4`
> +          host_mac=8000000000$o4
> +
> +          lrmac=00000000ff$id$jd
> +          lrip=`ip_to_hex 192 168 $id$jd 254`
> +
> +          arp=${lrmac}${host_mac}08060001080006040002${host_mac}${host_ip}${lrmac}${lrip}
> +
> +          echo
> +          echo
> +          echo
> +          hv=hv`vif_to_hv $d`
> +          as $hv ovs-appctl netdev-dummy/receive vif$d $arp
> +          #as $hv ovs-appctl ofproto/trace br-int in_port=$d $arp
> +          #as $hv ovs-ofctl dump-flows br-int table=19
> +
> +          host_ip_pretty=192.168.$id$jd.$o4
> +          host_mac_pretty=80:00:00:00:00:$o4
> +          echo lrp$id$jd,$host_ip_pretty,$host_mac_pretty >>
> mac_bindings.expected
> +        done
> +      done
> +    done
> +  done
> +done
> +
>   # Test router replies to ARP requests from all source ports:
>   #
> -# 4. Router replies to query for its MAC address from port's own IP address.
> +# 5. Router replies to query for its MAC address from port's own IP address.
>   #
> -# 5. Router replies to query for its MAC address from any random IP address
> +# 6. Router replies to query for its MAC address from any random IP address
>   #    in its subnet.
>   #
> -# 6. No reply to query for IP address other than router IP.
> +# 7. No reply to query for IP address other than router IP.
>   #
> -# 7. No reply to query from another subnet.
> -: > mac_bindings.expected
> +# 8. No reply to query from another subnet.
>   for i in 1 2 3; do
>     for j in 1 2 3; do
>       for k in 1 2 3; do
> @@ -2727,50 +2772,6 @@ for i in 1 2 3; do
>     done
>   done
> 
> -# 8. Generate an ARP reply for each of the IP addresses ARPed for
> -#    earlier as #3.
> -#
> -#    Here, the $s is the VIF that originated the ARP request and $d is
> -#    the VIF that sends the ARP reply, which is somewhat backward but
> -#    it means that $s and $d are the same as #3.
> -for is in 1 2 3; do
> -  for js in 1 2 3; do
> -    for ks in 1 2 3; do
> -      s=$is$js$ks
> -      for id in 1 2 3; do
> -        for jd in 1 2 3; do
> -          if test $is$js = $id$jd; then
> -            continue
> -          fi
> -
> -          kd=1
> -          d=$id$jd$kd
> -
> -          o4=`expr $is '*' 9 + $js '*' 3 + $ks + 10`
> -          host_ip=`ip_to_hex 192 168 $id$jd $o4`
> -          host_mac=8000000000$o4
> -
> -          lrmac=00000000ff$id$jd
> -          lrip=`ip_to_hex 192 168 $id$jd 254`
> -
> -          arp=${lrmac}${host_mac}08060001080006040002${host_mac}${host_ip}${lrmac}${lrip}
> -
> -          echo
> -          echo
> -          echo
> -          hv=hv`vif_to_hv $d`
> -          as $hv ovs-appctl netdev-dummy/receive vif$d $arp
> -          #as $hv ovs-appctl ofproto/trace br-int in_port=$d $arp
> -          #as $hv ovs-ofctl dump-flows br-int table=19
> -
> -          host_ip_pretty=192.168.$id$jd.$o4
> -          host_mac_pretty=80:00:00:00:00:$o4
> -          echo lrp$id$jd,$host_ip_pretty,$host_mac_pretty >>
> mac_bindings.expected
> -        done
> -      done
> -    done
> -  done
> -done
> 
>   # Allow some time for packet forwarding.
>   # XXX This can be improved.
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> 



More information about the dev mailing list