[ovs-dev] [PATCH ovn v4 2/2] system-ovn.at: Fix flaky "load-balancing" test (all servers targetted).

[Mark Michelson]

Hi Xavier,

Thanks for the fix on this. Looks good to me.

Acked-by: Mark Michelson <mmichels at redhat.com>

On 9/10/21 8:23 AM, Xavier Simonart wrote:
> This test sends requests to LB and checks that each server receives at
> least one request. However, even if 20 requests are sent to 3 servers,
> there was a possibility that one server did not get any request (0.3%).
> 
> Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=1989517
> Signed-off-by: Xavier Simonart <xsimonar at redhat.com>
> ---
>   tests/system-ovn.at | 55 ++++++++++++++++++++++-----------------------
>   1 file changed, 27 insertions(+), 28 deletions(-)
> 
> diff --git a/tests/system-ovn.at b/tests/system-ovn.at
> index aadd68634..345384223 100644
> --- a/tests/system-ovn.at
> +++ b/tests/system-ovn.at
> @@ -1449,42 +1449,42 @@ OVS_START_L7([bar2], [http])
>   OVS_START_L7([bar3], [http])
>   
>   dnl Should work with the virtual IP 30.0.0.1 address through NAT
> -for i in `seq 1 20`; do
> -    echo Request $i
> -    NS_CHECK_EXEC([foo1], [wget 30.0.0.1 -t 5 -T 1 --retry-connrefused -v -o wget$i.log])
> -done
> -
>   dnl Each server should have at least one connection.
> -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(30.0.0.1) | \
> -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> +dnl With 20 requests, one server might not receive any connection
> +dnl in 0.3% of cases, so run a few times.
> +
> +OVS_WAIT_FOR_OUTPUT([
> +    for i in `seq 1 20`; do
> +        ip netns exec foo1 wget 30.0.0.1 -t 5 -T 1 --retry-connrefused -v -o wget$i.log;
> +    done
> +    ovs-appctl dpctl/dump-conntrack | FORMAT_CT(30.0.0.1) | \
> +      sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
>   tcp,orig=(src=192.168.1.2,dst=30.0.0.1,sport=<cleared>,dport=<cleared>),reply=(src=172.16.1.2,dst=192.168.1.2,sport=<cleared>,dport=<cleared>),zone=<cleared>,labels=0x2,protoinfo=(state=<cleared>)
>   tcp,orig=(src=192.168.1.2,dst=30.0.0.1,sport=<cleared>,dport=<cleared>),reply=(src=172.16.1.3,dst=192.168.1.2,sport=<cleared>,dport=<cleared>),zone=<cleared>,labels=0x2,protoinfo=(state=<cleared>)
>   tcp,orig=(src=192.168.1.2,dst=30.0.0.1,sport=<cleared>,dport=<cleared>),reply=(src=172.16.1.4,dst=192.168.1.2,sport=<cleared>,dport=<cleared>),zone=<cleared>,labels=0x2,protoinfo=(state=<cleared>)
>   ])
>   
>   dnl Should work with the virtual IP 30.0.0.3 address through NAT
> -for i in `seq 1 20`; do
> -    echo Request $i
> -    NS_CHECK_EXEC([foo1], [wget 30.0.0.3 -t 5 -T 1 --retry-connrefused -v -o wget$i.log])
> -done
> -
>   dnl Each server should have at least one connection.
> -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(30.0.0.3) | \
> -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> +OVS_WAIT_FOR_OUTPUT([
> +    for i in `seq 1 20`; do
> +        ip netns exec foo1 wget 30.0.0.3 -t 5 -T 1 --retry-connrefused -v -o wget$i.log;
> +    done
> +    ovs-appctl dpctl/dump-conntrack | FORMAT_CT(30.0.0.3) | \
> +      sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
>   tcp,orig=(src=192.168.1.2,dst=30.0.0.3,sport=<cleared>,dport=<cleared>),reply=(src=172.16.1.2,dst=192.168.1.2,sport=<cleared>,dport=<cleared>),zone=<cleared>,labels=0x2,protoinfo=(state=<cleared>)
>   tcp,orig=(src=192.168.1.2,dst=30.0.0.3,sport=<cleared>,dport=<cleared>),reply=(src=172.16.1.3,dst=192.168.1.2,sport=<cleared>,dport=<cleared>),zone=<cleared>,labels=0x2,protoinfo=(state=<cleared>)
>   tcp,orig=(src=192.168.1.2,dst=30.0.0.3,sport=<cleared>,dport=<cleared>),reply=(src=172.16.1.4,dst=192.168.1.2,sport=<cleared>,dport=<cleared>),zone=<cleared>,labels=0x2,protoinfo=(state=<cleared>)
>   ])
>   
>   dnl Test load-balancing that includes L4 ports in NAT.
> -for i in `seq 1 20`; do
> -    echo Request $i
> -    NS_CHECK_EXEC([foo1], [wget 30.0.0.2:8000 -t 5 -T 1 --retry-connrefused -v -o wget$i.log])
> -done
> -
>   dnl Each server should have at least one connection.
> -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(30.0.0.2) | \
> -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> +OVS_WAIT_FOR_OUTPUT([
> +    for i in `seq 1 20`; do
> +        ip netns exec foo1 wget 30.0.0.2:8000 -t 5 -T 1 --retry-connrefused -v -o wget$i.log;
> +    done
> +    ovs-appctl dpctl/dump-conntrack | FORMAT_CT(30.0.0.2) | \
> +      sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
>   tcp,orig=(src=192.168.1.2,dst=30.0.0.2,sport=<cleared>,dport=<cleared>),reply=(src=172.16.1.2,dst=192.168.1.2,sport=<cleared>,dport=<cleared>),zone=<cleared>,labels=0x2,protoinfo=(state=<cleared>)
>   tcp,orig=(src=192.168.1.2,dst=30.0.0.2,sport=<cleared>,dport=<cleared>),reply=(src=172.16.1.3,dst=192.168.1.2,sport=<cleared>,dport=<cleared>),zone=<cleared>,labels=0x2,protoinfo=(state=<cleared>)
>   tcp,orig=(src=192.168.1.2,dst=30.0.0.2,sport=<cleared>,dport=<cleared>),reply=(src=172.16.1.4,dst=192.168.1.2,sport=<cleared>,dport=<cleared>),zone=<cleared>,labels=0x2,protoinfo=(state=<cleared>)
> @@ -1500,14 +1500,13 @@ OVS_WAIT_UNTIL([
>   AT_CHECK([ovs-appctl dpctl/flush-conntrack])
>   
>   dnl Test load-balancing that includes L4 ports in NAT.
> -for i in `seq 1 20`; do
> -    echo Request $i
> -    NS_CHECK_EXEC([foo1], [wget 30.0.0.2:8000 -t 5 -T 1 --retry-connrefused -v -o wget$i.log])
> -done
> -
>   dnl Each server should have at least one connection.
> -AT_CHECK([ovs-appctl dpctl/dump-conntrack | FORMAT_CT(30.0.0.2) | \
> -sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
> +OVS_WAIT_FOR_OUTPUT([
> +    for i in `seq 1 20`; do
> +        ip netns exec foo1 wget 30.0.0.2:8000 -t 5 -T 1 --retry-connrefused -v -o wget$i.log;
> +    done
> +    ovs-appctl dpctl/dump-conntrack | FORMAT_CT(30.0.0.2) | \
> +      sed -e 's/zone=[[0-9]]*/zone=<cleared>/'], [0], [dnl
>   tcp,orig=(src=192.168.1.2,dst=30.0.0.2,sport=<cleared>,dport=<cleared>),reply=(src=172.16.1.2,dst=192.168.1.2,sport=<cleared>,dport=<cleared>),zone=<cleared>,labels=0x2,protoinfo=(state=<cleared>)
>   tcp,orig=(src=192.168.1.2,dst=30.0.0.2,sport=<cleared>,dport=<cleared>),reply=(src=172.16.1.3,dst=192.168.1.2,sport=<cleared>,dport=<cleared>),zone=<cleared>,labels=0x2,protoinfo=(state=<cleared>)
>   tcp,orig=(src=192.168.1.2,dst=30.0.0.2,sport=<cleared>,dport=<cleared>),reply=(src=172.16.1.4,dst=192.168.1.2,sport=<cleared>,dport=<cleared>),zone=<cleared>,labels=0x2,protoinfo=(state=<cleared>)
>