[ovs-discuss] Warning logs flooding: failed to flow_del (No such file or directory)

[Andrey Korolyov]

Thanks Alex,

On Sat, Oct 4, 2014 at 5:39 AM, Alex Wang <alexw at nicira.com> wrote:
> Hey Andrey,
>
> Thanks for reporting the issue,  Please see my reply below,
>
>> Hello,
>>
>> getting the lookalike message on 2.1.3 where the problem described in
>> the conversation should not take a place:
>
>
> Yes, v2.1.3 contains the fix commit 3601bd879 (datapath: Use exact lookup
> for flow_get and flow_del.)

Sorry, it turns out that the datapath I am using is a pre-release
2.1.3 from early May, will update and report soon.

>
>
>
>
>>
>>
>> 2014-10-03T17:24:30.429Z|00026|dpif(revalidator_11)|WARN|system at ovs-system:
>> failed to flow_del (No such file or directory)
>>
>> skb_priority(0),in_port(2),skb_mark(0),eth(src=xx:xx:xx:xx:xx:xx,dst=yy:yy:yy:yy:yy:yy),eth_type(0x0800),ipv4(src=zz.zz.zz.zz,dst=ii.jj.kk.ll,proto=6,tos=0,ttl=118,frag=no),tcp(src=nnn,dst=mmm),tcp_flags(0x018)
>
>
>
> Could you help confirm the following things?
>
> 1. if the code you use contains commit 3601bd879
>
> 2. for the flows in 'failed to flow_del' logs, were they the same flow or
> same set of flows?  or totally random flows?

Looks like they are random, though occurences are very rare (about 1
for 10k OpenFlow flow_adds).
>
> One thing comes to my mind is that, for v2.1.3, the same dp flow could
> actually be dumped more than once.  since the flow dumping happens in batch,
> so for example assume the current batch dumped flow A, then the flow adds
> before the next dump move back the position of flow A in the link list, then
> the next dump will dump flow A again.
>
> So, the revalidator will process flow A twice.  And if the first process
> decides to delete flow A.  The next process will do the same thing but
> result in this 'failed to flow_del' error~
>
>
> 3. don't know if it is possible for your to provide the result of following
> commands:
>
> ovs-dpctl dump-flows | sed -n 's/^.*\(used:[0-9.]\+\).*$/\1/p'
>
>
>
>
>>
>>
>> Preconditions are following:
>>   - 30k installed non-overlapping ip mask flows, ttl=4h,
>>   - 3k dp flows,
>>   - moderate traffic, on about 200mbit and hitting 2000prefixes/s.
>>
>> The error appears long before flow` expiration time, so I think that
>> there is something wrong. Also even with very low flow_add ratio (~
>> 5/s) I am getting moderately high cpu consumption by vswitchd with
>> such a setup, on about 50% of single Xeon core, comparing to 20% for
>> proactive-only match for same traffic with small number of installed
>> rules (~100).
>
>
>
> For 'flow_add', do you mean OpenFlow flows?  If so, i could more time to
> add/del flows in classifier,
>
> Do you have more interfaces as compared to the other setup you mentioned?
> With more interfaces, it would take more time to update stats/status.
>
> Thanks,
> Alex Wang,