[ovs-dev] knowing when a kernel flow is really deleted

[Jesse Gross]

On Thu, Dec 15, 2011 at 10:48 AM, Ben Pfaff <blp at nicira.com> wrote:
> On Wed, Dec 14, 2011 at 06:34:04PM -0800, Jesse Gross wrote:
>> On Wed, Dec 14, 2011 at 4:12 PM, Ben Pfaff <blp at nicira.com> wrote:
>> > Why do we care? ??Current userspace isn't really affected. ??At most, we
>> > get flow statistics wrong by one packet (I guess multiple packets is
>> > theoretically possible?), if and when this happens. ??The place where it
>> > keeps coming up is in situations where we want userspace to be able to
>> > associate metadata with a packet sent from kernel to user. ??This came up
>> > in August regarding sflow:
>> > ?? ?? ?? ??http://openvswitch.org/pipermail/dev/2011-August/010384.html
>>
>> In theory the number of packets that you lose is unbounded if traffic
>> suddenly spikes, causing the CPU to hold onto the flow forever.  I
>> don't think it really matters and it is possible to do better than we
>> do today, though.
>
> I agree that memory freeing is deferred to the next grace period,
> which can effectively be "forever".  But I think that the number of
> packets whose statistics can be lost is limited by the time that it
> takes for a memory write to the flow table to become visible to other
> CPUs.  I think that would normally be a much shorter duration; my
> guess is that it would be hard for more than one packet to get "lost"
> that way.

That's a good point.

>> > ?? ?? ?? ??3. Somehow actually eliminate the problem with deleting flows,
>> > ?? ?? ?? ?? ?? so that when userspace receives the response to the flow
>> > ?? ?? ?? ?? ?? deletion we know that no more packets can go through the
>> > ?? ?? ?? ?? ?? flow. ??I don't know how to do this efficiently.
>>
>> I'm not sure that this is really a question of efficiency, so much as
>> it is complexity.  Basically you have to make userspace able to
>> tolerate blocking while the flow is deleted and then use
>> synchronize_rcu when removing flows.  Presumably this would mean that
>> you need some kind of worker threads.
>
> synchronize_rcu() is the obvious solution but the efficiency I was
> worried about is being able to delete flows at a reasonable pace.
> Wouldn't using synchronize_rcu() throttle back the speed of flow
> deletion to an unacceptable rate?  I remember from a long time ago
> that synchronize_rcu() can be ridiculously slow.  Oh yeah, here's the
> log message (which might be so old that it's not in the current OVS
> repo, not sure).  It's referring to testing that we did on whatever
> was the current XenServer version at the time, with a 2.6.18-based
> kernel:

Yeah, that was why I was saying that you probably need a pool of
worker threads that can block for a long time and then do whatever is
associated with deleting the flow (I'm not saying that this is a good
solution, just how I think this approach would look like).