[ovs-dev] Recirculation with milti-threaded miss handling

[Simon Horman]

Hi Ethan,

I am having a bit of difficulty re-implementing portions of my
recirculation patchset in order to work in conjunction with your recent
change, e1ec7dd ("ofproto-dpif: Implement multi-threaded miss handling.")

In short, it would make my life much easier if facets could be looked up
execute_flow_miss(). And I am interested to know if you have any plans in
that regards.

In more detail, the main problems I am having are:

1. When a miss for a recirculated packet is received the flow of
   the miss cannot be used directly to lookup a rule. That is because when
   a packet is recirculated flow differs from that of the original packet.

   Up until now recirculation has dealt with this by allowing
   facets to be looked up by their recirculation id as necessary.
   This allows the top-most facet of a chain of recirculation to
   be found and its flow to be used to lookup the rule.

   However, my understanding is that with your recent changes facets aren't
   available in the thread where misses are executed which seems to be when
   action translation occurs and the rule is required.


   I had planned to get around this by creating a new recirculator
   structure which would hold a recirculation id and flow. The flow
   holds the parent recirculation id and it should be possible to
   use this to find the top-most recirculator. This would provide
   the thus the top-most flow which could be use for rule lookup.

   At the heart of this plan the recirculator structure could be looked up
   based on its recirculation id; created if lookup failed and; is
   referenced counted.

   Later, when facets are created, the recirculator can be associated with
   the facet, allowing the facet side of recirculation, in particular
   revalidation and consistency checking, to work by looking up
   recirculator by their id.

   Thus thread-safety should be taken care of without impacting the
   performance of the common case where recirculation is not used.

   However I see two problems with this scheme:

   i.  If multiple misses are in-flight but handled in different
       flow_miss_batches then multiple recirculators would be created for
       what is in fact the same flow. This doesn't map comfortably to the
       idea associate a recirculator with a facet. Though in a pinch it
       ought to be possible to associate multiple recirculators with a
       facet.

       Another solution would be to allow recirculators to be looked up be
       classifier. But it seems this would require re-working or augmenting
       the thread-safety of classifiers as in the scheme I propose
       recirculators are created when translation occurs and this is not in
       the main thread where write-access to classifiers is thread-safe.

   ii.  A more acute problem seems to that it seems that revalidation
        should occur when rule lookup for a recirculated packet occurs.
        This is because the scheme described above involves using offsets,
        calculated during previous action translations, to determine which
        part of a rules ofpacts should be used when translating actions
        for a recirculated packet.

	I suppose it would be possible to add sufficient information
	to recirculators to make this possible. But at this point
	they are starting to look a lot like facets.

2. A facet may or may not be created for a miss

   This is determined after execution occurs,
   and I believe it takes place in a different thread.

   This is problematic for recirculation because in its current form if a
   facet is to be used then the packet is only executed up until
   recirculation.  But if a facet is not to be used then the packet is
   recirculated in ovs-vswtichd. Thus recirculation uses knowledge of
   weather a facet will be created or not at execution time. But that
   information is no longer available with your recent changes.

   A simple work-around is force facet creation if recirculation is used.
   It would be nice not to use this work-around.