[ovs-dev] [PATCH v8 0/3] Support dynamic rebalancing of offloaded flows

Eelco Chaudron echaudro at redhat.com
Fri Oct 19 14:27:02 UTC 2018

Hi Simon,

You might have missed my general comments email before you committed the 
patchset to master.

Just now I also sent my full review, and it looks like there is one 
nasty memory trashing one in 3/3 which need fixing. It’s the 
x2nrealloc() always allocating 1 entry, but we write to other offsets.


On 19 Oct 2018, at 11:28, Simon Horman wrote:

> On Thu, Oct 18, 2018 at 09:43:11PM +0530, Sriharsha Basavapatna via 
> dev wrote:
>> With the current OVS offload design, when an offload-device fails to 
>> add a
>> flow rule and returns an error, OVS adds the rule to the kernel 
>> datapath.
>> The flow gets processed by the kernel datapath for the entire life of 
>> that
>> flow. This is fine when an error is returned by the device due to 
>> lack of
>> support for certain keys or actions.
>> But when an error is returned due to temporary conditions such as 
>> lack of
>> resources to add a flow rule, the flow continues to be processed by 
>> kernel
>> even when resources become available later. That is, those flows 
>> never get
>> offloaded again. This problem becomes more pronounced when a flow 
>> that has
>> been initially offloaded may have a smaller packet rate than a later 
>> flow
>> that could not be offloaded due to lack of resources. This leads to
>> inefficient use of HW resources and wastage of host CPU cycles.
>> This patch-set addresses this issue by providing a way to detect 
>> temporary
>> offload resource constraints (Out-Of-Resource or OOR condition) and 
>> to
>> selectively and dynamically offload flows with a higher 
>> packets-per-second
>> (pps) rate. This dynamic rebalancing is done periodically on netdevs 
>> that
>> are in OOR state until resources become available to offload all 
>> pending
>> flows.
>> The patch-set involves the following changes at a high level:
>> 1. Detection of Out-Of-Resources (OOR) condition on an 
>> offload-capable
>>    netdev.
>> 2. Gathering flow offload selection criteria for all flows on an OOR 
>> netdev;
>>    i.e, packets-per-second (pps) rate of flows for offloaded and
>>    non-offloaded (pending) flows.
>> 3. Dynamically replacing offloaded flows with a lower pps-rate, with
>>    non-offloaded flows with a higher pps-rate, on an OOR netdev. A 
>> new
>>    OpenvSwitch configuration option - "offload-rebalance" to enable
>>    this policy.
>> Cost/benefits data points:
>> 1. Rough cost of the new rebalancing, in terms of CPU time:
>>    Ran a test that replaced 256 low pps-rate flows(pings) with 256 
>> high
>>    pps-rate flows(iperf), in a system with 4 cpus (Intel Xeon E5 @ 
>> 2.40GHz;
>>    2 cores with hw threads enabled, rest disabled). The data showed 
>> that cpu
>>    utilization increased by about ~20%. This increase occurs during 
>> the
>>    specific second in which rebalancing is done. And subsequently 
>> (from the
>>    next second), cpu utilization decreases significantly due to 
>> offloading
>>    of higher pps-rate flows. So effectively there's a bump in cpu 
>> utilization
>>    at the time of rebalancing, that is more than compensated by 
>> reduced cpu
>>    utilization once the right flows get offloaded.
>> 2. Rough benefits to the user in terms of offload performance:
>>    The benefits to the user is reduced cpu utilization in the host, 
>> since
>>    higher pps-rate flows get offloaded, replacing lower pps-rate 
>> flows.
>>    Replacing a single offloaded flood ping flow with an iperf flow 
>> (multiple
>>    connections), shows that the cpu %used that was originally 100% on 
>> a
>>    single cpu (rebalancing disabled) goes down to 35% (rebalancing 
>> enabled).
>>    That is, cpu utilization decreased 65% after rebalancing.
>> 3. Circumstances under which the benefits would show up:
>>    The rebalancing benefits would show up once offload resources are
>>    exhausted and new flows with higher pps-rate are initiated, that 
>> would
>>    otherwise be handled by kernel datapath costing host cpu cycles.
>>    This can be observed using 'ovs appctl dpctl/ dump-flows' command. 
>> Prior
>>    to rebalancing, any high pps-rate flows that couldn't be offloaded 
>> due to
>>    resource crunch would show up in the output of 'dump-flows 
>> type=ovs' and
>>    after rebalancing such flows would appear in the output of
>>    'dump-flows type=offloaded'.
> Thanks, applied to master.
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev

More information about the dev mailing list