[ovs-dev] [PATCH v8 0/3] Support dynamic rebalancing of offloaded flows
simon.horman at netronome.com
Fri Oct 19 09:28:56 UTC 2018
On Thu, Oct 18, 2018 at 09:43:11PM +0530, Sriharsha Basavapatna via dev wrote:
> With the current OVS offload design, when an offload-device fails to add a
> flow rule and returns an error, OVS adds the rule to the kernel datapath.
> The flow gets processed by the kernel datapath for the entire life of that
> flow. This is fine when an error is returned by the device due to lack of
> support for certain keys or actions.
> But when an error is returned due to temporary conditions such as lack of
> resources to add a flow rule, the flow continues to be processed by kernel
> even when resources become available later. That is, those flows never get
> offloaded again. This problem becomes more pronounced when a flow that has
> been initially offloaded may have a smaller packet rate than a later flow
> that could not be offloaded due to lack of resources. This leads to
> inefficient use of HW resources and wastage of host CPU cycles.
> This patch-set addresses this issue by providing a way to detect temporary
> offload resource constraints (Out-Of-Resource or OOR condition) and to
> selectively and dynamically offload flows with a higher packets-per-second
> (pps) rate. This dynamic rebalancing is done periodically on netdevs that
> are in OOR state until resources become available to offload all pending
> The patch-set involves the following changes at a high level:
> 1. Detection of Out-Of-Resources (OOR) condition on an offload-capable
> 2. Gathering flow offload selection criteria for all flows on an OOR netdev;
> i.e, packets-per-second (pps) rate of flows for offloaded and
> non-offloaded (pending) flows.
> 3. Dynamically replacing offloaded flows with a lower pps-rate, with
> non-offloaded flows with a higher pps-rate, on an OOR netdev. A new
> OpenvSwitch configuration option - "offload-rebalance" to enable
> this policy.
> Cost/benefits data points:
> 1. Rough cost of the new rebalancing, in terms of CPU time:
> Ran a test that replaced 256 low pps-rate flows(pings) with 256 high
> pps-rate flows(iperf), in a system with 4 cpus (Intel Xeon E5 @ 2.40GHz;
> 2 cores with hw threads enabled, rest disabled). The data showed that cpu
> utilization increased by about ~20%. This increase occurs during the
> specific second in which rebalancing is done. And subsequently (from the
> next second), cpu utilization decreases significantly due to offloading
> of higher pps-rate flows. So effectively there's a bump in cpu utilization
> at the time of rebalancing, that is more than compensated by reduced cpu
> utilization once the right flows get offloaded.
> 2. Rough benefits to the user in terms of offload performance:
> The benefits to the user is reduced cpu utilization in the host, since
> higher pps-rate flows get offloaded, replacing lower pps-rate flows.
> Replacing a single offloaded flood ping flow with an iperf flow (multiple
> connections), shows that the cpu %used that was originally 100% on a
> single cpu (rebalancing disabled) goes down to 35% (rebalancing enabled).
> That is, cpu utilization decreased 65% after rebalancing.
> 3. Circumstances under which the benefits would show up:
> The rebalancing benefits would show up once offload resources are
> exhausted and new flows with higher pps-rate are initiated, that would
> otherwise be handled by kernel datapath costing host cpu cycles.
> This can be observed using 'ovs appctl dpctl/ dump-flows' command. Prior
> to rebalancing, any high pps-rate flows that couldn't be offloaded due to
> resource crunch would show up in the output of 'dump-flows type=ovs' and
> after rebalancing such flows would appear in the output of
> 'dump-flows type=offloaded'.
Thanks, applied to master.
More information about the dev