[ovs-dev] [PATCH v8 0/3] Support dynamic rebalancing of offloaded flows

Sriharsha Basavapatna sriharsha.basavapatna at broadcom.com
Thu Oct 25 14:08:04 UTC 2018


Hi Eelco,

On Mon, Oct 22, 2018 at 12:30 PM Simon Horman
<simon.horman at netronome.com> wrote:
>
> Hi Eelco,
>
> Thanks for your review and sorry for missing it. I trust that Sriharsha will address it in follow-up patches.
>
>
> 2018/10/19 15:27 Eelco Chaudron <echaudro at redhat.com>:
>>
>> Hi Simon,
>>
>> You might have missed my general comments email before you committed the
>> patchset to master.
>>
>> Just now I also sent my full review, and it looks like there is one
>> nasty memory trashing one in 3/3 which need fixing. It’s the
>> x2nrealloc() always allocating 1 entry, but we write to other offsets.

I've responded to your comments on the 3 patches. I've also answered
to your above concern on x2nrealloc(); there's no issue. Please see my
email on 3/3 for details.

Thanks,
-Harsha
>>
>> //Eelco
>>
>>
>> On 19 Oct 2018, at 11:28, Simon Horman wrote:
>>
>> > On Thu, Oct 18, 2018 at 09:43:11PM +0530, Sriharsha Basavapatna via
>> > dev wrote:
>> >> With the current OVS offload design, when an offload-device fails to
>> >> add a
>> >> flow rule and returns an error, OVS adds the rule to the kernel
>> >> datapath.
>> >> The flow gets processed by the kernel datapath for the entire life of
>> >> that
>> >> flow. This is fine when an error is returned by the device due to
>> >> lack of
>> >> support for certain keys or actions.
>> >>
>> >> But when an error is returned due to temporary conditions such as
>> >> lack of
>> >> resources to add a flow rule, the flow continues to be processed by
>> >> kernel
>> >> even when resources become available later. That is, those flows
>> >> never get
>> >> offloaded again. This problem becomes more pronounced when a flow
>> >> that has
>> >> been initially offloaded may have a smaller packet rate than a later
>> >> flow
>> >> that could not be offloaded due to lack of resources. This leads to
>> >> inefficient use of HW resources and wastage of host CPU cycles.
>> >>
>> >> This patch-set addresses this issue by providing a way to detect
>> >> temporary
>> >> offload resource constraints (Out-Of-Resource or OOR condition) and
>> >> to
>> >> selectively and dynamically offload flows with a higher
>> >> packets-per-second
>> >> (pps) rate. This dynamic rebalancing is done periodically on netdevs
>> >> that
>> >> are in OOR state until resources become available to offload all
>> >> pending
>> >> flows.
>> >>
>> >> The patch-set involves the following changes at a high level:
>> >>
>> >> 1. Detection of Out-Of-Resources (OOR) condition on an
>> >> offload-capable
>> >>    netdev.
>> >> 2. Gathering flow offload selection criteria for all flows on an OOR
>> >> netdev;
>> >>    i.e, packets-per-second (pps) rate of flows for offloaded and
>> >>    non-offloaded (pending) flows.
>> >> 3. Dynamically replacing offloaded flows with a lower pps-rate, with
>> >>    non-offloaded flows with a higher pps-rate, on an OOR netdev. A
>> >> new
>> >>    OpenvSwitch configuration option - "offload-rebalance" to enable
>> >>    this policy.
>> >>
>> >> Cost/benefits data points:
>> >>
>> >> 1. Rough cost of the new rebalancing, in terms of CPU time:
>> >>
>> >>    Ran a test that replaced 256 low pps-rate flows(pings) with 256
>> >> high
>> >>    pps-rate flows(iperf), in a system with 4 cpus (Intel Xeon E5 @
>> >> 2.40GHz;
>> >>    2 cores with hw threads enabled, rest disabled). The data showed
>> >> that cpu
>> >>    utilization increased by about ~20%. This increase occurs during
>> >> the
>> >>    specific second in which rebalancing is done. And subsequently
>> >> (from the
>> >>    next second), cpu utilization decreases significantly due to
>> >> offloading
>> >>    of higher pps-rate flows. So effectively there's a bump in cpu
>> >> utilization
>> >>    at the time of rebalancing, that is more than compensated by
>> >> reduced cpu
>> >>    utilization once the right flows get offloaded.
>> >>
>> >> 2. Rough benefits to the user in terms of offload performance:
>> >>
>> >>    The benefits to the user is reduced cpu utilization in the host,
>> >> since
>> >>    higher pps-rate flows get offloaded, replacing lower pps-rate
>> >> flows.
>> >>    Replacing a single offloaded flood ping flow with an iperf flow
>> >> (multiple
>> >>    connections), shows that the cpu %used that was originally 100% on
>> >> a
>> >>    single cpu (rebalancing disabled) goes down to 35% (rebalancing
>> >> enabled).
>> >>    That is, cpu utilization decreased 65% after rebalancing.
>> >>
>> >> 3. Circumstances under which the benefits would show up:
>> >>
>> >>    The rebalancing benefits would show up once offload resources are
>> >>    exhausted and new flows with higher pps-rate are initiated, that
>> >> would
>> >>    otherwise be handled by kernel datapath costing host cpu cycles.
>> >>
>> >>    This can be observed using 'ovs appctl dpctl/ dump-flows' command.
>> >> Prior
>> >>    to rebalancing, any high pps-rate flows that couldn't be offloaded
>> >> due to
>> >>    resource crunch would show up in the output of 'dump-flows
>> >> type=ovs' and
>> >>    after rebalancing such flows would appear in the output of
>> >>    'dump-flows type=offloaded'.
>> >
>> > Thanks, applied to master.
>> > _______________________________________________
>> > dev mailing list
>> > dev at openvswitch.org
>> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev


More information about the dev mailing list