[ovs-dev] [RFC] Question about ovn-controller performance

Han Zhou zhouhan at gmail.com
Thu Sep 14 19:26:36 UTC 2017


Hi folks,

I'd like to discuss my thoughts, too. To simplify the discussion, let's
just use the Neutron sec-group use case as the context. If there are P
ports in same group and there are R rules in the sec-group (by default
there are 2 such ingress (OVN to-lport) rules in the neutron default
sec-group), then the number of flows (F) related to those rules on each HV
will be (assuming all ports are under same lswitch or different lswitch but
connected by lrouter(s)):

F = P x P x R x 3 (for each ACL there are 3 flows required to implement it)

Given the huge number of flows, it is expected that the CPU will be running
hot. But there are still 2 parts of the problem:
1. Reduce number of flows
2. Reduce CPU cost

============================
1. Reduce number of flows

a. Consider local bound ports only (not implemented)
As discussed in today's OVN meeting, in theory, for the flows related to
ACLs, only the ones has inport/outport in match condition being the local
ports is required on the HV. If there are L ports bound locally, it should
be:

F = L x P x R x 3

This is a huge difference considering that L is usually much smaller than P.

b. Conjuncture
This is mentioned by Miguel with the link to an earlier discussion. If I
understand correctly, there are several cases can be optimized potentially.

i). When there are multiple ports on the same HV belongs to same group
(thus has same security rules/acls), conjuncture can help if the rules can
be combined like: inport/outport in {port1, port2, port3 ...} <other match
conditions> ... . Assume the L port in above example belongs to G groups (L
> G). F become:

F = G x P x R x 3 + L

ii). When there are multiple sec-group rules with same "remote-group-id"
(by default it is the self group) but with different TCP/UDP ports.
Conjuncture can help if we can combine the rules like: ... tcp.src in {p1,
p2, p3 ...}. If R rules can be combined to S rules:

F = G x P x S x 3 + L + R

There can be other cases to utilize conjuncture but I think in OpenStack
Neutron these are the typical ones. However, it may be tricky where to do
the combining. For i), if Neutron has port-binding information it may do
some work, but it is not quit straightforward to me. For ii), Neutron
sec-group API doesn't support multiple tcp/udp ports in a single rule,
except for port range. So it seems ovn-controller is the best place to do
the combining.

2. Reduce CPU cost
There are some points for tuning.

a. Make sure the version used include this commit:
https://github.com/openvswitch/ovs/commit/74c760c8fe99d554b94423d49d13d5ca3dea0d9e
It saves CPU cycles dramatically. It is in 2.8 but not 2.7.

b. Disable ovsdb probe. Any input would trigger a full recomputing of
flows. OVN-SB probe is configurable by
external_ids:ovn-remote-probe-interval

c. Packet-in from local OVS, such as ECHO, will also trigger recomputing.
Multi-threading ovn-controller will solve the problem but it is not merged
yet. So there is no way to tune, but if you are sure there is no need for
the features require packet-in, you can disable it by hardcode (not
recommended though), maybe just for performance testing. And if you use
2.8, make sure "log" is not enabled in ACL (using Neutron won't have the
problem, since this is not integrated to Neutron yet).

Thanks,
Han

On Thu, Sep 14, 2017 at 11:51 AM, Miguel Angel Ajo Pelayo <
majopela at redhat.com> wrote:

> I will prepare a summary with what I found and post it on this thread.
>
> On Thu, Sep 14, 2017 at 12:28 PM, Ben Pfaff <blp at ovn.org> wrote:
>
> > On Thu, Sep 14, 2017 at 10:39:28AM +0800, wang.qianyu at zte.com.cn wrote:
> > > I configure 5 networks, every network have about 80 ports, the total
> > ports
> > > is 400, all in same security group.
> > >
> > > When I bind some port on HVs, the ovn-controller is always running with
> > > 100% cpu, and the total openflow table entities in ovs is more than
> > > 300,000. Most of the entities is table 52, worked as src ip filter.
> >
> > It sounds like some of the OVN folks at Red Hat have recently found the
> > same issue during testing.  I'm expecting a more detailed report from
> > them soon.
> > _______________________________________________
> > dev mailing list
> > dev at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>


More information about the dev mailing list