<div dir="ltr"><div class="gmail_extra">Thanks a lot Han and Ben for looking into this!</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 14, 2018 at 9:34 PM, Han Zhou <span dir="ltr"><<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><span class="gmail-"><div><br><br>On Wed, Feb 14, 2018 at 9:45 AM, Ben Pfaff <<a href="mailto:blp@ovn.org" target="_blank">blp@ovn.org</a>> wrote:<br>><br>> On Wed, Feb 14, 2018 at 11:27:11AM +0100, Daniel Alvarez Sanchez wrote:<br>> > Thanks for your inputs. I need to look more carefully into the patch you<br>> > submitted but it looks like, at least, we'll be reducing the number of<br>> > calls to Datum.__cmp__ which should be good.<br>><br>> Thanks. Please do take a look. It's a micro-optimization but maybe<br>> it'll help?<br>><br>> > I probably didn't explain it very well. Right now we have N processes<br>> > for Neutron server (in every node). Each of those opens a connection<br>> > to NB db and they subscribe to updates from certain tables. Each time<br>> > a change happens, ovsdb-server will send N update2 messages that has<br>> > to be processed in this "expensive" way by each of those N<br>> > processes. My proposal (yet to be refined) would be to now open N+1<br>> > connections to ovsdb-server and only subscribe to notifications from 1<br>> > of those. So every time a new change happens, ovsdb-server will send 1<br>> > update2 message. This message will be processed (using Py IDL as we do<br>> > now) and once processed, send it (mcast maybe?) to the rest N<br>> > processes. This msg could be simply a Python object serialized and<br>> > we'd be saving all this Datum, Atom, etc. processing by doing it just<br>> > once.<br>><br></div></span>Daniel, I understand that the update2 messages sending would consume NB ovsdb-server CPU and processing those update would consume neutron server process CPU. However, are we sure it is the bottleneck for port creation?<br></div></div></blockquote><div> </div><div>No, I don't think this is the bottleneck at this stage with only 800 ports being created</div><div>at that pace. As you say, I'm also not keen on doing such a big change right now but it's</div><div>something to bear in mind for the future as ~50% of total processing time being</div><div>spent in <a href="http://data.py/uuid.py">data.py/uuid.py</a> is not neglectable. At least, it'd be nice to try Ben's suggestions</div><div>and I'll do it as soon as I can.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br>From ovsdb-server point of view, sending updates to tens of clients should not be the bottleneck, considering that we have a lot more clients on HVs for SB ovsdb-server.<br></div></div></blockquote><div>Same would go for SB ovsdb-server though but yeah... I agree. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div>From clients point of view, I think it is more of memory overhead than CPU, and it also depends on how many neutron processes are running on the same node. I didn't find neutron process CPU in your charts. I am hesitate for such big change before we are clear about the bottleneck. The chart of port creation time is very nice, but do we know which part of code contributed to the linear growth? Do we have profiling for the time spent in ovn_client.add_acls()?<br></div></blockquote><div>This is my next step. It seems pretty clear that ACLs are the bottleneck.</div><div><br></div><div><br></div><div>As a note, unsurprisingly, same goes for port deletion:</div><div><div>$ date; for i in {1..800}; do openstack port delete port$i; done; date </div><div>Wed Feb 14 17:45:07 UTC 2018</div><div>Wed Feb 14 19:36:09 UTC 2018</div></div><div><br></div><div>Same behavior shown during deletion as can be seen at [0].</div><div><br></div><div>[0] <a href="https://imgur.com/a/Dam10">https://imgur.com/a/Dam10</a></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div class="gmail-h5"><br>> OK. It's an optimization that does the work in one place rather than N<br>> places, so definitely a win from a CPU cost point of view, but it trades<br>> performance for increased complexity. It sounds like performance is<br>> really important so maybe the increased complexity is a fair trade.<br>><br>> We might also be able to improve performance by using native code for<br>> some of the work. Were these tests done with the native code JSON<br>> parser that comes with OVS? It is dramatically faster than the Python<br>> code.<br></div></div></div></div></div></div></blockquote><div><br></div><div>Yes, these tests were done using the native JSON parser. I saw that Python</div><div>parsing had a 5% over the total execution time so recompiling with the</div><div>--enabled-shared made it use the native parser. However it didn't make a big</div><div>difference in the port creation times. In my first tests I also saw some reconnections</div><div>from clients to ovsdb-server especially when the number of ports increased. (not</div><div>many though, just a couple of them) during the 800 ports.</div><div>I then changed the inactivity probe to 180s for both NB and SB connections but</div><div>still the results were pretty much the same even though I didn't see any</div><div>reconnections this time.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div class="gmail-h5">><br>> > On Tue, Feb 13, 2018 at 8:32 PM, Ben Pfaff <<a href="mailto:blp@ovn.org" target="_blank">blp@ovn.org</a>> wrote:<br>> ><br>> > > Can you sketch the rows that are being inserted or modified when a port<br>> > > is added? I would expect something like this as a minimum:<br>> > ><br>> > > * Insert one Logical_Switch_Port row.<br>> > ><br>> > > * Add pointer to Logical_Switch_Port to ports column in one row<br>> > > in Logical_Switch.<br>> > ><br>> > > In addition it sounds like currently we're seeing:<br>> > ><br>> > > * Add one ACL row per security group rule.<br>> > ><br>> > > * Add pointers to ACL rows to acls column in one row in<br>> > > Logical_Switch.<br>> > ><br>> > This is what happens when we create a port in OpenStack (without<br>> > binding it) which belongs to a SG which allows ICMP and SSH traffic<br>> > and drops the rest [0]<br>> ><br>> > Basically, you were right and only thing missing was adding the new<br>> > address to the Address_Set table.<br>><br>> OK.<br>><br>> It sounds like the real scaling problem here is that for R security<br>> group rules and P ports, we have R*P rows in the ACL table. Is that<br>> correct? Should we aim to solve that problem?<br><br></div></div>I think this might be the most valuable point to optimize for the create_port scenario from Neutron.<br></div><div>I remember there was a patch for ACL group in OVN, so that instead of R*P rows we will have only R + P rows, but didn't see it went through.<br></div>Is this also a good use case of conjuncture?<br></div><div><div><br></div><div>> ______________________________<wbr>_________________<br>> discuss mailing list<br>> <a href="mailto:discuss@openvswitch.org" target="_blank">discuss@openvswitch.org</a><br>> <a href="https://mail.openvswitch.org/mailman/listinfo/ovs-discuss" target="_blank">https://mail.openvswitch.org/<wbr>mailman/listinfo/ovs-discuss</a><br><br></div></div></div></div>
</blockquote></div><br></div></div>