<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 31, 2020 at 4:14 PM Tony Liu <<a href="mailto:tonyliu0592@hotmail.com">tonyliu0592@hotmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I see the active ovn-northd takes much CPU (30% - 100%) when there is no<br>
configuration from OpenStack, nothing happening on all chassis nodes either.<br>
<br>
Is this expected? What is it busy with?<br>
<br></blockquote><div><br></div><div>Yes, this is expected. It is due to the OVSDB probe between ovn-northd and NB/SB OVSDB servers, which is used to detect the OVSDB connection failure.</div><div>Usually this is not a concern (unlike the probe with a large number of ovn-controller clients), because ovn-northd is a centralized component and the CPU cost when there is no configuration change doesn't matter that much. However, if it is a concern, the probe interval (default 5 sec) can be changed.</div><div>If you change, remember to change on both server side and client side.</div><div>For client side (ovn-northd), it is configured in the NB DB's NB_Global table's options:northd_probe_interval. See man page of ovn-nb(5).</div><div>For server side (NB and SB), it is configured in the NB and SB DB's Connection table's inactivity_probe column.</div><div><br></div><div>Thanks,</div><div>Han<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
========<br>
2020-07-31T23:08:09.511Z|04267|poll_loop|DBG|wakeup due to [POLLIN] on fd 8 (<a href="http://10.6.20.84:44358" rel="noreferrer" target="_blank">10.6.20.84:44358</a><-><a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>) at lib/stream-fd.c:157 (68% CPU usage)<br>
2020-07-31T23:08:09.512Z|04268|jsonrpc|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: received request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:09.512Z|04269|jsonrpc|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: send reply, result=[], id="echo"<br>
2020-07-31T23:08:12.777Z|04270|poll_loop|DBG|wakeup due to [POLLIN] on fd 9 (<a href="http://10.6.20.84:49158" rel="noreferrer" target="_blank">10.6.20.84:49158</a><-><a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>) at lib/stream-fd.c:157 (34% CPU usage)<br>
2020-07-31T23:08:12.777Z|04271|reconnect|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: idle 5002 ms, sending inactivity probe<br>
2020-07-31T23:08:12.777Z|04272|reconnect|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: entering IDLE<br>
2020-07-31T23:08:12.777Z|04273|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: send request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:12.777Z|04274|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: received request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:12.777Z|04275|reconnect|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: entering ACTIVE<br>
2020-07-31T23:08:12.777Z|04276|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: send reply, result=[], id="echo"<br>
2020-07-31T23:08:13.635Z|04277|poll_loop|DBG|wakeup due to [POLLIN] on fd 9 (<a href="http://10.6.20.84:49158" rel="noreferrer" target="_blank">10.6.20.84:49158</a><-><a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>) at lib/stream-fd.c:157 (34% CPU usage)<br>
2020-07-31T23:08:13.635Z|04278|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: received reply, result=[], id="echo"<br>
2020-07-31T23:08:14.480Z|04279|hmap|DBG|Dropped 129 log messages in last 5 seconds (most recently, 0 seconds ago) due to excessive rate<br>
2020-07-31T23:08:14.480Z|04280|hmap|DBG|lib/shash.c:112: 2 buckets with 6+ nodes, including 2 buckets with 6 nodes (32 nodes total across 32 buckets)<br>
2020-07-31T23:08:14.513Z|04281|poll_loop|DBG|wakeup due to 27-ms timeout at lib/reconnect.c:643 (34% CPU usage)<br>
2020-07-31T23:08:14.513Z|04282|reconnect|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: idle 5001 ms, sending inactivity probe<br>
2020-07-31T23:08:14.513Z|04283|reconnect|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: entering IDLE<br>
2020-07-31T23:08:14.513Z|04284|jsonrpc|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: send request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:15.370Z|04285|poll_loop|DBG|wakeup due to [POLLIN] on fd 8 (<a href="http://10.6.20.84:44358" rel="noreferrer" target="_blank">10.6.20.84:44358</a><-><a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>) at lib/stream-fd.c:157 (34% CPU usage)<br>
2020-07-31T23:08:15.370Z|04286|jsonrpc|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: received request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:15.370Z|04287|reconnect|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: entering ACTIVE<br>
2020-07-31T23:08:15.370Z|04288|jsonrpc|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: send reply, result=[], id="echo"<br>
2020-07-31T23:08:16.236Z|04289|poll_loop|DBG|wakeup due to 0-ms timeout at tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a> (100% CPU usage)<br>
2020-07-31T23:08:16.236Z|04290|jsonrpc|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: received reply, result=[], id="echo"<br>
2020-07-31T23:08:17.778Z|04291|poll_loop|DBG|wakeup due to [POLLIN] on fd 9 (<a href="http://10.6.20.84:49158" rel="noreferrer" target="_blank">10.6.20.84:49158</a><-><a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>) at lib/stream-fd.c:157 (100% CPU usage)<br>
2020-07-31T23:08:17.778Z|04292|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: received request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:17.778Z|04293|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: send reply, result=[], id="echo"<br>
2020-07-31T23:08:20.372Z|04294|poll_loop|DBG|wakeup due to [POLLIN] on fd 8 (<a href="http://10.6.20.84:44358" rel="noreferrer" target="_blank">10.6.20.84:44358</a><-><a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>) at lib/stream-fd.c:157 (41% CPU usage)<br>
2020-07-31T23:08:20.372Z|04295|reconnect|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: idle 5002 ms, sending inactivity probe<br>
2020-07-31T23:08:20.372Z|04296|reconnect|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: entering IDLE<br>
2020-07-31T23:08:20.372Z|04297|jsonrpc|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: send request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:20.372Z|04298|jsonrpc|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: received request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:20.372Z|04299|reconnect|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: entering ACTIVE<br>
2020-07-31T23:08:20.372Z|04300|jsonrpc|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: send reply, result=[], id="echo"<br>
2020-07-31T23:08:20.376Z|04301|hmap|DBG|Dropped 181 log messages in last 6 seconds (most recently, 1 seconds ago) due to excessive rate<br>
2020-07-31T23:08:20.376Z|04302|hmap|DBG|northd/ovn-northd.c:595: 2 buckets with 6+ nodes, including 2 buckets with 6 nodes (256 nodes total across 256 buckets)<br>
2020-07-31T23:08:21.222Z|04303|poll_loop|DBG|wakeup due to [POLLIN] on fd 8 (<a href="http://10.6.20.84:44358" rel="noreferrer" target="_blank">10.6.20.84:44358</a><-><a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>) at lib/stream-fd.c:157 (41% CPU usage)<br>
2020-07-31T23:08:21.223Z|04304|jsonrpc|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: received reply, result=[], id="echo"<br>
2020-07-31T23:08:22.779Z|04305|poll_loop|DBG|wakeup due to 706-ms timeout at lib/reconnect.c:643 (41% CPU usage)<br>
2020-07-31T23:08:22.779Z|04306|reconnect|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: idle 5001 ms, sending inactivity probe<br>
2020-07-31T23:08:22.779Z|04307|reconnect|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: entering IDLE<br>
2020-07-31T23:08:22.779Z|04308|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: send request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:23.632Z|04309|poll_loop|DBG|wakeup due to [POLLIN] on fd 9 (<a href="http://10.6.20.84:49158" rel="noreferrer" target="_blank">10.6.20.84:49158</a><-><a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>) at lib/stream-fd.c:157 (79% CPU usage)<br>
2020-07-31T23:08:23.632Z|04310|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: received request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:23.632Z|04311|reconnect|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: entering ACTIVE<br>
2020-07-31T23:08:23.632Z|04312|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: send reply, result=[], id="echo"<br>
2020-07-31T23:08:24.481Z|04313|poll_loop|DBG|wakeup due to 0-ms timeout at tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a> (79% CPU usage)<br>
2020-07-31T23:08:24.481Z|04314|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: received reply, result=[], id="echo"<br>
2020-07-31T23:08:25.372Z|04315|poll_loop|DBG|wakeup due to [POLLIN] on fd 8 (<a href="http://10.6.20.84:44358" rel="noreferrer" target="_blank">10.6.20.84:44358</a><-><a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>) at lib/stream-fd.c:157 (79% CPU usage)<br>
2020-07-31T23:08:25.372Z|04316|jsonrpc|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: received request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:25.372Z|04317|jsonrpc|DBG|tcp:<a href="http://10.6.20.84:6641" rel="noreferrer" target="_blank">10.6.20.84:6641</a>: send reply, result=[], id="echo"<br>
2020-07-31T23:08:28.634Z|04318|poll_loop|DBG|wakeup due to [POLLIN] on fd 9 (<a href="http://10.6.20.84:49158" rel="noreferrer" target="_blank">10.6.20.84:49158</a><-><a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>) at lib/stream-fd.c:157 (50% CPU usage)<br>
2020-07-31T23:08:28.634Z|04319|reconnect|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: idle 5002 ms, sending inactivity probe<br>
2020-07-31T23:08:28.634Z|04320|reconnect|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: entering IDLE<br>
2020-07-31T23:08:28.634Z|04321|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: send request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:28.634Z|04322|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: received request, method="echo", params=[], id="echo"<br>
2020-07-31T23:08:28.634Z|04323|reconnect|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: entering ACTIVE<br>
2020-07-31T23:08:28.634Z|04324|jsonrpc|DBG|tcp:<a href="http://10.6.20.85:6642" rel="noreferrer" target="_blank">10.6.20.85:6642</a>: send reply, result=[], id="echo"<br>
2020-07-31T23:08:28.638Z|04325|hmap|DBG|Dropped 266 log messages in last 9 seconds (most recently, 3 seconds ago) due to excessive rate<br>
2020-07-31T23:08:28.638Z|04326|hmap|DBG|northd/ovn-northd.c:595: 2 buckets with 6+ nodes, including 2 buckets with 6 nodes (256 nodes total across 256 buckets)<br>
========<br>
<br>
Thanks!<br>
<br>
Tony<br>
<br>
_______________________________________________<br>
discuss mailing list<br>
<a href="mailto:discuss@openvswitch.org" target="_blank">discuss@openvswitch.org</a><br>
<a href="https://mail.openvswitch.org/mailman/listinfo/ovs-discuss" rel="noreferrer" target="_blank">https://mail.openvswitch.org/mailman/listinfo/ovs-discuss</a><br>
</blockquote></div></div>