[ovs-discuss] OVN: Delay in handling unixctl commands in ovsdb-server

[Ben Pfaff]

On Wed, Feb 12, 2020 at 11:27:18PM +0530, Numan Siddique wrote:
> Hi Ben/All,
> 
> In an OVN deployment - with OVN dbs deployed as active/standby using
> pacemaker, we are seeing delays in response to unixctl command -
> ovsdb-server/sync-status.
> 
> Pacemaker periodically calls the OVN pacemaker OCF script to get the
> status and this script internally invokes - ovs-appctl -t
> /var/run/openvswitch/ovnsb_db.ctl ovsdb-server/sync-status. In a large
> deployment with lots of OVN resources we see that ovsdb-server takes a
> lot of time (sometimes > 60 seconds) to respond to this command. This
> causes pacemaker to stop the service in that node and move the master
> to another node. This causes a lot of disruption.
> 
> One approach of solving this issue is to handle unixctl commands in a
> separate thread. The commands like sync-status, get-** etc can be
> easily handled in the thread. Still, there are many commands like
> ovsdb-server/set-active-ovsdb-server, ovsdb-server/compact etc (which
> changes the state) which needs to be synchronized between the main
> ovsdb-server thread and the newly added thread using a mutex.
> 
> Does this approach makes sense ? I started working on it. But I wanted
> to check with the community before putting into more efforts.

It seems reasonable to me to support unixctl commands in multiple
threads.  The details of how you implement it will determine how usable
it is.  I suggest making the current case easy and common.