[ovs-dev] [ovs-discuss] [OVN] no response to inactivity probe

Tony Liu tonyliu0592 at hotmail.com
Thu Aug 6 18:11:13 UTC 2020


Inline... (please read with monospaced font:))

Thanks!

Tony
> -----Original Message-----
> From: Han Zhou <hzhou at ovn.org>
> Sent: Wednesday, August 5, 2020 11:48 PM
> To: Tony Liu <tonyliu0592 at hotmail.com>
> Cc: Han Zhou <hzhou at ovn.org>; Numan Siddique <numans at ovn.org>; ovs-dev
> <ovs-dev at openvswitch.org>; ovs-discuss <ovs-discuss at openvswitch.org>
> Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> 
> 
> 
> On Wed, Aug 5, 2020 at 9:14 PM Tony Liu <tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > wrote:
> 
> 
> 	I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb-db
> 	and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first node
> 	of cluster. Also ovn-openflow-probe-interval=30 on compute node.
> 	It seems helping. Not that many connect/drop/reconnect in logging.
> 	That "commit failure" is also gone.
> 	The issue I reported in another thread "packet drop" seems gone.
> 	And launching VM starts working.
> 
> 	How should I set connection table for all ovn-nb-db and ovn-sb-db
> 	nodes in the cluster to set inactivity_probe?
> 	One row with address 0.0.0.0 seems not working.
> 
> You can simply use 0.0.0.0 in the connection table, but don't specify
> the same connection method on the command line when starting ovsdb-
> server for NB/SB DB. Otherwise, these are conflicting and that's why you
> saw "Address already in use" error.

Could you share a bit details how it works?
I thought the row in connection table only tells nbdb and sbdb the
probe interval. Isn't that right? Does nbdb and sbdb also create
socket based on target column?

> 
> 	Is "external_ids:ovn-remote-probe-interval" in ovsdb-server on
> 	compute node for ovn-controller to probe ovn-sb-db?
> 
> OVSDB probe is bidirectional, so you need to set this value, too, if you
> don't want too many probes handled by the SB server. (setting the
> connection table for SB only changes the server side).

In that case, how do I set probe interval for ovn-controller?
My understanding is that, ovn-controller reads configuration from
ovsdb-server on the local compute node. Isn't that right?

> 	Is "external_ids:ovn-openflow-probe-interval" in ovsdb-server on
> 	compute node for ovn-controller to probe ovsdb-server?
> 
> It is for the OpenFlow connection between ovn-controller and ovs-
> vswitchd, which is part of the OpenFlow protocol.
> 
> 	What's probe interval for ovsdb-server to probe ovn-controller?
> 
> The local ovsdb connection uses unix socket, which doesn't send probe by
> default (if I remember correctly).

Here is how ovsdb-server and ovn-controller is invoked on compute node.
========
root     41129  0.0  0.0 157556 20532 ?        S    Jul30   1:51 /usr/sbin/ovsdb-server /var/lib/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/run/openvswitch/db.sock --remote=ptcp:6640:127.0.0.1 --remote=db:Open_vSwitch,Open_vSwitch,manager_options --log-file=/var/log/kolla/openvswitch/ovsdb-server.log --pidfile

root     63775 55.9  0.4 1477796 1224324 ?     Sl   Aug04 1360:55 /usr/bin/ovn-controller --pidfile=/run/ovn/ovn-controller.pid --log-file=/var/log/kolla/openvswitch/ovn-controller.log tcp:127.0.0.1:6640
========
Is that OK? Or UNIX socket method is recommended for ovn-controller
to connect to ovsdb-server?

Here is the configuration in open_vswitch table in ovsdb-server.
========
external_ids        : {ovn-encap-ip="10.6.30.22", ovn-encap-type=geneve, ovn-openflow-probe-interval="30", ovn-remote="tcp:10.6.20.84:6642,tcp:10.6.20.85:6642,tcp:10.6.20.86:6642", ovn-remote-probe-interval="60000", system-id="compute-3"}
========
ovn-controller connects to ovsdb-server and reads this configuration,
so it knows how to connect to all sbdb nodes, right?

If it's TCP between ovn-controller and ovsdb-server, is that probe
interval setting will also apply to the probe from ovn-controller to
ovsdb-server?

ovn-controller connects to ovs-vswitchd by UNIX socket to program
open-flow. ovs-vswitchd and ovsdb-server are connected by UNIX too.
So, is that ovn-openflow-probe-interval for the probe from ovn-controller
to ovs-vswitchd via UNIX?

As a summary for the probe setting,

+--------------+  driver configuration
|  ovn-driver  |
+--------------+
    ^    |
    |    v
+--------------+  inactivity_probe in table "Connection"
|  ovn-nb-db   |
+--------------+
    ^    |
    |    v
+--------------+  options:northd_probe_interval in table "NB_Global"
|  ovn-northd  |  in nbdb.
+--------------+
    ^    |
    |    v
+--------------+  inactivity_probe in table "Connection"
|  ovn-sb-db   |
+--------------+
    ^    |
    |    v
+--------------------------------+  in table "Open_vSwitch" in ovsdb-server
|        ovn-controller          |  ovn-remote-probe-interval for TCP
+--------------------------------+  probe to ovsdb-server,
    ^    |            ^    |        ovn-openflow-probe-interval for UNIX
    |    v TCP        |    v UNIX   probe to ovs-vswitchd
+--------------+  +--------------+
| ovsdb-server |  | ovs-vswitchd |
+--------------+  +--------------+

Is that correct?

> For ovn-controller, since it is implemented with incremental-processing,
> even if there are probes from openflow or local ovsdb, it doesn't matter.
> If there is no configuration change, ovn-controller simply replies the
> probe and there is no extra cost.

I did see errors about connecting to br-mgmt (UNIX connection to
ovs-vswitchd) in ovn-controller logging. After setting proble internal,
those errors are gone. It seems that it does matter.

> 
> 
> 
> 
> 	Thanks!
> 
> 	Tony
> 	> -----Original Message-----
> 	> From: discuss <ovs-discuss-bounces at openvswitch.org <mailto:ovs-
> discuss-bounces at openvswitch.org> > On Behalf Of Tony
> 	> Liu
> 	> Sent: Wednesday, August 5, 2020 4:29 PM
> 	> To: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org> >
> 	> Cc: ovs-dev <ovs-dev at openvswitch.org <mailto:ovs-
> dev at openvswitch.org> >; ovs-discuss <ovs-
> 	> discuss at openvswitch.org <mailto:discuss at openvswitch.org> >
> 	> Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> 	>
> 	> Hi Han,
> 	>
> 	> After setting connection target="ptcp:6642:0.0.0.0" for ovn-sb-db,
> I see
> 	> this error.
> 	> ========
> 	> 2020-08-
> 	> 05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.0
> <http://0.0.0.0> :
> 	> listen failed: Address already in use ======== Anything I am
> missing
> 	> here?
> 	>
> 	>
> 	> Thanks!
> 	>
> 	> Tony
> 	> > -----Original Message-----
> 	> > From: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org> >
> 	> > Sent: Tuesday, August 4, 2020 4:44 PM
> 	> > To: Tony Liu <tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> >
> 	> > Cc: Numan Siddique <numans at ovn.org <mailto:numans at ovn.org> >;
> Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org> >; ovs-
> 	> > discuss <ovs-discuss at openvswitch.org <mailto:ovs-
> discuss at openvswitch.org> >; ovs-dev
> 	> > <ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> >
> 	> > Subject: Re: [ovs-discuss] [OVN] no response to inactivity
> probe
> 	> >
> 	> >
> 	> >
> 	> > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu
> <tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> 	> > <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > > wrote:
> 	> >
> 	> >
> 	> >     Hi,
> 	> >
> 	> >     Since I have 3 OVN DB nodes, should I add 3 rows in
> connection
> 	> table
> 	> >     for the inactivity_probe? Or put 3 addresses into one row?
> 	> >
> 	> >     "set-connection" set one row only, and there is no "add-
> connection".
> 	> >     How should I add 3 rows into the table connection?
> 	> >
> 	> >
> 	> >
> 	> >
> 	> > You only need to set one row. Try this command:
> 	> >
> 	> > ovn-nbctl -- --id=@conn_uuid create Connection
> 	> > target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set
> NB_Global .
> 	> > connections=@conn_uuid
> 	> >
> 	> >
> 	> >
> 	> >     Thanks!
> 	> >
> 	> >     Tony
> 	> >
> 	> >     > -----Original Message-----
> 	> >     > From: Numan Siddique <numans at ovn.org
> <mailto:numans at ovn.org>  <mailto:numans at ovn.org
> <mailto:numans at ovn.org> > >
> 	> >     > Sent: Tuesday, August 4, 2020 12:36 AM
> 	> >     > To: Tony Liu <tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com>
> 	> > <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > >
> 	> >     > Cc: ovs-discuss <ovs-discuss at openvswitch.org <mailto:ovs-
> discuss at openvswitch.org>  <mailto:ovs- <mailto:ovs->
> 	> > discuss at openvswitch.org <mailto:discuss at openvswitch.org> > >;
> ovs-dev <ovs-
> 	> >     > dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > >
> 	> >     > Subject: Re: [ovs-discuss] [OVN] no response to
> inactivity probe
> 	> >     >
> 	> >     >
> 	> >     >
> 	> >     > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu
> <tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> 	> > <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> >
> 	> >     > <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com>
> 	> > <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > > > wrote:
> 	> >     >
> 	> >     >
> 	> >     >       In my deployment, on each Neutron server, there are
> 13
> 	> > Neutron
> 	> >     > server processes.
> 	> >     >       I see 12 of them (monitor, maintenance, RPC, API)
> connect
> 	> > to both
> 	> >     > ovn-nb-db
> 	> >     >       and ovn-sb-db. With 3 Neutron server nodes, that's
> 36 OVSDB
> 	> > clients.
> 	> >     >       Is so many clients OK?
> 	> >     >
> 	> >     >       Any suggestions how to figure out which side
> doesn't
> 	> > respond the
> 	> >     > probe,
> 	> >     >       if it's bi-directional? I don't see any activities
> from
> 	> > logging,
> 	> >     > other than
> 	> >     >       connect/drop and reconnect...
> 	> >     >
> 	> >     >       BTW, please let me know if this is not the right
> place to
> 	> > discuss
> 	> >     > Neutron OVN
> 	> >     >       ML2 driver.
> 	> >     >
> 	> >     >
> 	> >     >       Thanks!
> 	> >     >
> 	> >     >       Tony
> 	> >     >
> 	> >     >       > -----Original Message-----
> 	> >     >       > From: dev <ovs-dev-bounces at openvswitch.org
> <mailto:ovs-dev-bounces at openvswitch.org>  <mailto:ovs- <mailto:ovs->
> 	> > dev-bounces at openvswitch.org <mailto:dev-
> bounces at openvswitch.org> >  <mailto:ovs-dev- <mailto:ovs-dev->
> <mailto:ovs-dev- <mailto:ovs-dev-> >
> 	> >     > bounces at openvswitch.org <mailto:bounces at openvswitch.org>
> <mailto:bounces at openvswitch.org <mailto:bounces at openvswitch.org> > > >
> On
> 	> > Behalf Of Tony Liu
> 	> >     >       > Sent: Monday, August 3, 2020 7:45 PM
> 	> >     >       > To: ovs-discuss <ovs-discuss at openvswitch.org
> <mailto:ovs-discuss at openvswitch.org>  <mailto:ovs- <mailto:ovs->
> 	> > discuss at openvswitch.org <mailto:discuss at openvswitch.org> >
> <mailto:ovs- <mailto:ovs->  <mailto:ovs- <mailto:ovs-> >
> 	> >     > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> > > >;
> 	> > ovs-dev <ovs-
> 	> >     >       > dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> 	> > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > > >
> 	> >     >       > Subject: [ovs-dev] [OVN] no response to
> inactivity probe
> 	> >     >       >
> 	> >     >       > Hi,
> 	> >     >       >
> 	> >     >       > Neutron OVN ML2 driver was disconnected by ovn-
> nb-db.
> 	> > There are
> 	> >     > many
> 	> >     >       > error messages from ovn-nb-db leader.
> 	> >     >       > ========
> 	> >     >       > 2020-08-
> 	> > 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620
> <http://10.6.20.81:58620>
> 	> > <http://10.6.20.81:58620>
> 	> >     > <http://10.6.20.81:58620> : no
> 	> >     >       > response to inactivity probe after 5 seconds,
> 	> > disconnecting
> 	> >     >       > 2020-08-
> 	> > 04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300
> <http://10.6.20.81:58300>
> 	> > <http://10.6.20.81:58300>
> 	> >     > <http://10.6.20.81:58300> : no
> 	> >     >       > response to inactivity probe after 5 seconds,
> 	> > disconnecting
> 	> >     >       > 2020-08-
> 	> > 04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582
> <http://10.6.20.81:59582>
> 	> > <http://10.6.20.81:59582>
> 	> >     > <http://10.6.20.81:59582> : no
> 	> >     >       > response to inactivity probe after 5 seconds,
> 	> > disconnecting
> 	> >     >       > 2020-08-
> 	> > 04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626
> <http://10.6.20.83:42626>
> 	> > <http://10.6.20.83:42626>
> 	> >     > <http://10.6.20.83:42626> : no
> 	> >     >       > response to inactivity probe after 5 seconds,
> 	> > disconnecting
> 	> >     >       > 2020-08-
> 	> > 04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412
> <http://10.6.20.82:45412>
> 	> > <http://10.6.20.82:45412>
> 	> >     > <http://10.6.20.82:45412> : no
> 	> >     >       > response to inactivity probe after 5 seconds,
> 	> > disconnecting
> 	> >     >       > 2020-08-
> 	> > 04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416
> <http://10.6.20.81:59416>
> 	> > <http://10.6.20.81:59416>
> 	> >     > <http://10.6.20.81:59416> : no
> 	> >     >       > response to inactivity probe after 5 seconds,
> 	> > disconnecting
> 	> >     >       > 2020-08-
> 	> > 04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004
> <http://10.6.20.81:60004>
> 	> > <http://10.6.20.81:60004>
> 	> >     > <http://10.6.20.81:60004> : no
> 	> >     >       > response to inactivity probe after 5 seconds,
> 	> > disconnecting
> 	> >     > ========
> 	> >     >       >
> 	> >     >       > Could anyone share a bit details how this
> inactivity
> 	> > probe works?
> 	> >     >
> 	> >     >
> 	> >     >
> 	> >     > The inactivity probe is sent by both the server and
> clients
> 	> >     > independently.
> 	> >     > Meaning ovsdb-server will send an inactivity probe every
> 'x'
> 	> > configured
> 	> >     > seconds to all its connected clients and if it doesn't
> get a
> 	> reply
> 	> > from
> 	> >     > the client within some time, it disconnects the
> connection.
> 	> >     >
> 	> >     > The inactivity probe from the server side can be
> configured. Run
> 	> > "ovn-
> 	> >     > nbctl list connection"
> 	> >     > and you will see inactivity_probe column. You can set
> this column
> 	> > to
> 	> >     > desired value like - ovn-nbctl set connection .
> 	> > inactivity_probe=30000
> 	> >     > (for 30 seconds)
> 	> >     >
> 	> >     > The same thing for SB ovsdb-server.
> 	> >     >
> 	> >     > Similarly each client (ovn-northd, ovn-controller,
> neutron server)
> 	> > sends
> 	> >     > inactivity probe every 'y' seconds and if the client
> doesn't get
> 	> > any
> 	> >     > reply from ovsdb-server it will disconnect the connection
> and
> 	> > reconnect
> 	> >     > again.
> 	> >     >
> 	> >     > For ovn-northd you can configured this as - ovn-nbctl set
> 	> NB_Global
> 	> > .
> 	> >     > options:northd_probe_interval=30000
> 	> >     >
> 	> >     > For ovn-controllers - ovs-vsctl set open .
> external_ids:ovn-
> 	> > remote-
> 	> >     > probe-interval=30000
> 	> >     >
> 	> >     > There is also a probe interval for openflow connection
> from ovn-
> 	> >     > controller to ovs-vswitchd which you can configure as
> ovs-vsctl
> 	> set
> 	> >     > open . external_ids:ovn-openflow-probe-interval=30 (this
> is in
> 	> > seconds)
> 	> >     >
> 	> >     >
> 	> >     > Regarding the neutron server I think it is set to 60
> seconds.
> 	> > Please see
> 	> >     > this -
> 	> >     >
> 	> >
> 	>
> https://github.com/openstack/neutron/blob/master/neutron/conf/plugins/ml
> 	> >     > 2/drivers/ovn/ovn_conf.py#L80
> 	> >     >
> 	> >     > From the logs you shared, it looks like ovsdb-server is
> not
> 	> getting
> 	> > the
> 	> >     > probe reply from neutron server after 5 seconds and hence
> it is
> 	> >     > disconnecting. Not sure what's happening though.
> 	> >     >
> 	> >     > You can try increasing the inactivity probe interval on
> the
> 	> > ovsdb-server
> 	> >     > side with the first command I shared.
> 	> >     > Note: If "ovn-nbctl list connection" returns empty, you
> need to
> 	> > create a
> 	> >     > connection row like - ovn-nbctl set-connection
> ptcp:6641:<IP>
> 	> >     >
> 	> >     >
> 	> >     > Thanks
> 	> >     > Numan
> 	> >     >
> 	> >     >
> 	> >     >
> 	> >     >
> 	> >     >       > From OVN ML2 driver log, I see it connected to
> the leader,
> 	> > then
> 	> >     > the
> 	> >     >       > connection was closed by leader after 5 or 6
> seconds. Is
> 	> > this
> 	> >     > probe one-
> 	> >     >       > way or two-ways?
> 	> >     >       > Both sides are not busy, not taking much CPU
> cycles. Not
> 	> > sure how
> 	> >     > this
> 	> >     >       > could happen. Any thoughts?
> 	> >     >       >
> 	> >     >       >
> 	> >     >       > Thanks!
> 	> >     >       >
> 	> >     >       > Tony
> 	> >     >       >
> 	> >     >       >
> 	> >     >       >
> 	> >     >       > _______________________________________________
> 	> >     >       > dev mailing list
> 	> >     >       > dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> 	> > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > >
> 	> >     >       >
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> 	> >     >       _______________________________________________
> 	> >     >       discuss mailing list
> 	> >     >       discuss at openvswitch.org
> <mailto:discuss at openvswitch.org>  <mailto:discuss at openvswitch.org
> <mailto:discuss at openvswitch.org> >
> 	> > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> > >
> 	> >     >       https://mail.openvswitch.org/mailman/listinfo/ovs-
> discuss
> 	> >     >
> 	> >     >
> 	> >
> 	> >
> 	>
> 	> _______________________________________________
> 	> discuss mailing list
> 	> discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> 	> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> 



More information about the dev mailing list