[ovs-dev] [ovs-discuss] [OVN] no response to inactivity probe
Tony Liu
tonyliu0592 at hotmail.com
Thu Aug 6 19:07:42 UTC 2020
Inline...
Thanks!
Tony
> -----Original Message-----
> From: Han Zhou <hzhou at ovn.org>
> Sent: Thursday, August 6, 2020 11:37 AM
> To: Tony Liu <tonyliu0592 at hotmail.com>
> Cc: Han Zhou <hzhou at ovn.org>; Numan Siddique <numans at ovn.org>; ovs-dev
> <ovs-dev at openvswitch.org>; ovs-discuss <ovs-discuss at openvswitch.org>
> Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
>
>
>
> On Thu, Aug 6, 2020 at 11:11 AM Tony Liu <tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > wrote:
> >
> > Inline... (please read with monospaced font:))
> >
> > Thanks!
> >
> > Tony
> > > -----Original Message-----
> > > From: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org> >
> > > Sent: Wednesday, August 5, 2020 11:48 PM
> > > To: Tony Liu <tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> >
> > > Cc: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org> >; Numan Siddique
> > > <numans at ovn.org <mailto:numans at ovn.org> >; ovs-dev
> > > <ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> >;
> > > ovs-discuss <ovs-discuss at openvswitch.org
> > > <mailto:ovs-discuss at openvswitch.org> >
> > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> > >
> > >
> > >
> > > On Wed, Aug 5, 2020 at 9:14 PM Tony Liu <tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > > wrote:
> > >
> > >
> > > I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb-
> db
> > > and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first
> node
> > > of cluster. Also ovn-openflow-probe-interval=30 on compute
> node.
> > > It seems helping. Not that many connect/drop/reconnect in
> logging.
> > > That "commit failure" is also gone.
> > > The issue I reported in another thread "packet drop" seems
> gone.
> > > And launching VM starts working.
> > >
> > > How should I set connection table for all ovn-nb-db and ovn-
> sb-db
> > > nodes in the cluster to set inactivity_probe?
> > > One row with address 0.0.0.0 seems not working.
> > >
> > > You can simply use 0.0.0.0 in the connection table, but don't
> > > specify the same connection method on the command line when starting
> > > ovsdb- server for NB/SB DB. Otherwise, these are conflicting and
> > > that's why you saw "Address already in use" error.
> >
> > Could you share a bit details how it works?
> > I thought the row in connection table only tells nbdb and sbdb the
> > probe interval. Isn't that right? Does nbdb and sbdb also create
> > socket based on target column?
>
> >
>
> In --remote option of ovsdb-server, you can specify either a connection
> method directly, or specify the db,table,column which contains the
> connection information.
> Please see manpage ovsdb-server(1).
Here is how one of those 3 nbdb nodes invoked.
========
ovsdb-server -vconsole:off -vfile:info --log-file=/var/log/kolla/openvswitch/ovn-sb-db.log --remote=punix:/var/run/ovn/ovnsb_db.sock --pidfile=/run/ovn/ovnsb_db.pid --unixctl=/var/run/ovn/ovnsb_db.ctl --remote=db:OVN_Southbound,SB_Global,connections --private-key=db:OVN_Southbound,SSL,private_key --certificate=db:OVN_Southbound,SSL,certificate --ca-cert=db:OVN_Southbound,SSL,ca_cert --ssl-protocols=db:OVN_Southbound,SSL,ssl_protocols --ssl-ciphers=db:OVN_Southbound,SSL,ssl_ciphers --remote=ptcp:6642:10.6.20.84 /var/lib/openvswitch/ovn-sb/ov sb.db
========
It creates UNIX and TCP sockets, and takes configuration from DB.
Does that look ok?
Given that, what the target column should be for all nodes of the cluster?
And whatever target is set, ovsdb-server will create socket, right?
Oh... Should I do "--remote=ptcp:6642:0.0.0.0"? Then I can set the same
in connection table, and it won't cause conflict?
If --remote and connection target are the same, whoever comes in later
will be ignored, right?
In coding, does ovsdb-server create a connection object for each of
--remote and connection target, or it's one single connection object
for both of them because method:port:address is the same? I'd expect
the single object.
> > > Is "external_ids:ovn-remote-probe-interval" in ovsdb-server on
> > > compute node for ovn-controller to probe ovn-sb-db?
> > >
> > > OVSDB probe is bidirectional, so you need to set this value, too, if
> > > you don't want too many probes handled by the SB server. (setting
> > > the connection table for SB only changes the server side).
> >
> > In that case, how do I set probe interval for ovn-controller?
> > My understanding is that, ovn-controller reads configuration from
> > ovsdb-server on the local compute node. Isn't that right?
>
> >
>
> The configuration you mentioned "external_ids:ovn-remote-probe-interval"
> is exactly the way to set the ovn-controller -> SB probe interval.
> (SB -> ovn-controller probe is set in the connection table of SB)
>
>
> You are right that ovn-controller reads configuration from the local
> ovsdb-server. This setting is in local ovsdb-server.
>
>
> > > Is "external_ids:ovn-openflow-probe-interval" in ovsdb-server
> on
> > > compute node for ovn-controller to probe ovsdb-server?
> > >
> > > It is for the OpenFlow connection between ovn-controller and ovs-
> > > vswitchd, which is part of the OpenFlow protocol.
> > >
> > > What's probe interval for ovsdb-server to probe ovn-controller?
> > >
> > > The local ovsdb connection uses unix socket, which doesn't send
> > > probe by default (if I remember correctly).
> >
> > Here is how ovsdb-server and ovn-controller is invoked on compute node.
> > ========
> > root 41129 0.0 0.0 157556 20532 ? S Jul30 1:51
> /usr/sbin/ovsdb-server /var/lib/openvswitch/conf.db -vconsole:emer -
> vsyslog:err -vfile:info --remote=punix:/run/openvswitch/db.sock --
> remote=ptcp:6640:127.0.0.1 --
> remote=db:Open_vSwitch,Open_vSwitch,manager_options --log-
> file=/var/log/kolla/openvswitch/ovsdb-server.log --pidfile
> >
> > root 63775 55.9 0.4 1477796 1224324 ? Sl Aug04 1360:55
> /usr/bin/ovn-controller --pidfile=/run/ovn/ovn-controller.pid --log-
> file=/var/log/kolla/openvswitch/ovn-controller.log tcp:127.0.0.1:6640
> <http://127.0.0.1:6640>
> > ========
> > Is that OK? Or UNIX socket method is recommended for ovn-controller
>
> > to connect to ovsdb-server?
>
> If using TCP, by default it is 5s probe interval. I think it is better
> to use unix socket. (but maybe it doesn't matter that much)
>
>
> >
> > Here is the configuration in open_vswitch table in ovsdb-server.
> > ========
> > external_ids : {ovn-encap-ip="10.6.30.22", ovn-encap-
> type=geneve, ovn-openflow-probe-interval="30", ovn-
> remote="tcp:10.6.20.84:6642
> <http://10.6.20.84:6642> ,tcp:10.6.20.85:6642
> <http://10.6.20.85:6642> ,tcp:10.6.20.86:6642 <http://10.6.20.86:6642> ",
> ovn-remote-probe-interval="60000", system-id="compute-3"}
> > ========
> > ovn-controller connects to ovsdb-server and reads this configuration,
> > so it knows how to connect to all sbdb nodes, right?
>
> >
> Yes, you are right.
>
> > If it's TCP between ovn-controller and ovsdb-server, is that probe
> > interval setting will also apply to the probe from ovn-controller to
> > ovsdb-server?
>
> >
> No. They are not related.
>
>
> > ovn-controller connects to ovs-vswitchd by UNIX socket to program
> > open-flow. ovs-vswitchd and ovsdb-server are connected by UNIX too.
> > So, is that ovn-openflow-probe-interval for the probe from
> > ovn-controller to ovs-vswitchd via UNIX?
> >
> > As a summary for the probe setting,
> >
> > +--------------+ driver configuration
> > | ovn-driver |
> > +--------------+
> > ^ |
> > | v
> > +--------------+ inactivity_probe in table "Connection"
> > | ovn-nb-db |
> > +--------------+
> > ^ |
> > | v
> > +--------------+ options:northd_probe_interval in table "NB_Global"
> > | ovn-northd | in nbdb.
> > +--------------+
> > ^ |
> > | v
> > +--------------+ inactivity_probe in table "Connection"
> > | ovn-sb-db |
> > +--------------+
> > ^ |
> > | v
> > +--------------------------------+ in table "Open_vSwitch" in
> > +--------------------------------+ ovsdb-server
> > | ovn-controller | ovn-remote-probe-interval for TCP
> > +--------------------------------+ probe to ovsdb-server,
> > ^ | ^ | ovn-openflow-probe-interval for
> UNIX
> > | v TCP | v UNIX probe to ovs-vswitchd
> > +--------------+ +--------------+
> > | ovsdb-server | | ovs-vswitchd |
> > +--------------+ +--------------+
> >
>
> > Is that correct?
>
> Correct. Except that you don't have to use TCP between ovn-controller
> and the local ovsdb-server. Use UNIX and then you don't need to worry
> about the probe between them.
>
>
> >
> > > For ovn-controller, since it is implemented with
> > > incremental-processing, even if there are probes from openflow or
> local ovsdb, it doesn't matter.
> > > If there is no configuration change, ovn-controller simply replies
> > > the probe and there is no extra cost.
> >
> > I did see errors about connecting to br-mgmt (UNIX connection to
> > ovs-vswitchd) in ovn-controller logging. After setting proble
> > internal, those errors are gone. It seems that it does matter.
>
> >
> What's the error log? (I suspect maybe it is just some log but not error)
This is from ovn-controller.
========
2020-08-06T03:53:08.663Z|128534|rconn|ERR|unix:/var/run/openvswitch/br-int.mgmt: no response to inactivity probe after 10 seconds, disconnecting
========
It seems that there is probe on UNIX socket.
> > >
> > > Thanks!
> > >
> > > Tony
> > > > -----Original Message-----
> > > > From: discuss <ovs-discuss-bounces at openvswitch.org
> > > <mailto:ovs-discuss-bounces at openvswitch.org> <mailto:ovs-
> <mailto:ovs-> discuss-bounces at openvswitch.org <mailto:discuss-
> bounces at openvswitch.org> > > On Behalf Of Tony
> > > > Liu
> > > > Sent: Wednesday, August 5, 2020 4:29 PM
> > > > To: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org>
> <mailto:hzhou at ovn.org <mailto:hzhou at ovn.org> > >
> > > > Cc: ovs-dev <ovs-dev at openvswitch.org
> > > <mailto:ovs-dev at openvswitch.org> <mailto:ovs- <mailto:ovs->
> dev at openvswitch.org <mailto:dev at openvswitch.org> > >; ovs-discuss <ovs-
> > > > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> > >
> > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity
> probe
> > > >
> > > > Hi Han,
> > > >
> > > > After setting connection target="ptcp:6642:0.0.0.0" for
> > > ovn-sb-db, I see
> > > > this error.
> > > > ========
> > > > 2020-08-
> > > >
> > > 05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.0
> > > <http://0.0.0.0> :
> > > > listen failed: Address already in use ======== Anything I am
> > > missing
> > > > here?
> > > >
> > > >
> > > > Thanks!
> > > >
> > > > Tony
> > > > > -----Original Message-----
> > > > > From: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org>
> <mailto:hzhou at ovn.org <mailto:hzhou at ovn.org> > >
> > > > > Sent: Tuesday, August 4, 2020 4:44 PM
> > > > > To: Tony Liu <tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > >
> > > > > Cc: Numan Siddique <numans at ovn.org <mailto:numans at ovn.org>
> > > <mailto:numans at ovn.org <mailto:numans at ovn.org> > >; Han Zhou
> <hzhou at ovn.org <mailto:hzhou at ovn.org> <mailto:hzhou at ovn.org
> <mailto:hzhou at ovn.org> > >; ovs-
> > > > > discuss <ovs-discuss at openvswitch.org
> > > <mailto:ovs-discuss at openvswitch.org> <mailto:ovs- <mailto:ovs->
> discuss at openvswitch.org <mailto:discuss at openvswitch.org> > >; ovs-dev
> > > > > <ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>
> <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> > >
> > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity
> > > probe
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu
> > > <tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com> >
> > > > > <mailto:tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > > > wrote:
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > Since I have 3 OVN DB nodes, should I add 3 rows in
> > > connection
> > > > table
> > > > > for the inactivity_probe? Or put 3 addresses into one
> row?
> > > > >
> > > > > "set-connection" set one row only, and there is no
> "add-
> > > connection".
> > > > > How should I add 3 rows into the table connection?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > You only need to set one row. Try this command:
> > > > >
> > > > > ovn-nbctl -- --id=@conn_uuid create Connection
> > > > > target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set
> > > NB_Global .
> > > > > connections=@conn_uuid
> > > > >
> > > > >
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Tony
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Numan Siddique <numans at ovn.org
> <mailto:numans at ovn.org>
> > > <mailto:numans at ovn.org <mailto:numans at ovn.org> >
> > > <mailto:numans at ovn.org <mailto:numans at ovn.org>
> <mailto:numans at ovn.org <mailto:numans at ovn.org> > > >
> > > > > > Sent: Tuesday, August 4, 2020 12:36 AM
> > > > > > To: Tony Liu <tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com>
> > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com> >
> > > > > <mailto:tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > > >
> > > > > > Cc: ovs-discuss <ovs-discuss at openvswitch.org
> <mailto:ovs-discuss at openvswitch.org> <mailto:ovs- <mailto:ovs->
> > > discuss at openvswitch.org <mailto:discuss at openvswitch.org> >
> <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> >
> > > > > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > <mailto:discuss at openvswitch.org
> <mailto:discuss at openvswitch.org> > > >; ovs-dev <ovs-
> > > > > > dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > > >
> > > > > > Subject: Re: [ovs-discuss] [OVN] no response to
> > > inactivity probe
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu
> > > <tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com> >
> > > > > <mailto:tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > >
> > > > > > <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com>
> > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com> >
> > > > > <mailto:tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > > > > wrote:
> > > > > >
> > > > > >
> > > > > > In my deployment, on each Neutron server,
> there are
> > > 13
> > > > > Neutron
> > > > > > server processes.
> > > > > > I see 12 of them (monitor, maintenance, RPC,
> API)
> > > connect
> > > > > to both
> > > > > > ovn-nb-db
> > > > > > and ovn-sb-db. With 3 Neutron server nodes,
> that's
> > > 36 OVSDB
> > > > > clients.
> > > > > > Is so many clients OK?
> > > > > >
> > > > > > Any suggestions how to figure out which side
> > > doesn't
> > > > > respond the
> > > > > > probe,
> > > > > > if it's bi-directional? I don't see any
> activities
> > > from
> > > > > logging,
> > > > > > other than
> > > > > > connect/drop and reconnect...
> > > > > >
> > > > > > BTW, please let me know if this is not the
> right
> > > place to
> > > > > discuss
> > > > > > Neutron OVN
> > > > > > ML2 driver.
> > > > > >
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Tony
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: dev <ovs-dev-bounces at openvswitch.org
> <mailto:ovs-dev-bounces at openvswitch.org>
> > > <mailto:ovs-dev-bounces at openvswitch.org <mailto:ovs-dev-
> bounces at openvswitch.org> > <mailto:ovs- <mailto:ovs-> <mailto:ovs-
> <mailto:ovs-> >
> > > > > dev-bounces at openvswitch.org
> > > <mailto:dev-bounces at openvswitch.org> <mailto:dev- <mailto:dev->
> > > bounces at openvswitch.org <mailto:bounces at openvswitch.org> > >
> > > <mailto:ovs-dev- <mailto:ovs-dev-> <mailto:ovs-dev-
> > > <mailto:ovs-dev-> >
> > > <mailto:ovs-dev- <mailto:ovs-dev-> <mailto:ovs-dev- <mailto:ovs-
> dev-> > >
> > > > > > bounces at openvswitch.org
> <mailto:bounces at openvswitch.org> <mailto:bounces at openvswitch.org
> <mailto:bounces at openvswitch.org> >
> > > <mailto:bounces at openvswitch.org <mailto:bounces at openvswitch.org>
> > > <mailto:bounces at openvswitch.org
> <mailto:bounces at openvswitch.org> > > > > On
> > > > > Behalf Of Tony Liu
> > > > > > > Sent: Monday, August 3, 2020 7:45 PM
> > > > > > > To: ovs-discuss <ovs-discuss at openvswitch.org
> <mailto:ovs-discuss at openvswitch.org>
> > > <mailto:ovs-discuss at openvswitch.org <mailto:ovs-
> discuss at openvswitch.org> > <mailto:ovs- <mailto:ovs-> <mailto:ovs-
> <mailto:ovs-> >
> > > > > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> > >
> > > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> >
> <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> > >
> > > > > > discuss at openvswitch.org
> <mailto:discuss at openvswitch.org> <mailto:discuss at openvswitch.org
> <mailto:discuss at openvswitch.org> >
> > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> > > > >;
> > > > > ovs-dev <ovs-
> > > > > > > dev at openvswitch.org
> <mailto:dev at openvswitch.org> <mailto:dev at openvswitch.org
> <mailto:dev at openvswitch.org> >
> > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > >
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > > > >
> > > > > > > Subject: [ovs-dev] [OVN] no response to
> > > inactivity probe
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Neutron OVN ML2 driver was disconnected by
> ovn-
> > > nb-db.
> > > > > There are
> > > > > > many
> > > > > > > error messages from ovn-nb-db leader.
> > > > > > > ========
> > > > > > > 2020-08-
> > > > > 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620
> > > <http://10.6.20.81:58620> <http://10.6.20.81:58620>
> > > > > <http://10.6.20.81:58620>
> > > > > > <http://10.6.20.81:58620> : no
> > > > > > > response to inactivity probe after 5 seconds,
> > > > > disconnecting
> > > > > > > 2020-08-
> > > > > 04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300
> > > <http://10.6.20.81:58300> <http://10.6.20.81:58300>
> > > > > <http://10.6.20.81:58300>
> > > > > > <http://10.6.20.81:58300> : no
> > > > > > > response to inactivity probe after 5 seconds,
> > > > > disconnecting
> > > > > > > 2020-08-
> > > > > 04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582
> > > <http://10.6.20.81:59582> <http://10.6.20.81:59582>
> > > > > <http://10.6.20.81:59582>
> > > > > > <http://10.6.20.81:59582> : no
> > > > > > > response to inactivity probe after 5 seconds,
> > > > > disconnecting
> > > > > > > 2020-08-
> > > > > 04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626
> > > <http://10.6.20.83:42626> <http://10.6.20.83:42626>
> > > > > <http://10.6.20.83:42626>
> > > > > > <http://10.6.20.83:42626> : no
> > > > > > > response to inactivity probe after 5 seconds,
> > > > > disconnecting
> > > > > > > 2020-08-
> > > > > 04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412
> > > <http://10.6.20.82:45412> <http://10.6.20.82:45412>
> > > > > <http://10.6.20.82:45412>
> > > > > > <http://10.6.20.82:45412> : no
> > > > > > > response to inactivity probe after 5 seconds,
> > > > > disconnecting
> > > > > > > 2020-08-
> > > > > 04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416
> > > <http://10.6.20.81:59416> <http://10.6.20.81:59416>
> > > > > <http://10.6.20.81:59416>
> > > > > > <http://10.6.20.81:59416> : no
> > > > > > > response to inactivity probe after 5 seconds,
> > > > > disconnecting
> > > > > > > 2020-08-
> > > > > 04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004
> > > <http://10.6.20.81:60004> <http://10.6.20.81:60004>
> > > > > <http://10.6.20.81:60004>
> > > > > > <http://10.6.20.81:60004> : no
> > > > > > > response to inactivity probe after 5 seconds,
> > > > > disconnecting
> > > > > > ========
> > > > > > >
> > > > > > > Could anyone share a bit details how this
> > > inactivity
> > > > > probe works?
> > > > > >
> > > > > >
> > > > > >
> > > > > > The inactivity probe is sent by both the server and
> > > clients
> > > > > > independently.
> > > > > > Meaning ovsdb-server will send an inactivity probe
> every
> > > 'x'
> > > > > configured
> > > > > > seconds to all its connected clients and if it
> doesn't
> > > get a
> > > > reply
> > > > > from
> > > > > > the client within some time, it disconnects the
> > > connection.
> > > > > >
> > > > > > The inactivity probe from the server side can be
> > > configured. Run
> > > > > "ovn-
> > > > > > nbctl list connection"
> > > > > > and you will see inactivity_probe column. You can
> set
> > > this column
> > > > > to
> > > > > > desired value like - ovn-nbctl set connection .
> > > > > inactivity_probe=30000
> > > > > > (for 30 seconds)
> > > > > >
> > > > > > The same thing for SB ovsdb-server.
> > > > > >
> > > > > > Similarly each client (ovn-northd, ovn-controller,
> > > neutron server)
> > > > > sends
> > > > > > inactivity probe every 'y' seconds and if the client
> > > doesn't get
> > > > > any
> > > > > > reply from ovsdb-server it will disconnect the
> connection
> > > and
> > > > > reconnect
> > > > > > again.
> > > > > >
> > > > > > For ovn-northd you can configured this as - ovn-
> nbctl set
> > > > NB_Global
> > > > > .
> > > > > > options:northd_probe_interval=30000
> > > > > >
> > > > > > For ovn-controllers - ovs-vsctl set open .
> > > external_ids:ovn-
> > > > > remote-
> > > > > > probe-interval=30000
> > > > > >
> > > > > > There is also a probe interval for openflow
> connection
> > > from ovn-
> > > > > > controller to ovs-vswitchd which you can configure
> as
> > > ovs-vsctl
> > > > set
> > > > > > open . external_ids:ovn-openflow-probe-interval=30
> (this
> > > is in
> > > > > seconds)
> > > > > >
> > > > > >
> > > > > > Regarding the neutron server I think it is set to 60
> > > seconds.
> > > > > Please see
> > > > > > this -
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/openstack/neutron/blob/master/neutron/conf/plugins/ml
> > > > > > 2/drivers/ovn/ovn_conf.py#L80
> > > > > >
> > > > > > From the logs you shared, it looks like ovsdb-server
> is
> > > not
> > > > getting
> > > > > the
> > > > > > probe reply from neutron server after 5 seconds and
> hence
> > > it is
> > > > > > disconnecting. Not sure what's happening though.
> > > > > >
> > > > > > You can try increasing the inactivity probe interval
> on
> > > the
> > > > > ovsdb-server
> > > > > > side with the first command I shared.
> > > > > > Note: If "ovn-nbctl list connection" returns empty,
> you
> > > need to
> > > > > create a
> > > > > > connection row like - ovn-nbctl set-connection
> > > ptcp:6641:<IP>
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > > > Numan
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > From OVN ML2 driver log, I see it connected
> to
> > > the leader,
> > > > > then
> > > > > > the
> > > > > > > connection was closed by leader after 5 or 6
> > > seconds. Is
> > > > > this
> > > > > > probe one-
> > > > > > > way or two-ways?
> > > > > > > Both sides are not busy, not taking much CPU
> > > cycles. Not
> > > > > sure how
> > > > > > this
> > > > > > > could happen. Any thoughts?
> > > > > > >
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > Tony
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> _______________________________________________
> > > > > > > dev mailing list
> > > > > > > dev at openvswitch.org
> <mailto:dev at openvswitch.org> <mailto:dev at openvswitch.org
> <mailto:dev at openvswitch.org> >
> > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > >
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > > >
> > > > > > >
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > > > >
> _______________________________________________
> > > > > > discuss mailing list
> > > > > > discuss at openvswitch.org
> <mailto:discuss at openvswitch.org>
> > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> >
> > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> > >
> > > > > <mailto:discuss at openvswitch.org
> > > <mailto:discuss at openvswitch.org> <mailto:discuss at openvswitch.org
> <mailto:discuss at openvswitch.org> > <mailto:discuss at openvswitch.org
> <mailto:discuss at openvswitch.org> <mailto:discuss at openvswitch.org
> <mailto:discuss at openvswitch.org> > > >
> > > > > >
> https://mail.openvswitch.org/mailman/listinfo/ovs-
> > > discuss
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > _______________________________________________
> > > > discuss mailing list
> > > > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> >
> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> > >
> >
More information about the dev
mailing list