[ovs-discuss] [OVN] no response to inactivity probe
Han Zhou
hzhou at ovn.org
Thu Aug 6 18:36:38 UTC 2020
On Thu, Aug 6, 2020 at 11:11 AM Tony Liu <tonyliu0592 at hotmail.com> wrote:
>
> Inline... (please read with monospaced font:))
>
> Thanks!
>
> Tony
> > -----Original Message-----
> > From: Han Zhou <hzhou at ovn.org>
> > Sent: Wednesday, August 5, 2020 11:48 PM
> > To: Tony Liu <tonyliu0592 at hotmail.com>
> > Cc: Han Zhou <hzhou at ovn.org>; Numan Siddique <numans at ovn.org>; ovs-dev
> > <ovs-dev at openvswitch.org>; ovs-discuss <ovs-discuss at openvswitch.org>
> > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> >
> >
> >
> > On Wed, Aug 5, 2020 at 9:14 PM Tony Liu <tonyliu0592 at hotmail.com
> > <mailto:tonyliu0592 at hotmail.com> > wrote:
> >
> >
> > I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb-db
> > and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first node
> > of cluster. Also ovn-openflow-probe-interval=30 on compute node.
> > It seems helping. Not that many connect/drop/reconnect in logging.
> > That "commit failure" is also gone.
> > The issue I reported in another thread "packet drop" seems gone.
> > And launching VM starts working.
> >
> > How should I set connection table for all ovn-nb-db and ovn-sb-db
> > nodes in the cluster to set inactivity_probe?
> > One row with address 0.0.0.0 seems not working.
> >
> > You can simply use 0.0.0.0 in the connection table, but don't specify
> > the same connection method on the command line when starting ovsdb-
> > server for NB/SB DB. Otherwise, these are conflicting and that's why you
> > saw "Address already in use" error.
>
> Could you share a bit details how it works?
> I thought the row in connection table only tells nbdb and sbdb the
> probe interval. Isn't that right? Does nbdb and sbdb also create
> socket based on target column?
>
In --remote option of ovsdb-server, you can specify either a connection
method directly, or specify the db,table,column which contains the
connection information.
Please see manpage ovsdb-server(1).
> >
> > Is "external_ids:ovn-remote-probe-interval" in ovsdb-server on
> > compute node for ovn-controller to probe ovn-sb-db?
> >
> > OVSDB probe is bidirectional, so you need to set this value, too, if you
> > don't want too many probes handled by the SB server. (setting the
> > connection table for SB only changes the server side).
>
> In that case, how do I set probe interval for ovn-controller?
> My understanding is that, ovn-controller reads configuration from
> ovsdb-server on the local compute node. Isn't that right?
>
The configuration you mentioned "external_ids:ovn-remote-probe-interval" is
exactly the way to set the ovn-controller -> SB probe interval.
(SB -> ovn-controller probe is set in the connection table of SB)
You are right that ovn-controller reads configuration from the local
ovsdb-server. This setting is in local ovsdb-server.
> > Is "external_ids:ovn-openflow-probe-interval" in ovsdb-server on
> > compute node for ovn-controller to probe ovsdb-server?
> >
> > It is for the OpenFlow connection between ovn-controller and ovs-
> > vswitchd, which is part of the OpenFlow protocol.
> >
> > What's probe interval for ovsdb-server to probe ovn-controller?
> >
> > The local ovsdb connection uses unix socket, which doesn't send probe by
> > default (if I remember correctly).
>
> Here is how ovsdb-server and ovn-controller is invoked on compute node.
> ========
> root 41129 0.0 0.0 157556 20532 ? S Jul30 1:51
/usr/sbin/ovsdb-server /var/lib/openvswitch/conf.db -vconsole:emer
-vsyslog:err -vfile:info --remote=punix:/run/openvswitch/db.sock
--remote=ptcp:6640:127.0.0.1
--remote=db:Open_vSwitch,Open_vSwitch,manager_options
--log-file=/var/log/kolla/openvswitch/ovsdb-server.log --pidfile
>
> root 63775 55.9 0.4 1477796 1224324 ? Sl Aug04 1360:55
/usr/bin/ovn-controller --pidfile=/run/ovn/ovn-controller.pid
--log-file=/var/log/kolla/openvswitch/ovn-controller.log tcp:127.0.0.1:6640
> ========
> Is that OK? Or UNIX socket method is recommended for ovn-controller
> to connect to ovsdb-server?
If using TCP, by default it is 5s probe interval. I think it is better to
use unix socket. (but maybe it doesn't matter that much)
>
> Here is the configuration in open_vswitch table in ovsdb-server.
> ========
> external_ids : {ovn-encap-ip="10.6.30.22", ovn-encap-type=geneve,
ovn-openflow-probe-interval="30", ovn-remote="tcp:10.6.20.84:6642,tcp:
10.6.20.85:6642,tcp:10.6.20.86:6642", ovn-remote-probe-interval="60000",
system-id="compute-3"}
> ========
> ovn-controller connects to ovsdb-server and reads this configuration,
> so it knows how to connect to all sbdb nodes, right?
>
Yes, you are right.
> If it's TCP between ovn-controller and ovsdb-server, is that probe
> interval setting will also apply to the probe from ovn-controller to
> ovsdb-server?
>
No. They are not related.
> ovn-controller connects to ovs-vswitchd by UNIX socket to program
> open-flow. ovs-vswitchd and ovsdb-server are connected by UNIX too.
> So, is that ovn-openflow-probe-interval for the probe from ovn-controller
> to ovs-vswitchd via UNIX?
>
> As a summary for the probe setting,
>
> +--------------+ driver configuration
> | ovn-driver |
> +--------------+
> ^ |
> | v
> +--------------+ inactivity_probe in table "Connection"
> | ovn-nb-db |
> +--------------+
> ^ |
> | v
> +--------------+ options:northd_probe_interval in table "NB_Global"
> | ovn-northd | in nbdb.
> +--------------+
> ^ |
> | v
> +--------------+ inactivity_probe in table "Connection"
> | ovn-sb-db |
> +--------------+
> ^ |
> | v
> +--------------------------------+ in table "Open_vSwitch" in
ovsdb-server
> | ovn-controller | ovn-remote-probe-interval for TCP
> +--------------------------------+ probe to ovsdb-server,
> ^ | ^ | ovn-openflow-probe-interval for UNIX
> | v TCP | v UNIX probe to ovs-vswitchd
> +--------------+ +--------------+
> | ovsdb-server | | ovs-vswitchd |
> +--------------+ +--------------+
>
> Is that correct?
Correct. Except that you don't have to use TCP between ovn-controller and
the local ovsdb-server. Use UNIX and then you don't need to worry about the
probe between them.
>
> > For ovn-controller, since it is implemented with incremental-processing,
> > even if there are probes from openflow or local ovsdb, it doesn't
matter.
> > If there is no configuration change, ovn-controller simply replies the
> > probe and there is no extra cost.
>
> I did see errors about connecting to br-mgmt (UNIX connection to
> ovs-vswitchd) in ovn-controller logging. After setting proble internal,
> those errors are gone. It seems that it does matter.
>
What's the error log? (I suspect maybe it is just some log but not error)
> >
> >
> >
> >
> > Thanks!
> >
> > Tony
> > > -----Original Message-----
> > > From: discuss <ovs-discuss-bounces at openvswitch.org <mailto:ovs-
> > discuss-bounces at openvswitch.org> > On Behalf Of Tony
> > > Liu
> > > Sent: Wednesday, August 5, 2020 4:29 PM
> > > To: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org> >
> > > Cc: ovs-dev <ovs-dev at openvswitch.org <mailto:ovs-
> > dev at openvswitch.org> >; ovs-discuss <ovs-
> > > discuss at openvswitch.org <mailto:discuss at openvswitch.org> >
> > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> > >
> > > Hi Han,
> > >
> > > After setting connection target="ptcp:6642:0.0.0.0" for
ovn-sb-db,
> > I see
> > > this error.
> > > ========
> > > 2020-08-
> > >
05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.0
> > <http://0.0.0.0> :
> > > listen failed: Address already in use ======== Anything I am
> > missing
> > > here?
> > >
> > >
> > > Thanks!
> > >
> > > Tony
> > > > -----Original Message-----
> > > > From: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org> >
> > > > Sent: Tuesday, August 4, 2020 4:44 PM
> > > > To: Tony Liu <tonyliu0592 at hotmail.com
> > <mailto:tonyliu0592 at hotmail.com> >
> > > > Cc: Numan Siddique <numans at ovn.org <mailto:numans at ovn.org> >;
> > Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org> >; ovs-
> > > > discuss <ovs-discuss at openvswitch.org <mailto:ovs-
> > discuss at openvswitch.org> >; ovs-dev
> > > > <ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> >
> > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity
> > probe
> > > >
> > > >
> > > >
> > > > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu
> > <tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > <mailto:tonyliu0592 at hotmail.com
> > <mailto:tonyliu0592 at hotmail.com> > > wrote:
> > > >
> > > >
> > > > Hi,
> > > >
> > > > Since I have 3 OVN DB nodes, should I add 3 rows in
> > connection
> > > table
> > > > for the inactivity_probe? Or put 3 addresses into one row?
> > > >
> > > > "set-connection" set one row only, and there is no "add-
> > connection".
> > > > How should I add 3 rows into the table connection?
> > > >
> > > >
> > > >
> > > >
> > > > You only need to set one row. Try this command:
> > > >
> > > > ovn-nbctl -- --id=@conn_uuid create Connection
> > > > target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set
> > NB_Global .
> > > > connections=@conn_uuid
> > > >
> > > >
> > > >
> > > > Thanks!
> > > >
> > > > Tony
> > > >
> > > > > -----Original Message-----
> > > > > From: Numan Siddique <numans at ovn.org
> > <mailto:numans at ovn.org> <mailto:numans at ovn.org
> > <mailto:numans at ovn.org> > >
> > > > > Sent: Tuesday, August 4, 2020 12:36 AM
> > > > > To: Tony Liu <tonyliu0592 at hotmail.com
> > <mailto:tonyliu0592 at hotmail.com>
> > > > <mailto:tonyliu0592 at hotmail.com
> > <mailto:tonyliu0592 at hotmail.com> > >
> > > > > Cc: ovs-discuss <ovs-discuss at openvswitch.org <mailto:
ovs-
> > discuss at openvswitch.org> <mailto:ovs- <mailto:ovs->
> > > > discuss at openvswitch.org <mailto:discuss at openvswitch.org> > >;
> > ovs-dev <ovs-
> > > > > dev at openvswitch.org <mailto:dev at openvswitch.org>
> > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > >
> > > > > Subject: Re: [ovs-discuss] [OVN] no response to
> > inactivity probe
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu
> > <tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > <mailto:tonyliu0592 at hotmail.com
> > <mailto:tonyliu0592 at hotmail.com> >
> > > > > <mailto:tonyliu0592 at hotmail.com
> > <mailto:tonyliu0592 at hotmail.com>
> > > > <mailto:tonyliu0592 at hotmail.com
> > <mailto:tonyliu0592 at hotmail.com> > > > wrote:
> > > > >
> > > > >
> > > > > In my deployment, on each Neutron server, there
are
> > 13
> > > > Neutron
> > > > > server processes.
> > > > > I see 12 of them (monitor, maintenance, RPC, API)
> > connect
> > > > to both
> > > > > ovn-nb-db
> > > > > and ovn-sb-db. With 3 Neutron server nodes, that's
> > 36 OVSDB
> > > > clients.
> > > > > Is so many clients OK?
> > > > >
> > > > > Any suggestions how to figure out which side
> > doesn't
> > > > respond the
> > > > > probe,
> > > > > if it's bi-directional? I don't see any activities
> > from
> > > > logging,
> > > > > other than
> > > > > connect/drop and reconnect...
> > > > >
> > > > > BTW, please let me know if this is not the right
> > place to
> > > > discuss
> > > > > Neutron OVN
> > > > > ML2 driver.
> > > > >
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Tony
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: dev <ovs-dev-bounces at openvswitch.org
> > <mailto:ovs-dev-bounces at openvswitch.org> <mailto:ovs- <mailto:ovs->
> > > > dev-bounces at openvswitch.org <mailto:dev-
> > bounces at openvswitch.org> > <mailto:ovs-dev- <mailto:ovs-dev->
> > <mailto:ovs-dev- <mailto:ovs-dev-> >
> > > > > bounces at openvswitch.org <mailto:bounces at openvswitch.org>
> > <mailto:bounces at openvswitch.org <mailto:bounces at openvswitch.org> > > >
> > On
> > > > Behalf Of Tony Liu
> > > > > > Sent: Monday, August 3, 2020 7:45 PM
> > > > > > To: ovs-discuss <ovs-discuss at openvswitch.org
> > <mailto:ovs-discuss at openvswitch.org> <mailto:ovs- <mailto:ovs->
> > > > discuss at openvswitch.org <mailto:discuss at openvswitch.org> >
> > <mailto:ovs- <mailto:ovs-> <mailto:ovs- <mailto:ovs-> >
> > > > > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> > > >;
> > > > ovs-dev <ovs-
> > > > > > dev at openvswitch.org <mailto:dev at openvswitch.org>
> > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > > >
> > > > > > Subject: [ovs-dev] [OVN] no response to
> > inactivity probe
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Neutron OVN ML2 driver was disconnected by ovn-
> > nb-db.
> > > > There are
> > > > > many
> > > > > > error messages from ovn-nb-db leader.
> > > > > > ========
> > > > > > 2020-08-
> > > > 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620
> > <http://10.6.20.81:58620>
> > > > <http://10.6.20.81:58620>
> > > > > <http://10.6.20.81:58620> : no
> > > > > > response to inactivity probe after 5 seconds,
> > > > disconnecting
> > > > > > 2020-08-
> > > > 04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300
> > <http://10.6.20.81:58300>
> > > > <http://10.6.20.81:58300>
> > > > > <http://10.6.20.81:58300> : no
> > > > > > response to inactivity probe after 5 seconds,
> > > > disconnecting
> > > > > > 2020-08-
> > > > 04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582
> > <http://10.6.20.81:59582>
> > > > <http://10.6.20.81:59582>
> > > > > <http://10.6.20.81:59582> : no
> > > > > > response to inactivity probe after 5 seconds,
> > > > disconnecting
> > > > > > 2020-08-
> > > > 04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626
> > <http://10.6.20.83:42626>
> > > > <http://10.6.20.83:42626>
> > > > > <http://10.6.20.83:42626> : no
> > > > > > response to inactivity probe after 5 seconds,
> > > > disconnecting
> > > > > > 2020-08-
> > > > 04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412
> > <http://10.6.20.82:45412>
> > > > <http://10.6.20.82:45412>
> > > > > <http://10.6.20.82:45412> : no
> > > > > > response to inactivity probe after 5 seconds,
> > > > disconnecting
> > > > > > 2020-08-
> > > > 04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416
> > <http://10.6.20.81:59416>
> > > > <http://10.6.20.81:59416>
> > > > > <http://10.6.20.81:59416> : no
> > > > > > response to inactivity probe after 5 seconds,
> > > > disconnecting
> > > > > > 2020-08-
> > > > 04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004
> > <http://10.6.20.81:60004>
> > > > <http://10.6.20.81:60004>
> > > > > <http://10.6.20.81:60004> : no
> > > > > > response to inactivity probe after 5 seconds,
> > > > disconnecting
> > > > > ========
> > > > > >
> > > > > > Could anyone share a bit details how this
> > inactivity
> > > > probe works?
> > > > >
> > > > >
> > > > >
> > > > > The inactivity probe is sent by both the server and
> > clients
> > > > > independently.
> > > > > Meaning ovsdb-server will send an inactivity probe every
> > 'x'
> > > > configured
> > > > > seconds to all its connected clients and if it doesn't
> > get a
> > > reply
> > > > from
> > > > > the client within some time, it disconnects the
> > connection.
> > > > >
> > > > > The inactivity probe from the server side can be
> > configured. Run
> > > > "ovn-
> > > > > nbctl list connection"
> > > > > and you will see inactivity_probe column. You can set
> > this column
> > > > to
> > > > > desired value like - ovn-nbctl set connection .
> > > > inactivity_probe=30000
> > > > > (for 30 seconds)
> > > > >
> > > > > The same thing for SB ovsdb-server.
> > > > >
> > > > > Similarly each client (ovn-northd, ovn-controller,
> > neutron server)
> > > > sends
> > > > > inactivity probe every 'y' seconds and if the client
> > doesn't get
> > > > any
> > > > > reply from ovsdb-server it will disconnect the
connection
> > and
> > > > reconnect
> > > > > again.
> > > > >
> > > > > For ovn-northd you can configured this as - ovn-nbctl
set
> > > NB_Global
> > > > .
> > > > > options:northd_probe_interval=30000
> > > > >
> > > > > For ovn-controllers - ovs-vsctl set open .
> > external_ids:ovn-
> > > > remote-
> > > > > probe-interval=30000
> > > > >
> > > > > There is also a probe interval for openflow connection
> > from ovn-
> > > > > controller to ovs-vswitchd which you can configure as
> > ovs-vsctl
> > > set
> > > > > open . external_ids:ovn-openflow-probe-interval=30 (this
> > is in
> > > > seconds)
> > > > >
> > > > >
> > > > > Regarding the neutron server I think it is set to 60
> > seconds.
> > > > Please see
> > > > > this -
> > > > >
> > > >
> > >
> > https://github.com/openstack/neutron/blob/master/neutron/conf/plugins/ml
> > > > > 2/drivers/ovn/ovn_conf.py#L80
> > > > >
> > > > > From the logs you shared, it looks like ovsdb-server is
> > not
> > > getting
> > > > the
> > > > > probe reply from neutron server after 5 seconds and
hence
> > it is
> > > > > disconnecting. Not sure what's happening though.
> > > > >
> > > > > You can try increasing the inactivity probe interval on
> > the
> > > > ovsdb-server
> > > > > side with the first command I shared.
> > > > > Note: If "ovn-nbctl list connection" returns empty, you
> > need to
> > > > create a
> > > > > connection row like - ovn-nbctl set-connection
> > ptcp:6641:<IP>
> > > > >
> > > > >
> > > > > Thanks
> > > > > Numan
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > From OVN ML2 driver log, I see it connected to
> > the leader,
> > > > then
> > > > > the
> > > > > > connection was closed by leader after 5 or 6
> > seconds. Is
> > > > this
> > > > > probe one-
> > > > > > way or two-ways?
> > > > > > Both sides are not busy, not taking much CPU
> > cycles. Not
> > > > sure how
> > > > > this
> > > > > > could happen. Any thoughts?
> > > > > >
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Tony
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > dev mailing list
> > > > > > dev at openvswitch.org <mailto:dev at openvswitch.org>
> > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > >
> > > > > >
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > > > _______________________________________________
> > > > > discuss mailing list
> > > > > discuss at openvswitch.org
> > <mailto:discuss at openvswitch.org> <mailto:discuss at openvswitch.org
> > <mailto:discuss at openvswitch.org> >
> > > > <mailto:discuss at openvswitch.org <mailto:
discuss at openvswitch.org>
> > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> > >
> > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-
> > discuss
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > _______________________________________________
> > > discuss mailing list
> > > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200806/9c45221d/attachment-0001.html>
More information about the discuss
mailing list