[ovs-discuss] [OVN] no response to inactivity probe
Han Zhou
hzhou at ovn.org
Thu Aug 6 06:48:25 UTC 2020
On Wed, Aug 5, 2020 at 9:14 PM Tony Liu <tonyliu0592 at hotmail.com> wrote:
> I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb-db
> and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first node
> of cluster. Also ovn-openflow-probe-interval=30 on compute node.
> It seems helping. Not that many connect/drop/reconnect in logging.
> That "commit failure" is also gone.
> The issue I reported in another thread "packet drop" seems gone.
> And launching VM starts working.
>
> How should I set connection table for all ovn-nb-db and ovn-sb-db
> nodes in the cluster to set inactivity_probe?
> One row with address 0.0.0.0 seems not working.
>
You can simply use 0.0.0.0 in the connection table, but don't specify the
same connection method on the command line when starting ovsdb-server for
NB/SB DB. Otherwise, these are conflicting and that's why you saw "Address
already in use" error.
> Is "external_ids:ovn-remote-probe-interval" in ovsdb-server on
> compute node for ovn-controller to probe ovn-sb-db?
>
> OVSDB probe is bidirectional, so you need to set this value, too, if you
don't want too many probes handled by the SB server. (setting the
connection table for SB only changes the server side).
> Is "external_ids:ovn-openflow-probe-interval" in ovsdb-server on
> compute node for ovn-controller to probe ovsdb-server?
>
> It is for the OpenFlow connection between ovn-controller and ovs-vswitchd,
which is part of the OpenFlow protocol.
> What's probe interval for ovsdb-server to probe ovn-controller?
>
> The local ovsdb connection uses unix socket, which doesn't send probe by
default (if I remember correctly).
For ovn-controller, since it is implemented with incremental-processing,
even if there are probes from openflow or local ovsdb, it doesn't matter.
If there is no configuration change, ovn-controller simply replies the
probe and there is no extra cost.
> Thanks!
>
> Tony
> > -----Original Message-----
> > From: discuss <ovs-discuss-bounces at openvswitch.org> On Behalf Of Tony
> > Liu
> > Sent: Wednesday, August 5, 2020 4:29 PM
> > To: Han Zhou <hzhou at ovn.org>
> > Cc: ovs-dev <ovs-dev at openvswitch.org>; ovs-discuss <ovs-
> > discuss at openvswitch.org>
> > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> >
> > Hi Han,
> >
> > After setting connection target="ptcp:6642:0.0.0.0" for ovn-sb-db, I see
> > this error.
> > ========
> > 2020-08-
> > 05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.0:
> > listen failed: Address already in use ======== Anything I am missing
> > here?
> >
> >
> > Thanks!
> >
> > Tony
> > > -----Original Message-----
> > > From: Han Zhou <hzhou at ovn.org>
> > > Sent: Tuesday, August 4, 2020 4:44 PM
> > > To: Tony Liu <tonyliu0592 at hotmail.com>
> > > Cc: Numan Siddique <numans at ovn.org>; Han Zhou <hzhou at ovn.org>; ovs-
> > > discuss <ovs-discuss at openvswitch.org>; ovs-dev
> > > <ovs-dev at openvswitch.org>
> > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> > >
> > >
> > >
> > > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu <tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> > wrote:
> > >
> > >
> > > Hi,
> > >
> > > Since I have 3 OVN DB nodes, should I add 3 rows in connection
> > table
> > > for the inactivity_probe? Or put 3 addresses into one row?
> > >
> > > "set-connection" set one row only, and there is no
> "add-connection".
> > > How should I add 3 rows into the table connection?
> > >
> > >
> > >
> > >
> > > You only need to set one row. Try this command:
> > >
> > > ovn-nbctl -- --id=@conn_uuid create Connection
> > > target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set NB_Global .
> > > connections=@conn_uuid
> > >
> > >
> > >
> > > Thanks!
> > >
> > > Tony
> > >
> > > > -----Original Message-----
> > > > From: Numan Siddique <numans at ovn.org <mailto:numans at ovn.org> >
> > > > Sent: Tuesday, August 4, 2020 12:36 AM
> > > > To: Tony Liu <tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> >
> > > > Cc: ovs-discuss <ovs-discuss at openvswitch.org <mailto:ovs-
> > > discuss at openvswitch.org> >; ovs-dev <ovs-
> > > > dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> > > >
> > > >
> > > >
> > > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu <tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com>
> > > > <mailto:tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> > > wrote:
> > > >
> > > >
> > > > In my deployment, on each Neutron server, there are 13
> > > Neutron
> > > > server processes.
> > > > I see 12 of them (monitor, maintenance, RPC, API) connect
> > > to both
> > > > ovn-nb-db
> > > > and ovn-sb-db. With 3 Neutron server nodes, that's 36 OVSDB
> > > clients.
> > > > Is so many clients OK?
> > > >
> > > > Any suggestions how to figure out which side doesn't
> > > respond the
> > > > probe,
> > > > if it's bi-directional? I don't see any activities from
> > > logging,
> > > > other than
> > > > connect/drop and reconnect...
> > > >
> > > > BTW, please let me know if this is not the right place to
> > > discuss
> > > > Neutron OVN
> > > > ML2 driver.
> > > >
> > > >
> > > > Thanks!
> > > >
> > > > Tony
> > > >
> > > > > -----Original Message-----
> > > > > From: dev <ovs-dev-bounces at openvswitch.org <mailto:ovs-
> > > dev-bounces at openvswitch.org> <mailto:ovs-dev- <mailto:ovs-dev->
> > > > bounces at openvswitch.org <mailto:bounces at openvswitch.org> > > On
> > > Behalf Of Tony Liu
> > > > > Sent: Monday, August 3, 2020 7:45 PM
> > > > > To: ovs-discuss <ovs-discuss at openvswitch.org <mailto:
> ovs-
> > > discuss at openvswitch.org> <mailto:ovs- <mailto:ovs->
> > > > discuss at openvswitch.org <mailto:discuss at openvswitch.org> > >;
> > > ovs-dev <ovs-
> > > > > dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > >
> > > > > Subject: [ovs-dev] [OVN] no response to inactivity probe
> > > > >
> > > > > Hi,
> > > > >
> > > > > Neutron OVN ML2 driver was disconnected by ovn-nb-db.
> > > There are
> > > > many
> > > > > error messages from ovn-nb-db leader.
> > > > > ========
> > > > > 2020-08-
> > > 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620
> > > <http://10.6.20.81:58620>
> > > > <http://10.6.20.81:58620> : no
> > > > > response to inactivity probe after 5 seconds,
> > > disconnecting
> > > > > 2020-08-
> > > 04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300
> > > <http://10.6.20.81:58300>
> > > > <http://10.6.20.81:58300> : no
> > > > > response to inactivity probe after 5 seconds,
> > > disconnecting
> > > > > 2020-08-
> > > 04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582
> > > <http://10.6.20.81:59582>
> > > > <http://10.6.20.81:59582> : no
> > > > > response to inactivity probe after 5 seconds,
> > > disconnecting
> > > > > 2020-08-
> > > 04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626
> > > <http://10.6.20.83:42626>
> > > > <http://10.6.20.83:42626> : no
> > > > > response to inactivity probe after 5 seconds,
> > > disconnecting
> > > > > 2020-08-
> > > 04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412
> > > <http://10.6.20.82:45412>
> > > > <http://10.6.20.82:45412> : no
> > > > > response to inactivity probe after 5 seconds,
> > > disconnecting
> > > > > 2020-08-
> > > 04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416
> > > <http://10.6.20.81:59416>
> > > > <http://10.6.20.81:59416> : no
> > > > > response to inactivity probe after 5 seconds,
> > > disconnecting
> > > > > 2020-08-
> > > 04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004
> > > <http://10.6.20.81:60004>
> > > > <http://10.6.20.81:60004> : no
> > > > > response to inactivity probe after 5 seconds,
> > > disconnecting
> > > > ========
> > > > >
> > > > > Could anyone share a bit details how this inactivity
> > > probe works?
> > > >
> > > >
> > > >
> > > > The inactivity probe is sent by both the server and clients
> > > > independently.
> > > > Meaning ovsdb-server will send an inactivity probe every 'x'
> > > configured
> > > > seconds to all its connected clients and if it doesn't get a
> > reply
> > > from
> > > > the client within some time, it disconnects the connection.
> > > >
> > > > The inactivity probe from the server side can be configured. Run
> > > "ovn-
> > > > nbctl list connection"
> > > > and you will see inactivity_probe column. You can set this column
> > > to
> > > > desired value like - ovn-nbctl set connection .
> > > inactivity_probe=30000
> > > > (for 30 seconds)
> > > >
> > > > The same thing for SB ovsdb-server.
> > > >
> > > > Similarly each client (ovn-northd, ovn-controller, neutron
> server)
> > > sends
> > > > inactivity probe every 'y' seconds and if the client doesn't get
> > > any
> > > > reply from ovsdb-server it will disconnect the connection and
> > > reconnect
> > > > again.
> > > >
> > > > For ovn-northd you can configured this as - ovn-nbctl set
> > NB_Global
> > > .
> > > > options:northd_probe_interval=30000
> > > >
> > > > For ovn-controllers - ovs-vsctl set open . external_ids:ovn-
> > > remote-
> > > > probe-interval=30000
> > > >
> > > > There is also a probe interval for openflow connection from ovn-
> > > > controller to ovs-vswitchd which you can configure as ovs-vsctl
> > set
> > > > open . external_ids:ovn-openflow-probe-interval=30 (this is in
> > > seconds)
> > > >
> > > >
> > > > Regarding the neutron server I think it is set to 60 seconds.
> > > Please see
> > > > this -
> > > >
> > >
> > https://github.com/openstack/neutron/blob/master/neutron/conf/plugins/ml
> > > > 2/drivers/ovn/ovn_conf.py#L80
> > > >
> > > > From the logs you shared, it looks like ovsdb-server is not
> > getting
> > > the
> > > > probe reply from neutron server after 5 seconds and hence it is
> > > > disconnecting. Not sure what's happening though.
> > > >
> > > > You can try increasing the inactivity probe interval on the
> > > ovsdb-server
> > > > side with the first command I shared.
> > > > Note: If "ovn-nbctl list connection" returns empty, you need to
> > > create a
> > > > connection row like - ovn-nbctl set-connection ptcp:6641:<IP>
> > > >
> > > >
> > > > Thanks
> > > > Numan
> > > >
> > > >
> > > >
> > > >
> > > > > From OVN ML2 driver log, I see it connected to the
> leader,
> > > then
> > > > the
> > > > > connection was closed by leader after 5 or 6 seconds. Is
> > > this
> > > > probe one-
> > > > > way or two-ways?
> > > > > Both sides are not busy, not taking much CPU cycles. Not
> > > sure how
> > > > this
> > > > > could happen. Any thoughts?
> > > > >
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Tony
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > dev mailing list
> > > > > dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > > _______________________________________________
> > > > discuss mailing list
> > > > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> >
> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> > > >
> > > >
> > >
> > >
> >
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200805/dff89b27/attachment-0001.html>
More information about the discuss
mailing list