[ovs-dev] [ovs-discuss] [OVN] no response to inactivity probe
Tony Liu
tonyliu0592 at hotmail.com
Thu Aug 6 04:14:35 UTC 2020
I set the connection target="ptcp:6641:10.6.20.84" for ovn-nb-db
and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the first node
of cluster. Also ovn-openflow-probe-interval=30 on compute node.
It seems helping. Not that many connect/drop/reconnect in logging.
That "commit failure" is also gone.
The issue I reported in another thread "packet drop" seems gone.
And launching VM starts working.
How should I set connection table for all ovn-nb-db and ovn-sb-db
nodes in the cluster to set inactivity_probe?
One row with address 0.0.0.0 seems not working.
Is "external_ids:ovn-remote-probe-interval" in ovsdb-server on
compute node for ovn-controller to probe ovn-sb-db?
Is "external_ids:ovn-openflow-probe-interval" in ovsdb-server on
compute node for ovn-controller to probe ovsdb-server?
What's probe interval for ovsdb-server to probe ovn-controller?
Thanks!
Tony
> -----Original Message-----
> From: discuss <ovs-discuss-bounces at openvswitch.org> On Behalf Of Tony
> Liu
> Sent: Wednesday, August 5, 2020 4:29 PM
> To: Han Zhou <hzhou at ovn.org>
> Cc: ovs-dev <ovs-dev at openvswitch.org>; ovs-discuss <ovs-
> discuss at openvswitch.org>
> Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
>
> Hi Han,
>
> After setting connection target="ptcp:6642:0.0.0.0" for ovn-sb-db, I see
> this error.
> ========
> 2020-08-
> 05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.0:
> listen failed: Address already in use ======== Anything I am missing
> here?
>
>
> Thanks!
>
> Tony
> > -----Original Message-----
> > From: Han Zhou <hzhou at ovn.org>
> > Sent: Tuesday, August 4, 2020 4:44 PM
> > To: Tony Liu <tonyliu0592 at hotmail.com>
> > Cc: Numan Siddique <numans at ovn.org>; Han Zhou <hzhou at ovn.org>; ovs-
> > discuss <ovs-discuss at openvswitch.org>; ovs-dev
> > <ovs-dev at openvswitch.org>
> > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> >
> >
> >
> > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu <tonyliu0592 at hotmail.com
> > <mailto:tonyliu0592 at hotmail.com> > wrote:
> >
> >
> > Hi,
> >
> > Since I have 3 OVN DB nodes, should I add 3 rows in connection
> table
> > for the inactivity_probe? Or put 3 addresses into one row?
> >
> > "set-connection" set one row only, and there is no "add-connection".
> > How should I add 3 rows into the table connection?
> >
> >
> >
> >
> > You only need to set one row. Try this command:
> >
> > ovn-nbctl -- --id=@conn_uuid create Connection
> > target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set NB_Global .
> > connections=@conn_uuid
> >
> >
> >
> > Thanks!
> >
> > Tony
> >
> > > -----Original Message-----
> > > From: Numan Siddique <numans at ovn.org <mailto:numans at ovn.org> >
> > > Sent: Tuesday, August 4, 2020 12:36 AM
> > > To: Tony Liu <tonyliu0592 at hotmail.com
> > <mailto:tonyliu0592 at hotmail.com> >
> > > Cc: ovs-discuss <ovs-discuss at openvswitch.org <mailto:ovs-
> > discuss at openvswitch.org> >; ovs-dev <ovs-
> > > dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> > >
> > >
> > >
> > > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu <tonyliu0592 at hotmail.com
> > <mailto:tonyliu0592 at hotmail.com>
> > > <mailto:tonyliu0592 at hotmail.com
> > <mailto:tonyliu0592 at hotmail.com> > > wrote:
> > >
> > >
> > > In my deployment, on each Neutron server, there are 13
> > Neutron
> > > server processes.
> > > I see 12 of them (monitor, maintenance, RPC, API) connect
> > to both
> > > ovn-nb-db
> > > and ovn-sb-db. With 3 Neutron server nodes, that's 36 OVSDB
> > clients.
> > > Is so many clients OK?
> > >
> > > Any suggestions how to figure out which side doesn't
> > respond the
> > > probe,
> > > if it's bi-directional? I don't see any activities from
> > logging,
> > > other than
> > > connect/drop and reconnect...
> > >
> > > BTW, please let me know if this is not the right place to
> > discuss
> > > Neutron OVN
> > > ML2 driver.
> > >
> > >
> > > Thanks!
> > >
> > > Tony
> > >
> > > > -----Original Message-----
> > > > From: dev <ovs-dev-bounces at openvswitch.org <mailto:ovs-
> > dev-bounces at openvswitch.org> <mailto:ovs-dev- <mailto:ovs-dev->
> > > bounces at openvswitch.org <mailto:bounces at openvswitch.org> > > On
> > Behalf Of Tony Liu
> > > > Sent: Monday, August 3, 2020 7:45 PM
> > > > To: ovs-discuss <ovs-discuss at openvswitch.org <mailto:ovs-
> > discuss at openvswitch.org> <mailto:ovs- <mailto:ovs->
> > > discuss at openvswitch.org <mailto:discuss at openvswitch.org> > >;
> > ovs-dev <ovs-
> > > > dev at openvswitch.org <mailto:dev at openvswitch.org>
> > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > >
> > > > Subject: [ovs-dev] [OVN] no response to inactivity probe
> > > >
> > > > Hi,
> > > >
> > > > Neutron OVN ML2 driver was disconnected by ovn-nb-db.
> > There are
> > > many
> > > > error messages from ovn-nb-db leader.
> > > > ========
> > > > 2020-08-
> > 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620
> > <http://10.6.20.81:58620>
> > > <http://10.6.20.81:58620> : no
> > > > response to inactivity probe after 5 seconds,
> > disconnecting
> > > > 2020-08-
> > 04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300
> > <http://10.6.20.81:58300>
> > > <http://10.6.20.81:58300> : no
> > > > response to inactivity probe after 5 seconds,
> > disconnecting
> > > > 2020-08-
> > 04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582
> > <http://10.6.20.81:59582>
> > > <http://10.6.20.81:59582> : no
> > > > response to inactivity probe after 5 seconds,
> > disconnecting
> > > > 2020-08-
> > 04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626
> > <http://10.6.20.83:42626>
> > > <http://10.6.20.83:42626> : no
> > > > response to inactivity probe after 5 seconds,
> > disconnecting
> > > > 2020-08-
> > 04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412
> > <http://10.6.20.82:45412>
> > > <http://10.6.20.82:45412> : no
> > > > response to inactivity probe after 5 seconds,
> > disconnecting
> > > > 2020-08-
> > 04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416
> > <http://10.6.20.81:59416>
> > > <http://10.6.20.81:59416> : no
> > > > response to inactivity probe after 5 seconds,
> > disconnecting
> > > > 2020-08-
> > 04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004
> > <http://10.6.20.81:60004>
> > > <http://10.6.20.81:60004> : no
> > > > response to inactivity probe after 5 seconds,
> > disconnecting
> > > ========
> > > >
> > > > Could anyone share a bit details how this inactivity
> > probe works?
> > >
> > >
> > >
> > > The inactivity probe is sent by both the server and clients
> > > independently.
> > > Meaning ovsdb-server will send an inactivity probe every 'x'
> > configured
> > > seconds to all its connected clients and if it doesn't get a
> reply
> > from
> > > the client within some time, it disconnects the connection.
> > >
> > > The inactivity probe from the server side can be configured. Run
> > "ovn-
> > > nbctl list connection"
> > > and you will see inactivity_probe column. You can set this column
> > to
> > > desired value like - ovn-nbctl set connection .
> > inactivity_probe=30000
> > > (for 30 seconds)
> > >
> > > The same thing for SB ovsdb-server.
> > >
> > > Similarly each client (ovn-northd, ovn-controller, neutron server)
> > sends
> > > inactivity probe every 'y' seconds and if the client doesn't get
> > any
> > > reply from ovsdb-server it will disconnect the connection and
> > reconnect
> > > again.
> > >
> > > For ovn-northd you can configured this as - ovn-nbctl set
> NB_Global
> > .
> > > options:northd_probe_interval=30000
> > >
> > > For ovn-controllers - ovs-vsctl set open . external_ids:ovn-
> > remote-
> > > probe-interval=30000
> > >
> > > There is also a probe interval for openflow connection from ovn-
> > > controller to ovs-vswitchd which you can configure as ovs-vsctl
> set
> > > open . external_ids:ovn-openflow-probe-interval=30 (this is in
> > seconds)
> > >
> > >
> > > Regarding the neutron server I think it is set to 60 seconds.
> > Please see
> > > this -
> > >
> >
> https://github.com/openstack/neutron/blob/master/neutron/conf/plugins/ml
> > > 2/drivers/ovn/ovn_conf.py#L80
> > >
> > > From the logs you shared, it looks like ovsdb-server is not
> getting
> > the
> > > probe reply from neutron server after 5 seconds and hence it is
> > > disconnecting. Not sure what's happening though.
> > >
> > > You can try increasing the inactivity probe interval on the
> > ovsdb-server
> > > side with the first command I shared.
> > > Note: If "ovn-nbctl list connection" returns empty, you need to
> > create a
> > > connection row like - ovn-nbctl set-connection ptcp:6641:<IP>
> > >
> > >
> > > Thanks
> > > Numan
> > >
> > >
> > >
> > >
> > > > From OVN ML2 driver log, I see it connected to the leader,
> > then
> > > the
> > > > connection was closed by leader after 5 or 6 seconds. Is
> > this
> > > probe one-
> > > > way or two-ways?
> > > > Both sides are not busy, not taking much CPU cycles. Not
> > sure how
> > > this
> > > > could happen. Any thoughts?
> > > >
> > > >
> > > > Thanks!
> > > >
> > > > Tony
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > dev mailing list
> > > > dev at openvswitch.org <mailto:dev at openvswitch.org>
> > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > _______________________________________________
> > > discuss mailing list
> > > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> >
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> > >
> > >
> >
> >
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
More information about the dev
mailing list