[ovs-discuss] [OVN] no response to inactivity probe

Tony Liu tonyliu0592 at hotmail.com
Wed Aug 5 23:28:35 UTC 2020


Hi Han,

After setting connection target="ptcp:6642:0.0.0.0" for ovn-sb-db,
I see this error.
========
2020-08-05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.0: listen failed: Address already in use
========
Anything I am missing here?


Thanks!

Tony
> -----Original Message-----
> From: Han Zhou <hzhou at ovn.org>
> Sent: Tuesday, August 4, 2020 4:44 PM
> To: Tony Liu <tonyliu0592 at hotmail.com>
> Cc: Numan Siddique <numans at ovn.org>; Han Zhou <hzhou at ovn.org>; ovs-
> discuss <ovs-discuss at openvswitch.org>; ovs-dev <ovs-dev at openvswitch.org>
> Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> 
> 
> 
> On Tue, Aug 4, 2020 at 2:50 PM Tony Liu <tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > wrote:
> 
> 
> 	Hi,
> 
> 	Since I have 3 OVN DB nodes, should I add 3 rows in connection
> table
> 	for the inactivity_probe? Or put 3 addresses into one row?
> 
> 	"set-connection" set one row only, and there is no "add-connection".
> 	How should I add 3 rows into the table connection?
> 
> 
> 
> 
> You only need to set one row. Try this command:
> 
> ovn-nbctl -- --id=@conn_uuid create Connection
> target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 -- set NB_Global .
> connections=@conn_uuid
> 
> 
> 
> 	Thanks!
> 
> 	Tony
> 
> 	> -----Original Message-----
> 	> From: Numan Siddique <numans at ovn.org <mailto:numans at ovn.org> >
> 	> Sent: Tuesday, August 4, 2020 12:36 AM
> 	> To: Tony Liu <tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> >
> 	> Cc: ovs-discuss <ovs-discuss at openvswitch.org <mailto:ovs-
> discuss at openvswitch.org> >; ovs-dev <ovs-
> 	> dev at openvswitch.org <mailto:dev at openvswitch.org> >
> 	> Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> 	>
> 	>
> 	>
> 	> On Tue, Aug 4, 2020 at 9:12 AM Tony Liu <tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com>
> 	> <mailto:tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > > wrote:
> 	>
> 	>
> 	>       In my deployment, on each Neutron server, there are 13
> Neutron
> 	> server processes.
> 	>       I see 12 of them (monitor, maintenance, RPC, API) connect
> to both
> 	> ovn-nb-db
> 	>       and ovn-sb-db. With 3 Neutron server nodes, that's 36 OVSDB
> clients.
> 	>       Is so many clients OK?
> 	>
> 	>       Any suggestions how to figure out which side doesn't
> respond the
> 	> probe,
> 	>       if it's bi-directional? I don't see any activities from
> logging,
> 	> other than
> 	>       connect/drop and reconnect...
> 	>
> 	>       BTW, please let me know if this is not the right place to
> discuss
> 	> Neutron OVN
> 	>       ML2 driver.
> 	>
> 	>
> 	>       Thanks!
> 	>
> 	>       Tony
> 	>
> 	>       > -----Original Message-----
> 	>       > From: dev <ovs-dev-bounces at openvswitch.org <mailto:ovs-
> dev-bounces at openvswitch.org>  <mailto:ovs-dev- <mailto:ovs-dev->
> 	> bounces at openvswitch.org <mailto:bounces at openvswitch.org> > > On
> Behalf Of Tony Liu
> 	>       > Sent: Monday, August 3, 2020 7:45 PM
> 	>       > To: ovs-discuss <ovs-discuss at openvswitch.org <mailto:ovs-
> discuss at openvswitch.org>  <mailto:ovs- <mailto:ovs->
> 	> discuss at openvswitch.org <mailto:discuss at openvswitch.org> > >;
> ovs-dev <ovs-
> 	>       > dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > >
> 	>       > Subject: [ovs-dev] [OVN] no response to inactivity probe
> 	>       >
> 	>       > Hi,
> 	>       >
> 	>       > Neutron OVN ML2 driver was disconnected by ovn-nb-db.
> There are
> 	> many
> 	>       > error messages from ovn-nb-db leader.
> 	>       > ========
> 	>       > 2020-08-
> 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620
> <http://10.6.20.81:58620>
> 	> <http://10.6.20.81:58620> : no
> 	>       > response to inactivity probe after 5 seconds,
> disconnecting
> 	>       > 2020-08-
> 04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300
> <http://10.6.20.81:58300>
> 	> <http://10.6.20.81:58300> : no
> 	>       > response to inactivity probe after 5 seconds,
> disconnecting
> 	>       > 2020-08-
> 04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582
> <http://10.6.20.81:59582>
> 	> <http://10.6.20.81:59582> : no
> 	>       > response to inactivity probe after 5 seconds,
> disconnecting
> 	>       > 2020-08-
> 04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626
> <http://10.6.20.83:42626>
> 	> <http://10.6.20.83:42626> : no
> 	>       > response to inactivity probe after 5 seconds,
> disconnecting
> 	>       > 2020-08-
> 04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412
> <http://10.6.20.82:45412>
> 	> <http://10.6.20.82:45412> : no
> 	>       > response to inactivity probe after 5 seconds,
> disconnecting
> 	>       > 2020-08-
> 04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416
> <http://10.6.20.81:59416>
> 	> <http://10.6.20.81:59416> : no
> 	>       > response to inactivity probe after 5 seconds,
> disconnecting
> 	>       > 2020-08-
> 04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004
> <http://10.6.20.81:60004>
> 	> <http://10.6.20.81:60004> : no
> 	>       > response to inactivity probe after 5 seconds,
> disconnecting
> 	> ========
> 	>       >
> 	>       > Could anyone share a bit details how this inactivity
> probe works?
> 	>
> 	>
> 	>
> 	> The inactivity probe is sent by both the server and clients
> 	> independently.
> 	> Meaning ovsdb-server will send an inactivity probe every 'x'
> configured
> 	> seconds to all its connected clients and if it doesn't get a
> reply from
> 	> the client within some time, it disconnects the connection.
> 	>
> 	> The inactivity probe from the server side can be configured. Run
> "ovn-
> 	> nbctl list connection"
> 	> and you will see inactivity_probe column. You can set this column
> to
> 	> desired value like - ovn-nbctl set connection .
> inactivity_probe=30000
> 	> (for 30 seconds)
> 	>
> 	> The same thing for SB ovsdb-server.
> 	>
> 	> Similarly each client (ovn-northd, ovn-controller, neutron server)
> sends
> 	> inactivity probe every 'y' seconds and if the client doesn't get
> any
> 	> reply from ovsdb-server it will disconnect the connection and
> reconnect
> 	> again.
> 	>
> 	> For ovn-northd you can configured this as - ovn-nbctl set
> NB_Global .
> 	> options:northd_probe_interval=30000
> 	>
> 	> For ovn-controllers - ovs-vsctl set open . external_ids:ovn-
> remote-
> 	> probe-interval=30000
> 	>
> 	> There is also a probe interval for openflow connection from ovn-
> 	> controller to ovs-vswitchd which you can configure as ovs-vsctl
> set
> 	> open . external_ids:ovn-openflow-probe-interval=30 (this is in
> seconds)
> 	>
> 	>
> 	> Regarding the neutron server I think it is set to 60 seconds.
> Please see
> 	> this -
> 	>
> https://github.com/openstack/neutron/blob/master/neutron/conf/plugins/ml
> 	> 2/drivers/ovn/ovn_conf.py#L80
> 	>
> 	> From the logs you shared, it looks like ovsdb-server is not
> getting the
> 	> probe reply from neutron server after 5 seconds and hence it is
> 	> disconnecting. Not sure what's happening though.
> 	>
> 	> You can try increasing the inactivity probe interval on the
> ovsdb-server
> 	> side with the first command I shared.
> 	> Note: If "ovn-nbctl list connection" returns empty, you need to
> create a
> 	> connection row like - ovn-nbctl set-connection ptcp:6641:<IP>
> 	>
> 	>
> 	> Thanks
> 	> Numan
> 	>
> 	>
> 	>
> 	>
> 	>       > From OVN ML2 driver log, I see it connected to the leader,
> then
> 	> the
> 	>       > connection was closed by leader after 5 or 6 seconds. Is
> this
> 	> probe one-
> 	>       > way or two-ways?
> 	>       > Both sides are not busy, not taking much CPU cycles. Not
> sure how
> 	> this
> 	>       > could happen. Any thoughts?
> 	>       >
> 	>       >
> 	>       > Thanks!
> 	>       >
> 	>       > Tony
> 	>       >
> 	>       >
> 	>       >
> 	>       > _______________________________________________
> 	>       > dev mailing list
> 	>       > dev at openvswitch.org <mailto:dev at openvswitch.org>
> <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> 	>       > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> 	>       _______________________________________________
> 	>       discuss mailing list
> 	>       discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org> >
> 	>       https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> 	>
> 	>
> 
> 



More information about the discuss mailing list