[ovs-dev] [ovs-discuss] [OVN] no response to inactivity probe

Tony Liu tonyliu0592 at hotmail.com
Fri Aug 7 04:00:36 UTC 2020


Hi,

There are still some connection errors from ovn-controller.
Is that connection drop will cause flows to be deleted from vswitchd?
================
......
2020-08-07T03:55:22.269Z|03988|jsonrpc|WARN|tcp:127.0.0.1:6640: send error: Broken pipe
......
2020-08-07T03:55:31.551Z|03996|reconnect|WARN|tcp:127.0.0.1:6640: connection dropped (Broken pipe)
================

================
2020-08-07T03:55:22.268Z|03986|poll_loop|INFO|wakeup due to [POLLIN] on fd 14 (127.0.0.1:49514<->127.0.0.1:6640) at lib/stream-fd.c:157 (99% CPU usage)
2020-08-07T03:55:22.268Z|03987|poll_loop|INFO|wakeup due to [POLLIN] on fd 19 (10.6.20.91:42854<->10.6.20.84:6642) at lib/stream-fd.c:157 (99% CPU usage)
2020-08-07T03:55:22.269Z|03988|jsonrpc|WARN|tcp:127.0.0.1:6640: send error: Broken pipe
2020-08-07T03:55:31.549Z|03989|timeval|WARN|Unreasonably long 9280ms poll interval (9220ms user, 1ms system)
2020-08-07T03:55:31.550Z|03990|timeval|WARN|disk: 0 reads, 8 writes
2020-08-07T03:55:31.550Z|03991|timeval|WARN|context switches: 0 voluntary, 5 involuntary
2020-08-07T03:55:31.550Z|03992|coverage|INFO|Dropped 4 log messages in last 47 seconds (most recently, 9 seconds ago) due to excessive rate
2020-08-07T03:55:31.551Z|03993|coverage|INFO|Skipping details of duplicate event coverage for hash=824dd6ab
2020-08-07T03:55:31.551Z|03994|poll_loop|INFO|wakeup due to [POLLIN] on fd 20 (<->/var/run/openvswitch/br-int.mgmt) at lib/stream-fd.c:157 (100% CPU usage)
2020-08-07T03:55:31.551Z|03995|poll_loop|INFO|wakeup due to [POLLIN] on fd 19 (10.6.20.91:42854<->10.6.20.84:6642) at lib/stream-fd.c:157 (100% CPU usage)
2020-08-07T03:55:31.551Z|03996|reconnect|WARN|tcp:127.0.0.1:6640: connection dropped (Broken pipe)
2020-08-07T03:55:31.552Z|03997|poll_loop|INFO|wakeup due to 0-ms timeout at controller/ovn-controller.c:2123 (100% CPU usage)
2020-08-07T03:55:40.752Z|03998|timeval|WARN|Unreasonably long 9176ms poll interval (9118ms user, 0ms system)
2020-08-07T03:55:40.752Z|03999|timeval|WARN|context switches: 0 voluntary, 7 involuntary
2020-08-07T03:55:40.753Z|04000|poll_loop|INFO|Dropped 2 log messages in last 10 seconds (most recently, 10 seconds ago) due to excessive rate
2020-08-07T03:55:40.753Z|04001|poll_loop|INFO|wakeup due to 0-ms timeout at lib/reconnect.c:643 (99% CPU usage)
2020-08-07T03:55:40.754Z|04002|reconnect|INFO|tcp:127.0.0.1:6640: connecting...
2020-08-07T03:55:40.771Z|04003|reconnect|INFO|tcp:127.0.0.1:6640: connected
================

Thanks!

Tony
> -----Original Message-----
> From: discuss <ovs-discuss-bounces at openvswitch.org> On Behalf Of Tony
> Liu
> Sent: Thursday, August 6, 2020 8:23 PM
> To: Han Zhou <hzhou at ovn.org>; Numan Siddique <numans at ovn.org>
> Cc: ovs-dev <ovs-dev at openvswitch.org>; ovs-discuss <ovs-
> discuss at openvswitch.org>
> Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> 
> Interesting...
> 
> with this configuration on gateway (chassis) node, ========
> external_ids        : {ovn-bridge-mappings="physnet1:br-ex", ovn-cms-
> options=enable-chassis-as-gw, ovn-encap-ip="10.6.30.91", ovn-encap-
> type=geneve, ovn-openflow-probe-interval="30", ovn-
> remote="tcp:10.6.20.84:6642,tcp:10.6.20.85:6642,tcp:10.6.20.86:6642",
> ovn-remote-probe-interval="30000", system-id="gateway-1"}
> ========
> 
> I still see error from ovn-controller.
> ========
> 2020-08-07T03:17:48.186Z|02737|reconnect|ERR|tcp:127.0.0.1:6640: no
> response to inactivity probe after 8.74 seconds, disconnecting ========
> That tcp:127.0.0.1:6640 is the connection between ovn-controller and
> local ovsdb-server.
> 
> Any settings I missed?
> 
> 
> Thanks!
> 
> Tony
> > -----Original Message-----
> > From: dev <ovs-dev-bounces at openvswitch.org> On Behalf Of Tony Liu
> > Sent: Thursday, August 6, 2020 7:45 PM
> > To: Han Zhou <hzhou at ovn.org>; Numan Siddique <numans at ovn.org>
> > Cc: ovs-dev <ovs-dev at openvswitch.org>; ovs-discuss <ovs-
> > discuss at openvswitch.org>
> > Subject: Re: [ovs-dev] [ovs-discuss] [OVN] no response to inactivity
> > probe
> >
> > Hi Han and Numan,
> >
> > I'd like to have a few more clarifications.
> >
> > For inactivity probe:
> > From ovn-controller to ovn-sb-db: ovn-remote-probe-interval
> >
> > From ovn-controller to ovs-vswitchd: ovn-openflow-probe-interval
> >
> > From ovn-controller to local ovsdb: which interval?
> >
> > From local ovsdb to ovn-controller: which interval?
> >
> > From ovs-vswitchd to ovn-controller: which interval?
> >
> >
> > Regarding to the connection between ovn-controller and local ovsdb-
> > server, I recall that UNIX socket is lighter than TCP socket and UNIX
> > socket is recommended for local communication.
> > Is that right?
> >
> >
> > Thanks!
> >
> > Tony
> >
> > > -----Original Message-----
> > > From: Han Zhou <hzhou at ovn.org>
> > > Sent: Thursday, August 6, 2020 12:42 PM
> > > To: Tony Liu <tonyliu0592 at hotmail.com>
> > > Cc: Han Zhou <hzhou at ovn.org>; Numan Siddique <numans at ovn.org>;
> > > ovs-dev <ovs-dev at openvswitch.org>; ovs-discuss
> > > <ovs-discuss at openvswitch.org>
> > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> > >
> > >
> > >
> > > On Thu, Aug 6, 2020 at 12:07 PM Tony Liu <tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> > wrote:
> > > >
> > > > Inline...
> > > >
> > > > Thanks!
> > > >
> > > > Tony
> > > > > -----Original Message-----
> > > > > From: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org> >
> > > > > Sent: Thursday, August 6, 2020 11:37 AM
> > > > > To: Tony Liu <tonyliu0592 at hotmail.com
> > > > > <mailto:tonyliu0592 at hotmail.com> >
> > > > > Cc: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org> >; Numan
> > > > > Siddique <numans at ovn.org <mailto:numans at ovn.org> >; ovs-dev
> > > > > <ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org> >;
> > > > > ovs-discuss <ovs-discuss at openvswitch.org
> > > > > <mailto:ovs-discuss at openvswitch.org> >
> > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity probe
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Aug 6, 2020 at 11:11 AM Tony Liu
> > > > > <tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > <mailto:tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> > > wrote:
> > > > > >
> > > > > > Inline... (please read with monospaced font:))
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Tony
> > > > > > > -----Original Message-----
> > > > > > > From: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org>
> > > > > > > <mailto:hzhou at ovn.org <mailto:hzhou at ovn.org> > >
> > > > > > > Sent: Wednesday, August 5, 2020 11:48 PM
> > > > > > > To: Tony Liu <tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com> > >
> > > > > > > Cc: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org>
> > > > > > > <mailto:hzhou at ovn.org <mailto:hzhou at ovn.org> > >; Numan
> > > > > > > Siddique <numans at ovn.org <mailto:numans at ovn.org>
> > > > > > > <mailto:numans at ovn.org <mailto:numans at ovn.org> > >; ovs-dev
> > > > > > > <ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>
> > > > > > > <mailto:ovs-dev at openvswitch.org
> > > > > > > <mailto:ovs-dev at openvswitch.org>
> > > > > > > > >; ovs-discuss <ovs-discuss at openvswitch.org
> > > > > > > <mailto:ovs-discuss at openvswitch.org>
> > > > > > > <mailto:ovs-discuss at openvswitch.org
> > > > > > > <mailto:ovs-discuss at openvswitch.org> > >
> > > > > > > Subject: Re: [ovs-discuss] [OVN] no response to inactivity
> > > > > > > probe
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 5, 2020 at 9:14 PM Tony Liu
> > > > > > > <tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com> >
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > <mailto:tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> > > > wrote:
> > > > > > >
> > > > > > >
> > > > > > >       I set the connection target="ptcp:6641:10.6.20.84" for
> > > > > > > ovn-nb-
> > > > > db
> > > > > > >       and "ptcp:6642:10.6.20.84" for ovn-sb-db. .84 is the
> > > > > > > first
> > > > > node
> > > > > > >       of cluster. Also ovn-openflow-probe-interval=30 on
> > > > > > > compute
> > > > > node.
> > > > > > >       It seems helping. Not that many connect/drop/reconnect
> > > > > > > in
> > > > > logging.
> > > > > > >       That "commit failure" is also gone.
> > > > > > >       The issue I reported in another thread "packet drop"
> > > > > > > seems
> > > > > gone.
> > > > > > >       And launching VM starts working.
> > > > > > >
> > > > > > >       How should I set connection table for all ovn-nb-db
> > > > > > > and
> > > > > > > ovn-
> > > > > sb-db
> > > > > > >       nodes in the cluster to set inactivity_probe?
> > > > > > >       One row with address 0.0.0.0 seems not working.
> > > > > > >
> > > > > > > You can simply use 0.0.0.0 in the connection table, but
> > > > > > > don't specify the same connection method on the command line
> > > > > > > when starting
> > > > > > > ovsdb- server for NB/SB DB. Otherwise, these are conflicting
> > > > > > > and that's why you saw "Address already in use" error.
> > > > > >
> > > > > > Could you share a bit details how it works?
> > > > > > I thought the row in connection table only tells nbdb and sbdb
> > > > > > the probe interval. Isn't that right? Does nbdb and sbdb also
> > > > > > create socket based on target column?
> > > > >
> > > > > >
> > > > >
> > > > > In --remote option of ovsdb-server, you can specify either a
> > > > > connection method directly, or specify the db,table,column which
> > > > > contains the connection information.
> > > > > Please see manpage ovsdb-server(1).
> > > >
> > > > Here is how one of those 3 nbdb nodes invoked.
> > > > ========
> > > > ovsdb-server -vconsole:off -vfile:info
> > > > --log-file=/var/log/kolla/openvswitch/ovn-sb-db.log
> > > > --remote=punix:/var/run/ovn/ovnsb_db.sock --
> > > pidfile=/run/ovn/ovnsb_db.pid --unixctl=/var/run/ovn/ovnsb_db.ctl --
> > > remote=db:OVN_Southbound,SB_Global,connections --private-
> > > key=db:OVN_Southbound,SSL,private_key --
> > > certificate=db:OVN_Southbound,SSL,certificate --ca-
> > > cert=db:OVN_Southbound,SSL,ca_cert --ssl-
> > > protocols=db:OVN_Southbound,SSL,ssl_protocols --ssl-
> > > ciphers=db:OVN_Southbound,SSL,ssl_ciphers
> > > --remote=ptcp:6642:10.6.20.84 /var/lib/openvswitch/ovn-sb/ov sb.db
> > > ======== It creates UNIX and TCP sockets, and takes configuration
> > > from
> > DB.
> > > > Does that look ok?
> > > > Given that, what the target column should be for all nodes of the
> > > cluster?
> > > > And whatever target is set, ovsdb-server will create socket, right?
> > > > Oh... Should I do "--remote=ptcp:6642:0.0.0.0"? Then I can set the
> > > > same in connection table, and it won't cause conflict?
> > > > If --remote and connection target are the same, whoever comes in
> > > > later will be ignored, right?
> > > > In coding, does ovsdb-server create a connection object for each
> > > > of --remote and connection target, or it's one single connection
> > > > object for both of them because method:port:address is the same?
> > > > I'd expect the single object.
> > >
> > > >
> > >
> > > --remote=ptcp:6642:10.6.20.84 should be removed from the command.
> > >
> > > You already specifies
> > > --remote=db:OVN_Southbound,SB_Global,connections
> > > which should contain ptcp:6642:0.0.0.0 If you set both, it will
> > > result in conflict when ovsdb-server tries to create both sockets
> > > with same port.
> > >
> > >
> > > > > > >       Is "external_ids:ovn-remote-probe-interval" in ovsdb-
> > > server on
> > > > > > >       compute node for ovn-controller to probe ovn-sb-db?
> > > > > > >
> > > > > > > OVSDB probe is bidirectional, so you need to set this value,
> > > > > > > too, if you don't want too many probes handled by the SB
> > server.
> > > > > > > (setting the connection table for SB only changes the server
> > > side).
> > > > > >
> > > > > > In that case, how do I set probe interval for ovn-controller?
> > > > > > My understanding is that, ovn-controller reads configuration
> > > > > > from ovsdb-server on the local compute node. Isn't that right?
> > > > >
> > > > > >
> > > > >
> > > > > The configuration you mentioned "external_ids:ovn-remote-probe-
> > > interval"
> > > > > is exactly the way to set the ovn-controller -> SB probe
> interval.
> > > > > (SB -> ovn-controller probe is set in the connection table of
> > > > > SB)
> > > > >
> > > > >
> > > > > You are right that ovn-controller reads configuration from the
> > > > > local ovsdb-server. This setting is in local ovsdb-server.
> > > > >
> > > > >
> > > > > > >       Is "external_ids:ovn-openflow-probe-interval" in
> > > > > > > ovsdb-server
> > > > > on
> > > > > > >       compute node for ovn-controller to probe ovsdb-server?
> > > > > > >
> > > > > > > It is for the OpenFlow connection between ovn-controller and
> > > > > > > ovs- vswitchd, which is part of the OpenFlow protocol.
> > > > > > >
> > > > > > >       What's probe interval for ovsdb-server to probe ovn-
> > > controller?
> > > > > > >
> > > > > > > The local ovsdb connection uses unix socket, which doesn't
> > > > > > > send probe by default (if I remember correctly).
> > > > > >
> > > > > > Here is how ovsdb-server and ovn-controller is invoked on
> > > > > > compute
> > > node.
> > > > > > ========
> > > > > > root     41129  0.0  0.0 157556 20532 ?        S    Jul30
> 1:51
> > > > > /usr/sbin/ovsdb-server /var/lib/openvswitch/conf.db
> > > > > -vconsole:emer
> > > > > - vsyslog:err -vfile:info
> > > > > --remote=punix:/run/openvswitch/db.sock
> > > > > --
> > > > > remote=ptcp:6640:127.0.0.1 --
> > > > > remote=db:Open_vSwitch,Open_vSwitch,manager_options --log-
> > > > > file=/var/log/kolla/openvswitch/ovsdb-server.log --pidfile
> > > > > >
> > > > > > root     63775 55.9  0.4 1477796 1224324 ?     Sl   Aug04
> > 1360:55
> > > > > /usr/bin/ovn-controller --pidfile=/run/ovn/ovn-controller.pid
> > > > > --log- file=/var/log/kolla/openvswitch/ovn-controller.log
> > > > > tcp:127.0.0.1:6640 <http://127.0.0.1:6640>
> > > > > <http://127.0.0.1:6640>
> > > > > > ========
> > > > > > Is that OK? Or UNIX socket method is recommended for
> > > > > > ovn-controller
> > > > >
> > > > > > to connect to ovsdb-server?
> > > > >
> > > > > If using TCP, by default it is 5s probe interval. I think it is
> > > > > better to use unix socket. (but maybe it doesn't matter that
> > > > > much)
> > > > >
> > > > >
> > > > > >
> > > > > > Here is the configuration in open_vswitch table in ovsdb-
> server.
> > > > > > ========
> > > > > > external_ids        : {ovn-encap-ip="10.6.30.22", ovn-encap-
> > > > > type=geneve, ovn-openflow-probe-interval="30", ovn-
> > > > > remote="tcp:10.6.20.84:6642 <http://10.6.20.84:6642>
> > > > > <http://10.6.20.84:6642> ,tcp:10.6.20.85:6642
> > > > > <http://10.6.20.85:6642> <http://10.6.20.85:6642>
> > > > > ,tcp:10.6.20.86:6642 <http://10.6.20.86:6642>
> > > > > <http://10.6.20.86:6642> ", ovn-remote-probe-interval="60000",
> > > > > system-id="compute-3"}
> > > > > > ========
> > > > > > ovn-controller connects to ovsdb-server and reads this
> > > > > > configuration, so it knows how to connect to all sbdb nodes,
> > right?
> > > > >
> > > > > >
> > > > > Yes, you are right.
> > > > >
> > > > > > If it's TCP between ovn-controller and ovsdb-server, is that
> > > > > > probe interval setting will also apply to the probe from
> > > > > > ovn-controller to ovsdb-server?
> > > > >
> > > > > >
> > > > > No. They are not related.
> > > > >
> > > > >
> > > > > > ovn-controller connects to ovs-vswitchd by UNIX socket to
> > > > > > program open-flow. ovs-vswitchd and ovsdb-server are connected
> > by UNIX too.
> > > > > > So, is that ovn-openflow-probe-interval for the probe from
> > > > > > ovn-controller to ovs-vswitchd via UNIX?
> > > > > >
> > > > > > As a summary for the probe setting,
> > > > > >
> > > > > > +--------------+  driver configuration
> > > > > > |  ovn-driver  |
> > > > > > +--------------+
> > > > > >     ^    |
> > > > > >     |    v
> > > > > > +--------------+  inactivity_probe in table "Connection"
> > > > > > |  ovn-nb-db   |
> > > > > > +--------------+
> > > > > >     ^    |
> > > > > >     |    v
> > > > > > +--------------+  options:northd_probe_interval in table
> > > "NB_Global"
> > > > > > |  ovn-northd  |  in nbdb.
> > > > > > +--------------+
> > > > > >     ^    |
> > > > > >     |    v
> > > > > > +--------------+  inactivity_probe in table "Connection"
> > > > > > |  ovn-sb-db   |
> > > > > > +--------------+
> > > > > >     ^    |
> > > > > >     |    v
> > > > > > +--------------------------------+  in table "Open_vSwitch" in
> > > > > > +--------------------------------+ ovsdb-server
> > > > > > |        ovn-controller          |  ovn-remote-probe-interval
> > for
> > > TCP
> > > > > > +--------------------------------+  probe to ovsdb-server,
> > > > > >     ^    |            ^    |        ovn-openflow-probe-
> interval
> > > for
> > > > > UNIX
> > > > > >     |    v TCP        |    v UNIX   probe to ovs-vswitchd
> > > > > > +--------------+  +--------------+
> > > > > > | ovsdb-server |  | ovs-vswitchd |
> > > > > > +--------------+  +--------------+
> > > > > >
> > > > >
> > > > > > Is that correct?
> > > > >
> > > > > Correct. Except that you don't have to use TCP between
> > > > > ovn-controller and the local ovsdb-server. Use UNIX and then you
> > > > > don't need to worry about the probe between them.
> > > > >
> > > > >
> > > > > >
> > > > > > > For ovn-controller, since it is implemented with
> > > > > > > incremental-processing, even if there are probes from
> > > > > > > openflow or
> > > > > local ovsdb, it doesn't matter.
> > > > > > > If there is no configuration change, ovn-controller simply
> > > > > > > replies the probe and there is no extra cost.
> > > > > >
> > > > > > I did see errors about connecting to br-mgmt (UNIX connection
> > > > > > to
> > > > > > ovs-vswitchd) in ovn-controller logging. After setting proble
> > > > > > internal, those errors are gone. It seems that it does matter.
> > > > >
> > > > > >
> > > > > What's the error log? (I suspect maybe it is just some log but
> > > > > not
> > > > > error)
> > > >
> > > > This is from ovn-controller.
> > > > ========
> > > > 2020-08-06T03:53:08.663Z|128534|rconn|ERR|unix:/var/run/openvswitc
> > > > h/
> > > > br
> > > > -int.mgmt: no response to inactivity probe after 10 seconds,
> > > > disconnecting ======== It seems that there is probe on UNIX socket.
> > >
> > > >
> > > OK, if you see this it seems ovs-vswitchd is busy. If it happens
> > > occasionally due to the scale then disabling the probe using ovn-
> > > openflow-probe-interval should help.
> > >
> > >
> > > > > > >
> > > > > > >       Thanks!
> > > > > > >
> > > > > > >       Tony
> > > > > > >       > -----Original Message-----
> > > > > > >       > From: discuss <ovs-discuss-bounces at openvswitch.org
> > > > > > > <mailto:ovs-discuss-bounces at openvswitch.org>
> > > > > > > <mailto:ovs-discuss-bounces at openvswitch.org
> > > > > > > <mailto:ovs-discuss-bounces at openvswitch.org> >  <mailto:ovs-
> > > > > > > <mailto:ovs->
> > > > > <mailto:ovs- <mailto:ovs-> > discuss-bounces at openvswitch.org
> > > > > <mailto:discuss-bounces at openvswitch.org>  <mailto:discuss-
> > > > > <mailto:discuss-> bounces at openvswitch.org
> > > > > <mailto:bounces at openvswitch.org> > > > On Behalf Of Tony
> > > > > > >       > Liu
> > > > > > >       > Sent: Wednesday, August 5, 2020 4:29 PM
> > > > > > >       > To: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org>
> > > > > > > <mailto:hzhou at ovn.org <mailto:hzhou at ovn.org> >
> > > > > <mailto:hzhou at ovn.org <mailto:hzhou at ovn.org>
> > > > > <mailto:hzhou at ovn.org <mailto:hzhou at ovn.org> > > >
> > > > > > >       > Cc: ovs-dev <ovs-dev at openvswitch.org
> > > > > > > <mailto:ovs-dev at openvswitch.org>
> > > > > > > <mailto:ovs-dev at openvswitch.org
> > > > > > > <mailto:ovs-dev at openvswitch.org> >  <mailto:ovs-
> > > > > > > <mailto:ovs->
> > > > > > > <mailto:ovs- <mailto:ovs-> >
> > > > > dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > > >;
> > > > > ovs-discuss <ovs-
> > > > > > >       > discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > >
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > >
> > > > > >
> > > > > >
> > > > > > >       > Subject: Re: [ovs-discuss] [OVN] no response to
> > > > > > > inactivity
> > > > > probe
> > > > > > >       >
> > > > > > >       > Hi Han,
> > > > > > >       >
> > > > > > >       > After setting connection target="ptcp:6642:0.0.0.0"
> > > > > > > for ovn-sb-db, I see
> > > > > > >       > this error.
> > > > > > >       > ========
> > > > > > >       > 2020-08-
> > > > > > >       >
> > > > > > >
> > 05T23:01:26.819Z|06799|ovsdb_jsonrpc_server|ERR|ptcp:6642:0.0.0.
> > > > > > > 0
> > > > > > > <http://0.0.0.0> :
> > > > > > >       > listen failed: Address already in use ========
> > > > > > > Anything I am missing
> > > > > > >       > here?
> > > > > > >       >
> > > > > > >       >
> > > > > > >       > Thanks!
> > > > > > >       >
> > > > > > >       > Tony
> > > > > > >       > > -----Original Message-----
> > > > > > >       > > From: Han Zhou <hzhou at ovn.org
> > > > > > > <mailto:hzhou at ovn.org> <mailto:hzhou at ovn.org
> > > > > > > <mailto:hzhou at ovn.org> >
> > > > > <mailto:hzhou at ovn.org <mailto:hzhou at ovn.org>
> > > > > <mailto:hzhou at ovn.org <mailto:hzhou at ovn.org> > > >
> > > > > > >       > > Sent: Tuesday, August 4, 2020 4:44 PM
> > > > > > >       > > To: Tony Liu <tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com> >
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > >
> > > > > >
> > > > > >
> > > > > > >       > > Cc: Numan Siddique <numans at ovn.org
> > > > > > > <mailto:numans at ovn.org>  <mailto:numans at ovn.org
> > > > > > > <mailto:numans at ovn.org> > <mailto:numans at ovn.org
> > > > > > > <mailto:numans at ovn.org>  <mailto:numans at ovn.org
> > > > > > > <mailto:numans at ovn.org> > > >; Han Zhou
> > > > > <hzhou at ovn.org <mailto:hzhou at ovn.org>  <mailto:hzhou at ovn.org
> > > > > <mailto:hzhou at ovn.org> >  <mailto:hzhou at ovn.org
> > > > > <mailto:hzhou at ovn.org> <mailto:hzhou at ovn.org
> > > > > <mailto:hzhou at ovn.org>
> > > > > > > >; ovs-
> > > > > > >       > > discuss <ovs-discuss at openvswitch.org
> > > > > > > <mailto:ovs-discuss at openvswitch.org>
> > > > > > > <mailto:ovs-discuss at openvswitch.org
> > > > > > > <mailto:ovs-discuss at openvswitch.org> >  <mailto:ovs-
> > > > > > > <mailto:ovs->  <mailto:ovs- <mailto:ovs-> >
> > > > > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > >
> > > > > >
> > > > > >; ovs-dev
> > > > > > >       > > <ovs-dev at openvswitch.org
> > > > > > > <mailto:ovs-dev at openvswitch.org>
> > > > > > > <mailto:ovs-dev at openvswitch.org
> > > > > > > <mailto:ovs-dev at openvswitch.org>
> > > > > > > >
> > > > > <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>
> > > > > <mailto:ovs-dev at openvswitch.org <mailto:ovs-dev at openvswitch.org>
> > > > > >
> > > > > >
> > > > > >
> > > > > > >       > > Subject: Re: [ovs-discuss] [OVN] no response to
> > > > > > > inactivity probe
> > > > > > >       > >
> > > > > > >       > >
> > > > > > >       > >
> > > > > > >       > > On Tue, Aug 4, 2020 at 2:50 PM Tony Liu
> > > > > > > <tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > > > >
> > > > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > >
> > > > > >
> > > > > > >       > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com> >
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > <mailto:tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> > > > > wrote:
> > > > > > >       > >
> > > > > > >       > >
> > > > > > >       > >     Hi,
> > > > > > >       > >
> > > > > > >       > >     Since I have 3 OVN DB nodes, should I add 3
> rows
> > > in
> > > > > > > connection
> > > > > > >       > table
> > > > > > >       > >     for the inactivity_probe? Or put 3 addresses
> > into
> > > one
> > > > > row?
> > > > > > >       > >
> > > > > > >       > >     "set-connection" set one row only, and there
> is
> > no
> > > > > "add-
> > > > > > > connection".
> > > > > > >       > >     How should I add 3 rows into the table
> > connection?
> > > > > > >       > >
> > > > > > >       > >
> > > > > > >       > >
> > > > > > >       > >
> > > > > > >       > > You only need to set one row. Try this command:
> > > > > > >       > >
> > > > > > >       > > ovn-nbctl -- --id=@conn_uuid create Connection
> > > > > > >       > > target="ptcp\:6641\:0.0.0.0" inactivity_probe=0 --
> > > > > > > set NB_Global .
> > > > > > >       > > connections=@conn_uuid
> > > > > > >       > >
> > > > > > >       > >
> > > > > > >       > >
> > > > > > >       > >     Thanks!
> > > > > > >       > >
> > > > > > >       > >     Tony
> > > > > > >       > >
> > > > > > >       > >     > -----Original Message-----
> > > > > > >       > >     > From: Numan Siddique <numans at ovn.org
> > > <mailto:numans at ovn.org>
> > > > > <mailto:numans at ovn.org <mailto:numans at ovn.org> >
> > > > > > > <mailto:numans at ovn.org <mailto:numans at ovn.org>
> > > > > > > <mailto:numans at ovn.org <mailto:numans at ovn.org> > >
> > > > > > > <mailto:numans at ovn.org <mailto:numans at ovn.org>
> > > > > > > <mailto:numans at ovn.org <mailto:numans at ovn.org> >
> > > > > <mailto:numans at ovn.org <mailto:numans at ovn.org>
> > > > > <mailto:numans at ovn.org <mailto:numans at ovn.org> > > > >
> > > > > > >       > >     > Sent: Tuesday, August 4, 2020 12:36 AM
> > > > > > >       > >     > To: Tony Liu <tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com>
> > > > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > >
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com> > >
> > > > > > >       > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com> >
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >       > >     > Cc: ovs-discuss <ovs-discuss at openvswitch.org
> > > <mailto:ovs-discuss at openvswitch.org>
> > > > > <mailto:ovs-discuss at openvswitch.org
> > > > > <mailto:ovs-discuss at openvswitch.org> >  <mailto:ovs-
> > > > > <mailto:ovs->
> > > > > <mailto:ovs- <mailto:ovs-> >
> > > > > > > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > > >
> > > > > <mailto:ovs- <mailto:ovs->  <mailto:ovs- <mailto:ovs-> >
> > > > > <mailto:ovs- <mailto:ovs->  <mailto:ovs- <mailto:ovs-> > >
> > > > > > >       > > discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > >
> > > > > >
> > > > > > >; ovs-dev <ovs-
> > > > > > >       > >     > dev at openvswitch.org
> > <mailto:dev at openvswitch.org>
> > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > >
> > > > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > > > >
> > > > > > >       > >     > Subject: Re: [ovs-discuss] [OVN] no response
> > to
> > > > > > > inactivity probe
> > > > > > >       > >     >
> > > > > > >       > >     >
> > > > > > >       > >     >
> > > > > > >       > >     > On Tue, Aug 4, 2020 at 9:12 AM Tony Liu
> > > > > > > <tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > > > >
> > > > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > >
> > > > > >
> > > > > > >       > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com> >
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > >
> > > > > >
> > > > > >
> > > > > > >       > >     > <mailto:tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com>
> > > > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com>
> > > > > >
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > <mailto:tonyliu0592 at hotmail.com <mailto:tonyliu0592 at hotmail.com> > >
> > > > > > >       > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com> >
> > > > > > > <mailto:tonyliu0592 at hotmail.com
> > > > > > > <mailto:tonyliu0592 at hotmail.com>
> > > > > <mailto:tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> > > > > > wrote:
> > > > > > >       > >     >
> > > > > > >       > >     >
> > > > > > >       > >     >       In my deployment, on each Neutron
> server,
> > > > > there are
> > > > > > > 13
> > > > > > >       > > Neutron
> > > > > > >       > >     > server processes.
> > > > > > >       > >     >       I see 12 of them (monitor, maintenance,
> > > RPC,
> > > > > API)
> > > > > > > connect
> > > > > > >       > > to both
> > > > > > >       > >     > ovn-nb-db
> > > > > > >       > >     >       and ovn-sb-db. With 3 Neutron server
> > nodes,
> > > > > that's
> > > > > > > 36 OVSDB
> > > > > > >       > > clients.
> > > > > > >       > >     >       Is so many clients OK?
> > > > > > >       > >     >
> > > > > > >       > >     >       Any suggestions how to figure out
> which
> > > side
> > > > > > > doesn't
> > > > > > >       > > respond the
> > > > > > >       > >     > probe,
> > > > > > >       > >     >       if it's bi-directional? I don't see
> any
> > > > > activities
> > > > > > > from
> > > > > > >       > > logging,
> > > > > > >       > >     > other than
> > > > > > >       > >     >       connect/drop and reconnect...
> > > > > > >       > >     >
> > > > > > >       > >     >       BTW, please let me know if this is not
> > the
> > > > > right
> > > > > > > place to
> > > > > > >       > > discuss
> > > > > > >       > >     > Neutron OVN
> > > > > > >       > >     >       ML2 driver.
> > > > > > >       > >     >
> > > > > > >       > >     >
> > > > > > >       > >     >       Thanks!
> > > > > > >       > >     >
> > > > > > >       > >     >       Tony
> > > > > > >       > >     >
> > > > > > >       > >     >       > -----Original Message-----
> > > > > > >       > >     >       > From: dev <ovs-dev-
> > > bounces at openvswitch.org <mailto:ovs-dev-bounces at openvswitch.org>
> > > > > <mailto:ovs-dev-bounces at openvswitch.org
> > > > > <mailto:ovs-dev-bounces at openvswitch.org> >
> > > > > > > <mailto:ovs-dev-bounces at openvswitch.org
> > > > > > > <mailto:ovs-dev-bounces at openvswitch.org>  <mailto:ovs-dev-
> > > > > > > <mailto:ovs-dev->
> > > > > bounces at openvswitch.org <mailto:bounces at openvswitch.org> > >
> > > > > <mailto:ovs- <mailto:ovs->  <mailto:ovs- <mailto:ovs-> >
> > > > > <mailto:ovs- <mailto:ovs->
> > > > > <mailto:ovs- <mailto:ovs-> > >
> > > > > > >       > > dev-bounces at openvswitch.org
> > > > > > > <mailto:dev-bounces at openvswitch.org>
> > > > > > > <mailto:dev-bounces at openvswitch.org
> > > > > > > <mailto:dev-bounces at openvswitch.org> >  <mailto:dev-
> > > > > > > <mailto:dev->  <mailto:dev- <mailto:dev-> >
> > > > > > > bounces at openvswitch.org <mailto:bounces at openvswitch.org>
> > > > > > > <mailto:bounces at openvswitch.org
> > > > > > > <mailto:bounces at openvswitch.org>
> > > > > > > > > >
> > > > > > > <mailto:ovs-dev- <mailto:ovs-dev->  <mailto:ovs-dev-
> > > > > > > <mailto:ovs-dev-> >  <mailto:ovs-dev- <mailto:ovs-dev->
> > > > > > > <mailto:ovs-dev- <mailto:ovs-dev-> > >
> > > > > > > <mailto:ovs-dev- <mailto:ovs-dev->  <mailto:ovs-dev-
> > > > > > > <mailto:ovs-dev-> >  <mailto:ovs-dev- <mailto:ovs-dev->
> > > > > > > <mailto:ovs- <mailto:ovs->
> > > > > dev-> > >
> > > > > > >       > >     > bounces at openvswitch.org
> > > <mailto:bounces at openvswitch.org>
> > > > > <mailto:bounces at openvswitch.org <mailto:bounces at openvswitch.org>
> > > > > > <mailto:bounces at openvswitch.org
> > > > > <mailto:bounces at openvswitch.org> <mailto:bounces at openvswitch.org
> > > > > <mailto:bounces at openvswitch.org> >
> > > > > >
> > > > > > > <mailto:bounces at openvswitch.org
> > > > > > > <mailto:bounces at openvswitch.org>
> > > > > > > <mailto:bounces at openvswitch.org
> > > > > > > <mailto:bounces at openvswitch.org>
> > > > > > > > <mailto:bounces at openvswitch.org
> > > > > > > <mailto:bounces at openvswitch.org>
> > > > > <mailto:bounces at openvswitch.org <mailto:bounces at openvswitch.org>
> > > > > >
> > > > > >
> > > > > > > > On
> > > > > > >       > > Behalf Of Tony Liu
> > > > > > >       > >     >       > Sent: Monday, August 3, 2020 7:45 PM
> > > > > > >       > >     >       > To: ovs-discuss <ovs-
> > > discuss at openvswitch.org <mailto:ovs-discuss at openvswitch.org>
> > > > > <mailto:ovs-discuss at openvswitch.org
> > > > > <mailto:ovs-discuss at openvswitch.org> >
> > > > > > > <mailto:ovs-discuss at openvswitch.org
> > > > > > > <mailto:ovs-discuss at openvswitch.org>  <mailto:ovs-
> > > > > > > <mailto:ovs->
> > > > > discuss at openvswitch.org <mailto:discuss at openvswitch.org> > >
> > > > > <mailto:ovs- <mailto:ovs->  <mailto:ovs- <mailto:ovs-> >
> > > > > <mailto:ovs- <mailto:ovs->
> > > > > <mailto:ovs- <mailto:ovs-> > >
> > > > > > >       > > discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > > > >
> > > > > > > <mailto:ovs- <mailto:ovs->  <mailto:ovs- <mailto:ovs-> >
> > > > > > > <mailto:ovs- <mailto:ovs->  <mailto:ovs- <mailto:ovs-> > >
> > > > > <mailto:ovs- <mailto:ovs->  <mailto:ovs- <mailto:ovs-> >
> > > > > <mailto:ovs- <mailto:ovs->  <mailto:ovs- <mailto:ovs-> > > >
> > > > > > >       > >     > discuss at openvswitch.org
> > > <mailto:discuss at openvswitch.org>
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > > <mailto:discuss at openvswitch.org
> > > > > <mailto:discuss at openvswitch.org> <mailto:discuss at openvswitch.org
> > > > > <mailto:discuss at openvswitch.org> >
> > > > > >
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > >
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > >
> > > > > >
> > > > > > > >;
> > > > > > >       > > ovs-dev <ovs-
> > > > > > >       > >     >       > dev at openvswitch.org
> > > <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > >
> > > > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > > >
> > > > > > >       > > <mailto:dev at openvswitch.org
> > > > > > > <mailto:dev at openvswitch.org>  <mailto:dev at openvswitch.org
> > > > > > > <mailto:dev at openvswitch.org> > <mailto:dev at openvswitch.org
> > > > > > > <mailto:dev at openvswitch.org>  <mailto:dev at openvswitch.org
> > > > > > > <mailto:dev at openvswitch.org> > >
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > > > >
> > > > > >
> > > > > > >       > >     >       > Subject: [ovs-dev] [OVN] no response
> > to
> > > > > > > inactivity probe
> > > > > > >       > >     >       >
> > > > > > >       > >     >       > Hi,
> > > > > > >       > >     >       >
> > > > > > >       > >     >       > Neutron OVN ML2 driver was
> > disconnected
> > > by
> > > > > ovn-
> > > > > > > nb-db.
> > > > > > >       > > There are
> > > > > > >       > >     > many
> > > > > > >       > >     >       > error messages from ovn-nb-db leader.
> > > > > > >       > >     >       > ========
> > > > > > >       > >     >       > 2020-08-
> > > > > > >       > >
> > > > > > > 04T02:31:39.751Z|03138|reconnect|ERR|tcp:10.6.20.81:58620
> > > <http://10.6.20.81:58620> <http://10.6.20.81:58620>
> > > <http://10.6.20.81:58620>
> > > > > > >       > > <http://10.6.20.81:58620>
> > > > > > >       > >     > <http://10.6.20.81:58620> : no
> > > > > > >       > >     >       > response to inactivity probe after 5
> > > seconds,
> > > > > > >       > > disconnecting
> > > > > > >       > >     >       > 2020-08-
> > > > > > >       > >
> > > > > > > 04T02:31:42.484Z|03139|reconnect|ERR|tcp:10.6.20.81:58300
> > > <http://10.6.20.81:58300> <http://10.6.20.81:58300>
> > > <http://10.6.20.81:58300>
> > > > > > >       > > <http://10.6.20.81:58300>
> > > > > > >       > >     > <http://10.6.20.81:58300> : no
> > > > > > >       > >     >       > response to inactivity probe after 5
> > > seconds,
> > > > > > >       > > disconnecting
> > > > > > >       > >     >       > 2020-08-
> > > > > > >       > >
> > > > > > > 04T02:31:49.858Z|03140|reconnect|ERR|tcp:10.6.20.81:59582
> > > <http://10.6.20.81:59582> <http://10.6.20.81:59582>
> > > <http://10.6.20.81:59582>
> > > > > > >       > > <http://10.6.20.81:59582>
> > > > > > >       > >     > <http://10.6.20.81:59582> : no
> > > > > > >       > >     >       > response to inactivity probe after 5
> > > seconds,
> > > > > > >       > > disconnecting
> > > > > > >       > >     >       > 2020-08-
> > > > > > >       > >
> > > > > > > 04T02:31:53.057Z|03141|reconnect|ERR|tcp:10.6.20.83:42626
> > > <http://10.6.20.83:42626> <http://10.6.20.83:42626>
> > > <http://10.6.20.83:42626>
> > > > > > >       > > <http://10.6.20.83:42626>
> > > > > > >       > >     > <http://10.6.20.83:42626> : no
> > > > > > >       > >     >       > response to inactivity probe after 5
> > > seconds,
> > > > > > >       > > disconnecting
> > > > > > >       > >     >       > 2020-08-
> > > > > > >       > >
> > > > > > > 04T02:31:53.058Z|03142|reconnect|ERR|tcp:10.6.20.82:45412
> > > <http://10.6.20.82:45412> <http://10.6.20.82:45412>
> > > <http://10.6.20.82:45412>
> > > > > > >       > > <http://10.6.20.82:45412>
> > > > > > >       > >     > <http://10.6.20.82:45412> : no
> > > > > > >       > >     >       > response to inactivity probe after 5
> > > seconds,
> > > > > > >       > > disconnecting
> > > > > > >       > >     >       > 2020-08-
> > > > > > >       > >
> > > > > > > 04T02:31:54.067Z|03143|reconnect|ERR|tcp:10.6.20.81:59416
> > > <http://10.6.20.81:59416> <http://10.6.20.81:59416>
> > > <http://10.6.20.81:59416>
> > > > > > >       > > <http://10.6.20.81:59416>
> > > > > > >       > >     > <http://10.6.20.81:59416> : no
> > > > > > >       > >     >       > response to inactivity probe after 5
> > > seconds,
> > > > > > >       > > disconnecting
> > > > > > >       > >     >       > 2020-08-
> > > > > > >       > >
> > > > > > > 04T02:31:54.809Z|03144|reconnect|ERR|tcp:10.6.20.81:60004
> > > <http://10.6.20.81:60004> <http://10.6.20.81:60004>
> > > <http://10.6.20.81:60004>
> > > > > > >       > > <http://10.6.20.81:60004>
> > > > > > >       > >     > <http://10.6.20.81:60004> : no
> > > > > > >       > >     >       > response to inactivity probe after 5
> > > seconds,
> > > > > > >       > > disconnecting
> > > > > > >       > >     > ========
> > > > > > >       > >     >       >
> > > > > > >       > >     >       > Could anyone share a bit details how
> > > this
> > > > > > > inactivity
> > > > > > >       > > probe works?
> > > > > > >       > >     >
> > > > > > >       > >     >
> > > > > > >       > >     >
> > > > > > >       > >     > The inactivity probe is sent by both the
> > server
> > > and
> > > > > > > clients
> > > > > > >       > >     > independently.
> > > > > > >       > >     > Meaning ovsdb-server will send an inactivity
> > > probe
> > > > > every
> > > > > > > 'x'
> > > > > > >       > > configured
> > > > > > >       > >     > seconds to all its connected clients and if
> it
> > > > > doesn't
> > > > > > > get a
> > > > > > >       > reply
> > > > > > >       > > from
> > > > > > >       > >     > the client within some time, it disconnects
> > the
> > > > > > > connection.
> > > > > > >       > >     >
> > > > > > >       > >     > The inactivity probe from the server side
> can
> > be
> > > > > > > configured. Run
> > > > > > >       > > "ovn-
> > > > > > >       > >     > nbctl list connection"
> > > > > > >       > >     > and you will see inactivity_probe column.
> You
> > > can
> > > > > set
> > > > > > > this column
> > > > > > >       > > to
> > > > > > >       > >     > desired value like - ovn-nbctl set
> connection .
> > > > > > >       > > inactivity_probe=30000
> > > > > > >       > >     > (for 30 seconds)
> > > > > > >       > >     >
> > > > > > >       > >     > The same thing for SB ovsdb-server.
> > > > > > >       > >     >
> > > > > > >       > >     > Similarly each client (ovn-northd, ovn-
> > > controller,
> > > > > > > neutron server)
> > > > > > >       > > sends
> > > > > > >       > >     > inactivity probe every 'y' seconds and if
> the
> > > client
> > > > > > > doesn't get
> > > > > > >       > > any
> > > > > > >       > >     > reply from ovsdb-server it will disconnect
> the
> > > > > connection
> > > > > > > and
> > > > > > >       > > reconnect
> > > > > > >       > >     > again.
> > > > > > >       > >     >
> > > > > > >       > >     > For ovn-northd you can configured this as -
> > ovn-
> > > > > nbctl set
> > > > > > >       > NB_Global
> > > > > > >       > > .
> > > > > > >       > >     > options:northd_probe_interval=30000
> > > > > > >       > >     >
> > > > > > >       > >     > For ovn-controllers - ovs-vsctl set open .
> > > > > > > external_ids:ovn-
> > > > > > >       > > remote-
> > > > > > >       > >     > probe-interval=30000
> > > > > > >       > >     >
> > > > > > >       > >     > There is also a probe interval for openflow
> > > > > connection
> > > > > > > from ovn-
> > > > > > >       > >     > controller to ovs-vswitchd which you can
> > > configure
> > > > > as
> > > > > > > ovs-vsctl
> > > > > > >       > set
> > > > > > >       > >     > open . external_ids:ovn-openflow-probe-
> > > interval=30
> > > > > (this
> > > > > > > is in
> > > > > > >       > > seconds)
> > > > > > >       > >     >
> > > > > > >       > >     >
> > > > > > >       > >     > Regarding the neutron server I think it is
> set
> > > to 60
> > > > > > > seconds.
> > > > > > >       > > Please see
> > > > > > >       > >     > this -
> > > > > > >       > >     >
> > > > > > >       > >
> > > > > > >       >
> > > > > > >
> > > > > https://github.com/openstack/neutron/blob/master/neutron/conf/pl
> > > > > ug
> > > > > in
> > > > > s/ml
> > > > > > >       > >     > 2/drivers/ovn/ovn_conf.py#L80
> > > > > > >       > >     >
> > > > > > >       > >     > From the logs you shared, it looks like
> ovsdb-
> > > server
> > > > > is
> > > > > > > not
> > > > > > >       > getting
> > > > > > >       > > the
> > > > > > >       > >     > probe reply from neutron server after 5
> > seconds
> > > and
> > > > > hence
> > > > > > > it is
> > > > > > >       > >     > disconnecting. Not sure what's happening
> > though.
> > > > > > >       > >     >
> > > > > > >       > >     > You can try increasing the inactivity probe
> > > interval
> > > > > on
> > > > > > > the
> > > > > > >       > > ovsdb-server
> > > > > > >       > >     > side with the first command I shared.
> > > > > > >       > >     > Note: If "ovn-nbctl list connection" returns
> > > empty,
> > > > > you
> > > > > > > need to
> > > > > > >       > > create a
> > > > > > >       > >     > connection row like - ovn-nbctl set-
> connection
> > > > > > > ptcp:6641:<IP>
> > > > > > >       > >     >
> > > > > > >       > >     >
> > > > > > >       > >     > Thanks
> > > > > > >       > >     > Numan
> > > > > > >       > >     >
> > > > > > >       > >     >
> > > > > > >       > >     >
> > > > > > >       > >     >
> > > > > > >       > >     >       > From OVN ML2 driver log, I see it
> > > connected
> > > > > to
> > > > > > > the leader,
> > > > > > >       > > then
> > > > > > >       > >     > the
> > > > > > >       > >     >       > connection was closed by leader
> after
> > 5
> > > or 6
> > > > > > > seconds. Is
> > > > > > >       > > this
> > > > > > >       > >     > probe one-
> > > > > > >       > >     >       > way or two-ways?
> > > > > > >       > >     >       > Both sides are not busy, not taking
> > much
> > > CPU
> > > > > > > cycles. Not
> > > > > > >       > > sure how
> > > > > > >       > >     > this
> > > > > > >       > >     >       > could happen. Any thoughts?
> > > > > > >       > >     >       >
> > > > > > >       > >     >       >
> > > > > > >       > >     >       > Thanks!
> > > > > > >       > >     >       >
> > > > > > >       > >     >       > Tony
> > > > > > >       > >     >       >
> > > > > > >       > >     >       >
> > > > > > >       > >     >       >
> > > > > > >       > >     >       >
> > > > > _______________________________________________
> > > > > > >       > >     >       > dev mailing list
> > > > > > >       > >     >       > dev at openvswitch.org
> > > <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > >
> > > > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > > >
> > > > > > >       > > <mailto:dev at openvswitch.org
> > > > > > > <mailto:dev at openvswitch.org>  <mailto:dev at openvswitch.org
> > > > > > > <mailto:dev at openvswitch.org> > <mailto:dev at openvswitch.org
> > > > > > > <mailto:dev at openvswitch.org>  <mailto:dev at openvswitch.org
> > > > > > > <mailto:dev at openvswitch.org> > >
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> >
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>
> > > > > <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org> > > > >
> > > > > > >       > >     >       >
> > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > > > > > >       > >     >
> > > > > _______________________________________________
> > > > > > >       > >     >       discuss mailing list
> > > > > > >       > >     >       discuss at openvswitch.org
> > > <mailto:discuss at openvswitch.org>
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > >
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > >
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > >
> > > > > >
> > > > > >
> > > > > > >       > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org> >
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > >
> > > > > > <mailto:discuss at openvswitch.org
> > > > > > <mailto:discuss at openvswitch.org>
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > > <mailto:discuss at openvswitch.org
> > > > > <mailto:discuss at openvswitch.org> <mailto:discuss at openvswitch.org
> > > > > <mailto:discuss at openvswitch.org> >
> > > > > >
> > > > > > >
> > > > > > >       > >     >
> > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-
> > > > > > > discuss
> > > > > > >       > >     >
> > > > > > >       > >     >
> > > > > > >       > >
> > > > > > >       > >
> > > > > > >       >
> > > > > > >       > _______________________________________________
> > > > > > >       > discuss mailing list
> > > > > > >       > discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > <mailto:discuss at openvswitch.org
> > > > > > > <mailto:discuss at openvswitch.org>
> > > > > > > >
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > <mailto:discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > > > > >
> > > > > >
> > > > > > >       >
> > > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> > > > > > >
> > > > > >
> > > >
> >
> > _______________________________________________
> > dev mailing list
> > dev at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss


More information about the dev mailing list