[ovs-discuss] SB flows not being created in OVN K8 Stateful set

Brendan Doyle brendan.doyle at oracle.com
Thu Aug 6 09:54:45 UTC 2020


I don't see any ovn-northd.log log, I only see those when I'm running 
OVN outside the k8s cluster.
Before I start the Satefulset on my k8 nodes I run:

ovn-ctl stop_northd
ovn-ctl stop_ovsdb
rm -rf /usr/etc/ovn/*.db


The only logs I see are ovn-controller.log  (I'm running ovn controller 
on the K8 nodes) ovsdb-server-nb.log ovsdb-server-sb.log
These logs look normal

k8 node 1 logs
============
2020-08-05T13:44:51.958Z|00001|vlog|INFO|opened log file 
/var/log/ovn/ovsdb-server-nb.log
2020-08-05T13:44:51.961Z|00002|ovsdb_server|INFO|ovsdb-server (Open 
vSwitch) 2.13.90
2020-08-05T13:44:51.961Z|00003|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connecting...
2020-08-05T13:44:51.961Z|00004|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connected
2020-08-05T13:44:51.962Z|00005|raft_rpc|INFO|learned cluster ID 6e05
2020-08-05T13:44:51.962Z|00006|raft|INFO|tcp:[253.255.0.34]:6643: 
learned server ID c8a5
2020-08-05T13:44:51.962Z|00007|raft|INFO|server c8a5 is leader for term 1
2020-08-05T13:44:51.962Z|00008|raft|INFO|rejecting append_request 
because previous entry 1,5 not in local log (mismatch past end of log)
2020-08-05T13:44:51.964Z|00009|raft|INFO|server c8a5 added to configuration
2020-08-05T13:44:51.966Z|00010|raft|INFO|server 0d3d added to configuration
2020-08-05T13:44:51.966Z|00011|raft|INFO|tcp:253.255.0.34:50744: learned 
server ID c8a5
2020-08-05T13:44:51.966Z|00012|raft|INFO|tcp:253.255.0.34:50744: learned 
remote address tcp:[253.255.0.34]:6643
2020-08-05T13:44:52.015Z|00013|raft|INFO|tcp:253.255.0.35:51780: learned 
server ID 941e
2020-08-05T13:44:52.015Z|00014|raft|INFO|tcp:253.255.0.35:51780: learned 
remote address tcp:[253.255.0.35]:6643
2020-08-05T13:44:52.015Z|00015|raft|INFO|adding 941e (941e at 
tcp:[253.255.0.35]:6643) to cluster 6e05 failed (not leader)
2020-08-05T13:44:52.015Z|00016|raft|INFO|server 941e added to configuration
2020-08-05T13:44:52.016Z|00017|reconnect|INFO|tcp:[253.255.0.35]:6643: 
connecting...
2020-08-05T13:44:52.016Z|00018|reconnect|INFO|tcp:[253.255.0.35]:6643: 
connected
2020-08-05T13:45:01.962Z|00019|memory|INFO|8424 kB peak resident set 
size after 10.0 seconds
2020-08-05T13:45:01.962Z|00020|memory|INFO|cells:40 monitors:0 
raft-connections:4
2020-08-05T14:10:03.593Z|00021|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connection closed by peer
2020-08-05T14:10:03.595Z|00022|raft|INFO|server 941e is leader for term 2
2020-08-05T14:10:04.594Z|00023|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connecting...
2020-08-05T14:10:04.594Z|00024|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connection attempt failed (Connection refused)
2020-08-05T14:10:04.594Z|00025|reconnect|INFO|tcp:[253.255.0.34]:6643: 
waiting 2 seconds before reconnect
2020-08-05T14:10:06.594Z|00026|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connecting...
2020-08-05T14:10:06.594Z|00027|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connection attempt failed (Connection refused)
2020-08-05T14:10:06.594Z|00028|reconnect|INFO|tcp:[253.255.0.34]:6643: 
waiting 4 seconds before reconnect
2020-08-05T14:10:08.453Z|00029|raft|INFO|received leadership transfer 
from 941e in term 2
2020-08-05T14:10:08.453Z|00030|raft|INFO|term 3: starting election
2020-08-05T14:10:08.453Z|00031|reconnect|INFO|tcp:[253.255.0.35]:6643: 
connection closed by peer

2020-08-05T13:44:48.971Z|00001|vlog|INFO|opened log file 
/var/log/ovn/ovsdb-server-sb.log
2020-08-05T13:44:48.973Z|00002|ovsdb_server|INFO|ovsdb-server (Open 
vSwitch) 2.13.90
2020-08-05T13:44:48.973Z|00003|reconnect|INFO|tcp:[253.255.0.34]:6644: 
connecting...
2020-08-05T13:44:48.973Z|00004|reconnect|INFO|tcp:[253.255.0.34]:6644: 
connected
2020-08-05T13:44:48.973Z|00005|raft_rpc|INFO|learned cluster ID 281c
2020-08-05T13:44:48.973Z|00006|raft|INFO|tcp:[253.255.0.34]:6644: 
learned server ID eeed
2020-08-05T13:44:48.974Z|00007|raft|INFO|server eeed is leader for term 1
2020-08-05T13:44:48.974Z|00008|raft|INFO|rejecting append_request 
because previous entry 1,4 not in local log (mismatch past end of log)
2020-08-05T13:44:48.976Z|00009|raft|INFO|server eeed added to configuration
2020-08-05T13:44:48.977Z|00010|raft|INFO|server 5098 added to configuration
2020-08-05T13:44:48.977Z|00011|raft|INFO|tcp:253.255.0.34:50628: learned 
server ID eeed
2020-08-05T13:44:48.977Z|00012|raft|INFO|tcp:253.255.0.34:50628: learned 
remote address tcp:[253.255.0.34]:6644
2020-08-05T13:44:49.044Z|00013|raft|INFO|tcp:253.255.0.35:49594: learned 
server ID b9de
2020-08-05T13:44:49.044Z|00014|raft|INFO|tcp:253.255.0.35:49594: learned 
remote address tcp:[253.255.0.35]:6644
2020-08-05T13:44:49.044Z|00015|raft|INFO|adding b9de (b9de at 
tcp:[253.255.0.35]:6644) to cluster 281c failed (not leader)
2020-08-05T13:44:49.044Z|00016|raft|INFO|server b9de added to configuration
2020-08-05T13:44:49.044Z|00017|reconnect|INFO|tcp:[253.255.0.35]:6644: 
connecting...
2020-08-05T13:44:49.045Z|00018|reconnect|INFO|tcp:[253.255.0.35]:6644: 
connected
2020-08-05T13:44:58.975Z|00019|memory|INFO|8408 kB peak resident set 
size after 10.0 seconds
2020-08-05T13:44:58.975Z|00020|memory|INFO|cells:39 monitors:0 
raft-connections:4
2020-08-05T14:10:03.593Z|00021|reconnect|INFO|tcp:[253.255.0.34]:6644: 
connection closed by peer
2020-08-05T14:10:03.595Z|00022|raft|INFO|server b9de is leader for term 2
2020-08-05T14:10:04.595Z|00023|reconnect|INFO|tcp:[253.255.0.34]:6644: 
connecting...
2020-08-05T14:10:04.595Z|00024|reconnect|INFO|tcp:[253.255.0.34]:6644: 
connection attempt failed (Connection refused)
2020-08-05T14:10:04.595Z|00025|reconnect|INFO|tcp:[253.255.0.34]:6644: 
waiting 2 seconds before reconnect

2020-08-05T13:47:27.231Z|00001|vlog|INFO|opened log file 
/usr/var/log/ovn/ovn-controller.log
2020-08-05T13:47:27.233Z|00002|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock: 
connecting...
2020-08-05T13:47:27.233Z|00003|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock: 
connected
2020-08-05T13:47:27.235Z|00004|main|INFO|OVS IDL reconnected, force 
recompute.
2020-08-05T13:47:27.236Z|00005|reconnect|INFO|tcp:253.255.0.35:6642: 
connecting...
2020-08-05T13:47:27.236Z|00006|main|INFO|OVNSB IDL reconnected, force 
recompute.
2020-08-05T13:47:27.236Z|00007|reconnect|INFO|tcp:253.255.0.35:6642: 
connected
2020-08-05T13:47:27.237Z|00008|ofctrl|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting to switch
2020-08-05T13:47:27.237Z|00009|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting...
2020-08-05T13:47:27.237Z|00010|rconn|WARN|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connection failed (No such file or directory)
2020-08-05T13:47:27.237Z|00011|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
waiting 1 seconds before reconnect
2020-08-05T13:47:28.239Z|00012|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting...
2020-08-05T13:47:28.239Z|00013|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connected
2020-08-05T13:47:28.240Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting to switch
2020-08-05T13:47:28.240Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting...
2020-08-05T13:47:28.240Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connected
2020-08-05T13:47:28.436Z|00014|dpif_netlink|INFO|The kernel module does 
not support meters.
2020-08-05T14:10:05.486Z|00015|main|INFO|OVNSB commit failed, force 
recompute next time.
2020-08-05T14:10:05.486Z|00016|jsonrpc|WARN|tcp:253.255.0.35:6642: 
receive error: Connection reset by peer
2020-08-05T14:10:05.486Z|00017|reconnect|WARN|tcp:253.255.0.35:6642: 
connection dropped (Connection reset by peer)
2020-08-05T14:10:05.486Z|00018|reconnect|INFO|tcp:253.255.0.33:6642: 
connecting...
2020-08-05T14:10:05.486Z|00019|reconnect|INFO|tcp:253.255.0.33:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:10:05.487Z|00020|reconnect|INFO|tcp:253.255.0.34:6642: 
connecting...
2020-08-05T14:10:05.487Z|00021|reconnect|INFO|tcp:253.255.0.34:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:10:06.488Z|00022|reconnect|INFO|tcp:253.255.0.35:6642: 
connecting...
2020-08-05T14:10:06.488Z|00023|reconnect|INFO|tcp:253.255.0.35:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:10:06.488Z|00024|reconnect|INFO|tcp:253.255.0.35:6642: 
waiting 2 seconds before reconnect
2020-08-05T14:10:08.490Z|00025|reconnect|INFO|tcp:253.255.0.33:6642: 
connecting...
2020-08-05T14:10:08.490Z|00026|reconnect|INFO|tcp:253.255.0.33:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:10:08.490Z|00027|reconnect|INFO|tcp:253.255.0.33:6642: 
waiting 4 seconds before reconnect
2020-08-05T14:10:12.492Z|00028|reconnect|INFO|tcp:253.255.0.34:6642: 
connecting...
2020-08-05T14:10:12.492Z|00029|reconnect|INFO|tcp:253.255.0.34:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:10:12.492Z|00030|reconnect|INFO|tcp:253.255.0.34:6642: 
continuing to reconnect in the background but suppressing further logging

K8 node 2
---------------
2020-08-05T13:43:11.094Z|00001|vlog|INFO|opened log file 
/var/log/ovn/ovsdb-server-nb.log
2020-08-05T13:43:11.095Z|00002|raft|INFO|term 1: 40130328487 ms timeout 
expired, starting election
2020-08-05T13:43:11.095Z|00003|raft|INFO|term 1: elected leader by 1+ of 
1 servers
2020-08-05T13:43:11.098Z|00004|ovsdb_server|INFO|ovsdb-server (Open 
vSwitch) 2.13.90
2020-08-05T13:43:12.074Z|00005|jsonrpc|WARN|unix#4: send error: Broken pipe
2020-08-05T13:43:12.074Z|00006|reconnect|WARN|unix#4: connection dropped 
(Broken pipe)
2020-08-05T13:43:15.942Z|00007|jsonrpc|WARN|tcp:253.255.0.35:60448: 
receive error: Connection reset by peer
2020-08-05T13:43:15.942Z|00008|reconnect|WARN|tcp:253.255.0.35:60448: 
connection dropped (Connection reset by peer)
2020-08-05T13:43:15.947Z|00009|raft|INFO|tcp:253.255.0.33:46572: learned 
server ID 0d3d
2020-08-05T13:43:15.947Z|00010|raft|INFO|tcp:253.255.0.33:46572: learned 
remote address tcp:[253.255.0.33]:6643
2020-08-05T13:43:15.947Z|00011|raft|INFO|starting to add server 0d3d 
(0d3d at tcp:[253.255.0.33]:6643) to cluster 6e05
2020-08-05T13:43:15.950Z|00012|raft|INFO|cluster 6e05: installed 
snapshot on server 0d3d  up to 1:1
2020-08-05T13:43:15.951Z|00013|reconnect|INFO|tcp:[253.255.0.33]:6643: 
connecting...
2020-08-05T13:43:15.952Z|00014|reconnect|INFO|tcp:[253.255.0.33]:6643: 
connected
2020-08-05T13:43:15.952Z|00015|raft|INFO|adding 0d3d (0d3d at 
tcp:[253.255.0.33]:6643) to cluster 6e05 succeeded (completed)
2020-08-05T13:43:15.997Z|00016|raft|INFO|tcp:253.255.0.35:60500: learned 
server ID 941e
2020-08-05T13:43:15.997Z|00017|raft|INFO|tcp:253.255.0.35:60500: learned 
remote address tcp:[253.255.0.35]:6643
2020-08-05T13:43:15.997Z|00018|raft|INFO|starting to add server 941e 
(941e at tcp:[253.255.0.35]:6643) to cluster 6e05
2020-08-05T13:43:16.000Z|00019|raft|INFO|cluster 6e05: installed 
snapshot on server 941e  up to 1:1
2020-08-05T13:43:16.001Z|00020|reconnect|INFO|tcp:[253.255.0.35]:6643: 
connecting...
2020-08-05T13:43:16.001Z|00021|reconnect|INFO|tcp:[253.255.0.35]:6643: 
connected
2020-08-05T13:43:16.002Z|00022|raft|INFO|adding 941e (941e at 
tcp:[253.255.0.35]:6643) to cluster 6e05 succeeded (completed)
2020-08-05T13:43:21.101Z|00023|memory|INFO|8452 kB peak resident set 
size after 10.0 seconds
2020-08-05T13:43:21.101Z|00024|memory|INFO|cells:40 monitors:0 
raft-connections:4

2020-08-05T13:43:11.105Z|00001|vlog|INFO|opened log file 
/var/log/ovn/ovsdb-server-sb.log
2020-08-05T13:43:11.106Z|00002|raft|INFO|term 1: 40130328497 ms timeout 
expired, starting election
2020-08-05T13:43:11.106Z|00003|raft|INFO|term 1: elected leader by 1+ of 
1 servers
2020-08-05T13:43:11.108Z|00004|ovsdb_server|INFO|ovsdb-server (Open 
vSwitch) 2.13.90
2020-08-05T13:43:12.959Z|00005|raft|INFO|tcp:253.255.0.33:46140: learned 
server ID 5098
2020-08-05T13:43:12.959Z|00006|raft|INFO|tcp:253.255.0.33:46140: learned 
remote address tcp:[253.255.0.33]:6644
2020-08-05T13:43:12.959Z|00007|raft|INFO|starting to add server 5098 
(5098 at tcp:[253.255.0.33]:6644) to cluster 281c
2020-08-05T13:43:12.962Z|00008|raft|INFO|cluster 281c: installed 
snapshot on server 5098  up to 1:1
2020-08-05T13:43:12.963Z|00009|reconnect|INFO|tcp:[253.255.0.33]:6644: 
connecting...
2020-08-05T13:43:12.963Z|00010|reconnect|INFO|tcp:[253.255.0.33]:6644: 
connected
2020-08-05T13:43:12.963Z|00011|raft|INFO|adding 5098 (5098 at 
tcp:[253.255.0.33]:6644) to cluster 281c succeeded (completed)
2020-08-05T13:43:12.972Z|00012|jsonrpc|WARN|tcp:253.255.0.35:40706: 
receive error: Connection reset by peer
2020-08-05T13:43:12.972Z|00013|reconnect|WARN|tcp:253.255.0.35:40706: 
connection dropped (Connection reset by peer)
2020-08-05T13:43:13.027Z|00014|raft|INFO|tcp:253.255.0.35:44172: learned 
server ID b9de
2020-08-05T13:43:13.027Z|00015|raft|INFO|tcp:253.255.0.35:44172: learned 
remote address tcp:[253.255.0.35]:6644
2020-08-05T13:43:13.027Z|00016|raft|INFO|starting to add server b9de 
(b9de at tcp:[253.255.0.35]:6644) to cluster 281c
2020-08-05T13:43:13.029Z|00017|raft|INFO|cluster 281c: installed 
snapshot on server b9de  up to 1:1
2020-08-05T13:43:13.030Z|00018|reconnect|INFO|tcp:[253.255.0.35]:6644: 
connecting...
2020-08-05T13:43:13.030Z|00019|reconnect|INFO|tcp:[253.255.0.35]:6644: 
connected
2020-08-05T13:43:13.031Z|00020|raft|INFO|adding b9de (b9de at 
tcp:[253.255.0.35]:6644) to cluster 281c succeeded (completed)
2020-08-05T13:43:21.109Z|00021|memory|INFO|8604 kB peak resident set 
size after 10.0 seconds
2020-08-05T13:43:21.109Z|00022|memory|INFO|cells:39 monitors:0 
raft-connections:4

2020-08-05T13:45:52.338Z|00001|vlog|INFO|opened log file 
/usr/var/log/ovn/ovn-controller.log
2020-08-05T13:45:52.340Z|00002|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock: 
connecting...
2020-08-05T13:45:52.340Z|00003|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock: 
connected
2020-08-05T13:45:52.342Z|00004|main|INFO|OVS IDL reconnected, force 
recompute.
2020-08-05T13:45:52.342Z|00005|reconnect|INFO|tcp:253.255.0.35:6642: 
connecting...
2020-08-05T13:45:52.342Z|00006|main|INFO|OVNSB IDL reconnected, force 
recompute.
2020-08-05T13:45:52.342Z|00007|reconnect|INFO|tcp:253.255.0.35:6642: 
connected
2020-08-05T13:45:52.344Z|00008|ofctrl|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting to switch
2020-08-05T13:45:52.344Z|00009|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting...
2020-08-05T13:45:52.344Z|00010|rconn|WARN|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connection failed (No such file or directory)
2020-08-05T13:45:52.344Z|00011|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
waiting 1 seconds before reconnect
2020-08-05T13:45:53.344Z|00012|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting...
2020-08-05T13:45:53.345Z|00013|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connected
2020-08-05T13:45:53.346Z|00014|dpif_netlink|INFO|The kernel module does 
not support meters.
2020-08-05T13:45:53.348Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting to switch
2020-08-05T13:45:53.348Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting...
2020-08-05T13:45:53.348Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connected
2020-08-05T14:08:29.432Z|00015|main|INFO|OVNSB commit failed, force 
recompute next time.
2020-08-05T14:08:29.432Z|00016|jsonrpc|WARN|tcp:253.255.0.35:6642: 
receive error: Connection reset by peer
2020-08-05T14:08:29.432Z|00017|reconnect|WARN|tcp:253.255.0.35:6642: 
connection dropped (Connection reset by peer)
2020-08-05T14:08:29.432Z|00018|reconnect|INFO|tcp:253.255.0.33:6642: 
connecting...
2020-08-05T14:08:29.432Z|00019|reconnect|INFO|tcp:253.255.0.33:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:08:29.433Z|00020|reconnect|INFO|tcp:253.255.0.34:6642: 
connecting...
2020-08-05T14:08:29.433Z|00021|reconnect|INFO|tcp:253.255.0.34:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:08:30.434Z|00022|reconnect|INFO|tcp:253.255.0.35:6642: 
connecting...
2020-08-05T14:08:30.434Z|00023|reconnect|INFO|tcp:253.255.0.35:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:08:30.434Z|00024|reconnect|INFO|tcp:253.255.0.35:6642: 
waiting 2 seconds before reconnect
2020-08-05T14:08:32.436Z|00025|reconnect|INFO|tcp:253.255.0.33:6642: 
connecting...
2020-08-05T14:08:32.436Z|00026|reconnect|INFO|tcp:253.255.0.33:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:08:32.436Z|00027|reconnect|INFO|tcp:253.255.0.33:6642: 
waiting 4 seconds before reconnect
2020-08-05T14:08:36.439Z|00028|reconnect|INFO|tcp:253.255.0.34:6642: 
connecting...
2020-08-05T14:08:36.439Z|00029|reconnect|INFO|tcp:253.255.0.34:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:08:36.439Z|00030|reconnect|INFO|tcp:253.255.0.34:6642: 
continuing to reconnect in the background but suppressing further logging

k8 node 3
--------------
2020-08-05T13:44:52.007Z|00001|vlog|INFO|opened log file 
/var/log/ovn/ovsdb-server-nb.log
2020-08-05T13:44:52.009Z|00002|ovsdb_server|INFO|ovsdb-server (Open 
vSwitch) 2.13.90
2020-08-05T13:44:52.010Z|00003|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connecting...
2020-08-05T13:44:52.010Z|00004|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connected
2020-08-05T13:44:52.010Z|00005|raft_rpc|INFO|learned cluster ID 6e05
2020-08-05T13:44:52.010Z|00006|raft|INFO|tcp:[253.255.0.34]:6643: 
learned server ID c8a5
2020-08-05T13:44:52.011Z|00007|raft|INFO|server c8a5 is leader for term 1
2020-08-05T13:44:52.011Z|00008|raft|INFO|rejecting append_request 
because previous entry 1,6 not in local log (mismatch past end of log)
2020-08-05T13:44:52.013Z|00009|raft|INFO|server c8a5 added to configuration
2020-08-05T13:44:52.014Z|00010|raft|INFO|server 0d3d added to configuration
2020-08-05T13:44:52.014Z|00011|reconnect|INFO|tcp:[253.255.0.33]:6643: 
connecting...
2020-08-05T13:44:52.014Z|00012|reconnect|INFO|tcp:[253.255.0.33]:6643: 
connected
2020-08-05T13:44:52.015Z|00013|raft|INFO|server 941e added to configuration
2020-08-05T13:44:52.015Z|00014|raft|INFO|tcp:253.255.0.34:52448: learned 
server ID c8a5
2020-08-05T13:44:52.015Z|00015|raft|INFO|tcp:253.255.0.34:52448: learned 
remote address tcp:[253.255.0.34]:6643
2020-08-05T13:44:52.015Z|00016|raft|INFO|tcp:253.255.0.33:56762: learned 
server ID 0d3d
2020-08-05T13:44:52.015Z|00017|raft|INFO|tcp:253.255.0.33:56762: learned 
remote address tcp:[253.255.0.33]:6643
2020-08-05T13:45:02.010Z|00018|memory|INFO|8884 kB peak resident set 
size after 10.0 seconds
2020-08-05T13:45:02.010Z|00019|memory|INFO|cells:40 monitors:0 
raft-connections:4
2020-08-05T14:10:03.593Z|00020|raft|INFO|received leadership transfer 
from c8a5 in term 1
2020-08-05T14:10:03.593Z|00021|raft|INFO|term 2: starting election
2020-08-05T14:10:03.593Z|00022|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connection closed by peer
2020-08-05T14:10:03.594Z|00023|raft|INFO|term 2: elected leader by 2+ of 
3 servers
2020-08-05T14:10:04.594Z|00024|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connecting...
2020-08-05T14:10:04.594Z|00025|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connection attempt failed (Connection refused)
2020-08-05T14:10:04.594Z|00026|reconnect|INFO|tcp:[253.255.0.34]:6643: 
waiting 2 seconds before reconnect
2020-08-05T14:10:06.595Z|00027|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connecting...
2020-08-05T14:10:06.595Z|00028|reconnect|INFO|tcp:[253.255.0.34]:6643: 
connection attempt failed (Connection refused)
2020-08-05T14:10:06.595Z|00029|reconnect|INFO|tcp:[253.255.0.34]:6643: 
waiting 4 seconds before reconnect


cat /usr/var/log/ovn//backups/ovsdb-server-sb.13

2020-08-05T13:44:49.037Z|00001|vlog|INFO|opened log file 
/var/log/ovn/ovsdb-server-sb.log
2020-08-05T13:44:49.039Z|00002|ovsdb_server|INFO|ovsdb-server (Open 
vSwitch) 2.13.90
2020-08-05T13:44:49.040Z|00003|reconnect|INFO|tcp:[253.255.0.34]:6644: 
connecting...
2020-08-05T13:44:49.040Z|00004|reconnect|INFO|tcp:[253.255.0.34]:6644: 
connected
2020-08-05T13:44:49.040Z|00005|raft_rpc|INFO|learned cluster ID 281c
2020-08-05T13:44:49.040Z|00006|raft|INFO|tcp:[253.255.0.34]:6644: 
learned server ID eeed
2020-08-05T13:44:49.040Z|00007|raft|INFO|server eeed is leader for term 1
2020-08-05T13:44:49.040Z|00008|raft|INFO|rejecting append_request 
because previous entry 1,5 not in local log (mismatch past end of log)
2020-08-05T13:44:49.043Z|00009|raft|INFO|server eeed added to configuration
2020-08-05T13:44:49.043Z|00010|raft|INFO|server 5098 added to configuration
2020-08-05T13:44:49.043Z|00011|reconnect|INFO|tcp:[253.255.0.33]:6644: 
connecting...
2020-08-05T13:44:49.043Z|00012|reconnect|INFO|tcp:[253.255.0.33]:6644: 
connected
2020-08-05T13:44:49.044Z|00013|raft|INFO|server b9de added to configuration
2020-08-05T13:44:49.044Z|00014|raft|INFO|tcp:253.255.0.34:33550: learned 
server ID eeed
2020-08-05T13:44:49.044Z|00015|raft|INFO|tcp:253.255.0.34:33550: learned 
remote address tcp:[253.255.0.34]:6644
2020-08-05T13:44:49.044Z|00016|raft|INFO|tcp:253.255.0.33:58002: learned 
server ID 5098
2020-08-05T13:44:49.044Z|00017|raft|INFO|tcp:253.255.0.33:58002: learned 
remote address tcp:[253.255.0.33]:6644
2020-08-05T13:44:59.041Z|00018|memory|INFO|8652 kB peak resident set 
size after 10.0 seconds
2020-08-05T13:44:59.041Z|00019|memory|INFO|cells:39 monitors:0 
raft-connections:4
2020-08-05T14:10:03.593Z|00020|raft|INFO|received leadership transfer 
from eeed in term 1
2020-08-05T14:10:03.593Z|00021|raft|INFO|term 2: starting election
2020-08-05T14:10:03.593Z|00022|reconnect|INFO|tcp:[253.255.0.34]:6644: 
connection closed by peer
2020-08-05T14:10:03.594Z|00023|raft|INFO|term 2: elected leader by 2+ of 
3 servers
2020-08-05T14:10:04.594Z|00024|reconnect|INFO|tcp:[253.255.0.34]:6644: 
connecting...
2020-08-05T14:10:04.594Z|00025|reconnect|INFO|tcp:[253.255.0.34]:6644: 
connection attempt failed (Connection refused)
2020-08-05T14:10:04.594Z|00026|reconnect|INFO|tcp:[253.255.0.34]:6644: 
waiting 2 seconds before reconnect
2020-08-05T14:10:05.378Z|00027|reconnect|INFO|tcp:[253.255.0.33]:6644: 
connection closed by peer


2020-08-05T13:47:29.329Z|00001|vlog|INFO|opened log file 
/usr/var/log/ovn/ovn-controller.log
2020-08-05T13:47:29.331Z|00002|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock: 
connecting...
2020-08-05T13:47:29.331Z|00003|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock: 
connected
2020-08-05T13:47:29.334Z|00004|main|INFO|OVS IDL reconnected, force 
recompute.
2020-08-05T13:47:29.334Z|00005|reconnect|INFO|tcp:253.255.0.34:6642: 
connecting...
2020-08-05T13:47:29.334Z|00006|main|INFO|OVNSB IDL reconnected, force 
recompute.
2020-08-05T13:47:29.334Z|00007|reconnect|INFO|tcp:253.255.0.34:6642: 
connected
2020-08-05T13:47:29.336Z|00008|ofctrl|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting to switch
2020-08-05T13:47:29.336Z|00009|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting...
2020-08-05T13:47:29.336Z|00010|rconn|WARN|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connection failed (No such file or directory)
2020-08-05T13:47:29.336Z|00011|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
waiting 1 seconds before reconnect
2020-08-05T13:47:30.337Z|00012|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting...
2020-08-05T13:47:30.338Z|00013|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connected
2020-08-05T13:47:30.340Z|00014|dpif_netlink|INFO|The kernel module does 
not support meters.
2020-08-05T13:47:30.341Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting to switch
2020-08-05T13:47:30.341Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connecting...
2020-08-05T13:47:30.341Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: 
connected
2020-08-05T14:10:03.592Z|00015|main|INFO|OVNSB commit failed, force 
recompute next time.
2020-08-05T14:10:03.593Z|00016|jsonrpc|WARN|tcp:253.255.0.34:6642: 
receive error: Connection reset by peer
2020-08-05T14:10:03.593Z|00017|reconnect|WARN|tcp:253.255.0.34:6642: 
connection dropped (Connection reset by peer)
2020-08-05T14:10:03.593Z|00018|reconnect|INFO|tcp:253.255.0.33:6642: 
connecting...
2020-08-05T14:10:03.593Z|00019|reconnect|INFO|tcp:253.255.0.33:6642: 
connected
2020-08-05T14:10:05.378Z|00020|main|INFO|OVNSB commit failed, force 
recompute next time.
2020-08-05T14:10:05.378Z|00021|jsonrpc|WARN|tcp:253.255.0.33:6642: 
receive error: Connection reset by peer
2020-08-05T14:10:05.378Z|00022|reconnect|WARN|tcp:253.255.0.33:6642: 
connection dropped (Connection reset by peer)
2020-08-05T14:10:05.379Z|00023|reconnect|INFO|tcp:253.255.0.35:6642: 
connecting...
2020-08-05T14:10:05.379Z|00024|reconnect|INFO|tcp:253.255.0.35:6642: 
connected
2020-08-05T14:10:05.485Z|00025|main|INFO|OVNSB commit failed, force 
recompute next time.
2020-08-05T14:10:05.485Z|00026|jsonrpc|WARN|tcp:253.255.0.35:6642: 
receive error: Connection reset by peer
2020-08-05T14:10:05.485Z|00027|reconnect|WARN|tcp:253.255.0.35:6642: 
connection dropped (Connection reset by peer)
2020-08-05T14:10:06.486Z|00028|reconnect|INFO|tcp:253.255.0.34:6642: 
connecting...
2020-08-05T14:10:06.487Z|00029|reconnect|INFO|tcp:253.255.0.34:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:10:06.487Z|00030|reconnect|INFO|tcp:253.255.0.34:6642: 
waiting 2 seconds before reconnect
2020-08-05T14:10:08.488Z|00031|reconnect|INFO|tcp:253.255.0.33:6642: 
connecting...
2020-08-05T14:10:08.488Z|00032|reconnect|INFO|tcp:253.255.0.33:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:10:08.488Z|00033|reconnect|INFO|tcp:253.255.0.33:6642: 
waiting 4 seconds before reconnect
2020-08-05T14:10:12.492Z|00034|reconnect|INFO|tcp:253.255.0.35:6642: 
connecting...
2020-08-05T14:10:12.493Z|00035|reconnect|INFO|tcp:253.255.0.35:6642: 
connection attempt failed (Connection refused)
2020-08-05T14:10:12.493Z|00036|reconnect|INFO|tcp:253.255.0.35:6642: 
continuing to reconnect in the background but suppressing further logging





On 05/08/2020 20:54, Dumitru Ceara wrote:
> On 8/5/20 5:14 PM, Brendan Doyle wrote:
>> Folks,
>>
>> I'm stumped here, I have the k8 ovnkube-db-raft Stateful set up and
>> running.
>> But when I create a simple network, no SB flows are generated.
>>
>> ovn-nbctl show shows my network. ovn-sbctl show shows the physicals
>> systems in my network.
>> But I can't ping between any hosts because ovn-sbctl lflow-list is
>> empty, and there are no
>> errors or warnings in the logs. The ovn cluster says it is up and healthy.
>>
>> Anybody got any ideas why this might be?
>>
> Hi Brendan,
>
> Maybe I missed it but is ovn-northd running anywhere? If so, could you
> please share its logs?
>
> Thanks,
> Dumitru
>



More information about the discuss mailing list