[ovs-discuss] SB flows not being created in OVN K8 Stateful set

Thu Aug 6 12:03:59 UTC 2020

On 06/08/2020 12:31, Dumitru Ceara wrote:
> On 8/6/20 11:54 AM, Brendan Doyle wrote:
>> I don't see any ovn-northd.log log, I only see those when I'm running
>> OVN outside the k8s cluster.
>> Before I start the Satefulset on my k8 nodes I run:
>>
>> ovn-ctl stop_northd
>> ovn-ctl stop_ovsdb
>> rm -rf /usr/etc/ovn/*.db
>>
>>
>> The only logs I see are ovn-controller.log  (I'm running ovn controller
>> on the K8 nodes) ovsdb-server-nb.log ovsdb-server-sb.log
>> These logs look normal
>>
> ovn-northd is the daemon that translates OVN_Northbound DB records to
> OVN_Southbound DB records (including logical flows). If your deployment
> doesn't start ovn-northd SB won't get populated.

Well I use the ovnkube-db-raft.yaml and ovn-setup.yaml as per the 
documentation, and they call
ovndb-raft-functions.sh and start the OVN procs as:

run_as_ovs_user_if_needed \
       ${OVNCTL_PATH} run_${db}_ovsdb --no-monitor \
       --db-${db}-cluster-local-addr=[${ovn_db_host}] \
       --db-${db}-cluster-local-port=${raft_port} \
       --db-${db}-cluster-local-proto=${transport} \
       ${db_ssl_opts} \
       --ovn-${db}-log="${ovn_loglevel_db}" &

Are you saying that there is more to do?? as in invoke run-ovn-northd() 
in ovnkube.sh which is called
from the ovnkube-master.yaml file? I see that 
https://github.com/ovn-org/ovn-kubernetes says

Apply OVN DaemonSet and Deployment yamls.

|# Create OVN namespace, service accounts, ovnkube-db headless service, 
configmap, and policies kubectl create -f 
$HOME/work/src/github.com/ovn-org/ovn-kubernetes/dist/yaml/ovn-setup.yaml 
# Run ovnkube-db deployment. kubectl create -f 
$HOME/work/src/github.com/ovn-org/ovn-kubernetes/dist/yaml/ovnkube-db.yaml 
# Run ovnkube-master deployment. kubectl create -f 
$HOME/work/src/github.com/ovn-org/ovn-kubernetes/dist/yaml/ovnkube-master.yaml 
# Run ovnkube daemonset for nodes kubectl create -f 
$HOME/work/src/github.com/ovn-org/ovn-kubernetes/dist/yaml/ovnkube-node.yaml 
|

But makes no mention of ovnkube-db-raft.yaml ?? so is the above is the 
correct procedure?
what then is ovnkube-db-raft.yaml for?

Note I don't want to replace flannel with OVN as the CNI, I just want to 
run OVN central in a k8
StatefulSet, that use flannel as the CNI.

Is it documented anywhere how I can do this? I would have thought that 
ovnkube-db-raft.yaml
was the way to go, and that it would start all that is needed??

Do I need to run all of the above yamls too, or just a subset? and will 
they interfere with ovnkube-db-raft.yaml
and/or flannel? Do I need to just drop ovnkube-db-raft.yaml???

Can someone provide the exact yamls I need to do this please

Thanks

Brendan.

> Regards,
> Dumitru
>
>> k8 node 1 logs
>> ============
>> 2020-08-05T13:44:51.958Z|00001|vlog|INFO|opened log file
>> /var/log/ovn/ovsdb-server-nb.log
>> 2020-08-05T13:44:51.961Z|00002|ovsdb_server|INFO|ovsdb-server (Open
>> vSwitch) 2.13.90
>> 2020-08-05T13:44:51.961Z|00003|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connecting...
>> 2020-08-05T13:44:51.961Z|00004|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connected
>> 2020-08-05T13:44:51.962Z|00005|raft_rpc|INFO|learned cluster ID 6e05
>> 2020-08-05T13:44:51.962Z|00006|raft|INFO|tcp:[253.255.0.34]:6643:
>> learned server ID c8a5
>> 2020-08-05T13:44:51.962Z|00007|raft|INFO|server c8a5 is leader for term 1
>> 2020-08-05T13:44:51.962Z|00008|raft|INFO|rejecting append_request
>> because previous entry 1,5 not in local log (mismatch past end of log)
>> 2020-08-05T13:44:51.964Z|00009|raft|INFO|server c8a5 added to configuration
>> 2020-08-05T13:44:51.966Z|00010|raft|INFO|server 0d3d added to configuration
>> 2020-08-05T13:44:51.966Z|00011|raft|INFO|tcp:253.255.0.34:50744: learned
>> server ID c8a5
>> 2020-08-05T13:44:51.966Z|00012|raft|INFO|tcp:253.255.0.34:50744: learned
>> remote address tcp:[253.255.0.34]:6643
>> 2020-08-05T13:44:52.015Z|00013|raft|INFO|tcp:253.255.0.35:51780: learned
>> server ID 941e
>> 2020-08-05T13:44:52.015Z|00014|raft|INFO|tcp:253.255.0.35:51780: learned
>> remote address tcp:[253.255.0.35]:6643
>> 2020-08-05T13:44:52.015Z|00015|raft|INFO|adding 941e (941e at
>> tcp:[253.255.0.35]:6643) to cluster 6e05 failed (not leader)
>> 2020-08-05T13:44:52.015Z|00016|raft|INFO|server 941e added to configuration
>> 2020-08-05T13:44:52.016Z|00017|reconnect|INFO|tcp:[253.255.0.35]:6643:
>> connecting...
>> 2020-08-05T13:44:52.016Z|00018|reconnect|INFO|tcp:[253.255.0.35]:6643:
>> connected
>> 2020-08-05T13:45:01.962Z|00019|memory|INFO|8424 kB peak resident set
>> size after 10.0 seconds
>> 2020-08-05T13:45:01.962Z|00020|memory|INFO|cells:40 monitors:0
>> raft-connections:4
>> 2020-08-05T14:10:03.593Z|00021|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connection closed by peer
>> 2020-08-05T14:10:03.595Z|00022|raft|INFO|server 941e is leader for term 2
>> 2020-08-05T14:10:04.594Z|00023|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connecting...
>> 2020-08-05T14:10:04.594Z|00024|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:04.594Z|00025|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> waiting 2 seconds before reconnect
>> 2020-08-05T14:10:06.594Z|00026|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connecting...
>> 2020-08-05T14:10:06.594Z|00027|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:06.594Z|00028|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> waiting 4 seconds before reconnect
>> 2020-08-05T14:10:08.453Z|00029|raft|INFO|received leadership transfer
>> from 941e in term 2
>> 2020-08-05T14:10:08.453Z|00030|raft|INFO|term 3: starting election
>> 2020-08-05T14:10:08.453Z|00031|reconnect|INFO|tcp:[253.255.0.35]:6643:
>> connection closed by peer
>>
>> 2020-08-05T13:44:48.971Z|00001|vlog|INFO|opened log file
>> /var/log/ovn/ovsdb-server-sb.log
>> 2020-08-05T13:44:48.973Z|00002|ovsdb_server|INFO|ovsdb-server (Open
>> vSwitch) 2.13.90
>> 2020-08-05T13:44:48.973Z|00003|reconnect|INFO|tcp:[253.255.0.34]:6644:
>> connecting...
>> 2020-08-05T13:44:48.973Z|00004|reconnect|INFO|tcp:[253.255.0.34]:6644:
>> connected
>> 2020-08-05T13:44:48.973Z|00005|raft_rpc|INFO|learned cluster ID 281c
>> 2020-08-05T13:44:48.973Z|00006|raft|INFO|tcp:[253.255.0.34]:6644:
>> learned server ID eeed
>> 2020-08-05T13:44:48.974Z|00007|raft|INFO|server eeed is leader for term 1
>> 2020-08-05T13:44:48.974Z|00008|raft|INFO|rejecting append_request
>> because previous entry 1,4 not in local log (mismatch past end of log)
>> 2020-08-05T13:44:48.976Z|00009|raft|INFO|server eeed added to configuration
>> 2020-08-05T13:44:48.977Z|00010|raft|INFO|server 5098 added to configuration
>> 2020-08-05T13:44:48.977Z|00011|raft|INFO|tcp:253.255.0.34:50628: learned
>> server ID eeed
>> 2020-08-05T13:44:48.977Z|00012|raft|INFO|tcp:253.255.0.34:50628: learned
>> remote address tcp:[253.255.0.34]:6644
>> 2020-08-05T13:44:49.044Z|00013|raft|INFO|tcp:253.255.0.35:49594: learned
>> server ID b9de
>> 2020-08-05T13:44:49.044Z|00014|raft|INFO|tcp:253.255.0.35:49594: learned
>> remote address tcp:[253.255.0.35]:6644
>> 2020-08-05T13:44:49.044Z|00015|raft|INFO|adding b9de (b9de at
>> tcp:[253.255.0.35]:6644) to cluster 281c failed (not leader)
>> 2020-08-05T13:44:49.044Z|00016|raft|INFO|server b9de added to configuration
>> 2020-08-05T13:44:49.044Z|00017|reconnect|INFO|tcp:[253.255.0.35]:6644:
>> connecting...
>> 2020-08-05T13:44:49.045Z|00018|reconnect|INFO|tcp:[253.255.0.35]:6644:
>> connected
>> 2020-08-05T13:44:58.975Z|00019|memory|INFO|8408 kB peak resident set
>> size after 10.0 seconds
>> 2020-08-05T13:44:58.975Z|00020|memory|INFO|cells:39 monitors:0
>> raft-connections:4
>> 2020-08-05T14:10:03.593Z|00021|reconnect|INFO|tcp:[253.255.0.34]:6644:
>> connection closed by peer
>> 2020-08-05T14:10:03.595Z|00022|raft|INFO|server b9de is leader for term 2
>> 2020-08-05T14:10:04.595Z|00023|reconnect|INFO|tcp:[253.255.0.34]:6644:
>> connecting...
>> 2020-08-05T14:10:04.595Z|00024|reconnect|INFO|tcp:[253.255.0.34]:6644:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:04.595Z|00025|reconnect|INFO|tcp:[253.255.0.34]:6644:
>> waiting 2 seconds before reconnect
>>
>> 2020-08-05T13:47:27.231Z|00001|vlog|INFO|opened log file
>> /usr/var/log/ovn/ovn-controller.log
>> 2020-08-05T13:47:27.233Z|00002|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock:
>> connecting...
>> 2020-08-05T13:47:27.233Z|00003|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock:
>> connected
>> 2020-08-05T13:47:27.235Z|00004|main|INFO|OVS IDL reconnected, force
>> recompute.
>> 2020-08-05T13:47:27.236Z|00005|reconnect|INFO|tcp:253.255.0.35:6642:
>> connecting...
>> 2020-08-05T13:47:27.236Z|00006|main|INFO|OVNSB IDL reconnected, force
>> recompute.
>> 2020-08-05T13:47:27.236Z|00007|reconnect|INFO|tcp:253.255.0.35:6642:
>> connected
>> 2020-08-05T13:47:27.237Z|00008|ofctrl|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting to switch
>> 2020-08-05T13:47:27.237Z|00009|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting...
>> 2020-08-05T13:47:27.237Z|00010|rconn|WARN|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connection failed (No such file or directory)
>> 2020-08-05T13:47:27.237Z|00011|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> waiting 1 seconds before reconnect
>> 2020-08-05T13:47:28.239Z|00012|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting...
>> 2020-08-05T13:47:28.239Z|00013|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connected
>> 2020-08-05T13:47:28.240Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting to switch
>> 2020-08-05T13:47:28.240Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting...
>> 2020-08-05T13:47:28.240Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connected
>> 2020-08-05T13:47:28.436Z|00014|dpif_netlink|INFO|The kernel module does
>> not support meters.
>> 2020-08-05T14:10:05.486Z|00015|main|INFO|OVNSB commit failed, force
>> recompute next time.
>> 2020-08-05T14:10:05.486Z|00016|jsonrpc|WARN|tcp:253.255.0.35:6642:
>> receive error: Connection reset by peer
>> 2020-08-05T14:10:05.486Z|00017|reconnect|WARN|tcp:253.255.0.35:6642:
>> connection dropped (Connection reset by peer)
>> 2020-08-05T14:10:05.486Z|00018|reconnect|INFO|tcp:253.255.0.33:6642:
>> connecting...
>> 2020-08-05T14:10:05.486Z|00019|reconnect|INFO|tcp:253.255.0.33:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:05.487Z|00020|reconnect|INFO|tcp:253.255.0.34:6642:
>> connecting...
>> 2020-08-05T14:10:05.487Z|00021|reconnect|INFO|tcp:253.255.0.34:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:06.488Z|00022|reconnect|INFO|tcp:253.255.0.35:6642:
>> connecting...
>> 2020-08-05T14:10:06.488Z|00023|reconnect|INFO|tcp:253.255.0.35:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:06.488Z|00024|reconnect|INFO|tcp:253.255.0.35:6642:
>> waiting 2 seconds before reconnect
>> 2020-08-05T14:10:08.490Z|00025|reconnect|INFO|tcp:253.255.0.33:6642:
>> connecting...
>> 2020-08-05T14:10:08.490Z|00026|reconnect|INFO|tcp:253.255.0.33:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:08.490Z|00027|reconnect|INFO|tcp:253.255.0.33:6642:
>> waiting 4 seconds before reconnect
>> 2020-08-05T14:10:12.492Z|00028|reconnect|INFO|tcp:253.255.0.34:6642:
>> connecting...
>> 2020-08-05T14:10:12.492Z|00029|reconnect|INFO|tcp:253.255.0.34:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:12.492Z|00030|reconnect|INFO|tcp:253.255.0.34:6642:
>> continuing to reconnect in the background but suppressing further logging
>>
>> K8 node 2
>> ---------------
>> 2020-08-05T13:43:11.094Z|00001|vlog|INFO|opened log file
>> /var/log/ovn/ovsdb-server-nb.log
>> 2020-08-05T13:43:11.095Z|00002|raft|INFO|term 1: 40130328487 ms timeout
>> expired, starting election
>> 2020-08-05T13:43:11.095Z|00003|raft|INFO|term 1: elected leader by 1+ of
>> 1 servers
>> 2020-08-05T13:43:11.098Z|00004|ovsdb_server|INFO|ovsdb-server (Open
>> vSwitch) 2.13.90
>> 2020-08-05T13:43:12.074Z|00005|jsonrpc|WARN|unix#4: send error: Broken pipe
>> 2020-08-05T13:43:12.074Z|00006|reconnect|WARN|unix#4: connection dropped
>> (Broken pipe)
>> 2020-08-05T13:43:15.942Z|00007|jsonrpc|WARN|tcp:253.255.0.35:60448:
>> receive error: Connection reset by peer
>> 2020-08-05T13:43:15.942Z|00008|reconnect|WARN|tcp:253.255.0.35:60448:
>> connection dropped (Connection reset by peer)
>> 2020-08-05T13:43:15.947Z|00009|raft|INFO|tcp:253.255.0.33:46572: learned
>> server ID 0d3d
>> 2020-08-05T13:43:15.947Z|00010|raft|INFO|tcp:253.255.0.33:46572: learned
>> remote address tcp:[253.255.0.33]:6643
>> 2020-08-05T13:43:15.947Z|00011|raft|INFO|starting to add server 0d3d
>> (0d3d at tcp:[253.255.0.33]:6643) to cluster 6e05
>> 2020-08-05T13:43:15.950Z|00012|raft|INFO|cluster 6e05: installed
>> snapshot on server 0d3d  up to 1:1
>> 2020-08-05T13:43:15.951Z|00013|reconnect|INFO|tcp:[253.255.0.33]:6643:
>> connecting...
>> 2020-08-05T13:43:15.952Z|00014|reconnect|INFO|tcp:[253.255.0.33]:6643:
>> connected
>> 2020-08-05T13:43:15.952Z|00015|raft|INFO|adding 0d3d (0d3d at
>> tcp:[253.255.0.33]:6643) to cluster 6e05 succeeded (completed)
>> 2020-08-05T13:43:15.997Z|00016|raft|INFO|tcp:253.255.0.35:60500: learned
>> server ID 941e
>> 2020-08-05T13:43:15.997Z|00017|raft|INFO|tcp:253.255.0.35:60500: learned
>> remote address tcp:[253.255.0.35]:6643
>> 2020-08-05T13:43:15.997Z|00018|raft|INFO|starting to add server 941e
>> (941e at tcp:[253.255.0.35]:6643) to cluster 6e05
>> 2020-08-05T13:43:16.000Z|00019|raft|INFO|cluster 6e05: installed
>> snapshot on server 941e  up to 1:1
>> 2020-08-05T13:43:16.001Z|00020|reconnect|INFO|tcp:[253.255.0.35]:6643:
>> connecting...
>> 2020-08-05T13:43:16.001Z|00021|reconnect|INFO|tcp:[253.255.0.35]:6643:
>> connected
>> 2020-08-05T13:43:16.002Z|00022|raft|INFO|adding 941e (941e at
>> tcp:[253.255.0.35]:6643) to cluster 6e05 succeeded (completed)
>> 2020-08-05T13:43:21.101Z|00023|memory|INFO|8452 kB peak resident set
>> size after 10.0 seconds
>> 2020-08-05T13:43:21.101Z|00024|memory|INFO|cells:40 monitors:0
>> raft-connections:4
>>
>> 2020-08-05T13:43:11.105Z|00001|vlog|INFO|opened log file
>> /var/log/ovn/ovsdb-server-sb.log
>> 2020-08-05T13:43:11.106Z|00002|raft|INFO|term 1: 40130328497 ms timeout
>> expired, starting election
>> 2020-08-05T13:43:11.106Z|00003|raft|INFO|term 1: elected leader by 1+ of
>> 1 servers
>> 2020-08-05T13:43:11.108Z|00004|ovsdb_server|INFO|ovsdb-server (Open
>> vSwitch) 2.13.90
>> 2020-08-05T13:43:12.959Z|00005|raft|INFO|tcp:253.255.0.33:46140: learned
>> server ID 5098
>> 2020-08-05T13:43:12.959Z|00006|raft|INFO|tcp:253.255.0.33:46140: learned
>> remote address tcp:[253.255.0.33]:6644
>> 2020-08-05T13:43:12.959Z|00007|raft|INFO|starting to add server 5098
>> (5098 at tcp:[253.255.0.33]:6644) to cluster 281c
>> 2020-08-05T13:43:12.962Z|00008|raft|INFO|cluster 281c: installed
>> snapshot on server 5098  up to 1:1
>> 2020-08-05T13:43:12.963Z|00009|reconnect|INFO|tcp:[253.255.0.33]:6644:
>> connecting...
>> 2020-08-05T13:43:12.963Z|00010|reconnect|INFO|tcp:[253.255.0.33]:6644:
>> connected
>> 2020-08-05T13:43:12.963Z|00011|raft|INFO|adding 5098 (5098 at
>> tcp:[253.255.0.33]:6644) to cluster 281c succeeded (completed)
>> 2020-08-05T13:43:12.972Z|00012|jsonrpc|WARN|tcp:253.255.0.35:40706:
>> receive error: Connection reset by peer
>> 2020-08-05T13:43:12.972Z|00013|reconnect|WARN|tcp:253.255.0.35:40706:
>> connection dropped (Connection reset by peer)
>> 2020-08-05T13:43:13.027Z|00014|raft|INFO|tcp:253.255.0.35:44172: learned
>> server ID b9de
>> 2020-08-05T13:43:13.027Z|00015|raft|INFO|tcp:253.255.0.35:44172: learned
>> remote address tcp:[253.255.0.35]:6644
>> 2020-08-05T13:43:13.027Z|00016|raft|INFO|starting to add server b9de
>> (b9de at tcp:[253.255.0.35]:6644) to cluster 281c
>> 2020-08-05T13:43:13.029Z|00017|raft|INFO|cluster 281c: installed
>> snapshot on server b9de  up to 1:1
>> 2020-08-05T13:43:13.030Z|00018|reconnect|INFO|tcp:[253.255.0.35]:6644:
>> connecting...
>> 2020-08-05T13:43:13.030Z|00019|reconnect|INFO|tcp:[253.255.0.35]:6644:
>> connected
>> 2020-08-05T13:43:13.031Z|00020|raft|INFO|adding b9de (b9de at
>> tcp:[253.255.0.35]:6644) to cluster 281c succeeded (completed)
>> 2020-08-05T13:43:21.109Z|00021|memory|INFO|8604 kB peak resident set
>> size after 10.0 seconds
>> 2020-08-05T13:43:21.109Z|00022|memory|INFO|cells:39 monitors:0
>> raft-connections:4
>>
>> 2020-08-05T13:45:52.338Z|00001|vlog|INFO|opened log file
>> /usr/var/log/ovn/ovn-controller.log
>> 2020-08-05T13:45:52.340Z|00002|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock:
>> connecting...
>> 2020-08-05T13:45:52.340Z|00003|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock:
>> connected
>> 2020-08-05T13:45:52.342Z|00004|main|INFO|OVS IDL reconnected, force
>> recompute.
>> 2020-08-05T13:45:52.342Z|00005|reconnect|INFO|tcp:253.255.0.35:6642:
>> connecting...
>> 2020-08-05T13:45:52.342Z|00006|main|INFO|OVNSB IDL reconnected, force
>> recompute.
>> 2020-08-05T13:45:52.342Z|00007|reconnect|INFO|tcp:253.255.0.35:6642:
>> connected
>> 2020-08-05T13:45:52.344Z|00008|ofctrl|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting to switch
>> 2020-08-05T13:45:52.344Z|00009|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting...
>> 2020-08-05T13:45:52.344Z|00010|rconn|WARN|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connection failed (No such file or directory)
>> 2020-08-05T13:45:52.344Z|00011|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> waiting 1 seconds before reconnect
>> 2020-08-05T13:45:53.344Z|00012|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting...
>> 2020-08-05T13:45:53.345Z|00013|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connected
>> 2020-08-05T13:45:53.346Z|00014|dpif_netlink|INFO|The kernel module does
>> not support meters.
>> 2020-08-05T13:45:53.348Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting to switch
>> 2020-08-05T13:45:53.348Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting...
>> 2020-08-05T13:45:53.348Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connected
>> 2020-08-05T14:08:29.432Z|00015|main|INFO|OVNSB commit failed, force
>> recompute next time.
>> 2020-08-05T14:08:29.432Z|00016|jsonrpc|WARN|tcp:253.255.0.35:6642:
>> receive error: Connection reset by peer
>> 2020-08-05T14:08:29.432Z|00017|reconnect|WARN|tcp:253.255.0.35:6642:
>> connection dropped (Connection reset by peer)
>> 2020-08-05T14:08:29.432Z|00018|reconnect|INFO|tcp:253.255.0.33:6642:
>> connecting...
>> 2020-08-05T14:08:29.432Z|00019|reconnect|INFO|tcp:253.255.0.33:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:08:29.433Z|00020|reconnect|INFO|tcp:253.255.0.34:6642:
>> connecting...
>> 2020-08-05T14:08:29.433Z|00021|reconnect|INFO|tcp:253.255.0.34:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:08:30.434Z|00022|reconnect|INFO|tcp:253.255.0.35:6642:
>> connecting...
>> 2020-08-05T14:08:30.434Z|00023|reconnect|INFO|tcp:253.255.0.35:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:08:30.434Z|00024|reconnect|INFO|tcp:253.255.0.35:6642:
>> waiting 2 seconds before reconnect
>> 2020-08-05T14:08:32.436Z|00025|reconnect|INFO|tcp:253.255.0.33:6642:
>> connecting...
>> 2020-08-05T14:08:32.436Z|00026|reconnect|INFO|tcp:253.255.0.33:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:08:32.436Z|00027|reconnect|INFO|tcp:253.255.0.33:6642:
>> waiting 4 seconds before reconnect
>> 2020-08-05T14:08:36.439Z|00028|reconnect|INFO|tcp:253.255.0.34:6642:
>> connecting...
>> 2020-08-05T14:08:36.439Z|00029|reconnect|INFO|tcp:253.255.0.34:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:08:36.439Z|00030|reconnect|INFO|tcp:253.255.0.34:6642:
>> continuing to reconnect in the background but suppressing further logging
>>
>> k8 node 3
>> --------------
>> 2020-08-05T13:44:52.007Z|00001|vlog|INFO|opened log file
>> /var/log/ovn/ovsdb-server-nb.log
>> 2020-08-05T13:44:52.009Z|00002|ovsdb_server|INFO|ovsdb-server (Open
>> vSwitch) 2.13.90
>> 2020-08-05T13:44:52.010Z|00003|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connecting...
>> 2020-08-05T13:44:52.010Z|00004|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connected
>> 2020-08-05T13:44:52.010Z|00005|raft_rpc|INFO|learned cluster ID 6e05
>> 2020-08-05T13:44:52.010Z|00006|raft|INFO|tcp:[253.255.0.34]:6643:
>> learned server ID c8a5
>> 2020-08-05T13:44:52.011Z|00007|raft|INFO|server c8a5 is leader for term 1
>> 2020-08-05T13:44:52.011Z|00008|raft|INFO|rejecting append_request
>> because previous entry 1,6 not in local log (mismatch past end of log)
>> 2020-08-05T13:44:52.013Z|00009|raft|INFO|server c8a5 added to configuration
>> 2020-08-05T13:44:52.014Z|00010|raft|INFO|server 0d3d added to configuration
>> 2020-08-05T13:44:52.014Z|00011|reconnect|INFO|tcp:[253.255.0.33]:6643:
>> connecting...
>> 2020-08-05T13:44:52.014Z|00012|reconnect|INFO|tcp:[253.255.0.33]:6643:
>> connected
>> 2020-08-05T13:44:52.015Z|00013|raft|INFO|server 941e added to configuration
>> 2020-08-05T13:44:52.015Z|00014|raft|INFO|tcp:253.255.0.34:52448: learned
>> server ID c8a5
>> 2020-08-05T13:44:52.015Z|00015|raft|INFO|tcp:253.255.0.34:52448: learned
>> remote address tcp:[253.255.0.34]:6643
>> 2020-08-05T13:44:52.015Z|00016|raft|INFO|tcp:253.255.0.33:56762: learned
>> server ID 0d3d
>> 2020-08-05T13:44:52.015Z|00017|raft|INFO|tcp:253.255.0.33:56762: learned
>> remote address tcp:[253.255.0.33]:6643
>> 2020-08-05T13:45:02.010Z|00018|memory|INFO|8884 kB peak resident set
>> size after 10.0 seconds
>> 2020-08-05T13:45:02.010Z|00019|memory|INFO|cells:40 monitors:0
>> raft-connections:4
>> 2020-08-05T14:10:03.593Z|00020|raft|INFO|received leadership transfer
>> from c8a5 in term 1
>> 2020-08-05T14:10:03.593Z|00021|raft|INFO|term 2: starting election
>> 2020-08-05T14:10:03.593Z|00022|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connection closed by peer
>> 2020-08-05T14:10:03.594Z|00023|raft|INFO|term 2: elected leader by 2+ of
>> 3 servers
>> 2020-08-05T14:10:04.594Z|00024|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connecting...
>> 2020-08-05T14:10:04.594Z|00025|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:04.594Z|00026|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> waiting 2 seconds before reconnect
>> 2020-08-05T14:10:06.595Z|00027|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connecting...
>> 2020-08-05T14:10:06.595Z|00028|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:06.595Z|00029|reconnect|INFO|tcp:[253.255.0.34]:6643:
>> waiting 4 seconds before reconnect
>>
>>
>> cat /usr/var/log/ovn//backups/ovsdb-server-sb.13
>>
>> 2020-08-05T13:44:49.037Z|00001|vlog|INFO|opened log file
>> /var/log/ovn/ovsdb-server-sb.log
>> 2020-08-05T13:44:49.039Z|00002|ovsdb_server|INFO|ovsdb-server (Open
>> vSwitch) 2.13.90
>> 2020-08-05T13:44:49.040Z|00003|reconnect|INFO|tcp:[253.255.0.34]:6644:
>> connecting...
>> 2020-08-05T13:44:49.040Z|00004|reconnect|INFO|tcp:[253.255.0.34]:6644:
>> connected
>> 2020-08-05T13:44:49.040Z|00005|raft_rpc|INFO|learned cluster ID 281c
>> 2020-08-05T13:44:49.040Z|00006|raft|INFO|tcp:[253.255.0.34]:6644:
>> learned server ID eeed
>> 2020-08-05T13:44:49.040Z|00007|raft|INFO|server eeed is leader for term 1
>> 2020-08-05T13:44:49.040Z|00008|raft|INFO|rejecting append_request
>> because previous entry 1,5 not in local log (mismatch past end of log)
>> 2020-08-05T13:44:49.043Z|00009|raft|INFO|server eeed added to configuration
>> 2020-08-05T13:44:49.043Z|00010|raft|INFO|server 5098 added to configuration
>> 2020-08-05T13:44:49.043Z|00011|reconnect|INFO|tcp:[253.255.0.33]:6644:
>> connecting...
>> 2020-08-05T13:44:49.043Z|00012|reconnect|INFO|tcp:[253.255.0.33]:6644:
>> connected
>> 2020-08-05T13:44:49.044Z|00013|raft|INFO|server b9de added to configuration
>> 2020-08-05T13:44:49.044Z|00014|raft|INFO|tcp:253.255.0.34:33550: learned
>> server ID eeed
>> 2020-08-05T13:44:49.044Z|00015|raft|INFO|tcp:253.255.0.34:33550: learned
>> remote address tcp:[253.255.0.34]:6644
>> 2020-08-05T13:44:49.044Z|00016|raft|INFO|tcp:253.255.0.33:58002: learned
>> server ID 5098
>> 2020-08-05T13:44:49.044Z|00017|raft|INFO|tcp:253.255.0.33:58002: learned
>> remote address tcp:[253.255.0.33]:6644
>> 2020-08-05T13:44:59.041Z|00018|memory|INFO|8652 kB peak resident set
>> size after 10.0 seconds
>> 2020-08-05T13:44:59.041Z|00019|memory|INFO|cells:39 monitors:0
>> raft-connections:4
>> 2020-08-05T14:10:03.593Z|00020|raft|INFO|received leadership transfer
>> from eeed in term 1
>> 2020-08-05T14:10:03.593Z|00021|raft|INFO|term 2: starting election
>> 2020-08-05T14:10:03.593Z|00022|reconnect|INFO|tcp:[253.255.0.34]:6644:
>> connection closed by peer
>> 2020-08-05T14:10:03.594Z|00023|raft|INFO|term 2: elected leader by 2+ of
>> 3 servers
>> 2020-08-05T14:10:04.594Z|00024|reconnect|INFO|tcp:[253.255.0.34]:6644:
>> connecting...
>> 2020-08-05T14:10:04.594Z|00025|reconnect|INFO|tcp:[253.255.0.34]:6644:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:04.594Z|00026|reconnect|INFO|tcp:[253.255.0.34]:6644:
>> waiting 2 seconds before reconnect
>> 2020-08-05T14:10:05.378Z|00027|reconnect|INFO|tcp:[253.255.0.33]:6644:
>> connection closed by peer
>>
>>
>> 2020-08-05T13:47:29.329Z|00001|vlog|INFO|opened log file
>> /usr/var/log/ovn/ovn-controller.log
>> 2020-08-05T13:47:29.331Z|00002|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock:
>> connecting...
>> 2020-08-05T13:47:29.331Z|00003|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock:
>> connected
>> 2020-08-05T13:47:29.334Z|00004|main|INFO|OVS IDL reconnected, force
>> recompute.
>> 2020-08-05T13:47:29.334Z|00005|reconnect|INFO|tcp:253.255.0.34:6642:
>> connecting...
>> 2020-08-05T13:47:29.334Z|00006|main|INFO|OVNSB IDL reconnected, force
>> recompute.
>> 2020-08-05T13:47:29.334Z|00007|reconnect|INFO|tcp:253.255.0.34:6642:
>> connected
>> 2020-08-05T13:47:29.336Z|00008|ofctrl|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting to switch
>> 2020-08-05T13:47:29.336Z|00009|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting...
>> 2020-08-05T13:47:29.336Z|00010|rconn|WARN|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connection failed (No such file or directory)
>> 2020-08-05T13:47:29.336Z|00011|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> waiting 1 seconds before reconnect
>> 2020-08-05T13:47:30.337Z|00012|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting...
>> 2020-08-05T13:47:30.338Z|00013|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connected
>> 2020-08-05T13:47:30.340Z|00014|dpif_netlink|INFO|The kernel module does
>> not support meters.
>> 2020-08-05T13:47:30.341Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting to switch
>> 2020-08-05T13:47:30.341Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connecting...
>> 2020-08-05T13:47:30.341Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt:
>> connected
>> 2020-08-05T14:10:03.592Z|00015|main|INFO|OVNSB commit failed, force
>> recompute next time.
>> 2020-08-05T14:10:03.593Z|00016|jsonrpc|WARN|tcp:253.255.0.34:6642:
>> receive error: Connection reset by peer
>> 2020-08-05T14:10:03.593Z|00017|reconnect|WARN|tcp:253.255.0.34:6642:
>> connection dropped (Connection reset by peer)
>> 2020-08-05T14:10:03.593Z|00018|reconnect|INFO|tcp:253.255.0.33:6642:
>> connecting...
>> 2020-08-05T14:10:03.593Z|00019|reconnect|INFO|tcp:253.255.0.33:6642:
>> connected
>> 2020-08-05T14:10:05.378Z|00020|main|INFO|OVNSB commit failed, force
>> recompute next time.
>> 2020-08-05T14:10:05.378Z|00021|jsonrpc|WARN|tcp:253.255.0.33:6642:
>> receive error: Connection reset by peer
>> 2020-08-05T14:10:05.378Z|00022|reconnect|WARN|tcp:253.255.0.33:6642:
>> connection dropped (Connection reset by peer)
>> 2020-08-05T14:10:05.379Z|00023|reconnect|INFO|tcp:253.255.0.35:6642:
>> connecting...
>> 2020-08-05T14:10:05.379Z|00024|reconnect|INFO|tcp:253.255.0.35:6642:
>> connected
>> 2020-08-05T14:10:05.485Z|00025|main|INFO|OVNSB commit failed, force
>> recompute next time.
>> 2020-08-05T14:10:05.485Z|00026|jsonrpc|WARN|tcp:253.255.0.35:6642:
>> receive error: Connection reset by peer
>> 2020-08-05T14:10:05.485Z|00027|reconnect|WARN|tcp:253.255.0.35:6642:
>> connection dropped (Connection reset by peer)
>> 2020-08-05T14:10:06.486Z|00028|reconnect|INFO|tcp:253.255.0.34:6642:
>> connecting...
>> 2020-08-05T14:10:06.487Z|00029|reconnect|INFO|tcp:253.255.0.34:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:06.487Z|00030|reconnect|INFO|tcp:253.255.0.34:6642:
>> waiting 2 seconds before reconnect
>> 2020-08-05T14:10:08.488Z|00031|reconnect|INFO|tcp:253.255.0.33:6642:
>> connecting...
>> 2020-08-05T14:10:08.488Z|00032|reconnect|INFO|tcp:253.255.0.33:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:08.488Z|00033|reconnect|INFO|tcp:253.255.0.33:6642:
>> waiting 4 seconds before reconnect
>> 2020-08-05T14:10:12.492Z|00034|reconnect|INFO|tcp:253.255.0.35:6642:
>> connecting...
>> 2020-08-05T14:10:12.493Z|00035|reconnect|INFO|tcp:253.255.0.35:6642:
>> connection attempt failed (Connection refused)
>> 2020-08-05T14:10:12.493Z|00036|reconnect|INFO|tcp:253.255.0.35:6642:
>> continuing to reconnect in the background but suppressing further logging
>>
>>
>>
>>
>>
>> On 05/08/2020 20:54, Dumitru Ceara wrote:
>>> On 8/5/20 5:14 PM, Brendan Doyle wrote:
>>>> Folks,
>>>>
>>>> I'm stumped here, I have the k8 ovnkube-db-raft Stateful set up and
>>>> running.
>>>> But when I create a simple network, no SB flows are generated.
>>>>
>>>> ovn-nbctl show shows my network. ovn-sbctl show shows the physicals
>>>> systems in my network.
>>>> But I can't ping between any hosts because ovn-sbctl lflow-list is
>>>> empty, and there are no
>>>> errors or warnings in the logs. The ovn cluster says it is up and
>>>> healthy.
>>>>
>>>> Anybody got any ideas why this might be?
>>>>
>>> Hi Brendan,
>>>
>>> Maybe I missed it but is ovn-northd running anywhere? If so, could you
>>> please share its logs?
>>>
>>> Thanks,
>>> Dumitru
>>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200806/1e2dedf8/attachment-0001.html>