[ovs-discuss] ovsdb cluster locking not working (*)

Matthew Booth mbooth at redhat.com
Fri Jun 19 16:43:11 UTC 2020


(*) ... as expected by me.

I'm currently attempting to configure a 3 node OVN control plane on
kubernetes. This consists of 3 pods, each running a nb, sb, and
northd. I expected only 1 of the northds to be active at any one time
as (IIUC) they are supposed to hold a northd lock in the south db.
However, I can see from the logs that they are all active. All 3
northds report:

2020-06-19T15:53:20Z|00009|ovn_northd|INFO|ovn-northd lock acquired.
This ovn-northd instance is now active.

I executed the following on 2 different nodes:

$ ovsdb-client lock unix:/pod-run/ovnnb_db.sock foo_lock

Both locks were granted concurrently. I can confirm the same behaviour
when connecting over tcp, except when connecting to the same db node
over tcp. I would assume that the nodes are not clustered correctly,
but they appear to be:

# ovs-appctl -t /pod-run/ovnsb_db.ctl cluster/status OVN_Southbound
5c61
Name: OVN_Southbound
Cluster ID: aa50 (aa505b7b-f0c4-4219-b357-bf7d0e2b79e9)
Server ID: 5c61 (5c6196d8-6a08-4b55-bbd9-b039d57a8e6d)
Address: tcp:10.131.0.12:6644
Status: cluster member
Role: candidate
Term: 3
Leader: 4cba
Vote: self

Log: [2, 8]
Entries not yet committed: 0
Entries not yet applied: 0
Connections: ->4cba ->7283 <-7283 <-4cba
Servers:
    5c61 (5c61 at tcp:10.131.0.12:6644) (self) (voted for 5c61)
    4cba (4cba at tcp:10.128.2.15:6644) (voted for 4cba)
    7283 (7283 at tcp:10.129.2.18:6644)

The dbs are launched slightly differently on node 0 from nodes 1 and 2
to account for bootstrapping. The bootstrapping node runs:

            exec /usr/share/openvswitch/scripts/ovn-ctl \
            --no-monitor \
            --db-nb-create-insecure-remote=yes \
            --db-nb-cluster-local-addr="${LOCAL_IP}" \
            --db-nb-cluster-local-proto=tcp \
            --ovn-nb-log="-vconsole:${OVN_LOG_LEVEL} -vfile:off" \
            run_sb_ovsdb

The others run:

            exec /usr/share/openvswitch/scripts/ovn-ctl \
            --no-monitor \
            --db-nb-create-insecure-remote=yes \
            --db-nb-cluster-remote-addr="${BOOTSTRAP_IP}" \
            --db-nb-cluster-local-addr="${LOCAL_IP}" \
            --db-nb-cluster-loca-proto=tcp \
            --db-nb-cluster-remote-proto=tcp \
            --ovn-nb-log="-vconsole:${OVN_LOG_LEVEL} -vfile:off" \
            run_sb_ovsdb

The north db is the same, and northd in each pod connects the local
unix sockets for both north and south dbs.

OVS_RUNDIR=/pod-run and OVS_DBDIR=/var/lib/openvswitch in all cases.

Any idea what's going on here?

Thanks,

Matt
-- 
Matthew Booth
Red Hat OpenStack Engineer, Compute DFG

Phone: +442070094448 (UK)



More information about the discuss mailing list