[ovs-dev] [PATCH v3] ovn-ctl: Support starting clustered OVN dbs

aginwala aginwala at asu.edu
Wed Apr 4 19:28:55 UTC 2018


Thanks Ben and Numan for upgrade feature and further optimizations.

Sorry couldn't test revised patch by you guys as I am doing some
simulations in my scale test to see how the cluster is behaving with more
HVs. Adding 1k HVs worked well even-though I intentionally kept killing old
and new leaders in the middle of chassis registration. Planning to do it
for 10k lports today for overlay setup. I hope to get some findings soon
with some intentional cluster crashing when ports are getting added.

However, I have one more finding here in the existing test:
So if I give different remote either for nb/sb while restoring a crashed
node, it still remembers the old remote IP. Hence, I delete dbs from all
three nodes and re-create the cluster. So need to handle the way to update
the remote IP in the old cluster.

e.g. I am using new remote 192.168.220.151. However, it still remembers the
old remote 192.168.220.101

/usr/share/openvswitch/scripts/ovn-ctl --db-nb-addr=192.168.220.103
--db-nb-port=6641 --db-nb-cluster-local-addr=tcp:192.168.220.103:6645
--db-nb-cluster-remote-addr="tcp:192.168.220.151:6645"
--db-nb-create-insecure-remote=yes start_nb_ovsdb

018-03-23T01:48:59.117Z|00013|reconnect|INFO|tcp:192.168.220.101:6645:
connecting...
2018-03-23T01:49:02.112Z|00014|reconnect|INFO|tcp:192.168.220.101:6645:
connection attempt failed (No route to host)
2018-03-23T01:49:02.113Z|00015|reconnect|INFO|tcp:192.168.220.101:6645:
continuing to reconnect in the background but suppressing further logging


Will test upgrade patch after this soon. :)


Regards,
Aliasgar

On Wed, Apr 4, 2018 at 11:32 AM, Numan Siddique <nusiddiq at redhat.com> wrote:

> On Wed, Apr 4, 2018 at 10:51 PM, Ben Pfaff <blp at ovn.org> wrote:
>
> > Hi Numan, thanks for the new version.
> >
> > Did you (or someone else) re-test this with the changes that I made?  I
> > did not test them myself, so I have low confidence in them.  The
> > upgrade_cluster() function especially needs testing.
> >
> > Thanks,
> >
> >
> Hi Ben,
>
> I did testing with upgrade_cluster and it worked fine. But I found one
> issue.
>
> In my testing, I stopped all the servers in my 3 node cluster. Upgraded ovn
> db schema by adding a tst column in logical_switch table and updated the
> ovs and ovn rpms in all the 3 nodes.
> Then when I started the cluster on node 1,  all the services were started,
> but I saw the below error
>
> *******
> 2018-04-04T14:28:13Z|00002|ovsdb_idl|WARN|Logical_Switch table in
> OVN_Northbound database lacks tst2 column (database needs upgrade?)
> Waiting for OVN_Northbound to come up                      [  OK  ]
> Upgrading database OVN_Northbound from schema version 5.10.2 to 5.10.3
> 2018-04-04T14:28:13Z|00001|ovsdb|WARN|/usr/share/
> openvswitch/ovn-nb.ovsschema:
> changed 2 columns in 'OVN_Northbound' database from ephemeral to
> persistent, including 'status' column in 'Connection' table, because
> clusters do not support ephemeral columns
> 2018-04-04T14:28:43Z|00002|fatal_signal|WARN|terminating with signal 14
> (Alarm clock)
> /usr/share/openvswitch/scripts/ovs-lib: line 600: 18504 Alarm clock
>      "$@"
>                                                            [FAILED]
> 2018-04-04T14:28:43Z|00002|ovsdb_idl|WARN|Logical_Switch table in
> OVN_Northbound database lacks tst2 column (database needs upgrade?)
> Waiting for OVN_Southbound to come up                      [  OK  ]
> Starting ovn-northd                                        [  OK  ]
>
> [root at vm1 vagrant]# ovn-nbctl show
> 2018-04-04T14:29:31Z|00001|ovsdb_idl|WARN|Logical_Switch table in
> OVN_Northbound database lacks tst2 column (database needs upgrade?)
> switch b5a0e5d3-2587-4f15-a3c4-8262fb20c65d (sw0)
> ********
>
> Even though I saw the above warning when I ran "ovn-nbctl show", everything
> worked fine. When I ran "ovn-nbctl list logical_switch" I could see the new
> column "tst".
>  And when I started the ovsdb-servers on the other nodes, the above warning
> went away.
>
> I also tested by restarting the ovsdb-servers on all the nodes almost at
> the same time and I didn't notice the message
> "/usr/share/openvswitch/scripts/ovs-lib:
> line 600: 18504 Alarm clock ..."  by ovn-ctl.
>
>
> Other than this I didn't notice anything odd. I was thinking to share these
> details tomorrow :).
>
> It would definitely help if some one could test this patch out.
>
> Thanks
> Numan
>
>
> Ben.
> >
> > On Wed, Apr 04, 2018 at 09:56:54PM +0530, nusiddiq at redhat.com wrote:
> > > From: Numan Siddique <nusiddiq at redhat.com>
> > >
> > > This patch adds the options to start clustered OVN db servers in
> ovn-ctl.
> > > To support this, following options are added -
> > '--db-(nb/sb)-cluster-local-addr',
> > > '--db-(nb/sb)-cluster-local-port', '--db-(nb/sb)-cluster-local-proto',
> > > '--db-(nb/sb)-cluster-remote-addr', '--db-(nb/sb)-cluster-remote-port'
> > and
> > > '--db-(nb/sb)-cluster-remote-proto'.
> > >
> > > If only '--db-(nb/sb)-cluster-local-addr' is defined then clustered db
> > is created
> > > (using ovsdb-tool create-cluster). If both are defined, then the db is
> > added to
> > > the cluster (using ovsdb-tool join-cluster)
> > >
> > > This patch also adds the support to configure ovn-northd to point to
> all
> > the servers
> > > in the cluster using the options - '--ovn-northd-nb-db' and
> > 'ovn-northd-sb-db'.
> > >
> > > Presently this patch doesn't handle the schema update scenario when
> > restarting the
> > > clustered ovsdb-servers. This will be handled in a separate patch.
> > >
> > > The initial versions of these commands are tested by Aliasgar
> <aginwala <
> > aginwala at asu.edu>
> > > and the discussion on this can be found here -
> > > https://mail.openvswitch.org/pipermail/ovs-discuss/2018-
> > March/046470.html
> > >
> > > (There are 4 checkpatch warnings 'Line length is >79-characters long'
> in
> > ovn-ctl.8.xml
> > > which I couldn't resolve without losing proper rendering when "man
> > ovn-ctl" is run.)
> > >
> > > Co-authored-by: Ben Pfaff <blp at ovn.org>
> > > Signed-off-by: Numan Siddique <nusiddiq at redhat.com>
> > > Signed-off-by: Ben Pfaff <blp at ovn.org>
> > > ---
> > >  ovn/utilities/ovn-ctl       | 191 +++++++++++++++++++++++++++++-
> > --------------
> > >  ovn/utilities/ovn-ctl.8.xml |  71 ++++++++++++++++
> > >  utilities/ovs-lib.in        |  62 ++++++++++++--
> > >  3 files changed, 255 insertions(+), 69 deletions(-)
> > >
> > > diff --git a/ovn/utilities/ovn-ctl b/ovn/utilities/ovn-ctl
> > > index dc0c26159..25dda524c 100755
> > > --- a/ovn/utilities/ovn-ctl
> > > +++ b/ovn/utilities/ovn-ctl
> > > @@ -93,84 +93,103 @@ promote_ovnsb() {
> > >      ovs-appctl -t $rundir/ovnsb_db.ctl ovsdb-server/disconnect-
> > active-ovsdb-server
> > >  }
> > >
> > > -start_nb_ovsdb() {
> > > -    # Check and eventually start ovsdb-server for Northbound DB
> > > -    if ! pidfile_is_running $DB_NB_PID; then
> > > -        upgrade_db "$DB_NB_FILE" "$DB_NB_SCHEMA" 1>/dev/null
> 2>/dev/null
> > > +start_ovsdb__() {
> > > +    local DB=$1 db=$2 schema_name=$3 table_name=$4
> > > +    eval local pid=\$DB_${DB}_PID
> > > +    eval local cluster_local_addr=\$DB_${DB}_CLUSTER_LOCAL_ADDR
> > > +    eval local cluster_local_port=\$DB_${DB}_CLUSTER_LOCAL_PORT
> > > +    eval local cluster_local_proto=\$DB_${DB}_CLUSTER_LOCAL_PROTO
> > > +    eval local cluster_remote_addr=\$DB_${DB}_CLUSTER_REMOTE_ADDR
> > > +    eval local cluster_remote_port=\$DB_${DB}_CLUSTER_REMOTE_PORT
> > > +    eval local cluster_remote_proto=\$DB_${DB}_CLUSTER_REMOTE_PROTO
> > > +    eval local sync_from_proto=\$DB_${DB}_SYNC_FROM_PROTO
> > > +    eval local sync_from_addr=\$DB_${DB}_SYNC_FROM_ADDR
> > > +    eval local sync_from_port=\$DB_${DB}_SYNC_FROM_PORT
> > > +    eval local file=\$DB_${DB}_FILE
> > > +    eval local schema=\$DB_${DB}_SCHEMA
> > > +    eval local logfile=\$OVN_${DB}_LOGFILE
> > > +    eval local log=\$OVN_${DB}_LOG
> > > +    eval local sock=\$DB_${DB}_SOCK
> > > +    eval local detach=\$DB_${DB}_DETACH
> > > +    eval local create_insecure_remote=\$DB_${
> DB}_CREATE_INSECURE_REMOTE
> > > +    eval local port=\$DB_${DB}_PORT
> > > +    eval local addr=\$DB_${DB}_ADDR
> > > +    eval local active_conf_file=\$ovn${db}_active_conf_file
> > > +
> > > +    # Check and eventually start ovsdb-server for DB
> > > +    if pidfile_is_running $pid; then
> > > +        return
> > > +    fi
> > >
> > > -        set ovsdb-server
> > > +    if test ! -z "$cluster_local_addr"; then
> > > +        mode=cluster
> > > +    elif test ! -z "$sync_from_addr"; then
> > > +        mode=active_passive
> > > +        echo "$sync_from_proto:$sync_from_addr:\
> > > +$sync_from_port" > $active_conf_file
> > > +    else
> > > +        mode=standalone
> > > +    fi
> > >
> > > -        if test X"$DB_NB_DETACH" != Xno; then
> > > -            set "$@" --detach --monitor
> > > +    if test $mode = cluster; then
> > > +        local local=$cluster_local_proto:$cluster_local_addr:\
> > > +$cluster_local_port
> > > +        local remote=$cluster_remote_proto:$cluster_remote_addr:\
> > > +$cluster_remote_port
> > > +        if test -n "$cluster_remote_addr"; then
> > > +            join_cluster "$file" "$schema_name" "$local" "$remote"
> > >          else
> > > -            set exec "$@"
> > > +            create_cluster "$file" "$schema" "$local"
> > >          fi
> > > +    else
> > > +        upgrade_db "$file" "$schema"
> > > +    fi
> > >
> > > -        set "$@" $OVN_NB_LOG --log-file=$OVN_NB_LOGFILE
> > > -        set "$@" --remote=punix:$DB_NB_SOCK --pidfile=$DB_NB_PID
> > > -        set "$@" --remote=db:OVN_Northbound,NB_Global,connections
> > > -        set "$@" --unixctl=ovnnb_db.ctl
> > > -        set "$@" --private-key=db:OVN_Northbound,SSL,private_key
> > > -        set "$@" --certificate=db:OVN_Northbound,SSL,certificate
> > > -        set "$@" --ca-cert=db:OVN_Northbound,SSL,ca_cert
> > > -        set "$@" --ssl-protocols=db:OVN_Northbound,SSL,ssl_protocols
> > > -        set "$@" --ssl-ciphers=db:OVN_Northbound,SSL,ssl_ciphers
> > > -
> > > -        if test X"$DB_NB_CREATE_INSECURE_REMOTE" = Xyes; then
> > > -            set "$@" --remote=ptcp:$DB_NB_PORT:$DB_NB_ADDR
> > > -        fi
> > > +    set ovsdb-server
> > > +    set "$@" $log --log-file=$logfile
> > > +    set "$@" --remote=punix:$sock --pidfile=$pid
> > > +    set "$@" --unixctl=ovn${db}_db.ctl
> > >
> > > -        if test ! -z "$DB_NB_SYNC_FROM_ADDR"; then
> > > -            echo "$DB_NB_SYNC_FROM_PROTO:$DB_
> > NB_SYNC_FROM_ADDR:$DB_NB_SYNC_FROM_PORT" > $ovnnb_active_conf_file
> > > -        fi
> > > +    if test X"$detach" != Xno; then
> > > +        set "$@" --detach --monitor
> > > +    else
> > > +        set exec "$@"
> > > +    fi
> > >
> > > -        if test -e $ovnnb_active_conf_file; then
> > > -            set "$@" --sync-from=`cat $ovnnb_active_conf_file`
> > > -        fi
> > > +    set "$@" --remote=db:$schema_name,$table_name,connections
> > > +    set "$@" --private-key=db:$schema_name,SSL,private_key
> > > +    set "$@" --certificate=db:$schema_name,SSL,certificate
> > > +    set "$@" --ca-cert=db:$schema_name,SSL,ca_cert
> > > +    set "$@" --ssl-protocols=db:$schema_name,SSL,ssl_protocols
> > > +    set "$@" --ssl-ciphers=db:$schema_name,SSL,ssl_ciphers
> > >
> > > -        $@ $DB_NB_FILE
> > > -        ovn-nbctl init
> > > +    if test X"$create_insecure_remote" = Xyes; then
> > > +        set "$@" --remote=ptcp:$port:$addr
> > >      fi
> > > -}
> > >
> > > -start_sb_ovsdb() {
> > > -    # Check and eventually start ovsdb-server for Southbound DB
> > > -    if ! pidfile_is_running $DB_SB_PID; then
> > > -        upgrade_db "$DB_SB_FILE" "$DB_SB_SCHEMA" 1>/dev/null
> 2>/dev/null
> > > -
> > > -        set ovsdb-server
> > > +    if test $mode = active_passive; then
> > > +        set "$@" --sync-from=`cat $active_conf_file`
> > > +    fi
> > >
> > > -        if test X"$DB_SB_DETACH" != Xno; then
> > > -            set "$@" --detach --monitor
> > > -        else
> > > -            set exec "$@"
> > > -        fi
> > > +    "$@" "$file"
> > >
> > > -        set "$@" $OVN_SB_LOG --log-file=$OVN_SB_LOGFILE
> > > -        set "$@" --remote=punix:$DB_SB_SOCK --pidfile=$DB_SB_PID
> > > -        set "$@" --remote=db:OVN_Southbound,SB_Global,connections
> > > -        set "$@" --unixctl=ovnsb_db.ctl
> > > -        set "$@" --private-key=db:OVN_Southbound,SSL,private_key
> > > -        set "$@" --certificate=db:OVN_Southbound,SSL,certificate
> > > -        set "$@" --ca-cert=db:OVN_Southbound,SSL,ca_cert
> > > -        set "$@" --ssl-protocols=db:OVN_Southbound,SSL,ssl_protocols
> > > -        set "$@" --ssl-ciphers=db:OVN_Southbound,SSL,ssl_ciphers
> > > -
> > > -        if test X"$DB_SB_CREATE_INSECURE_REMOTE" = Xyes; then
> > > -            set "$@" --remote=ptcp:$DB_SB_PORT:$DB_SB_ADDR
> > > -        fi
> > > +    # Initialize the database if it's running standalone,
> > > +    # active-passive, or is the first server in a cluster.
> > > +    if test -z "$cluster_remote_addr"; then
> > > +        ovn-nbctl init
> > > +    fi
> > >
> > > -        if test ! -z "$DB_SB_SYNC_FROM_ADDR"; then
> > > -            echo "$DB_SB_SYNC_FROM_PROTO:$DB_
> > SB_SYNC_FROM_ADDR:$DB_SB_SYNC_FROM_PORT" > $ovnsb_active_conf_file
> > > -        fi
> > > +    if test $mode = cluster; then
> > > +        upgrade_cluster "$schema" "unix:$sock"
> > > +    fi
> > > +}
> > >
> > > -        if test -e $ovnsb_active_conf_file; then
> > > -            set "$@" --sync-from=`cat $ovnsb_active_conf_file`
> > > -        fi
> > > +start_nb_ovsdb() {
> > > +    start_ovsdb__ NB nb OVN_Northbound NB_Global
> > > +}
> > >
> > > -        $@ $DB_SB_FILE
> > > -        ovn-sbctl init
> > > -    fi
> > > +start_sb_ovsdb() {
> > > +    start_ovsdb__ SB sb OVN_Southbound SB_Global
> > >  }
> > >
> > >  start_ovsdb () {
> > > @@ -236,7 +255,8 @@ start_northd () {
> > >                  exit
> > >              fi
> > >          fi
> > > -        ovn_northd_params="--ovnnb-db=unix:$DB_NB_SOCK
> > --ovnsb-db=unix:$DB_SB_SOCK"
> > > +        ovn_northd_params="--ovnnb-db=$OVN_NORTHD_NB_DB \
> > > +        --ovnsb-db=$OVN_NORTHD_SB_DB"
> > >      else
> > >          ovn_northd_params="`cat $ovn_northd_db_conf_file`"
> > >      fi
> > > @@ -406,6 +426,23 @@ set_defaults () {
> > >
> > >      DB_NB_DETACH="yes"
> > >      DB_SB_DETACH="yes"
> > > +
> > > +    DB_NB_CLUSTER_LOCAL_ADDR=""
> > > +    DB_NB_CLUSTER_LOCAL_PROTO="tcp"
> > > +    DB_NB_CLUSTER_LOCAL_PORT=6643
> > > +    DB_NB_CLUSTER_REMOTE_ADDR=""
> > > +    DB_NB_CLUSTER_REMOTE_PROTO="tcp"
> > > +    DB_NB_CLUSTER_REMOTE_PORT=6643
> > > +
> > > +    DB_SB_CLUSTER_LOCAL_ADDR=""
> > > +    DB_SB_CLUSTER_LOCAL_PROTO="tcp"
> > > +    DB_SB_CLUSTER_LOCAL_PORT=6644
> > > +    DB_SB_CLUSTER_REMOTE_ADDR=""
> > > +    DB_SB_CLUSTER_REMOTE_PROTO="tcp"
> > > +    DB_SB_CLUSTER_REMOTE_PORT=6644
> > > +
> > > +    OVN_NORTHD_NB_DB="unix:$DB_NB_SOCK"
> > > +    OVN_NORTHD_SB_DB="unix:$DB_SB_SOCK"
> > >  }
> > >
> > >  set_option () {
> > > @@ -494,6 +531,32 @@ File location options:
> > >    --db-sb-sync-from-port=ADDR OVN Southbound active db tcp port
> > (default: $DB_SB_SYNC_FROM_PORT)
> > >    --db-sb-sync-from-proto=PROTO OVN Southbound active db transport
> > (default: $DB_SB_SYNC_FROM_PROTO)
> > >    --db-sb-create-insecure-remote=yes|no Create ptcp OVN Southbound
> > remote (default: $DB_SB_CREATE_INSECURE_REMOTE)
> > > +  --db-nb-cluster-local-addr=ADDR OVN_Northbound cluster local
> address
> > \
> > > +  (default: $DB_NB_CLUSTER_LOCAL_ADDR)
> > > +  --db-nb-cluster-local-port=PORT OVN_Northbound cluster local tcp
> > port \
> > > +  (default: $DB_NB_CLUSTER_LOCAL_PORT)
> > > +  --db-nb-cluster-local-proto=PROTO OVN_Northbound cluster local db
> > transport \
> > > +  (default: $DB_NB_CLUSTER_LOCAL_PROTO)
> > > +  --db-nb-cluster-remote-addr=ADDR OVN_Northbound cluster remote
> > address \
> > > +  (default: $DB_NB_CLUSTER_REMOTE_ADDR)
> > > +  --db-nb-cluster-remote-port=PORT OVN_Northbound cluster remote tcp
> > port \
> > > +  (default: $DB_NB_CLUSTER_REMOTE_PORT)
> > > +  --db-nb-cluster-remote-proto=PROTO OVN_Northbound cluster remote
> db \
> > > +  transport (default: $DB_NB_CLUSTER_REMOTE_PROTO)
> > > +  --db-sb-cluster-local-addr=ADDR OVN_Southbound cluster local
> address
> > \
> > > +  (default: $DB_SB_CLUSTER_LOCAL_ADDR)
> > > +  --db-sb-cluster-local-port=PORT OVN_Southbound cluster local tcp
> > port \
> > > +  (default: $DB_SB_CLUSTER_LOCAL_PORT)
> > > +  --db-sb-cluster-local-proto=PROTO OVN_Southbound cluster local db
> > transport \
> > > +  (default: $DB_SB_CLUSTER_LOCAL_PROTO)
> > > +  --db-sb-cluster-remote-addr=ADDR OVN_Southbound cluster remote
> > address \
> > > +  (default: $DB_SB_CLUSTER_REMOTE_ADDR)
> > > +  --db-sb-cluster-remote-port=PORT OVN_Southbound cluster remote tcp
> > port \
> > > +  (default: $DB_SB_CLUSTER_REMOTE_PORT)
> > > +  --db-sb-cluster-remote-proto=PROTO OVN_Southbound cluster remote
> db \
> > > +  transport (default: $DB_SB_CLUSTER_REMOTE_PROTO)
> > > +  --ovn-northd-nb-db=NB DB address(es) (default: $OVN_NORTHD_NB_DB)
> > > +  --ovn-northd-sb-db=SB DB address(es) (default: $OVN_NORTHD_SB_DB)
> > >
> > >  Default directories with "configure" option and environment variable
> > override:
> > >    logs: /usr/local/var/log/openvswitch (--with-logdir, OVS_LOGDIR)
> > > diff --git a/ovn/utilities/ovn-ctl.8.xml b/ovn/utilities/ovn-ctl.8.xml
> > > index 40defc9ec..02235fe1e 100644
> > > --- a/ovn/utilities/ovn-ctl.8.xml
> > > +++ b/ovn/utilities/ovn-ctl.8.xml
> > > @@ -66,6 +66,31 @@
> > >      <p><code>--db-sb-sync-from-addr=<var>IP ADDRESS</var></code></p>
> > >      <p><code>--db-sb-sync-from-port=<var>PORT NUMBER</var></code></p>
> > >      <p><code>--db-sb-sync-from-proto=<var>PROTO</var></code></p>
> > > +    <p>
> > > +      <code>
> > > +        --ovn-northd-nb-db=<var>PROTO</var>:<var>IP ADDRESS</var>:
> > > +        <var>PORT</var>..
> > > +      </code>
> > > +    </p>
> > > +    <p>
> > > +      <code>
> > > +        --ovn-northd-sb-db=<var>PROTO</var>:<var>IP ADDRESS</var>:
> > > +        <var>PORT</var>..
> > > +      </code>
> > > +    </p>
> > > +    <h1> Clustering options </h1>
> > > +    <p><code>--db-nb-cluster-local-addr=<var>IP
> > ADDRESS</var></code></p>
> > > +    <p><code>--db-nb-cluster-local-port=<var>PORT
> > NUMBER</var></code></p>
> > > +    <p><code>--db-nb-cluster-local-proto=<var>PROTO
> > (tcp/ssl)</var></code></p>
> > > +    <p><code>--db-nb-cluster-remote-addr=<var>IP
> > ADDRESS</var></code></p>
> > > +    <p><code>--db-nb-cluster-remote-port=<var>PORT
> > NUMBER</var></code></p>
> > > +    <p><code>--db-nb-cluster-remote-proto=<var>PROTO
> > (tcp/ssl)</var></code></p>
> > > +    <p><code>--db-sb-cluster-local-addr=<var>IP
> > ADDRESS</var></code></p>
> > > +    <p><code>--db-sb-cluster-local-port=<var>PORT
> > NUMBER</var></code></p>
> > > +    <p><code>--db-sb-cluster-local-proto=<var>PROTO
> > (tcp/ssl)</var></code></p>
> > > +    <p><code>--db-sb-cluster-remote-addr=<var>IP
> > ADDRESS</var></code></p>
> > > +    <p><code>--db-sb-cluster-remote-port=<var>PORT
> > NUMBER</var></code></p>
> > > +    <p><code>--db-sb-cluster-remote-proto=<var>PROTO
> > (tcp/ssl)</var></code></p>
> > >
> > >      <h1>Configuration files</h1>
> > >      <p>Following are the optional configuration files. If present, it
> > should be located in the etc dir</p>
> > > @@ -125,4 +150,50 @@
> > >      <p><code># ovn-ctl promote_ovnsb</code></p>
> > >      <p><code># ovn-ctl --db-nb-sync-from-addr=x.x.x.x
> > --db-nb-sync-from-port=6641 demote_ovnnb</code></p>
> > >      <p><code># ovn-ctl --db-sb-sync-from-addr=x.x.x.x
> > --db-sb-sync-from-port=6642 demote_ovnsb</code></p>
> > > +
> > > +    <h2>Creating a clustered db on 3 nodes with IPs x.x.x.x, y.y.y.y
> > and z.z.z.z</h2>
> > > +    <h3>Starting OVN ovsdb servers and ovn-northd on the node with IP
> > x.x.x.x</h3>
> > > +    <p>
> > > +      <code>
> > > +        # ovn-ctl --db-nb-addr=x.x.x.x --db-nb-create-insecure-
> > remote=yes
> > > +        --db-sb-addr=x.x.x.x --db-sb-create-insecure-remote=yes
> > > +        --db-nb-cluster-local-addr=x.x.x.x
> > > +        --db-sb-cluster-local-addr=x.x.x.x
> > > +        --ovn-northd-nb-db=tcp:x.x.x.x:6641,tcp:y.y.y.y:6641,tcp:z.
> > z.z.z:6641
> > > +        --ovn-northd-sb-db=tcp:x.x.x.x:6642,tcp:y.y.y.y:6642,tcp:z.
> > z.z.z:6642
> > > +        start_northd
> > > +      </code>
> > > +    </p>
> > > +
> > > +    <h3>Starting OVN ovsdb-servers and ovn-northd on the node with IP
> > y.y.y.y and joining the cluster started at x.x.x.x</h3>
> > > +    <p>
> > > +      <code>
> > > +        # ovn-ctl --db-nb-addr=y.y.y.y --db-nb-create-insecure-
> > remote=yes
> > > +        --db-sb-addr=y.y.y.y --db-sb-create-insecure-remote=yes
> > > +        --db-nb-cluster-local-addr=y.y.y.y
> > > +        --db-sb-cluster-local-addr=y.y.y.y
> > > +        --db-nb-cluster-remote-addr=x.x.x.x
> > > +        --db-sb-cluster-remote-addr=x.x.x.x
> > > +        --ovn-northd-nb-db=tcp:x.x.x.x:6641,tcp:y.y.y.y:6641,tcp:z.
> > z.z.z:6641
> > > +        --ovn-northd-sb-db=tcp:x.x.x.x:6642,tcp:y.y.y.y:6642,tcp:z.
> > z.z.z:6642
> > > +        start_northd
> > > +      </code>
> > > +    </p>
> > > +
> > > +    <h3>Starting OVN ovsdb-servers and ovn-northd on the node with IP
> > z.z.z.z and joining the cluster started at x.x.x.x</h3>
> > > +    <p>
> > > +      <code>
> > > +        # ovn-ctl --db-nb-addr=z.z.z.z
> > > +          --db-nb-create-insecure-remote=yes
> > > +          --db-nb-cluster-local-addr=z.z.z.z
> > > +          --db-sb-addr=z.z.z.z
> > > +          --db-sb-create-insecure-remote=yes
> > > +          --db-sb-cluster-local-addr=z.z.z.z
> > > +          --db-nb-cluster-remote-addr=x.x.x.x
> > > +          --db-sb-cluster-remote-addr=x.x.x.x
> > > +          --ovn-northd-nb-db=tcp:x.x.x.x:6641,tcp:y.y.y.y:6641,tcp:z.
> > z.z.z:6641
> > > +          --ovn-northd-sb-db=tcp:x.x.x.x:6642,tcp:y.y.y.y:6642,tcp:z.
> > z.z.z:6642
> > > +          start_northd
> > > +      </code>
> > > +    </p>
> > >  </manpage>
> > > diff --git a/utilities/ovs-lib.in b/utilities/ovs-lib.in
> > > index 4b08f0f53..4dc315118 100644
> > > --- a/utilities/ovs-lib.in
> > > +++ b/utilities/ovs-lib.in
> > > @@ -401,6 +401,14 @@ create_db () {
> > >      action "Creating empty database $DB_FILE" ovsdb_tool create
> > "$DB_FILE" "$DB_SCHEMA"
> > >  }
> > >
> > > +backup_db () {
> > > +    # Back up the old version.
> > > +    version=`ovsdb_tool db-version "$DB_FILE"`
> > > +    cksum=`ovsdb_tool db-cksum "$DB_FILE" | awk '{print $1}'`
> > > +    backup=$DB_FILE.backup$version-$cksum
> > > +    action "Backing up database to $backup" cp "$DB_FILE" "$backup" ||
> > return 1
> > > +}
> > > +
> > >  upgrade_db () {
> > >      DB_FILE="$1"
> > >      DB_SCHEMA="$2"
> > > @@ -411,11 +419,7 @@ upgrade_db () {
> > >          install_dir `dirname $DB_FILE`
> > >          create_db "$DB_FILE" "$DB_SCHEMA"
> > >      elif test X"`ovsdb_tool needs-conversion "$DB_FILE" "$DB_SCHEMA"`"
> > = Xyes; then
> > > -        # Back up the old version.
> > > -        version=`ovsdb_tool db-version "$DB_FILE"`
> > > -        cksum=`ovsdb_tool db-cksum "$DB_FILE" | awk '{print $1}'`
> > > -        backup=$DB_FILE.backup$version-$cksum
> > > -        action "Backing up database to $backup" cp "$DB_FILE"
> "$backup"
> > || return 1
> > > +        backup_db || return 1
> > >
> > >          # Compact database.  This is important if the old schema did
> > not enable
> > >          # garbage collection (i.e. if it did not have any tables with
> > "isRoot":
> > > @@ -443,6 +447,54 @@ upgrade_db () {
> > >      fi
> > >  }
> > >
> > > +upgrade_cluster () {
> > > +    local DB_SCHEMA=$1 DB_SERVER=$2
> > > +    local schema_name=$(ovsdb-tool schema-name $1) || return 1
> > > +
> > > +    action "Waiting for $schema_name to come up" ovsdb-client -t 30
> > wait "$DB_SERVER" "$schema_name" connected || return $?
> > > +    local db_version=$(ovsdb-client -t 10 get-schema-version
> > "$DB_SERVER" "$schema_name") || return $?
> > > +    local target_version=$(ovsdb-tool schema-version "$DB_SCHEMA") ||
> > return $?
> > > +
> > > +    if ovsdb-tool compare-versions "$db_version" == "$target_version";
> > then
> > > +        :
> > > +    elif ovsdb-tool compare-versions "$db_version" ">"
> > "$target_version"; then
> > > +        log_warning_msg "Database $schema_name has newer schema
> version
> > ($db_version) than our local schema ($target_version), possibly an
> upgrade
> > is partially complete?"
> > > +    else
> > > +        action "Upgrading database $schema_name from schema version
> > $db_version to $target_version" ovsdb-client -t 30 convert "$DB_SERVER"
> > "$DB_SCHEMA"
> > > +    fi
> > > +}
> > > +
> > > +create_cluster () {
> > > +    DB_FILE="$1"
> > > +    DB_SCHEMA="$2"
> > > +    LOCAL_ADDR="$3"
> > > +
> > > +    if test ! -e "$DB_FILE"; then
> > > +        action "Creating cluster database $DB_FILE" ovsdb_tool
> > create-cluster "$DB_FILE" "$DB_SCHEMA" "$LOCAL_ADDR"
> > > +    elif ovsdb_tool db-is-standalone "$DB_FILE"; then
> > > +        # Convert standalone database to clustered.
> > > +        backup_db || return 1
> > > +        action "Creating cluster database $DB_FILE from existing one"
> \
> > > +               ovsdb_tool create-cluster "$DB_FILE" "$backup"
> > "$LOCAL_ADDR"
> > > +    fi
> > > +}
> > > +
> > > +join_cluster() {
> > > +    DB_FILE="$1"
> > > +    SCHEMA_NAME="$2"
> > > +    LOCAL_ADDR="$3"
> > > +    REMOTE_ADDR="$4"
> > > +
> > > +    if test ! -e "$DB_FILE"; then
> > > +        ovsdb_tool join-cluster "$DB_FILE" "$SCHEMA_NAME"
> "$LOCAL_ADDR"
> > "$REMOTE_ADDR"
> > > +    elif ovsdb_tool db-is-standalone "$DB_FILE"; then
> > > +        # Backup standalone database and join cluster.
> > > +        backup_db || return 1
> > > +        action "Joining $DB_FILE to cluster" \
> > > +               ovsdb_tool join-cluster "$DB_FILE" "$SCHEMA_NAME"
> > "$LOCAL_ADDR"
> > > +    fi
> > > +}
> > > +
> > >  ovs_vsctl () {
> > >      ovs-vsctl --no-wait "$@"
> > >  }
> > > --
> > > 2.14.3
> > >
> >
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>


More information about the dev mailing list