[ovs-dev] [v3] ovsdb: Replication usability improvements

Andy Zhou azhou at ovn.org
Thu Sep 1 02:32:41 UTC 2016


Added the '--no-sync' option base on feedbacks of current
implementation.

Added appctl command "ovsdb-server/sync-status" based on feedbacks
of current implementation.

Added the RPL_S_INIT state in the state machine. This state is
not strictly necessary for the replication state machine, but is
introduced to make sure the state is update immediately when
the state machine is reset, via replication_init(). Without it
ovsdb/sync-status may display "replicating" or crash, if the command
is issued between after replication_init() is called, but before
the state variable is updated from replication_run().

Added a test to simulate the integration of HA manager with OVSDB
server using replication.

Other documentation and API improvements.

Tested-by: Numan Siddique <nusiddiq at redhat.com>
Signed-off-by: Andy Zhou <azhou at ovn.org>
---
 Documentation/OVSDB-replication.md |   2 +-
 ovsdb/jsonrpc-server.c             |   6 +
 ovsdb/jsonrpc-server.h             |   1 +
 ovsdb/ovsdb-server.1.in            |  24 ++--
 ovsdb/ovsdb-server.c               | 251 ++++++++++++++++++++++++++-----------
 ovsdb/replication.c                |  89 ++++++++++---
 ovsdb/replication.h                |  28 ++++-
 ovsdb/replication.man              |  15 ++-
 tests/ovsdb-server.at              | 105 ++++++++++++++--
 9 files changed, 408 insertions(+), 113 deletions(-)

diff --git a/Documentation/OVSDB-replication.md b/Documentation/OVSDB-replication.md
index 74c9500..a9ab5cc 100644
--- a/Documentation/OVSDB-replication.md
+++ b/Documentation/OVSDB-replication.md
@@ -144,7 +144,7 @@ commands are the following.
     between the active server and frees the memory used for the replication
     configuration.
 
--   ovsdb-server/set-sync-excluded-tables {db:table,...}: sets the tables list
+-   ovsdb-server/set-sync-exclude-tables {db:table,...}: sets the tables list
     that will be excluded from being replicated.
 
 -   ovsdb-server/get-sync-excluded-tables: gets the tables list that is
diff --git a/ovsdb/jsonrpc-server.c b/ovsdb/jsonrpc-server.c
index 45975a3..427026d 100644
--- a/ovsdb/jsonrpc-server.c
+++ b/ovsdb/jsonrpc-server.c
@@ -343,6 +343,12 @@ ovsdb_jsonrpc_server_reconnect(struct ovsdb_jsonrpc_server *svr, bool read_only)
     }
 }
 
+bool
+ovsdb_jsonrpc_server_is_read_only(struct ovsdb_jsonrpc_server *svr)
+{
+    return svr->read_only;
+}
+
 void
 ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *svr)
 {
diff --git a/ovsdb/jsonrpc-server.h b/ovsdb/jsonrpc-server.h
index 955bbe4..f04b1e9 100644
--- a/ovsdb/jsonrpc-server.h
+++ b/ovsdb/jsonrpc-server.h
@@ -65,6 +65,7 @@ void ovsdb_jsonrpc_server_reconnect(struct ovsdb_jsonrpc_server *, bool read_onl
 
 void ovsdb_jsonrpc_server_run(struct ovsdb_jsonrpc_server *);
 void ovsdb_jsonrpc_server_wait(struct ovsdb_jsonrpc_server *);
+bool ovsdb_jsonrpc_server_is_read_only(struct ovsdb_jsonrpc_server *);
 
 void ovsdb_jsonrpc_server_get_memory_usage(const struct ovsdb_jsonrpc_server *,
                                            struct simap *usage);
diff --git a/ovsdb/ovsdb-server.1.in b/ovsdb/ovsdb-server.1.in
index d1ba83b..e2e96ae 100644
--- a/ovsdb/ovsdb-server.1.in
+++ b/ovsdb/ovsdb-server.1.in
@@ -36,7 +36,7 @@ If none is specified, the default is \fB at DBDIR@/conf.db\fR.  The database
 files must already have been created and initialized using, for
 example, \fBovsdb\-tool create\fR.
 .
-.SH "ACTIVE and BACKUP "
+.SH "ACTIVE and BACKUP"
 \fBovsdb\-server\fR runs either as a backup server, or as an active server.
 When  \fBovsdb\-server\fR is running as a backup server, all transactions that
 can modify the database content, including the lock commands are rejected.
@@ -45,12 +45,12 @@ When \fBovsdb\-server\fR role changes, all existing client connection are
 reset, requiring clients to reconnect to the server.
 .PP
 By default, \fBovsdb\-server\fR runs as an active server, except when the
-\fB\-\-sync\-from=\fIserver\fR command line option is specified.  During
-runtime, \fBovsdb\-server\fR role can be switch by using appctl commands.
+\fB\-\-sync\-from=\fIserver\fR command line option is specified and without
+the \fB\-\-no\-sync option\fR.  During runtime, \fBovsdb\-server\fR role can be switch by using appctl commands.
 .PP
 \fBovsdb-server/connect\-active\-ovsdb\-server\fR switches
-\fBovsdb\-server\fR role into a backup server, Conversely,
-\fBovsdb-server/disconnect\-active\-ovsdb\-server\fR changes server into
+\fBovsdb\-server\fR into a backup server, Conversely,
+\fBovsdb-server/disconnect\-active\-ovsdb\-server\fR switches server into
 an active one.
 .
 .SH OPTIONS
@@ -220,12 +220,22 @@ specified by \fBovsdb\-server/set\-active\-ovsdb\-server\fR.
 .IP "\fBovsdb\-server/disconnect\-active\-ovsdb\-server"
 Causes \fBovsdb\-server\fR to  stop  synchronizing  its  databases with a active server.
 .
-.IP "\fBovsdb\-server/set\-sync\-excluded\-tables \fIdb\fB:\fItable\fR[\fB,\fIdb\fB:\fItable\fR]..."
+.IP "\fBovsdb\-server/set\-sync\-exclude\-tables \fIdb\fB:\fItable\fR[\fB,\fIdb\fB:\fItable\fR]..."
 Sets the \fItable\fR whitin \fIdb\fR that will be excluded from synchronization.
 .
-.IP "\fBovsdb\-server/get\-sync\-excluded\-tables"
+.IP "\fBovsdb\-server/get\-sync\-exclude\-tables"
 Gets  the  tables  that are currently excluded from synchronization.
 .
+.IP "\fBovsdb\-server/sync\-status"
+Prints a summary of replication run time information. The \fBstate\fR
+information is always provided, indicating whether the server is running
+in the \fIactive\fR or the \fIbackup\fR mode.
+When running in backup mode, replication connection status, which
+can be either \fIconnecting\fR, \fIreplicating\fR or \fIerror\fR, are shown.
+When the connection is in \fIreplicating\fR state, further output shows
+the list of databases currently replicating, and the tables that are
+excluded.
+.
 .so lib/vlog-unixctl.man
 .so lib/memory-unixctl.man
 .so lib/coverage-unixctl.man
diff --git a/ovsdb/ovsdb-server.c b/ovsdb/ovsdb-server.c
index 4de370c..29092e0 100644
--- a/ovsdb/ovsdb-server.c
+++ b/ovsdb/ovsdb-server.c
@@ -76,9 +76,6 @@ static char *certificate_file;
 static char *ca_cert_file;
 static bool bootstrap_ca_cert;
 
-/* Replication current state. */
-static bool is_backup_server;
-
 static unixctl_cb_func ovsdb_server_exit;
 static unixctl_cb_func ovsdb_server_compact;
 static unixctl_cb_func ovsdb_server_reconnect;
@@ -89,13 +86,17 @@ static unixctl_cb_func ovsdb_server_set_active_ovsdb_server;
 static unixctl_cb_func ovsdb_server_get_active_ovsdb_server;
 static unixctl_cb_func ovsdb_server_connect_active_ovsdb_server;
 static unixctl_cb_func ovsdb_server_disconnect_active_ovsdb_server;
-static unixctl_cb_func ovsdb_server_set_sync_excluded_tables;
-static unixctl_cb_func ovsdb_server_get_sync_excluded_tables;
+static unixctl_cb_func ovsdb_server_set_sync_exclude_tables;
+static unixctl_cb_func ovsdb_server_get_sync_exclude_tables;
+static unixctl_cb_func ovsdb_server_get_sync_status;
 
 struct server_config {
     struct sset *remotes;
     struct shash *all_dbs;
     FILE *config_tmpfile;
+    char **sync_from;
+    char **sync_exclude;
+    bool *is_backup;
     struct ovsdb_jsonrpc_server *jsonrpc;
 };
 static unixctl_cb_func ovsdb_server_add_remote;
@@ -111,7 +112,8 @@ static void close_db(struct db *db);
 
 static void parse_options(int *argc, char **argvp[],
                           struct sset *remotes, char **unixctl_pathp,
-                          char **run_command);
+                          char **run_command, char **sync_from,
+                          char **sync_exclude, bool *is_backup);
 OVS_NO_RETURN static void usage(void);
 
 static char *reconfigure_remotes(struct ovsdb_jsonrpc_server *,
@@ -125,15 +127,19 @@ static void update_remote_status(const struct ovsdb_jsonrpc_server *jsonrpc,
                                  struct shash *all_dbs);
 
 static void save_config__(FILE *config_file, const struct sset *remotes,
-                          const struct sset *db_filenames);
+                          const struct sset *db_filenames,
+                          const char *sync_from, const char *sync_exclude,
+                          bool is_backup);
 static void save_config(struct server_config *);
 static void load_config(FILE *config_file, struct sset *remotes,
-                        struct sset *db_filenames);
+                        struct sset *db_filenames, char **sync_from,
+                        char **sync_exclude, bool *is_backup);
 
 static void
-ovsdb_replication_init(struct shash *all_dbs)
+ovsdb_replication_init(const char *sync_from, const char *exclude,
+                       struct shash *all_dbs)
 {
-    replication_init();
+    replication_init(sync_from, exclude);
     struct shash_node *node;
     SHASH_FOR_EACH (node, all_dbs) {
         struct db *db = node->data;
@@ -144,11 +150,12 @@ ovsdb_replication_init(struct shash *all_dbs)
 static void
 main_loop(struct ovsdb_jsonrpc_server *jsonrpc, struct shash *all_dbs,
           struct unixctl_server *unixctl, struct sset *remotes,
-          struct process *run_process, bool *exiting)
+          struct process *run_process, bool *exiting, bool *is_backup)
 {
     char *remotes_error, *ssl_error;
     struct shash_node *node;
     long long int status_timer = LLONG_MIN;
+    bool last_role = *is_backup;
 
     *exiting = false;
     ssl_error = NULL;
@@ -172,13 +179,13 @@ main_loop(struct ovsdb_jsonrpc_server *jsonrpc, struct shash *all_dbs,
         /* Run unixctl_server_run() before reconfigure_remotes() because
          * ovsdb-server/add-remote and ovsdb-server/remove-remote can change
          * the set of remotes that reconfigure_remotes() uses. */
-        bool last_role = is_backup_server;
         unixctl_server_run(unixctl);
 
-        /* In case unixctl commands change the role of ovsdb-server,
-         *  from active to backup or vise versa, recoonect jsonrpc server.  */
-        if (last_role != is_backup_server) {
-            ovsdb_jsonrpc_server_reconnect(jsonrpc, is_backup_server);
+        /* In ovsdb-server's role (active or backup) has changed, restart
+         * the ovsdb jsonrpc server.  */
+        if (last_role != *is_backup) {
+            bool read_only = last_role = *is_backup;
+            ovsdb_jsonrpc_server_reconnect(jsonrpc, read_only);
         }
 
         report_error_if_changed(
@@ -187,7 +194,7 @@ main_loop(struct ovsdb_jsonrpc_server *jsonrpc, struct shash *all_dbs,
         report_error_if_changed(reconfigure_ssl(all_dbs), &ssl_error);
         ovsdb_jsonrpc_server_run(jsonrpc);
 
-        if (is_backup_server) {
+        if (*is_backup) {
             replication_run();
             if (!replication_is_alive()) {
                 int retval = replication_get_last_error();
@@ -213,7 +220,7 @@ main_loop(struct ovsdb_jsonrpc_server *jsonrpc, struct shash *all_dbs,
         }
 
         memory_wait();
-        if (is_backup_server) {
+        if (*is_backup) {
             replication_wait();
         }
 
@@ -247,6 +254,8 @@ main(int argc, char *argv[])
     struct unixctl_server *unixctl;
     struct ovsdb_jsonrpc_server *jsonrpc;
     struct sset remotes, db_filenames;
+    char *sync_from, *sync_exclude;
+    bool is_backup;
     const char *db_filename;
     struct process *run_process;
     bool exiting;
@@ -264,7 +273,11 @@ main(int argc, char *argv[])
     fatal_ignore_sigpipe();
     process_init();
 
-    parse_options(&argc, &argv, &remotes, &unixctl_path, &run_command);
+    bool no_sync = false;
+    parse_options(&argc, &argv, &remotes, &unixctl_path, &run_command,
+                  &sync_from, &sync_exclude, &no_sync);
+    is_backup = sync_from && !no_sync;
+
     daemon_become_new_user(false);
 
     /* Create and initialize 'config_tmpfile' as a temporary file to hold
@@ -291,17 +304,26 @@ main(int argc, char *argv[])
     server_config.remotes = &remotes;
     server_config.config_tmpfile = config_tmpfile;
 
-    save_config__(config_tmpfile, &remotes, &db_filenames);
+    save_config__(config_tmpfile, &remotes, &db_filenames, sync_from,
+                  sync_exclude, is_backup);
 
     daemonize_start(false);
 
     /* Load the saved config. */
-    load_config(config_tmpfile, &remotes, &db_filenames);
-    jsonrpc = ovsdb_jsonrpc_server_create(is_backup_server);
+    load_config(config_tmpfile, &remotes, &db_filenames, &sync_from,
+                &sync_exclude, &is_backup);
+
+    /* Start ovsdb jsonrpc server. When running as a backup server,
+     * jsonrpc connections are read only. Otherwise, both read
+     * and write transactions are allowed.  */
+    jsonrpc = ovsdb_jsonrpc_server_create(is_backup);
 
     shash_init(&all_dbs);
     server_config.all_dbs = &all_dbs;
     server_config.jsonrpc = jsonrpc;
+    server_config.sync_from = &sync_from;
+    server_config.sync_exclude = &sync_exclude;
+    server_config.is_backup = &is_backup;
 
     perf_counters_init();
 
@@ -373,34 +395,39 @@ main(int argc, char *argv[])
                              ovsdb_server_perf_counters_show, NULL);
     unixctl_command_register("ovsdb-server/perf-counters-clear", "", 0, 0,
                              ovsdb_server_perf_counters_clear, NULL);
-
-    unixctl_command_register("ovsdb-server/set-active-ovsdb-server", "", 0, 1,
-                              ovsdb_server_set_active_ovsdb_server, NULL);
+    unixctl_command_register("ovsdb-server/set-active-ovsdb-server", "", 1, 1,
+                             ovsdb_server_set_active_ovsdb_server,
+                             &server_config);
     unixctl_command_register("ovsdb-server/get-active-ovsdb-server", "", 0, 0,
-                              ovsdb_server_get_active_ovsdb_server, NULL);
+                             ovsdb_server_get_active_ovsdb_server,
+                             &server_config);
     unixctl_command_register("ovsdb-server/connect-active-ovsdb-server", "",
                              0, 0, ovsdb_server_connect_active_ovsdb_server,
-                             &all_dbs);
+                             &server_config);
     unixctl_command_register("ovsdb-server/disconnect-active-ovsdb-server", "",
                              0, 0, ovsdb_server_disconnect_active_ovsdb_server,
+                             &server_config);
+    unixctl_command_register("ovsdb-server/set-sync-exclude-tables", "",
+                             0, 1, ovsdb_server_set_sync_exclude_tables,
+                             &server_config);
+    unixctl_command_register("ovsdb-server/get-sync-exclude-tables", "",
+                             0, 0, ovsdb_server_get_sync_exclude_tables,
                              NULL);
-    unixctl_command_register("ovsdb-server/set-sync-excluded-tables", "",
-                             0, 1, ovsdb_server_set_sync_excluded_tables,
-                             &all_dbs);
-    unixctl_command_register("ovsdb-server/get-sync-excluded-tables", "",
-                             0, 0, ovsdb_server_get_sync_excluded_tables,
-                             NULL);
+    unixctl_command_register("ovsdb-server/sync-status", "",
+                             0, 0, ovsdb_server_get_sync_status,
+                             &server_config);
 
     /* Simulate the behavior of OVS release prior to version 2.5 that
      * does not support the monitor_cond method.  */
     unixctl_command_register("ovsdb-server/disable-monitor-cond", "", 0, 0,
                              ovsdb_server_disable_monitor_cond, jsonrpc);
 
-    if (is_backup_server) {
-        ovsdb_replication_init(&all_dbs);
+    if (is_backup) {
+        ovsdb_replication_init(sync_from, sync_exclude, &all_dbs);
     }
 
-    main_loop(jsonrpc, &all_dbs, unixctl, &remotes, run_process, &exiting);
+    main_loop(jsonrpc, &all_dbs, unixctl, &remotes, run_process, &exiting,
+              &is_backup);
 
     ovsdb_jsonrpc_server_destroy(jsonrpc);
     SHASH_FOR_EACH_SAFE(node, next, &all_dbs) {
@@ -411,6 +438,8 @@ main(int argc, char *argv[])
     shash_destroy(&all_dbs);
     sset_destroy(&remotes);
     sset_destroy(&db_filenames);
+    free(sync_from);
+    free(sync_exclude);
     unixctl_server_destroy(unixctl);
     replication_destroy();
 
@@ -1097,9 +1126,16 @@ report_error_if_changed(char *error, char **last_errorp)
 static void
 ovsdb_server_set_active_ovsdb_server(struct unixctl_conn *conn,
                                      int argc OVS_UNUSED, const char *argv[],
-                                     void *arg_ OVS_UNUSED)
+                                     void *config_)
 {
-    set_active_ovsdb_server(argv[1]);
+    struct server_config *config = config_;
+
+    if (*config->sync_from) {
+        free(*config->sync_from);
+    }
+    *config->sync_from = xstrdup(argv[1]);
+    save_config(config);
+
     unixctl_command_reply(conn, NULL);
 }
 
@@ -1107,49 +1143,65 @@ static void
 ovsdb_server_get_active_ovsdb_server(struct unixctl_conn *conn,
                                      int argc OVS_UNUSED,
                                      const char *argv[] OVS_UNUSED,
-                                     void *arg_ OVS_UNUSED)
+                                     void *config_ )
 {
-    unixctl_command_reply(conn, get_active_ovsdb_server());
+    struct server_config *config = config_;
+
+    unixctl_command_reply(conn, *config->sync_from);
 }
 
 static void
 ovsdb_server_connect_active_ovsdb_server(struct unixctl_conn *conn,
                                          int argc OVS_UNUSED,
                                          const char *argv[] OVS_UNUSED,
-                                         void *all_dbs_)
+                                         void *config_)
 {
-    struct shash *all_dbs = all_dbs_;
+    struct server_config *config = config_;
+    char *msg = NULL;
 
-    if (!is_backup_server) {
-        ovsdb_replication_init(all_dbs);
+    if ( !*config->sync_from) {
+        msg = "Unable to connect: active server is not specified.\n";
+    } else {
+        ovsdb_replication_init(*config->sync_from, *config->sync_exclude,
+                               config->all_dbs);
+        if (!*config->is_backup) {
+            *config->is_backup = true;
+            save_config(config);
+        }
     }
-    is_backup_server = true;
-    unixctl_command_reply(conn, NULL);
+    unixctl_command_reply(conn, msg);
 }
 
 static void
 ovsdb_server_disconnect_active_ovsdb_server(struct unixctl_conn *conn,
                                             int argc OVS_UNUSED,
                                             const char *argv[] OVS_UNUSED,
-                                            void *arg_ OVS_UNUSED)
+                                            void *config_)
 {
+    struct server_config *config = config_;
+
     disconnect_active_server();
-    is_backup_server = false;
+    *config->is_backup = false;
+    save_config(config);
     unixctl_command_reply(conn, NULL);
 }
 
 static void
-ovsdb_server_set_sync_excluded_tables(struct unixctl_conn *conn,
-                                      int argc OVS_UNUSED,
-                                      const char *argv[],
-                                      void *all_dbs_)
+ovsdb_server_set_sync_exclude_tables(struct unixctl_conn *conn,
+                                     int argc OVS_UNUSED,
+                                     const char *argv[],
+                                     void *config_)
 {
-    struct shash *all_dbs = all_dbs_;
-    char *err = set_blacklist_tables(argv[1], true);
+    struct server_config *config = config_;
 
+    char *err = set_blacklist_tables(argv[1], true);
     if (!err) {
-        if (is_backup_server) {
-            ovsdb_replication_init(all_dbs);
+        free(*config->sync_exclude);
+        *config->sync_exclude = xstrdup(argv[1]);
+        save_config(config);
+        if (*config->is_backup) {
+            ovsdb_replication_init(*config->sync_from, *config->sync_exclude,
+                                   config->all_dbs);
         }
         err = set_blacklist_tables(argv[1], false);
     }
@@ -1158,10 +1210,10 @@ ovsdb_server_set_sync_excluded_tables(struct unixctl_conn *conn,
 }
 
 static void
-ovsdb_server_get_sync_excluded_tables(struct unixctl_conn *conn,
-                                 int argc OVS_UNUSED,
-                                 const char *argv[] OVS_UNUSED,
-                                 void *arg_ OVS_UNUSED)
+ovsdb_server_get_sync_exclude_tables(struct unixctl_conn *conn,
+                                     int argc OVS_UNUSED,
+                                     const char *argv[] OVS_UNUSED,
+                                     void *arg_ OVS_UNUSED)
 {
     unixctl_command_reply(conn, get_blacklist_tables());
 }
@@ -1206,9 +1258,10 @@ ovsdb_server_disable_monitor_cond(struct unixctl_conn *conn,
                                   void *jsonrpc_)
 {
     struct ovsdb_jsonrpc_server *jsonrpc = jsonrpc_;
+    bool read_only = ovsdb_jsonrpc_server_is_read_only(jsonrpc);
 
     ovsdb_jsonrpc_disable_monitor_cond();
-    ovsdb_jsonrpc_server_reconnect(jsonrpc, is_backup_server);
+    ovsdb_jsonrpc_server_reconnect(jsonrpc, read_only);
     unixctl_command_reply(conn, NULL);
 }
 
@@ -1263,8 +1316,9 @@ ovsdb_server_reconnect(struct unixctl_conn *conn, int argc OVS_UNUSED,
                        const char *argv[] OVS_UNUSED, void *jsonrpc_)
 {
     struct ovsdb_jsonrpc_server *jsonrpc = jsonrpc_;
+    bool read_only = ovsdb_jsonrpc_server_is_read_only(jsonrpc);
 
-    ovsdb_jsonrpc_server_reconnect(jsonrpc, is_backup_server);
+    ovsdb_jsonrpc_server_reconnect(jsonrpc, read_only);
     unixctl_command_reply(conn, NULL);
 }
 
@@ -1350,8 +1404,9 @@ ovsdb_server_add_database(struct unixctl_conn *conn, int argc OVS_UNUSED,
     error = open_db(config, filename);
     if (!error) {
         save_config(config);
-        if (is_backup_server) {
-            ovsdb_replication_init(config->all_dbs);
+        if (*config->is_backup) {
+            ovsdb_replication_init(*config->sync_from, *config->sync_exclude,
+                                   config->all_dbs);
         }
         unixctl_command_reply(conn, NULL);
     } else {
@@ -1383,8 +1438,9 @@ ovsdb_server_remove_database(struct unixctl_conn *conn, int argc OVS_UNUSED,
     shash_delete(config->all_dbs, node);
 
     save_config(config);
-    if (is_backup_server) {
-        ovsdb_replication_init(config->all_dbs);
+    if (*config->is_backup) {
+        ovsdb_replication_init(*config->sync_from, *config->sync_exclude,
+                               config->all_dbs);
     }
     unixctl_command_reply(conn, NULL);
 }
@@ -1412,8 +1468,27 @@ ovsdb_server_list_databases(struct unixctl_conn *conn, int argc OVS_UNUSED,
 }
 
 static void
+ovsdb_server_get_sync_status(struct unixctl_conn *conn, int argc OVS_UNUSED,
+                             const char *argv[] OVS_UNUSED, void *config_)
+{
+    struct server_config *config = config_;
+    bool is_backup = *config->is_backup;
+    struct ds ds = DS_EMPTY_INITIALIZER;
+
+    ds_put_format(&ds, "state: %s\n", is_backup ? "backup" : "active");
+
+    if (is_backup) {
+        ds_put_and_free_cstr(&ds, replication_status());
+    }
+
+    unixctl_command_reply(conn, ds_cstr(&ds));
+    ds_destroy(&ds);
+}
+
+static void
 parse_options(int *argcp, char **argvp[],
-              struct sset *remotes, char **unixctl_pathp, char **run_command)
+              struct sset *remotes, char **unixctl_pathp, char **run_command,
+              char **sync_from, char **sync_exclude, bool *no_sync)
 {
     enum {
         OPT_REMOTE = UCHAR_MAX + 1,
@@ -1423,6 +1498,7 @@ parse_options(int *argcp, char **argvp[],
         OPT_PEER_CA_CERT,
         OPT_SYNC_FROM,
         OPT_SYNC_EXCLUDE,
+        OPT_NO_SYNC,
         VLOG_OPTION_ENUMS,
         DAEMON_OPTION_ENUMS
     };
@@ -1443,12 +1519,15 @@ parse_options(int *argcp, char **argvp[],
         {"ca-cert",     required_argument, NULL, 'C'},
         {"sync-from",   required_argument, NULL, OPT_SYNC_FROM},
         {"sync-exclude-tables", required_argument, NULL, OPT_SYNC_EXCLUDE},
+        {"no-sync", no_argument, NULL, OPT_NO_SYNC},
         {NULL, 0, NULL, 0},
     };
     char *short_options = ovs_cmdl_long_options_to_short_options(long_options);
     int argc = *argcp;
     char **argv = *argvp;
 
+    *sync_from = NULL;
+    *sync_exclude = NULL;
     sset_init(remotes);
     for (;;) {
         int c;
@@ -1504,8 +1583,7 @@ parse_options(int *argcp, char **argvp[],
             break;
 
         case OPT_SYNC_FROM:
-            set_active_ovsdb_server(optarg);
-            is_backup_server = true;
+            *sync_from = xstrdup(optarg);
             break;
 
         case OPT_SYNC_EXCLUDE: {
@@ -1513,8 +1591,12 @@ parse_options(int *argcp, char **argvp[],
             if (err) {
                 ovs_fatal(0, "%s", err);
             }
+            *sync_exclude = xstrdup(optarg);
             break;
         }
+        case OPT_NO_SYNC:
+            *no_sync = true;
+            break;
 
         case '?':
             exit(EXIT_FAILURE);
@@ -1568,7 +1650,8 @@ sset_to_json(const struct sset *sset)
  * 'remotes' and 'db_filenames'. */
 static void
 save_config__(FILE *config_file, const struct sset *remotes,
-              const struct sset *db_filenames)
+              const struct sset *db_filenames, const char *sync_from,
+              const char *sync_exclude, bool is_backup)
 {
     struct json *obj;
     char *s;
@@ -1581,6 +1664,15 @@ save_config__(FILE *config_file, const struct sset *remotes,
     obj = json_object_create();
     json_object_put(obj, "remotes", sset_to_json(remotes));
     json_object_put(obj, "db_filenames", sset_to_json(db_filenames));
+    if (sync_from) {
+        json_object_put(obj, "sync_from", json_string_create(sync_from));
+    }
+    if (sync_exclude) {
+        json_object_put(obj, "sync_exclude",
+                        json_string_create(sync_exclude));
+    }
+    json_object_put(obj, "is_backup", json_boolean_create(is_backup));
+
     s = json_to_string(obj, 0);
     json_destroy(obj);
 
@@ -1606,7 +1698,9 @@ save_config(struct server_config *config)
         sset_add(&db_filenames, db->filename);
     }
 
-    save_config__(config->config_tmpfile, config->remotes, &db_filenames);
+    save_config__(config->config_tmpfile, config->remotes, &db_filenames,
+                  *config->sync_from, *config->sync_exclude,
+                  *config->is_backup);
 
     sset_destroy(&db_filenames);
 }
@@ -1628,7 +1722,8 @@ sset_from_json(struct sset *sset, const struct json *array)
 /* Clears and replaces 'remotes' and 'dbnames' by a configuration read from
  * 'config_file', which must have been previously written by save_config(). */
 static void
-load_config(FILE *config_file, struct sset *remotes, struct sset *db_filenames)
+load_config(FILE *config_file, struct sset *remotes, struct sset *db_filenames,
+            char **sync_from, char **sync_exclude, bool *is_backup)
 {
     struct json *json;
 
@@ -1644,5 +1739,17 @@ load_config(FILE *config_file, struct sset *remotes, struct sset *db_filenames)
     sset_from_json(remotes, shash_find_data(json_object(json), "remotes"));
     sset_from_json(db_filenames,
                    shash_find_data(json_object(json), "db_filenames"));
+
+    struct json *string;
+    string = shash_find_data(json_object(json), "sync_from");
+    free(*sync_from);
+    *sync_from = string ? xstrdup(json_string(string)) : NULL;
+
+    string = shash_find_data(json_object(json), "sync_exclude");
+    free(*sync_exclude);
+    *sync_exclude = string ? xstrdup(json_string(string)) : NULL;
+
+    *is_backup = json_boolean(shash_find_data(json_object(json), "is_backup"));
+
     json_destroy(json);
 }
diff --git a/ovsdb/replication.c b/ovsdb/replication.c
index 2245566..40a9fd0 100644
--- a/ovsdb/replication.c
+++ b/ovsdb/replication.c
@@ -37,7 +37,7 @@
 
 VLOG_DEFINE_THIS_MODULE(replication);
 
-static char *active_ovsdb_server;
+static char *sync_from;
 static struct jsonrpc_session *session = NULL;
 static unsigned int session_seqno = UINT_MAX;
 
@@ -87,6 +87,7 @@ static void request_ids_destroy(void);
 void request_ids_clear(void);
 
 enum ovsdb_replication_state {
+    RPL_S_INIT,
     RPL_S_DB_REQUESTED,
     RPL_S_SCHEMA_REQUESTED,
     RPL_S_MONITOR_REQUESTED,
@@ -108,8 +109,15 @@ static struct ovsdb* find_db(const char *db_name);
 
 
 void
-replication_init(void)
+replication_init(const char *sync_from_, const char *exclude_tables)
 {
+    free(sync_from);
+    sync_from = xstrdup(sync_from_);
+    char *err = set_blacklist_tables(exclude_tables, false);
+    /* Caller should have verified that the 'exclude_tables' is
+     * parseable. An error here is unexpected. */
+    ovs_assert(!err);
+
     shash_destroy(replication_dbs);
     replication_dbs = NULL;
 
@@ -118,8 +126,9 @@ replication_init(void)
         jsonrpc_session_close(session);
     }
 
-    session = jsonrpc_session_open(active_ovsdb_server, true);
+    session = jsonrpc_session_open(sync_from, true);
     session_seqno = UINT_MAX;
+    state = RPL_S_INIT;
 }
 
 void
@@ -143,7 +152,7 @@ replication_run(void)
         unsigned int seqno;
 
         seqno = jsonrpc_session_get_seqno(session);
-        if (seqno != session_seqno) {
+        if (seqno != session_seqno || state == RPL_S_INIT) {
             session_seqno = seqno;
             request_ids_clear();
             struct jsonrpc_msg *request;
@@ -156,6 +165,7 @@ replication_run(void)
             replication_dbs = replication_db_clone(&local_dbs);
 
             state = RPL_S_DB_REQUESTED;
+            VLOG_DBG("Send list_dbs request");
         }
 
         msg = jsonrpc_session_recv(session);
@@ -216,6 +226,7 @@ replication_run(void)
                             }
                         }
                     }
+                    VLOG_DBG("Send schema requests");
                     state = RPL_S_SCHEMA_REQUESTED;
                 }
                 break;
@@ -271,6 +282,7 @@ replication_run(void)
 
                             request_ids_add(request->id, db);
                             jsonrpc_session_send(session, request);
+                            VLOG_DBG("Send monitor requests");
                             state = RPL_S_MONITOR_REQUESTED;
                         }
                     }
@@ -289,6 +301,7 @@ replication_run(void)
                     /* Transition to replicating state after receiving
                      * all replies of "monitor" requests. */
                     if (hmap_is_empty(&request_ids)) {
+                        VLOG_DBG("Listening to monitor updates");
                         state = RPL_S_REPLICATING;
                     }
                 }
@@ -299,6 +312,7 @@ replication_run(void)
                 /* Ignore all messages */
                 break;
 
+            case RPL_S_INIT:
             case RPL_S_REPLICATING:
             default:
                 OVS_NOT_REACHED();
@@ -318,18 +332,6 @@ replication_wait(void)
     }
 }
 
-void
-set_active_ovsdb_server(const char *active_server)
-{
-    active_ovsdb_server = nullable_xstrdup(active_server);
-}
-
-const char *
-get_active_ovsdb_server(void)
-{
-    return active_ovsdb_server;
-}
-
 /* Parse 'blacklist' to rebuild 'blacklist_tables'.  If 'dryrun' is false, the
  * current black list tables will be wiped out, regardless of whether
  * 'blacklist' can be parsed.  If 'dryrun' is true, only parses 'blacklist' and
@@ -457,9 +459,9 @@ replication_destroy(void)
     blacklist_tables_clear();
     shash_destroy(&blacklist_tables);
 
-    if (active_ovsdb_server) {
-        free(active_ovsdb_server);
-        active_ovsdb_server = NULL;
+    if (sync_from) {
+        free(sync_from);
+        sync_from = NULL;
     }
 
     request_ids_destroy();
@@ -855,12 +857,59 @@ replication_get_last_error(void)
     return err;
 }
 
+char *
+replication_status(void)
+{
+    bool alive = session && jsonrpc_session_is_alive(session);
+    struct ds ds = DS_EMPTY_INITIALIZER;
+
+    if (alive) {
+        switch(state) {
+        case RPL_S_INIT:
+        case RPL_S_DB_REQUESTED:
+        case RPL_S_SCHEMA_REQUESTED:
+        case RPL_S_MONITOR_REQUESTED:
+            ds_put_format(&ds, "connecting: %s", sync_from);
+            break;
+        case RPL_S_REPLICATING: {
+            struct shash_node *node;
+
+            ds_put_format(&ds, "replicating: %s\n", sync_from);
+            ds_put_cstr(&ds, "database:");
+            SHASH_FOR_EACH (node, replication_dbs) {
+                ds_put_format(&ds, " %s,", node->name);
+            }
+            ds_chomp(&ds, ',');
+
+            if (!shash_is_empty(&blacklist_tables)) {
+                ds_put_char(&ds, '\n');
+                ds_put_cstr(&ds, "exclude: ");
+                ds_put_and_free_cstr(&ds, get_blacklist_tables());
+            }
+            break;
+        }
+        case RPL_S_ERR:
+            ds_put_format(&ds, "Replication to (%s) failed\n", sync_from);
+            break;
+        default:
+            OVS_NOT_REACHED();
+            break;
+        }
+    } else {
+        ds_put_format(&ds, "not connected to %s", sync_from);
+    }
+    return ds_steal_cstr(&ds);
+}
+
 void
 replication_usage(void)
 {
     printf("\n\
 Syncing options:\n\
   --sync-from=SERVER      sync DATABASE from active SERVER\n\
+                          start in backup mode, except when\n\
+                          --no-sync is also specified\n\
   --sync-exclude-tables=DB:TABLE,...\n\
-                          exclude the TABLE in DB from syncing\n");
+                          exclude the TABLE in DB from syncing\n\
+  --no-sync               start in active mode\n");
 }
diff --git a/ovsdb/replication.h b/ovsdb/replication.h
index c873648..622d355 100644
--- a/ovsdb/replication.h
+++ b/ovsdb/replication.h
@@ -21,7 +21,30 @@
 #include <stdbool.h>
 struct ovsdb;
 
-void replication_init(void);
+/* Replication module runs when OVSDB server runs in the backup mode.
+ *
+ * API Usage
+ *===========
+ *
+ * - replication_init() needs to be called whenever OVSDB server switches into
+ *   the backup mode.
+ *
+ * - replication_add_local_db() should be called immediately after to add all
+ *   known database that OVSDB server owns, one at a time.
+ *
+ * - replication_destroy() should be called when OVSDB server shutdown to
+ *   reclaim resources.
+ *
+ * - replication_run(), replication_wait(), replication_is_alive() and
+ *   replication_get_last_error() should be call within the main loop
+ *   whenever OVSDB server runs in the backup mode.
+ *
+ *  - set_blacklist_tables(), get_blacklist_tables(),
+ *    disconnect_active_server() and replication_usage() are support functions
+ *    used mainly by uinxctl commands.
+ */
+
+void replication_init(const char *sync_from, const char *exclude_tables);
 void replication_run(void);
 void replication_wait(void);
 void replication_destroy(void);
@@ -29,9 +52,8 @@ void replication_usage(void);
 void replication_add_local_db(const char *databse, struct ovsdb *db);
 bool replication_is_alive(void);
 int replication_get_last_error(void);
+char *replication_status(void);
 
-void set_active_ovsdb_server(const char *remote_server);
-const char *get_active_ovsdb_server(void);
 char *set_blacklist_tables(const char *blacklist, bool dryrun)
     OVS_WARN_UNUSED_RESULT;
 char *get_blacklist_tables(void) OVS_WARN_UNUSED_RESULT;
diff --git a/ovsdb/replication.man b/ovsdb/replication.man
index 26420bc..be827b9 100644
--- a/ovsdb/replication.man
+++ b/ovsdb/replication.man
@@ -3,6 +3,17 @@ with another running \fBovsdb\-server\fR.
 .TP
 \fB\-\-sync\-from=\fIserver\fR
 Causes  \fBovsdb\-server\fR to synchronize its databases with the databases in
-\fIserver\fR.  Every transaction committed by \fIserver\fR  will  be  replicated
+\fIserver\fR.  Every transaction committed by \fIserver\fR  will be replicated
 to \fBovsdb\-server\fR. \fIserver\fR is an active connection method in one of
-the forms documented in \fBovsdb\-client(1).
+the forms documented in \fBovsdb\-client(1)\fR.
+However if the \fB\-\-no\-sync\fR is also specified, replication will be
+postponed until \fBovs-appctl(1)\fR command
+\fBovsdb\-server/connect\-active\-ovsdb\-server\fR is issued.
+.TP
+\fB\-\-sync\-exclude-tables=\fIdb:table[,db:table]...\fR
+Causes the specified tables to be excluded from replication.
+.TP
+\fB\-\-no\-sync\fR
+Always start the server as an active server. Use this option to allow
+the syncing options to be specified using command line options, yet start
+the server, as the default, active server.
diff --git a/tests/ovsdb-server.at b/tests/ovsdb-server.at
index d2f3089..e7daefd 100644
--- a/tests/ovsdb-server.at
+++ b/tests/ovsdb-server.at
@@ -1100,22 +1100,22 @@ AT_CHECK([ovs-appctl -t ovsdb-server ovsdb-server/get-active-ovsdb-server],
 ])
 AT_CLEANUP
 
-#ovsdb-server/get-sync-excluded-tables command
-AT_SETUP([ovsdb-server/get-sync-excluded-tables])
-AT_KEYWORDS([ovsdb server replication get-excluded-tables])
+#ovsdb-server/get-sync-exclude-tables command
+AT_SETUP([ovsdb-server/get-sync-exclude-tables])
+AT_KEYWORDS([ovsdb server replication get-exclude-tables])
 ordinal_schema > schema
 AT_CHECK([ovsdb-tool create db schema], [0], [ignore], [ignore])
 on_exit 'kill `cat *.pid`'
 AT_CHECK([ovsdb-server --detach --no-chdir --pidfile --sync-exclude-tables=mydb:db1,mydb:db2 db])
 
-AT_CHECK([ovs-appctl -t ovsdb-server ovsdb-server/get-sync-excluded-tables],
+AT_CHECK([ovs-appctl -t ovsdb-server ovsdb-server/get-sync-exclude-tables],
   [0], [mydb:db1,mydb:db2
 ])
 AT_CLEANUP
 
-#ovsdb-server/set-sync-excluded-tables command
-AT_SETUP([ovsdb-server/set-sync-excluded-tables])
-AT_KEYWORDS([ovsdb server replication set-excluded-tables])
+#ovsdb-server/set-sync-exclude-tables command
+AT_SETUP([ovsdb-server/set-sync-exclude-tables])
+AT_KEYWORDS([ovsdb server replication set-exclude-tables])
 replication_schema > schema
 AT_CHECK([ovsdb-tool create db1 schema], [0], [stdout], [ignore])
 AT_CHECK([ovsdb-tool create db2 schema], [0], [stdout], [ignore])
@@ -1126,7 +1126,7 @@ on_exit 'test ! -e pid || kill `cat pid`'
 AT_CHECK([ovsdb-server --detach --no-chdir --log-file=ovsdb-server2.log --pidfile="`pwd`"/pid2 --remote=punix:db2.sock --unixctl="`pwd`"/unixctl2 --sync-from=unix:db.sock db2], [0], [ignore], [ignore])
 on_exit 'test ! -e pid2 || kill `cat pid2`'
 
-AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/set-sync-excluded-tables mydb:b], [0], [ignore], [ignore], [test ! -e pid || kill `cat pid`; test ! -e pid2 || kill `cat pid2`])
+AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/set-sync-exclude-tables mydb:b], [0], [ignore], [ignore], [test ! -e pid || kill `cat pid`; test ! -e pid2 || kill `cat pid2`])
 
 AT_CHECK([ovsdb-client transact unix:db.sock \
  '[["mydb",
@@ -1174,6 +1174,11 @@ on_exit 'test ! -e pid || kill `cat pid`'
 AT_CHECK([ovsdb-server --detach --no-chdir --log-file=ovsdb-server2.log --pidfile="`pwd`"/pid2 --remote=punix:db2.sock --unixctl="`pwd`"/unixctl2 db2], [0], [ignore], [ignore])
 on_exit 'test ! -e pid2 || kill `cat pid2`'
 
+dnl Try to connect without specifying the active server.
+AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/connect-active-ovsdb-server], [0],
+[Unable to connect: active server is not specified.
+], [ignore])
+
 AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/set-active-ovsdb-server unix:db.sock], [0], [stdout], [ignore])
 
 AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/connect-active-ovsdb-server], [0], [stdout], [ignore])
@@ -1278,3 +1283,87 @@ _uuid                                name number
 OVSDB_SERVER_SHUTDOWN
 OVSDB_SERVER_SHUTDOWN2
 AT_CLEANUP
+
+#ovsdb-server/active-backup-role-switching
+AT_SETUP([ovsdb-server/active-backup-role-switching])
+AT_KEYWORDS([ovsdb server replication active-backup-switching])
+replication_schema > schema
+AT_CHECK([ovsdb-tool create db1 schema], [0], [stdout], [ignore])
+AT_CHECK([ovsdb-tool create db2 schema], [0], [stdout], [ignore])
+
+dnl Add some data to both DBs
+AT_CHECK([ovsdb-tool transact db1 \
+'[["mydb",
+  {"op": "insert",
+   "table": "a",
+   "row": {"number": 9, "name": "nine"}}]]'], [0], [ignore], [ignore])
+
+AT_CHECK([ovsdb-tool transact db2 \
+'[["mydb",
+  {"op": "insert",
+   "table": "a",
+   "row": {"number": 9, "name": "nine"}}]]'], [0], [ignore], [ignore])
+
+dnl Start both 'db1' and 'db2' in backup mode. Let them backup from each
+dnl other. This is not an supported operation state, but to simulate a start
+dnl up condition where an HA manger can select which one to be an active
+dnl server soon after.
+AT_CHECK([ovsdb-server --detach --no-chdir --log-file=ovsdb-server1.log --pidfile="`pwd`"/pid --remote=punix:db.sock --unixctl="`pwd`"/unixctl db1 --sync-from=unix:db2.sock --no-sync ], [0], [ignore], [ignore])
+on_exit 'test ! -e pid || kill `cat pid`'
+
+AT_CHECK([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/connect-active-ovsdb-server])
+
+AT_CHECK([ovsdb-server --detach --no-chdir --log-file=ovsdb-server2.log --pidfile="`pwd`"/pid2 --remote=punix:db2.sock --unixctl="`pwd`"/unixctl2 --sync-from=unix:db.sock db2], [0], [ignore], [ignore])
+on_exit 'test ! -e pid2 || kill `cat pid2`'
+
+dnl
+dnl make sure both servers reached the replication state
+OVS_WAIT_UNTIL([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/sync-status |grep replicating])
+OVS_WAIT_UNTIL([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/sync-status |grep replicating])
+
+dnl Switch the 'db1' to active
+AT_CHECK([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/disconnect-active-ovsdb-server])
+AT_CHECK([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/sync-status], [0], [state: active
+])
+
+dnl Issue a transaction to 'db1'
+AT_CHECK([ovsdb-client transact unix:db.sock \
+'[["mydb",
+  {"op": "insert",
+   "table": "a",
+   "row": {"number": 0, "name": "zero"}}]]'], [0], [ignore])
+
+dnl It should be replicated to 'db2'
+OVS_WAIT_UNTIL([ovsdb-client dump unix:db2.sock | grep zero])
+
+dnl Flip the role of 'db1' and 'db2'.  'db1' becomes backup, and db2 becomes active
+AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/disconnect-active-ovsdb-server])
+AT_CHECK([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/connect-active-ovsdb-server])
+
+dnl Verify the change happend
+OVS_WAIT_UNTIL([ovs-appctl -t "`pwd`"/unixctl ovsdb-server/sync-status |grep replicating])
+AT_CHECK([ovs-appctl -t "`pwd`"/unixctl2 ovsdb-server/sync-status], [0], [state: active
+])
+
+dnl Issue an transaction to 'db2' which is now active.
+AT_CHECK([ovsdb-client transact unix:db2.sock \
+'[["mydb",
+  {"op": "insert",
+   "table": "b",
+   "row": {"number": 1, "name": "one"}}]]'], [0], [ignore])
+
+dnl The transaction should be replicated to 'db1'
+OVS_WAIT_UNTIL([ovsdb-client dump unix:db.sock | grep one])
+
+dnl Both servers should have the same content.
+AT_CHECK([ovsdb-client dump unix:db.sock], [0], [stdout])
+cat stdout > dump1
+
+AT_CHECK([ovsdb-client dump unix:db2.sock], [0], [stdout])
+cat stdout > dump2
+
+AT_CHECK([diff dump1 dump2])
+
+dnl OVSDB_SERVER_SHUTDOWN
+dnl OVSDB_SERVER_SHUTDOWN2
+AT_CLEANUP
-- 
1.9.1




More information about the dev mailing list