<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 9:49 AM, aginwala <span dir="ltr">&lt;<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Numan:<div><br></div><div>Yup agree with the locking part. For now; yes I am running northd on one node. I might right a script to monitor northd  in cluster so that if the node where it&#39;s running goes down, script can spin up northd on one other active nodes as a dirty hack.</div><div><br></div></div></blockquote><div>The &quot;dirty hack&quot; is pacemaker :)<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div>Sure, will await for the inputs from Ben too on this and see how complex would it be to roll out this feature.</div><div><br></div><div><br></div><div>Regards,</div><div><div class="h5"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 5:43 AM, Numan Siddique <span dir="ltr">&lt;<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Aliasgar,<div><br></div><div>ovsdb-server maintains locks per each connection and not across the db. A workaround for you now would be to configure all the ovn-northd instances to connect to one ovsdb-server if you want to have active/standy.</div><div><br></div><div>Probably Ben can answer if there is a plan to support ovsdb locks across the db. We also need this support in networking-ovn as it also uses ovsdb locks.</div><div><br></div><div>Thanks</div><span class="m_6021648639681040183m_-5349386396769448156HOEnZb"><font color="#888888"><div>Numan</div></font></span><div><div class="m_6021648639681040183m_-5349386396769448156h5"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 1:40 PM, aginwala <span dir="ltr">&lt;<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Numan:</div><div><br></div><div>Just figured out that ovn-northd is running as active on all 3 nodes instead of one active instance as I continued to test further which results in db errors as per logs.</div><div><br></div><div><br></div><div># on node 3, I run ovn-nbctl ls-add ls2 ; it populates below logs in  ovn-north</div><div>2018-03-21T06:01:59.442Z|00007<wbr>|ovsdb_idl|WARN|transaction error: {&quot;details&quot;:&quot;Transaction causes multiple rows in \&quot;Datapath_Binding\&quot; table to have identical values (1) for index on column \&quot;tunnel_key\&quot;.  First row, with UUID 8c5d9342-2b90-4229-8ea1-001a73<wbr>3a915c, was inserted by this transaction.  Second row, with UUID 8e06f919-4cc7-4ffc-9a79-20ce66<wbr>63b683, existed in the database before this transaction and was not modified by the transaction.&quot;,&quot;error&quot;:&quot;constra<wbr>int violation&quot;}</div><div><br></div><div>In southbound datapath list, 2 duplicate records gets created for same switch.</div><div><br></div><div># ovn-sbctl list Datapath</div><div>_uuid               : b270ae30-3458-445f-95d2-b14e8e<wbr>bddd01</div><div>external_ids        : {logical-switch=&quot;4d6674e3-ff9f<wbr>-4f38-b050-0fa9bec9e34d&quot;, name=&quot;ls2&quot;}</div><div>tunnel_key          : 2</div><div><br></div><div>_uuid               : 8e06f919-4cc7-4ffc-9a79-20ce66<wbr>63b683</div><div>external_ids        : {logical-switch=&quot;4d6674e3-ff9f<wbr>-4f38-b050-0fa9bec9e34d&quot;, name=&quot;ls2&quot;}</div><div>tunnel_key          : 1</div><div><br></div><div><br></div><div><br></div><div># on nodes 1 and 2 where northd is running, it gives below error:</div><div>2018-03-21T06:01:59.437Z|00008<wbr>|ovsdb_idl|WARN|transaction error: {&quot;details&quot;:&quot;cannot delete Datapath_Binding row 8e06f919-4cc7-4ffc-9a79-20ce66<wbr>63b683 because of 17 remaining reference(s)&quot;,&quot;error&quot;:&quot;referen<wbr>tial integrity violation&quot;}</div><div><br></div><div>As per commit message, for northd I re-tried setting --ovnnb-db=&quot;tcp:<a href="http://10.169.125.152:6641" target="_blank">10.169.125.152<wbr>:6641</a>,tcp:<a href="http://10.169.125.131:6641" target="_blank">10.169.125.131:6641</a>,<wbr>tcp:<a href="http://10.148.181.162:6641" target="_blank">10.148.181.162:6641</a>&quot;  and --ovnsb-db=&quot;tcp:<a href="http://10.169.125.152:6642" target="_blank">10.169.125.152<wbr>:6642</a>,tcp:<a href="http://10.169.125.131:6642" target="_blank">10.169.125.131:6642</a>,<wbr>tcp:<a href="http://10.148.181.162:6642" target="_blank">10.148.181.162:6642</a>&quot; and it did not help either.</div><div> </div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">There is no issue if I keep running only one instance of northd on any of these 3 nodes. </span>Hence, wanted to know is there something else missing here to make only one northd instance as active and rest as standby? </div><div><br></div><div><br></div><div>Regards,</div></div><div class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104HOEnZb"><div class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 15, 2018 at 3:09 AM, Numan Siddique <span dir="ltr">&lt;<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">That&#39;s great<span class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792HOEnZb"><font color="#888888"><div><br></div><div>Numan</div><div><br></div></font></span></div><div class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792HOEnZb"><div class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 15, 2018 at 2:57 AM, aginwala <span dir="ltr">&lt;<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Numan:<div><br></div><div>I tried on new nodes (kernel : 4.4.0-104-generic , Ubuntu 16.04)with fresh installation and it worked super fine for <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">both sb and nb dbs</span>. <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Seems like some kernel issue on the previous nodes when I re-installed raft patch as I was running different ovs version on those nodes before.</span></div><div><br></div><div><br></div><div>For 2 HVs, I now set <span style="color:rgb(51,51,51);font-family:monospace;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ovn-remote</span>=&quot;tcp:<a href="http://10.169.125.152:6642" target="_blank">10.169.125.152<wbr>:6642</a>, tcp:<a href="http://10.169.125.131:6642" target="_blank">10.169.125.131:6642</a>, tcp:<a href="http://10.148.181.162:6642" target="_blank">10.148.181.162:6642</a>&quot;  and started controller and it works super fine.  </div><div><br></div><div><br></div><div>Did some failover testing by rebooting/killing the leader (<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">10.169.125.152</span>) and bringing it back up and it works as expected. Nothing weird noted so far. </div><div><br></div><div># check-cluster gives below data one of the node(<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">10.148.181.162)</span> post leader failure</div><div><br></div><div>ovsdb-tool check-cluster /etc/openvswitch/ovnsb_db.db</div><div>ovsdb-tool: leader /etc/openvswitch/ovnsb_db.db for term 2 has log entries only up to index 18446744073709551615, but index 9 was committed in a previous term (e.g. by /etc/openvswitch/ovnsb_db.db)</div><div><br></div><div><br></div><div>For check-cluster, are we planning to add more output showing which node is active(leader), etc in upcoming versions ? </div><div><br></div><div><br></div><div>Thanks a ton for helping sort this out.  I think the patch looks good to be merged post addressing of the comments by Justin along with the man page details for ovsdb-tool.<br></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">I will do some more crash testing for the cluster along with the scale test and keep you posted if something unexpected is noted.</span></div><div><br></div><div> </div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div>Regards,</div><div><br></div><div><br></div></div><div class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882HOEnZb"><div class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 13, 2018 at 11:07 PM, Numan Siddique <span dir="ltr">&lt;<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, Mar 14, 2018 at 7:51 AM, aginwala <span dir="ltr">&lt;<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sure. <div><br></div><div>To add on , I also ran for nb db <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">too </span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">using different port </span> and Node2 crashes with same error :</div><div># Node 2</div><div>/usr/share/openvswitch/scripts<wbr>/ovn-ctl --db-nb-addr=10.99.152.138 --db-nb-port=6641 --db-nb-cluster-remote-addr=&quot;t<wbr>cp:<a href="http://10.99.152.148:6645" target="_blank">10.99.152.148:6645</a>&quot; --db-nb-cluster-local-addr=&quot;tc<wbr>p:<a href="http://10.99.152.138:6645" target="_blank">10.99.152.138:6645</a>&quot; start_nb_ovsdb<br></div><div><div>ovsdb-server: ovsdb error: /etc/openvswitch/ovnnb_db.db: cannot identify file type</div></div><div><br></div><div><br></div></div></blockquote><div><br></div></span><div>Hi Aliasgar,<br></div><div><br></div><div>It worked for me. Can you delete the old db files in /etc/openvswitch/ and try running the commands again ?</div><div><br></div><div>Below are the commands I ran in my setup.</div><div><br></div><div>Node 1</div><div>-------</div><div>sudo /usr/share/openvswitch/scripts<wbr>/ovn-ctl  --db-sb-addr=192.168.121.91 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-local-addr=tcp<wbr>:<a href="http://192.168.121.91:6644" target="_blank">192.168.121.91:6644</a> start_sb_ovsdb<br></div><div><br></div><div>Node 2</div><div>---------</div><div>sudo /usr/share/openvswitch/scripts<wbr>/ovn-ctl  --db-sb-addr=192.168.121.87 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-local-addr=&quot;tc<wbr>p:<a href="http://192.168.121.87:6644" target="_blank">192.168.121.87:6644</a>&quot; --db-sb-cluster-remote-addr=&quot;t<wbr>cp:<a href="http://192.168.121.91:6644" target="_blank">192.168.121.91:6644</a>&quot;  start_sb_ovsdb<br></div><div><br></div><div>Node 3</div><div>---------</div><div><div>sudo /usr/share/openvswitch/scripts<wbr>/ovn-ctl  --db-sb-addr=192.168.121.78 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-local-addr=&quot;tc<wbr>p:<a href="http://192.168.121.78:6644" target="_blank">192.168.121.78:6644</a>&quot; --db-sb-cluster-remote-addr=&quot;t<wbr>cp:<a href="http://192.168.121.91:6644" target="_blank">192.168.121.91:6644</a>&quot;  start_sb_ovsdb</div></div><div><br></div><div><br></div><div><br></div><div>Thanks</div><span class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681HOEnZb"><font color="#888888"><div>Numan</div></font></span><div><div class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681h5"><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div></div><div class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-HOEnZb"><div class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 13, 2018 at 9:40 AM, Numan Siddique <span dir="ltr">&lt;<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Tue, Mar 13, 2018 at 9:46 PM, aginwala <span dir="ltr">&lt;<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks Numan for the response.<div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">There is no command <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">start_cluster_sb_ovsdb in the source code too. Is that in a separate commit somewhere? Hence, I used <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">start_sb_ovsdb which I think would not be a right choice? </span></span></div></div></div></blockquote><div><br></div></span><div>Sorry, I meant start_sb_ovsdb. Strange that it didn&#39;t work for you. Let me try it out again and update this thread.</div><div><br></div><div>Thanks</div><span class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891HOEnZb"><font color="#888888"><div>Numan</div></font></span><div><div class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> </span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"># Node1  <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">came up as expected. </span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ovn-ctl --db-sb-addr=10.99.152.148 --db-sb-port=6642 --db-sb-create-insecure-</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">remote<wbr>=yes --db-sb-cluster-local-addr=&quot;</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">tc<wbr>p:</span><a href="http://10.99.152.148:6644/" style="color:rgb(17,85,204);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" target="_blank">10.99.152.148:6644</a><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">&quot; start_sb_ovsdb<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">. </span></span><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"># verifying its a clustered db with </span></span>ovsdb-tool db-local-address /etc/openvswitch/ovnsb_db.db</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">tcp:<a href="http://10.99.152.148:6644" target="_blank">10.99.152.148:6644</a></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"># ovn-sbctl show works fine and chassis are being populated correctly.<br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">#Node 2 fails with error: </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div>/usr/share/openvswitch/scripts<wbr>/ovn-ctl --db-sb-addr=10.99.152.138 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-remote-addr=&quot;t<wbr>cp:<a href="http://10.99.152.148:6644" target="_blank">10.99.152.148:6644</a>&quot; --db-sb-cluster-local-addr=&quot;tc<wbr>p:<a href="http://10.99.152.138:6644" target="_blank">10.99.152.138:6644</a>&quot; start_sb_ovsdb</div><span class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891m_7002243843969463772gmail-"><div>ovsdb-server: ovsdb error: /etc/openvswitch/ovnsb_db.db: cannot identify file type<br></div><div><br></div></span><div># So i did start the sb db the usual way using start_ovsdb to just get the db file created and killed the sb pid and re-ran the command which gave actual error where it complains for join-cluster command that is being called internally</div><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">/usr/share/openvswitch/scripts<wbr>/ovn-ctl --db-sb-addr=10.99.152.138 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-remote-addr=&quot;t<wbr>cp:<a href="http://10.99.152.148:6644" target="_blank">10.99.152.148:6644</a>&quot; --db-sb-cluster-local-addr=&quot;tc<wbr>p:<a href="http://10.99.152.138:6644" target="_blank">10.99.152.138:6644</a>&quot; start_sb_ovsdb</div></div><div>ovsdb-tool: /etc/openvswitch/ovnsb_db.db: not a clustered database</div><div> * Backing up database to /etc/openvswitch/ovnsb_db.db.b<wbr>ackup1.15.0-70426956</div><div>ovsdb-tool: &#39;join-cluster&#39; command requires at least 4 arguments</div><div> * Creating cluster database /etc/openvswitch/ovnsb_db.db from existing one</div></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"># based on above error I killed the sb db pid again and  try to create a local cluster on node  then re-ran the join operation as per the source code function. </div><div style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">ovsdb-tool join-cluster /etc/openvswitch/ovnsb_db.db OVN_Southbound tcp:<a href="http://10.99.152.138:6644" target="_blank">10.99.152.138:6644</a> tcp:<a href="http://10.99.152.148:6644" target="_blank">10.99.152.148:6644</a> which still complains</div><div style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">ovsdb-tool: I/O error: /etc/openvswitch/ovnsb_db.db: create failed (File exists)</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"># Node 3: I did not try as I am assuming the same failure as node 2</div><br></div><div><br></div><div>Let me know may know further. <div><div class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891m_7002243843969463772gmail-h5"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 13, 2018 at 3:08 AM, Numan Siddique <span dir="ltr">&lt;<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Aliasgar,</span><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891m_7002243843969463772gmail-m_-5297065311753068359gmail-m_-4985231986926937825gmail-">On Tue, Mar 13, 2018 at 7:11 AM, aginwala <span dir="ltr">&lt;<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Ben/Noman:<div><br></div><div>I am trying to setup 3 node southbound db cluster  using <a href="https://patchwork.ozlabs.org/patch/854298/" target="_blank">raft10</a> in review. </div><div><div><br></div><div># Node 1 create-cluster</div><div>ovsdb-tool create-cluster /etc/openvswitch/ovnsb_db.db /root/ovs-reviews/ovn/ovn-sb.o<wbr>vsschema tcp:<a href="http://10.99.152.148:6642" target="_blank">10.99.152.148:6642</a></div></div></div></blockquote><div><br></div></span><div>A different port is used for RAFT. So you have to choose another port like 6644 for example.  </div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891m_7002243843969463772gmail-m_-5297065311753068359gmail-m_-4985231986926937825gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div></div><div><br></div><div># Node 2 </div><div>ovsdb-tool join-cluster /etc/openvswitch/ovnsb_db.db OVN_Southbound tcp:<a href="http://10.99.152.138:6642" target="_blank">10.99.152.138:6642</a> tcp:<a href="http://10.99.152.148:6642" target="_blank">10.99.152.148:6642</a> --cid 5dfcb678-bb1d-4377-b02d-a380ed<wbr>ec2982<br></div><div><br></div><div>#Node 3 </div><div>ovsdb-tool join-cluster /etc/openvswitch/ovnsb_db.db OVN_Southbound tcp:<a href="http://10.99.152.101:6642" target="_blank">10.99.152.101:6642</a> tcp:<a href="http://10.99.152.138:6642" target="_blank">10.99.152.138:6642</a> tcp:<a href="http://10.99.152.148:6642" target="_blank">10.99.152.148:6642</a> --cid 5dfcb678-bb1d-4377-b02d-a380ed<wbr>ec2982<br></div><div><br></div><div># ovn remote is set to all 3 nodes</div><div>external_ids:ovn-remote=&quot;tcp:<a href="http://10.99.152.148:6642" target="_blank">1<wbr>0.99.152.148:6642</a>, tcp:<a href="http://10.99.152.138:6642" target="_blank">10.99.152.138:6642</a>, tcp:<a href="http://10.99.152.101:6642" target="_blank">10.99.152.101:6642</a>&quot;<br></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div># Starting sb db on node 1 using below command on node 1:</div><div><br></div><div>ovsdb-server --detach --monitor -vconsole:off -vraft -vjsonrpc --log-file=/var/log/openvswitc<wbr>h/ovsdb-server-sb.log --pidfile=/var/run/openvswitch<wbr>/ovnsb_db.pid --remote=db:OVN_Southbound,SB_<wbr>Global,connections --unixctl=ovnsb_db.ctl --private-key=db:OVN_Southboun<wbr>d,SSL,private_key --certificate=db:OVN_Southboun<wbr>d,SSL,certificate --ca-cert=db:OVN_Southbound,SS<wbr>L,ca_cert --ssl-protocols=db:OVN_Southbo<wbr>und,SSL,ssl_protocols --ssl-ciphers=db:OVN_Southboun<wbr>d,SSL,ssl_ciphers --remote=punix:/var/run/openvs<wbr>witch/ovnsb_db.sock /etc/openvswitch/ovnsb_db.db<br></div><div><br></div><div># check-cluster is returning nothing <div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">ovsdb-tool check-cluster /etc/openvswitch/ovnsb_db.db</div></div><div><br></div><div># ovsdb-server-sb.log below shows the leader is elected with only one server and there are rbac related debug logs with rpc replies and empty params with no errors</div><div><br></div><div><div>2018-03-13T01:12:02Z|00002|raf<wbr>t|DBG|server 63d1 added to configuration</div><div>2018-03-13T01:12:02Z|00003|raf<wbr>t|INFO|term 6: starting election</div><div>2018-03-13T01:12:02Z|00004|raf<wbr>t|INFO|term 6: elected leader by 1+ of 1 servers</div></div><div><br></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Now Starting the ovsdb-server on the other clusters fails saying </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">ovsdb-server: ovsdb error: /etc/openvswitch/ovnsb_db.db: cannot identify file type</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Also noticed that man ovsdb-tool is missing cluster details. Might want to address it in the same patch or different. </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891m_7002243843969463772gmail-m_-5297065311753068359gmail-m_-4985231986926937825gmail-m_-2613566027972923721gmail-m_6242263053178116842gmail-Apple-interchange-newline">Please advise to what is missing here for running ovn-sbctl show as this command hangs.</span><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><br class="m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891m_7002243843969463772gmail-m_-5297065311753068359gmail-m_-4985231986926937825gmail-m_-2613566027972923721gmail-m_6242263053178116842gmail-Apple-interchange-newline"></div></div></div></blockquote><div><br></div><div><br></div></span><div>I think you can use the ovn-ctl command &quot;start_cluster_sb_ovsdb&quot; for your testing (atleast for now)</div><div><br></div><div>For your setup, I think you can start the cluster as</div><div><br></div><div># Node 1</div><div>ovn-ctl --db-sb-addr=10.99.152.148 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-local-addr=&quot;tc<wbr>p:<a href="http://10.99.152.148:6644" target="_blank">10.99.152.148:6644</a>&quot; start_cluster_sb_ovsdb</div><div><br></div><div># Node 2</div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ovn-ctl --db-sb-addr=10.99.152.138 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-local-addr=&quot;tc<wbr>p:10.99.152.138</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">:6644&quot;  --db-sb-cluster-remote-addr=&quot;t<wbr>cp:<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><a href="http://10.99.152.148" target="_blank">10.99.152.148</a>:</span>6644&quot; start_cluster_sb_ovsdb</span><br></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"># Node 3</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);float:none;display:inline">ovn-ctl --db-sb-addr=10.99.152.101 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-local-addr=&quot;tc<wbr>p:10.99.152.101</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);float:none;display:inline">:6644&quot;  --db-sb-cluster-remote-addr=&quot;t<wbr>cp:<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><a href="http://10.99.152.148" target="_blank">10.99.152.148</a>:</span>6644&quot;<span> </span>start_c<wbr>luster_sb_ovsdb</span><br></span></div><div></div></div></div></div></blockquote><div><br></div><div><br></div><div><br></div><div><br></div><div></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Let me know how it goes.</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Thanks</span></div><div>Numan</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"> </div><br></div></div>
<br>______________________________<wbr>_________________<br>
discuss mailing list<br>
<a href="mailto:discuss@openvswitch.org" target="_blank">discuss@openvswitch.org</a><br>
<a href="https://mail.openvswitch.org/mailman/listinfo/ovs-discuss" rel="noreferrer" target="_blank">https://mail.openvswitch.org/m<wbr>ailman/listinfo/ovs-discuss</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div></div></div></div></div>
<br>______________________________<wbr>_________________<br>
discuss mailing list<br>
<a href="mailto:discuss@openvswitch.org">discuss@openvswitch.org</a><br>
<a href="https://mail.openvswitch.org/mailman/listinfo/ovs-discuss" rel="noreferrer" target="_blank">https://mail.openvswitch.org/<wbr>mailman/listinfo/ovs-discuss</a><br>
<br></blockquote></div><br></div></div>