<div dir="ltr">Hu Numan:<div><br></div><div>You need to use --db as you are now running db in cluster, you can access data from any of the three dbs. </div><div><br></div><div>So if the leader crashes, it re-elects from the other two. Below is the e.g. command:</div><div><br></div><div># export remote="tcp:<a href="http://192.168.220.103:6641" target="_blank">192.168.220.103:66<wbr>41</a>,tcp:<a href="http://192.168.220.102:6641" target="_blank">192.168.220.102:6641</a>,tc<wbr>p:<a href="http://192.168.220.101:6641" target="_blank">192.168.220.101:6641</a>"</div><div># kill -9 3985<br></div><div># ovn-nbctl --db=$remote show<br></div><div>switch 1d86ab4e-c8bf-4747-a716-8832a2<wbr>85d58c (ls1)</div><div># ovn-nbctl --db=$remote ls-del ls1</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Hope it helps! </div><div><br></div><div>Regards,</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 27, 2018 at 10:01 AM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Hi </span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);float:none;display:inline">Aliasgar,</span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">In your setup, if you kill the leader what is the behaviour ? Are you still able to create or delete any resources ? Is a new leader elected ?</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">In my setup, the command "ovn-nbctl ls-add" for example blocks until I restart the ovsdb-server in node 1. And I don't see any other ovsdb-server becoming leader. May be I have configured wrongly.</span></div><div><span style="font-size:12.8px">Could you please test this scenario if not yet please and let me know your observations if possible.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thanks</span></div><span class="HOEnZb"><font color="#888888"><div><span style="font-size:12.8px">Numan</span></div></font></span><div><div class="h5"><div><span style="font-size:12.8px"><br></span><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 22, 2018 at 12:28 PM, Han Zhou <span dir="ltr"><<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Sounds good.<br><br></div>Just checked the patch, by default the C IDL has "leader_only" as true, which ensures that connection is to leader only. This is the case for northd. So the lock works for northd hot active-standby purpose if all the ovsdb endpoints of a cluster are specified to northd, since all northds are connecting to the same DB, the leader.<br><br></div>For neutron networking-ovn, this may not work yet, since I didn't see such logic in the python IDL in current patch series. It would be good if we add similar logic for python IDL. (@ben/numan, correct me if I am wrong)<div><div class="m_2804075142644265844h5"><br><div><div><div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 6:49 PM, aginwala <span dir="ltr"><<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi :</div><div><br></div><div>Just sorted out the correct settings and northd also works in ha in raft.</div><div><br></div><div>There were 2 issues in the setup:</div><div>1. I had started nb db without --db-nb-create-insecure-remote</div><div>2. I also started northd locally on all 3 without remote which is like all three northd trying to lock the ovsdb locally.</div><div><br></div><div>Hence, the duplicate logs were populated in the southbound datapath due to multiple northd trying to write the local copy. </div><div><br></div><div>So, I now start nb db with --db-nb-create-insecure-remote and northd on all 3 nodes using below command:</div><div><br></div><div>ovn-northd -vconsole:emer -vsyslog:err -vfile:info --ovnnb-db="tcp:<a href="http://10.169.125.152:6641" target="_blank">10.169.125.152<wbr>:6641</a>,tcp:<a href="http://10.169.125.131:6641" target="_blank">10.169.125.131:6641</a>,<wbr>tcp:<a href="http://10.148.181.162:6641" target="_blank">10.148.181.162:6641</a>" --ovnsb-db="tcp:<a href="http://10.169.125.152:6642" target="_blank">10.169.125.152<wbr>:6642</a>,tcp:<a href="http://10.169.125.131:6642" target="_blank">10.169.125.131:6642</a>,<wbr>tcp:<a href="http://10.148.181.162:6642" target="_blank">10.148.181.162:6642</a>" --no-chdir --log-file=/var/log/openvswitc<wbr>h/ovn-northd.log --pidfile=/var/run/openvswitch<wbr>/ovn-northd.pid --detach --monitor</div><div><br></div><div><br></div><div>#At start, northd went active on the leader node and standby on other two nodes.</div><div><br></div><div>#After old leader crashed and new leader got elected, northd goes active on any of the remaining 2 nodes as per sample logs below from non-leader node:</div><div>2018-03-22T00:20:30.732Z|00023<wbr>|ovn_northd|INFO|ovn-northd lock lost. This ovn-northd instance is now on standby.</div><div>2018-03-22T00:20:30.743Z|00024<wbr>|ovn_northd|INFO|ovn-northd lock acquired. This ovn-northd instance is now active.</div><div><br></div><div># Also ovn-controller works similar way if leader goes down and connects to any of the remaining 2 nodes:</div><div>2018-03-22T01:21:56.250Z|00029<wbr>|ovsdb_idl|INFO|tcp:<a href="http://10.148.181.162:6642" target="_blank">10.148.181<wbr>.162:6642</a>: clustered database server is disconnected from cluster; trying another server</div><div>2018-03-22T01:21:56.250Z|00030<wbr>|reconnect|INFO|tcp:<a href="http://10.148.181.162:6642" target="_blank">10.148.181<wbr>.162:6642</a>: connection attempt timed out</div><div>2018-03-22T01:21:56.250Z|00031<wbr>|reconnect|INFO|tcp:<a href="http://10.148.181.162:6642" target="_blank">10.148.181<wbr>.162:6642</a>: waiting 4 seconds before reconnect</div><div>2018-03-22T01:23:52.417Z|00043<wbr>|reconnect|INFO|tcp:<a href="http://10.148.181.162:6642" target="_blank">10.148.181<wbr>.162:6642</a>: connected</div><div><br></div><div><br></div><div><br></div><div>Above settings will also work if we put all the nodes behind the vip and updates the ovn configs to use vips. So we don't need pacemaker explicitly for northd HA :).</div><div><br></div><div>Since the setup is complete now, I will populate the same in scale test env and see how it behaves. </div><div><br></div><div>@Numan: We can try the same with networking-ovn integration and see if we find anything weird there too. Not sure if you have any exclusive findings for this case. </div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Let me know if something else is missed here.</span></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Regards,</div></div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011HOEnZb"><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 2:50 PM, Han Zhou <span dir="ltr"><<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Ali, sorry if I misunderstand what you are saying, but pacemaker here is for northd HA. pacemaker itself won't point to any ovsdb cluster node. All northds can point to a LB VIP for the ovsdb cluster, so if a member of ovsdb cluster is down it won't have impact to northd.<br><br></div>Without clustering support of the ovsdb lock, I think this is what we have now for northd HA. Please suggest if anyone has any other idea. Thanks :)<br></div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124HOEnZb"><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 1:12 PM, aginwala <span dir="ltr"><<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">:) The only thing is while using pacemaker, if the node that pacemaker if pointing to is down, all the active/standby northd nodes have to be updated to new node from the cluster. But will dig in more to see what else I can find. <div><br></div><div>@Ben: Any suggestions further?<br><div><br></div><div><br></div><div>Regards,</div></div></div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929HOEnZb"><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 10:22 AM, Han Zhou <span dir="ltr"><<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, Mar 21, 2018 at 9:49 AM, aginwala <span dir="ltr"><<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Numan:<div><br></div><div>Yup agree with the locking part. For now; yes I am running northd on one node. I might right a script to monitor northd in cluster so that if the node where it's running goes down, script can spin up northd on one other active nodes as a dirty hack.</div><div><br></div></div></blockquote></span><div>The "dirty hack" is pacemaker :)<br> <br></div><div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div>Sure, will await for the inputs from Ben too on this and see how complex would it be to roll out this feature.</div><div><br></div><div><br></div><div>Regards,</div><div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641h5"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 5:43 AM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Aliasgar,<div><br></div><div>ovsdb-server maintains locks per each connection and not across the db. A workaround for you now would be to configure all the ovn-northd instances to connect to one ovsdb-server if you want to have active/standy.</div><div><br></div><div>Probably Ben can answer if there is a plan to support ovsdb locks across the db. We also need this support in networking-ovn as it also uses ovsdb locks.</div><div><br></div><div>Thanks</div><span class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156HOEnZb"><font color="#888888"><div>Numan</div></font></span><div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156h5"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 1:40 PM, aginwala <span dir="ltr"><<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Numan:</div><div><br></div><div>Just figured out that ovn-northd is running as active on all 3 nodes instead of one active instance as I continued to test further which results in db errors as per logs.</div><div><br></div><div><br></div><div># on node 3, I run ovn-nbctl ls-add ls2 ; it populates below logs in ovn-north</div><div>2018-03-21T06:01:59.442Z|00007<wbr>|ovsdb_idl|WARN|transaction error: {"details":"Transaction causes multiple rows in \"Datapath_Binding\" table to have identical values (1) for index on column \"tunnel_key\". First row, with UUID 8c5d9342-2b90-4229-8ea1-001a73<wbr>3a915c, was inserted by this transaction. Second row, with UUID 8e06f919-4cc7-4ffc-9a79-20ce66<wbr>63b683, existed in the database before this transaction and was not modified by the transaction.","error":"constra<wbr>int violation"}</div><div><br></div><div>In southbound datapath list, 2 duplicate records gets created for same switch.</div><div><br></div><div># ovn-sbctl list Datapath</div><div>_uuid : b270ae30-3458-445f-95d2-b14e8e<wbr>bddd01</div><div>external_ids : {logical-switch="4d6674e3-ff9f<wbr>-4f38-b050-0fa9bec9e34d", name="ls2"}</div><div>tunnel_key : 2</div><div><br></div><div>_uuid : 8e06f919-4cc7-4ffc-9a79-20ce66<wbr>63b683</div><div>external_ids : {logical-switch="4d6674e3-ff9f<wbr>-4f38-b050-0fa9bec9e34d", name="ls2"}</div><div>tunnel_key : 1</div><div><br></div><div><br></div><div><br></div><div># on nodes 1 and 2 where northd is running, it gives below error:</div><div>2018-03-21T06:01:59.437Z|00008<wbr>|ovsdb_idl|WARN|transaction error: {"details":"cannot delete Datapath_Binding row 8e06f919-4cc7-4ffc-9a79-20ce66<wbr>63b683 because of 17 remaining reference(s)","error":"referen<wbr>tial integrity violation"}</div><div><br></div><div>As per commit message, for northd I re-tried setting --ovnnb-db="tcp:<a href="http://10.169.125.152:6641" target="_blank">10.169.125.152<wbr>:6641</a>,tcp:<a href="http://10.169.125.131:6641" target="_blank">10.169.125.131:6641</a>,<wbr>tcp:<a href="http://10.148.181.162:6641" target="_blank">10.148.181.162:6641</a>" and --ovnsb-db="tcp:<a href="http://10.169.125.152:6642" target="_blank">10.169.125.152<wbr>:6642</a>,tcp:<a href="http://10.169.125.131:6642" target="_blank">10.169.125.131:6642</a>,<wbr>tcp:<a href="http://10.148.181.162:6642" target="_blank">10.148.181.162:6642</a>" and it did not help either.</div><div> </div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">There is no issue if I keep running only one instance of northd on any of these 3 nodes. </span>Hence, wanted to know is there something else missing here to make only one northd instance as active and rest as standby? </div><div><br></div><div><br></div><div>Regards,</div></div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104HOEnZb"><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 15, 2018 at 3:09 AM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">That's great<span class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792HOEnZb"><font color="#888888"><div><br></div><div>Numan</div><div><br></div></font></span></div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792HOEnZb"><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 15, 2018 at 2:57 AM, aginwala <span dir="ltr"><<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Numan:<div><br></div><div>I tried on new nodes (kernel : 4.4.0-104-generic , Ubuntu 16.04)with fresh installation and it worked super fine for <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">both sb and nb dbs</span>. <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Seems like some kernel issue on the previous nodes when I re-installed raft patch as I was running different ovs version on those nodes before.</span></div><div><br></div><div><br></div><div>For 2 HVs, I now set <span style="color:rgb(51,51,51);font-family:monospace;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ovn-remote</span>="tcp:<a href="http://10.169.125.152:6642" target="_blank">10.169.125.152<wbr>:6642</a>, tcp:<a href="http://10.169.125.131:6642" target="_blank">10.169.125.131:6642</a>, tcp:<a href="http://10.148.181.162:6642" target="_blank">10.148.181.162:6642</a>" and started controller and it works super fine. </div><div><br></div><div><br></div><div>Did some failover testing by rebooting/killing the leader (<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">10.169.125.152</span>) and bringing it back up and it works as expected. Nothing weird noted so far. </div><div><br></div><div># check-cluster gives below data one of the node(<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">10.148.181.162)</span> post leader failure</div><div><br></div><div>ovsdb-tool check-cluster /etc/openvswitch/ovnsb_db.db</div><div>ovsdb-tool: leader /etc/openvswitch/ovnsb_db.db for term 2 has log entries only up to index 18446744073709551615, but index 9 was committed in a previous term (e.g. by /etc/openvswitch/ovnsb_db.db)</div><div><br></div><div><br></div><div>For check-cluster, are we planning to add more output showing which node is active(leader), etc in upcoming versions ? </div><div><br></div><div><br></div><div>Thanks a ton for helping sort this out. I think the patch looks good to be merged post addressing of the comments by Justin along with the man page details for ovsdb-tool.<br></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">I will do some more crash testing for the cluster along with the scale test and keep you posted if something unexpected is noted.</span></div><div><br></div><div> </div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div>Regards,</div><div><br></div><div><br></div></div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882HOEnZb"><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 13, 2018 at 11:07 PM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, Mar 14, 2018 at 7:51 AM, aginwala <span dir="ltr"><<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sure. <div><br></div><div>To add on , I also ran for nb db <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">too </span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">using different port </span> and Node2 crashes with same error :</div><div># Node 2</div><div>/usr/share/openvswitch/scripts<wbr>/ovn-ctl --db-nb-addr=10.99.152.138 --db-nb-port=6641 --db-nb-cluster-remote-addr="t<wbr>cp:<a href="http://10.99.152.148:6645" target="_blank">10.99.152.148:6645</a>" --db-nb-cluster-local-addr="tc<wbr>p:<a href="http://10.99.152.138:6645" target="_blank">10.99.152.138:6645</a>" start_nb_ovsdb<br></div><div><div>ovsdb-server: ovsdb error: /etc/openvswitch/ovnnb_db.db: cannot identify file type</div></div><div><br></div><div><br></div></div></blockquote><div><br></div></span><div>Hi Aliasgar,<br></div><div><br></div><div>It worked for me. Can you delete the old db files in /etc/openvswitch/ and try running the commands again ?</div><div><br></div><div>Below are the commands I ran in my setup.</div><div><br></div><div>Node 1</div><div>-------</div><div>sudo /usr/share/openvswitch/scripts<wbr>/ovn-ctl --db-sb-addr=192.168.121.91 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-local-addr=tcp<wbr>:<a href="http://192.168.121.91:6644" target="_blank">192.168.121.91:6644</a> start_sb_ovsdb<br></div><div><br></div><div>Node 2</div><div>---------</div><div>sudo /usr/share/openvswitch/scripts<wbr>/ovn-ctl --db-sb-addr=192.168.121.87 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-local-addr="tc<wbr>p:<a href="http://192.168.121.87:6644" target="_blank">192.168.121.87:6644</a>" --db-sb-cluster-remote-addr="t<wbr>cp:<a href="http://192.168.121.91:6644" target="_blank">192.168.121.91:6644</a>" start_sb_ovsdb<br></div><div><br></div><div>Node 3</div><div>---------</div><div><div>sudo /usr/share/openvswitch/scripts<wbr>/ovn-ctl --db-sb-addr=192.168.121.78 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-local-addr="tc<wbr>p:<a href="http://192.168.121.78:6644" target="_blank">192.168.121.78:6644</a>" --db-sb-cluster-remote-addr="t<wbr>cp:<a href="http://192.168.121.91:6644" target="_blank">192.168.121.91:6644</a>" start_sb_ovsdb</div></div><div><br></div><div><br></div><div><br></div><div>Thanks</div><span class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681HOEnZb"><font color="#888888"><div>Numan</div></font></span><div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681h5"><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div></div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-HOEnZb"><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 13, 2018 at 9:40 AM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Tue, Mar 13, 2018 at 9:46 PM, aginwala <span dir="ltr"><<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks Numan for the response.<div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">There is no command <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">start_cluster_sb_ovsdb in the source code too. Is that in a separate commit somewhere? Hence, I used <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">start_sb_ovsdb which I think would not be a right choice? </span></span></div></div></div></blockquote><div><br></div></span><div>Sorry, I meant start_sb_ovsdb. Strange that it didn't work for you. Let me try it out again and update this thread.</div><div><br></div><div>Thanks</div><span class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891HOEnZb"><font color="#888888"><div>Numan</div></font></span><div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> </span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"># Node1 <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">came up as expected. </span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ovn-ctl --db-sb-addr=10.99.152.148 --db-sb-port=6642 --db-sb-create-insecure-</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">remote<wbr>=yes --db-sb-cluster-local-addr="</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">tc<wbr>p:</span><a href="http://10.99.152.148:6644/" style="color:rgb(17,85,204);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255)" target="_blank">10.99.152.148:6644</a><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">" start_sb_ovsdb<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">. </span></span><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"># verifying its a clustered db with </span></span>ovsdb-tool db-local-address /etc/openvswitch/ovnsb_db.db</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">tcp:<a href="http://10.99.152.148:6644" target="_blank">10.99.152.148:6644</a></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"># ovn-sbctl show works fine and chassis are being populated correctly.<br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">#Node 2 fails with error: </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><div>/usr/share/openvswitch/scripts<wbr>/ovn-ctl --db-sb-addr=10.99.152.138 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-remote-addr="t<wbr>cp:<a href="http://10.99.152.148:6644" target="_blank">10.99.152.148:6644</a>" --db-sb-cluster-local-addr="tc<wbr>p:<a href="http://10.99.152.138:6644" target="_blank">10.99.152.138:6644</a>" start_sb_ovsdb</div><span class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891m_7002243843969463772gmail-"><div>ovsdb-server: ovsdb error: /etc/openvswitch/ovnsb_db.db: cannot identify file type<br></div><div><br></div></span><div># So i did start the sb db the usual way using start_ovsdb to just get the db file created and killed the sb pid and re-ran the command which gave actual error where it complains for join-cluster command that is being called internally</div><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">/usr/share/openvswitch/scripts<wbr>/ovn-ctl --db-sb-addr=10.99.152.138 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-remote-addr="t<wbr>cp:<a href="http://10.99.152.148:6644" target="_blank">10.99.152.148:6644</a>" --db-sb-cluster-local-addr="tc<wbr>p:<a href="http://10.99.152.138:6644" target="_blank">10.99.152.138:6644</a>" start_sb_ovsdb</div></div><div>ovsdb-tool: /etc/openvswitch/ovnsb_db.db: not a clustered database</div><div> * Backing up database to /etc/openvswitch/ovnsb_db.db.b<wbr>ackup1.15.0-70426956</div><div>ovsdb-tool: 'join-cluster' command requires at least 4 arguments</div><div> * Creating cluster database /etc/openvswitch/ovnsb_db.db from existing one</div></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"># based on above error I killed the sb db pid again and try to create a local cluster on node then re-ran the join operation as per the source code function. </div><div style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">ovsdb-tool join-cluster /etc/openvswitch/ovnsb_db.db OVN_Southbound tcp:<a href="http://10.99.152.138:6644" target="_blank">10.99.152.138:6644</a> tcp:<a href="http://10.99.152.148:6644" target="_blank">10.99.152.148:6644</a> which still complains</div><div style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">ovsdb-tool: I/O error: /etc/openvswitch/ovnsb_db.db: create failed (File exists)</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"># Node 3: I did not try as I am assuming the same failure as node 2</div><br></div><div><br></div><div>Let me know may know further. <div><div class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891m_7002243843969463772gmail-h5"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 13, 2018 at 3:08 AM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Aliasgar,</span><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891m_7002243843969463772gmail-m_-5297065311753068359gmail-m_-4985231986926937825gmail-">On Tue, Mar 13, 2018 at 7:11 AM, aginwala <span dir="ltr"><<a href="mailto:aginwala@asu.edu" target="_blank">aginwala@asu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Ben/Noman:<div><br></div><div>I am trying to setup 3 node southbound db cluster using <a href="https://patchwork.ozlabs.org/patch/854298/" target="_blank">raft10</a> in review. </div><div><div><br></div><div># Node 1 create-cluster</div><div>ovsdb-tool create-cluster /etc/openvswitch/ovnsb_db.db /root/ovs-reviews/ovn/ovn-sb.o<wbr>vsschema tcp:<a href="http://10.99.152.148:6642" target="_blank">10.99.152.148:6642</a></div></div></div></blockquote><div><br></div></span><div>A different port is used for RAFT. So you have to choose another port like 6644 for example. </div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891m_7002243843969463772gmail-m_-5297065311753068359gmail-m_-4985231986926937825gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div></div><div><br></div><div># Node 2 </div><div>ovsdb-tool join-cluster /etc/openvswitch/ovnsb_db.db OVN_Southbound tcp:<a href="http://10.99.152.138:6642" target="_blank">10.99.152.138:6642</a> tcp:<a href="http://10.99.152.148:6642" target="_blank">10.99.152.148:6642</a> --cid 5dfcb678-bb1d-4377-b02d-a380ed<wbr>ec2982<br></div><div><br></div><div>#Node 3 </div><div>ovsdb-tool join-cluster /etc/openvswitch/ovnsb_db.db OVN_Southbound tcp:<a href="http://10.99.152.101:6642" target="_blank">10.99.152.101:6642</a> tcp:<a href="http://10.99.152.138:6642" target="_blank">10.99.152.138:6642</a> tcp:<a href="http://10.99.152.148:6642" target="_blank">10.99.152.148:6642</a> --cid 5dfcb678-bb1d-4377-b02d-a380ed<wbr>ec2982<br></div><div><br></div><div># ovn remote is set to all 3 nodes</div><div>external_ids:ovn-remote="tcp:<a href="http://10.99.152.148:6642" target="_blank">1<wbr>0.99.152.148:6642</a>, tcp:<a href="http://10.99.152.138:6642" target="_blank">10.99.152.138:6642</a>, tcp:<a href="http://10.99.152.101:6642" target="_blank">10.99.152.101:6642</a>"<br></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div># Starting sb db on node 1 using below command on node 1:</div><div><br></div><div>ovsdb-server --detach --monitor -vconsole:off -vraft -vjsonrpc --log-file=/var/log/openvswitc<wbr>h/ovsdb-server-sb.log --pidfile=/var/run/openvswitch<wbr>/ovnsb_db.pid --remote=db:OVN_Southbound,SB_<wbr>Global,connections --unixctl=ovnsb_db.ctl --private-key=db:OVN_Southboun<wbr>d,SSL,private_key --certificate=db:OVN_Southboun<wbr>d,SSL,certificate --ca-cert=db:OVN_Southbound,SS<wbr>L,ca_cert --ssl-protocols=db:OVN_Southbo<wbr>und,SSL,ssl_protocols --ssl-ciphers=db:OVN_Southboun<wbr>d,SSL,ssl_ciphers --remote=punix:/var/run/openvs<wbr>witch/ovnsb_db.sock /etc/openvswitch/ovnsb_db.db<br></div><div><br></div><div># check-cluster is returning nothing <div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">ovsdb-tool check-cluster /etc/openvswitch/ovnsb_db.db</div></div><div><br></div><div># ovsdb-server-sb.log below shows the leader is elected with only one server and there are rbac related debug logs with rpc replies and empty params with no errors</div><div><br></div><div><div>2018-03-13T01:12:02Z|00002|raf<wbr>t|DBG|server 63d1 added to configuration</div><div>2018-03-13T01:12:02Z|00003|raf<wbr>t|INFO|term 6: starting election</div><div>2018-03-13T01:12:02Z|00004|raf<wbr>t|INFO|term 6: elected leader by 1+ of 1 servers</div></div><div><br></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Now Starting the ovsdb-server on the other clusters fails saying </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">ovsdb-server: ovsdb error: /etc/openvswitch/ovnsb_db.db: cannot identify file type</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Also noticed that man ovsdb-tool is missing cluster details. Might want to address it in the same patch or different. </div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891m_7002243843969463772gmail-m_-5297065311753068359gmail-m_-4985231986926937825gmail-m_-2613566027972923721gmail-m_6242263053178116842gmail-Apple-interchange-newline">Please advise to what is missing here for running ovn-sbctl show as this command hangs.</span><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><br class="m_2804075142644265844m_2781121317021346311m_1800271113111618011m_-8399299218989951124m_-8920890933544636929m_3421646272532524144m_1594826320034701641m_6021648639681040183m_-5349386396769448156m_-4081634083131670776m_2213493976743944104m_-9072150127864706792m_2258937762538382882m_8777951427526992681m_-1142771216983740967gmail-m_-8667090609025108891m_7002243843969463772gmail-m_-5297065311753068359gmail-m_-4985231986926937825gmail-m_-2613566027972923721gmail-m_6242263053178116842gmail-Apple-interchange-newline"></div></div></div></blockquote><div><br></div><div><br></div></span><div>I think you can use the ovn-ctl command "start_cluster_sb_ovsdb" for your testing (atleast for now)</div><div><br></div><div>For your setup, I think you can start the cluster as</div><div><br></div><div># Node 1</div><div>ovn-ctl --db-sb-addr=10.99.152.148 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-local-addr="tc<wbr>p:<a href="http://10.99.152.148:6644" target="_blank">10.99.152.148:6644</a>" start_cluster_sb_ovsdb</div><div><br></div><div># Node 2</div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ovn-ctl --db-sb-addr=10.99.152.138 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-local-addr="tc<wbr>p:10.99.152.138</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">:6644" --db-sb-cluster-remote-addr="t<wbr>cp:<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><a href="http://10.99.152.148" target="_blank">10.99.152.148</a>:</span>6644" start_cluster_sb_ovsdb</span><br></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"># Node 3</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);float:none;display:inline">ovn-ctl --db-sb-addr=10.99.152.101 --db-sb-port=6642 --db-sb-create-insecure-remote<wbr>=yes --db-sb-cluster-local-addr="tc<wbr>p:10.99.152.101</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);float:none;display:inline">:6644" --db-sb-cluster-remote-addr="t<wbr>cp:<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><a href="http://10.99.152.148" target="_blank">10.99.152.148</a>:</span>6644"<span> </span>start_c<wbr>luster_sb_ovsdb</span><br></span></div><div></div></div></div></div></blockquote><div><br></div><div><br></div><div><br></div><div><br></div><div></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Let me know how it goes.</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Thanks</span></div><div>Numan</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"> </div><br></div></div>
<br>______________________________<wbr>_________________<br>
discuss mailing list<br>
<a href="mailto:discuss@openvswitch.org" target="_blank">discuss@openvswitch.org</a><br>
<a href="https://mail.openvswitch.org/mailman/listinfo/ovs-discuss" rel="noreferrer" target="_blank">https://mail.openvswitch.org/m<wbr>ailman/listinfo/ovs-discuss</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div></div></div></div></div>
<br>______________________________<wbr>_________________<br>
discuss mailing list<br>
<a href="mailto:discuss@openvswitch.org" target="_blank">discuss@openvswitch.org</a><br>
<a href="https://mail.openvswitch.org/mailman/listinfo/ovs-discuss" rel="noreferrer" target="_blank">https://mail.openvswitch.org/m<wbr>ailman/listinfo/ovs-discuss</a><br>
<br></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div></div></div></div></div></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div>