<div dir="ltr">Thanks Numan, in my environment, it's worse, it's even not getting started and the monitor is only called once other than repeatedly for both master/slave or none, do you know if any problem could cause pacemaker have this decision? other resource are good.</div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 1, 2017 at 2:08 AM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span style="font-size:12.8px">Hi HuiXiang,</span><div><span style="font-size:12.8px">Even I am seeing the issue where no node is promoted as master. I will test more, fix and and submit patch set v3.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thanks</span></div><span class="HOEnZb"><font color="#888888"><div><span style="font-size:12.8px">Numan</span></div></font></span><div><div class="h5"><div><span style="font-size:12.8px"><br></span><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 30, 2017 at 4:10 PM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="m_-2119459686872868904gmail-">On Thu, Nov 30, 2017 at 1:15 PM, Hui Xiang <span dir="ltr"><<a href="mailto:xianghuir@gmail.com" target="_blank">xianghuir@gmail.com</a>></span> wrote:<br></span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Numan,<div><br></div><span class="m_-2119459686872868904gmail-"><div>Thanks for helping, I am following your pcs example, but still with no lucky, </div><div><br></div><div>1. Before running any configuration, I stopped all of the ovsdb-server for OVN, and ovn-northd. Deleted ovnnb_active.conf/ovnsb_active<wbr>.conf.</div><div><br></div><div>2. Since I have already had an vip in the cluster, so I chose to use it, it's status is OK.</div><div><div>[root@node-1 ~]# pcs resource show</div><div> vip__management_old<span style="white-space:pre-wrap">        </span>(ocf::es:ns_IPaddr2):<span style="white-space:pre-wrap">        </span>Started node-1.domain.tld</div></div><div><br></div><div>3. Use pcs to create ovndb-servers and constraint</div><div><div>[root@node-1 ~]# pcs resource create tst-ovndb ocf:ovn:ovndb-servers manage_northd=yes master_ip=192.168.0.2 nb_master_port=6641 sb_master_port=6642 master</div><div> ([root@node-1 ~]# pcs resource meta tst-ovndb-master notify=true</div><div> Error: unable to find a resource/clone/master/group: tst-ovndb-master) ## returned error, so I changed into below command.</div></div></span></div></blockquote><div><br></div><div>Hi HuiXiang,</div><div>This command is very important. Without which, pacemaker do not notify the status change and ovsdb-servers would not be promoted or demoted.</div><div>Hence you don't see the notify action getting called in ovn ocf script.</div><div><br></div><div>Can you try with the other command which I shared in my previous email. These commands work fine for me.</div><div><br></div><div>Let me know how it goes.</div><div><br></div><div>Thanks</div><span class="m_-2119459686872868904gmail-HOEnZb"><font color="#888888"><div>Numan</div></font></span><div><div class="m_-2119459686872868904gmail-h5"><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>[root@node-1 ~]# pcs resource master tst-ovndb-master tst-ovndb notify=true</div><div>[root@node-1 ~]# pcs constraint colocation add master tst-ovndb-master with vip__management_old</div><div><br></div><div>4. pcs status</div><div>[root@node-1 ~]# pcs status</div><div> vip__management_old<span style="white-space:pre-wrap">        </span>(ocf::es:ns_IPaddr2):<span style="white-space:pre-wrap">        </span>Started node-1.domain.tld</div><div> Master/Slave Set: tst-ovndb-master [tst-ovndb]</div><div> Stopped: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]</div><div><br></div><div>5. pcs resource show XXX</div><div>[root@node-1 ~]# pcs resource show vip__management_old</div><div> Resource: vip__management_old (class=ocf provider=es type=ns_IPaddr2)</div><div> Attributes: nic=br-mgmt base_veth=br-mgmt-hapr ns_veth=hapr-m ip=192.168.0.2 iflabel=ka cidr_netmask=24 ns=haproxy gateway=none gateway_metric=0 iptables_start_rules=false iptables_stop_rules=false iptables_comment=default-comme<wbr>nt </div><div> Meta Attrs: migration-threshold=3 failure-timeout=60 resource-stickiness=1 </div><div> Operations: monitor interval=3 timeout=30 (vip__management_old-monitor-3<wbr>)</div><div> start interval=0 timeout=30 (vip__management_old-start-0)</div><div> stop interval=0 timeout=30 (vip__management_old-stop-0)</div><div>[root@node-1 ~]# pcs resource show tst-ovndb-master</div><div> Master: tst-ovndb-master</div><div> Meta Attrs: notify=true </div><div> Resource: tst-ovndb (class=ocf provider=ovn type=ovndb-servers)</div><div> Attributes: manage_northd=yes master_ip=192.168.0.2 nb_master_port=6641 sb_master_port=6642 </div><div> Operations: start interval=0s timeout=30s (tst-ovndb-start-timeout-30s)</div><div> stop interval=0s timeout=20s (tst-ovndb-stop-timeout-20s)</div><div> promote interval=0s timeout=50s (tst-ovndb-promote-timeout-50s<wbr>)</div><div> demote interval=0s timeout=50s (tst-ovndb-demote-timeout-50s)</div><div> monitor interval=30s timeout=20s (tst-ovndb-monitor-interval-30<wbr>s)</div><div> monitor interval=10s role=Master timeout=20s (tst-ovndb-monitor-interval-10<wbr>s-role-Master)</div><div> monitor interval=30s role=Slave timeout=20s (tst-ovndb-monitor-interval-30<wbr>s-role-Slave)</div></div><div><br></div><div><br></div><div>6. I have put log in every ovndb-servers op, seems only the monitor op is being called, no promoted by the pacemaker DC:</div><div><div><30>Nov 30 15:22:19 node-1 ovndb-servers(tst-ovndb)[29808<wbr>60]: INFO: ovsdb_server_monitor</div><div><30>Nov 30 15:22:19 node-1 ovndb-servers(tst-ovndb)[29808<wbr>60]: INFO: ovsdb_server_check_status</div><div><30>Nov 30 15:22:19 node-1 ovndb-servers(tst-ovndb)[29808<wbr>60]: INFO: return OCFOCF_NOT_RUNNINGG</div><div><30>Nov 30 15:22:20 node-1 ovndb-servers(tst-ovndb)[29808<wbr>60]: INFO: ovsdb_server_master_update: 7}</div><div><30>Nov 30 15:22:20 node-1 ovndb-servers(tst-ovndb)[29808<wbr>60]: INFO: ovsdb_server_master_update end}</div><div><30>Nov 30 15:22:20 node-1 ovndb-servers(tst-ovndb)[29808<wbr>60]: INFO: monitor is going to return 7</div><div><30>Nov 30 15:22:20 node-1 ovndb-servers(undef)[2980970]: INFO: metadata exit OCF_SUCCESS}</div></div><div><br></div><div><br></div><div>Please take a look, thank you very much.</div><span class="m_-2119459686872868904gmail-m_-3745470431214182794HOEnZb"><font color="#888888"><div>Hui.</div><div><br></div><div><br></div><div><br></div></font></span></div><div class="m_-2119459686872868904gmail-m_-3745470431214182794HOEnZb"><div class="m_-2119459686872868904gmail-m_-3745470431214182794h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 29, 2017 at 11:03 PM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, Nov 29, 2017 at 4:16 PM, Hui Xiang <span dir="ltr"><<a href="mailto:xianghuir@gmail.com" target="_blank">xianghuir@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">FYI, If I have configured a good ovndb-server cluster with one active two slaves, then start pacemaker ovn-servers resource agents, they are all becoming slaves...</div></blockquote><div><br></div></span><div>You don't need to start ovndb-servers. When you create pacemaker resources it would automatically start them and promote on of them.</div><div><br></div><div>One thing which is very important is to create an IPaddr2 resource before and add a colocation constraint so that pacemaker would promote the ovsdb-server in the node</div><div>where IPaddr2 resource is running. This IPaddr2 resource ip should be your master ip.</div><div><br></div><div>Can you please do "pcs resource show <name_of_the_resource>" and share the output ?</div><div><br></div><div>Below is how I normally use for my testing.</div><div><br></div><div>############</div><div><div style="font-size:12.8px"><span class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-il">pcs</span> cluster cib tmp-cib.xml</div><div style="font-size:12.8px">cp tmp-cib.xml tmp-cib.xml.deltasrc</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><span class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-il">pcs</span> -f tmp-cib.xml <span class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-il">resource</span> create tst-ovndb ocf:ovn:ovndb-servers manage_northd=yes master_ip=192.168.24.10 nb_master_port=6641 sb_master_port=6642 master</div><div style="font-size:12.8px"><span class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-il">pcs</span> -f tmp-cib.xml <span class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-il">resource</span> meta tst-ovndb-master notify=true</div><div style="font-size:12.8px"><span class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-il">pcs</span> -f tmp-cib.xml constraint colocation add master tst-ovndb-master with ip-192.168.24.10</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><span class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-il">pcs</span> cluster cib-push tmp-cib.xml diff-against=tmp-cib.xml.delta<wbr>src</div><div style="font-size:12.8px"><span class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-il">pcs</span> status</div></div><div style="font-size:12.8px">##############</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">In the above example, "<span style="font-size:12.8px">ip-192.168.24.10" is the IPaddr2 resource.</span></div><div style="font-size:12.8px"><span style="font-size:12.8px"><br></span></div><div style="font-size:12.8px"><span style="font-size:12.8px">Thanks</span></div><div style="font-size:12.8px"><span style="font-size:12.8px">Numan</span></div><span><div style="font-size:12.8px"><span style="font-size:12.8px"><br></span></div><div style="font-size:12.8px"><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-m_7916908338232400053HOEnZb"><div class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-m_7916908338232400053h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 28, 2017 at 10:48 PM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Tue, Nov 28, 2017 at 2:29 PM, Hui Xiang <span dir="ltr"><<a href="mailto:xianghuir@gmail.com" target="_blank">xianghuir@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Numan,<div><br></div><div><br></div><div>Finally figure it out what's wrong when running ovndb-servers ocf in my environment.</div><div><br></div><div>1. There is no default ovnnb and ovnsb running in my environment, I thought it should be started by pacemaker as the usual way other typical resource agent do it.</div><div>when I create the ovndb_servers resource, nothing happened, no operation is executed except monitor, which is really hard to debug for a while.</div><div>In the ovsdb_server_monitor() function, first it will check the status, here, it will be return NOT_RUNNING, then in the ovsdb_server_master_update<wbr>() function, "CRM_MASTER -D" is being executed, which appears stopped every following action, I am not very clear what work it did.<br></div><div><br></div><div>So, do the ovn_nb and ovn_sb needs to be running previouly before pacemaker ovndb_servers resource create? Is there any such documentation referred?</div></div></blockquote></span></div></div></div></blockquote></div></div></div></div></blockquote></span><div>No they don't need to be.</div><div><div class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-m_7916908338232400053HOEnZb"><div class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-m_7916908338232400053h5"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>2. Without your patch every nodes executing ovsdb_server_monitor and return OCF_SUCCESS </div><div><div>However, the first node of the three nodes cluster is executed ovsdb_server_stop action, the reason showed below:</div><div><27>Nov 28 15:35:11 node-1 pengine[1897010]: error: clone_color: ovndb_servers:0 is running on node-1.domain.tld which isn't allowed<br></div></div><div>Did I miss anything? I don't understand why it isn't allowed.</div><div><br></div><div>3. Regard your patch[1]</div><div>It first reports "/usr/lib/ocf/resource.d/ovn/o<wbr>vndb-servers: line 26: ocf_attribute_target: command not found ]" in my environment(pacemaker 1.1.12)</div></div></blockquote><div><br></div></span><div>Thanks. I will come back to you on your other points. The function "ocf_attribute_target" action must be added in 1.1.16-12.</div><div><br></div><div>I think it makes sense to either remove "ocf_attribute_target" or find a way so that even older versions work.</div><div><br></div><div>I will spin a v2.</div><div>Thanks</div><span class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-m_7916908338232400053m_7530973181775659281HOEnZb"><font color="#888888"><div>Numan</div></font></span><div><div class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-m_7916908338232400053m_7530973181775659281h5"><div><br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The log showed same as item2, but I have seen very shortly different state from "pcs status" as below shown:</div><div><div> Master/Slave Set: ovndb_servers-master [ovndb_servers]</div><div> Slaves: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]</div></div><div>There is no promote action being executed.</div><div><br></div><div><br></div><div>Thanks for looking and help.</div><div><br></div><div><span style="font-size:12.8px">[1] - </span><a href="https://patchwork.ozlabs.org/patch/839022/" style="font-size:12.8px" target="_blank">https://patchwork.ozlabs.org<wbr>/patch/839022/</a><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><span class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-m_7916908338232400053m_7530973181775659281m_6477589643099522897gmail-">On Fri, Nov 24, 2017 at 10:54 PM, Numan Siddique <span dir="ltr"><<a href="mailto:nusiddiq@redhat.com" target="_blank">nusiddiq@redhat.com</a>></span> wrote:<br></span><div><div class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-m_7916908338232400053m_7530973181775659281m_6477589643099522897gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Hui Xiang,<div><br></div><div>Can you please try with this patch [1] and see if it works for you ? Please let me know how it goes. But I am not sure, if the patch would fix the issue.</div><div><br></div><div>To brief, the OVN OCF script doesn't add monitor action for "Master" role. So pacemaker Resource agent would not check for the status of ovn db servers periodically. In case ovn db servers are killed, pacemaker wont know about it.</div><div><br></div><div><br></div><div><br></div><div><br></div><div>You can also take a look at this [1] to know how it is used in openstack with tripleo installation.</div><div><br></div><div>[1] - <a href="https://patchwork.ozlabs.org/patch/839022/" target="_blank">https://patchwork.ozlabs.org/p<wbr>atch/839022/</a></div><div>[2] - <a href="https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/pacemaker/ovn_northd.pp" target="_blank">https://github.com/openstack<wbr>/puppet-tripleo/blob/master/ma<wbr>nifests/profile/pacemaker/ovn_<wbr>northd.pp</a></div><div><br></div><div><br></div><div>Thanks</div><div>Numan</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-m_7916908338232400053m_7530973181775659281m_6477589643099522897gmail-m_-6827037725983782493h5">On Fri, Nov 24, 2017 at 3:00 PM, Hui Xiang <span dir="ltr"><<a href="mailto:xianghuir@gmail.com" target="_blank">xianghuir@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-2119459686872868904gmail-m_-3745470431214182794m_-5823501525314324389m_-5040426341463572524gmail-m_7916908338232400053m_7530973181775659281m_6477589643099522897gmail-m_-6827037725983782493h5"><div dir="ltr">Hi folks,<div><br></div><div> I am following what suggested on doc[1] to configure the ovndb_servers HA, however, it's so unluck with upgrading pacemaker packages from 1.12 to 1.16, do almost every kind of changes, there still not a ovndb_servers master promoted, is there any special recipe for it to run? so frustrated on it, sigh.</div><div><br></div><div>It always showed:</div><div><div> Master/Slave Set: ovndb_servers-master [ovndb_servers]</div><div> Stopped: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]</div></div><div><br></div><div>Even if I tried below steps:</div><div>1. pcs resource debug-stop ovndb_server on every nodes. ovn-ctl status_ovnxb: running/backup </div><div>2. pcs resource debug-start ovndb_server on every nodes. ovn-ctl status_ovnxb: running/backup <br></div><div>3. pcs resource debug-promote ovndb_server on one nodes. ovn-ctl status_ovnxb: running/active<br></div><div><br></div><div>With above status, the pcs status still showed as:</div><div><div> Master/Slave Set: ovndb_servers-master [ovndb_servers]</div><div> Stopped: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]</div></div><div><br></div><div><br></div><div>[1]. <a href="https://github.com/openvswitch/ovs/blob/master/Documentation/topics/integration.rst" target="_blank">https://github.com/openvs<wbr>witch/ovs/blob/master/Document<wbr>ation/topics/integration.rst</a></div><div><br></div><div>Appreciated any hint.</div><div><br></div><div><br></div></div>
<br></div></div>______________________________<wbr>_________________<br>
discuss mailing list<br>
<a href="mailto:discuss@openvswitch.org" target="_blank">discuss@openvswitch.org</a><br>
<a href="https://mail.openvswitch.org/mailman/listinfo/ovs-discuss" rel="noreferrer" target="_blank">https://mail.openvswitch.org/m<wbr>ailman/listinfo/ovs-discuss</a><br>
<br></blockquote></div><br></div>
</blockquote></div></div></div><br></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div></div></div></div>
</blockquote></div><br></div>