[ovs-discuss] ovn ovsdb clustering question

[Satish Patel]

Hi Numan,

Finally i have created small blog on OVSDB clustering so it help out
new folks - https://satishdotpatel.github.io/openstack-ansible-ovn-clustering/

On Mon, May 24, 2021 at 5:58 PM Satish Patel <satish.txt at gmail.com> wrote:
>
> Hi Numan,
>
> Sorry for delay reply but I figured it out how to setup cluster basically I have to remove everything from /var/lib/ovn directory on each new nodes before I join them to cluster.
>
> Now I have other issue where I messed up my cluster using leave command and don’t know how to rejoin it. So basically I destroy cluster and rebuild it but in that process i lost my ovsdb database and now I don’t know how to restore previous data? I found some backup files inside /var/lib/ovn/ directories and I did restore using ovsdb-client utility but they didn’t help.
>
> I love ovn design but missing documents specially related cluster operation and whenever I google something all I found people asking question but not good answer. It would be great to have nice cluster operation and management doc. (I’m planning to write blog once I get everything working in lab)
>
> Sent from my iPhone
>
> > On May 21, 2021, at 4:35 PM, Numan Siddique <numans at ovn.org> wrote:
> >
> > On Fri, May 21, 2021 at 9:53 AM Satish Patel <satish.txt at gmail.com> wrote:
> >>
> >> Folks,
> >>
> >> I have 3 controller nodes and am trying to setup clustering for OVN
> >> deployment, but lack enough documentation and have some issues and
> >> questions regarding it.
> >>
> >> I found this document but again its little confusing and very old -
> >> https://mail.openvswitch.org/pipermail/ovs-discuss/2018-March/046470.html
> >>
> >> # controller 1
> >>
> >> /usr/share/ovn/scripts/ovn-ctl --db-nb-addr=172.30.40.93 \
> >> --db-nb-create-insecure-remote=yes \
> >> --db-sb-addr=172.30.40.93 \
> >> --db-sb-create-insecure-remote=yes \
> >> --db-nb-cluster-local-addr=172.30.40.93 \
> >> --db-sb-cluster-local-addr=172.30.40.93 \
> >> --ovn-northd-nb-db=tcp:172.30.40.93:6641,tcp:172.30.40.25:6641,tcp:172.30.40.177:6641
> >> \
> >> --ovn-northd-sb-db=tcp:172.30.40.93:6642,tcp:172.30.40.25:6642,tcp:172.30.40.177:6642
> >> \
> >> start_northd
> >>
> >>
> >> # controller 2
> >>
> >> /usr/share/ovn/scripts/ovn-ctl --db-nb-addr=172.30.40.25 \
> >> --db-nb-create-insecure-remote=yes \
> >> --db-sb-addr=172.30.40.25 \
> >> --db-sb-create-insecure-remote=yes \
> >> --db-nb-cluster-local-addr=172.30.40.25 \
> >> --db-sb-cluster-local-addr=172.30.40.25 \
> >> --db-nb-cluster-remote-addr=172.30.40.93 \
> >> --db-sb-cluster-remote-addr=172.30.40.93 \
> >> --ovn-northd-nb-db=tcp:172.30.40.93:6641,tcp:172.30.40.25:6641,tcp:172.30.40.177:6641
> >> \
> >> --ovn-northd-sb-db=tcp:172.30.40.93:6642,tcp:172.30.40.25:6642,tcp:172.30.40.177:6642
> >> \
> >> start_northd
> >>
> >>
> >> # controller 3
> >>
> >> /usr/share/ovn/scripts/ovn-ctl --db-nb-addr=172.30.40.177 \
> >> --db-nb-create-insecure-remote=yes \
> >> --db-nb-cluster-local-addr=172.30.40.177 \
> >> --db-sb-addr=172.30.40.177 \
> >> --db-sb-create-insecure-remote=yes \
> >> --db-sb-cluster-local-addr=172.30.40.177 \
> >> --db-nb-cluster-remote-addr=172.30.40.93 \
> >> --db-sb-cluster-remote-addr=172.30.40.93 \
> >> --ovn-northd-nb-db=tcp:172.30.40.93:6641,tcp:172.30.40.25:6641,tcp:172.30.40.177:6641
> >> \
> >> --ovn-northd-sb-db=tcp:172.30.40.93:6642,tcp:172.30.40.25:6642,tcp:172.30.40.177:6642
> >> \
> >> start_northd
> >>
> >> ## Validation steps
> >>
> >> controller-2# export\
> >> remote="tcp:172.30.40.93:6641,tcp:172.30.40.25:6641,tcp:172.30.40.177:6641"
> >>
> >> controller-2# ovn-nbctl --db=$remote show
> >> controller-2#
> >>
> >> In the above command i am seeing output only when it hit controller-1
> >> node, but for node-2 and note-3 giving me empty output that means data
> >> replication doesn't work. what is the command to verify
> >> synchronization working between all 3 nodes?
> >>
> >> Do I need to restart any other services?
> >
> > Hi Satish,
> >
> > Can you check out this script and try it -
> > https://github.com/ovn-org/ovn-fake-multinode/blob/master/ovn_cluster.sh#L353
> >
> > Probably there is no need for you to start northd on all three nodes.
> > If you want to start northd on 3 nodes,
> > then they should be configured to connect to the clustered databases.
> > Probably in your case, ovn-norths
> > are connecting to the local unix sockets of ovsdb servers.
> >
> > Thanks
> > Numan
> >
> >> _______________________________________________
> >> discuss mailing list
> >> discuss at openvswitch.org
> >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> >>