[ovs-discuss] HA OVN "Central" as a kubernetes service

aginwala aginwala at asu.edu
Mon Jul 6 20:10:33 UTC 2020


On Mon, Jul 6, 2020 at 4:33 AM Brendan Doyle <brendan.doyle at oracle.com>
wrote:

> Hi,
>
> So I'm really confused by what you have pointed me to here. As stated I do
> NOT
> want to use OVN as a CNI. I have a k8s cluster that use flannel as the
> CNI. I simply
> want to create an OVN "central" cluster as a Stateful set in my *existing*
> K8
> config.
>
> This repo:
>
> https://github.com/ovn-org/ovn-kubernetes/commit/a07b1a01af7e37b15c2e5f179ffad2b9f25a083d
>
> Seems to be for setting up a K8s cluster to use OVN as the CNI??
> Have you tried this?
> What IP do the ovn-controllers use to reach the OVN "central cluster?
> It seems to use an OVN docker image from docker.io, I want to use my own
> OVN src
> Do I use/modify the dist/images/Dockerfile in this repo? that has loads of
> references to CNI
> like I said I don't want to use OVN as the CNI??
>
> A pre-req for running ovn central as a k8s app is containerize ovn central
components. Hence, you need to start your own containers using docker.
Either you follow the approach from ovn-k8s repo as to how to build ovn
images or refer to the docker instructions in ovn repo. Since this app (ovn
central) will run behind a k8s service, ovn-controller should point to the
service ip of ovn central k8s app. k8s folks can comment on how to build
image that is in k8s pod specs e.g
http://docker.io/ovnkube/ovn-daemonset:latest

>
> The instructions here
> https://github.com/ovn-org/ovn/blob/d6b56b1629d5984ef91864510f918e232efb89de/Documentation/intro/install/general.rst
> seem more promising, if not a little confusing:
>
> IN the section "Starting OVN Central services in containers"
>
> Export following variables in .env and place it under project root:
>
> $ OVN_BRANCH=<BRANCH>
> $ OVN_VERSION=<VERSION>
> $ DISTRO=<LINUX_DISTRO>
> $ KERNEL_VERSION=<LINUX_KERNEL_VERSION>
> $ GITHUB_SRC=<GITHUB_URL>
> $ DOCKER_REPO=<REPO_TO_PUSH_IMAGE>
>
>
> Does it mean create a file called ".env" and place it in the toplevel dir
> of the cloned ovn repo?
> Or does it mean just add these to you shell environment (i.e put them in
> .bashrc)?
>
> You can just export OVN_BRANCH=xx in your shell for all variables and
build your containers with desired distro/version using make build
>
> Then we have:
>
> 1)
>
> Start OVN containers using below command:
>
> $ docker run -itd --net=host --name=ovn-nb \
>   <docker_repo>:<tag> ovn-nb-tcp
>
> $ docker run -itd --net=host --name=ovn-sb \
>   <docker_repo>:<tag> ovn-sb-tcp
>
> $ docker run -itd --net=host --name=ovn-northd \
>   <docker_repo>:<tag> ovn-northd-tcp
>
> followed by
>
> 2)
>
> $ docker run -e "host_ip=<host_ip>" -e "nb_db_port=<port>" -itd \
>   --name=ovn-nb-raft --net=host --privileged <docker_repo>:<tag> \
>   ovn-nb-cluster-create
>
> $ docker run -e "host_ip=<host_ip>" -e "sb_db_port=<port>" -itd \
>   --name=ovn-sb-raft --net=host --privileged <docker_repo>:<tag> \
>   ovn-sb-cluster-create
>
> $ docker run -e "OVN_NB_DB=tcp:<node1>:6641,tcp:<node2>:6641,\
>   tcp:<node3>:6641" -e "OVN_SB_DB=tcp:<node1>:6642,tcp:<node2>:6642,\
>   tcp:<node3>:6642" -itd --name=ovn-northd-raft <docker_repo>:<tag> \
>   ovn-northd-cluster
>
> Does it mean do 1), then 2) or does it mean do 1) for non HA OVN central
> *OR* 2)
> for HA/clustered OVN Central?
>
> Doc says Start OVN containers in cluster mode using below command on
node2 and node3 to make them join the peer using below command:. Hence, you
can even play with just docker on 3 nodes where you run step1 on node1 that
creates cluster and do the join-cluster on rest two nodes to give you a
clear idea before moving to pod in k8s. Not sure if you need more details
to update doc. We can always improvise. Upstream ovn-k8s does the same for
pods where e.g. ovn-kube0 pod creates a cluster and rest two pods joins

> It's not clear
>
> Thanks
>
>
>
>
>
>
> On 25/06/2020 17:36, aginwala wrote:
>
> Hi:
>
> There are a couple of options as I have been exploring this too:
>
> 1. Upstream ovn-k8s patches (
> https://github.com/ovn-org/ovn-kubernetes/commit/a07b1a01af7e37b15c2e5f179ffad2b9f25a083d)
> uses statefulset and headless service for starting ovn central raft cluster
> with 3 replicas. Cluster startup code and pod specs are pretty neat that
> addresses most of the doubts.
>
> OVN components have been containerized too to start them in pods. You can
> also refer to
> https://github.com/ovn-org/ovn/blob/d6b56b1629d5984ef91864510f918e232efb89de/Documentation/intro/install/general.rst
>  for the same and use them to make it work in pod specs too.
>
>
> 2. Write a new ovn operator similar to etcd operator
> https://github.com/coreos/etcd-operator which just takes the count of
> raft replicas and does the job in the background.
>
> I also added ovn-k8s group so they can comment on any other ideas too.
> Hope it helps.
>
>
>
> On Thu, Jun 25, 2020 at 7:15 AM Brendan Doyle <brendan.doyle at oracle.com>
> wrote:
>
>> Hi,
>>
>> So I'm trying to find information on setting up an OVN "Central/Master"
>> high availability (HA)
>> Not as Active-Backup with Pacemaker, but as a cluster. But I want to
>> deploy and manage that
>> cluster as a Kubernetes service .
>>
>> I can find lots of stuff on "ovn-kube" but this seems to be using OVN as
>> a  kubernetes CNI instead of
>> Flannel etc.  But this is not what I want to do, I have a kubernetes
>> cluster using Flannel as the CNI,
>> now  I want to deploy a HA OVN "Central" as a kubernetes service. Kind
>> of like how you can deploy
>> a MySQL cluster in kubernetes using a SatefulSet deployment.
>>
>> I have found this:
>>   https://github.com/ovn-org/ovn-kubernetes#readme
>>
>> But it is not clear to me if this is how to setup OVN as a kubernetes
>> CNI or it's how to setup a HA
>> OVN central as kubernetes service.
>>
>> Can anybody comment, has anyone done this?
>>
>>
>> I guess I could run an OVN central as standalone and use a kubernetes
>> deployment with 3
>>   replica sets and "export" as a NodePort service. And have a
>> floating/VIP on my kubernetes
>> nodes. And direct ovn-controllers to the VIP. So only the pod that holds
>> the VIP would service
>> requests. This would work and give HA, but you don't get the performance
>> of an OVN
>> clustered Database Model, where each OVN central could service requests.
>>
>>
>>
>>
>> Thanks
>>
>>
>> Rdgs
>> Brendan
>>
>> _______________________________________________
>> discuss mailing list
>> discuss at openvswitch.org
>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200706/70515fe1/attachment.html>


More information about the discuss mailing list