[ovs-discuss] etcd for OVN status update (was: Re: more about etcd (can it support big transactions and many monitors?))

Ben Pfaff blp at ovn.org
Wed Jun 15 17:02:02 UTC 2016


On Fri, Jun 10, 2016 at 03:59:34PM -0700, Ben Pfaff wrote:
> After talking to Justin, I think I'm going to take a few days (maybe
> Wednesday through Friday next week) to hack on etcd related stuff, with
> the goal being to come up with a detailed to-do list and try to verify
> that the stuff that I think should work really will.  (For example, Andy
> thinks that the gRPC implementation for C is incomplete, but neither of
> us has really tried anything.)
> 
> I guess the first step for me personally is an x86-64 container or
> chroot or something.

Here are the questions I started out with:

1. Can I get etcd v3 working on an i386 laptop somehow, given etcd's
   hostility to non-x86-64 architectures?

2. Can I get a C program to talk to etcd v3, given:

   - etcd is layered on grpc, which does not have a full set of bindings
     for C but instead infrastructure bindings on which the other
     language bindings are layered.

   - etcd uses protobufs (as is typical) inside grpc, but protobufs do
     not have official C bindings either.

3. How should the OVN databases be arranged within etcd?  There are
   multiple possibilities:

   - Define OVSDB bindings to etcd and implement those bindings in the
     OVSDB client libraries (C and Python).

   - Define OVSDB bindings to etcd and implement those bindings in the
     OVSDB server (so that ovsdb-server uses etcd as a storage layer).

   - Define a native etcd schema for OVN SB (and probably NB) database
     and make ovn-controller and ovn-northd use it natively.

Here's what I've accomplished so far:

* Managed to get etcd v3 beta running in a docker container on my
  laptop.  This solves the problem I personally have due to running i386
  userspace on my laptop, because Docker containers run x86-64 userspace
  internally regardless of whether the host uses i386 or x86-64
  userspace.

  (This does not answer the larger question of how archs other than
  x86-64 will use OVN.)

  I suspect this is not the same reason that the rest of the world is so
  excited about containers, but it works for me.

* Managed to interact with etcd in the x86-64 container from the etcdctl
  v3 command-line client running on my i386 host.  This does not seem to
  have the same arch problem as running the etcd server, or at least the
  client does useful work instead of immediately aborting with an error
  due to an unsupported arch.

* Managed to build the grpc C libraries.

* Managed to use "nanopb" (https://github.com/nanopb/nanopb) to generate
  C bindings for the Protocol Buffer definitions for the etcd v3
  protocol.  This took a little bit of coaxing.  I have not yet tried to
  compile or use what it generated.  (grpc itself uses nanopb
  internally, which gives me hope that nanopb works OK.)

Here's what is left:

* Write a C test program that tries to tie together the grpc C library
  and the nanopb-generated etcd bindings into something that can
  actually talk to etcd, probably something comparable to a simplified
  etcdctl in C.

  This has multiple steps, at least the following:

  - Figure out what's needed to tie together all the grpc C
    infrastructure code into a C library that can actually implement a
    grpc service.  I don't know whether this is on the spectrum from
    trivial to troublesome.

  - Layer a library on top of the previous one from the previous bullet
    that can send and receive etcd messages.

  - Layer a test program on top of the previous library.

  - Get grpc and nanopb-generated code and everything else to actually
    link together.  This is always more of a challenge than it ought to
    be.

* Figure out OVN database handling (question 3 above).

So far, I'm feeling fairly optimistic.

Thanks,

Ben.



More information about the discuss mailing list