[ovs-dev] Update on DDlog port of ovn-northd

Mark Michelson mmichels at redhat.com
Thu Jul 18 17:18:12 UTC 2019


Hi, just wanted to send a follow-up and a test result.

I got everything up and running (yay!). So I set up a server and ran a 
sandbox test using ovn with traditional C-based ovn-northd and then ran 
the same test using ovn with DDlog-based ovn-northd.

The test script can be found here: 
https://gist.github.com/putnopvut/3df7156b6d44b81d75a598262c5e959b

The test sets up a network with 159 logical switches all connected to a 
single logical router. Each logical switch has 92 logical switch ports. 
For every 2 logical switch ports we add, we create a new port group, and 
ACLs are created for each port group. The important part of this test is 
that the final call to ovn-nbctl in each iteration of the loop uses 
--wait=sb. In other words, each new logical switch port we add has to 
wait for ovn-northd to process it completely before we can start adding 
the next logical switch port.

With traditional C-based ovn-northd:
[root at wsfd-netdev67 tutorial]# time ./scale-daemon-new-pg.sh

real	128m59.280s
user	1m27.601s
sys	2m0.986s

With DDlog-based ovn-northd:
[root at wsfd-netdev67 tutorial]# time ./scale-daemon-new-pg.sh

real	33m40.586s
user	1m26.976s
sys	1m59.784s

As you can see, the test takes about a quarter of the time to complete 
when running with DDlog. So far, it seems like a good improvement.

As a point of reference, if I remove the --wait=sb from the test, then 
the result is:

[root at wsfd-netdev67 tutorial]# time ./scale-daemon-new-pg.sh

real	3m59.696s
user	1m18.937s
sys	1m44.669s

This test was run with DDlog-based ovn-northd, but the result is nearly 
identical with C-based ovn-northd since the test no longer is bound by 
ovn-northd's performance.

I'm a bit surprised that with DDlog, adding --wait=sb to each iteration 
causes such a large difference in the overall test time (~4 minutes up 
to ~34 minutes).

My next step is going to be to plot the time of each individual 
iteration during the test. I'll share those results when I have them 
ready. I don't want to speculate on anything until I have that data to 
share.

Thanks,
Mark Michelson

On 7/15/19 12:34 PM, Leonid Ryzhyk wrote:
> Hi Mark,
> 
> Thanks for the feedback. We will add the `--version` flag.
> 
> Yes, `stack install` merely copies the two DDlog executable to 
> `~/.local/bin/` (or a custom path specified using `--local-bin-path 
> <custom_path>`).
> 
> Also, just want to mention that we maintain up-to-date binary releases 
> of DDlog at `https://github.com/vmware/differential-datalog/releases`, 
> so most users should not need to install the Haskell tool stack and 
> compile DDlog. When using binary releases, all you have to do is add the 
> `ddlog/bin` directory to `$PATH`.
> 
> Leonid
> 
> On Mon, Jul 15, 2019 at 6:30 AM Mark Michelson <mmichels at redhat.com 
> <mailto:mmichels at redhat.com>> wrote:
> 
>     I did install DDlog originally back in early December last year. So
>     there are probably remnants of that still present. However, I did
>     perform a `stack install` using an updated pull of master on Friday. So
>     I guess the `stack install` didn't get rid of the old installation?
> 
>     Also, there's no `ddlog --version` or anything similar to see what
>     version of DDlog is installed. That could be a nice feature to have in
>     the near future.
> 
>     On 7/12/19 12:40 PM, Leonid Ryzhyk wrote:
>      > Thanks for trying it out!  Sounds like you have a very old
>     version of
>      > ddlog. If you install from source, please make sure that you run
>     `stack
>      > install` and that there is no other version off ddlog in your
>     path other
>      > than the one created by `stack install`.
>      >
>      > Leonid
>      >
>      > On Fri, Jul 12, 2019, 9:05 AM Mark Michelson <mmichels at redhat.com
>     <mailto:mmichels at redhat.com>
>      > <mailto:mmichels at redhat.com <mailto:mmichels at redhat.com>>> wrote:
>      >
>      >     On 7/12/19 3:29 AM, Leonid Ryzhyk wrote:
>      >      > Dear OVN developers,
>      >      >
>      >      > This is a brief update on the state of the DDlog port of
>     ovn-northd.
>      >      >
>      >      > We completed the initial implementation of ovn-northd in DDlog
>      >     few months
>      >      > ago.  Justin kindly
>      >      > helped to integrate it with OVN, so that it can be used as
>     a drop-in
>      >      > replacement for the C
>      >      > version (and passes all the tests in the OVN test suite). 
>     The DDlog
>      >      > implementation does
>      >      > not have any of the new features/improvements added in
>     April 2019
>      >     or later.
>      >      >
>      >      > ## Repository
>      >      >
>      >      > The code is in the `ddlog-dev` branch of the `ovn-org/ovn`
>      >     repository:
>      >      > https://github.com/ovn-org/ovn/tree/ddlog-dev
>      >
>      >     Hi Leonid,
>      >
>      >     I ran into an issue when attempting to build ovn-northd. I
>     successfully
>      >     installed DDLog, but then I encountered this issue when
>     building OVN:
>      >
>      >     ddlog -i ovn/northd/ovn_northd.dl -L
>      >     /home/putnopvut/differential-datalog/lib
>      >     ddlog: Failed to parse input file: "ovn/northd/ovn_northd.dl"
>     (line 94,
>      >     column 5):
>      >     unexpected "&"
>      >     expecting "not", variable name, relation name, "var",
>     expression or "."
>      >
>      >     The line in question looks like this:
>      >
>      >           &SwitchPort(.lsp = lsp, .sw = &sw),
>      >
>      >     Any idea what's gone wrong here?
>      >
>      >     Thanks,
>      >
>      >     Mark Michelson
>      >
>      >
>      >
>      >      >
>      >      > ## Documentation
>      >      >
>      >      > Building and using ovn-northd-ddlog:
>      >      >
>      >
>     https://github.com/ovn-org/ovn/blob/ddlog-dev/ovn/northd/docs/design.md
>      >      >
>      >      > Debugging ovn-northd-ddlog:
>      >      >
>      >
>     https://github.com/ovn-org/ovn/blob/ddlog-dev/ovn/northd/docs/debugging.md
>      >      >
>      >      > ## Preliminary performance results
>      >      >
>      >      > Han Zhou kindly tested ovn-northd-ddlog with his OVN scale
>     test
>      >     and even
>      >      > found a nasty
>      >      > performance bug in the process (thanks, Han!).  He reports
>     that DDlog
>      >      > speeds up the test
>      >      > by almost a factor of 10:
>      >      >
>      >      > - ddlog version: 7:39min
>      >      > - C version: 67:47min
>      >      >
>      >      > This is great, and in fact profiling shows that there is still
>      >     plenty of
>      >      > space for
>      >      > improvement.  He also reports a 10+ times increase in memory
>      >     footprint:
>      >      >
>      >      > - ddlog version: 1944696KB
>      >      > - C version: 147984KB
>      >      >
>      >      > Again, we are working on a number of optimizations, which
>     should
>      >     reduce
>      >      > this overhead; although it will never be as low as C,
>     since DDlog
>      >      > fundamentally
>      >      > needs to cache more state to enable fast incremental
>     computation.
>      >      >
>      >      > Han also used DDlog's record&replay feature to capture all
>     northd
>      >      > transactions
>      >      > performed by the scale test in a format that can be replayed
>      >     against the
>      >      > standalone DDlog executable without having to reproduce Han's
>      >     OpenStack
>      >      > setup.
>      >      > The replay file is here: http://ryzhyk.net/replay.tgz
>      >      >
>      >      > Instructions for replaying this script:
>      >      >
>      >
>     https://github.com/ovn-org/ovn/blob/ddlog-dev/ovn/northd/docs/debugging.md#record-and-replay-ddlog-execution
>      >      > The script will run for a few
>      >      > minutes and finally print some profiling information,
>     including the
>      >      > breakdown of
>      >      > DDlog's CPU and memory usage.
>      >      >
>      >      > ## Next steps
>      >      >
>      >      > We seek help from the OVN community in maintaining
>      >     ovn-northd-ddlog.  The
>      >      > first
>      >      > step is to start porting new OVN features introduced in
>     the last
>      >     few months
>      >      > to
>      >      > DDlog.
>      >      >
>      >      > Leonid
>      >
>      >     Hi Leonid,
>      >
>      >     I attempted
>      >
>      >      > _______________________________________________
>      >      > dev mailing list
>      >      > dev at openvswitch.org <mailto:dev at openvswitch.org>
>     <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>>
>      >      > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>      >      >
>      >
> 



More information about the dev mailing list