[ovs-dev] Update on DDlog port of ovn-northd

Mark Michelson mmichels at redhat.com
Thu Jul 18 20:51:05 UTC 2019


I've now performed tests where I measured the wall clock time of each 
iteration through the test. I've created a plot of the C vs. the DDlog 
ovn-northd here: https://i.imgur.com/bIsbeMN.png

This confirms for me that the DDlog implementation does a better job of 
staying consistent in its execution time even as the network grows 
larger. You can see that the C implementation gets noticeably slower as 
the network grows. The DDlog implementation also gets slower but the 
slope of the trend is much less severe compared to C.

Through my initial tests, I'm happy with what DDlog is doing. It appears 
to be handling large-scale networks much better than the C 
implementation. The incremental processing allows for the network size 
not to affect the performance of ovn-northd as much.

On 7/18/19 1:18 PM, Mark Michelson wrote:
> Hi, just wanted to send a follow-up and a test result.
> 
> I got everything up and running (yay!). So I set up a server and ran a 
> sandbox test using ovn with traditional C-based ovn-northd and then ran 
> the same test using ovn with DDlog-based ovn-northd.
> 
> The test script can be found here: 
> https://gist.github.com/putnopvut/3df7156b6d44b81d75a598262c5e959b
> 
> The test sets up a network with 159 logical switches all connected to a 
> single logical router. Each logical switch has 92 logical switch ports. 
> For every 2 logical switch ports we add, we create a new port group, and 
> ACLs are created for each port group. The important part of this test is 
> that the final call to ovn-nbctl in each iteration of the loop uses 
> --wait=sb. In other words, each new logical switch port we add has to 
> wait for ovn-northd to process it completely before we can start adding 
> the next logical switch port.
> 
> With traditional C-based ovn-northd:
> [root at wsfd-netdev67 tutorial]# time ./scale-daemon-new-pg.sh
> 
> real    128m59.280s
> user    1m27.601s
> sys    2m0.986s
> 
> With DDlog-based ovn-northd:
> [root at wsfd-netdev67 tutorial]# time ./scale-daemon-new-pg.sh
> 
> real    33m40.586s
> user    1m26.976s
> sys    1m59.784s
> 
> As you can see, the test takes about a quarter of the time to complete 
> when running with DDlog. So far, it seems like a good improvement.
> 
> As a point of reference, if I remove the --wait=sb from the test, then 
> the result is:
> 
> [root at wsfd-netdev67 tutorial]# time ./scale-daemon-new-pg.sh
> 
> real    3m59.696s
> user    1m18.937s
> sys    1m44.669s
> 
> This test was run with DDlog-based ovn-northd, but the result is nearly 
> identical with C-based ovn-northd since the test no longer is bound by 
> ovn-northd's performance.
> 
> I'm a bit surprised that with DDlog, adding --wait=sb to each iteration 
> causes such a large difference in the overall test time (~4 minutes up 
> to ~34 minutes).
> 
> My next step is going to be to plot the time of each individual 
> iteration during the test. I'll share those results when I have them 
> ready. I don't want to speculate on anything until I have that data to 
> share.
> 
> Thanks,
> Mark Michelson
> 
> On 7/15/19 12:34 PM, Leonid Ryzhyk wrote:
>> Hi Mark,
>>
>> Thanks for the feedback. We will add the `--version` flag.
>>
>> Yes, `stack install` merely copies the two DDlog executable to 
>> `~/.local/bin/` (or a custom path specified using `--local-bin-path 
>> <custom_path>`).
>>
>> Also, just want to mention that we maintain up-to-date binary releases 
>> of DDlog at `https://github.com/vmware/differential-datalog/releases`, 
>> so most users should not need to install the Haskell tool stack and 
>> compile DDlog. When using binary releases, all you have to do is add 
>> the `ddlog/bin` directory to `$PATH`.
>>
>> Leonid
>>
>> On Mon, Jul 15, 2019 at 6:30 AM Mark Michelson <mmichels at redhat.com 
>> <mailto:mmichels at redhat.com>> wrote:
>>
>>     I did install DDlog originally back in early December last year. So
>>     there are probably remnants of that still present. However, I did
>>     perform a `stack install` using an updated pull of master on 
>> Friday. So
>>     I guess the `stack install` didn't get rid of the old installation?
>>
>>     Also, there's no `ddlog --version` or anything similar to see what
>>     version of DDlog is installed. That could be a nice feature to 
>> have in
>>     the near future.
>>
>>     On 7/12/19 12:40 PM, Leonid Ryzhyk wrote:
>>      > Thanks for trying it out!  Sounds like you have a very old
>>     version of
>>      > ddlog. If you install from source, please make sure that you run
>>     `stack
>>      > install` and that there is no other version off ddlog in your
>>     path other
>>      > than the one created by `stack install`.
>>      >
>>      > Leonid
>>      >
>>      > On Fri, Jul 12, 2019, 9:05 AM Mark Michelson <mmichels at redhat.com
>>     <mailto:mmichels at redhat.com>
>>      > <mailto:mmichels at redhat.com <mailto:mmichels at redhat.com>>> wrote:
>>      >
>>      >     On 7/12/19 3:29 AM, Leonid Ryzhyk wrote:
>>      >      > Dear OVN developers,
>>      >      >
>>      >      > This is a brief update on the state of the DDlog port of
>>     ovn-northd.
>>      >      >
>>      >      > We completed the initial implementation of ovn-northd in 
>> DDlog
>>      >     few months
>>      >      > ago.  Justin kindly
>>      >      > helped to integrate it with OVN, so that it can be used as
>>     a drop-in
>>      >      > replacement for the C
>>      >      > version (and passes all the tests in the OVN test 
>> suite).     The DDlog
>>      >      > implementation does
>>      >      > not have any of the new features/improvements added in
>>     April 2019
>>      >     or later.
>>      >      >
>>      >      > ## Repository
>>      >      >
>>      >      > The code is in the `ddlog-dev` branch of the `ovn-org/ovn`
>>      >     repository:
>>      >      > https://github.com/ovn-org/ovn/tree/ddlog-dev
>>      >
>>      >     Hi Leonid,
>>      >
>>      >     I ran into an issue when attempting to build ovn-northd. I
>>     successfully
>>      >     installed DDLog, but then I encountered this issue when
>>     building OVN:
>>      >
>>      >     ddlog -i ovn/northd/ovn_northd.dl -L
>>      >     /home/putnopvut/differential-datalog/lib
>>      >     ddlog: Failed to parse input file: "ovn/northd/ovn_northd.dl"
>>     (line 94,
>>      >     column 5):
>>      >     unexpected "&"
>>      >     expecting "not", variable name, relation name, "var",
>>     expression or "."
>>      >
>>      >     The line in question looks like this:
>>      >
>>      >           &SwitchPort(.lsp = lsp, .sw = &sw),
>>      >
>>      >     Any idea what's gone wrong here?
>>      >
>>      >     Thanks,
>>      >
>>      >     Mark Michelson
>>      >
>>      >
>>      >
>>      >      >
>>      >      > ## Documentation
>>      >      >
>>      >      > Building and using ovn-northd-ddlog:
>>      >      >
>>      >
>>     
>> https://github.com/ovn-org/ovn/blob/ddlog-dev/ovn/northd/docs/design.md
>>      >      >
>>      >      > Debugging ovn-northd-ddlog:
>>      >      >
>>      >
>>     
>> https://github.com/ovn-org/ovn/blob/ddlog-dev/ovn/northd/docs/debugging.md 
>>
>>      >      >
>>      >      > ## Preliminary performance results
>>      >      >
>>      >      > Han Zhou kindly tested ovn-northd-ddlog with his OVN scale
>>     test
>>      >     and even
>>      >      > found a nasty
>>      >      > performance bug in the process (thanks, Han!).  He reports
>>     that DDlog
>>      >      > speeds up the test
>>      >      > by almost a factor of 10:
>>      >      >
>>      >      > - ddlog version: 7:39min
>>      >      > - C version: 67:47min
>>      >      >
>>      >      > This is great, and in fact profiling shows that there is 
>> still
>>      >     plenty of
>>      >      > space for
>>      >      > improvement.  He also reports a 10+ times increase in 
>> memory
>>      >     footprint:
>>      >      >
>>      >      > - ddlog version: 1944696KB
>>      >      > - C version: 147984KB
>>      >      >
>>      >      > Again, we are working on a number of optimizations, which
>>     should
>>      >     reduce
>>      >      > this overhead; although it will never be as low as C,
>>     since DDlog
>>      >      > fundamentally
>>      >      > needs to cache more state to enable fast incremental
>>     computation.
>>      >      >
>>      >      > Han also used DDlog's record&replay feature to capture all
>>     northd
>>      >      > transactions
>>      >      > performed by the scale test in a format that can be 
>> replayed
>>      >     against the
>>      >      > standalone DDlog executable without having to reproduce 
>> Han's
>>      >     OpenStack
>>      >      > setup.
>>      >      > The replay file is here: http://ryzhyk.net/replay.tgz
>>      >      >
>>      >      > Instructions for replaying this script:
>>      >      >
>>      >
>>     
>> https://github.com/ovn-org/ovn/blob/ddlog-dev/ovn/northd/docs/debugging.md#record-and-replay-ddlog-execution 
>>
>>      >      > The script will run for a few
>>      >      > minutes and finally print some profiling information,
>>     including the
>>      >      > breakdown of
>>      >      > DDlog's CPU and memory usage.
>>      >      >
>>      >      > ## Next steps
>>      >      >
>>      >      > We seek help from the OVN community in maintaining
>>      >     ovn-northd-ddlog.  The
>>      >      > first
>>      >      > step is to start porting new OVN features introduced in
>>     the last
>>      >     few months
>>      >      > to
>>      >      > DDlog.
>>      >      >
>>      >      > Leonid
>>      >
>>      >     Hi Leonid,
>>      >
>>      >     I attempted
>>      >
>>      >      > _______________________________________________
>>      >      > dev mailing list
>>      >      > dev at openvswitch.org <mailto:dev at openvswitch.org>
>>     <mailto:dev at openvswitch.org <mailto:dev at openvswitch.org>>
>>      >      > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>>      >      >
>>      >
>>
> 



More information about the dev mailing list