[ovs-dev] Update on DDlog port of ovn-northd

[Leonid Ryzhyk]

Dear OVN developers,

This is a brief update on the state of the DDlog port of ovn-northd.

We completed the initial implementation of ovn-northd in DDlog few months
ago.  Justin kindly
helped to integrate it with OVN, so that it can be used as a drop-in
replacement for the C
version (and passes all the tests in the OVN test suite).  The DDlog
implementation does
not have any of the new features/improvements added in April 2019 or later.

## Repository

The code is in the `ddlog-dev` branch of the `ovn-org/ovn` repository:
https://github.com/ovn-org/ovn/tree/ddlog-dev

## Documentation

Building and using ovn-northd-ddlog:
https://github.com/ovn-org/ovn/blob/ddlog-dev/ovn/northd/docs/design.md

Debugging ovn-northd-ddlog:
https://github.com/ovn-org/ovn/blob/ddlog-dev/ovn/northd/docs/debugging.md

## Preliminary performance results

Han Zhou kindly tested ovn-northd-ddlog with his OVN scale test and even
found a nasty
performance bug in the process (thanks, Han!).  He reports that DDlog
speeds up the test
by almost a factor of 10:

- ddlog version: 7:39min
- C version: 67:47min

This is great, and in fact profiling shows that there is still plenty of
space for
improvement.  He also reports a 10+ times increase in memory footprint:

- ddlog version: 1944696KB
- C version: 147984KB

Again, we are working on a number of optimizations, which should reduce
this overhead; although it will never be as low as C, since DDlog
fundamentally
needs to cache more state to enable fast incremental computation.

Han also used DDlog's record&replay feature to capture all northd
transactions
performed by the scale test in a format that can be replayed against the
standalone DDlog executable without having to reproduce Han's OpenStack
setup.
The replay file is here: http://ryzhyk.net/replay.tgz

Instructions for replaying this script:
https://github.com/ovn-org/ovn/blob/ddlog-dev/ovn/northd/docs/debugging.md#record-and-replay-ddlog-execution
The script will run for a few
minutes and finally print some profiling information, including the
breakdown of
DDlog's CPU and memory usage.

## Next steps

We seek help from the OVN community in maintaining ovn-northd-ddlog.  The
first
step is to start porting new OVN features introduced in the last few months
to
DDlog.

Leonid