[ovs-dev] [PATCH 4/4] ovn-northd: Reduce amount of flow hashing.

[Ben Pfaff]

On Fri, Feb 16, 2018 at 05:10:29PM +0100, Jakub Sitnicki wrote:
> Ben,
> 
> On Wed, Feb 14, 2018 at 09:54 PM GMT, Ben Pfaff wrote:
> > Jakub Sitnicki demonstrated that repeatedly calculating row hashes is
> > expensive, so this should improve ovn-northd performance.
> >
> > Reported-by: Jakub Sitnicki <jkbs at redhat.com>
> > Reported-at: https://mail.openvswitch.org/pipermail/ovs-dev/2018-February/344404.html
> > Signed-off-by: Ben Pfaff <blp at ovn.org>
> > ---
> >  ovn/lib/ovn-sb-idl.ann  | 20 ++++++++++++++++++++
> >  ovn/lib/ovn-util.c      | 27 +++++++++++++++++++++++++++
> >  ovn/lib/ovn-util.h      |  7 +++++++
> >  ovn/northd/ovn-northd.c | 28 +++++++++++++++++-----------
> >  4 files changed, 71 insertions(+), 11 deletions(-)
> >
> 
> With this series applied the total of CPU cycles consumed by northd and
> the per-function profile changes dramatically when running a simple
> benchmark of creating 15 lswitches, 100 lports per each lswitch.
> 
> `perf stat` for ovn-northd looks like so:
> 
> before:
> 
>  Performance counter stats for process id '7091':
> 
>       85263.043641      task-clock (msec)         #    0.843 CPUs utilized
>              9,859      context-switches          #    0.116 K/sec
>                575      cpu-migrations            #    0.007 K/sec
>              2,243      page-faults               #    0.026 K/sec
>    225,663,620,029      cycles                    #    2.647 GHz
>    302,290,105,647      instructions              #    1.34  insn per cycle
>     53,556,381,940      branches                  #  628.131 M/sec
>        435,374,510      branch-misses             #    0.81% of all branches
> 
>      101.202109604 seconds time elapsed
> 
> after:
> 
>  Performance counter stats for process id '25306':
> 
>       50362.124282      task-clock (msec)         #    0.474 CPUs utilized
>              5,120      context-switches          #    0.102 K/sec
>              1,025      cpu-migrations            #    0.020 K/sec
>              9,546      page-faults               #    0.190 K/sec
>    134,756,308,237      cycles                    #    2.676 GHz
>    154,810,279,583      instructions              #    1.15  insn per cycle
>     27,350,003,179      branches                  #  543.067 M/sec
>        216,219,142      branch-misses             #    0.79% of all branches
> 
>      106.207558198 seconds time elapsed
> 
> CPU time spent in ovn_lflow_find becomes insignificant (<0.1%) after the
> changes, so instead let's look at the total cycles spent processing NBDB
> contents:
> 
>   Children      Self  Command     Shared Object       Symbol
> before:
>     75.95%     0.01%  ovn-northd  ovn-northd          [.] ovnnb_db_run
> after:
>      8.55%     0.02%  ovn-northd  ovn-northd          [.] ovnnb_db_run
> 
> Honestly, I am not sure where the huge boost comes from. I need to read
> through the changes more closely. Feel free to add my:
> 
> Tested-by: Jakub Sitnicki <jkbs at redhat.com>

Thanks a lot for the testing.  Please do feel free to continue reading
the patch series.  I am also surprised that there was so much impact--I
guess that hashing text strings is really expensive!

Thanks,

Ben.