[ovs-git] [ovn-org/ovn] f9c302: northd.c: Optimize parallel build performance with...

Han Zhou noreply at github.com
Tue Oct 5 17:15:44 UTC 2021


  Branch: refs/heads/master
  Home:   https://github.com/ovn-org/ovn
  Commit: f9c302d3317a8f48451294cf8979c97d2a9a1aef
      https://github.com/ovn-org/ovn/commit/f9c302d3317a8f48451294cf8979c97d2a9a1aef
  Author: Han Zhou <hzhou at ovn.org>
  Date:   2021-10-05 (Tue, 05 Oct 2021)

  Changed paths:
    M northd/northd.c

  Log Message:
  -----------
  northd.c: Optimize parallel build performance with hash based locks.

The current implementation of parallel build in northd with dp-groups
enabled results in bad performance when the below assumption is not
true:

 * 3. Most RW ops are not flow adds, but changes to the
 * od groups.

In fact most (if not all) of the ovn port based flows don't share
dp-groups, so the assumption doesn't hold in the reality, and in a scale
test environment with ovn-k8s topology of 800 nodes, the parallel build
shows 5 times more time spent for one northd iteration than without
parallel build on a test machine with 24 cores (24 northd worker
threads). This is because of lock contension on the global rw lock
protecting the lflow hmap.

This patch optimizes it by using an array of bash based locks instead of
a single global lock. It is similar to the approach prior to the commit
8c980ce6, but with two major differences:

1) It uses a fixed length mutex array instead of the dynamic array of
   the struct hashrow_locks. It is equally efficient considering the low
   chance of contention in a large array of locks, but without the burden
   of resizing every time the hmap size changes. The uniqueness of the
   lock is guaranteed by combining the masks of both the hmap and the
   mutex array.

2) It fixes the corrupted hmap size after each round of parallel flow
   build. The hash based lock protects the list in each bucket, but
   doesn't protect the hmap size. The patch uses thread-local counters
   and aggregate them at the end of each iteration, which is lock free.
   This approach has lower cost than alternatively using atomic
   incrementing a global counter.

This patch ends up with 8 times speedup than the current parallel build
with dp-group enabled for the same scale test (which is 30% faster than
without parallel).

Test command: ovn-nbctl --print-wait-time --wait=sb sync

Before:

no parallel:
ovn-northd completion: 7807ms

parallel:
ovn-northd completion: 41267ms

After:

no parallel: (no change)

parallel:
ovn-northd completion: 5081ms
(8x faster than before, 30% faster than no parallel)

Note: all the above tests are with dp-groups enabled)

Signed-off-by: Han Zhou <hzhou at ovn.org>
Acked-by: Numan Siddique <numans at ovn.org>


  Commit: 3fb397b63663297acbcbf794e1233951222ae5af
      https://github.com/ovn-org/ovn/commit/3fb397b63663297acbcbf794e1233951222ae5af
  Author: Han Zhou <hzhou at ovn.org>
  Date:   2021-10-05 (Tue, 05 Oct 2021)

  Changed paths:
    M lib/ovn-parallel-hmap.h

  Log Message:
  -----------
  ovn-parallel-hmap.h: Minor fixes for hashrow_lock.

Although not used currently, it is better to fix:
1. The type of the mask field should be the same as hmap->mask: size_t.
2. Calculating the index is better to use & instead of %.

Signed-off-by: Han Zhou <hzhou at ovn.org>
Acked-by: Numan Siddique <numans at ovn.org>


Compare: https://github.com/ovn-org/ovn/compare/ab4bdb55cd04...3fb397b63663


More information about the git mailing list