[ovs-git] [openvswitch/ovs] eca34e: raft: Set threshold on backlog for raft connections.

Ilya Maximets noreply at github.com
Tue Nov 10 08:07:09 UTC 2020


  Branch: refs/heads/master
  Home:   https://github.com/openvswitch/ovs
  Commit: eca34ebd7c418c0351eb92ae615d07edc31a9404
      https://github.com/openvswitch/ovs/commit/eca34ebd7c418c0351eb92ae615d07edc31a9404
  Author: Ilya Maximets <i.maximets at ovn.org>
  Date:   2020-11-10 (Tue, 10 Nov 2020)

  Changed paths:
    M NEWS
    M lib/jsonrpc.c
    M lib/jsonrpc.h
    M ovsdb/raft.c

  Log Message:
  -----------
  raft: Set threshold on backlog for raft connections.

RAFT messages could be fairly big.  If something abnormal happens to
one of the servers in a cluster it may not be able to process all the
incoming messages in a timely manner.  This results in jsonrpc backlog
growth on the sender's side.  For example if follower gets many new
clients at once that it needs to serve, or it decides to take a
snapshot in a period of high number of database changes.
If backlog grows large enough it becomes harder and harder for follower
to process incoming raft messages, it sends outdated replies and
starts receiving snapshots and the whole raft log from the leader.
Sometimes backlog grows too high (60GB in this example):

      jsonrpc|INFO|excessive sending backlog, jsonrpc: ssl:<ip>,
                   num of msgs: 15370, backlog: 61731060773.

In this case OS might actually decide to kill the sender to free some
memory.  Anyway, It could take a lot of time for such a server to catch
up with the rest of the cluster if it has so much data to receive and
process.

Introducing backlog thresholds for jsonrpc connections.
If sending backlog will exceed particular values (500 messages or
4GB in size), connection will be dropped and re-created.  This will
allow to drop all the current backlog and start over increasing
chances of cluster recovery.

Reported-at: https://bugzilla.redhat.com/show_bug.cgi?id=1888829
Acked-by: Dumitru Ceara <dceara at redhat.com>
Signed-off-by: Ilya Maximets <i.maximets at ovn.org>


  Commit: 80e3becdc1eea9b92253a391c0071e6218dda7d8
      https://github.com/openvswitch/ovs/commit/80e3becdc1eea9b92253a391c0071e6218dda7d8
  Author: Ilya Maximets <i.maximets at ovn.org>
  Date:   2020-11-10 (Tue, 10 Nov 2020)

  Changed paths:
    M NEWS
    M ovsdb/ovsdb-server.1.in
    M ovsdb/raft.c

  Log Message:
  -----------
  raft: Make backlog thresholds configurable.

New appctl 'cluster/set-backlog-threshold' to configure thresholds
on backlog of raft jsonrpc connections.  Could be used, for example,
in some extreme conditions where size of a database expected to be
very large, i.e. comparable with default 4GB threshold.

Acked-by: Dumitru Ceara <dceara at redhat.com>
Signed-off-by: Ilya Maximets <i.maximets at ovn.org>


  Commit: 1090a949ac920b4e7ee901cee36008408a1c2386
      https://github.com/openvswitch/ovs/commit/1090a949ac920b4e7ee901cee36008408a1c2386
  Author: Yi-Hung Wei <yihung.wei at gmail.com>
  Date:   2020-11-10 (Tue, 10 Nov 2020)

  Changed paths:
    M ovsdb/log.c

  Log Message:
  -----------
  ovsdb: Remove read permission of *.db from others.

Currently, when ovsdb *.db is created by ovsdb-tool it grants read
permission to others.  This may incur security concerns, for example,
IPsec Pre-shared keys are stored in ovs-vsitchd.conf.db.
This patch addresses the concerns by removing permission for others.

Reported-by: Antonin Bas <abas at vmware.com>
Acked-by: Mark Gray <mark.d.gray at redhat.com>
Signed-off-by: Yi-Hung Wei <yihung.wei at gmail.com>
Signed-off-by: Ilya Maximets <i.maximets at ovn.org>


  Commit: 588821eacf03e85263dc1f3b9755746a859c68d9
      https://github.com/openvswitch/ovs/commit/588821eacf03e85263dc1f3b9755746a859c68d9
  Author: Eli Britstein <elibr at nvidia.com>
  Date:   2020-11-10 (Tue, 10 Nov 2020)

  Changed paths:
    M lib/netdev-offload-dpdk.c

  Log Message:
  -----------
  netdev-offload-dpdk: Preserve HW statistics for modified flows.

In case of a flow modification, preserve the HW statistics of the old HW
flow to the new one.

Fixes: 3c7330ebf036 ("netdev-offload-dpdk: Support offload of output action.")
Signed-off-by: Eli Britstein <elibr at nvidia.com>
Reviewed-by: Gaetan Rivet <gaetanr at nvidia.com>
Acked-by: Sriharsha Basavapatna <sriharsha.basavapatna at broadcom.com>
Tested-by: Emma Finn <emma.finn at intel.com>
Signed-off-by: Ilya Maximets <i.maximets at ovn.org>


  Commit: 2fe34c03078f1fe01a39b4d963a9a367ba468bad
      https://github.com/openvswitch/ovs/commit/2fe34c03078f1fe01a39b4d963a9a367ba468bad
  Author: Tonghao Zhang <xiangxia.m.yue at gmail.com>
  Date:   2020-11-10 (Tue, 10 Nov 2020)

  Changed paths:
    M NEWS
    M lib/dpctl.c
    M lib/dpctl.man
    M tests/pmd.at

  Log Message:
  -----------
  dpctl: Add the option 'pmd' for dump-flows.

"ovs-appctl dpctl/dump-flows" added the option
"pmd" which allow user to dump pmd specified.

That option is useful to dump rules of pmd
when we have a large number of rules in dp.

Signed-off-by: Tonghao Zhang <xiangxia.m.yue at gmail.com>
Acked-by: Gaetan Rivet <grive at u256.net>
Signed-off-by: Ilya Maximets <i.maximets at ovn.org>


  Commit: ed8cf18733fd1f4865d8f16fa06180bf5a772ebe
      https://github.com/openvswitch/ovs/commit/ed8cf18733fd1f4865d8f16fa06180bf5a772ebe
  Author: Ilya Maximets <i.maximets at ovn.org>
  Date:   2020-11-10 (Tue, 10 Nov 2020)

  Changed paths:
    M Documentation/faq/releases.rst

  Log Message:
  -----------
  releases: Mark 2.13 as a new LTS release.

2.5 release is 4.5 years old and I'm not aware of anyone who actually
uses it today.  Release process documentation says that there is no
strict time period for nominating a new LTS release and that usually
it happens once in a two years.  So, proposing to nominate 2.13 as
our new LTS release since it's a first release that doesn't include
OVN inside, so we will formally not have to support it in this
repository in case there are major issues that might be hard to fix.

Suggested-by: Ben Pfaff <blp at ovn.org>
Acked-by: Flavio Leitner <fbl at sysclose.org>
Acked-by: Ian Stokes <ian.stokes at intel.com>
Acked-by: Kevin Traynor <ktraynor at redhat.com>
Reviewed-by: Simon Horman <simon.horman at netronome.com>
Signed-off-by: Ilya Maximets <i.maximets at ovn.org>


  Commit: 15177c4dadcc0470cf668cfbd9691d728f5f0721
      https://github.com/openvswitch/ovs/commit/15177c4dadcc0470cf668cfbd9691d728f5f0721
  Author: Ilya Maximets <i.maximets at ovn.org>
  Date:   2020-11-10 (Tue, 10 Nov 2020)

  Changed paths:
    M Documentation/internals/release-process.rst

  Log Message:
  -----------
  release-process: Add transition period for LTS releases.

While LTS change happens, according to release-process.rst, we're
immediately dropping support for the old LTS and, according to
backporting-patches.rst could stop backporting bug fixes to branches
older than new LTS.  While this might be OK for an upstream project
(some upstream projects like QEMU doesn't support anything at all
except the last release) it doesn't sound like a user-friendly policy.

Below addition to the release process might make the process a bit
smoother in terms that we will continue support of branches a little
bit longer even after changing current LTS, i.e. providing at least a
minimal transition period (1 release frame) for users of old LTS.

Effectively, this change means that we will support branch-2.5 until
2.15 release, i.e. we will provide the last release, if any, on
branch-2.5 somewhere around Feb 2021. (I don't actually expect many
fixes there)

Signed-off-by: Ilya Maximets <i.maximets at ovn.org>
Acked-by: Flavio Leitner <fbl at sysclose.org>
Acked-by: Kevin Traynor <ktraynor at redhat.com>


  Commit: 8c6944f6913c0c819e495bee8a5e74c218dc72b2
      https://github.com/openvswitch/ovs/commit/8c6944f6913c0c819e495bee8a5e74c218dc72b2
  Author: Ilya Maximets <i.maximets at ovn.org>
  Date:   2020-11-10 (Tue, 10 Nov 2020)

  Changed paths:
    M Documentation/internals/release-process.rst

  Log Message:
  -----------
  release-process: Standardize designation of new LTS releases.

Standardize that we will mark a new release as LTS every two years
to avoid situation where we have a really old LTS branch that no-one
actually uses, but we have to support and provide releases for it.

This will also make release process more predictable, so users will
be able to rely on it and plan their upgrades accordingly.

As a bonus, 2 years support cycle kind of aligns with 2 years support
cycle of DPDK LTS releases.

Still keeping a window for us to discuss and avoid marking some
particular release as LTS in case of significant issues with it.

Signed-off-by: Ilya Maximets <i.maximets at ovn.org>
Acked-by: Flavio Leitner <fbl at sysclose.org>
Acked-by: Kevin Traynor <ktraynor at redhat.com>


  Commit: 193995f81c07347847190b03bd9da23948d497a6
      https://github.com/openvswitch/ovs/commit/193995f81c07347847190b03bd9da23948d497a6
  Author: Ilya Maximets <i.maximets at ovn.org>
  Date:   2020-11-10 (Tue, 10 Nov 2020)

  Changed paths:
    M Documentation/internals/contributing/backporting-patches.rst
    M Documentation/internals/release-process.rst

  Log Message:
  -----------
  release-process: Policy for unmaintained branches.

While only 2 branches are formally maintained (LTS and latest release),
OVS team usually provides stable releases for other branches too, at
least for branches between LTS and latest.

When transition period ends for an old LTS, we, according to
backporting-patches.rst, could stop backporting bug fixes to branches
older than new LTS.  While this might be OK for an upstream project
it doesn't sound like a user-friendly policy just because it means
that we're dropping support for branches released less than a year
ago.

Below addition to the release process might make the process a bit
smoother in terms that we will not drop support for not so old branches
even after the transition period, if committers will follow the
"as far as it goes" backporting policy.  And we will provide stable
releases for these branches for at least 2 years (these releases could
be less frequent than releases on LTS branches).

After 2 year period (4 releases) committers are still free to backport
fixes they think are needed on older branches, however we will likely
not provide actual releases on these branches, unless it's specially
requested and discussed.

Additionally, "4 releases" policy aligns with the DPDK LTS support
policy, i.e. we will be able to validate and release last OVS releases
with the last available DPDK LTS, e.g. OVS 2.11 last stable release
will likely be released with the 18.11 EOL release validated.

Signed-off-by: Ilya Maximets <i.maximets at ovn.org>
Acked-by: Flavio Leitner <fbl at sysclose.org>
Acked-by: Kevin Traynor <ktraynor at redhat.com>


Compare: https://github.com/openvswitch/ovs/compare/c4bc03d872db...193995f81c07


More information about the git mailing list