[ovs-dev] [PATCH v6 0/8] Add OVS DPDK keep-alive functionality.

Fischetti, Antonio antonio.fischetti at intel.com
Fri Dec 15 14:23:45 UTC 2017


Hi Bhanu,
I've tested v6 and LGTM, works as expected.

To check the behavior I blocked one or more of the PMD with the trick of 
attaching cgdb. So with 

   utilities/ovs-appctl keepalive/pmd-health-show

I could see for example 2 of the 3 PMDs - those I intentionally blocked - 
correctly reported as DEAD first, then GONE like

                Keepalive status

keepalive status   : Enabled
keepalive interval : 1000 ms
keepalive init time: 15 Dec 2017 13:20:54
PMD threads        : 3

 PMD    CORE    STATE   LAST SEEN TIMESTAMP(UTC)
pmd62    5      GONE    15 Dec 2017 13:22:39
pmd65    7      ALIVE   15 Dec 2017 13:22:50
pmd63    4      GONE    15 Dec 2017 13:22:14


Thanks,
Antonio

> -----Original Message-----
> From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-
> bounces at openvswitch.org] On Behalf Of Bhanuprakash Bodireddy
> Sent: Friday, December 8, 2017 12:04 PM
> To: dev at openvswitch.org
> Subject: [ovs-dev] [PATCH v6 0/8] Add OVS DPDK keep-alive functionality.
> 
> Keepalive feature is aimed at achieving Fastpath Service Assurance
> in OVS-DPDK deployments. It adds support for monitoring the packet
> processing threads by dispatching heartbeats at regular intervals.
> 
> keepalive feature can be enabled through below OVSDB settings.
> 
>     enable-keepalive=true
>       - Keepalive feature is disabled by default and should be enabled
>         at startup before ovs-vswitchd daemon is started.
> 
>     keepalive-interval="5000"
>       - Timer interval in milliseconds for monitoring the packet
>         processing cores.
> 
> TESTING:
>     The testing of keepalive is done using stress cmd (simulating the
> stalls).
>       - pmd-cpu-mask=0xf [MQ enabled on DPDK ports]
>       - stress -c 1 &          [tid is usually the __tid + 1 of the
> output]
>       - chrt -r -p 99 <tid>    [set realtime priority for stress thread]
>       - taskset -p 0x8 <tid>   [Pin the stress thread to the core PMD is
> running]
>       - PMD thread will be descheduled due to its normal priority and
> yields
>         core to stress thread.
> 
>       - ovs-appctl keepalive/pmd-health-show   [Display that the thread
> is GONE]
>       - ./ovsdb/ovsdb-client monitor Open_vSwitch  [Should update the
> status]
> 
>       - taskset -p 0x10 <tid>  [This brings back pmd thread to life as
> stress thread
>                                 is moved to idle core]
> 
>       (watch out for stress threads, and carefully pin them to core not
> to hang your DUTs
>        during tesing).
> 
> v5 -> v6
>   * Remove 2 patches from series
>      - xnanosleep was applied to master as part of high resolution
> timeout support.
>      - Extend get_process_info() API was also applied to master earlier.
>   * Remove KA_STATE_DOZING as it was initially meant to handle Core C
> states, not needed
>     for now.
>   * Fixed ka_destroy(), to fix unit test cases 536, 537.
>   * A minor performance degradation(0.5%) is observed with Keepalive
> enabled.
>     [Tested with loopback case using 1000 IXIA streams/64 byte udp pkts
> and
>     1 PMD thread(9.239 vs 9.177Mpps) at 10ms ka-interval timeout]
>   * Verified with sparse, MSVC compilers(appveyor).
> 
> v4 -> v5
>   * Add 3 more patches to the series
>      - xnanosleep()
>      - Documentation
>      - Update to NEWS
>   * Remove all references to core_id and instead implemented thread
> based tracking.
>   * Addressed most of the comments in v4.
> 
> v3 -> v4
>   * Split the functionality in to 2 parts. This patch series only
> updates
>     PMD status to OVSDB. The incremental patch series to handle false
> positives,
>     negatives and more checking and stats.
>   * Remove code from netdev layer and dependency on rte_keepalive lib.
>   * Merged few patches and simplified the patch series.
>   * Timestamp in human readable form.
> 
> v2 -> v3
>   * Rebase.
>   * Verified with dpdk-stable-17.05.1 release.
>   * Fixed build issues with MSVC and cross checked with appveyor.
> 
> v1 -> v2
>   * Rebase
>   * Drop 01/20 Patch "Consolidate process related APIs" of V1 as it
>     is already applied as separate patch.
> 
> RFCv3 -> v1
>   * Made changes to fix failures in some unit test cases.
>   * some more code cleanup w.r.t process related APIs.
> 
> RFCv2 -> RFCv3
>   * Remove POSIX shared memory block implementation (suggested by
> Aaron).
>   * Rework the logic to register and track threads instead of cores.
> This way
>     in the future any thread can be registered to KA framework. For now
> only PMD
>     threads are tracked (suggested by Aaron).
>   * Refactor few APIs and further clean up the code.
> 
> RFCv1 -> RFCv2
>   * Merged the xml and schema commits to later commit where the actual
>     implementation is done(suggested by Ben).
>   * Fix ovs-appctl keepalive/* hang issue when KA disabled.
>   * Fixed memory leaks with appctl commands for keepalive/pmd-health-
> show,
>     pmd-xstats-show.
>   * Refactor code and fixed APIs dealing with PMD health monitoring.
> 
> 
> Bhanuprakash Bodireddy (8):
>   Keepalive: Add initial keepalive configuration.
>   dpif-netdev: Register packet processing cores to KA framework.
>   dpif-netdev: Enable heartbeats for DPDK datapath.
>   keepalive: Retrieve PMD status periodically.
>   bridge: Update keepalive status in OVSDB.
>   keepalive: Add support to query keepalive status and statistics.
>   Documentation: Update DPDK doc with Keepalive feature.
>   NEWS: Add keepalive support information in NEWS.
> 
>  Documentation/howto/dpdk.rst | 112 +++++++++
>  NEWS                         |   2 +
>  lib/automake.mk              |   2 +
>  lib/dpif-netdev.c            |  92 ++++++++
>  lib/keepalive.c              | 552
> +++++++++++++++++++++++++++++++++++++++++++
>  lib/keepalive.h              | 109 +++++++++
>  lib/ovs-thread.c             |   6 +
>  lib/ovs-thread.h             |   1 +
>  lib/util.c                   |  22 ++
>  lib/util.h                   |   1 +
>  vswitchd/bridge.c            |  29 +++
>  vswitchd/vswitch.ovsschema   |   8 +-
>  vswitchd/vswitch.xml         |  49 ++++
>  13 files changed, 983 insertions(+), 2 deletions(-)
>  create mode 100644 lib/keepalive.c
>  create mode 100644 lib/keepalive.h
> 
> --
> 2.4.11
> 
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev


More information about the dev mailing list