[ovs-dev] [PATCH v10 3/3] dpif-netdev: Detection and logging of suspicious PMD iterations

[Stokes, Ian]

> This patch enhances dpif-netdev-perf to detect iterations with suspicious
> statistics according to the following criteria:
> 
> - iteration lasts longer than US_THR microseconds (default 250).
>   This can be used to capture events where a PMD is blocked or
>   interrupted for such a period of time that there is a risk for
>   dropped packets on any of its Rx queues.
> 
> - max vhost qlen exceeds a threshold Q_THR (default 128). This can
>   be used to infer virtio queue overruns and dropped packets inside
>   a VM, which are not visible in OVS otherwise.
> 
> Such suspicious iterations can be logged together with their iteration
> statistics to be able to correlate them to packet drop or other events
> outside OVS.
> 
> A new command is introduced to enable/disable logging at run-time and to
> adjust the above thresholds for suspicious iterations:
> 
> ovs-appctl dpif-netdev/pmd-perf-log-set on | off
>     [-b before] [-a after] [-e|-ne] [-us usec] [-q qlen]
> 
> Turn logging on or off at run-time (on|off).
> 
> -b before:  The number of iterations before the suspicious iteration to
>             be logged (default 5).
> -a after:   The number of iterations after the suspicious iteration to
>             be logged (default 5).
> -e:         Extend logging interval if another suspicious iteration is
>             detected before logging occurs.
> -ne:        Do not extend logging interval (default).
> -q qlen:    Suspicious vhost queue fill level threshold. Increase this
>             to 512 if the Qemu supports 1024 virtio queue length.
>             (default 128).
> -us usec:   change the duration threshold for a suspicious iteration
>             (default 250 us).
> 
> Note: Logging of suspicious iterations itself consumes a considerable
> amount of processing cycles of a PMD which may be visible in the iteration
> history. In the worst case this can lead OVS to detect another suspicious
> iteration caused by logging.
> 
> If more than 100 iterations around a suspicious iteration have been logged
> once, OVS falls back to the safe default values (-b 5/-a 5/-ne) to avoid
> that logging itself causes continuos further logging.
> 
> Signed-off-by: Jan Scheurich <jan.scheurich at ericsson.com>
> Acked-by: Billy O'Mahony <billy.o.mahony at intel.com>
> ---
>  NEWS                        |   2 +
>  lib/dpif-netdev-perf.c      | 201
> ++++++++++++++++++++++++++++++++++++++++++++
>  lib/dpif-netdev-perf.h      |  42 +++++++++
>  lib/dpif-netdev-unixctl.man |  59 +++++++++++++
>  lib/dpif-netdev.c           |   5 ++
>  5 files changed, 309 insertions(+)
> 
> diff --git a/NEWS b/NEWS
> index 8f66fd3..61148b1 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -76,6 +76,8 @@ v2.9.0 - 19 Feb 2018
>       * Commands ovs-appctl dpif-netdev/pmd-*-show can now work on a
> single PMD
>       * Detailed PMD performance metrics available with new command
>           ovs-appctl dpif-netdev/pmd-perf-show
> +     * Supervision of PMD performance metrics and logging of suspicious
> +       iterations
>     - vswitchd:
>       * Datapath IDs may now be specified as 0x1 (etc.) instead of 16
> digits.
>       * Configuring a controller, or unconfiguring all controllers, now
> deletes diff --git a/lib/dpif-netdev-perf.c b/lib/dpif-netdev-perf.c index
> 2b36410..410a209 100644
> --- a/lib/dpif-netdev-perf.c
> +++ b/lib/dpif-netdev-perf.c
> @@ -25,6 +25,24 @@
> 
>  VLOG_DEFINE_THIS_MODULE(pmd_perf);
> 
> +#define ITER_US_THRESHOLD 250   /* Warning threshold for iteration
> duration
> +                                   in microseconds. */
> +#define VHOST_QUEUE_FULL 128    /* Size of the virtio TX queue. */
> +#define LOG_IT_BEFORE 5         /* Number of iterations to log before
> +                                   suspicious iteration. */
> +#define LOG_IT_AFTER 5          /* Number of iterations to log after
> +                                   suspicious iteration. */
> +
> +bool log_enabled = false;
> +bool log_extend = false;

Will cause compilation error 'error: symbol 'log_extend' was not declared. Should it be static?'

You could declare it as an extern similar to 'bool log_enabled' in dpif-netdev-perf.h.

Thanks
Ian