<div dir="ltr"><div><font face="arial, sans-serif">[PATCH V2 1/3] coverage: Reimplement the &quot;ovs-appclt coverage/show&quot; command</font><br></div><div><font face="arial, sans-serif"><br></font></div><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">This commit changes the &quot;ovs-appclt coverage/show&quot; command to show the</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">per-second, per-minute and per-hour rates of function invocation. More</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">importantly, this makes using coverage counter an easy way to monitor</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">the execution of specific functions.</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">Signed-off-by: Alex Wang &lt;</span><a href="mailto:alexw@nicira.com" style="font-family:arial,sans-serif;font-size:13.333333969116211px">alexw@nicira.com</a><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">&gt;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">---</span><div><br></div><div style>v1 -&gt; v2:</div><div style>- abandon the wakeup timer and coverage_wait() function</div><div style><br></div>
<div>---<br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> lib/coverage-unixctl.man |    3 +-</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> lib/coverage.c           |   95 ++++++++++++++++++++++++++++++</span><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+++++++++-------</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> lib/coverage.h           |   17 +++++++--</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> lib/timeval.c            |    1 -</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> vswitchd/ovs-vswitchd.c  |    2 +</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> 5 files changed, 99 insertions(+), 19 deletions(-)</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">diff --git a/lib/coverage-unixctl.man b/lib/coverage-unixctl.man</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">index 9718894..2d283dc 100644</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">--- a/lib/coverage-unixctl.man</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+++ b/lib/coverage-unixctl.man</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">@@ -8,4 +8,5 @@ main loop takes unusually long to run.</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> Coverage counters are useful mainly for performance analysis and</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> debugging.</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> .IP &quot;\fBcoverage/show\fR&quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-Displays the values of all of the coverage counters.</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+Displays the per-second, per-minute and per-hour rates, and total count</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+of all of the coverage counters.</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">diff --git a/lib/coverage.c b/lib/coverage.c</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">index f152474..e3e7610 100644</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">--- a/lib/coverage.c</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+++ b/lib/coverage.c</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">@@ -47,9 +47,21 @@ struct coverage_counter *coverage_counters[] = {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> #define n_coverage_counters ARRAY_SIZE(coverage_counters)</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> #endif  /* !USE_LINKER_SECTIONS */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-static unsigned int epoch;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+/* Makes coverage_run run every 5 seconds. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+#define COVERAGE_RUN_INTERVAL    5000</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+static long long int coverage_run_time = LLONG_MIN;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+/* Defines the moving average array index variables. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+static unsigned int min_idx = 0;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+static unsigned int hr_idx = 0;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+/* Index counter used to compute the moving average array&#39;s index. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+static unsigned int idx_count = 0;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> static void coverage_read(struct svec *);</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+static unsigned int coverage_array_sum(const unsigned int *arr,</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                                       const unsigned int len);</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> static void</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> coverage_unixctl_show(struct unixctl_conn *conn, int argc OVS_UNUSED,</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">@@ -170,7 +182,7 @@ coverage_log(void)</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">         uint32_t hash = coverage_hash();</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">         if (coverage_hit(hash)) {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">             VLOG_INFO(&quot;Skipping details of duplicate event coverage for &quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-                      &quot;hash=%08&quot;PRIx32&quot; in epoch %u&quot;, hash, epoch);</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                      &quot;hash=%08&quot;PRIx32, hash);</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">         } else {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">             struct svec lines;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">             const char *line;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">@@ -189,8 +201,16 @@ coverage_log(void)</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> static void</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> coverage_read_counter(struct svec *lines, const struct coverage_counter *c)</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-    svec_add_nocopy(lines, xasprintf(&quot;%-24s %5u / %9llu&quot;,</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-                                     c-&gt;name, c-&gt;count, c-&gt;count + c-&gt;total));</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    /* The per second rate is calculated via averaging the counts in</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+     * last 5-second interval. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    svec_add_nocopy(lines,</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+            xasprintf(&quot;%-24s %5.1f/sec %7u/min &quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                      &quot;%9u/hr   total: %llu&quot;,</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                      c-&gt;name,</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                      c-&gt;sec_to_min[(min_idx + 11) % MIN_LEN] / 5.0,</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                      coverage_array_sum(c-&gt;sec_to_</span><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">min, MIN_LEN),</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                      coverage_array_sum(c-&gt;min_to_</span><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">hr,  HR_LEN),</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                      c-&gt;count + c-&gt;total));</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> }</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> /* Adds coverage counter information to &#39;lines&#39;. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">@@ -204,8 +224,8 @@ coverage_read(struct svec *lines)</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">     hash = coverage_hash();</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">     n_never_hit = 0;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-    svec_add_nocopy(lines, xasprintf(&quot;Event coverage (epoch %u/entire run), &quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-                                     &quot;hash=%08&quot;PRIx32&quot;:&quot;, epoch, hash));</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    svec_add_nocopy(lines, xasprintf(&quot;Event coverage (frequency), &quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                                     &quot;hash=%08&quot;PRIx32&quot;:&quot;, hash));</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">     for (i = 0; i &lt; n_coverage_counters; i++) {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">         struct coverage_counter *c = coverage_counters[i];</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">         if (c-&gt;count) {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">@@ -225,16 +245,65 @@ coverage_read(struct svec *lines)</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">     svec_add_nocopy(lines, xasprintf(&quot;%zu events never hit&quot;, n_never_hit));</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> }</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-/* Advances to the next epoch of coverage, resetting all the counters to 0. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+/* Runs every second to update the coverage counters&#39;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+ * sec_to_min array and min_to_hr array. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> void</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-coverage_clear(void)</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+coverage_run(void)</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    long long int cur_time = time_msec();</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    /* Initialize the coverage_run_time. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    if (coverage_run_time == LLONG_MIN) {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+        coverage_run_time = time_msec();</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    }</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    if (cur_time &gt;= coverage_run_time) {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+        size_t i, j;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+        /* Computes the number of 5-second slots. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+        int slots = (cur_time - coverage_run_time) / COVERAGE_RUN_INTERVAL</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                    + 1;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+        /* This loop is necessary, since it is possible that the actual</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+         * run interval is multiple of 5 seconds. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+        for (i = 0; i &lt; slots; i++) {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+            /* Updates the index variables. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+            /* The min_idx is increased from 0 to 11. Every time the</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+             * min_idx finishes a cycle (a cycle is one minute),</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+             * the hr_idx is incremented by 1. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+            min_idx = idx_count % MIN_LEN;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+            hr_idx  = idx_count / MIN_LEN;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+            for (j = 0; j &lt; n_coverage_counters; j++) {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                struct coverage_counter *c = coverage_counters[j];</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                /* Calculates the portion to be added, which is the average</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                 * of counts over the time. Note, the last loop will also</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                 * include the remainder. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                int portion = c-&gt;count / slots + (i == slots - 1 ?</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                              c-&gt;count % slots : 0);</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                c-&gt;total += portion;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                c-&gt;count = (i == slots -1 ? 0 : c-&gt;count) ;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                c-&gt;sec_to_min[min_idx] = portion;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                /* Note, only adds up the latest count portion here. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                c-&gt;min_to_hr[hr_idx] = min_idx == 0 ?</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                     portion : (c-&gt;min_to_hr[hr_idx] + portion);</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+            }</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+            /* This is to guarantee that mov_avg_hr_idx ranges from 0 to 59. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+            idx_count = (idx_count + 1) % 720;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+            /* Updates the run time. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+            coverage_run_time += COVERAGE_RUN_INTERVAL;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+        }</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    }</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+}</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+static unsigned int</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+coverage_array_sum(const unsigned int *arr, const unsigned int len)</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+{</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    unsigned int sum = 0;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">     size_t i;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-    epoch++;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-    for (i = 0; i &lt; n_coverage_counters; i++) {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-        struct coverage_counter *c = coverage_counters[i];</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-        c-&gt;total += c-&gt;count;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-        c-&gt;count = 0;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    for (i = 0; i &lt; len; i++) {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+        sum += arr[i];</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">     }</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    return sum;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> }</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">diff --git a/lib/coverage.h b/lib/coverage.h</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">index 968c489..26837ed 100644</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">--- a/lib/coverage.h</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+++ b/lib/coverage.h</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">@@ -29,11 +29,19 @@</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> #include &quot;vlog.h&quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+/* Defines the moving average array length. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+#define MIN_LEN 12   /* 12 * 5 seconds = 1 minute */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+#define HR_LEN  60   /* 60 * 1 minute = 1 hour */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> /* A coverage counter. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> struct coverage_counter {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">     const char *name;           /* Textual name. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-    unsigned int count;         /* Count within the current epoch. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-    unsigned long long int total; /* Total count over all epochs. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    unsigned int count;         /* Count within the current second. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    unsigned long long int total; /* Total count over all time. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    /* The moving average arrays. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    unsigned int sec_to_min[MIN_LEN];</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+    unsigned int min_to_hr[HR_LEN];</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> };</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> /* Defines COUNTER.  There must be exactly one such definition at file scope</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">@@ -57,11 +65,12 @@ struct coverage_counter {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> void coverage_init(void);</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> void coverage_log(void);</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-void coverage_clear(void);</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+void coverage_run(void);</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> /* Implementation detail. */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> #define COVERAGE_DEFINE__(COUNTER)                              \</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">         extern struct coverage_counter counter_##COUNTER;       \</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-        struct coverage_counter counter_##COUNTER = { #COUNTER, 0, 0 }</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+        struct coverage_counter counter_##COUNTER = { #COUNTER, 0, 0, \</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+                                                      {0}, {0} }</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> #endif /* coverage.h */</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">diff --git a/lib/timeval.c b/lib/timeval.c</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">index 5ff59bc..fe4f2a7 100644</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">--- a/lib/timeval.c</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+++ b/lib/timeval.c</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">@@ -333,7 +333,6 @@ time_poll(struct pollfd *pollfds, int n_pollfds, long long int timeout_when,</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">     if (last_wakeup) {</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">         log_poll_interval(last_wakeup)</span><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">     }</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">-    coverage_clear();</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">     start = time_msec();</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">     blocked = false;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">diff --git a/vswitchd/ovs-vswitchd.c b/vswitchd/ovs-vswitchd.c</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">index e11febd..5194d97 100644</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">--- a/vswitchd/ovs-vswitchd.c</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+++ b/vswitchd/ovs-vswitchd.c</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">@@ -28,6 +28,7 @@</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> #include &quot;bridge.h&quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> #include &quot;command-line.h&quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> #include &quot;compiler.h&quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+#include &quot;coverage.h&quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> #include &quot;daemon.h&quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> #include &quot;dirs.h&quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> #include &quot;dpif.h&quot;</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">@@ -126,6 +127,7 @@ main(int argc, char *argv[])</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">         bridge_run_fast();</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">         unixctl_server_run(unixctl);</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">         netdev_run();</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">+        coverage_run();</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<br style="font-family:arial,sans-serif;font-size:13.333333969116211px"><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">         worker_wait();</span><br style="font-family:arial,sans-serif;font-size:13.333333969116211px">
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px">         signal_wait(sighup);</span><div class="" style="font-family:arial,sans-serif;font-size:13.333333969116211px"><div id=":133" class="" tabindex="0">
<img class="" src="https://mail.google.com/mail/u/0/images/cleardot.gif" style=""></div></div><div class="" style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br></div><span class="" style="font-family:arial,sans-serif;font-size:13.333333969116211px"><font color="#888888">--<br>
1.7.9.5</font></span><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 17, 2013 at 11:11 AM, Alex Wang <span dir="ltr">&lt;<a href="mailto:alexw@nicira.com" target="_blank">alexw@nicira.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Ben,<div class="gmail_extra"><br><br><div class="gmail_quote"><div class="im">On Wed, Jun 12, 2013 at 4:20 PM, Ben Pfaff <span dir="ltr">&lt;<a href="mailto:blp@nicira.com" target="_blank">blp@nicira.com</a>&gt;</span> wrote:<br>





<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>On Wed, Jun 12, 2013 at 11:38:41AM -0700, Alex Wang wrote:<br>





&gt; This commit changes the &quot;ovs-appclt coverage/show&quot; command to show the<br>
&gt; per-second, per-minute and per-hour rates of function invocation. More<br>
&gt; importantly, this makes using coverage counter an easy way to monitor<br>
&gt; the execution of specific functions.<br>
&gt;<br>
&gt; Signed-off-by: Alex Wang &lt;<a href="mailto:alexw@nicira.com" target="_blank">alexw@nicira.com</a>&gt;<br>
&gt; ---<br>
&gt; Possible Issue:<br>
&gt;<br>
&gt; The &quot;coverage_run&quot; runs every second. And the number of addition operations is<br>
&gt; not very large. If it is still considered to be time-consuming, I&#39;ll adjust<br>
&gt; further.<br>
&gt;<br>
<br>
</div>I have two high-level design questions.<br>
<br>
First is the cost, not of the periodic summations (I doubt that cost<br>
matters) but of forcing a wakeup every second.  This probably doesn&#39;t<br>
matter in ovs-vswitchd, but other daemons that don&#39;t wake up so<br>
frequently also use the poll_loop library, and it would be a shame to<br>
drive up system load unnecessarily.  So I would prefer, if possible,<br>
for the coverage code not to use a timer but instead to just do work<br>
in coverage_run() whenever the process happens to wake up.<br></blockquote><div><br></div><div><br></div></div><div>This is really important to know.</div><div class="im"><div> </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">






Second is accuracy.  I believe that the current code considers any<br>
interval greater than or equal to a second as exactly a second.<br>
Because wakeups aren&#39;t precise, that tends to stretch out the<br>
definition of a second to somewhat longer.  If you take my suggestion<br>
about cost, above, that makes the problem worse.  So I think that it<br>
may be worthwhile to measure the length of the actual interval for a<br>
&quot;second&quot; and divide by its length.  For example, if there were 100<br>
events of some type over 1.5 seconds, we would count that as 100 / 1.5<br>
== 66.67 events per second, and use that in the average.  That might<br>
require using &quot;double&quot;s for the average counters, or we could use<br>
fixed-point arithmetic; I think either solution could be OK.<br></blockquote><div><br></div><div><br></div></div><div>Yes, this will achieve the per-second rate. But will not be very easy to compute the</div><div>per-minute and hourly rate. So far, I cannot find a very efficient way to calculate</div>



<div>these (maybe use long linked-lists that count to 1 minute and 1 hour?).</div><div><br></div><div><div>I&#39;ll put detailed comments in v2.</div></div><div class="im"><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">





I didn&#39;t do a detailed review yet because I&#39;d like to hear your<br>
thoughts on these points first.  I did notice that<br>
coverage_array_sum() is only used in coverage.c, so I would define it<br>
there (and not &quot;inline&quot;), not in coverage.h.<br></blockquote><div><br></div><div><br></div></div><div>Thanks for pointing this out.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">


Thanks,<br>
<br>
Ben.</blockquote></div></div></div>
</blockquote></div><br></div>