[ovs-dev] [PATCH v4] lib: Use a more accurate value for CPU count (sched_getaffinity).

[Quentin Monnet]

From: David Marchand <david.marchand at 6wind.com>

Relying on /proc/cpuinfo to count the number of available cores is not
the best option:

- The code is x86-specific.
- If the process is started with a different CPU affinity, then it will
  wrongly try to start too many threads (for an example, imagine an OVS
  daemon restricted to 4 CPU threads on a 128 threads system).

This commit removes /proc/cpuinfo parsing. For Linux systems, it
introduces instead a call to sched_getaffinity(), which is
architecture-independant, in order to retrieve the list of CPU threads
available to the current process and to count them. Other UNIX-like
systems only use _SC_NPROCESSORS_ONLN.

Signed-off-by: David Marchand <david.marchand at 6wind.com>
Signed-off-by: Liu Xiaofeng <xiaofeng.liu at 6wind.com>
Signed-off-by: Quentin Monnet <quentin.monnet at 6wind.com>
---
v4:
sched_getaffinity() is specific to Linux, do not use it for other systems

v3:
Add `CPU_FREE(set);`, at the correct location

v2:
Add `CPU_FREE(set);`

Tests: on a VM with 3 cores:
- OVS master:
# taskset -c 1 ovs-vswitchd --pidfile
2016-06-02T11:05:47Z|00001|ovs_numa|INFO|Discovered 3 CPU cores on NUMA node 0
2016-06-02T11:05:47Z|00002|ovs_numa|INFO|Discovered 1 NUMA nodes and 3 CPU cores
[…]
2016-06-02T11:05:57Z|00019|memory|INFO|5072 kB peak resident set size after 10.0 seconds
2016-06-02T11:05:57Z|00020|memory|INFO|handlers:2 ports:2 revalidators:1 rules:5 udpif keys:1

- OVS with patch:
# taskset -c 1 ovs-vswitchd --pidfile
2016-06-02T11:10:12Z|00001|ovs_numa|INFO|Discovered 3 CPU cores on NUMA node 0
2016-06-02T11:10:12Z|00002|ovs_numa|INFO|Discovered 1 NUMA nodes and 3 CPU cores
[…]
2016-06-02T11:10:22Z|00019|memory|INFO|5052 kB peak resident set size after 10.0 seconds
2016-06-02T11:10:22Z|00020|memory|INFO|handlers:1 ports:2 revalidators:1 rules:5

 lib/ovs-thread.c | 78 +++++++++++---------------------------------------------
 1 file changed, 15 insertions(+), 63 deletions(-)

diff --git a/lib/ovs-thread.c b/lib/ovs-thread.c
index 6e6345273c7c..eee8d9e45b79 100644
--- a/lib/ovs-thread.c
+++ b/lib/ovs-thread.c
@@ -547,67 +547,8 @@ ovs_thread_stats_next_bucket(const struct ovsthread_stats *stats, size_t i)
 }
 
 
-/* Parses /proc/cpuinfo for the total number of physical cores on this system
- * across all CPU packages, not counting hyper-threads.
- *
- * Sets *n_cores to the total number of cores on this system, or 0 if the
- * number cannot be determined. */
-static void
-parse_cpuinfo(long int *n_cores)
-{
-    static const char file_name[] = "/proc/cpuinfo";
-    char line[128];
-    uint64_t cpu = 0; /* Support up to 64 CPU packages on a single system. */
-    long int cores = 0;
-    FILE *stream;
-
-    stream = fopen(file_name, "r");
-    if (!stream) {
-        VLOG_DBG("%s: open failed (%s)", file_name, ovs_strerror(errno));
-        return;
-    }
-
-    while (fgets(line, sizeof line, stream)) {
-        unsigned int id;
-
-        /* Find the next CPU package. */
-        if (ovs_scan(line, "physical id%*[^:]: %u", &id)) {
-            if (id > 63) {
-                VLOG_WARN("Counted over 64 CPU packages on this system. "
-                          "Parsing %s for core count may be inaccurate.",
-                          file_name);
-                cores = 0;
-                break;
-            }
-
-            if (cpu & (1ULL << id)) {
-                /* We've already counted this package's cores. */
-                continue;
-            }
-            cpu |= 1ULL << id;
-
-            /* Find the number of cores for this package. */
-            while (fgets(line, sizeof line, stream)) {
-                int count;
-
-                if (ovs_scan(line, "cpu cores%*[^:]: %u", &count)) {
-                    cores += count;
-                    break;
-                }
-            }
-        }
-    }
-    fclose(stream);
-
-    *n_cores = cores;
-}
-
-/* Returns the total number of cores on this system, or 0 if the number cannot
- * be determined.
- *
- * Tries not to count hyper-threads, but may be inaccurate - particularly on
- * platforms that do not provide /proc/cpuinfo, but also if /proc/cpuinfo is
- * formatted different to the layout that parse_cpuinfo() expects. */
+/* Returns the total number of cores available to this process, or 0 if the
+ * number cannot be determined. */
 int
 count_cpu_cores(void)
 {
@@ -616,10 +557,21 @@ count_cpu_cores(void)
 
     if (ovsthread_once_start(&once)) {
 #ifndef _WIN32
-        parse_cpuinfo(&n_cores);
-        if (!n_cores) {
-            n_cores = sysconf(_SC_NPROCESSORS_ONLN);
+        n_cores = sysconf(_SC_NPROCESSORS_ONLN);
+#ifdef __linux__
+        if (n_cores > 0) {
+            cpu_set_t *set = CPU_ALLOC(n_cores);
+
+            if (set) {
+                size_t size = CPU_ALLOC_SIZE(n_cores);
+
+                if (!sched_getaffinity(0, size, set)) {
+                    n_cores = CPU_COUNT_S(size, set);
+                }
+                CPU_FREE(set);
+            }
         }
+#endif
 #else
         SYSTEM_INFO sysinfo;
         GetSystemInfo(&sysinfo);
-- 
2.1.4