[ovs-dev] [PATCH] ovsdb: add user space instruction counters

[Ben Pfaff]

On Wed, Apr 15, 2015 at 09:01:14AM -0700, Andy Zhou wrote:
> Add a set of user space instruction counters to help break down
> where instructions are spent.
> 
> Here is an example of screen capture of adding a port to database
> without any monitoring client.
> 
> ~/projs/ovs/tutorial$ ovs-appctl -t
> ovsdb-server ovsdb-server/perf-counters-clear
> ~/projs/ovs/tutorial$ ovs-vsctl add-port br0 p3
> ~/projs/ovs/tutorial$ ovs-appctl -t
> ovsdb-server ovsdb-server/perf-counters-show
> ovsdb_execute_comment       1        1069 1069.0
> ovsdb_execute_insert        2       40849 20424.5
> ovsdb_execute_mutate        1       12166 12166.0
> ovsdb_execute_select        1        9086 9086.0
> ovsdb_execute_update        4      107140 26785.0
> ovsdb_execute_wait          6       86628 14438.0
> ovsdb_txn_commit            2      906922 453461.0
> 
> However, with 300 monitoring clients, add a port yield:
> 
> ovsdb_execute_comment       1        1069 1069.0
> ovsdb_execute_insert        2       43304 21652.0
> ovsdb_execute_mutate        1       15173 15173.0
> ovsdb_execute_select        1       10560 10560.0
> ovsdb_execute_update        4      118056 29514.0
> ovsdb_execute_wait          8      121386 15173.2
> ovsdb_txn_commit            2    27976269 13988134.5
> 
> It is clear that ovsdb_txn_commit has the biggest scaling issue w.r.t
> the number of monitoring clients.
> 
> Signed-off-by: Andy Zhou <azhou at nicira.com>

This makes hundreds of tests fail on my system.  Example backtrace:

Core was generated by `ovsdb-tool transact db ["Open_vSwitch",
          {"op": "insert",
           "'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x08063fe2 in hmap_first_with_hash (hmap=<optimized out>, 
    hmap=<optimized out>, hash=4284692858) at ../lib/hmap.h:286
286	    return hmap_next_with_hash__(hmap->buckets[hash & hmap->mask], hash);
(gdb) bt
#0  0x08063fe2 in hmap_first_with_hash (hmap=<optimized out>, 
    hmap=<optimized out>, hash=4284692858) at ../lib/hmap.h:286
#1  shash_find__ (sh=<optimized out>, 
    name=name at entry=0x80798c0 <__func__.5835> "ovsdb_execute_insert", 
    name_len=20, hash=4284692858) at ../lib/shash.c:197
#2  0x080644df in shash_find (sh=0xff63397a, 
    name=0x80798c0 <__func__.5835> "ovsdb_execute_insert")
    at ../lib/shash.c:209
#3  0x080644f4 in shash_add_once (sh=0x80a01fc <perf_counters>, 
    name=0x80798c0 <__func__.5835> "ovsdb_execute_insert", 
    data=0x809d6c8 <x__>) at ../lib/shash.c:135
#4  0x08064544 in shash_add_assert (sh=0x80a01fc <perf_counters>, 
    name=0x80798c0 <__func__.5835> "ovsdb_execute_insert", 
    data=0x809d6c8 <x__>) at ../lib/shash.c:146
#5  0x08061a4c in perf_counter_init (counter=0x809d6c8 <x__>)
    at ../lib/perf-counter.c:86
#6  perf_counter_accumulate (counter=0x809d6c8 <x__>, start_count=0)
    at ../lib/perf-counter.c:95
#7  0x0804cdde in ovsdb_execute_insert (x=0xffc44dac, parser=0x0, 
    result=0x99147d8) at ../ovsdb/execution.c:350
#8  0x0804de76 in ovsdb_execute (db=0x9904108, session=0x0, params=0x1, 
    elapsed_msec=689410940416052624, timeout_msec=0x0)
    at ../ovsdb/execution.c:151
#9  0x0804bbcd in transact (read_only=<optimized out>, argc=<optimized out>, 
    argv=<optimized out>) at ../ovsdb/ovsdb-tool.c:369
#10 0x080568dd in ovs_cmdl_run_command (ctx=0xffc44e84, 
    commands=0x8079500 <all_commands>) at ../lib/command-line.c:121
#11 0x0804b44c in main (argc=4, argv=0xffc44f54) at ../ovsdb/ovsdb-tool.c:64