[ovs-dev] [PATCH V13] Function tracer to trace all function calls

[Ryan Moats]

"dev" <dev-bounces at openvswitch.org> wrote on 07/08/2016 07:04:06 PM:

> From: Nirapada Ghosh/San Jose/IBM at IBMUS
> To: dev at openvswitch.org
> Date: 07/08/2016 07:04 PM
> Subject: [ovs-dev] [PATCH V13] Function tracer to trace all function
calls
> Sent by: "dev" <dev-bounces at openvswitch.org>
>
> From: Nirapada Ghosh <nghosh at us.ibm.com>
>
> In some circumstances, we might need to figure out where in
> code, the CPU time is being spent most, so as to pinpoint
> the bottleneck and thereby resolve it with proper changes.
> Using '-finstrument-functions' flag, that can be achieved, and
> this patch exactly does that.
>
> There is a python file [generate_ft_report.py] with the patch,
> that may be used to convert this trace output to a human readable
> format with symbol names instead of address and their execution
> times. This tool uses addr2line that expects the executable to
> be built with -g flag.
>
> To enable this feature, ovs needs needs to be configured with
> "--enable-ft" command line argument [i.e. configure --enable-ft]
>
> This instrumentation logs the tracing output in separate log files
> namely func_trace_<pid>.log. It does not use VLOG mechanism for
> logging as that will make the patch very complicated to avoid
> recursion in the trace routine.
>
> This feature starts dumping output, only in debug mode, which means
> ovs-appctl -t <module> vlog/set any:any:dbg should be used to enable
> this logging.
>
> Currently, only ovn-northd, ovn-controller, vswitchd are instrumented.

After trying this out in some scaling scenarios, it looks like
it could use some reworking - even when not actually logging
data, it's resulting in what look like significantly slower
performance times of the instrumented binaries.

If you can provide some data showing that the degradation is not
significant, then I'll remove my objection...

Ryan