[ovs-dev] [PATCH] python: Allow specifying the iterations in Idl.run

Terry Wilson twilson at redhat.com
Thu Mar 24 18:05:22 UTC 2016


On Wed, Mar 23, 2016 at 2:31 PM, Jason Kölker <jason at koelker.net> wrote:
> On Wed, Mar 23, 2016 at 6:29 PM, Ben Pfaff <blp at ovn.org> wrote:
>> On Sun, Mar 20, 2016 at 03:29:06AM +0000, Jason Kölker wrote:
>>> Callers to Idl.run() should be able to specify the number of iterations
>>> to run in one call. Instrumentation revealed that a call to Idl.run() with a
>>> can block up to a few seconds, with most of the time being spent parsing
>>> messages.
>>>
>>> Since parsing is a CPU only task eventlet and friends are unable to
>>> to provide concurrency. Specifying a configurable number of iterations
>>> will allow calling code to better manage concurrency.
>>>
>>> Signed-off-by: Jason Kölker <jason at koelker.net>
>>
>> A couple of seconds is really extreme.  I'm actually shocked to hear
>> that parsing 4 kB * 200 == 800 kB of data can take multiple seconds.
>
> I'm still working on tracking it all down. I *think* it has something to do
> with the socket stream buffering. But as we only see it from time to time
> on our controller and running just the Idl under eventlet (no other services)
> doesn't have the same behavior; its been slow getting a reproducible case.
>
>>  If it takes that long, then I'd suggest an approach different from this
>> patch.  First, reduce the iteration count from 50 to something smaller.
>> Second, figure out why JSON parsing is so slow, and fix it.
>
> Agreed. This was next on my list to start going through the JSON parser and
> instrumenting it to track it down, once I can get a good test to
> reproduce reliably.
>
>> stackoverflow points to some JSON push parsers:
>> http://stackoverflow.com/questions/444380/is-there-a-streaming-api-for-json
>
> Thanks! I'll look through the them and compare and see if there are some speed
> ups we can use. I briefly looked at what it would take to replace the
> parser with
> ijson. That might be the direction I try to go long term, but I'd
> rather not have an
> external dependency. I'll carry this patch internally for the time
> being until I can
> better understand whats really going on.

I remember looking into the OVS python JSON library a while back and
it was over 60x slower than most other JSON parsers, including the
in-tree C-based version it is essentially a copy of. The built-in
python JSON library doesn't fit the streaming use case that we need,
but py-yajl was mostly a good fit architecturally. I made a pull
request (https://github.com/rtyler/py-yajl/pull/38) to fix the support
that we would need, but never got a response on it. The other solution
I thought of at the time was just to do bindings to the C version
in-tree and see how fast it was.



More information about the dev mailing list