<div dir="ltr">Hello,<br><br>Daniel and Lucas were doing some scale testing with OpenShift<br>deployed on top of OpenStack with OVN (OVS/OVN 2.9) in last<br>couple of weeks. They noticed very high CPU usage during the test runs.<br><br>Below are some of the observations we found<br> -  ovn-controller is taking a lot of time in processing the logical flows.<br> -  The setup had close to 37000 logical flows and the function lflow_run()<br>    was taking ~10 seconds to complete.<br> -  Since main loop took &gt; 5 seconds to complete, poll_block() was returning<br>    immediately to handle the probe inactivity for the openflow unix socket connection.<br>    This caused continuous 100% CPU usage. After applying this patch - <a href="https://patchwork.ozlabs.org/patch/1125106/">https://patchwork.ozlabs.org/patch/1125106/</a><br>    and configuring higher inactivity probe value, the CPU usage settled down.<br><br>We wanted to understand why lflow_run() is taking so much time. So<br>we did some dirty hacks and did the following in the function consider_logical_flow() in lflow.c<br>and measured the time taken for the lflow_run()<br><br>  - returned just before calling expr_parse_string() - The time taken for lflow_run was 40 msec<br>  - returned just after calling expr_parse_string()  - The time taken was ~425 msec<br>  - returned just after expr_annotate()              - The time taken was ~1800 msec<br>  - returned just after expr_normalize()             - The time taken was ~4900 msec<br>  - returned just after expr_to_matches()            - The time taken was ~8900 msec<br><br>Looks like these expr_*() functions are doing lots of mallocs and string operations.<br><br>Below is the output of &quot;perf top -t `pidof ovn-controller` -K&quot;<br><br>*******<br>16.30%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] _int_malloc                                                                                                                                                                                                <br>  11.74%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] _int_free                                                                                                                                                                                                  <br>   6.38%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] malloc                                                                                                                                                                                                     <br>   3.51%  ovn-controller      [.] expr_normalize_and                                                                                                                                                                                         <br>   3.41%  ovn-controller      [.] match_hash                                                                                                                                                                                                 <br>   3.40%  ovn-controller      [.] flow_wildcards_hash                                                                                                                                                                                        <br>   3.39%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] __memcpy_ssse3_back                                                                                                                                                                                        <br>   2.81%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] malloc_consolidate                                                                                                                                                                                         <br>   2.64%  ovn-controller      [.] is_all_zeros                                                                                                                                                                                               <br>   2.49%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] __strcmp_sse42                                                                                                                                                                                             <br>   2.30%  ovn-controller      [.] add_conjunction                                                                                                                                                                                            <br>   2.17%  ovn-controller      [.] bitwise_rscan                                                                                                                                                                                              <br>   1.76%  ovn-controller      [.] expr_destroy                                                                                                                                                                                               <br>   1.61%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] __libc_calloc                                                                                                                                                                                              <br>   1.57%  ovn-controller      [.] bitwise_copy                                                                                                                                                                                               <br>   1.54%  ovn-controller      [.] ovsdb_idl_read                                                                                                                                                                                             <br>   1.46%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] __memcmp_sse4_1                                                                                                                                                                                            <br>   1.43%  ovn-controller      [.] ovn_flow_lookup                                                                                                                                                                                            <br>   1.39%  ovn-controller      [.] consider_logical_flow.isra.2                                                                                                                                                                               <br>   1.26%  ovn-controller      [.] ovsdb_datum_compare_3way                                                                                                                                                                                   <br>   1.24%  ovn-controller      [.] mf_subvalue_intersect                                                                                                                                                                                      <br>   1.17%  ovn-controller      [.] expr_match_new                                                                                                                                                                                             <br>   1.17%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] free                                                                                                                                                                                                       <br>   1.12%  ovn-controller      [.] resize                                                                                                                                                                                                     <br>   1.02%  ovn-controller      [.] expr_matches_destroy                                                                                                                                                                                       <br>   1.01%  ovn-controller      [.] ovsdb_idl_index_generic_comparer                                                                                                                                                                           <br>   0.91%  ovn-controller      [.] ofctrl_add_flow                                                                                                                                                                                            <br>   0.87%  ovn-controller      [.] expr_clone                                                                                                                                                                                                 <br>   0.87%  ovn-controller      [.] expr_match_add                                                                                                                                                                                             <br>   0.84%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] vfprintf                   <br>   0.80%  ovn-controller      [.] match_init_catchall                                                                                                                                                                                        <br>   0.72%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] msort_with_tmp.part.0                                                                                                                                                                                      <br>   0.71%  ovn-controller      [.] hash_bytes                                                                                                                                                                                                 <br>   0.64%  ovn-controller      [.] skiplist_forward_to_                                                                                                                                                                                       <br>   0.62%  ovn-controller      [.] expr_simplify                                                                                                                                                                                              <br>   0.54%  ovn-controller      [.] match_set_metadata                                                                                                                                                                                         <br>   0.52%  ovn-controller      [.] ovsdb_atom_compare_3way                                                                                                                                                                                    <br>   0.49%  ovn-controller      [.] parse_constant                                                                                                                                                                                             <br>   0.49%  ovn-controller      [.] expr_to_matches                                                                                                                                                                                            <br>   0.48%  ovn-controller      [.] ofctrl_put                                                                                                                                                                                                 <br>   0.46%  ovn-controller      [.] crush_and_numeric                                                                                                                                                                                          <br>   0.41%  ovn-controller      [.] constrain_match                                                                                                                                                                                            <br>   0.40%  ovn-controller      [.] lex_token_parse                                                                                                                                                                                            <br>   0.35%  ovn-controller      [.] expr_is_cmp                                                                                                                                                                                                <br>   0.30%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] __memcpy_sse2                                                                                                                                                                                              <br>   0.29%  ovn-controller      [.] mf_get_value                                                                                                                                                                                               <br>   0.29%  ovn-controller      [.] lex_parse_integer__                                                                                                                                                                                        <br>   0.28%  ovn-controller      [.] expr_normalize_or                                                                                                                                                                                          <br>   0.27%  ovn-controller      [.] xmalloc                                                                                                                                                                                                    <br>   0.26%  ovn-controller      [.] flow_wildcards_init_catchall                                                                                                                                                                               <br>   0.25%  ovn-controller      [.] expr_annotate__                                                                                                                                                                                            <br>   0.24%  ovn-controller      [.] mf_set                                                                                                                                                                                                     <br>   0.23%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] _IO_default_xsputn                                                                                                                                                                                         <br>   0.23%  ovn-controller      [.] expr_combine                                                                                                                                                                                               <br>   0.23%  <a href="http://libc-2.17.so">libc-2.17.so</a>        [.] __strspn_sse2<br>*******<br><br>I think we may have to optimize the expr_*() functions.<br>I don&#39;t have any solid idea at this point of time. Dumitru and I are planning to<br>explore more on this in the coming weeks.<br><br>Any comments/suggestions/thoughts on this ?<br><br>Thanks<br>Numan<br><br><br></div>