<div dir="ltr">Happy new year everyone, I believe the assert is benign and open the below issue for tracking:<div><br></div><div><a href="https://github.com/openvswitch/ovs-issues/issues/199">https://github.com/openvswitch/ovs-issues/issues/199</a><br></div><div><br></div><div>Thanks,</div><div>Eric</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Dec 24, 2020 at 9:38 AM Eric Li <<a href="mailto:erictheone@gmail.com">erictheone@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi All,<div><br></div><div>My first post, so please be gentle on me :)</div><div><br></div><div>We have an app that wants to add a lot of openflow rules quickly. To speed things up, the app is creating multiple threads to do its logic and then call into ovs library code directly. I am getting the below assert about 50% of time on fast machines. Need help to understand what that assert means or suggest a better approach.</div><div><br></div><div>Questions in my mind:</div><div><br></div><div>1. I am thinking about using "bundle" to push in more openflow rules in one shot, what is a good number of rules to bundle together? 10, 100, 1000?</div><div>2. I tried to use a shared vonn in multiple threads but hitting other weird openflow errors, I don't think that's the right approach right?</div><div><br></div><div>Thanks a lot!</div><div>Eric</div><div><br></div><div>poll-loop.c: line 111:</div><div><br></div><div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">static void</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">poll_create_node(int fd, HANDLE wevent, short int events, const char *where)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">{</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>struct poll_loop *loop = poll_loop();</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>struct poll_node *node;</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>COVERAGE_INC(poll_create_node);</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>/* Both 'fd' and 'wevent' cannot be set. */</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>ovs_assert(!fd != !wevent); <-- !!!assert hit here because wevent=0 and fd=0!!!</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue";min-height:14px"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>/* Check for duplicate.<span> </span>If found, "or" the events. */</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>node = find_poll_node(loop, fd, wevent);</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>if (node) {</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>node->pollfd.events |= events;</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>} else {</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>node = xzalloc(sizeof *node);</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>hmap_insert(&loop->poll_nodes, &node->hmap_node,</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>hash_2words(fd, (uint32_t)wevent));</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>node->pollfd.fd = fd;</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>node->pollfd.events = events;</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#ifdef _WIN32</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>if (!wevent) {</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>wevent = CreateEvent(NULL, FALSE, FALSE, NULL);</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>}</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#endif</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>node->wevent = wevent;</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>node->where = where;</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>}</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">}</p><div><br></div><div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">(gdb) bt</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#0<span> </span>__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#1<span> </span>0x00007ffff61c18b1 in __GI_abort () at abort.c:79</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#2<span> </span>0x00007ffff6ea1ade in ovs_abort_valist (err_no=err_no@entry=0, format=format@entry=0x7ffff6f21bd0 "%s: assertion %s failed in %s()",<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>args=args@entry=0x7fffeeffb890) at lib/util.c:419</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#3<span> </span>0x00007ffff6ea9de0 in vlog_abort_valist (module_=<optimized out>, message=0x7ffff6f21bd0 "%s: assertion %s failed in %s()", args=args@entry=0x7fffeeffb890)</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>at lib/vlog.c:1227</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#4<span> </span>0x00007ffff6ea9e74 in vlog_abort (module=module@entry=0x7ffff71b6960 <this_module>, message=message@entry=0x7ffff6f21bd0 "%s: assertion %s failed in %s()")</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>at lib/vlog.c:1241</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#5<span> </span>0x00007ffff6ea17bc in ovs_assert_failure (where=where@entry=0x7ffff6f1d73f "lib/poll-loop.c:111",<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>function=function@entry=0x7ffff6f1d7a0 <__func__.7424> "poll_create_node", condition=condition@entry=0x7ffff6f1d730 "!fd != !wevent") at lib/util.c:86</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#6<span> </span>0x00007ffff6e826b6 in poll_create_node (wevent=0, where=0x7ffff6f023a4 "lib/fatal-signal.c:224", events=1, fd=0) at lib/poll-loop.c:111</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#7<span> </span>poll_fd_wait_at (fd=0, events=events@entry=1, where=where@entry=0x7ffff6f023a4 "lib/fatal-signal.c:224") at lib/poll-loop.c:149</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#8<span> </span>0x00007ffff6de6684 in fatal_signal_wait () at lib/fatal-signal.c:224</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#9<span> </span>0x00007ffff6e82777 in poll_block () at lib/poll-loop.c:331</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#10 0x00007ffff6ea66eb in vconn_connect_block (vconn=0x7fffe0001bf0, timeout=<optimized out>) at lib/vconn.c:722</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#11 0x00005555558320c9 in ovs_control::OVS_Control::open_vconn (this=0x555555c32de8 <ovs_control::OVS_Control::get_instance()::instance>,<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>name=0x5555559869c8 "br-tun", vconnp=0x7fffeeffbbb0) at /root/aca-dev/src/ovs/ovs_control.cpp:642</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#12 0x00005555558317e7 in ovs_control::OVS_Control::open_vconn_for_flow_mod (this=0x555555c32de8 <ovs_control::OVS_Control::get_instance()::instance>,<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>remote=0x5555559869c8 "br-tun", vconnp=0x7fffeeffbbb0,<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>usable_protocols=(OFPUTIL_P_OF10_NXM_TID | OFPUTIL_P_OF12_OXM | OFPUTIL_P_OF13_OXM | OFPUTIL_P_OF14_OXM | OFPUTIL_P_OF15_OXM))</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>at /root/aca-dev/src/ovs/ovs_control.cpp:501</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#13 0x000055555583157a in ovs_control::OVS_Control::flow_mod__ (this=0x555555c32de8 <ovs_control::OVS_Control::get_instance()::instance>,<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>remote=0x5555559869c8 "br-tun", fms=0x7fffeeffbc40, n_fms=1,<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>usable_protocols=(OFPUTIL_P_OF10_NXM_TID | OFPUTIL_P_OF12_OXM | OFPUTIL_P_OF13_OXM | OFPUTIL_P_OF14_OXM | OFPUTIL_P_OF15_OXM))</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>at /root/aca-dev/src/ovs/ovs_control.cpp:442</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#14 0x0000555555831365 in ovs_control::OVS_Control::flow_mod (this=0x555555c32de8 <ovs_control::OVS_Control::get_instance()::instance>,<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>bridge=0x5555559869c8 "br-tun",<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>flow=0x7fffe0001c80 "table=20,priority=50,dl_vlan=1,dl_dst:fa:16:3e:d7:f2:6c,actions=strip_vlan,load:20->NXM_NX_TUN_ID[],set_field:123.0.0.4->tun_dst,output:100", command=0) at /root/aca-dev/src/ovs/ovs_control.cpp:410</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">#15 0x00005555558310dd in ovs_control::OVS_Control::add_flow (this=0x555555c32de8 <ovs_control::OVS_Control::get_instance()::instance>,<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>bridge=0x5555559869c8 "br-tun",<span> </span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue""><span> </span>flow=0x7fffe0001c80 "table=20,priority=50,dl_vlan=1,dl_dst:fa:16:3e:d7:f2:6c,actions=strip_vlan,load:20->NXM_NX_TUN_ID[],set_field:123.0.0.4->tun_dst,output:100") at /root/aca-dev/src/ovs/ovs_control.cpp:375</p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Helvetica Neue"">...</p></div></div></div>
</blockquote></div>