<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello,<br>
<br>
I'm seeing a consistent crash when trying to run an iperf test using qede interface and DPDK. Running one iperf test on its own works fine but as soon as I attempt to run two in parallel through the same interface the crash occurs.<br>
<br>
OVS configuration is:<br>
<br>
ovs-vsctl set Open_vSwitch . other_config:dpdk-socket-mem="4096,0"
<div>ovs-vsctl --no-wait set Open_vSwitch . other_config:vhost-iommu-support=true</div>
<div>ovs-vsctl --timeout 10 set Open_vSwitch . other_config:max-idle=30000</div>
<div>ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x1e</div>
<div><br>
</div>
<div>ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk \</div>
<div> options:dpdk-devargs=0000:45:00.0 \</div>
<div> options:n_rxq=1 other_config:pmd-rxq-affinity="0:1" \</div>
<div> options:flow-ctrl-autoneg=true \</div>
<div> ofport=1</div>
<div><br>
</div>
<div>ovs-vsctl -- set Interface br1 mtu_request=1500</div>
<div>ovs-vsctl add-port br1 dpdk-p1 -- set Interface dpdk-p1 type=dpdk \</div>
<div> options:dpdk-devargs=0000:45:00.1 \</div>
<div> options:n_rxq=3 other_config:pmd-rxq-affinity="0:2,1:3,2:4" options:n_rxq_desc=2048 options:n_txq_desc=2048 \</div>
<div> options:flow-ctrl-autoneg=true \</div>
<div> mtu_request=1504 ofport=1</div>
<div><br>
</div>
<div>ip a add 192.168.190.77/25 dev br0</div>
<div>ip a add 192.168.190.200/25 dev br1</div>
<div><br>
</div>
<div>rm -f /dev/hugepages/*</div>
<div><br>
</div>
<div>ip link set br0 up</div>
<div>ip link set br1 up</div>
<div><br>
</div>
<div>ip route add default via 192.168.190.1 dev br0</div>
<div><br>
</div>
<div>ovs-vsctl set port dpdk-p1 qos=@qos -- \</div>
<div> --id=@qos create qos type=trtcm-policer other-config:cir=250000000 other-config:cbs=30080 other_config:eir=12500000 other-config:ebs=30080 queues:1=@queue1 queues:2=@queue2 queues:3=@queue3 -- \</div>
<div> --id=@queue1 create queue other-config:cir=220000000 other-config:eir=10000000 other-config:cbs=30080 other-config:ebs=30080 -- \</div>
<div> --id=@queue2 create queue other-config:cir=20000000 other-config:eir=220000000 other-config:cbs=30080 other-config:ebs=30080 -- \</div>
<div> --id=@queue3 create queue other-config:cir=10000000 other-config:eir=10000000 other-config:cbs=30080 other-config:ebs=30080</div>
<div><br>
</div>
<div>ovs-ofctl del-flows br1</div>
<div><br>
</div>
<div>ovs-ofctl add-flow br1 'table=0,priority=1,ip,ip_dst=0.0.0.201/0.0.0.255,actions=set_queue:0,resubmit(,1)'</div>
<div>ovs-ofctl add-flow br1 'table=0,priority=1,ip,ip_dst=0.0.0.202/0.0.0.255,actions=set_queue:1,resubmit(,1)'</div>
<div>ovs-ofctl add-flow br1 'table=0,priority=1,ip,ip_dst=0.0.0.203/0.0.0.255,actions=set_queue:2,resubmit(,1)'</div>
<div>ovs-ofctl add-flow br1 'table=0,priority=0,ip,actions=resubmit(,1)'</div>
<div><br>
</div>
<div>ovs-ofctl add-flow br1 'table=0,priority=0,arp,actions=all'</div>
<div><br>
</div>
<div>ovs-ofctl add-flow br1 'table=0,mpls,actions=pop_mpls:0x0800,local'</div>
<div><br>
</div>
ovs-ofctl add-flow br1 'table=1,ip,actions=move:ip_src[0..7]->reg0[0..7],push_mpls:0x8847,move:reg0[0..7]->mpls_label[0..7],set_field:2->mpls_tc,set_field:255->mpls_ttl,1'<br>
<br>
The crash produces the following back-trace:<br>
<br>
Process 2800 (ovs-vswitchd) of user 0 dumped core.
<div> </div>
<div> Stack trace of thread 2800:</div>
<div> #0 0x00007f2f94f3470f raise (libc.so.6)</div>
<div> #1 0x00007f2f94f1eb25 abort (libc.so.6)</div>
<div> #2 0x00007f2f94f77897 __libc_message (libc.so.6)</div>
<div> #3 0x00007f2f94f7dfdc malloc_printerr (libc.so.6)</div>
<div> #4 0x00007f2f94f7fe44 _int_free (libc.so.6)</div>
<div> #5 0x000000000050881a dp_packet_delete (ovs-vswitchd)</div>
<div> #6 0x000000000050883e dp_packet_delete_batch (ovs-vswitchd)</div>
<div> #7 0x000000000050a114 dp_packet_delete_batch (ovs-vswitchd)</div>
<div> #8 0x000000000047d243 netdev_send (ovs-vswitchd)</div>
<div> #9 0x0000000000458910 dp_netdev_pmd_flush_output_on_port (ovs-vswitchd)</div>
<div> #10 0x0000000000458a1a dp_netdev_pmd_flush_output_packets (ovs-vswitchd)</div>
<div> #11 0x000000000045d705 dp_netdev_process_rxq_port (ovs-vswitchd)</div>
<div> #12 0x000000000045deb5 dpif_netdev_run (ovs-vswitchd)</div>
<div> #13 0x0000000000429e10 type_run (ovs-vswitchd)</div>
<div> #14 0x0000000000418b10 ofproto_type_run (ovs-vswitchd)</div>
<div> #15 0x000000000040c101 bridge_run__ (ovs-vswitchd)</div>
<div> #16 0x0000000000410f3d bridge_run (ovs-vswitchd)</div>
<div> #17 0x0000000000409802 main (ovs-vswitchd)</div>
<div> #18 0x00007f2f94f206a3 __libc_start_main (libc.so.6)</div>
<div> #19 0x000000000040a2de _start (ovs-vswitchd)</div>
<div> </div>
<div> Stack trace of thread 2959:</div>
<div> #0 0x00007f2f94fedf21 __poll (libc.so.6)</div>
<div> #1 0x00000000004dd2e9 time_poll (ovs-vswitchd)</div>
<div> #2 0x00000000004cdb35 poll_block (ovs-vswitchd)</div>
<div> #3 0x000000000052e9b3 clean_thread_main (ovs-vswitchd)</div>
<div> #4 0x00000000004beba0 ovsthread_wrapper (ovs-vswitchd)</div>
<div> #5 0x00007f2f95d9f2de start_thread (libpthread.so.0)</div>
<div> #6 0x00007f2f94ff8e83 __clone (libc.so.6)</div>
<div> </div>
<div> Stack trace of thread 3069:</div>
<div> #0 0x000000000045d981 pmd_thread_main (ovs-vswitchd)</div>
<div> #1 0x00000000004beba0 ovsthread_wrapper (ovs-vswitchd)</div>
<div> #2 0x00007f2f95d9f2de start_thread (libpthread.so.0)</div>
<div> #3 0x00007f2f94ff8e83 __clone (libc.so.6)</div>
<div> </div>
<div> Stack trace of thread 3071:</div>
<div> #0 0x000000000045d5d7 rte_rdtsc (ovs-vswitchd)</div>
<div> #1 0x000000000045da10 pmd_thread_main (ovs-vswitchd)</div>
<div> #2 0x00000000004beba0 ovsthread_wrapper (ovs-vswitchd)</div>
<div> #3 0x00007f2f95d9f2de start_thread (libpthread.so.0)</div>
<div> #4 0x00007f2f94ff8e83 __clone (libc.so.6)</div>
<div> </div>
<div> Stack trace of thread 3072:</div>
<div> #0 0x00007ffd370139c9 n/a (linux-vdso.so.1)</div>
<div> #1 0x00007ffd37013bcc __vdso_clock_gettime (linux-vdso.so.1)</div>
<div> #2 0x00007f2f95006ea2 __clock_gettime (libc.so.6)</div>
<div> #3 0x00000000004dceba xclock_gettime (ovs-vswitchd)</div>
<div> #4 0x00000000004dcefa time_timespec__ (ovs-vswitchd)</div>
<div> #5 0x00000000004dd421 time_usec__ (ovs-vswitchd)</div>
<div> #6 0x000000000045d95f pmd_thread_ctx_time_update (ovs-vswitchd)</div>
<div> #7 0x00000000004beba0 ovsthread_wrapper (ovs-vswitchd)</div>
<div> #8 0x00007f2f95d9f2de start_thread (libpthread.so.0)</div>
<div> #9 0x00007f2f94ff8e83 __clone (libc.so.6)</div>
<div> </div>
<div> Stack trace of thread 3070:</div>
<div> #0 0x00000000004dced3 time_timespec__ (ovs-vswitchd)</div>
<div> #1 0x00000000004dd421 time_usec__ (ovs-vswitchd)</div>
<div> #2 0x000000000045d95f pmd_thread_ctx_time_update (ovs-vswitchd)</div>
<div> #3 0x00000000004beba0 ovsthread_wrapper (ovs-vswitchd)</div>
<div> #4 0x00007f2f95d9f2de start_thread (libpthread.so.0)</div>
#5 0x00007f2f94ff8e83 __clone (libc.so.6)<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div> Stack trace of thread 2803:</div>
<div> #0 0x00007f2f94fc5238 __nanosleep (libc.so.6)</div>
<div> #1 0x00007f2f94fc513e sleep (libc.so.6)</div>
<div> #2 0x00000000004e431e xsleep (ovs-vswitchd)</div>
<div> #3 0x0000000000506ef7 dpdk_watchdog (ovs-vswitchd)</div>
<div> #4 0x00000000004beba0 ovsthread_wrapper (ovs-vswitchd)</div>
<div> #5 0x00007f2f95d9f2de start_thread (libpthread.so.0)</div>
<div> #6 0x00007f2f94ff8e83 __clone (libc.so.6)</div>
<div> </div>
<div> Stack trace of thread 2801:</div>
<div> #0 0x00007f2f94ff91b7 epoll_wait (libc.so.6)</div>
<div> #1 0x00007f2f97560a44 eal_intr_thread_main (librte_eal.so.20.0)</div>
<div> #2 0x00007f2f95d9f2de start_thread (libpthread.so.0)</div>
<div> #3 0x00007f2f94ff8e83 __clone (libc.so.6)</div>
<div> </div>
<div> Stack trace of thread 2804:</div>
<div> #0 0x00007f2f94fedf21 __poll (libc.so.6)</div>
<div> #1 0x00000000004dd2e9 time_poll (ovs-vswitchd)</div>
<div> #2 0x00000000004cdb35 poll_block (ovs-vswitchd)</div>
<div> #3 0x00000000004bd334 ovsrcu_postpone_thread (ovs-vswitchd)</div>
<div> #4 0x00000000004beba0 ovsthread_wrapper (ovs-vswitchd)</div>
<div> #5 0x00007f2f95d9f2de start_thread (libpthread.so.0)</div>
<div> #6 0x00007f2f94ff8e83 __clone (libc.so.6)</div>
<div> </div>
<div> Stack trace of thread 2967:</div>
<div> #0 0x00007f2f94fedf21 __poll (libc.so.6)</div>
<div> #1 0x00000000004dd2e9 time_poll (ovs-vswitchd)</div>
<div> #2 0x00000000004cdb35 poll_block (ovs-vswitchd)</div>
<div> #3 0x000000000043589b udpif_upcall_handler (ovs-vswitchd)</div>
<div> #4 0x00000000004beba0 ovsthread_wrapper (ovs-vswitchd)</div>
<div> #5 0x00007f2f95d9f2de start_thread (libpthread.so.0)</div>
<div> #6 0x00007f2f94ff8e83 __clone (libc.so.6)</div>
<div> </div>
<div> Stack trace of thread 2960:</div>
<div> #0 0x00007f2f94fedf21 __poll (libc.so.6)</div>
<div> #1 0x00000000004dd2e9 time_poll (ovs-vswitchd)</div>
<div> #2 0x00000000004cdb35 poll_block (ovs-vswitchd)</div>
<div> #3 0x0000000000469dce ipf_clean_thread_main (ovs-vswitchd)</div>
<div> #4 0x00000000004beba0 ovsthread_wrapper (ovs-vswitchd)</div>
<div> #5 0x00007f2f95d9f2de start_thread (libpthread.so.0)</div>
<div> #6 0x00007f2f94ff8e83 __clone (libc.so.6)</div>
<div> </div>
<div> Stack trace of thread 2970:</div>
<div> #0 0x00007f2f94fedf21 __poll (libc.so.6)</div>
<div> #1 0x00000000004dd2e9 time_poll (ovs-vswitchd)</div>
<div> #2 0x00000000004cdb35 poll_block (ovs-vswitchd)</div>
<div> #3 0x000000000043589b udpif_upcall_handler (ovs-vswitchd)</div>
<div> #4 0x00000000004beba0 ovsthread_wrapper (ovs-vswitchd)</div>
<div> #5 0x00007f2f95d9f2de start_thread (libpthread.so.0)</div>
<div> #6 0x00007f2f94ff8e83 __clone (libc.so.6)</div>
<div> </div>
<div> Stack trace of thread 2976:</div>
<div> #0 0x00007f2f94fedf21 __poll (libc.so.6)</div>
<div> #1 0x00000000004dd2e9 time_poll (ovs-vswitchd)</div>
<div> #2 0x00000000004cdb35 poll_block (ovs-vswitchd)</div>
<div> #3 0x000000000043589b udpif_upcall_handler (ovs-vswitchd)</div>
<div> #4 0x00000000004beba0 ovsthread_wrapper (ovs-vswitchd)</div>
<div> #5 0x00007f2f95d9f2de start_thread (libpthread.so.0)</div>
#6 0x00007f2f94ff8e83 __clone (libc.so.6)<br>
<br>
<br>
Any idea as to what I might be doing wrong? Or is this a bug?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Bill Kozak<br>
</div>
</body>
</html>