[ovs-dev] [PATCH] dpdk: Deprecate pdump support.
David Marchand
david.marchand at redhat.com
Thu Nov 14 16:41:18 UTC 2019
Hello Reshma,
Has pdump been tested (recently) with OVS?
On Mon, Nov 11, 2019 at 7:53 PM Ilya Maximets <i.maximets at ovn.org> wrote:
>
> The conventional way for packet dumping in OVS is to use ovs-tcpdump
> that works via traffic mirroring. DPDK pdump could probably be used
> for some lower level debugging, but it is not commonly used for
> various reasons.
>
> There are lots of limitations for using this functionality in practice.
> Most of them connected with running secondary pdump process and
> memory layout issues like requirement to disable ASLR in kernel.
> More details are available in DPDK guide:
> https://doc.dpdk.org/guides/prog_guide/multi_proc_support.html#multi-process-limitations
>
> Beside the functional limitations it's also hard to use this
> functionality correctly. User must be sure that OVS and pdump utility
> are running on different CPU cores, which is hard because non-PMD
> threads could float over available CPU cores. This or any other
> misconfiguration will likely lead to crash of the pdump utility
> or/and OVS.
>
> Another problem is that the user must actually have this special pdump
> utility in a system and it might be not available in distributions.
>
> This change disables pdump support by default introducing special
> configuration option '--enable-dpdk-pdump'. Deprecation warnings will
> be shown to users on configuration and in runtime.
>
> Claiming to completely remove this functionality from OVS in one
> of the next releases.
>
> Signed-off-by: Ilya Maximets <i.maximets at ovn.org>
> Acked-by: Aaron Conole <aconole at redhat.com>
- Recompiled from scratch, on OVS master (before this patch) with dpdk 18.11.2.
other_config : {dpdk-init="true", pmd-cpu-mask="0x00008002"}
2 physical ports, 2 vhost ports.
2019-11-14T14:13:09.596Z|00018|dpdk|INFO|Using DPDK 18.11.2
2019-11-14T14:13:09.596Z|00019|dpdk|INFO|DPDK Enabled - initializing...
2019-11-14T14:13:09.596Z|00020|dpdk|INFO|No vhost-sock-dir provided -
defaulting to //var/run/openvswitch
2019-11-14T14:13:09.596Z|00021|dpdk|INFO|IOMMU support for
vhost-user-client disabled.
2019-11-14T14:13:09.596Z|00022|dpdk|INFO|POSTCOPY support for
vhost-user-client disabled.
2019-11-14T14:13:09.596Z|00023|dpdk|INFO|Per port memory for DPDK
devices disabled.
2019-11-14T14:13:09.596Z|00024|dpdk|INFO|EAL ARGS: ovs-vswitchd
--socket-mem 1024 --socket-limit 1024 -l 0.
2019-11-14T14:13:09.600Z|00025|dpdk|INFO|EAL: Detected 28 lcore(s)
2019-11-14T14:13:09.600Z|00026|dpdk|INFO|EAL: Detected 1 NUMA nodes
2019-11-14T14:13:09.602Z|00027|dpdk|INFO|EAL: Multi-process socket
/var/run/openvswitch/dpdk/rte/mp_socket
2019-11-14T14:13:09.618Z|00028|dpdk|INFO|EAL: Probing VFIO support...
2019-11-14T14:13:09.618Z|00029|dpdk|INFO|EAL: VFIO support initialized
2019-11-14T14:13:14.612Z|00030|dpdk|INFO|EAL: PCI device 0000:01:00.0
on NUMA socket 0
2019-11-14T14:13:14.612Z|00031|dpdk|INFO|EAL: probe driver:
8086:10fb net_ixgbe
2019-11-14T14:13:14.613Z|00032|dpdk|INFO|EAL: using IOMMU type 1 (Type 1)
2019-11-14T14:13:14.744Z|00033|dpdk|INFO|EAL: Ignore mapping IO port bar(2)
2019-11-14T14:13:15.090Z|00034|dpdk|INFO|EAL: PCI device 0000:01:00.1
on NUMA socket 0
2019-11-14T14:13:15.090Z|00035|dpdk|INFO|EAL: probe driver:
8086:10fb net_ixgbe
2019-11-14T14:13:15.199Z|00036|dpdk|INFO|EAL: Ignore mapping IO port bar(2)
2019-11-14T14:13:15.530Z|00037|dpdk|INFO|EAL: PCI device 0000:07:00.0
on NUMA socket 0
2019-11-14T14:13:15.530Z|00038|dpdk|INFO|EAL: probe driver:
8086:1521 net_e1000_igb
2019-11-14T14:13:15.530Z|00039|dpdk|INFO|EAL: PCI device 0000:07:00.1
on NUMA socket 0
2019-11-14T14:13:15.530Z|00040|dpdk|INFO|EAL: probe driver:
8086:1521 net_e1000_igb
...
2019-11-14T14:13:15.802Z|00042|dpdk|INFO|DPDK pdump packet capture enabled
2019-11-14T14:13:15.803Z|00043|dpdk|INFO|DPDK Enabled - initialized
- Attached a gdb to ovs-vswitchd.
- Started pdump:
# sudo -u openvswitch XDG_RUNTIME_DIR=/var/run/openvswitch
./v18.11.2/app/dpdk-pdump -- --pdump
'port=0,queue=*,rx-dev=/tmp/pkts.pcap'
EAL: Detected 28 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket
/var/run/openvswitch/dpdk/rte/mp_socket_83791_549cdfd05e328e
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: using IOMMU type 1 (Type 1)
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:07:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:07:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 net_e1000_igb
Port 3 MAC: 02 70 63 61 70 00
- Sent one packet to the first physical port from my tgen
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f4840659700 (LWP 84336)]
bucket_dequeue_orphans (n_orphans=251, obj_table=0x14fe5af50,
bd=0x14fdad880) at
/root/dpdk/drivers/mempool/bucket/rte_mempool_bucket.c:190
190 objptr = bucket_stack_pop(bd->buckets[rte_lcore_id()]);
(gdb) bt
#0 bucket_dequeue_orphans (n_orphans=251, obj_table=0x14fe5af50,
bd=0x14fdad880) at
/root/dpdk/drivers/mempool/bucket/rte_mempool_bucket.c:190
#1 bucket_dequeue (mp=<optimized out>, obj_table=0x14fe5af50,
n=<optimized out>) at
/root/dpdk/drivers/mempool/bucket/rte_mempool_bucket.c:288
#2 0x00000000004eeeef in rte_mempool_ops_dequeue_bulk (n=251,
obj_table=0x14fe5af50, mp=0x14fe2dac0) at
/root/dpdk/v18.11.2/include/rte_mempool.h:657
#3 __mempool_generic_get (cache=0x14fe5af40, n=1,
obj_table=0x7f4840656fe0, mp=0x14fe2dac0) at
/root/dpdk/v18.11.2/include/rte_mempool.h:1363
#4 rte_mempool_generic_get (cache=0x14fe5af40, n=1,
obj_table=0x7f4840656fe0, mp=0x14fe2dac0) at
/root/dpdk/v18.11.2/include/rte_mempool.h:1426
#5 rte_mempool_get_bulk (n=1, obj_table=0x7f4840656fe0,
mp=0x14fe2dac0) at /root/dpdk/v18.11.2/include/rte_mempool.h:1459
#6 rte_mempool_get (obj_p=0x7f4840656fe0, mp=0x14fe2dac0) at
/root/dpdk/v18.11.2/include/rte_mempool.h:1485
#7 rte_mbuf_raw_alloc (mp=0x14fe2dac0) at
/root/dpdk/v18.11.2/include/rte_mbuf.h:1078
#8 rte_pktmbuf_alloc (mp=0x14fe2dac0) at
/root/dpdk/v18.11.2/include/rte_mbuf.h:1331
#9 pdump_pktmbuf_copy (mp=0x14fe2dac0, m=0x1509ea100) at
/root/dpdk/lib/librte_pdump/rte_pdump.c:99
#10 pdump_copy (pkts=<optimized out>, nb_pkts=<optimized out>,
user_params=<optimized out>) at
/root/dpdk/lib/librte_pdump/rte_pdump.c:151
#11 0x00000000004eff31 in pdump_rx (port=<optimized out>,
qidx=<optimized out>, pkts=<optimized out>, nb_pkts=<optimized out>,
max_pkts=<optimized out>, user_params=<optimized out>)
at /root/dpdk/lib/librte_pdump/rte_pdump.c:172
#12 0x00000000009f25fa in rte_eth_rx_burst (nb_pkts=32,
rx_pkts=0x7f4840657110, queue_id=0, port_id=0) at
/usr/local/include/dpdk/rte_ethdev.h:3888
#13 netdev_dpdk_rxq_recv (rxq=0x1501bd940, batch=0x7f4840657100,
qfill=0x0) at ../lib/netdev-dpdk.c:2287
#14 0x000000000093dab1 in netdev_rxq_recv (rx=<optimized out>,
batch=batch at entry=0x7f4840657100, qfill=<optimized out>) at
../lib/netdev.c:724
#15 0x0000000000911694 in dp_netdev_process_rxq_port
(pmd=pmd at entry=0x7f484065a010, rxq=0x2711ae0, port_no=3) at
../lib/dpif-netdev.c:4268
#16 0x0000000000911af9 in pmd_thread_main (f_=<optimized out>) at
../lib/dpif-netdev.c:5526
#17 0x000000000099355d in ovsthread_wrapper (aux_=<optimized out>) at
../lib/ovs-thread.c:383
#18 0x00007f4875402dd5 in start_thread (arg=0x7f4840659700) at
pthread_create.c:307
#19 0x00007f4874920ead in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) p bd->buckets[rte_lcore_id()]
$2 = (struct bucket_stack *) 0x0
We can see a "bucket" mempool.
ovs is using "ring_mp_mc" default mempool.
So maybe something unaligned here.
On OVS side, we can see the pdump code expects that this mempool has
some resource initialised for a lcore 15.
Not sure who is responsible for doing this part, primary ? secondary ?
The pdump application in 18.11.2 has a hardwired core mask as 0x1.
If I shoot this (backporting a commit that did this) and start pdump
on lcores 0, 1 and 15 (to mimic OVS running on master core 0 + lcore 1
and 15), I get a segfault a little bit later, but again with an
uninitialised resource on OVS side.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff795324700 (LWP 176548)]
bucket_dequeue_orphans (n_orphans=251, obj_table=0x14fe30bd0,
bd=0x14fdad880) at
/root/dpdk/drivers/mempool/bucket/rte_mempool_bucket.c:194
194 rc = rte_ring_dequeue(bd->shared_bucket_ring,
(gdb) bt
#0 bucket_dequeue_orphans (n_orphans=251, obj_table=0x14fe30bd0,
bd=0x14fdad880) at
/root/dpdk/drivers/mempool/bucket/rte_mempool_bucket.c:194
#1 bucket_dequeue (mp=<optimized out>, obj_table=0x14fe30bd0,
n=<optimized out>) at
/root/dpdk/drivers/mempool/bucket/rte_mempool_bucket.c:288
#2 0x00000000004eeeef in rte_mempool_ops_dequeue_bulk (n=251,
obj_table=0x14fe30bd0, mp=0x14fe2dac0) at
/root/dpdk/v18.11.2/include/rte_mempool.h:657
#3 __mempool_generic_get (cache=0x14fe30bc0, n=1,
obj_table=0x7ff795321fe0, mp=0x14fe2dac0) at
/root/dpdk/v18.11.2/include/rte_mempool.h:1363
#4 rte_mempool_generic_get (cache=0x14fe30bc0, n=1,
obj_table=0x7ff795321fe0, mp=0x14fe2dac0) at
/root/dpdk/v18.11.2/include/rte_mempool.h:1426
#5 rte_mempool_get_bulk (n=1, obj_table=0x7ff795321fe0,
mp=0x14fe2dac0) at /root/dpdk/v18.11.2/include/rte_mempool.h:1459
#6 rte_mempool_get (obj_p=0x7ff795321fe0, mp=0x14fe2dac0) at
/root/dpdk/v18.11.2/include/rte_mempool.h:1485
#7 rte_mbuf_raw_alloc (mp=0x14fe2dac0) at
/root/dpdk/v18.11.2/include/rte_mbuf.h:1078
#8 rte_pktmbuf_alloc (mp=0x14fe2dac0) at
/root/dpdk/v18.11.2/include/rte_mbuf.h:1331
#9 pdump_pktmbuf_copy (mp=0x14fe2dac0, m=0x1509ea100) at
/root/dpdk/lib/librte_pdump/rte_pdump.c:99
#10 pdump_copy (pkts=<optimized out>, nb_pkts=<optimized out>,
user_params=<optimized out>) at
/root/dpdk/lib/librte_pdump/rte_pdump.c:151
#11 0x00000000004eff31 in pdump_rx (port=<optimized out>,
qidx=<optimized out>, pkts=<optimized out>, nb_pkts=<optimized out>,
max_pkts=<optimized out>, user_params=<optimized out>)
at /root/dpdk/lib/librte_pdump/rte_pdump.c:172
#12 0x00000000009f25fa in rte_eth_rx_burst (nb_pkts=32,
rx_pkts=0x7ff795322110, queue_id=0, port_id=0) at
/usr/local/include/dpdk/rte_ethdev.h:3888
#13 netdev_dpdk_rxq_recv (rxq=0x1501bd940, batch=0x7ff795322100,
qfill=0x0) at ../lib/netdev-dpdk.c:2287
#14 0x000000000093dab1 in netdev_rxq_recv (rx=<optimized out>,
batch=batch at entry=0x7ff795322100, qfill=<optimized out>) at
../lib/netdev.c:724
#15 0x0000000000911694 in dp_netdev_process_rxq_port
(pmd=pmd at entry=0x7ff795325010, rxq=0x2376b30, port_no=3) at
../lib/dpif-netdev.c:4268
#16 0x0000000000911af9 in pmd_thread_main (f_=<optimized out>) at
../lib/dpif-netdev.c:5526
#17 0x000000000099355d in ovsthread_wrapper (aux_=<optimized out>) at
../lib/ovs-thread.c:383
#18 0x00007ff7cd81ddd5 in start_thread (arg=0x7ff795324700) at
pthread_create.c:307
#19 0x00007ff7ccd3bead in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) p bd->shared_bucket_ring
$1 = (struct rte_ring *) 0x0
--
David Marchand
More information about the dev
mailing list