[ovs-discuss] handler threads
Chandran, Sugesh
sugesh.chandran at intel.com
Thu Aug 18 12:15:51 UTC 2016
Thank you Joe for the help!
I am looking at how handler threads handling upcalls in OVS-DPDK data path.
From the GDB log, the upcall processing is happening in the PMD thread itself, see the logs below.
---------------------------------
Breakpoint 4, upcall_receive (upcall=0x7f9883fec2d0, backer=0x12e5000, packet=0x7f961357eb80, type=DPIF_UC_MISS, userdata=0x0,
flow=0x7f9883fecfa0, mru=0, ufid=0x7f9883fece80, pmd_id=3) at ofproto/ofproto-dpif-upcall.c:1017
1017 error = xlate_lookup(backer, flow, &upcall->ofproto, &upcall->ipfix,
(gdb) thread
[Current thread is 2 (Thread 0x7f9883fef700 (LWP 134776))]
(gdb) bt
#0 upcall_receive (upcall=0x7f9883fec2d0, backer=0x12e5000, packet=0x7f961357eb80, type=DPIF_UC_MISS, userdata=0x0, flow=0x7f9883fecfa0, mru=0,
ufid=0x7f9883fece80, pmd_id=3) at ofproto/ofproto-dpif-upcall.c:1017
#1 0x00000000005d9d9e in upcall_cb (packet=0x7f961357eb80, flow=0x7f9883fecfa0, ufid=0x7f9883fece80, pmd_id=3, type=DPIF_UC_MISS, userdata=0x0,
actions=0x7f9883fedd20, wc=0x7f9883fed1e8, put_actions=0x7f9883fedd60, aux=0x12e79e0) at ofproto/ofproto-dpif-upcall.c:1152
#2 0x000000000061bb77 in dp_netdev_upcall (pmd=0x1792570, packet_=0x7f961357eb80, flow=0x7f9883fecfa0, wc=0x7f9883fed1e8, ufid=0x7f9883fece80,
type=DPIF_UC_MISS, userdata=0x0, actions=0x7f9883fedd20, put_actions=0x7f9883fedd60) at lib/dpif-netdev.c:3834
#3 0x000000000061c2b6 in handle_packet_upcall (pmd=0x1792570, packet=0x7f961357eb80, key=0x7f9883fee430, actions=0x7f9883fedd20,
put_actions=0x7f9883fedd60, lost_cnt=0x7f9883fedcb4, now=95247337) at lib/dpif-netdev.c:4048
#4 0x000000000061c89f in fast_path_processing (pmd=0x1792570, packets_=0x7f9883fee950, keys=0x7f9883fee430, batches=0x7f9883fee1f0,
n_batches=0x7f9883fee860, in_port=2, now=95247337) at lib/dpif-netdev.c:4160
#5 0x000000000061cdc9 in dp_netdev_input__ (pmd=0x1792570, packets=0x7f9883fee950, md_is_valid=false, port_no=2) at lib/dpif-netdev.c:4229
#6 0x000000000061cea0 in dp_netdev_input (pmd=0x1792570, packets=0x7f9883fee950, port_no=2) at lib/dpif-netdev.c:4246
#7 0x00000000006190de in dp_netdev_process_rxq_port (pmd=0x1792570, port=0x1384540, rxq=0x7f9612a38980) at lib/dpif-netdev.c:2856
#8 0x0000000000619cd2 in pmd_thread_main (f_=0x1792570) at lib/dpif-netdev.c:3115
#9 0x00000000006b784c in ovsthread_wrapper (aux_=0x136fa60) at lib/ovs-thread.c:342
#10 0x00007f98a67a76aa in start_thread (arg=0x7f9883fef700) at pthread_create.c:333
#11 0x00007f98a5fcd13d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) c
Continuing.
Breakpoint 5, process_upcall (udpif=0x12e79e0, upcall=0x7f9883fec2d0, odp_actions=0x7f9883fedd20, wc=0x7f9883fed1e8)
at ofproto/ofproto-dpif-upcall.c:1201
1201 {
(gdb) thread
[Current thread is 2 (Thread 0x7f9883fef700 (LWP 134776))]
(gdb) bt
#0 process_upcall (udpif=0x12e79e0, upcall=0x7f9883fec2d0, odp_actions=0x7f9883fedd20, wc=0x7f9883fed1e8) at ofproto/ofproto-dpif-upcall.c:1201
#1 0x00000000005d9de0 in upcall_cb (packet=0x7f961357eb80, flow=0x7f9883fecfa0, ufid=0x7f9883fece80, pmd_id=3, type=DPIF_UC_MISS, userdata=0x0,
actions=0x7f9883fedd20, wc=0x7f9883fed1e8, put_actions=0x7f9883fedd60, aux=0x12e79e0) at ofproto/ofproto-dpif-upcall.c:1158
#2 0x000000000061bb77 in dp_netdev_upcall (pmd=0x1792570, packet_=0x7f961357eb80, flow=0x7f9883fecfa0, wc=0x7f9883fed1e8, ufid=0x7f9883fece80,
type=DPIF_UC_MISS, userdata=0x0, actions=0x7f9883fedd20, put_actions=0x7f9883fedd60) at lib/dpif-netdev.c:3834
#3 0x000000000061c2b6 in handle_packet_upcall (pmd=0x1792570, packet=0x7f961357eb80, key=0x7f9883fee430, actions=0x7f9883fedd20,
put_actions=0x7f9883fedd60, lost_cnt=0x7f9883fedcb4, now=95247337) at lib/dpif-netdev.c:4048
#4 0x000000000061c89f in fast_path_processing (pmd=0x1792570, packets_=0x7f9883fee950, keys=0x7f9883fee430, batches=0x7f9883fee1f0,
n_batches=0x7f9883fee860, in_port=2, now=95247337) at lib/dpif-netdev.c:4160
#5 0x000000000061cdc9 in dp_netdev_input__ (pmd=0x1792570, packets=0x7f9883fee950, md_is_valid=false, port_no=2) at lib/dpif-netdev.c:4229
#6 0x000000000061cea0 in dp_netdev_input (pmd=0x1792570, packets=0x7f9883fee950, port_no=2) at lib/dpif-netdev.c:4246
#7 0x00000000006190de in dp_netdev_process_rxq_port (pmd=0x1792570, port=0x1384540, rxq=0x7f9612a38980) at lib/dpif-netdev.c:2856
#8 0x0000000000619cd2 in pmd_thread_main (f_=0x1792570) at lib/dpif-netdev.c:3115
#9 0x00000000006b784c in ovsthread_wrapper (aux_=0x136fa60) at lib/ovs-thread.c:342
#10 0x00007f98a67a76aa in start_thread (arg=0x7f9883fef700) at pthread_create.c:333
#11 0x00007f98a5fcd13d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) c
Continuing.
[New Thread 0x7f98a7488b00 (LWP 135068)]
[Thread 0x7f98a7488b00 (LWP 135068) exited]
^C
Program received signal SIGINT, Interrupt.
[Switching to Thread 0x7f98a7479b00 (LWP 134677)]
0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) info threads
Id Target Id Frame
35 Thread 0x7f98a5ec5700 (LWP 134678) "eal-intr-thread" 0x00007f98a5fcd743 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
34 Thread 0x7f98a56c4700 (LWP 134679) "dpdk_watchdog2" 0x00007f98a5f90e7d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
33 Thread 0x7f98a4ec3700 (LWP 134680) "vhost_thread1" 0x00007f98a5fc3983 in select () at ../sysdeps/unix/syscall-template.S:81
32 Thread 0x7f989ffff700 (LWP 134681) "urcu3" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
31 Thread 0x7f989f7fe700 (LWP 134691) "ct_clean32" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
30 Thread 0x7f98847f0700 (LWP 134735) "handler60" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
29 Thread 0x7f9884ff1700 (LWP 134736) "revalidator59" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
28 Thread 0x7f98857f2700 (LWP 134737) "revalidator58" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
27 Thread 0x7f9885ff3700 (LWP 134738) "revalidator57" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
26 Thread 0x7f989effd700 (LWP 134739) "revalidator56" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
25 Thread 0x7f989e7fc700 (LWP 134740) "revalidator43" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
24 Thread 0x7f989dffb700 (LWP 134741) "revalidator44" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
23 Thread 0x7f989d7fa700 (LWP 134742) "revalidator46" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
22 Thread 0x7f989cff9700 (LWP 134743) "revalidator47" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
21 Thread 0x7f9897fff700 (LWP 134744) "revalidator48" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
20 Thread 0x7f98977fe700 (LWP 134745) "revalidator41" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
19 Thread 0x7f9896ffd700 (LWP 134746) "revalidator54" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
18 Thread 0x7f98967fc700 (LWP 134747) "revalidator49" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
17 Thread 0x7f9895ffb700 (LWP 134748) "revalidator33" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
16 Thread 0x7f98957fa700 (LWP 134749) "revalidator40" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
15 Thread 0x7f9894ff9700 (LWP 134750) "revalidator34" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
14 Thread 0x7f988bfff700 (LWP 134751) "revalidator35" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
13 Thread 0x7f988b7fe700 (LWP 134752) "revalidator50" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
12 Thread 0x7f988affd700 (LWP 134753) "revalidator36" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
11 Thread 0x7f988a7fc700 (LWP 134754) "revalidator51" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
10 Thread 0x7f9889ffb700 (LWP 134755) "revalidator55" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
9 Thread 0x7f98897fa700 (LWP 134756) "revalidator37" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
8 Thread 0x7f9888ff9700 (LWP 134757) "revalidator52" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
7 Thread 0x7f98887f8700 (LWP 134758) "revalidator42" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
6 Thread 0x7f9887ff7700 (LWP 134759) "revalidator38" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
5 Thread 0x7f98877f6700 (LWP 134760) "revalidator39" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
4 Thread 0x7f9886ff5700 (LWP 134761) "revalidator45" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
3 Thread 0x7f98867f4700 (LWP 134762) "revalidator53" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
2 Thread 0x7f9883fef700 (LWP 134776) "pmd62" 0x0000000000614dae in non_atomic_ullong_add (var=0x1b92660, n=116) at lib/dpif-netdev.c:1161
* 1 Thread 0x7f98a7479b00 (LWP 134677) "ovs-vswitchd" 0x00007f98a5fc1b1d in poll () at ../sysdeps/unix/syscall-template.S:81
(gdb)
--------------------
As far as I understood the upcall_receive() has to be executed by the handler thread, not the PMD.
Am I missing anything here?
Regards
_Sugesh
> -----Original Message-----
> From: Joe Stringer [mailto:joe at ovn.org]
> Sent: Tuesday, August 16, 2016 6:09 PM
> To: Chandran, Sugesh <sugesh.chandran at intel.com>
> Cc: discuss at openvswitch.org
> Subject: Re: handler threads
>
> Hi Sugesh,
>
> On 11 August 2016 at 08:31, Chandran, Sugesh
> <sugesh.chandran at intel.com> wrote:
> > Hi,
> > I am looking at the data-path flow insertion in OVS. Specifically the upcall
> handling in handler threads. According to my understanding, for every new
> packet-in, the handler threads process the upcall and insert the
> corresponding rule in dpif.
> > And its done in "udpif_upcall_handler" --> "recv_upcalls"(ofproto-dpif-
> upcall.c).
> >
> > My questions are
> > 1) Is it the place where the upcall handling really happening?
>
> Yes, for the kernel datapath. For userspace datapath, you'll be more
> interested in upcall_cb(). The two meet at upcall_receive().
>
> > 2) When I try to debug the handler with gdb, "recv_upcalls" is not
> > called for the new packet in/flow insert (Please note the number of
> > handler and revalidator threads are set to 1)
>
> This is a statement.
>
> > 3) How can I adjust the timeout for revalidator and handler to debug the
> threads with gdb. Sometimes the threads are crashing because of it.
>
> Which timeout do you mean? If threads are crashing, then gdb should help
> you to find out why.
More information about the discuss
mailing list