[ovs-dev] OVS will hit an assert if encap(nsh) is done in bucket of group

Jan Scheurich jan.scheurich at ericsson.com
Fri Mar 23 07:51:45 UTC 2018


Hi Yi,

Could you please provide the OF pipeline (flows and groups) and an ofproto/trace command that triggers that fault? 

Thanks, Jan

> -----Original Message-----
> From: Yang, Yi [mailto:yi.y.yang at intel.com]
> Sent: Friday, 23 March, 2018 04:53
> To: dev at openvswitch.org
> Cc: Jan Scheurich <jan.scheurich at ericsson.com>; Zoltán Balogh <zoltan.balogh at ericsson.com>
> Subject: OVS will hit an assert if encap(nsh) is done in bucket of group
> 
> Hi, guys
> 
> A NSH user found OVS will hit the below assert in function
> commit_set_nsh_action in file lib/odp-util.c if encap(nsh) is done in
> bucket of group
> 
>     ovs_assert(flow->nsh.mdtype == base_flow->nsh.mdtype &&
>                flow->nsh.np == base_flow->nsh.np);
> 
> But it isn't an issue if encap(nsh) in actions=.
> 
> I debugged this issue but can't find the root cause, basically
> xlate_generic_encap_action is called twice for a packet in different
> code path in group bucket use case. one is upcall process, another one
> is normal process in two code paths, gdb call stack dump is as followed.
> 
> The second call is obviously based on the result of the first call which
> has been committed by xlate_commit_actions in first call
> xlate_generic_encap_action, so flow->nsh.np will be set to NSH_P_NSH,
> this is wrong, but it can work normally if I comment out the above
> assert. I really don't know why xlate_generic_encap_action is called
> twice in group bucket use case, so look forward to your insights,
> appriciate your feedback heartfeltly.
> 
> (gdb) bt
> #0  xlate_generic_encap_action (encap=0x7ffd9458f460,
> ctx=0x7ffd94590f30)
>     at ofproto/ofproto-dpif-xlate.c:5913
> #1  do_xlate_actions (ofpacts=<optimized out>, ofpacts_len=<optimized
> out>,
>     ctx=ctx at entry=0x7ffd94590f30,
> is_last_action=is_last_action at entry=false)
>     at ofproto/ofproto-dpif-xlate.c:6499
> #2  0x000000000074add0 in xlate_group_bucket
> (ctx=ctx at entry=0x7ffd94590f30,
>     is_last_action=<optimized out>, bucket=0x2214f90, bucket=0x2214f90)
>     at ofproto/ofproto-dpif-xlate.c:4090
> #3  0x0000000000749ee4 in xlate_all_group (is_last_action=false,
>     group=0x2214bc0, ctx=0x7ffd94590f30) at
> ofproto/ofproto-dpif-xlate.c:4150
> #4  xlate_group_action__ (is_last_action=<optimized out>,
> group=0x2214bc0,
>     ctx=0x7ffd94590f30) at ofproto/ofproto-dpif-xlate.c:4304
> #5  xlate_group_action (is_last_action=<optimized out>,
>     group_id=<optimized out>, ctx=0x7ffd94590f30)
>     at ofproto/ofproto-dpif-xlate.c:4335
> #6  do_xlate_actions (ofpacts=ofpacts at entry=0x2212558,
>     ofpacts_len=ofpacts_len at entry=8, ctx=ctx at entry=0x7ffd94590f30,
>     is_last_action=is_last_action at entry=true)
>     at ofproto/ofproto-dpif-xlate.c:6177
> #7  0x0000000000750041 in xlate_actions (xin=xin at entry=0x7ffd945919d0,
>     xout=xout at entry=0x7ffd94591dd0) at ofproto/ofproto-dpif-xlate.c:7090
> #8  0x0000000000741f00 in upcall_xlate (wc=0x7ffd94592ff8,
> ---Type <return> to continue, or q <return> to quit---
>     odp_actions=0x7ffd945927d0, upcall=0x7ffd94591d70, udpif=0x1805440)
>     at ofproto/ofproto-dpif-upcall.c:1162
> #9  process_upcall (udpif=udpif at entry=0x1805440,
>     upcall=upcall at entry=0x7ffd94591d70,
>     odp_actions=odp_actions at entry=0x7ffd945927d0,
> wc=wc at entry=0x7ffd94592ff8)
>     at ofproto/ofproto-dpif-upcall.c:1361
> #10 0x000000000074244b in upcall_cb (packet=<optimized out>,
>     flow=0x7ffd94592d60, ufid=<optimized out>, pmd_id=<optimized out>,
>     type=<optimized out>, userdata=<optimized out>,
> actions=0x7ffd945927d0,
>     wc=0x7ffd94592ff8, put_actions=0x7ffd94592810, aux=0x1805440)
>     at ofproto/ofproto-dpif-upcall.c:1263
> #11 0x000000000076b2d6 in dp_netdev_upcall
> (packet_=packet_ at entry=0x2211680,
>     flow=flow at entry=0x7ffd94592d60, wc=wc at entry=0x7ffd94592ff8,
>     ufid=ufid at entry=0x7ffd945927b0, type=type at entry=DPIF_UC_MISS,
>     userdata=userdata at entry=0x0, actions=actions at entry=0x7ffd945927d0,
>     put_actions=put_actions at entry=0x7ffd94592810, pmd=<optimized out>,
>     pmd=<optimized out>) at lib/dpif-netdev.c:4868
> #12 0x00000000007725fd in handle_packet_upcall
> (put_actions=0x7ffd94592810,
>     actions=0x7ffd945927d0, key=0x7ffd94593c40, packet=0x2211680,
>     pmd=0x18a3e90) at lib/dpif-netdev.c:5079
> #13 fast_path_processing (pmd=pmd at entry=0x18a3e90,
>     packets_=packets_ at entry=0x7ffd94593fa0,
> keys=keys at entry=0x7ffd94593c40,
> ---Type <return> to continue, or q <return> to quit---
>     batches=batches at entry=0x7ffd94593ae0,
>     n_batches=n_batches at entry=0x7ffd94593f28, in_port=<optimized out>)
>     at lib/dpif-netdev.c:5187
> #14 0x0000000000772ea8 in dp_netdev_input__ (pmd=pmd at entry=0x18a3e90,
>     packets=packets at entry=0x7ffd94593fa0,
>     md_is_valid=md_is_valid at entry=false, port_no=port_no at entry=5)
>     at lib/dpif-netdev.c:5259
> #15 0x000000000077310d in dp_netdev_input (port_no=5,
> packets=0x7ffd94593fa0,
>     pmd=0x18a3e90) at lib/dpif-netdev.c:5287
> #16 dp_netdev_process_rxq_port (pmd=pmd at entry=0x18a3e90, rxq=0x1d4f2a0,
>     port_no=5) at lib/dpif-netdev.c:3286
> #17 0x0000000000773b02 in dpif_netdev_run (dpif=<optimized out>)
>     at lib/dpif-netdev.c:3940
> #18 0x0000000000733a18 in type_run (type=<optimized out>)
>     at ofproto/ofproto-dpif.c:342
> #19 0x000000000071f9cf in ofproto_type_run (datapath_type=<optimized
> out>,
>     datapath_type at entry=0x1d50ab0 "netdev") at ofproto/ofproto.c:1707
> #20 0x000000000070f955 in bridge_run__ () at vswitchd/bridge.c:2931
> #21 0x00000000007153c8 in bridge_run () at vswitchd/bridge.c:2995
> #22 0x0000000000415485 in main (argc=5, argv=0x7ffd94594568)
>     at vswitchd/ovs-vswitchd.c:120
> (gdb) bt
> #0  xlate_generic_encap_action (encap=0x7ffd9458f478,
> ctx=0x7ffd94590f30)
>     at ofproto/ofproto-dpif-xlate.c:5913
> #1  do_xlate_actions (ofpacts=<optimized out>, ofpacts_len=<optimized
> out>,
>     ctx=ctx at entry=0x7ffd94590f30,
> is_last_action=is_last_action at entry=false)
>     at ofproto/ofproto-dpif-xlate.c:6499
> #2  0x000000000074add0 in xlate_group_bucket
> (ctx=ctx at entry=0x7ffd94590f30,
>     is_last_action=<optimized out>, bucket=0x2214f90, bucket=0x2214f90)
>     at ofproto/ofproto-dpif-xlate.c:4090
> #3  0x0000000000749ee4 in xlate_all_group (is_last_action=false,
>     group=0x2214bc0, ctx=0x7ffd94590f30) at
> ofproto/ofproto-dpif-xlate.c:4150
> #4  xlate_group_action__ (is_last_action=<optimized out>,
> group=0x2214bc0,
>     ctx=0x7ffd94590f30) at ofproto/ofproto-dpif-xlate.c:4304
> #5  xlate_group_action (is_last_action=<optimized out>,
>     group_id=<optimized out>, ctx=0x7ffd94590f30)
>     at ofproto/ofproto-dpif-xlate.c:4335
> #6  do_xlate_actions (ofpacts=ofpacts at entry=0x2212558,
>     ofpacts_len=ofpacts_len at entry=8, ctx=ctx at entry=0x7ffd94590f30,
>     is_last_action=is_last_action at entry=true)
>     at ofproto/ofproto-dpif-xlate.c:6177
> #7  0x0000000000750041 in xlate_actions (xin=xin at entry=0x7ffd945919d0,
>     xout=xout at entry=0x7ffd94591dd0) at ofproto/ofproto-dpif-xlate.c:7090
> #8  0x0000000000741f00 in upcall_xlate (wc=0x7ffd94592ff8,
> ---Type <return> to continue, or q <return> to quit---
>     odp_actions=0x7ffd945927d0, upcall=0x7ffd94591d70, udpif=0x1805440)
>     at ofproto/ofproto-dpif-upcall.c:1162
> #9  process_upcall (udpif=udpif at entry=0x1805440,
>     upcall=upcall at entry=0x7ffd94591d70,
>     odp_actions=odp_actions at entry=0x7ffd945927d0,
> wc=wc at entry=0x7ffd94592ff8)
>     at ofproto/ofproto-dpif-upcall.c:1361
> #10 0x000000000074244b in upcall_cb (packet=<optimized out>,
>     flow=0x7ffd94592d60, ufid=<optimized out>, pmd_id=<optimized out>,
>     type=<optimized out>, userdata=<optimized out>,
> actions=0x7ffd945927d0,
>     wc=0x7ffd94592ff8, put_actions=0x7ffd94592810, aux=0x1805440)
>     at ofproto/ofproto-dpif-upcall.c:1263
> #11 0x000000000076b2d6 in dp_netdev_upcall
> (packet_=packet_ at entry=0x2211680,
>     flow=flow at entry=0x7ffd94592d60, wc=wc at entry=0x7ffd94592ff8,
>     ufid=ufid at entry=0x7ffd945927b0, type=type at entry=DPIF_UC_MISS,
>     userdata=userdata at entry=0x0, actions=actions at entry=0x7ffd945927d0,
>     put_actions=put_actions at entry=0x7ffd94592810, pmd=<optimized out>,
>     pmd=<optimized out>) at lib/dpif-netdev.c:4868
> #12 0x00000000007725fd in handle_packet_upcall
> (put_actions=0x7ffd94592810,
>     actions=0x7ffd945927d0, key=0x7ffd94593c40, packet=0x2211680,
>     pmd=0x18a3e90) at lib/dpif-netdev.c:5079
> #13 fast_path_processing (pmd=pmd at entry=0x18a3e90,
>     packets_=packets_ at entry=0x7ffd94593fa0,
> keys=keys at entry=0x7ffd94593c40,
> ---Type <return> to continue, or q <return> to quit---
>     batches=batches at entry=0x7ffd94593ae0,
>     n_batches=n_batches at entry=0x7ffd94593f28, in_port=<optimized out>)
>     at lib/dpif-netdev.c:5187
> #14 0x0000000000772ea8 in dp_netdev_input__ (pmd=pmd at entry=0x18a3e90,
>     packets=packets at entry=0x7ffd94593fa0,
>     md_is_valid=md_is_valid at entry=false, port_no=port_no at entry=5)
>     at lib/dpif-netdev.c:5259
> #15 0x000000000077310d in dp_netdev_input (port_no=5,
> packets=0x7ffd94593fa0,
>     pmd=0x18a3e90) at lib/dpif-netdev.c:5287
> #16 dp_netdev_process_rxq_port (pmd=pmd at entry=0x18a3e90, rxq=0x1d4f2a0,
>     port_no=5) at lib/dpif-netdev.c:3286
> #17 0x0000000000773b02 in dpif_netdev_run (dpif=<optimized out>)
>     at lib/dpif-netdev.c:3940
> #18 0x0000000000733a18 in type_run (type=<optimized out>)
>     at ofproto/ofproto-dpif.c:342
> #19 0x000000000071f9cf in ofproto_type_run (datapath_type=<optimized
> out>,
>     datapath_type at entry=0x1d50ab0 "netdev") at ofproto/ofproto.c:1707
> #20 0x000000000070f955 in bridge_run__ () at vswitchd/bridge.c:2931
> #21 0x00000000007153c8 in bridge_run () at vswitchd/bridge.c:2995
> #22 0x0000000000415485 in main (argc=5, argv=0x7ffd94594568)
>     at vswitchd/ovs-vswitchd.c:120
> (gdb)
> 
> (gdb) bt
> #0  xlate_generic_encap_action (encap=0x7f7df3ff81c0,
> ctx=0x7f7df3ff9c90)
>     at ofproto/ofproto-dpif-xlate.c:5913
> #1  do_xlate_actions (ofpacts=<optimized out>, ofpacts_len=<optimized
> out>,
>     ctx=ctx at entry=0x7f7df3ff9c90,
> is_last_action=is_last_action at entry=false)
>     at ofproto/ofproto-dpif-xlate.c:6499
> #2  0x000000000074add0 in xlate_group_bucket
> (ctx=ctx at entry=0x7f7df3ff9c90,
>     is_last_action=<optimized out>, bucket=0x2214f90, bucket=0x2214f90)
>     at ofproto/ofproto-dpif-xlate.c:4090
> #3  0x0000000000749ee4 in xlate_all_group (is_last_action=false,
>     group=0x2214bc0, ctx=0x7f7df3ff9c90) at
> ofproto/ofproto-dpif-xlate.c:4150
> #4  xlate_group_action__ (is_last_action=<optimized out>,
> group=0x2214bc0,
>     ctx=0x7f7df3ff9c90) at ofproto/ofproto-dpif-xlate.c:4304
> #5  xlate_group_action (is_last_action=<optimized out>,
>     group_id=<optimized out>, ctx=0x7f7df3ff9c90)
>     at ofproto/ofproto-dpif-xlate.c:4335
> #6  do_xlate_actions (ofpacts=ofpacts at entry=0x2212558,
>     ofpacts_len=ofpacts_len at entry=8, ctx=ctx at entry=0x7f7df3ff9c90,
>     is_last_action=is_last_action at entry=true)
>     at ofproto/ofproto-dpif-xlate.c:6177
> #7  0x0000000000750041 in xlate_actions (xin=xin at entry=0x7f7df3ffa130,
>     xout=xout at entry=0x7f7df3ffaa50) at ofproto/ofproto-dpif-xlate.c:7090
> #8  0x000000000073fccd in xlate_key (key=<optimized out>,
> ---Type <return> to continue, or q <return> to quit---
>     len=<optimized out>, push=push at entry=0x7f7df3ffa4d0,
>     ctx=ctx at entry=0x7f7df3ffaa30, udpif=<optimized out>)
>     at ofproto/ofproto-dpif-upcall.c:2055
> #9  0x000000000074031a in xlate_ukey (ukey=0x2214300, ukey=0x2214300,
>     ctx=0x7f7df3ffaa30, tcp_flags=<optimized out>, udpif=0x1805440)
>     at ofproto/ofproto-dpif-upcall.c:2070
> #10 revalidate_ukey__ (udpif=udpif at entry=0x1805440,
>     ukey=ukey at entry=0x2214300, tcp_flags=<optimized out>,
>     odp_actions=0x7f7df3ffae60, recircs=recircs at entry=0x7f7df3ffae50,
>     xcache=<optimized out>) at ofproto/ofproto-dpif-upcall.c:2116
> #11 0x00000000007405a6 in revalidate_ukey (udpif=udpif at entry=0x1805440,
>     ukey=ukey at entry=0x2214300, stats=stats at entry=0x7f7df3ffbb98,
>     odp_actions=odp_actions at entry=0x7f7df3ffae60,
>     reval_seq=reval_seq at entry=527957,
> recircs=recircs at entry=0x7f7df3ffae50)
>     at ofproto/ofproto-dpif-upcall.c:2218
> #12 0x0000000000743523 in revalidate (revalidator=0x18078b0)
>     at ofproto/ofproto-dpif-upcall.c:2522
> #13 0x000000000074362b in udpif_revalidator (arg=0x18078b0)
>     at ofproto/ofproto-dpif-upcall.c:910
> #14 0x00000000007eb344 in ovsthread_wrapper (aux_=<optimized out>)
>     at lib/ovs-thread.c:348
> #15 0x00007f7e06697184 in start_thread (arg=0x7f7df3fff700)
> ---Type <return> to continue, or q <return> to quit---
>     at pthread_create.c:312
> #16 0x00007f7e05cab03d in clone ()
>     at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
> (gdb)


More information about the dev mailing list