[ovs-dev] [PATCH v4 2/2] netdev-dpdk: Add new DPDK RFC 4115 egress policer

Eelco Chaudron echaudro at redhat.com
Tue Jan 14 16:21:40 UTC 2020



On 14 Jan 2020, at 16:21, Stokes, Ian wrote:

> On 1/14/2020 2:13 PM, Eelco Chaudron wrote:
>>
>>
>> On 14 Jan 2020, at 12:23, Stokes, Ian wrote:
>>
>>> On 1/13/2020 8:32 PM, Stokes, Ian wrote:
>>
>> <SNIP>
>>
>>> Hi Eelco, I'm seeing a crash in OVS while running this with just a 
>>> port and a default queue 0 (phy to phy setup). It seems related to 
>>> the call to rte_meter_trtcm_rfc4115_color_blind_check. I've provided 
>>> more detail below in the trtcm_policer_run_single_packet function, 
>>> just wondering if you've come across it?
>>>
>>> Heres the output for the qos configuration I'm using
>>>
>>> -bash-4.4$ovs-appctl -t ovs-vswitchd qos/show dpdk1
>>> QoS: dpdk1 trtcm-policer
>>> eir: 52000
>>> cbs: 2048
>>> ebs: 2048
>>> cir: 52000
>>>
>>> Default:
>>>   eir: 52000
>>>   cbs: 2048
>>>   ebs: 2048
>>>   cir: 52000
>>>   tx_packets: 672150
>>>   tx_bytes: 30918900
>>>   tx_errors: 489562233
>>>
>>> I'll try to investigate further with DPDK and GDB also.
>>
>> I tried to replicate this, but I’m not able to do so. How did you 
>> test? Reconfiguring it and start, etc. etc.?
>
> Starting a fresh instance of OVS (cleared previous OVSDB etc.).
>
> dpdk-socket-mem="1024,0"
> dpdk-lcore-mask="0x2"
> pmd-cpu-mask="0xC"
>
> 2 phy ports only, 1 rxq per phy port.
>
> Flow rules are basic (in port 1 out port 2)
>
> Traffic profile is IPv4 UDP 64 byte packets at line rate (10G)
>
> QoS Setup with the following
>
> sudo $OVS_DIR/utilities/ovs-vsctl --timeout=5 set port dpdk1 
> qos=@myqos -- \
> --id=@myqos create qos type=trtcm-policer \
> other-config:cir=52000 other-config:cbs=2048 \
> other-config:eir=52000 other-config:ebs=2048
>
> From there it's a case of leaving traffic run (between 10 to 15 mins) 
> before the segfault occurs.

Tried a couple of runs, but no luck…

>>
>> <SNIP>
>>
>>>
>>> A few times during testing I have seen OVS crash with the following
>>>
>>> ./launch_vswitch.sh: line 66: 11694 Floating point exception sudo 
>>> $OVS_DIR/vswitchd/ovs-vswitchd unix:$DB_SOCK --pidfile
>>>
>>> Looking into it with GDB it sems related to the 
>>> rte_meter_trtcm_rfc4115_color_blind_check above. See GDB output 
>>> below.
>>>
>>> Thread 12 "pmd-c03/id:9" received signal SIGFPE, Arithmetic 
>>> exception.
>>> [Switching to Thread 0x7f3dce734700 (LWP 26465)]
>>> 0x0000000000d4b92d in rte_meter_trtcm_rfc4115_color_blind_check 
>>> (m=0x328e178, p=0x328e148, time=29107058565136113, pkt_len=46) at 
>>> /opt/istokes/dpdk-19.11//x86_64-native-linuxapp-gcc/include/rte_meter.h:599
>>> 599             n_periods_te = time_diff_te / 
>>> p->eir_period;
>>> (gdb) bt
>>> #0  0x0000000000d4b92d in rte_meter_trtcm_rfc4115_color_blind_check 
>>> (m=0x328e178, p=0x328e148, time=29107058565136113, pkt_len=46) at 
>>> /opt/istokes/dpdk-19.11//x86_64-native-linuxapp-gcc/include/rte_meter.h:599
>>> #1  0x0000000000d4abc1 in trtcm_policer_run_single_packet 
>>> (policer=0x2774200, pkt=0x1508fbd40, time=29107058565136113) at 
>>> lib/netdev-dpdk.c:4649
>>> #2  0x0000000000d4ad24 in trtcm_policer_run (conf=0x2774200, 
>>> pkts=0x7f3db8005100, pkt_cnt=32, should_steal=true) at 
>>> lib/netdev-dpdk.c:4691
>>> #3  0x0000000000d45299 in netdev_dpdk_qos_run (dev=0x17fd68840, 
>>> pkts=0x7f3db8005100, cnt=32, should_steal=true) at 
>>> lib/netdev-dpdk.c:2421
>>> #4  0x0000000000d45db0 in netdev_dpdk_send__ (dev=0x17fd68840, 
>>> qid=1, batch=0x7f3db80050f0, concurrent_txq=false) at 
>>> lib/netdev-dpdk.c:2683
>>> #5  0x0000000000d45ee9 in netdev_dpdk_eth_send (netdev=0x17fd688c0, 
>>> qid=1, batch=0x7f3db80050f0, concurrent_txq=false) at 
>>> lib/netdev-dpdk.c:2710
>>> #6  0x0000000000c342ba in netdev_send (netdev=0x17fd688c0, qid=1, 
>>> batch=0x7f3db80050f0, concurrent_txq=false) at lib/netdev.c:814
>>> #7  0x0000000000beb3de in dp_netdev_pmd_flush_output_on_port 
>>> (pmd=0x7f3dce735010, p=0x7f3db80050c0) at lib/dpif-netdev.c:4224
>>> #8  0x0000000000beb5c4 in dp_netdev_pmd_flush_output_packets 
>>> (pmd=0x7f3dce735010, force=false) at lib/dpif-netdev.c:4264
>>> #9  0x0000000000beb814 in dp_netdev_process_rxq_port 
>>> (pmd=0x7f3dce735010, rxq=0x328d930, port_no=2) at 
>>> lib/dpif-netdev.c:4319
>>> #10 0x0000000000bef432 in pmd_thread_main (f_=0x7f3dce735010) at 
>>> lib/dpif-netdev.c:5556
>>> #11 0x0000000000cb24e5 in ovsthread_wrapper (aux_=0x326d220) at 
>>> lib/ovs-thread.c:383
>>> #12 0x00007f3de11d236d in start_thread (arg=0x7f3dce734700) at 
>>> pthread_create.c:456
>>> #13 0x00007f3de06bab4f in clone () at 
>>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
>>
>> Looks like a divide by zero, but that was fixed and seems to be in 
>> DPDK v19.11, ebe3a769911071450acb808153ec2a2496726906
>
> I've confirmed I'm testing against 19.11.0 and that commit is present.
>
>
>>
>> So for some reason rte_meter_get_tb_params() might return 0 in 
>> eir_period. Looking at the code, I would say this could only really 
>> happen if rte_get_tsc_hz() returns 0, which seems odd… Could this 
>> happen in your system for some reason?
>
> I don't think rte_get_tsc_hz is returning 0, at east the values for 
> the function call in GDB don't seem to suggest this. Snippet below 
> from rte_meter_trtcm_rfc4115_color_blind_check that I'm checking with 
> GDB.
>
> rte_meter_trtcm_rfc4115_color_blind_check(struct 
> rte_meter_trtcm_rfc4115 *m, struct rte_meter_trtcm_rfc4115_profile *p, 
> uint64_t time, uint32_t pkt_len)
>
> {
>
>
>
>     uint64_t time_diff_tc, time_diff_te, n_periods_tc, n_periods_te,
>     tc, te;
>
>
>
>
>
>
>     /* Bucket update */
>     time_diff_tc = time - m->time_tc;
>     time_diff_te = time - m->time_te;
>
>
>
>     n_periods_tc = time_diff_tc / p->cir_period;
>     n_periods_te = time_diff_te / p->eir_period;
>
> Looking at values with GDB gives the following
>
> (gdb) p *p
> $1 = {cbs = 2048, ebs = 2048, cir_period = 44230, cir_bytes_per_period 
> = 1, eir_period = 44230, eir_bytes_per_period = 1}
> (gdb) p time_diff_tc
> $2 = 29137292739849292
> (gdb) p n_periods_tc
> $3 = 13937399
> (gdb) p time
> $4 = 29137292739849292
> (gdb) p m->time_tc
> $5 = 29137292739858117
> (gdb) p *m
> $6 = {time_tc = 29137292739858117, time_te = 29137292739858117, tc = 
> 2048, te = 2048}
> (gdb) p time_diff_te
> $7 = 29137292739849289
> (gdb) p p->eir_period
> $8 = 44230
> (gdb) p n_periods_te
> $9 = 140260075883992
>
> I don't have another board to test on at the moment but will try.

Odd, do not see how this would create a SIGFPE…



More information about the dev mailing list