[ovs-dev] [PATCH v4 2/2] netdev-dpdk: Add new DPDK RFC 4115 egress policer

Tue Jan 14 15:21:14 UTC 2020

On 1/14/2020 2:13 PM, Eelco Chaudron wrote:
> 
> 
> On 14 Jan 2020, at 12:23, Stokes, Ian wrote:
> 
>> On 1/13/2020 8:32 PM, Stokes, Ian wrote:
> 
> <SNIP>
> 
>> Hi Eelco, I'm seeing a crash in OVS while running this with just a 
>> port and a default queue 0 (phy to phy setup). It seems related to the 
>> call to rte_meter_trtcm_rfc4115_color_blind_check. I've provided more 
>> detail below in the trtcm_policer_run_single_packet function, just 
>> wondering if you've come across it?
>>
>> Heres the output for the qos configuration I'm using
>>
>> -bash-4.4$ovs-appctl -t ovs-vswitchd qos/show dpdk1
>> QoS: dpdk1 trtcm-policer
>> eir: 52000
>> cbs: 2048
>> ebs: 2048
>> cir: 52000
>>
>> Default:
>>   eir: 52000
>>   cbs: 2048
>>   ebs: 2048
>>   cir: 52000
>>   tx_packets: 672150
>>   tx_bytes: 30918900
>>   tx_errors: 489562233
>>
>> I'll try to investigate further with DPDK and GDB also.
> 
> I tried to replicate this, but I’m not able to do so. How did you test? 
> Reconfiguring it and start, etc. etc.?

Starting a fresh instance of OVS (cleared previous OVSDB etc.).

dpdk-socket-mem="1024,0"
dpdk-lcore-mask="0x2"
pmd-cpu-mask="0xC"

2 phy ports only, 1 rxq per phy port.

Flow rules are basic (in port 1 out port 2)

Traffic profile is IPv4 UDP 64 byte packets at line rate (10G)

QoS Setup with the following

sudo $OVS_DIR/utilities/ovs-vsctl --timeout=5 set port dpdk1 qos=@myqos -- \
--id=@myqos create qos type=trtcm-policer \
other-config:cir=52000 other-config:cbs=2048 \
other-config:eir=52000 other-config:ebs=2048

 From there it's a case of leaving traffic run (between 10 to 15 mins) 
before the segfault occurs.

> 
> <SNIP>
> 
>>
>> A few times during testing I have seen OVS crash with the following
>>
>> ./launch_vswitch.sh: line 66: 11694 Floating point exception sudo 
>> $OVS_DIR/vswitchd/ovs-vswitchd unix:$DB_SOCK --pidfile
>>
>> Looking into it with GDB it sems related to the 
>> rte_meter_trtcm_rfc4115_color_blind_check above. See GDB output below.
>>
>> Thread 12 "pmd-c03/id:9" received signal SIGFPE, Arithmetic exception.
>> [Switching to Thread 0x7f3dce734700 (LWP 26465)]
>> 0x0000000000d4b92d in rte_meter_trtcm_rfc4115_color_blind_check 
>> (m=0x328e178, p=0x328e148, time=29107058565136113, pkt_len=46) at 
>> /opt/istokes/dpdk-19.11//x86_64-native-linuxapp-gcc/include/rte_meter.h:599 
>>
>> 599             n_periods_te = time_diff_te / p->eir_period;
>> (gdb) bt
>> #0  0x0000000000d4b92d in rte_meter_trtcm_rfc4115_color_blind_check 
>> (m=0x328e178, p=0x328e148, time=29107058565136113, pkt_len=46) at 
>> /opt/istokes/dpdk-19.11//x86_64-native-linuxapp-gcc/include/rte_meter.h:599 
>>
>> #1  0x0000000000d4abc1 in trtcm_policer_run_single_packet 
>> (policer=0x2774200, pkt=0x1508fbd40, time=29107058565136113) at 
>> lib/netdev-dpdk.c:4649
>> #2  0x0000000000d4ad24 in trtcm_policer_run (conf=0x2774200, 
>> pkts=0x7f3db8005100, pkt_cnt=32, should_steal=true) at 
>> lib/netdev-dpdk.c:4691
>> #3  0x0000000000d45299 in netdev_dpdk_qos_run (dev=0x17fd68840, 
>> pkts=0x7f3db8005100, cnt=32, should_steal=true) at lib/netdev-dpdk.c:2421
>> #4  0x0000000000d45db0 in netdev_dpdk_send__ (dev=0x17fd68840, qid=1, 
>> batch=0x7f3db80050f0, concurrent_txq=false) at lib/netdev-dpdk.c:2683
>> #5  0x0000000000d45ee9 in netdev_dpdk_eth_send (netdev=0x17fd688c0, 
>> qid=1, batch=0x7f3db80050f0, concurrent_txq=false) at 
>> lib/netdev-dpdk.c:2710
>> #6  0x0000000000c342ba in netdev_send (netdev=0x17fd688c0, qid=1, 
>> batch=0x7f3db80050f0, concurrent_txq=false) at lib/netdev.c:814
>> #7  0x0000000000beb3de in dp_netdev_pmd_flush_output_on_port 
>> (pmd=0x7f3dce735010, p=0x7f3db80050c0) at lib/dpif-netdev.c:4224
>> #8  0x0000000000beb5c4 in dp_netdev_pmd_flush_output_packets 
>> (pmd=0x7f3dce735010, force=false) at lib/dpif-netdev.c:4264
>> #9  0x0000000000beb814 in dp_netdev_process_rxq_port 
>> (pmd=0x7f3dce735010, rxq=0x328d930, port_no=2) at lib/dpif-netdev.c:4319
>> #10 0x0000000000bef432 in pmd_thread_main (f_=0x7f3dce735010) at 
>> lib/dpif-netdev.c:5556
>> #11 0x0000000000cb24e5 in ovsthread_wrapper (aux_=0x326d220) at 
>> lib/ovs-thread.c:383
>> #12 0x00007f3de11d236d in start_thread (arg=0x7f3dce734700) at 
>> pthread_create.c:456
>> #13 0x00007f3de06bab4f in clone () at 
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> 
> Looks like a divide by zero, but that was fixed and seems to be in DPDK 
> v19.11, ebe3a769911071450acb808153ec2a2496726906

I've confirmed I'm testing against 19.11.0 and that commit is present.

> 
> So for some reason rte_meter_get_tb_params() might return 0 in 
> eir_period. Looking at the code, I would say this could only really 
> happen if rte_get_tsc_hz() returns 0, which seems odd… Could this happen 
> in your system for some reason?

I don't think rte_get_tsc_hz is returning 0, at east the values for the 
function call in GDB don't seem to suggest this. Snippet below from 
rte_meter_trtcm_rfc4115_color_blind_check that I'm checking with GDB.

rte_meter_trtcm_rfc4115_color_blind_check(struct rte_meter_trtcm_rfc4115 
*m, struct rte_meter_trtcm_rfc4115_profile *p, uint64_t time, uint32_t 
pkt_len) 

{ 

     uint64_t time_diff_tc, time_diff_te, n_periods_tc, n_periods_te,
     tc, te;

     /* Bucket update */
     time_diff_tc = time - m->time_tc;
     time_diff_te = time - m->time_te;

     n_periods_tc = time_diff_tc / p->cir_period;
     n_periods_te = time_diff_te / p->eir_period;

Looking at values with GDB gives the following

(gdb) p *p
$1 = {cbs = 2048, ebs = 2048, cir_period = 44230, cir_bytes_per_period = 
1, eir_period = 44230, eir_bytes_per_period = 1}
(gdb) p time_diff_tc
$2 = 29137292739849292
(gdb) p n_periods_tc
$3 = 13937399
(gdb) p time
$4 = 29137292739849292
(gdb) p m->time_tc
$5 = 29137292739858117
(gdb) p *m
$6 = {time_tc = 29137292739858117, time_te = 29137292739858117, tc = 
2048, te = 2048}
(gdb) p time_diff_te
$7 = 29137292739849289
(gdb) p p->eir_period
$8 = 44230
(gdb) p n_periods_te
$9 = 140260075883992

I don't have another board to test on at the moment but will try.

Ian

> 
> //Eelco
> 
> 
> <SNIP>
> 