[ovs-discuss] line rate traffic in dpdk & ovs rx errors and dead port.

Daniele Di Proietto diproiettod at vmware.com
Mon Sep 28 18:16:57 UTC 2015


First of all I would update to OVS 2.4 or master.

That said, the first issue you're facing might be related to a
packet leak (I suspect in the code that moves the packets to the
slow path). You could check the hit/miss counters with
`ovs-appctl dpif/show` (we added ovs-appctl dpif-netdev/pmd-stats-show
in 2.4, which gives per pmd thread statistics).

I'm not sure about the second problem, but I would suggest again strongly
to update to a more recent version.

Thanks,

Daniele

On 26/09/2015 21:04, "David Evans" <davidjoshuaevans at gmail.com> wrote:

>Hi Daniele,
>
>
>I have a way to mitigate the problem by stopping  rte_eth_rx_burst until
>all the rules have settled. I do this with an options: argument in
>other_config (amongst other arguments using this free form field - like
>dedup and defrag features i am experimenting
> with) My rules are static so this is ok for me to drop packets while
>rules are all being added as configuration is an infrequent event for my
>implementation.
>
>
>but occasionally after setting port options on a netdev using pmd328 via
>ovsdb 'options' in other_config i get:
>2015-09-26T19:41:14.186Z|00297|ovs_rcu|WARN|blocked 1000 ms waiting for
>pmd327 to quiesce
>
>
>from which it never recovers but exponentially backs off waiting.
>
>
>
>
>Cheers,
>Dave
>
>
>On Fri, Sep 25, 2015 at 10:17 PM, David Evans
><davidjoshuaevans at gmail.com> wrote:
>
>yes i suspect a packet leak.
>As the drop count when it stops is usually about 520K packets. :(
>Where do i start looking in the source for that kind of symptom?
>
>
>On Fri, Sep 25, 2015 at 9:58 PM, David Evans
><davidjoshuaevans at gmail.com> wrote:
>
>Hi Daniele!
>Thank you for your reply.
>I find that it reliably starts too if i have 6 or less NIC ports.
>
>
>Any more than 6 ports (even if there is only one receiving traffic) and
>the one receivng traffic will just keep upping the rx_errors.
>
>
>
>Ovs is automatically adding 11pmd's because i have a FFF mask for core
>usage and it is using 10G worth of 1G hugepages. I have about 524288 9k
>packet buffers on one  numa node memory. and 262K on the other.
>
>
>If i have my 12 ports included, and if i wait till it has completed
>starting up before sending any traffic it easily copes with at least
>80+Gbps throughput. But alas if traffic is running on startup i just get
>rx_errors.
>
>
>On my bridge i add a drop all flow at low priority on table 3 and have
>various other rules in table 0, 1 and 2 and 3 for
>detection/filtering/forwarding. The problem is exacerbated if i add more
>bridges and connect the dpdk ports to other bridges in the
> same switch.
>
>
>Could it have leaked out all it's packet buffers?
>Or the threads be busy wait somewhere else because of a race condition
>with all those PMD's?
>Maybe a thread synchronize, or quiescing issue?
>
>
>Can you think of a neat way i can track thread synchronization & packet
>usage?
>
>
>btw i am using  ovs 2.3.2+29bae5412ea3c6a1e20d79afd25c11b968dffa05-r0
>
>
>Regards,
>
>
>Dave.
> 
>
>
>On Fri, Sep 25, 2015 at 11:50 AM, Daniele Di Proietto
><diproiettod at vmware.com> wrote:
>
>I regularly start ovs with dpdk while traffic is flowing.
>
>Is there anything suspicious in the ovs-vswitchd log?
>
>Have you tried other DPDK apps (e.g. l2fwd)?
>
>
>On 25/09/2015 17:28, "David Evans" <davidjoshuaevans at gmail.com> wrote:
>
>>Hi all,
>>
>>
>>Has anyone seen that when starting ovs (with dpdk) that the port will
>>show many rx-errors and the device will stop receiving traffic?
>>Is there a mitigation for this.
>>I have tried adding drop all flows to the bridge before all my other
>>flows are set up.
>>I have tried disabling the port rx too. But still get this behaviour when
>>starting ovs if there is traffic running to it.
>>
>>
>>Everything works fine if I have no traffic running to the dpdk port at
>>startup and initiate the traffic after the switch is steady and
>>configured.
>>
>>
>>Regards,
>>Dave.
>>
>>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>




More information about the discuss mailing list