[ovs-dev] [PATCH RFC 4/6] dpif-netdev: XPS (Transmit Packet Steering) implementation.

[Ryan Moats]

"dev" <dev-bounces at openvswitch.org> wrote on 05/12/2016 08:43:13 AM:

> From: Ilya Maximets <i.maximets at samsung.com>
> To: dev at openvswitch.org, Daniele Di Proietto <diproiettod at vmware.com>
> Cc: Dyasly Sergey <s.dyasly at samsung.com>, Flavio Leitner
> <fbl at sysclose.org>, Ilya Maximets <i.maximets at samsung.com>, Kevin
> Traynor <kevin.traynor at intel.com>
> Date: 05/12/2016 08:44 AM
> Subject: [ovs-dev] [PATCH RFC 4/6] dpif-netdev: XPS (Transmit Packet
> Steering) implementation.
> Sent by: "dev" <dev-bounces at openvswitch.org>
>
> If CPU number in pmd-cpu-mask is not divisible by the number of queues
and
> in a few more complex situations there may be unfair distribution of TX
> queue-ids between PMD threads.
>
> For example, if we have 2 ports with 4 queues and 6 CPUs in pmd-cpu-mask
> such distribution is possible:
>
<------------------------------------------------------------------------>
> # ovs-appctl dpif-netdev/pmd-rxq-show
> pmd thread numa_id 0 core_id 13:
>         port: vhost-user1       queue-id: 1
>         port: dpdk0     queue-id: 3
> pmd thread numa_id 0 core_id 14:
>         port: vhost-user1       queue-id: 2
> pmd thread numa_id 0 core_id 16:
>         port: dpdk0     queue-id: 0
> pmd thread numa_id 0 core_id 17:
>         port: dpdk0     queue-id: 1
> pmd thread numa_id 0 core_id 12:
>         port: vhost-user1       queue-id: 0
>         port: dpdk0     queue-id: 2
> pmd thread numa_id 0 core_id 15:
>         port: vhost-user1       queue-id: 3
>
<------------------------------------------------------------------------>
>
> As we can see above dpdk0 port polled by threads on cores:
>    12, 13, 16 and 17.
>
> By design of dpif-netdev, there is only one TX queue-id assigned to each
> pmd thread. This queue-id's are sequential similar to core-id's. And
> thread will send packets to queue with exact this queue-id regardless
> of port.
>
> In previous example:
>
>    pmd thread on core 12 will send packets to tx queue 0
>    pmd thread on core 13 will send packets to tx queue 1
>    ...
>    pmd thread on core 17 will send packets to tx queue 5
>
> So, for dpdk0 port after truncating in netdev-dpdk:
>
>    core 12 --> TX queue-id 0 % 4 == 0
>    core 13 --> TX queue-id 1 % 4 == 1
>    core 16 --> TX queue-id 4 % 4 == 0
>    core 17 --> TX queue-id 5 % 4 == 1
>
> As a result only 2 of 4 queues used.
>
> To fix this issue some kind of XPS implemented in following way:
>
>    * TX queue-ids are allocated dynamically.
>    * When PMD thread first time tries to send packets to new port
>      it allocates less used TX queue for this port.
>    * PMD threads periodically performes revalidation of
>      allocated TX queue-ids. If queue wasn't used in last XPS_CYCLES
>      it will be freed while revalidation.
>
> Reported-by: Zhihong Wang <zhihong.wang at intel.com>
> Signed-off-by: Ilya Maximets <i.maximets at samsung.com>
> ---

As an FTI, the patch to lib/netdev-dummy.c didn't apply cleanly
for me.

Ryan