[ovs-discuss] [ovs-dev] mbuf pool sizing

[Kevin Traynor]

On 01/23/2018 11:42 AM, Kevin Traynor wrote:
> On 01/17/2018 07:48 PM, Venkatesan Pradeep wrote:
>> Hi,
>>
>> Assuming that all ports use the same MTU,  in OVS2.8 and earlier, a single mempool of 256K buffers (MAX_NB_MBUF = 4096 * 64) will be created and shared by all the ports
>>
>> With the OVS2.9 mempool patches, we have port specific allocation and the number of mbufs created for each port is based on the following formula (with a lower limit of MIN_NB_MBUF = 4096*4)
>>        n_mbufs = dev->requested_n_rxq * dev->requested_rxq_size
>>               + dev->requested_n_txq * dev->requested_txq_size
>>               + MIN(RTE_MAX_LCORE, dev->requested_n_rxq) * NETDEV_MAX_BURST
>>               + MIN_NB_MBUF;
>>
>> Using minimal value (1) for n_rxq and n_rxq and default value (2048) for requested_rxq_size and requested_txq_size, the above translates to
>>       n_mbufs = 1*2048 + 1*2048 + 1*32 + 4096*4  = 20512
>>
>> Assuming all ports have the same MTU, this means that approximately 13 ports in OVS2.9 will consume as much memory as the single mempool shared by all ports in OVS2.8 (256*1024 / 20512) .
>>
>> When a node is upgraded from OVS2.8 to OVS2.9 it is quite possible that the memory set aside for OVS may be insufficient. I'm not sure if this aspect has been discussed previously and wanted to bring this up for discussion.
>>
> 
> Hi Pradeep, I don't think it has been discussed. I guess the thinking
> was that with a giant shared mempool, it was over provisioning when
> there was a few ports, and in the case where there was a lot of ports
> there could be some starvation at run time. It also meant if you had a
> mix of different MTU's you had multiple giant shared mempools and could
> run out of memory very quickly at config or run time then also.
> 
> So I can see the argument for having a mempool per port, as it is more
> fine grained and if you are going to run short of memory, it will at
> least be at config time. The problem is if you give some over provision
> to each port and you have a lot of ports, you hit the situation you are
> seeing.
> 
> I think some amount of over provision per port is needed because you
> don't want to be cutting it so fine that you run into memory issues at
> run time about local mbuf caches on cores running out, or even if
> someone used dpdk rings to send the mbuf somewhere else for a time.
> There may be other corner cases too. Perhaps as compromise the min size
> could be reduce from 4096*4 to 4096*2 or 4096.
> 
> Thoughts?
> 

I just sent a compile tested only RFC here
https://mail.openvswitch.org/pipermail/ovs-dev/2018-January/343581.html

> Kevin.
> 
>> Regards,
>>
>> Pradeep
>>
>>
>> _______________________________________________
>> dev mailing list
>> dev at openvswitch.org
>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>>
> 
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>