[ovs-dev] [PATCH v2] netdev-dpdk: Limit rate of DPDK logs.

Ilya Maximets i.maximets at samsung.com
Fri Mar 23 12:54:18 UTC 2018


On 23.03.2018 15:18, Stokes, Ian wrote:
>> DPDK could produce huge amount of logs. For example, in case of exhausting
>> of a mempool in vhost-user port, following message will be printed on each
>> call to 'rte_vhost_dequeue_burst()':
>>
>>     |ERR|VHOST_DATA: Failed to allocate memory for mbuf.
>>
>> These messages are increasing ovs-vswitchd.log size extremely fast making
>> it unreadable and non-parsable by a common linux utils like grep, less
>> etc. Moreover continuously growing log could exhaust the HDD space in a
>> few hours breaking normal operation of the whole system.
>>
>> To avoid such issues, DPDK log rate limited to 600 messages per minute.
>> This value is high, because we still want to see many big logs like vhost-
>> user configuration sequence. The debug messages are treated separately to
>> avoid looss of errors/warnings in case of intensive debug enabled in DPDK.
>>
>> Signed-off-by: Ilya Maximets <i.maximets at samsung.com>
> 
> Thanks for this Ilya, LGTM.
> 
> Do you think this is worth backporting to previous releases also, the same issue would be there also I assume?

Yes, sure.
The described issue was found with 2.8 release.

Best regards, Ilya Maximets.

> 
> Ian
>> ---
>>
>> Version 2:
>> 	* Separate limit for DEBUG level.
>>
>>  lib/dpdk.c | 10 ++++++----
>>  1 file changed, 6 insertions(+), 4 deletions(-)
>>
>> diff --git a/lib/dpdk.c b/lib/dpdk.c
>> index 3f5a55f..00dd974 100644
>> --- a/lib/dpdk.c
>> +++ b/lib/dpdk.c
>> @@ -272,20 +272,22 @@ static ssize_t
>>  dpdk_log_write(void *c OVS_UNUSED, const char *buf, size_t size)  {
>>      char *str = xmemdup0(buf, size);
>> +    static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(600, 600);
>> +    static struct vlog_rate_limit dbg_rl = VLOG_RATE_LIMIT_INIT(600,
>> + 600);
>>
>>      switch (rte_log_cur_msg_loglevel()) {
>>          case RTE_LOG_DEBUG:
>> -            VLOG_DBG("%s", str);
>> +            VLOG_DBG_RL(&dbg_rl, "%s", str);
>>              break;
>>          case RTE_LOG_INFO:
>>          case RTE_LOG_NOTICE:
>> -            VLOG_INFO("%s", str);
>> +            VLOG_INFO_RL(&rl, "%s", str);
>>              break;
>>          case RTE_LOG_WARNING:
>> -            VLOG_WARN("%s", str);
>> +            VLOG_WARN_RL(&rl, "%s", str);
>>              break;
>>          case RTE_LOG_ERR:
>> -            VLOG_ERR("%s", str);
>> +            VLOG_ERR_RL(&rl, "%s", str);
>>              break;
>>          case RTE_LOG_CRIT:
>>          case RTE_LOG_ALERT:
>> --
>> 2.7.4
> 
> 
> 
> 


More information about the dev mailing list