[ovs-dev] [PATCH v2] netdev-dpdk: Limit rate of DPDK logs.
Ilya Maximets
i.maximets at samsung.com
Fri Mar 23 12:54:18 UTC 2018
On 23.03.2018 15:18, Stokes, Ian wrote:
>> DPDK could produce huge amount of logs. For example, in case of exhausting
>> of a mempool in vhost-user port, following message will be printed on each
>> call to 'rte_vhost_dequeue_burst()':
>>
>> |ERR|VHOST_DATA: Failed to allocate memory for mbuf.
>>
>> These messages are increasing ovs-vswitchd.log size extremely fast making
>> it unreadable and non-parsable by a common linux utils like grep, less
>> etc. Moreover continuously growing log could exhaust the HDD space in a
>> few hours breaking normal operation of the whole system.
>>
>> To avoid such issues, DPDK log rate limited to 600 messages per minute.
>> This value is high, because we still want to see many big logs like vhost-
>> user configuration sequence. The debug messages are treated separately to
>> avoid looss of errors/warnings in case of intensive debug enabled in DPDK.
>>
>> Signed-off-by: Ilya Maximets <i.maximets at samsung.com>
>
> Thanks for this Ilya, LGTM.
>
> Do you think this is worth backporting to previous releases also, the same issue would be there also I assume?
Yes, sure.
The described issue was found with 2.8 release.
Best regards, Ilya Maximets.
>
> Ian
>> ---
>>
>> Version 2:
>> * Separate limit for DEBUG level.
>>
>> lib/dpdk.c | 10 ++++++----
>> 1 file changed, 6 insertions(+), 4 deletions(-)
>>
>> diff --git a/lib/dpdk.c b/lib/dpdk.c
>> index 3f5a55f..00dd974 100644
>> --- a/lib/dpdk.c
>> +++ b/lib/dpdk.c
>> @@ -272,20 +272,22 @@ static ssize_t
>> dpdk_log_write(void *c OVS_UNUSED, const char *buf, size_t size) {
>> char *str = xmemdup0(buf, size);
>> + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(600, 600);
>> + static struct vlog_rate_limit dbg_rl = VLOG_RATE_LIMIT_INIT(600,
>> + 600);
>>
>> switch (rte_log_cur_msg_loglevel()) {
>> case RTE_LOG_DEBUG:
>> - VLOG_DBG("%s", str);
>> + VLOG_DBG_RL(&dbg_rl, "%s", str);
>> break;
>> case RTE_LOG_INFO:
>> case RTE_LOG_NOTICE:
>> - VLOG_INFO("%s", str);
>> + VLOG_INFO_RL(&rl, "%s", str);
>> break;
>> case RTE_LOG_WARNING:
>> - VLOG_WARN("%s", str);
>> + VLOG_WARN_RL(&rl, "%s", str);
>> break;
>> case RTE_LOG_ERR:
>> - VLOG_ERR("%s", str);
>> + VLOG_ERR_RL(&rl, "%s", str);
>> break;
>> case RTE_LOG_CRIT:
>> case RTE_LOG_ALERT:
>> --
>> 2.7.4
>
>
>
>
More information about the dev
mailing list