[ovs-dev] [PATCH] ofproto: Fix resource usage explosion while processing bundled FLOW_MOD.

[Vladislav Odintsov]

Hi Ilya,

I’ve tested both patches, the problem with high CPU seems to be solved, thanks.
I noticed see that if I add/delete one lsp to/from port group with a negative match rule, ovs-vswitchd each time (add or remove) consumes +4-18 MB RSS.

# ovn-nbctl pg-set-ports pg_1 lsp1 lsp2 lsp3 lsp4

[root at ovn-1 ~]# while :; do ps-mem -p $(pidof ovs-vswitchd) 2> /dev/null | grep ovs; sleep 1; done
 32.6 MiB +   1.4 MiB =  34.0 MiB	ovs-vswitchd
 32.6 MiB +   1.4 MiB =  34.0 MiB	ovs-vswitchd
 32.6 MiB +   1.4 MiB =  34.0 MiB	ovs-vswitchd
 32.5 MiB +   1.4 MiB =  33.8 MiB	ovs-vswitchd
 88.7 MiB +   1.4 MiB =  90.1 MiB	ovs-vswitchd
130.7 MiB +   1.4 MiB = 132.1 MiB	ovs-vswitchd  # ovn-nbctl pg-set-ports pg_1 lsp1 lsp2 lsp3
130.7 MiB +   1.4 MiB = 132.1 MiB	ovs-vswitchd
130.7 MiB +   1.4 MiB = 132.1 MiB	ovs-vswitchd
130.7 MiB +   1.4 MiB = 132.1 MiB	ovs-vswitchd
130.7 MiB +   1.4 MiB = 132.1 MiB	ovs-vswitchd
130.7 MiB +   1.4 MiB = 132.1 MiB	ovs-vswitchd
141.2 MiB +   1.4 MiB = 142.6 MiB	ovs-vswitchd  # ovn-nbctl pg-set-ports pg_1 lsp1 lsp2 lsp3 lsp4
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
158.4 MiB +   1.4 MiB = 159.8 MiB	ovs-vswitchd
162.3 MiB +   1.4 MiB = 163.7 MiB	ovs-vswitchd  # ovn-nbctl pg-set-ports pg_1 lsp1 lsp2 lsp3
162.3 MiB +   1.4 MiB = 163.7 MiB	ovs-vswitchd
162.3 MiB +   1.4 MiB = 163.7 MiB	ovs-vswitchd
162.3 MiB +   1.4 MiB = 163.7 MiB	ovs-vswitchd
162.3 MiB +   1.4 MiB = 163.7 MiB	ovs-vswitchd
162.3 MiB +   1.4 MiB = 163.7 MiB	ovs-vswitchd
162.3 MiB +   1.4 MiB = 163.7 MiB	ovs-vswitchd
162.3 MiB +   1.4 MiB = 163.7 MiB	ovs-vswitchd
170.9 MiB +   1.4 MiB = 172.2 MiB	ovs-vswitchd  # ovn-nbctl pg-set-ports pg_1 lsp1 lsp2 lsp3 lsp4
170.9 MiB +   1.4 MiB = 172.2 MiB	ovs-vswitchd
170.9 MiB +   1.4 MiB = 172.2 MiB	ovs-vswitchd
170.9 MiB +   1.4 MiB = 172.2 MiB	ovs-vswitchd

Can this be also related to above problem?

Regards,
Vladislav Odintsov

> On 22 Nov 2021, at 18:25, Ilya Maximets <i.maximets at ovn.org> wrote:
> 
> On 11/22/21 10:18, Vladislav Odintsov wrote:
>> Ilya,
>> 
>> there’s a problem still in place, just with another case.
>> Initially I’ve tested only creation of topology, but didn’t think about testing the modification of those flows.
>> 
>> Create topology from initial mail, and then modify it somehow. For instance, change the LSPs in port group.
>> Consider we’ve got lsp1, lsp2, lsp3, lsp4 in pg1 with ACL’s negative match, then remove lsp4 from pg1 (ovn-nbctl pg-set-ports lsp1 lsp2 lsp3).
>> Symptoms are the same as it was with initial addition of flows:
>> high ovn-controller & ovs-vswitchd CPU usage and ovs-vswitchd memory consumed all memory and then got killed by OOM-killer.
>> 
>> Let me know if you need any additional info.
> 
> Thanks for testing!  I can reproduce that too.
> It's a very similar issue, but with a different flavor, so
> it will require a separate fix.
> 
> Could you try the following patch together with the previous one:
>  https://patchwork.ozlabs.org/project/openvswitch/patch/20211122152310.3546130-1-i.maximets@ovn.org/
> ?
> 
> Bets regards, Ilya Maximets.
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev