<div dir="ltr">Hello Dumitru, Han,<div><br></div><div>So, we applied this patchset and gave it a spin on our large scale cluster and saw a significant reduction in the number of logical flows in lr_in_ip_input table. Before this patch there were around 1.6M flows in lr_in_ip_input table. However, after the patch we see about 26K flows. So that is significant reduction in number of logical flows.</div><div><br></div><div>In lr_in_ip_input, I see</div><div><ul><li>priority 92 flows matching ARP requests for dnat_and_snat IPs on distributed gateway port with is_chassis_resident() and corresponding ARP reply</li><li>priority 91 flows matching ARP requests for dnat_and_snat IPs on distributed gateway port with !is_chassis_resident() and corresponding drop</li><li>priority 90 flow matching ARP request for dnat_and_snat IPs and corresponding ARP replies</li></ul><div>So far so good. </div><div><br></div><div>However, not directly related to this patch per-se but directly related to the behaviour of ARP and dnat_and_snat IP, on the OVN chassis we are seeing a significant number of OpenFlow flows in table 27 (around 2.3M OpenFlow flows). This table gets populated from logical flows in table=19 (ls_in_l2_lkup) of logical switch.</div></div><div><br></div><div>The two logical flows in l2_in_l2_lkup that are contributing to huge number of OpenFlow flows are: (for the entire logical flow entry, please see: <a href="https://gist.github.com/girishmg/57b3005030d421c59b30e6c36cfc9c18">https://gist.github.com/girishmg/57b3005030d421c59b30e6c36cfc9c18</a>)<br></div><div><br></div><div>Priority=75 flow <br>=============</div><div>This flow looks like below (where <a href="http://169.254.0.0/29">169.254.0.0/29</a> is dnat_and_snat subnet and 192.168.0.1 is the logical_switch's gateway IP)<br><br></div><div><font face="monospace">table=19(ls_in_l2_lkup ), priority=75 , match=(flags[1] == 0 && arp.op == 1 && arp.tpa == { 169.254.3.107, 169.254.1.85, 192.168.0.1, 169.254.10.155, 169.254.1.6}), action=(outport = "stor-sdn-test1"; output;)</font><br></div><div><font face="monospace"><br></font></div><div>What this flow says is that any ARP request packet from the switch heading towards the default gateway or any of those 1-to-1 nat send it out through the port towards the ovn_cluster_router’s ingress pipeline. Question though is why any Pod on the logical switch would send an ARP for an IP that is not in its subnet. A packet from a Pod towards a non-subnet IP should ARP only for the default gateway IP.<br></div><div><br></div><div>Priority=80 Flow</div><div>=============<br>This flow looks like below</div><div><br></div><div><font face="monospace">table=19(ls_in_l2_lkup ), priority=80 , match=(eth.src == { 0a:58:c0:a8:00:01, 6a:93:f4:55:aa:a7, ae:92:2d:33:24:ea, ba:0a:d3:7d:bc:e8, b2:2f:40:4d:d9:2b} && (arp.op == 1 || nd_ns)), action=(outport = "_MC_flood"; output;)</font><br></div><div><font face="monospace"><br></font></div><div><font face="arial, sans-serif">The question again for this flow is why will there be a self-originated arp requests for the dnat_and_snat IPs from inside of the node's logical switch. I can see how this is a possibility on the switch that has `localnet port` on it and to which the distributed router connects to through a gateway port. </font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Regards,</font></div><div><font face="arial, sans-serif">~Girish</font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 24, 2020 at 8:55 AM Dumitru Ceara <<a href="mailto:dceara@redhat.com">dceara@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Girish,<br>
<br>
I sent a patch series to implement Han's suggestion:<br>
<a href="https://patchwork.ozlabs.org/project/openvswitch/list/?series=185580" rel="noreferrer" target="_blank">https://patchwork.ozlabs.org/project/openvswitch/list/?series=185580</a><br>
<a href="https://mail.openvswitch.org/pipermail/ovs-dev/2020-June/372005.html" rel="noreferrer" target="_blank">https://mail.openvswitch.org/pipermail/ovs-dev/2020-June/372005.html</a><br>
<br>
It would be great if you could give it a run on your setup too.<br>
<br>
Thanks,<br>
Dumitru<br>
<br>
On 6/16/20 5:18 PM, Girish Moodalbail wrote:<br>
> Thanks Han for the update.<br>
> <br>
> Regards,<br>
> ~Girish <br>
> <br>
> On Mon, Jun 15, 2020 at 12:55 PM Han Zhou <<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a><br>
> <mailto:<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a>>> wrote:<br>
> <br>
> Sorry Girish, I can't promise for now. I will see if I have time in<br>
> the next couple of weeks, but welcome anyone to volunteer on this if<br>
> it is urgent.<br>
> <br>
> On Mon, Jun 15, 2020 at 10:56 AM Girish Moodalbail<br>
> <<a href="mailto:gmoodalbail@gmail.com" target="_blank">gmoodalbail@gmail.com</a> <mailto:<a href="mailto:gmoodalbail@gmail.com" target="_blank">gmoodalbail@gmail.com</a>>> wrote:<br>
> <br>
> Hello Han,<br>
> <br>
> On Wed, Jun 3, 2020 at 9:39 PM Han Zhou <<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a><br>
> <mailto:<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a>>> wrote:<br>
> <br>
> <br>
> <br>
> On Wed, Jun 3, 2020 at 7:16 PM Girish Moodalbail<br>
> <<a href="mailto:gmoodalbail@gmail.com" target="_blank">gmoodalbail@gmail.com</a> <mailto:<a href="mailto:gmoodalbail@gmail.com" target="_blank">gmoodalbail@gmail.com</a>>> wrote:<br>
> <br>
> Hello all,<br>
> <br>
> While working on an extension, see the diagram below, to<br>
> the existing OVN logical topology for the ovn-kubernetes<br>
> project, I am seeing an explosion of the "Reply to ARP<br>
> requests" logical flows in the `lr_in_ip_input` table<br>
> for the distributed router (ovn_cluster_router)<br>
> configured with gateway port (rtol-LS)<br>
> <br>
> internet <br>
> ---------+--------------> <br>
> | <br>
> | <br>
> +----------localnet-port---------+ <br>
> |LS | <br>
> +-----------------ltor-LS--------+ <br>
> | <br>
> | <br>
> +---------------------rtol-LS------------+<br>
> | ovn_cluster_router |<br>
> | (Distributed Router) |<br>
> +-rtos-ls0------rtos-ls1--------rtos-ls2-+<br>
> | | | <br>
> | | | <br>
> +-----+-+ +----+--+ +-----+-+ <br>
> | LS0 | | LS1 | | LS2 | <br>
> +-+-----+ +-+-----+ +-+-----+ <br>
> | | | <br>
> p0 p1 p2 <br>
> IA0 IA1 IA2 <br>
> EA0 EA1 EA2 <br>
> (Node0) (Node1) (Node2)<br>
> <br>
> In the topology above, each of the three logical switch<br>
> port has an internal address of IAx and an external<br>
> address of EAx (dnat_and_snat IP). They are all bound to<br>
> their respective nodes (Nodex). A packet from `p0`<br>
> heading towards the internet will be SNAT'ed to EA0 on<br>
> the local hypervisor and then sent out through the LS's<br>
> localnet-port on that hypervisor. Basically, they are<br>
> configured for distributed NATing.<br>
> <br>
> I am seeing interesting "Reply to ARP requests" flows<br>
> for arp.tpa set to "EAX". Flows are like this:<br>
> <br>
> For EA0<br>
> priority=90, match=(inport == "rtos-ls0" && arp.tpa ==<br>
> EA0 && arp.op == 1), action=(/* ARP reply */)<br>
> priority=90, match=(inport == "rtos-ls1" && arp.tpa ==<br>
> EA0 && arp.op == 1), action=(/* ARP reply */)<br>
> priority=90, match=(inport == "rtos-ls2" && arp.tpa ==<br>
> EA0 && arp.op == 1), action=(/* ARP reply */)<br>
> <br>
> For EA1<br>
> priority=90, match=(inport == "rtos-ls0" && arp.tpa ==<br>
> EA1 && arp.op == 1), action=(/* ARP reply */)<br>
> priority=90, match=(inport == "rtos-ls1" && arp.tpa ==<br>
> EA0 && arp.op == 1), action=(/* ARP reply */)<br>
> priority=90, match=(inport == "rtos-ls2" && arp.tpa ==<br>
> EA1 && arp.op == 1), action=(/* ARP reply */)<br>
> <br>
> Similarly, for EA2.<br>
> <br>
> So, we have N * N "Reply to ARP requests" flows for N<br>
> nodes each with 1 dnat_and_snat ip. <br>
> This is causing scale issues.<br>
> <br>
> If you look at the flows for `EA0`, i am confused as to<br>
> why is it needed?<br>
> <br>
> 1. When will one see an ARP request for the EA0 from<br>
> any of the LS{0,1,2}'s logical switch port.<br>
> 2. If it is needed at all, can't we just remove the<br>
> `inport` thing altogether since the flow is<br>
> configured for every port of logical router port<br>
> except for the distributed gateway port rtol-LS. For<br>
> this port, we could add an higher priority rule with<br>
> action set to `next`.<br>
> 3. Say, we don't need east-west NAT connectivity. Is<br>
> there a way to make these ARPs be learnt<br>
> dynamically, like we are doing for join and external<br>
> logical switch (the other thread [1]).<br>
> <br>
> Regards,<br>
> ~Girish<br>
> <br>
> [1] <a href="https://mail.openvswitch.org/pipermail/ovs-discuss/2020-May/049994.html" rel="noreferrer" target="_blank">https://mail.openvswitch.org/pipermail/ovs-discuss/2020-May/049994.html</a> <br>
> <br>
> <br>
> In general, these flows should be per router instead of per<br>
> router port, since the nat addresses are not attached to any<br>
> router port. For distributed gateway ports, there will need<br>
> per-port flows to match<br>
> is_chassis_resident(gateway-chassis). I think this can be<br>
> handled by:<br>
> - priority X + 20 flows for each distributed gateway port<br>
> with is_chassis_resident(), reply ARP<br>
> - priority X + 10 flows for each distributed gateway port<br>
> without is_chassis_resident(), drop<br>
> - priority X flows for each router (no need to match<br>
> inport), reply ARP<br>
> <br>
> This way, there are N * (2D + 1) flows per router. N =<br>
> number of NAT IPs, D = number of distributed gateway ports.<br>
> This would optimize the above scenario where there is only 1<br>
> distributed gateway port but many regular router ports.<br>
> Thoughts?<br>
> <br>
> <br>
> We went ahead and added support for this topology in<br>
> ovn-kubernetes project in this commit<br>
> <a href="https://github.com/ovn-org/ovn-kubernetes/commit/edb24e6a71142f2e835b67b29c11e1688c645683" rel="noreferrer" target="_blank">https://github.com/ovn-org/ovn-kubernetes/commit/edb24e6a71142f2e835b67b29c11e1688c645683</a> <br>
> <br>
> Han, was curious to know if the above fix is in your radar? Thanks. <br>
> <br>
> The number of OpenFlow flows in each of the hypervisors is<br>
> insanely high and is consuming a lot of memory.<br>
> <br>
> Regards,<br>
> ~Girish<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> Thanks,<br>
> Han<br>
> <br>
> -- <br>
> You received this message because you are subscribed to the<br>
> Google Groups "ovn-kubernetes" group.<br>
> To unsubscribe from this group and stop receiving emails from<br>
> it, send an email to <a href="mailto:ovn-kubernetes%2Bunsubscribe@googlegroups.com" target="_blank">ovn-kubernetes+unsubscribe@googlegroups.com</a><br>
> <mailto:<a href="mailto:ovn-kubernetes%2Bunsubscribe@googlegroups.com" target="_blank">ovn-kubernetes+unsubscribe@googlegroups.com</a>>.<br>
> To view this discussion on the web visit<br>
> <a href="https://groups.google.com/d/msgid/ovn-kubernetes/CAAF2STTOrzx-zy48TKpbxx4yxxQ_X5bN05VPqBHA79gpCBQfwg%40mail.gmail.com" rel="noreferrer" target="_blank">https://groups.google.com/d/msgid/ovn-kubernetes/CAAF2STTOrzx-zy48TKpbxx4yxxQ_X5bN05VPqBHA79gpCBQfwg%40mail.gmail.com</a><br>
> <<a href="https://groups.google.com/d/msgid/ovn-kubernetes/CAAF2STTOrzx-zy48TKpbxx4yxxQ_X5bN05VPqBHA79gpCBQfwg%40mail.gmail.com?utm_medium=email&utm_source=footer" rel="noreferrer" target="_blank">https://groups.google.com/d/msgid/ovn-kubernetes/CAAF2STTOrzx-zy48TKpbxx4yxxQ_X5bN05VPqBHA79gpCBQfwg%40mail.gmail.com?utm_medium=email&utm_source=footer</a>>.<br>
> <br>
<br>
</blockquote></div>