<div dir="ltr">Thanks Han for the update.<div><br></div><div>Regards,</div><div>~Girish </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 15, 2020 at 12:55 PM Han Zhou <<a href="mailto:zhouhan@gmail.com">zhouhan@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sorry Girish, I can't promise for now. I will see if I have time in the next couple of weeks, but welcome anyone to volunteer on this if it is urgent.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 15, 2020 at 10:56 AM Girish Moodalbail <<a href="mailto:gmoodalbail@gmail.com" target="_blank">gmoodalbail@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello Han,</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 3, 2020 at 9:39 PM Han Zhou <<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 3, 2020 at 7:16 PM Girish Moodalbail <<a href="mailto:gmoodalbail@gmail.com" target="_blank">gmoodalbail@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><font face="arial, sans-serif">Hello all,</font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">While working on an extension, </font>see the diagram below, <span style="font-family:arial,sans-serif">to the existing OVN logical topology for the ovn-kubernetes project, I am seeing an explosion of the "Reply to ARP requests" logical flows in the `lr_in_ip_input` table for the distributed router (ovn_cluster_router) configured with gateway port (rtol-LS)</span></div><font face="monospace"><div><font face="monospace"><br></font></div> internet <br> ---------+--------------> <br> | <br> | <br> +----------localnet-port---------+ <br> |LS | <br> +-----------------ltor-LS--------+ <br> | <br> | <br> +---------------------rtol-LS------------+<br> | ovn_cluster_router |<br> | (Distributed Router) |<br> +-rtos-ls0------rtos-ls1--------rtos-ls2-+<br> | | | <br> | | | <br>+-----+-+ +----+--+ +-----+-+ <br>| LS0 | | LS1 | | LS2 | <br>+-+-----+ +-+-----+ +-+-----+ <br> | | | <br> p0 p1 p2 <br> IA0 IA1 IA2 <br> EA0 EA1 EA2 </font><div><font face="monospace">(Node0) (Node1) (Node2)</font><div><font face="monospace"><br></font></div><div><font face="arial, sans-serif">In the topology above, each of the three logical switch port has an internal address of IAx and an external address of EAx (dnat_and_snat IP). They are all bound to their respective nodes (Nodex). A packet from `p0` heading towards the internet will be SNAT'ed to EA0 on the local hypervisor and then sent out through the LS's localnet-port on that hypervisor. Basically, they are configured for distributed NATing.</font></div></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">I am seeing interesting "Reply to ARP requests" flows for arp.tpa set to "EAX". Flows are like this:</font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">For EA0</font></div><div><div><font face="arial, sans-serif">priority=90, match=(inport == "rtos-ls0" && arp.tpa == EA0 && arp.op == 1), action=(/* ARP reply */)</font></div><div></div></div><div><font face="arial, sans-serif">priority=90, match=(inport == "rtos-ls1" && arp.tpa == EA0 && arp.op == 1), action=(/* ARP reply */)</font></div><div><font face="arial, sans-serif">priority=90, match=(inport == "rtos-ls2" && arp.tpa == EA0 && arp.op == 1), action=(/* ARP reply */)</font></div><div><font face="arial, sans-serif"><br></font></div><div><div><font face="arial, sans-serif">For EA1</font></div><div><font face="arial, sans-serif">priority=90, match=(inport == "rtos-ls0" && arp.tpa == EA1 && arp.op == 1), action=(/* ARP reply */)</font></div><div><div><font face="arial, sans-serif">priority=90, match=(inport == "rtos-ls1" && arp.tpa == EA0 && arp.op == 1), action=(/* ARP reply */)</font></div><div></div></div><div><font face="arial, sans-serif">priority=90, match=(inport == "rtos-ls2" && arp.tpa == EA1 && arp.op == 1), action=(/* ARP reply */)</font></div></div><div><br></div><div>Similarly, for EA2.</div><div><br>So, we have N * N "Reply to ARP requests" flows for N nodes each with 1 dnat_and_snat ip. </div><div>This is causing scale issues.</div><div><br></div><div>If you look at the flows for `EA0`, i am confused as to why is it needed?</div><div><ol><li>When will one see an ARP request for the EA0 from any of the LS{0,1,2}'s logical switch port.<br></li><li>If it is needed at all, can't we just remove the `inport` thing altogether since the flow is configured for every port of logical router port except for the distributed gateway port rtol-LS. For this port, we could add an higher priority rule with action set to `next`.</li><li>Say, we don't need east-west NAT connectivity. Is there a way to make these ARPs be learnt dynamically, like we are doing for join and external logical switch (the other thread [1]).</li></ol><div>Regards,</div></div><div>~Girish</div><div><br></div><div>[1] <a href="https://mail.openvswitch.org/pipermail/ovs-discuss/2020-May/049994.html" target="_blank">https://mail.openvswitch.org/pipermail/ovs-discuss/2020-May/049994.html</a> </div></div></blockquote><div><br></div><div>In general, these flows should be per router instead of per router port, since the nat addresses are not attached to any router port. For distributed gateway ports, there will need per-port flows to match is_chassis_resident(gateway-chassis). I think this can be handled by:</div><div>- priority X + 20 flows for each distributed gateway port with is_chassis_resident(), reply ARP<br></div><div>- priority X + 10 flows for each distributed gateway port without is_chassis_resident(), drop</div><div>- priority X flows for each router (no need to match inport), reply ARP</div><div><br></div><div>This way, there are N * (2D + 1) flows per router. N = number of NAT IPs, D = number of distributed gateway ports. This would optimize the above scenario where there is only 1 distributed gateway port but many regular router ports. Thoughts?</div></div></div></blockquote><div><br></div><div>We went ahead and added support for this topology in ovn-kubernetes project in this commit</div><div><a href="https://github.com/ovn-org/ovn-kubernetes/commit/edb24e6a71142f2e835b67b29c11e1688c645683" target="_blank">https://github.com/ovn-org/ovn-kubernetes/commit/edb24e6a71142f2e835b67b29c11e1688c645683</a> <br></div><div><br></div><div>Han, was curious to know if the above fix is in your radar? Thanks. </div><div><br></div><div>The number of OpenFlow flows in each of the hypervisors is insanely high and is consuming a lot of memory.</div><div><br></div><div>Regards,</div><div>~Girish</div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>Thanks,</div><div>Han<br></div></div></div>
</blockquote></div></div>
<p></p>
-- <br>
You received this message because you are subscribed to the Google Groups "ovn-kubernetes" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:ovn-kubernetes+unsubscribe@googlegroups.com" target="_blank">ovn-kubernetes+unsubscribe@googlegroups.com</a>.<br>
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/ovn-kubernetes/CAAF2STTOrzx-zy48TKpbxx4yxxQ_X5bN05VPqBHA79gpCBQfwg%40mail.gmail.com?utm_medium=email&utm_source=footer" target="_blank">https://groups.google.com/d/msgid/ovn-kubernetes/CAAF2STTOrzx-zy48TKpbxx4yxxQ_X5bN05VPqBHA79gpCBQfwg%40mail.gmail.com</a>.<br>
</blockquote></div>
</blockquote></div>