<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 28, 2017 at 12:52 PM, Darrell Ball <span dir="ltr"><<a href="mailto:dball@vmware.com" target="_blank">dball@vmware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="m_5602431797921295604WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-family:Calibri;color:black">From:
</span></b><span style="font-family:Calibri;color:black">Hui Xiang <<a href="mailto:xianghuir@gmail.com" target="_blank">xianghuir@gmail.com</a>><br>
<b>Date: </b>Thursday, July 27, 2017 at 8:10 PM</span></p><div><div class="h5"><br>
<b>To: </b>Darrell Ball <<a href="mailto:dball@vmware.com" target="_blank">dball@vmware.com</a>><br>
<b>Cc: </b>"<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a>" <<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a>><br>
<b>Subject: </b>Re: [ovs-discuss] OVS-DPDK IP fragmentation require<u></u><u></u></div></div><p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
<div><div><div class="h5">
<p class="MsoNormal" style="margin-left:.5in">On Fri, Jul 28, 2017 at 10:54 AM, Darrell Ball <<a href="mailto:dball@vmware.com" target="_blank">dball@vmware.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">
<span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:1.0in">
<b><span style="font-family:Calibri;color:black">From: </span></b><span style="font-family:Calibri;color:black">Hui Xiang <<a href="mailto:xianghuir@gmail.com" target="_blank">xianghuir@gmail.com</a>><br>
<b>Date: </b>Thursday, July 27, 2017 at 6:59 PM<br>
<span class="m_5602431797921295604gmail-"><b>To: </b>Darrell Ball <<a href="mailto:dball@vmware.com" target="_blank">dball@vmware.com</a>></span><br>
<span class="m_5602431797921295604gmail-"><b>Cc: </b>"<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a>" <<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a>></span><br>
<span class="m_5602431797921295604gmail-"><b>Subject: </b>Re: [ovs-discuss] OVS-DPDK IP fragmentation require</span></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">
<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:1.0in">
<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:1.0in">
On Fri, Jul 28, 2017 at 1:12 AM, Darrell Ball <<a href="mailto:dball@vmware.com" target="_blank">dball@vmware.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">
<span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in">
<span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:1.5in">
<b><span style="font-family:Calibri;color:black">From: </span></b><span style="font-family:Calibri;color:black">Hui Xiang <<a href="mailto:xianghuir@gmail.com" target="_blank">xianghuir@gmail.com</a>><br>
<b>Date: </b>Thursday, July 27, 2017 at 3:18 AM<br>
<b>To: </b>Darrell Ball <<a href="mailto:dball@vmware.com" target="_blank">dball@vmware.com</a>><br>
<b>Cc: </b>"<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a>" <<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a>><br>
<b>Subject: </b>Re: [ovs-discuss] OVS-DPDK IP fragmentation require</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
<u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
Blow is the diagram (using OVS-DPDK):<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
1. For packets coming to vm1 from internet where could have MTU 1500, there could be including some fragmented packets,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
how does the ALC/Security groups handle these fragmented packets? do nothing and pass it next which may pass the packets<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
should be dropped or any special handling?<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in">
Lets assume the fragments get thru. the physical switch and/or firewall.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in">
Are you using DPDK in GW and using OVS kernel datapath in br-int where you apply ACL/Security groups policy ?<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:1.0in">
All are using DPDK, the ACL/Security groups policy said is OVS-DPDK conntrack implementation.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.0in">
With the case that we should have dropped some packets by creating special security group rules, but now due to they are fragmented and get thru by default, this is not what is expected.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
I would check your configuration.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
The dpdk connection tracker marks fragments as ‘invalid’ today and your rules should drop ‘invalid’.<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:.5in">OK, thanks. here are the two scenarios we are discussing: <u></u><u></u></p>
</div>
</div></div><div><div><div class="h5">
<p class="m_5602431797921295604MsoListParagraph" style="margin-left:.75in">
<u></u><span>1.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>For packets out from vms, use Jumbo Frame supported physical switches/routers within OpenStack cloud and enable it in all OVS-DPDK or do not allow application to send large frames.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div></div><p class="MsoNormal">Try to use jumbo frames for performance reasons.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">On going out, if you get fragmentation done in HW at the physical switches, then<u></u><u></u></p>
<p class="m_5602431797921295604MsoListParagraph"><u></u><span>1)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>If it could go back into one of your dpdk networks, then encourage using smaller packets<u></u><u></u></p>
<p class="m_5602431797921295604MsoListParagraph"><u></u><span>2)<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>If it goes somewhere else, then it does not matter, keep bigger packets<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.25in">Are you sure the physical switches do not support jumbo frames?<u></u><u></u></p>
<p class="m_5602431797921295604MsoListParagraph">Maybe it is just a config. change fix there.<u></u><u></u></p>
<p class="MsoNormal"><u></u> </p></div></div></div></div></div></div></blockquote><div>Few physical switches in my lab probably just support max MTU 2000.. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="white" lang="EN-US" link="blue" vlink="purple"><div class="m_5602431797921295604WordSection1"><div><div><div><div><p class="MsoNormal"><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div><span class="">
<div>
<p class="MsoNormal" style="margin-left:.5in">2. For packets coming from internet to OVS-DPDK, fragmented packets could be arrived, they are all dropped due to marks as 'invalid'.<u></u><u></u></p>
</div>
</span><div><span class="">
<p class="MsoNormal" style="margin-left:.5in"> With above analysis, if these fragments are marked as 'invalid' and being dropped, the best way I can think about is to not use security group in OVS-DPDK if there could be fragments generated.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</span><p class="MsoNormal">If you already trust what gets to GW because you have a HW firewall, yes<u></u><u></u></p>
<p class="MsoNormal">This assumes internally generated is always safe.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Otherwise, you want to keep security groups and ‘encourage’ no fragments coming in, if you can<u></u><u></u></p>
<p class="MsoNormal">‘Encourage’ can be done by dropping and triggering checking why the fragments got generated in the first place<u></u><u></u></p>
<p class="MsoNormal">Fragments may also indicate an exploit attempt, in which case, dropping is good.</p></div></div></div></div></div></div></blockquote><div>Thanks Darrell, yep these are the solutions so far. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="white" lang="EN-US" link="blue" vlink="purple"><div class="m_5602431797921295604WordSection1"><div><div><div><div><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div><span class="">
<div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Please correct me if I misunderstand anything.<u></u><u></u></p>
</div>
</span><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<div><span class="">
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
2. For packets egress from vm1, if all internal physical switch support Jumbo Frame, that's fine, but if there are some physical swithes<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
just support 1500/2000 MTU, then fragmented packets generated again. The ACL/Security groups face problem as item 1 as well.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.5in">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.5in">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in">
For packets that reach the physical switches on the way out, then the decision how to handle them is at the physical switch/router<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in">
The packets may be fragmented at this point; depending on the switch; there will be HW firewall policies to contend with, so depends.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.0in">
<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:1.0in">
Here, again what I mean is the packets are fragmented by the physical switch/router, and they are switching/routing to a next node where has the OVS-DPDK set with security group, and OVS-DPDK may let them thru with ignoring the security group rules. <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
Sorry, you lost me a bit here; in point ‘2’ above you said packets are going from vm1 to internet and are fine until they hit the physical switch<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
Where you are assuming they are fragmented because the mtu is lower.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
If this is not going to the internet but rather another set of nodes running dpdk, then this is another variation of ‘1’ and hence we don’t<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">
need to discuss it.<u></u><u></u></p>
</div>
</span><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
<img border="0" width="472" height="472" id="m_5602431797921295604gmail-m_-8924721896389721608m_5024491747323286630_x005f_x005f_x005f_x0000_i1025" src="cid:image001.png@01D30722.A7D4CCE0" alt="ine image 1"><u></u><u></u></p>
</div>
</div><div><div class="h5">
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
On Thu, Jul 27, 2017 at 2:49 PM, Darrell Ball <<a href="mailto:dball@vmware.com" target="_blank">dball@vmware.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-left:1.5in">
<span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.5in">
<span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:2.0in">
<b><span style="font-family:Calibri;color:black">From: </span></b><span style="font-family:Calibri;color:black">Hui Xiang <<a href="mailto:xianghuir@gmail.com" target="_blank">xianghuir@gmail.com</a>><br>
<b>Date: </b>Wednesday, July 26, 2017 at 9:43 PM<br>
<b>To: </b>Darrell Ball <<a href="mailto:dball@vmware.com" target="_blank">dball@vmware.com</a>><br>
<b>Cc: </b>"<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a>" <<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a>><br>
<b>Subject: </b>Re: [ovs-discuss] OVS-DPDK IP fragmentation require</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:2.0in">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:2.0in">
Thanks Darrell, comment inline.<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:2.0in">
<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:2.0in">
On Thu, Jul 27, 2017 at 12:08 PM, Darrell Ball <<a href="mailto:dball@vmware.com" target="_blank">dball@vmware.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-left:2.0in">
<span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:2.0in">
<span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:2.5in">
<b><span style="font-family:Calibri;color:black">From: </span></b><span style="font-family:Calibri;color:black"><<a href="mailto:ovs-discuss-bounces@openvswitch.org" target="_blank">ovs-discuss-bounces@<wbr>openvswitch.org</a>> on behalf of Hui Xiang <<a href="mailto:xianghuir@gmail.com" target="_blank">xianghuir@gmail.com</a>><br>
<b>Date: </b>Wednesday, July 26, 2017 at 7:47 PM<br>
<b>To: </b>"<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a>" <<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a>><br>
<b>Subject: </b>[ovs-discuss] OVS-DPDK IP fragmentation require</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:2.5in">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:2.5in">
Hi guys, <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:2.5in">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:2.5in">
Seems OVS-DPDK still missing IP fragmentation support, is there any schedule to have it?<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:2.0in">
OVS 2.9<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:2.5in">
I'm transferring to use OVN, but for those nodes which have external network connection, they may face this problem,<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:2.5in">
except to configure Jumbo frames, is there any other workaround?<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:2.0in">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:2.0in">
I am not clear on the situation however.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:2.0in">
You mention about configuring jumbo frames which means you can avoid the fragments by doing this ?<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:2.0in">
No, I can't guarantee that, only can do it inside OpenStack, it is limited. <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:2.0in">
If this is true, then this is the best way to proceed since performance will be better.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:2.0in">
What is wrong with jumbo frames ?<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="margin-left:2.0in">
It's good but it's limited can't be guaranteed, so I am asking is there any other way without IP fragmentation so far. <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.5in">
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.5in">
It sounds like you want to avoid IP fragmentation; so far so good.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.5in">
I am not sure I understand the whole picture though.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:1.5in">
Maybe you can describe what you see ?; maybe a simple diagram would help ?<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:2.0in">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:2.5in">
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:2.5in">
BR.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:2.5in">
Hui. <u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:2.0in">
<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:1.5in">
<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:1.0in">
<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</blockquote></div><br></div></div>