<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
        {font-family:SimSun;
        panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
        {font-family:"\@SimSun";
        panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
        {mso-style-name:msonormal;
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
p.gmail-m-5984112266810193446msolistparagraph, li.gmail-m-5984112266810193446msolistparagraph, div.gmail-m-5984112266810193446msolistparagraph
        {mso-style-name:gmail-m_-5984112266810193446msolistparagraph;
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
p.gmail-m-5984112266810193446gmail-m-3642746635662152373gmail-m8687681736471031303msolistparagraph, li.gmail-m-5984112266810193446gmail-m-3642746635662152373gmail-m8687681736471031303msolistparagraph, div.gmail-m-5984112266810193446gmail-m-3642746635662152373gmail-m8687681736471031303msolistparagraph
        {mso-style-name:gmail-m_-5984112266810193446gmail-m-3642746635662152373gmail-m8687681736471031303msolistparagraph;
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
span.EmailStyle20
        {mso-style-type:personal-reply;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-family:"Calibri",sans-serif;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
/* List Definitions */
@list l0
        {mso-list-id:509223771;
        mso-list-template-ids:297049358;}
@list l0:level1
        {mso-level-start-at:2;
        mso-level-tab-stop:.5in;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l1
        {mso-list-id:966282560;
        mso-list-template-ids:-1919618816;}
@list l2
        {mso-list-id:1171218278;
        mso-list-template-ids:667158626;}
@list l2:level1
        {mso-level-start-at:2;
        mso-level-tab-stop:.5in;
        mso-level-number-position:left;
        text-indent:-.25in;}
@list l3
        {mso-list-id:1218854782;
        mso-list-template-ids:207151654;}
@list l4
        {mso-list-id:1449426575;
        mso-list-template-ids:-236391586;}
@list l5
        {mso-list-id:1821580585;
        mso-list-template-ids:-954009010;}
@list l5:level1
        {mso-level-start-at:2;
        mso-level-tab-stop:.5in;
        mso-level-number-position:left;
        text-indent:-.25in;}
ol
        {margin-bottom:0in;}
ul
        {margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">The thing is, I don’t see empty TCP packet drops on DPDK computes, I nevertheless applied the patch HAN mentioned on DPDK computes, no difference.<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">The issues we see is on OVS computes.<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal">Jing<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<p class="MsoNormal"><b>From:</b> Darrell Ball &lt;dlu998@gmail.com&gt; <br>
<b>Sent:</b> Friday, May 03, 2019 3:34 PM<br>
<b>To:</b> Zhang, Jing C. (Nokia - CA/Ottawa) &lt;jing.c.zhang@nokia.com&gt;<br>
<b>Cc:</b> Han Zhou &lt;zhouhan@gmail.com&gt;; ovs-discuss@openvswitch.org<br>
<b>Subject:</b> Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets continued<o:p></o:p></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, May 3, 2019 at 10:44 AM Zhang, Jing C. (Nokia - CA/Ottawa) &lt;<a href="mailto:jing.c.zhang@nokia.com">jing.c.zhang@nokia.com</a>&gt; wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<ol start="1" type="1">
<li class="gmail-m-5984112266810193446msolistparagraph" style="mso-list:l1 level1 lfo1">
<span lang="EN-CA">The hybrid firewall refers to Linux bridge based firewall. To debug the issue, we switch the neutron OVS agent to use native firewall.<o:p></o:p></span></li></ol>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">[securitygroup]<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">#firewall_driver=iptables_hybrid<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">firewall_driver=openvswitch<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA"># ovs-ofctl dump-flows br-int | grep ct_state<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">cookie=0xddb977285e2ba9b6, duration=185.322s, table=71, n_packets=0, n_bytes=0, idle_age=185, priority=110,ct_state=&#43;trk actions=ct_clear,resubmit(,71)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, idle_age=184, priority=75,ct_state=&#43;est-rel-rpl,icmp,reg5=0x1 actions=resubmit(,73)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204, n_bytes=16642, idle_age=18, priority=77,ct_state=&#43;est-rel-rpl,udp,reg5=0x1 actions=resubmit(,73)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, idle_age=184, priority=77,ct_state=&#43;est-rel-rpl,tcp,reg5=0x1 actions=resubmit(,73)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, idle_age=184, priority=75,ct_state=&#43;new-est,icmp,reg5=0x1 actions=resubmit(,73)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204, n_bytes=16642, idle_age=18, priority=77,ct_state=&#43;new-est,udp,reg5=0x1 actions=resubmit(,73)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, idle_age=184, priority=77,ct_state=&#43;new-est,tcp,reg5=0x1 actions=resubmit(,73)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, idle_age=184, priority=74,ct_state=&#43;est-rel-rpl,ipv6,reg5=0x1 actions=resubmit(,73)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, idle_age=184, priority=74,ct_state=&#43;est-rel-rpl,ip,reg5=0x1 actions=resubmit(,73)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, idle_age=184, priority=74,ct_state=&#43;new-est,ipv6,reg5=0x1 actions=resubmit(,73)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0, n_bytes=0, idle_age=184, priority=74,ct_state=&#43;new-est,ip,reg5=0x1 actions=resubmit(,73)<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">My understanding is Centos 7 packs the OVS tree, that how conntrack is supported before kernel 4.3.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA"><a href="https://cbs.centos.org/koji/buildinfo?buildID=24381" target="_blank">https://cbs.centos.org/koji/buildinfo?buildID=24381</a><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">I assumed you are using Linux tree OVS kernel module<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">&nbsp;<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<ol start="2" type="1">
<li class="gmail-m-5984112266810193446msolistparagraph" style="mso-list:l5 level1 lfo2">
<span lang="EN-CA">I back-ported the patch pointed by Han to OVS v2.9.0, it does not solve the packet drop on the OVS compute.<o:p></o:p></span></li></ol>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks for confirming<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">We don't know your full topology, but if you want to send packets following a path that goes thru an OVS userspace datapath then<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">that patch would be applicable.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Did you apply the patch to ALL userspace dataspath instances that could be in you packet path ?&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">What is the path followed in the problem case ?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">&nbsp;<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<ol start="2" type="1">
<li class="gmail-m-5984112266810193446msolistparagraph" style="mso-list:l0 level1 lfo3">
<span lang="EN-CA"><o:p>&nbsp;</o:p></span></li></ol>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">The nature of the issue is same, OVS native firewall drops packets less than 60 bytes.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">Pls correct me and advise if the issue on OVS compute is fixable.<o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">You could check the OVS tree kernel module.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<div>
<p class="MsoNormal">&nbsp;<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">JIng<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> Darrell Ball &lt;<a href="mailto:dlu998@gmail.com" target="_blank">dlu998@gmail.com</a>&gt;
<br>
<b>Sent:</b> Friday, May 3, 2019 11:55 AM<br>
<b>To:</b> Zhang, Jing C. (Nokia - CA/Ottawa) &lt;<a href="mailto:jing.c.zhang@nokia.com" target="_blank">jing.c.zhang@nokia.com</a>&gt;<br>
<b>Cc:</b> Han Zhou &lt;<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a>&gt;;
<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a><br>
<b>Subject:</b> Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets continued<span lang="EN-CA"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">couple corrections inline<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">On Fri, May 3, 2019 at 8:52 AM Darrell Ball &lt;<a href="mailto:dlu998@gmail.com" target="_blank">dlu998@gmail.com</a>&gt; wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">On Fri, May 3, 2019 at 8:29 AM Zhang, Jing C. (Nokia - CA/Ottawa) &lt;<a href="mailto:jing.c.zhang@nokia.com" target="_blank">jing.c.zhang@nokia.com</a>&gt; wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<ol start="1" type="1">
<li class="gmail-m-5984112266810193446gmail-m-3642746635662152373gmail-m8687681736471031303msolistparagraph" style="mso-list:l4 level1 lfo4">
<span lang="EN-CA">This issue is with native OVS firewall where the data flows are subject to conntrack rules, there is no issue for hybrid firewall<o:p></o:p></span></li></ol>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">1/ Does 'native OVS firewall' mean either kernel datapath or userpace datapath ?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">2/ Pls define 'hybrid datapath' in your context ?<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">2/ Pls define 'hybrid firewall' in your context ?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<ol start="1" type="1">
<li class="gmail-m-5984112266810193446gmail-m-3642746635662152373gmail-m8687681736471031303msolistparagraph" style="mso-list:l3 level1 lfo5">
<span lang="EN-CA">&nbsp;<o:p></o:p></span></li></ol>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<ol start="2" type="1">
<li class="gmail-m-5984112266810193446gmail-m-3642746635662152373gmail-m8687681736471031303msolistparagraph" style="mso-list:l2 level1 lfo6">
<span lang="EN-CA">Below is from DPDK compute:<o:p></o:p></span></li></ol>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA"># ovs-vsctl --no-wait get Open_vSwitch . other_config&nbsp;<o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">3/ dpdk is not initialized<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">An info log is also present when dpdk is initialized: &quot;DPDK Enabled - initialized&quot;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">btw: '--no-wait' is needed for get commands<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">btw: '--no-wait' is NOT needed for get commands<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA"># ovs-vsctl -- list bridge br-int | grep datapath<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">datapath_id&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : &quot;00001a9b5b9ec94e&quot;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">datapath_type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : netdev<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">datapath_version&nbsp;&nbsp;&nbsp; : &quot;&lt;built-in&gt;&quot;<o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">4/ You are using userspace datapath on this particular node without dpdk support<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp; &nbsp; Is that intentional ?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> Darrell Ball &lt;<a href="mailto:dlu998@gmail.com" target="_blank">dlu998@gmail.com</a>&gt;
<br>
<b>Sent:</b> Friday, May 3, 2019 11:24 AM<br>
<b>To:</b> Zhang, Jing C. (Nokia - CA/Ottawa) &lt;<a href="mailto:jing.c.zhang@nokia.com" target="_blank">jing.c.zhang@nokia.com</a>&gt;<br>
<b>Cc:</b> Han Zhou &lt;<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a>&gt;;
<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a><br>
<b>Subject:</b> Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets continued<span lang="EN-CA"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">The node you are displaying below is running kernel datapath<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">fyi: The fix Han pointed you to is for userspace datapath/conntrack<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">On Fri, May 3, 2019 at 8:14 AM Zhang, Jing C. (Nokia - CA/Ottawa) &lt;<a href="mailto:jing.c.zhang@nokia.com" target="_blank">jing.c.zhang@nokia.com</a>&gt; wrote:<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">We have both OVS and OVS-dpdk computes.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">Below is from OVS compute:<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA"># ovs-vsctl --no-wait get Open_vSwitch . other_config<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">{}<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA"># ovs-vsctl -- list bridge br-int | grep datapath<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">datapath_id&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : &quot;0000aaf62aaf3546&quot;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">datapath_type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; : system<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">datapath_version&nbsp;&nbsp;&nbsp; : &quot;&lt;unknown&gt;&quot;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> Darrell Ball &lt;<a href="mailto:dlu998@gmail.com" target="_blank">dlu998@gmail.com</a>&gt;
<br>
<b>Sent:</b> Friday, May 3, 2019 12:19 AM<br>
<b>To:</b> Han Zhou &lt;<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a>&gt;; Zhang, Jing C. (Nokia - CA/Ottawa) &lt;<a href="mailto:jing.c.zhang@nokia.com" target="_blank">jing.c.zhang@nokia.com</a>&gt;;
<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a><br>
<b>Subject:</b> Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets continued<span lang="EN-CA"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">What do the following commands yield ?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">sudo ovs-vsctl -- get bridge &lt;bridge name&gt; datapath_type<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">sudo ovs-vsctl --no-wait get Open_vSwitch . other_config<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-CA">&nbsp;<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<span lang="EN-CA"><o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">&lt;<a href="mailto:ovs-discuss-bounces@openvswitch.org" target="_blank">ovs-discuss-bounces@openvswitch.org</a>&gt; on behalf of Han Zhou &lt;<a href="mailto:zhouhan@gmail.com" target="_blank">zhouhan@gmail.com</a>&gt;<br>
<b>Date: </b>Thursday, May 2, 2019 at 7:12 PM<br>
<b>To: </b>&quot;Zhang, Jing C. (Nokia - CA/Ottawa)&quot; &lt;<a href="mailto:jing.c.zhang@nokia.com" target="_blank">jing.c.zhang@nokia.com</a>&gt;<br>
<b>Cc: </b>&quot;<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a>&quot; &lt;<a href="mailto:ovs-discuss@openvswitch.org" target="_blank">ovs-discuss@openvswitch.org</a>&gt;<br>
<b>Subject: </b>Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty payload TCP packets continued</span><span lang="EN-CA"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<span lang="EN-CA"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
<br>
On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) &lt;<a href="mailto:jing.c.zhang@nokia.com" target="_blank">jing.c.zhang@nokia.com</a>&gt; wrote:<br>
&gt;<br>
&gt; We (our VNFs) continue to observe the same empty payload TCP (ACK) packet drop with native firewall (see original post below) after upgrading to Centos 7.6. This packet drop results in unacceptable TCP performance, by that native firewall still can not be
 enabled in product.<br>
&gt; &nbsp;<br>
&gt; <a href="https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.openvswitch.org%2Fpipermail%2Fovs-discuss%2F2018-August%2F047263.html&amp;data=02%7C01%7Cdball%40vmware.com%7Cd358d5a23b2640ebd28708d6cf6cc524%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636924463524642583&amp;sdata=ihTE%2BcOA9d8yNflCbqJYHXOJWhFuqvq4yJmu7H9lGwo%3D&amp;reserved=0" target="_blank">
https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html</a><br>
&gt; &nbsp;<br>
&gt; $ uname -a<br>
&gt; Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux<br>
&gt; &nbsp;<br>
&gt; $ ovs-vswitchd --version<br>
&gt; ovs-vswitchd (Open vSwitch) 2.9.0<br>
&gt; DPDK 17.11.0<br>
&gt; &nbsp;<br>
&gt; The scenario: OVS provider VLAN network is used<br>
&gt; &nbsp;<br>
&gt;<br>
&gt; in physical interface of ovs compute zero length tcp payload packet arrives as padded to 64 bytes (and vlan tag is included in ethernet header)<br>
&gt; same packet does not appear anymore in the tcpdump taken from tap-xyz interface (once vlan tag is removed and packet is cut by 4 bytes to 60 bytes)<br>
&gt;<br>
&gt; &nbsp;<br>
&gt; Tcpdump on physical port:<br>
&gt; &nbsp;<br>
&gt; 00:25:24.468423 fa:16:3e:d7:bb:2c &gt; fa:16:3e:ff:dd:29, ethertype 802.1Q (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 6893, offset 0, flags [DF], proto TCP (6), length 2656)<br>
&gt; &nbsp; &nbsp; 192.168.10.52.80 &gt; 192.168.10.60.57576: Flags [P.], cksum 0xa013 (incorrect -&gt; 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP<br>
&gt; 00:25:24.468593 fa:16:3e:ff:dd:29 &gt; fa:16:3e:d7:bb:2c, ethertype 802.1Q (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 56318, offset 0, flags [DF], proto TCP (6), length 40)<br>
&gt; &nbsp; &nbsp; 192.168.10.60.57576 &gt; 192.168.10.52.80: Flags [.], cksum 0x1d34 (correct), seq 78, ack 11577, win 391, length 0<br>
&gt; 00:25:24.475848 fa:16:3e:ff:dd:29 &gt; fa:16:3e:d7:bb:2c, ethertype 802.1Q (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 56319, offset 0, flags [DF], proto TCP (6), length 40)<br>
&gt; &nbsp; &nbsp; 192.168.10.60.57576 &gt; 192.168.10.52.80: Flags [F.], cksum 0x1d33 (correct), seq 78, ack 11577, win 391, length 0<br>
&gt; 00:25:24.480337 fa:16:3e:d7:bb:2c &gt; fa:16:3e:ff:dd:29, ethertype 802.1Q (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 6894, offset 0, flags [DF], proto TCP (6), length 2656)<br>
&gt; &nbsp; &nbsp; 192.168.10.52.80 &gt; 192.168.10.60.57576: Flags [P.], cksum 0xa013 (incorrect -&gt; 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP<br>
&gt; &nbsp;<br>
&gt; Tcpdump on vm tap interface:<br>
&gt; &nbsp;<br>
&gt; 00:25:24.468419 fa:16:3e:d7:bb:2c &gt; fa:16:3e:ff:dd:29, ethertype IPv4 (0x0800), length 2670: (tos 0x0, ttl 64, id 6893, offset 0, flags [DF], proto TCP (6), length 2656)<br>
&gt; &nbsp; &nbsp; 192.168.10.52.80 &gt; 192.168.10.60.57576: Flags [P.], cksum 0xa013 (incorrect -&gt; 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP<br>
&gt; 00:25:24.480331 fa:16:3e:d7:bb:2c &gt; fa:16:3e:ff:dd:29, ethertype IPv4 (0x0800), length 2670: (tos 0x0, ttl 64, id 6894, offset 0, flags [DF], proto TCP (6), length 2656)<br>
&gt; &nbsp; &nbsp; 192.168.10.52.80 &gt; 192.168.10.60.57576: Flags [P.], cksum 0xa013 (incorrect -&gt; 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP<br>
&gt; &nbsp;<br>
&gt; Very straightforward to see the issue:<br>
&gt; &nbsp;<br>
&gt;<br>
&gt; Configure neutron OVS agent to use native firewall<br>
&gt; Create a pair of VMs on separate computes on provider vLAN<br>
&gt; Disable TCP timestamp inside the VMs<br>
&gt; Exchange TCP traffic between the VMs, e.g. http download.<br>
&gt; Tcpdump on the physical and vm port, and compare.<br>
&gt;<br>
&gt; &nbsp;<br>
&gt; I wonder why such obvious issue is not widely discussed?<br>
&gt; &nbsp;<br>
&gt; Jing<br>
&gt; <span lang="EN-CA"><o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<span lang="EN-CA"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Maybe it's fixed by:
<span lang="EN-CA"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenvswitch%2Fovs%2Fcommit%2F9171c63532ee9cbc63bb8cfae364ab071f44389b&amp;data=02%7C01%7Cdball%40vmware.com%7Cd358d5a23b2640ebd28708d6cf6cc524%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636924463524652585&amp;sdata=OLr263fKPdgKA5nga2UlGNF0WtB6srXSIg9a7aHkf44%3D&amp;reserved=0" target="_blank">https://github.com/openvswitch/ovs/commit/9171c63532ee9cbc63bb8cfae364ab071f44389b</a><span lang="EN-CA"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<span lang="EN-CA"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>