[ovs-discuss] FW: OVS 2.9.0 native firewall drops empty payload TCP packets continued

Darrell Ball dlu998 at gmail.com
Fri May 3 22:07:19 UTC 2019 

Thanks for reconfirming Jing

Darrell

On Fri, May 3, 2019 at 3:02 PM Zhang, Jing C. (Nokia - CA/Ottawa) <
jing.c.zhang at nokia.com> wrote:

> The thing is, I don’t see empty TCP packet drops on DPDK computes, I
> nevertheless applied the patch HAN mentioned on DPDK computes, no
> difference.
>
>
>
> The issues we see is on OVS computes.
>
>
>
> Jing
>
>
>
> *From:* Darrell Ball <dlu998 at gmail.com>
> *Sent:* Friday, May 03, 2019 3:34 PM
> *To:* Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang at nokia.com>
> *Cc:* Han Zhou <zhouhan at gmail.com>; ovs-discuss at openvswitch.org
> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
>
>
>
>
> On Fri, May 3, 2019 at 10:44 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zhang at nokia.com> wrote:
>
>
>    1. The hybrid firewall refers to Linux bridge based firewall. To debug
>    the issue, we switch the neutron OVS agent to use native firewall.
>
>
>
> [securitygroup]
>
> #firewall_driver=iptables_hybrid
>
> firewall_driver=openvswitch
>
>
>
> # ovs-ofctl dump-flows br-int | grep ct_state
>
> cookie=0xddb977285e2ba9b6, duration=185.322s, table=71, n_packets=0,
> n_bytes=0, idle_age=185, priority=110,ct_state=+trk
> actions=ct_clear,resubmit(,71)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=75,ct_state=+est-rel-rpl,icmp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204,
> n_bytes=16642, idle_age=18, priority=77,ct_state=+est-rel-rpl,udp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=77,ct_state=+est-rel-rpl,tcp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=75,ct_state=+new-est,icmp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=204,
> n_bytes=16642, idle_age=18, priority=77,ct_state=+new-est,udp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=77,ct_state=+new-est,tcp,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=74,ct_state=+est-rel-rpl,ipv6,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=74,ct_state=+est-rel-rpl,ip,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=74,ct_state=+new-est,ipv6,reg5=0x1
> actions=resubmit(,73)
>
> cookie=0xddb977285e2ba9b6, duration=182.170s, table=72, n_packets=0,
> n_bytes=0, idle_age=184, priority=74,ct_state=+new-est,ip,reg5=0x1
> actions=resubmit(,73)
>
>
>
> My understanding is Centos 7 packs the OVS tree, that how conntrack is
> supported before kernel 4.3.
>
>
>
> https://cbs.centos.org/koji/buildinfo?buildID=24381
>
>
>
> I assumed you are using Linux tree OVS kernel module
>
>
>
>
>
>
>
>
>
>    1. I back-ported the patch pointed by Han to OVS v2.9.0, it does not
>    solve the packet drop on the OVS compute.
>
>
>
> Thanks for confirming
>
> We don't know your full topology, but if you want to send packets
> following a path that goes thru an OVS userspace datapath then
>
> that patch would be applicable.
>
> Did you apply the patch to ALL userspace dataspath instances that could be
> in you packet path ?
>
> What is the path followed in the problem case ?
>
>
>
>
>
>
>    1.
>
>
>
> The nature of the issue is same, OVS native firewall drops packets less
> than 60 bytes.
>
>
>
> Pls correct me and advise if the issue on OVS compute is fixable.
>
>
>
> You could check the OVS tree kernel module.
>
>
>
>
>
>
>
> JIng
>
>
>
> *From:* Darrell Ball <dlu998 at gmail.com>
> *Sent:* Friday, May 3, 2019 11:55 AM
> *To:* Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang at nokia.com>
> *Cc:* Han Zhou <zhouhan at gmail.com>; ovs-discuss at openvswitch.org
> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
> couple corrections inline
>
>
>
> On Fri, May 3, 2019 at 8:52 AM Darrell Ball <dlu998 at gmail.com> wrote:
>
>
>
>
>
> On Fri, May 3, 2019 at 8:29 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zhang at nokia.com> wrote:
>
>
>    1. This issue is with native OVS firewall where the data flows are
>    subject to conntrack rules, there is no issue for hybrid firewall
>
>
>
> 1/ Does 'native OVS firewall' mean either kernel datapath or userpace
> datapath ?
>
> 2/ Pls define 'hybrid datapath' in your context ?
>
>
>
> 2/ Pls define 'hybrid firewall' in your context ?
>
>
>
>
>
>
>    1.
>
>
>
>    1. Below is from DPDK compute:
>
>
>
> # ovs-vsctl --no-wait get Open_vSwitch . other_config
>
>
>
> 3/ dpdk is not initialized
>
>
>
> An info log is also present when dpdk is initialized: "DPDK Enabled -
> initialized"
>
>
>
> btw: '--no-wait' is needed for get commands
>
>
>
> btw: '--no-wait' is NOT needed for get commands
>
>
>
>
>
>
>
> # ovs-vsctl -- list bridge br-int | grep datapath
>
> datapath_id         : "00001a9b5b9ec94e"
>
> datapath_type       : netdev
>
> datapath_version    : "<built-in>"
>
>
>
> 4/ You are using userspace datapath on this particular node without dpdk
> support
>
>     Is that intentional ?
>
>
>
>
>
>
>
> *From:* Darrell Ball <dlu998 at gmail.com>
> *Sent:* Friday, May 3, 2019 11:24 AM
> *To:* Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang at nokia.com>
> *Cc:* Han Zhou <zhouhan at gmail.com>; ovs-discuss at openvswitch.org
> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
> The node you are displaying below is running kernel datapath
>
>
>
> fyi: The fix Han pointed you to is for userspace datapath/conntrack
>
>
>
>
>
>
>
> On Fri, May 3, 2019 at 8:14 AM Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zhang at nokia.com> wrote:
>
> We have both OVS and OVS-dpdk computes.
>
>
>
> Below is from OVS compute:
>
>
>
> # ovs-vsctl --no-wait get Open_vSwitch . other_config
>
> {}
>
> # ovs-vsctl -- list bridge br-int | grep datapath
>
> datapath_id         : "0000aaf62aaf3546"
>
> datapath_type       : system
>
> datapath_version    : "<unknown>"
>
>
>
> *From:* Darrell Ball <dlu998 at gmail.com>
> *Sent:* Friday, May 3, 2019 12:19 AM
> *To:* Han Zhou <zhouhan at gmail.com>; Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zhang at nokia.com>; ovs-discuss at openvswitch.org
> *Subject:* Re: FW: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
> What do the following commands yield ?
>
>
>
> sudo ovs-vsctl -- get bridge <bridge name> datapath_type
>
>
>
> sudo ovs-vsctl --no-wait get Open_vSwitch . other_config
>
>
>
>
>
>
>
> *From: *<ovs-discuss-bounces at openvswitch.org> on behalf of Han Zhou <
> zhouhan at gmail.com>
> *Date: *Thursday, May 2, 2019 at 7:12 PM
> *To: *"Zhang, Jing C. (Nokia - CA/Ottawa)" <jing.c.zhang at nokia.com>
> *Cc: *"ovs-discuss at openvswitch.org" <ovs-discuss at openvswitch.org>
> *Subject: *Re: [ovs-discuss] OVS 2.9.0 native firewall drops empty
> payload TCP packets continued
>
>
>
>
>
> On Thu, May 2, 2019 at 6:04 PM Zhang, Jing C. (Nokia - CA/Ottawa) <
> jing.c.zhang at nokia.com> wrote:
> >
> > We (our VNFs) continue to observe the same empty payload TCP (ACK)
> packet drop with native firewall (see original post below) after upgrading
> to Centos 7.6. This packet drop results in unacceptable TCP performance, by
> that native firewall still can not be enabled in product.
> >
> >
> https://mail.openvswitch.org/pipermail/ovs-discuss/2018-August/047263.html
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.openvswitch.org%2Fpipermail%2Fovs-discuss%2F2018-August%2F047263.html&data=02%7C01%7Cdball%40vmware.com%7Cd358d5a23b2640ebd28708d6cf6cc524%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636924463524642583&sdata=ihTE%2BcOA9d8yNflCbqJYHXOJWhFuqvq4yJmu7H9lGwo%3D&reserved=0>
> >
> > $ uname -a
> > Linux overcloud-sriovperformancecompute-0 3.10.0-957.10.1.el7.x86_64 #1
> SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> >
> > $ ovs-vswitchd --version
> > ovs-vswitchd (Open vSwitch) 2.9.0
> > DPDK 17.11.0
> >
> > The scenario: OVS provider VLAN network is used
> >
> >
> > in physical interface of ovs compute zero length tcp payload packet
> arrives as padded to 64 bytes (and vlan tag is included in ethernet header)
> > same packet does not appear anymore in the tcpdump taken from tap-xyz
> interface (once vlan tag is removed and packet is cut by 4 bytes to 60
> bytes)
> >
> >
> > Tcpdump on physical port:
> >
> > 00:25:24.468423 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
> 6893, offset 0, flags [DF], proto TCP (6), length 2656)
> >     192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> > 00:25:24.468593 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
> 56318, offset 0, flags [DF], proto TCP (6), length 40)
> >     192.168.10.60.57576 > 192.168.10.52.80: Flags [.], cksum 0x1d34
> (correct), seq 78, ack 11577, win 391, length 0
> > 00:25:24.475848 fa:16:3e:ff:dd:29 > fa:16:3e:d7:bb:2c, ethertype 802.1Q
> (0x8100), length 60: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
> 56319, offset 0, flags [DF], proto TCP (6), length 40)
> >     192.168.10.60.57576 > 192.168.10.52.80: Flags [F.], cksum 0x1d33
> (correct), seq 78, ack 11577, win 391, length 0
> > 00:25:24.480337 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype 802.1Q
> (0x8100), length 2674: vlan 3837, p 0, ethertype IPv4, (tos 0x0, ttl 64, id
> 6894, offset 0, flags [DF], proto TCP (6), length 2656)
> >     192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> >
> > Tcpdump on vm tap interface:
> >
> > 00:25:24.468419 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4
> (0x0800), length 2670: (tos 0x0, ttl 64, id 6893, offset 0, flags [DF],
> proto TCP (6), length 2656)
> >     192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> > 00:25:24.480331 fa:16:3e:d7:bb:2c > fa:16:3e:ff:dd:29, ethertype IPv4
> (0x0800), length 2670: (tos 0x0, ttl 64, id 6894, offset 0, flags [DF],
> proto TCP (6), length 2656)
> >     192.168.10.52.80 > 192.168.10.60.57576: Flags [P.], cksum 0xa013
> (incorrect -> 0x772d), seq 8961:11577, ack 78, win 210, length 2616: HTTP
> >
> > Very straightforward to see the issue:
> >
> >
> > Configure neutron OVS agent to use native firewall
> > Create a pair of VMs on separate computes on provider vLAN
> > Disable TCP timestamp inside the VMs
> > Exchange TCP traffic between the VMs, e.g. http download.
> > Tcpdump on the physical and vm port, and compare.
> >
> >
> > I wonder why such obvious issue is not widely discussed?
> >
> > Jing
> >
>
>
>
> Maybe it's fixed by:
>
>
> https://github.com/openvswitch/ovs/commit/9171c63532ee9cbc63bb8cfae364ab071f44389b
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenvswitch%2Fovs%2Fcommit%2F9171c63532ee9cbc63bb8cfae364ab071f44389b&data=02%7C01%7Cdball%40vmware.com%7Cd358d5a23b2640ebd28708d6cf6cc524%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636924463524652585&sdata=OLr263fKPdgKA5nga2UlGNF0WtB6srXSIg9a7aHkf44%3D&reserved=0>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20190503/dd073327/attachment-0001.html>

More information about the discuss mailing list