[ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Thu Nov 2 16:30:55 UTC 2017

From: <ovs-discuss-bounces at openvswitch.org> on behalf of Rohith Basavaraja <rohith.basavaraja at ericsson.com>
Date: Wednesday, November 1, 2017 at 10:28 PM
To: "ovs-discuss at openvswitch.org" <ovs-discuss at openvswitch.org>
Cc: Jan Scheurich <jan.scheurich at ericsson.com>
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi,

It’ been quite some time I raised this issue, thought will update the thread with our findings.
Following is the summary of our findings and analysis, and we think OVS user datapath conntrack implementation
Needs to be fixed otherwise some of the security group deployments mentioned below might fail.

Analysis/Findings
===============
Currently OVS kernel datapath implementation have the ct_state (conntrack state) semantics as described
In the following document.
http://www.iptables.info/en/connection-state.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.iptables.info_en_connection-2Dstate.html&d=DwMGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=cTQCN_hF1TN3jk_TJWYNMha6aUqSOURuVHEc5RCep1Y&s=yCJI8jfdp7_jUXkdLDee-j25N93R7kNawJUgmugy2-M&e=>[1]

OVS user datapath doesn’t follow above semantics and also the ct_state description in the  OVS specification
(http://openvswitch.org/support/dist-docs/ovs-fields.7.pdf)<https://urldefense.proofpoint.com/v2/url?u=http-3A__openvswitch.org_support_dist-2Ddocs_ovs-2Dfields.7.pdf-29&d=DwMGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih-uZnsw&m=cTQCN_hF1TN3jk_TJWYNMha6aUqSOURuVHEc5RCep1Y&s=eGwS0b14Sw_uvAefFrhp3aKmzckD4UOt2Y0nwXsPPS8&e=>[2] is not correct as explained below.

The main issue is when the conntrack  state “CS_ESTABLISHED” is set for a tracked flow. In the kernel datapath and iptables a tracked
flow moves to established state only once it sees a reply packet in the reverse direction.

The user-space conntrack, in contrast, moves a
tracked connection to established state as soon as a newly tracked connection is looked up the first time, irrespectively of the direction
of the packet.

[Darrell] are you sure ?
               The expectation is that the Userspace Datapath 2.6 behavior adheres to the OVS specification below.
               Please provide a test case that shows this is not the case; I would be interested ?

Finally, OVS specification [2] defines the “est” state as “est (0x02) Part of an existing connection. Set to 1 if this is a committed
connection”. This means that the tracked connection would move to established state when the ct(commit) action is executed and the
semantics don’t match either the kernel or user-space behaviour.

Because of the above difference some of the Security Group(SGs) use cases are failing for eg:
VMs that have SGs that shall not allow communication among them are not working when VMs are on the same compute node.

[Darrell] We had a lengthy offline email discussion about this from 8/23 to 8/30. The last few exchanges are below.

///////////////////////////////////////////////

From: Rohith Basavaraja <rohith.basavaraja at ericsson.com>
Date: Wednesday, August 30, 2017 at 9:05 AM
To: Darrel Ball <dball at vmware.com>
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrell,

Thanks, a lot for the help and sharing the useful information.

Thanks
Rohith

From: Darrell Ball <dball at vmware.com>
Date: Wednesday, 30 August 2017 at 9:18 PM
To: Rohith Basavaraja <rohith.basavaraja at ericsson.com>
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Rohith

From: Rohith Basavaraja <rohith.basavaraja at ericsson.com>
Date: Wednesday, August 30, 2017 at 3:46 AM
To: Darrel Ball <dball at vmware.com>
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrell,

For user datapath  do we have any other tools to dump conntrack entries
Other than  *ovs-appctl dpctl/dump-conntrack*

[Darrell]
Right now, we have

           "  dump-conntrack [DP] [zone=ZONE]  " \
               "display conntrack entries for ZONE\n"
           "  flush-conntrack [DP] [zone=ZONE] " \
               "delete all conntrack entries in ZONE\n"
           "  ct-stats-show [DP] [zone=ZONE] [verbose] " \
               "CT connections grouped by protocol\n"
           "  ct-bkts [DP] [gt=N] display connections per CT bucket\n"

This is from ./utilities/ovs-dpctl.c

For kernel datapath I see that we can use conntrack –L  to dump the entries,
Is conntrack tool is only for kernel datapath only?

[Darrell]
Yes

In general, any other conntrack commands or tools available for userdatapath?

[Darrell]
Right now, the ones mentioned above
Of course, the well known commands also tell lots about what is happening in conntrack indirectly
sudo ovs-ofctl dump-flows br0
sudo ovs-appctl dpif/dump-flows br0

Sorry for too many queries,  Pl let me know if it’s bothering you.

[Darrell]
No problem at all.

Thanks
Rohith

From: Rohith Basavaraja <rohith.basavaraja at ericsson.com>
Date: Wednesday, 30 August 2017 at 1:57 PM
To: Darrell Ball <dball at vmware.com>
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrell,

Thanks for the suggestions.

Thanks
Rohith

From: Darrell Ball <dball at vmware.com>
Date: Wednesday, 30 August 2017 at 12:17 PM
To: Rohith Basavaraja <rohith.basavaraja at ericsson.com>
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Ho Rohith

So the previous answer is the solution then; elaborating:

You want a committed connection VM2 -> VM1 (i.e. originated from VM2); this allows VM1 to send replies to VM2.

You want to prevent creating a committed connection from VM1 -> VM2
This can be done in various ways by using in_port, zones (per logical ports), dl_src, dl_dst etc

So traffic originated from VM1 -> VM2 will always be new

Thanks Darrell

From: Rohith Basavaraja <rohith.basavaraja at ericsson.com>
Date: Tuesday, August 29, 2017 at 11:19 PM
To: Darrel Ball <dball at vmware.com>
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrell,

Just to clarify following is the usecase.

  1.  VM1 can originate/initiate traffic to VM2
  2.  VM1 can receive traffic from VM2
  3.  VM2 should not receive any new connection from VM1
  4.  VM2 can originate/initiate traffic to VM1

Thanks
Rohith

From: Darrell Ball <dball at vmware.com>
Date: Wednesday, 30 August 2017 at 11:37 AM
To: Rohith Basavaraja <rohith.basavaraja at ericsson.com>
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Rohith

Just to confirm:
1/ VM1 can never send traffic to VM2 (originate or reply) ?
OR
2/ VM1 cannot originate traffic to VM2 but VM1 can send reply traffic to VM2. ?

I have now been assuming ‘2’ ?

Thanks Darrell

///////////////////////////////////////////////

Thanks
Rohith

From: Rohith Basavaraja <rohith.basavaraja at ericsson.com>
Date: Thursday, 24 August 2017 at 12:43 AM
To: Darrell Ball <dball at vmware.com>, "ovs-discuss at openvswitch.org" <ovs-discuss at openvswitch.org>
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Darrell,

Yes the expected outcome is to drop new or non related connection, and allow only related or established connections.

Just for the clarity adding  the details of the topology and pipeline rules.

Description about the topology
=========================

VM1 and VM4 VMs  are on same compute node but with different SGs.

For VM4, security rules configured are as below:
Egress/Ingress  Allow all

For VM1,
Egress  Allow all
Ingress  Allow only from VMs which are in same security group.

For above combination, all conntrack flows required (in tables 213, 214 on VM egress side and 243, 244) are properly programmed in the OVS.

For traffic sent from VM4 to VM1 , conntrack is allowing traffic which should have been dropped as the ingress for later is to be allowed only from the VMs of the same SG. For VM1 , conntrack is directly sending traffic to "ct_state==-new+est-rel-inv+trk " flow by-passing "ct_state=+new+trk" flow in the ingress direction.

Following is the pipe line rules details
==============================

VM4 is on 112/15: (dpdkvhostuser: configured_rx_queues=1, configured_tx_queues=1, mtu=2140, requested_rx_queues=1, requested_tx_queues=1)

VM1 is on 108/11: (dpdkvhostuser: configured_rx_queues=1, configured_tx_queues=1, mtu=2140, requested_rx_queues=1, requested_tx_queues=1)

VM4 IP: 172.20.1.113  MAC: fa:16:3e:55:9e:33

VM1 IP: 172.20.1.117 MAC : fa:16:3e:f7:72:d3

I am doing Ping from VM4 (172.20.1.113 ) to VM1 (172.20.1.117).

cookie=0x8000000, duration=2809.367s, table=0, n_packets=74, n_bytes=10426, priority=4,in_port=112,vlan_tci=0x0000/0x1fff actions=write_metadata:0x19f40000000000/0xffffff0000000001,goto_table:17

cookie=0x6900000, duration=2809.343s, table=17, n_packets=74, n_bytes=10426, priority=10,metadata=0x19f40000000000/0xffffff0000000000 actions=write_metadata:0x4019f40000000000/0xfffffffffffffffe,goto_table:211

cookie=0x6900000, duration=2809.313s, table=211, n_packets=54, n_bytes=8674, priority=61010,ip,metadata=0x19f40000000000/0x1fffff0000000000,dl_src=fa:16:3e:55:9e:33,nw_src=172.20.1.113 actions=goto_table:212

cookie=0x6900000, duration=15546.529s, table=212, n_packets=3669, n_bytes=361562, priority=61010,icmp actions=goto_table:213

cookie=0x6900000, duration=2809.308s, table=213, n_packets=54, n_bytes=8674, priority=61010,ip,metadata=0x19f40000000000/0x1fffff0000000000 actions=ct(table=214,zone=5021)

cookie=0x6900000, duration=15546.544s, table=214, n_packets=3660, n_bytes=367508, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,17)
cookie=0x6900001, duration=2809.304s, table=214, n_packets=2, n_bytes=180, priority=62015,ct_state=+inv+trk,metadata=0x19f40000000000/0x1fffff0000000000 actions=drop
cookie=0x6900000, duration=2809.295s, table=214, n_packets=10, n_bytes=908, priority=1000,ct_state=+new+trk,ip,metadata=0x19f40000000000/0x1fffff0000000000 actions=ct(commit,zone=5021),resubmit(,17)

cookie=0x6800001, duration=2807.340s, table=17, n_packets=72, n_bytes=10246, priority=10,metadata=0x4019f40000000000/0xffffff0000000000 actions=write_metadata:0xc019f40000000000/0xfffffffffffffffe,goto_table:60

cookie=0x6800000, duration=15546.596s, table=60, n_packets=262265, n_bytes=19604926, priority=0 actions=resubmit(,17)

cookie=0x8040000, duration=2807.338s, table=17, n_packets=70, n_bytes=9562, priority=10,metadata=0xc019f40000000000/0xffffff0000000000 actions=write_metadata:0xe019f4139d000000/0xfffffffffffffffe,goto_table:48

cookie=0x8500000, duration=15546.528s, table=48, n_packets=302458, n_bytes=34190652, priority=0 actions=resubmit(,49),resubmit(,50)

cookie=0x805139d, duration=2808.378s, table=50, n_packets=70, n_bytes=9562, priority=20,metadata=0x19f4139d000000/0x1fffffffff000000,dl_src=fa:16:3e:55:9e:33 actions=goto_table:51

cookie=0x803139d, duration=2818.232s, table=51, n_packets=34, n_bytes=4613, priority=20,metadata=0x139d000000/0xffff000000,dl_dst=fa:16:3e:f7:72:d3 actions=load:0x1a5300->NXM_NX_REG6[],resubmit(,220)

cookie=0x6900000, duration=2819.193s, table=220, n_packets=3455, n_bytes=207541, priority=6,reg6=0x1a5300 actions=load:0xe01a5300->NXM_NX_REG6[],write_metadata:0xe01a530000000000/0xfffffffffffffffe,goto_table:241

cookie=0x6900000, duration=2819.237s, table=241, n_packets=32, n_bytes=4851, priority=61010,ip,metadata=0x1a530000000000/0x1fffff0000000000,dl_dst=fa:16:3e:f7:72:d3,nw_dst=172.20.1.117 actions=goto_table:242

cookie=0x6900000, duration=15546.579s, table=242, n_packets=3738, n_bytes=368372, priority=61010,icmp actions=goto_table:243

cookie=0x6900000, duration=2819.235s, table=243, n_packets=32, n_bytes=4851, priority=61010,ip,metadata=0x1a530000000000/0x1fffff0000000000 actions=ct(table=244,zone=5021)

cookie=0x6900001, duration=2819.230s, table=244, n_packets=2, n_bytes=196, priority=50,ct_state=+new+trk,metadata=0x1a530000000000/0x1fffff0000000000 actions=drop

cookie=0x6900000, duration=15546.577s, table=244, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,220)
cookie=0x6900000, duration=15546.552s, table=244, n_packets=3819, n_bytes=431050, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,220)

cookie=0x8000007, duration=2819.193s, table=220, n_packets=107, n_bytes=9431, priority=7,reg6=0xe01a5300 actions=output:108

Thanks
Rohith

From: Darrell Ball <dball at vmware.com>
Date: Thursday, 24 August 2017 at 12:20 AM
To: Rohith Basavaraja <rohith.basavaraja at ericsson.com>, "ovs-discuss at openvswitch.org" <ovs-discuss at openvswitch.org>
Subject: Re: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi Rohith

I might have missed the alias earlier.

From the below o/p, I see the rule
cookie=0x6900000, duration=15546.577s, table=244, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,220)
not being hit.

I also see the rule
cookie=0x6900001, duration=2819.230s, table=244, n_packets=2, n_bytes=196, priority=50, ct_state=+new+trk,metadata=0x1a530000000000/0x1fffff0000000000 actions=drop
having a drop action.

What is the expectation of the test ?
Is table 244 intended to drop non-related and non-established packets ?

Thanks Darrell

From: <ovs-discuss-bounces at openvswitch.org> on behalf of Rohith Basavaraja <rohith.basavaraja at ericsson.com>
Date: Wednesday, August 23, 2017 at 3:03 AM
To: "ovs-discuss at openvswitch.org" <ovs-discuss at openvswitch.org>
Subject: [ovs-discuss] Conntrack issue in OVS (2.6)+DPDK

Hi,

I see that if I have following rules, i.e not allow any new connections and allow only established and related flows,

cookie=0x6900001, duration=2819.230s, table=244, n_packets=2, n_bytes=196, priority=50, ct_state=+new+trk,metadata=0x1a530000000000/0x1fffff0000000000 actions=drop
cookie=0x6900000, duration=15546.577s, table=244, n_packets=0, n_bytes=0, priority=62020,ct_state=-new-est+rel-inv+trk actions=resubmit(,220)
cookie=0x6900000, duration=15546.552s, table=244, n_packets=3819, n_bytes=431050, priority=62020,ct_state=-new+est-rel-inv+trk actions=resubmit(,220)

We are still seeing that new connections are getting allowed, we see this behavior/issue only OVS + DPDK and not in OVS kernel mode.

Wanted to check if this issue is already reported elsewhere or it’s new issue.

Thanks
Rohith

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20171102/a674ca21/attachment-0001.html>