[ovs-discuss] OVS conntrack

Matt Warner matt at warnertechnology.com
Fri May 12 17:26:38 UTC 2017


On Fri, May 12, 2017 at 8:49 AM, Joe Stringer <joe at ovn.org> wrote:

> On 11 May 2017 at 19:46, Matt Warner <matt at warnertechnology.com> wrote:
> >
> > On Thu, May 11, 2017 at 4:28 PM, Joe Stringer <joe at ovn.org> wrote:
> >>
> >> On 11 May 2017 at 13:38, Matt Warner <matt at warnertechnology.com> wrote:
> >> > Good afternoon. I've been experimenting with OVS and the conntrack
> >> > feature.
> >> > I'd like to be able to use conntrack to maintain state. Specifically,
> in
> >> > the
> >> > man pages it says that this configuration should allow traffic to flow
> >> > only
> >> > one way.
> >> >
> >> > I was testing 2.5.x and hit problems, so I compiled and installed
> 2.7.0
> >> > on
> >> > Ubuntu 16.04.2 LTS, but the problem persists.
> >> >
> >> > In testing, these rules don't seem to allow any traffic at all. I'd
> like
> >> > to
> >> > be able to pass traffic unidirectionally, in a stateful way, per the
> man
> >> > page:
> >> >
> >> > table=0,priority=1,action=drop
> >> > table=0,priority=10,arp,action=normal
> >> > table=0,priority=100,ip,ct_state=-trk,action=ct(table=1)
> >> > table=1,in_port=1,ip,ct_state=+trk+new,action=ct(commit),2
> >> > table=1,in_port=1,ip,ct_state=+trk+est,action=2
> >> > table=1,in_port=2,ip,ct_state=+trk+new,action=drop
> >> > table=1,in_port=2,ip,ct_state=+trk+est,action=1
> >> >
> >> >
> >> > The conntrack kernel module is loaded, and on startup OVS indicates
> that
> >> > CT
> >> > is available:
> >> >
> >> > 2017-05-11T20:23:31Z|00001|vlog|INFO|opened log file
> >> > /usr/local/var/log/openvswitch/ovsdb-server.log
> >> > 2017-05-11T20:23:31Z|00001|vlog|INFO|opened log file
> >> > /usr/local/var/log/openvswitch/ovs-vswitchd.log
> >> > 2017-05-11T20:23:31Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on
> NUMA
> >> > node
> >> > 0
> >> > 2017-05-11T20:23:31Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and
> 4
> >> > CPU
> >> > cores
> >> >
> >> > 2017-05-11T20:23:31Z|00004|reconnect|INFO|unix:/usr/
> local/var/run/openvswitch/db.sock:
> >> > connecting...
> >> >
> >> > 2017-05-11T20:23:31Z|00005|reconnect|INFO|unix:/usr/
> local/var/run/openvswitch/db.sock:
> >> > connected
> >> > 2017-05-11T20:23:31Z|00006|ofproto_dpif|INFO|system at ovs-system:
> Datapath
> >> > supports recirculation
> >> > 2017-05-11T20:23:31Z|00007|ofproto_dpif|INFO|system at ovs-system: MPLS
> >> > label
> >> > stack length probed as 1
> >> > 2017-05-11T20:23:31Z|00008|ofproto_dpif|INFO|system at ovs-system:
> Datapath
> >> > does not support truncate action
> >> > 2017-05-11T20:23:31Z|00009|ofproto_dpif|INFO|system at ovs-system:
> Datapath
> >> > supports unique flow ids
> >> > 2017-05-11T20:23:31Z|00010|ofproto_dpif|INFO|system at ovs-system:
> Datapath
> >> > supports ct_state  <-----
> >> > 2017-05-11T20:23:31Z|00011|ofproto_dpif|INFO|system at ovs-system:
> Datapath
> >> > supports ct_zone
> >> > 2017-05-11T20:23:31Z|00012|ofproto_dpif|INFO|system at ovs-system:
> Datapath
> >> > supports ct_mark
> >> > 2017-05-11T20:23:31Z|00013|ofproto_dpif|INFO|system at ovs-system:
> Datapath
> >> > supports ct_label
> >> > 2017-05-11T20:23:31Z|00014|ofproto_dpif|INFO|system at ovs-system:
> Datapath
> >> > does not support ct_state_nat
> >> > 2017-05-11T20:23:31Z|00001|ofproto_dpif_upcall(handler1)|
> INFO|received
> >> > packet on unassociated datapath port 0
> >> > 2017-05-11T20:23:31Z|00015|bridge|INFO|bridge matt_bridge: added
> >> > interface
> >> > ens33 on port 1
> >> > 2017-05-11T20:23:31Z|00016|bridge|INFO|bridge matt_bridge: added
> >> > interface
> >> > ens34 on port 2
> >> > 2017-05-11T20:23:31Z|00017|bridge|INFO|bridge matt_bridge: added
> >> > interface
> >> > matt_bridge on port 65534
> >> > 2017-05-11T20:23:31Z|00018|bridge|INFO|bridge matt_bridge: using
> >> > datapath ID
> >> > 000000900b2aec9b
> >> > 2017-05-11T20:23:31Z|00019|connmgr|INFO|matt_bridge: added service
> >> > controller "punix:/usr/local/var/run/openvswitch/matt_bridge.mgmt"
> >> >
> >> > root at demo:~# lsmod |egrep 'open|conn'
> >> > openvswitch            98304  4
> >> > nf_defrag_ipv6         36864  1 openvswitch
> >> > nf_conntrack          106496  1 openvswitch
> >> > libcrc32c              16384  2 raid456,openvswitch
> >> >
> >> >
> >> > Here's the ruleset applied:
> >> >
> >> > root at demo:~# ovs-appctl bridge/dump-flows matt_bridge
> >> > duration=18s, n_packets=4, n_bytes=406,
> >> > priority=100,ct_state=-trk,ip,actions=ct(table=1)
> >> > duration=18s, n_packets=2, n_bytes=120, priority=10,arp,actions=NORMAL
> >> > duration=99s, n_packets=9, n_bytes=1124, priority=0,actions=NORMAL
> >> > duration=18s, n_packets=0, n_bytes=0, priority=1,actions=drop
> >> > table_id=1, duration=18s, n_packets=0, n_bytes=0,
> >> > ct_state=+new+trk,ip,in_port=1,actions=ct(commit),output:2
> >> > table_id=1, duration=18s, n_packets=0, n_bytes=0,
> >> > ct_state=+new+trk,ip,in_port=2,actions=drop
> >> > table_id=1, duration=18s, n_packets=0, n_bytes=0,
> >> > ct_state=+est+trk,ip,in_port=1,actions=output:2
> >> > table_id=1, duration=18s, n_packets=0, n_bytes=0,
> >> > ct_state=+est+trk,ip,in_port=2,actions=output:1
> >> > table_id=254, duration=99s, n_packets=0, n_bytes=0,
> >> > priority=2,recirc_id=0,actions=drop
> >> > table_id=254, duration=99s, n_packets=0, n_bytes=0,
> >> > priority=0,reg0=0x1,actions=controller(reason=)
> >> > table_id=254, duration=99s, n_packets=4, n_bytes=406,
> >> > priority=0,reg0=0x2,actions=drop
> >> > table_id=254, duration=99s, n_packets=0, n_bytes=0,
> >> > priority=0,reg0=0x3,actions=drop
> >> >
> >> >
> >> > Trying to initiate traffic from either connected device increments
> >> > counters
> >> > in table 0, but nothing seems to get to table 1:
> >> >
> >> > root at demo:~# ovs-ofctl dump-flows matt_bridge
> >> > NXST_FLOW reply (xid=0x4):
> >> >  cookie=0x0, duration=127.968s, table=0, n_packets=25, n_bytes=2402,
> >> > idle_age=7, priority=100,ct_state=-trk,ip actions=ct(table=1)
> >> >  cookie=0x0, duration=127.982s, table=0, n_packets=4, n_bytes=240,
> >> > idle_age=45, priority=10,arp actions=NORMAL
> >> >  cookie=0x0, duration=209.003s, table=0, n_packets=9, n_bytes=1124,
> >> > idle_age=136, priority=0 actions=NORMAL
> >> >  cookie=0x0, duration=128.001s, table=0, n_packets=1, n_bytes=126,
> >> > idle_age=31, priority=1 actions=drop
> >> >  cookie=0x0, duration=127.954s, table=1, n_packets=0, n_bytes=0,
> >> > idle_age=127, ct_state=+new+trk,ip,in_port=1
> actions=ct(commit),output:2
> >> >  cookie=0x0, duration=127.922s, table=1, n_packets=0, n_bytes=0,
> >> > idle_age=127, ct_state=+new+trk,ip,in_port=2 actions=drop
> >> >  cookie=0x0, duration=127.938s, table=1, n_packets=0, n_bytes=0,
> >> > idle_age=127, ct_state=+est+trk,ip,in_port=1 actions=output:2
> >> >  cookie=0x0, duration=127.906s, table=1, n_packets=0, n_bytes=0,
> >> > idle_age=127, ct_state=+est+trk,ip,in_port=2 actions=output:1
> >> >
> >> >
> >> > Here's what appears to be the status of the in-flight packets (here
> I'm
> >> > trying to initiate a connection from both sides simultaneously):
> >> >
> >> > root at demo:~# ovs-appctl dpif/dump-flows matt_bridge
> >> > recirc_id(0),in_port(1),ct_state(-trk),eth_type(0x0800),
> ipv4(frag=no),
> >> > packets:14, bytes:1732, used:0.284s, flags:S, actions:ct,recirc(0xc)
> >> >
> >> > recirc_id(0xd),in_port(2),ct_state(-new-est+trk),eth_type(
> 0x0800),ipv4(frag=no),
> >> > packets:3, bytes:222, used:1.836s, flags:S, actions:drop
> >> >
> >> > recirc_id(0),in_port(1),ct_state(-trk),eth(src=00:3e:e1:
> c0:9b:92,dst=02:08:20:05:74:f7),eth_type(0x0806),
> >> > packets:0, bytes:0, used:never, actions:2
> >> >
> >> > recirc_id(0),in_port(2),ct_state(-trk),eth(src=02:08:20:
> 05:74:f7,dst=00:3e:e1:c0:9b:92),eth_type(0x0806),
> >> > packets:0, bytes:0, used:never, actions:1
> >> > recirc_id(0),in_port(2),ct_state(-trk),eth_type(0x0800),
> ipv4(frag=no),
> >> > packets:3, bytes:222, used:1.836s, flags:S, actions:ct,recirc(0xd)
> >> >
> >> > recirc_id(0xc),in_port(1),ct_state(-new-est+trk),eth_type(
> 0x0800),ipv4(frag=no),
> >> > packets:14, bytes:1732, used:0.284s, flags:S, actions:drop
> >> >
> >> >
> >> > I also noticed that if I clear out table1 and use these two entries
> >> > traffic
> >> > is allowed, but bi-directionally:
> >> >
> >> > root at demo:~# ovs-ofctl add-flow matt_bridge
> >> >
> >> > 'table=1,in_port=1,priority=65532,ip,ct_state=+trk-est-
> rpl-rel-new,action=ct(commit),2'
> >> > root at demo:~# ovs-ofctl add-flow matt_bridge
> >> > 'table=1,in_port=2,priority=65533,ip,ct_state=+trk,action=ct(),1'
> >> >
> >> > It seems like the "+new" flag isn't actually being set or something...
> >> >
> >> > Any pointers about what I'm doing wrong here?
> >>
> >> Hi Matt,
> >>
> >> I have a few thoughts.
> >>
> >> * Which Linux kernel version are you using?
> >> * Are you using the DKMS module distributed with openvswitch or the
> >> one that comes with your upstream kernel ('modinfo openvswitch')?
> >> * Is nf_conntrack_ipv4 module loaded?
> >> * Any insight into what traffic you're running?
> >>
> >> You might be interested to add a flow like this to see if the
> >> connection tracker determines any traffic is invalid:
> >> table=1,in_port=1,ip,ct_state=+trk+inv,action=drop
> >>
> >> Depending on the topology, it's possible to have packets ingress to
> >> OVS with ct_state already populated (from bridge or internal type
> >> port). To check if any of that is happening, you could add a flow to
> >> table 0:
> >> table=0,ip,ct_state=+trk, priority=100 actions=goto_table(1)
> >>
> >> With regards to your final flows which allow traffic to pass, that is
> >> because they do not check the ct() result; they unconditionally output
> >> at the end of the flow.
> >
> >
> > The kernel version is 4.4.0 (see below). No DPDK as far as I know—I
> thought
> > I saw that the hardware had to support Vt-d?
> >
> > root at demo:~# modinfo openvswitch
> > filename:
> > /lib/modules/4.4.0-72-generic/kernel/net/openvswitch/openvswitch.ko
> > license:        GPL
> > description:    Open vSwitch switching datapath
> > srcversion:     F3C723D4112A63EB96CF3A2
> > depends:        nf_conntrack,libcrc32c,nf_defrag_ipv6
> > intree:         Y
> > vermagic:       4.4.0-72-generic SMP mod_unload modversions
>
> OK, 'intree: Y' indicates that you are using the kernel module that
> comes with the distribution's Linux 4.4 package. I suspect that later
> versions will autoload nf_conntrack_ipv4 for you, but with this
> version you'll have to load it yourself.
>
> >
> >
> > I see no nf_conntrack_ipv4 listed, just nf_conntrack.
> >
> > Great suggestion to check for invalid traffic. I removed all the table1
> > entries and added 'table=1,in_port=1,ip,ct_state=+trk+inv,action=drop'
> and
> > the counter is incrementing:
> >
> > cookie=0x0, duration=25.057s, table=1, n_packets=9, n_bytes=830,
> > reset_counts ct_state=+inv+trk,ip,in_port=1 actions=drop
> >
> >
> > So OVS sees traffic it considers invalid. I have no idea what that
> means...
>
> Invalid is a 'catch-all' that indicates the Linux netfilter code was
> unable to identify the connection, or the protocol handlers saw
> something that seems strange about the packets. In this case I suspect
> it's the former, because the IPv4 handler isn't loaded.
>
> If you look at ovs-fields(7) man pages and search for 'invalid' under
> the connection tracking state field, you'll see a further description:
> http://openvswitch.org/support/dist-docs/ovs-fields.7.html
>
> > I also added the check to see if ct_state was already being populated,
> but
> > the counter is not incrementing:
> >
> > cookie=0x0, duration=23.721s, table=0, n_packets=0, n_bytes=0,
> reset_counts
> > priority=100,ct_state=+trk,ip actions=resubmit(,1)
> >
> >
> > So invalid traffic...?
>
> Try 'modprobe nf_connntrack_ipv4'?
>

Ahh! Like magic it now works.

Thanks so much!

Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20170512/ce0c67a8/attachment-0001.html>


More information about the discuss mailing list