[ovs-discuss] OVN with stateful ACLs connectivity issue.

Odintsov Vladislav VlOdintsov at croc.ru
Thu Jan 23 19:49:49 UTC 2020


Hi all.

I’m trying to configure OVN manually on my CentOS 7.5 server and have two connectivity issues when I use stateful acls.
Could somebody point me on any type of misconfiguration or say that such topology shouldn’t work?
I use 2.12.0 version for openvswitch, openvswitch-kmod and ovn (taken from openvswitch.org<http://openvswitch.org>).


(apologize for long read)

First configuration and issue.
All tests are done on all-in-one node.
I’ve got next topology (see ascii scheme):

                                                       +---------+
                                  +--------------------- lswitch |
                                  |                    +|-------|+
                                  |                     |       |
                                  |                     |       |
                                  |      +--------------|-------|------------+
                         +--------|--+   |    +---------|-+   +-|----------+ |
                         |   lport1  |   |    |   lport2  |   |   lport3   | |
                         | 172.31.0.1|   |    | 172.31.0.4|   | 172.31.0.5 | |
                         +---------|-+   |    +---------|-+   +-|----------+ |
                                   |     | port_group_1 |       |            |
                                   |     +--------------|-------|------------+
                                   |                    |       |
                                   |                    |       |
            +----------------------|--+  +--------------|-+  +--|----------------------+
            | router-vm         +--|-+|  | vm1      +---|+|  | +|---+              vm2 |
            |         172.31.0.1|eth0||  |172.31.0.4|eth0||  | |eth0|172.31.0.5        |
            |                   +----+|  |          +----+|  | +----+                  |
            | # ip r                  |  |                |  | net.ipv4.ip_forward = 1 |
            | 172.31.0.0/24 dev eth0  |  |                |  | +---------------+       |
            | 10.0.0.1 via 172.31.0.5 |  |                |  | |lo: 10.0.0.1/32|       |
            |                         |  |                |  | +---------------+       |
            +-------------------------+  +----------------+  +-------------------------+



             acl
            +------------------------------------------------------------+---------------+
            |to-lport  | 1001  |outport == @port_group_1 && ip           |drop           | (default-drop)
            |to-lport  | 1002  |outport == @port_group_1 && ip4 && icmp4 |allow-related  |
            |from-lport| 1001  |inport == @port_group_1 && ip            |drop           | (default-drop)
            |from-lport| 1002  |inport == @port_group_1 && ip            |allow-related  |
            +------------------------------------------------------------+---------------+


  *   One logical switch with three VIF logical ports attached to QEMU virtual machines with tap interfaces.
  *   lport2 and port3 reside in one port group (port_group_1).
  *   each port is assigned with mac address and ipv4 address in NorthBound, same mac and IP assigned inside vms.
  *   port_group_1 has assigned acls for ingress and egress default drop rules (see table above) and allow-related `all` for egress (from the VM point of view, from-lport) and allow-related icmp4 for ingress (from VM point of view, to-lport).
  *   lport1 and lswitch have no assigned port_group/acls.
  *   vm2 has assigned IP from another subnet (10.0.0.1/32), enabled net.ipv4.ip_forward =1 and no iptables rules with accept policy.
  *   router-vm (lport1) has route to 10.0.0.1/32 via vm2 (172.31.0.5), enabled net.ipv4.ip_forward =1 and no iptables rules with accept policy.

When I start ping from vm1 to vm2 loopback address (172.31.0.4 -> 10.0.0.1), ping succeeds (it goes through router-vm, reaches vm2 and reply goes directly from vm2 to vm1.
When I start ping from vm2 to vm1 from loopback address (10.0.0.1 as a source IP), I have 100% packet loss. Reply packet goes out vm1 and get dropped in the OVS datapath (I see drop datapath flow with in_port - VM1, and incresing counter).

What do I do wrong?

# ovs-dpctl dump-flows | grep drop
recirc_id(0xbd5b),in_port(14),ct_state(+inv+trk),eth(),eth_type(0x0800),ipv4(frag=no), packets:1, bytes:98, used:0.803s, actions:drop


[root at dev ~]# grep icmp /proc/net/nf_conntrack
ipv4     2 icmp     1 29 src=10.0.0.1 dst=172.31.0.4 type=8 code=0 id=22278 src=172.31.0.4 dst=10.0.0.1 type=0 code=0 id=22278 mark=0 zone=7 use=2
ipv4     2 icmp     1 29 src=10.0.0.1 dst=172.31.0.4 type=8 code=0 id=22278 [UNREPLIED] src=172.31.0.4 dst=10.0.0.1 type=0 code=0 id=22278 mark=0 zone=8 use=2

When I add next route on VM1: 10.0.0.1/32 via 172.31.0.5, ping succeeds in both directions.
Is it possible to make this topology work without this specific route? I need to route this traffic through router-vm.


Second configuration and issue.

Here the second server with CentOS 7.5 added. It has a role VTEP Emulator. ovs-controller-vtep and ovs-vtep python scripts from OVS source tree runs. open_vswitch and hardware_vtep databases configured. I used instructions from http://docs.openvswitch.org/en/latest/howto/vtep/


                                      +-----------+
                                      |  lswitch  |
                                      +-|-------|-+
                                        |       |
                                        |       |
                          +-------------|-+    +--------------------------+
                          | +-----------|+|    | lport2                   |
                          | |lport1      ||    | type=vtep                |
                          | |172.31.0.4  ||    | addresses=unknown        |
                          | +-----------|+|    | vtep_physical_switch=br0 |
                          |             | |    | vtep_logical_switch=ls1  |
                          | port_group_1| |    +--------------------------+
                          |             | |
                          +-------------|-+
                                        |
                          +-------------|-------+
                          |+------------|-+     |
                          ||eth0          |     |
                          ||172.31.0.4/24 |     |
                          |+--------------+     |
                          |                     |
                          |                     |
                          |vm1                  |
                          +---------------------+


             acl
            +------------------------------------------------------------+---------------+
            |to-lport  | 1001  |outport == @port_group_1 && ip           |drop           | (default-drop)
            |to-lport  | 1002  |outport == @port_group_1 && ip4 && icmp4 |allow-related  |
            |from-lport| 1001  |inport == @port_group_1 && ip            |drop           | (default-drop)
            |from-lport| 1002  |inport == @port_group_1 && ip            |allow-related  |
            +------------------------------------------------------------+---------------+



  *   there is one logical switch lswitch
  *   to logical ports attached to lswitch - lport1 and lport2
  *   lport1 is a normal VIF port with configured mac and ip address in North DB, same l2/l3 addresses configured inside VM attached to this lport via tap interface
  *   lport1 assigned to port_group_1
  *   port_group_1 is assigned to same acls as in previous case (see scheme above). ingress (to-lport icmp4 allow-related)
  *   lport2 has type vtep, addresses set to unknown, vtep_logical_switch configured to appropriate logical switch from vtep DB and vtep_hardware_switch configured to HW VTEP switch PS name.

On the vtep emulator side:

  *   created lswitch in hardware_vtep DB
  *   configured binding for lswitch, outgoing port (OVS internal port `vlans` in bridge br0) and vlan 10.
  *   on top of `vlans` OVS port I created virtual port vlan10 (ip li add link vlans name vlan10 vlan id 10), assigned IP 172.31.0.200/24 and brought it up.



If I ping from VTEP side (vlan10) to VM (172.31.0.200 -> 172.31.0.4), ping succeeds.
If I ping in the reverse direction (172.31.0.4 -> 172.31.0.200), I have 100% ping loss. And at the same time with tcpdump I see, that ICMP request successfully went from node with ovn-controller to VTEP emulator, it reached vlan10 interface, and the ICMP response came to hypervisor node. On this node it got dropped. I see datapath drop flow:

# ovs-dpctl dump-flows | grep drop
recirc_id(0xc3),tunnel(tun_id=0x1,src=192.168.0.13,dst=192.168.0.9,flags(-df-csum+key)),in_port(2),ct_state(+inv+trk),eth(),eth_type(0x0800),ipv4(frag=no), packets:3635, bytes:356230, used:0.072s, actions:drop

192.168.0.9 - hypervisor tunnel IP
192.168.0.13 - VTEP emulator tunnel IP

[root at dev ~]# tcpdump -eni eth0 port 4789
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:33:57.597093 0a:00:b7:77:b1:80 > 0a:00:7f:6a:6f:00, ethertype IPv4 (0x0800), length 148: 192.168.0.9.37376 > 192.168.0.13.4789: VXLAN, flags [I] (0x08), vni 1
0a:00:f6:a1:93:e0 > 46:fc:d2:cc:ae:0e, ethertype IPv4 (0x0800), length 98: 172.31.0.4 > 172.31.0.200: ICMP echo request, id 40705, seq 1873, length 64
19:33:57.597380 0a:00:7f:6a:6f:00 > 0a:00:b7:77:b1:80, ethertype IPv4 (0x0800), length 148: 192.168.0.13.44458 > 192.168.0.9.4789: VXLAN, flags [I] (0x08), vni 1
46:fc:d2:cc:ae:0e > 0a:00:f6:a1:93:e0, ethertype IPv4 (0x0800), length 98: 172.31.0.200 > 172.31.0.4: ICMP echo reply, id 40705, seq 1873, length 64

[root at dev ~]# grep zone=4 /proc/net/nf_conntrack
ipv4     2 icmp     1 29 src=172.31.0.4 dst=172.31.0.200 type=8 code=0 id=44033 [UNREPLIED] src=172.31.0.200 dst=172.31.0.4 type=0 code=0 id=44033 mark=0 zone=4 use=2

How to understand, why does conntrack decide these packets as invalid?
And what do I miss in this configuration?

Thank you if read till the end. If you have any suggestions they’d be highly appreciated.


Regards,

Vladislav Odintsov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200123/fa9d2533/attachment-0001.html>


More information about the discuss mailing list