[ovs-discuss] [ovn] TCP/UDP traffic dropped between IP addresses defined in allowed address pairs when remote security group is used

Dumitru Ceara dceara at redhat.com
Tue Dec 15 11:20:59 UTC 2020


On 12/15/20 12:02 PM, Krzysztof Klimonda wrote:
> Hi Dumitru,
> 
> Thanks for checking it out.
> 
> On Tue, Dec 15, 2020, at 10:45, Dumitru Ceara wrote:
>> Hi Krzysztof,
>>
>> Thanks for the DBs and all the details.
>>
>> I gave it a try on my local setup, using your DBs.  The behavior below
>> is identical on v20.06.2, branch-20.12 and current master.
> 
> I have to admit I'm not sure why in your tests the behaviour is the same in 20.06.2 - unfortunately I no longer have access to the environment with that version, so I can't re-verify that on my end and I based it on the fact that this SG was created by magnum for kubernetes cluster deployed on top of openstack, and the cluster was being tested successfully. The earliest I can try recreating this with 20.06.2 release is next year, so I'm not sure if this will be of any help.
> 

I just rechecked with 20.06.2 and I'm 100% sure that with the DBs you
provided and the behavior is exactly the same as with newer branches
(including latest master).

> [...]
>>
>> For TCP traffic 172.16.0.11 -> 172.16.0.10 we don't hit any of the
>> allow-related ACLs.  172.16.0.11 is not set as a port address on port
>> 81d23182-37ac-4d3d-815e-4c25d26fe154 so it will not be included in the
>> auto-generated address_set pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4.
>>
>> On the other hand, for TCP traffic 10.0.0.11 -> 172.16.0.10 we hit the
>> 4th allow-related ACL:
>> to-lport  1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed &&
>> ip4 && ip4.src == $pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4 && tcp)
>> allow-related
>>
>> ICMP traffic matches the 5th allow-related ACL which doesn't check
>> source IPs:
>> to-lport  1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed &&
>> ip4 && ip4.src == 0.0.0.0/0 && icmp4) allow-related
>>
>> So, unless I'm missing something, OVN is doing what it's supposed to do
>> based on the current configuration.  To avoid adding a new ACL, one
>> option would be to add the 172.16.0.10 and 172.16.0.11 IPs to the
>> corresponding logical_switch_ports "addresses" column, i.e., something like:
> 
> The problem with adding those addresses to LSP is that they are outside of OVN/neutron control - this is a kubernetes cluster deployed with magnum, where CNI plugin (calico) doesn't use any tunneling between k8s nodes, each pod has address in 172.16.0.0/16 subnet and uses addresses from 172.16.0.0/16 to communicate witch other pods. To accomodate that ports have 172.16.0.0/16 added to their allowed addresses.
> 
> I'm assuming that, if OVN is doing what it's supposed to be doing based on the configuration[1], then there is a mismatch between neutron and OVN behaviour in regards to SG with allowed address pairs? I guess someone from neutron team would have to comment whether it's 

I agree, some input from neutron would be great. (cc-ing Daniel).

> 
> [1] I'm not entirely convinced that's the case given that ICMP traffic is being forwarded - I see how it's doing what programmed flows are telling it to do, but that doesn't seem to be expected result.
> 

I'm confused, why wouldn't this be expected?

The ICMP ACL explicitly allows traffic to any port in the port group if
it's ICMP and comes from any IP source:

to-lport  1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed &&
ip4 && ip4.src == 0.0.0.0/0 && icmp4) allow-related

For TCP the match is more restrictive:

The TCP ACL allows traffic to any port in the port group if the IP
source is part of the addresses defined on the logical ports that are
part of the port group:

to-lport  1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed
&&ip4 && ip4.src == $pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4 && tcp)
allow-related

But the traffic comes from 172.16.0.11 which is not an address on the
logical switch port.  So traffic doesn't match the allow ACL and instead
matches the deny-all ACL.

OVN cannot automatically deduce anything from the fact that
port_security includes 172.16.0.0/16 and without help from the CMS it
cannot decide to allow traffic from 172.16.0.11.

Regards,
Dumitru

> Best Regards,
>   Chris
> 
>>
>> Thanks,
>> Dumitru
>>
>>
>> On 12/14/20 4:43 PM, Krzysztof Klimonda wrote:
>>> Hi Numan,
>>>
>>> https://signal.klimonda.com/ovnnb.db-broken.txt - this is the "initial" state where TCP is not being established (but ping works)
>>> https://signal.klimonda.com/ovnnb.db-working.txt - this is after I create a separate IP-based rule to allow TCP traffic
>>>
>>> In both examples, security group in question is ed081ef3-754a-492f-80b2-fb73cd2dceed which is mapped to pg_ed081ef3_754a_492f_80b2_fb73cd2dceed port group.
>>>
>>> In the second ovnnb.db (ovnnb.db-working.txt), there is an extra ACL fb464efc-f63b-494b-b59b-6c2860dcecba added from CLI via:
>>>
>>> `openstack security group rule create --ingress --protocol tcp --remote-ip 172.16.0.0/24 default`
>>>
>>> -- Krzysztof Klimonda kklimonda at syntaxhighlighted.com On Mon, Dec 14,
>>> 2020, at 14:14, Numan Siddique wrote:
>>>> On Mon, Dec 14, 2020 at 4:01 PM Krzysztof Klimonda
>>>> <kklimonda at syntaxhighlighted.com> wrote:
>>>>> Hi,
>>>>>
>>>>> After upgrading to OVN 20.12.0 snapshot d8bc0377c I've noticed a problem in communication between VMs that use allowed address pairs and remote group id in security groups. I believe it has worked properly with OVN 20.06.2 release (although I have no way of verifying it right now).
>>>>>
>>>> Thanks for reporting the issue.
>>>>
>>>> Is it possible for you to share the OVN NB DB somewhere ?
>>>>
>>>> It would be easier to reproduce the issue with the DB.
>>>>
>>>> Thanks
>>>> Numan
>>>>
>>>>> Given the following scenario:
>>>>>
>>>>> - 2 VMs with IP addresses: vm-a with IP addresses 10.0.0.10 and 172.16.0.10 and vm-b with IP addresses 10.0.0.11 and 172.16.0.11 where 10.0.0.0/8 addresses are set on ports, and 172.16.0.0/16 addresses are set in allowed-address for on ports
>>>>> - There is single security group attached to both ports allowing for ingress tcp traffic coming from the same security group (remote-group)
>>>>> - There is a TCP service listening on 10.0.0.10 on port 8000
>>>>>
>>>>> When I try accessing service from vm-b using 10.0.0.10 address, ovn forwards traffic properly. However, when I try accessing same service via 172.16.0.10 traffic is dropped.
>>>>>
>>>>> When I trace packets between VMs using ovn-trace, for first scenario the last step is:
>>>>>
>>>>> ----8<----8<----
>>>>> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
>>>>> ---------------------------------------------------------------
>>>>>  4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid ab5a233e
>>>>>     reg0[8] = 1;
>>>>>     reg0[10] = 1;
>>>>>     next;
>>>>>  5. ls_out_acl (ovn-northd.c:5498): reg0[8] == 1 && (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed && ip4 && ip4.src == $pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4 && tcp), priority 2002, uuid d92706d4
>>>>>     next;
>>>>>  9. ls_out_port_sec_ip (ovn-northd.c:4525): outport == "864929" && eth.dst == fa:16:3e:bc:20:10 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.0.0.10, 172.16.0.0/16}, priority 90, uuid ff3390b1
>>>>>     next;
>>>>> 10. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "864929" && eth.dst == {fa:16:3e:bc:20:10}, priority 50, uuid af91c05c
>>>>>     output;
>>>>>     /* output to "864929", type "" */
>>>>> ----8<----8<----
>>>>>
>>>>> However, when I use 172.16.0.0/24 addresses, the last step changes to:
>>>>>
>>>>> ----8<----8<----
>>>>> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
>>>>> ---------------------------------------------------------------
>>>>>  4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid ab5a233e
>>>>>     reg0[8] = 1;
>>>>>     reg0[10] = 1;
>>>>>     next;
>>>>>  5. ls_out_acl (ovn-northd.c:5553): reg0[10] == 1 && (outport == @neutron_pg_drop && ip), priority 2001, uuid e36c0840
>>>>>     ct_commit { ct_label.blocked = 1; };
>>>>> ----8<----8<----
>>>>>
>>>>> Further notes:
>>>>>
>>>>> - ICMP traffic between 172.16.0.0/24 addresses is forwarded correctly, with last step of ovn-trace being:
>>>>>
>>>>> ----8<----8<----
>>>>> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
>>>>> ---------------------------------------------------------------
>>>>>  4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid ab5a233e
>>>>>     reg0[8] = 1;
>>>>>     reg0[10] = 1;
>>>>>     next;
>>>>>  5. ls_out_acl (ovn-northd.c:5498): reg0[8] == 1 && (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed && ip4 && ip4.src == 0.0.0.0/0 && icmp4), priority 2002, uuid cd1705d8
>>>>>     next;
>>>>>  9. ls_out_port_sec_ip (ovn-northd.c:4525): outport == "864929" && eth.dst == fa:16:3e:bc:20:10 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.0.0.10, 172.16.0.0/16}, priority 90, uuid ff3390b1
>>>>>     next;
>>>>> 10. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "864929" && eth.dst == {fa:16:3e:bc:20:10}, priority 50, uuid af91c05c
>>>>>     output;
>>>>>     /* output to "864929", type "" */
>>>>> ----8<----8<----
>>>>>
>>>>> - If I replace security group rule, changing remote group to remote ip, traffic is forwarded correctly and last step in ovn-trace is:
>>>>>
>>>>> ----8<----8<----
>>>>> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
>>>>> ---------------------------------------------------------------
>>>>>  4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid ab5a233e
>>>>>     reg0[8] = 1;
>>>>>     reg0[10] = 1;
>>>>>     next;
>>>>>  5. ls_out_acl (ovn-northd.c:5498): reg0[8] == 1 && (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed && ip4 && ip4.src == 172.16.0.0/24 && tcp), priority 2002, uuid a0871ca2
>>>>>     next;
>>>>>  9. ls_out_port_sec_ip (ovn-northd.c:4525): outport == "864929" && eth.dst == fa:16:3e:bc:20:10 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.0.0.10, 172.16.0.0/16}, priority 90, uuid ff3390b1
>>>>>     next;
>>>>> 10. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "864929" && eth.dst == {fa:16:3e:bc:20:10}, priority 50, uuid af91c05c
>>>>>     output;
>>>>>     /* output to "864929", type "" */
>>>>> ----8<----8<----
>>>>>
>>>>> --
>>>>>   Krzysztof Klimonda
>>>>>   kklimonda at syntaxhighlighted.com
>>>>> _______________________________________________
>>>>> discuss mailing list
>>>>> discuss at openvswitch.org
>>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>>>>
>>> _______________________________________________
>>> discuss mailing list
>>> discuss at openvswitch.org
>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>>
>>
>>
> 



More information about the discuss mailing list