[ovs-discuss] [ovn] TCP/UDP traffic dropped between IP addresses defined in allowed address pairs when remote security group is used

Flavio Fernandes flavio at flaviof.com
Tue Dec 15 11:37:11 UTC 2020



> On Dec 15, 2020, at 6:20 AM, Dumitru Ceara <dceara at redhat.com> wrote:
> 
> On 12/15/20 12:02 PM, Krzysztof Klimonda wrote:
>> Hi Dumitru,
>> 
>> Thanks for checking it out.
>> 
>> On Tue, Dec 15, 2020, at 10:45, Dumitru Ceara wrote:
>>> Hi Krzysztof,
>>> 
>>> Thanks for the DBs and all the details.
>>> 
>>> I gave it a try on my local setup, using your DBs.  The behavior below
>>> is identical on v20.06.2, branch-20.12 and current master.
>> 
>> I have to admit I'm not sure why in your tests the behaviour is the same in 20.06.2 - unfortunately I no longer have access to the environment with that version, so I can't re-verify that on my end and I based it on the fact that this SG was created by magnum for kubernetes cluster deployed on top of openstack, and the cluster was being tested successfully. The earliest I can try recreating this with 20.06.2 release is next year, so I'm not sure if this will be of any help.
>> 
> 
> I just rechecked with 20.06.2 and I'm 100% sure that with the DBs you
> provided and the behavior is exactly the same as with newer branches
> (including latest master).
> 
>> [...]
>>> 
>>> For TCP traffic 172.16.0.11 -> 172.16.0.10 we don't hit any of the
>>> allow-related ACLs.  172.16.0.11 is not set as a port address on port
>>> 81d23182-37ac-4d3d-815e-4c25d26fe154 so it will not be included in the
>>> auto-generated address_set pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4.
>>> 
>>> On the other hand, for TCP traffic 10.0.0.11 -> 172.16.0.10 we hit the
>>> 4th allow-related ACL:
>>> to-lport  1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed &&
>>> ip4 && ip4.src == $pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4 && tcp)
>>> allow-related
>>> 
>>> ICMP traffic matches the 5th allow-related ACL which doesn't check
>>> source IPs:
>>> to-lport  1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed &&
>>> ip4 && ip4.src == 0.0.0.0/0 && icmp4) allow-related
>>> 
>>> So, unless I'm missing something, OVN is doing what it's supposed to do
>>> based on the current configuration.  To avoid adding a new ACL, one
>>> option would be to add the 172.16.0.10 and 172.16.0.11 IPs to the
>>> corresponding logical_switch_ports "addresses" column, i.e., something like:
>> 
>> The problem with adding those addresses to LSP is that they are outside of OVN/neutron control - this is a kubernetes cluster deployed with magnum, where CNI plugin (calico) doesn't use any tunneling between k8s nodes, each pod has address in 172.16.0.0/16 subnet and uses addresses from 172.16.0.0/16 to communicate witch other pods. To accomodate that ports have 172.16.0.0/16 added to their allowed addresses.
>> 
>> I'm assuming that, if OVN is doing what it's supposed to be doing based on the configuration[1], then there is a mismatch between neutron and OVN behaviour in regards to SG with allowed address pairs? I guess someone from neutron team would have to comment whether it's
> 
> I agree, some input from neutron would be great. (cc-ing Daniel).

Hi folks. I'm caching up on this thread and have a follow up question/request.

@Krzysztof: can you tell me a little more about how the ips 172.16.0.10 and 172.16.0.11 are added as
secondary on the neutron ports? Are you using multiple fixed ip addresses? Do you have 2 neutron subnets configured
on the same neutron network?

The reason I ask is because a neutron network and its associated neutron subnet(s) are "aggregated" into a single logical switch
in OVN. In fact, neutron subnets have no corresponding row in OVN. So I wonder if what you really want are 2 separate neutron
ports for each vm, instead of a single port with overlapping subnets.

Thanks,

-- flaviof

> 
>> 
>> [1] I'm not entirely convinced that's the case given that ICMP traffic is being forwarded - I see how it's doing what programmed flows are telling it to do, but that doesn't seem to be expected result.
>> 
> 
> I'm confused, why wouldn't this be expected?
> 
> The ICMP ACL explicitly allows traffic to any port in the port group if
> it's ICMP and comes from any IP source:
> 
> to-lport  1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed &&
> ip4 && ip4.src == 0.0.0.0/0 && icmp4) allow-related
> 
> For TCP the match is more restrictive:
> 
> The TCP ACL allows traffic to any port in the port group if the IP
> source is part of the addresses defined on the logical ports that are
> part of the port group:
> 
> to-lport  1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed
> &&ip4 && ip4.src == $pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4 && tcp)
> allow-related
> 
> But the traffic comes from 172.16.0.11 which is not an address on the
> logical switch port.  So traffic doesn't match the allow ACL and instead
> matches the deny-all ACL.
> 
> OVN cannot automatically deduce anything from the fact that
> port_security includes 172.16.0.0/16 and without help from the CMS it
> cannot decide to allow traffic from 172.16.0.11.
> 
> Regards,
> Dumitru
> 
>> Best Regards,
>>  Chris
>> 
>>> 
>>> Thanks,
>>> Dumitru
>>> 
>>> 
>>> On 12/14/20 4:43 PM, Krzysztof Klimonda wrote:
>>>> Hi Numan,
>>>> 
>>>> https://signal.klimonda.com/ovnnb.db-broken.txt - this is the "initial" state where TCP is not being established (but ping works)
>>>> https://signal.klimonda.com/ovnnb.db-working.txt - this is after I create a separate IP-based rule to allow TCP traffic
>>>> 
>>>> In both examples, security group in question is ed081ef3-754a-492f-80b2-fb73cd2dceed which is mapped to pg_ed081ef3_754a_492f_80b2_fb73cd2dceed port group.
>>>> 
>>>> In the second ovnnb.db (ovnnb.db-working.txt), there is an extra ACL fb464efc-f63b-494b-b59b-6c2860dcecba added from CLI via:
>>>> 
>>>> `openstack security group rule create --ingress --protocol tcp --remote-ip 172.16.0.0/24 default`
>>>> 
>>>> -- Krzysztof Klimonda kklimonda at syntaxhighlighted.com On Mon, Dec 14,
>>>> 2020, at 14:14, Numan Siddique wrote:
>>>>> On Mon, Dec 14, 2020 at 4:01 PM Krzysztof Klimonda
>>>>> <kklimonda at syntaxhighlighted.com> wrote:
>>>>>> Hi,
>>>>>> 
>>>>>> After upgrading to OVN 20.12.0 snapshot d8bc0377c I've noticed a problem in communication between VMs that use allowed address pairs and remote group id in security groups. I believe it has worked properly with OVN 20.06.2 release (although I have no way of verifying it right now).
>>>>>> 
>>>>> Thanks for reporting the issue.
>>>>> 
>>>>> Is it possible for you to share the OVN NB DB somewhere ?
>>>>> 
>>>>> It would be easier to reproduce the issue with the DB.
>>>>> 
>>>>> Thanks
>>>>> Numan
>>>>> 
>>>>>> Given the following scenario:
>>>>>> 
>>>>>> - 2 VMs with IP addresses: vm-a with IP addresses 10.0.0.10 and 172.16.0.10 and vm-b with IP addresses 10.0.0.11 and 172.16.0.11 where 10.0.0.0/8 addresses are set on ports, and 172.16.0.0/16 addresses are set in allowed-address for on ports
>>>>>> - There is single security group attached to both ports allowing for ingress tcp traffic coming from the same security group (remote-group)
>>>>>> - There is a TCP service listening on 10.0.0.10 on port 8000
>>>>>> 
>>>>>> When I try accessing service from vm-b using 10.0.0.10 address, ovn forwards traffic properly. However, when I try accessing same service via 172.16.0.10 traffic is dropped.
>>>>>> 
>>>>>> When I trace packets between VMs using ovn-trace, for first scenario the last step is:
>>>>>> 
>>>>>> ----8<----8<----
>>>>>> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
>>>>>> ---------------------------------------------------------------
>>>>>> 4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid ab5a233e
>>>>>>    reg0[8] = 1;
>>>>>>    reg0[10] = 1;
>>>>>>    next;
>>>>>> 5. ls_out_acl (ovn-northd.c:5498): reg0[8] == 1 && (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed && ip4 && ip4.src == $pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4 && tcp), priority 2002, uuid d92706d4
>>>>>>    next;
>>>>>> 9. ls_out_port_sec_ip (ovn-northd.c:4525): outport == "864929" && eth.dst == fa:16:3e:bc:20:10 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.0.0.10, 172.16.0.0/16}, priority 90, uuid ff3390b1
>>>>>>    next;
>>>>>> 10. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "864929" && eth.dst == {fa:16:3e:bc:20:10}, priority 50, uuid af91c05c
>>>>>>    output;
>>>>>>    /* output to "864929", type "" */
>>>>>> ----8<----8<----
>>>>>> 
>>>>>> However, when I use 172.16.0.0/24 addresses, the last step changes to:
>>>>>> 
>>>>>> ----8<----8<----
>>>>>> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
>>>>>> ---------------------------------------------------------------
>>>>>> 4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid ab5a233e
>>>>>>    reg0[8] = 1;
>>>>>>    reg0[10] = 1;
>>>>>>    next;
>>>>>> 5. ls_out_acl (ovn-northd.c:5553): reg0[10] == 1 && (outport == @neutron_pg_drop && ip), priority 2001, uuid e36c0840
>>>>>>    ct_commit { ct_label.blocked = 1; };
>>>>>> ----8<----8<----
>>>>>> 
>>>>>> Further notes:
>>>>>> 
>>>>>> - ICMP traffic between 172.16.0.0/24 addresses is forwarded correctly, with last step of ovn-trace being:
>>>>>> 
>>>>>> ----8<----8<----
>>>>>> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
>>>>>> ---------------------------------------------------------------
>>>>>> 4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid ab5a233e
>>>>>>    reg0[8] = 1;
>>>>>>    reg0[10] = 1;
>>>>>>    next;
>>>>>> 5. ls_out_acl (ovn-northd.c:5498): reg0[8] == 1 && (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed && ip4 && ip4.src == 0.0.0.0/0 && icmp4), priority 2002, uuid cd1705d8
>>>>>>    next;
>>>>>> 9. ls_out_port_sec_ip (ovn-northd.c:4525): outport == "864929" && eth.dst == fa:16:3e:bc:20:10 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.0.0.10, 172.16.0.0/16}, priority 90, uuid ff3390b1
>>>>>>    next;
>>>>>> 10. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "864929" && eth.dst == {fa:16:3e:bc:20:10}, priority 50, uuid af91c05c
>>>>>>    output;
>>>>>>    /* output to "864929", type "" */
>>>>>> ----8<----8<----
>>>>>> 
>>>>>> - If I replace security group rule, changing remote group to remote ip, traffic is forwarded correctly and last step in ovn-trace is:
>>>>>> 
>>>>>> ----8<----8<----
>>>>>> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
>>>>>> ---------------------------------------------------------------
>>>>>> 4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid ab5a233e
>>>>>>    reg0[8] = 1;
>>>>>>    reg0[10] = 1;
>>>>>>    next;
>>>>>> 5. ls_out_acl (ovn-northd.c:5498): reg0[8] == 1 && (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed && ip4 && ip4.src == 172.16.0.0/24 && tcp), priority 2002, uuid a0871ca2
>>>>>>    next;
>>>>>> 9. ls_out_port_sec_ip (ovn-northd.c:4525): outport == "864929" && eth.dst == fa:16:3e:bc:20:10 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.0.0.10, 172.16.0.0/16}, priority 90, uuid ff3390b1
>>>>>>    next;
>>>>>> 10. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "864929" && eth.dst == {fa:16:3e:bc:20:10}, priority 50, uuid af91c05c
>>>>>>    output;
>>>>>>    /* output to "864929", type "" */
>>>>>> ----8<----8<----
>>>>>> 
>>>>>> --
>>>>>>  Krzysztof Klimonda
>>>>>>  kklimonda at syntaxhighlighted.com
>>>>>> _______________________________________________
>>>>>> discuss mailing list
>>>>>> discuss at openvswitch.org
>>>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>>>>> 
>>>> _______________________________________________
>>>> discuss mailing list
>>>> discuss at openvswitch.org
>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>>> 
>>> 
>>> 
>> 
> 
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20201215/44c37b89/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20201215/44c37b89/attachment-0001.sig>


More information about the discuss mailing list