[ovs-discuss] [ovn] TCP/UDP traffic dropped between IP addresses defined in allowed address pairs when remote security group is used
Krzysztof Klimonda
kklimonda at syntaxhighlighted.com
Tue Dec 15 14:55:43 UTC 2020
Daniel,
Sure, let move this over to openstack-discuss@ - I'll open a new thread linking to this message in ovs-discuss archive.
As for verification with ml2/ovs I won't be able to do that until second half of January - I'm taking the rest of the year off until then, and I don't have a lab environment to test it before then, but it's possible that related people (magnum devs and users) can respond before I have a chance to do that.
--
Best Regards,
- Chris
On Tue, Dec 15, 2020, at 15:42, Daniel Alvarez Sanchez wrote:
>
>
> On Tue, Dec 15, 2020 at 2:58 PM Krzysztof Klimonda <kklimonda at syntaxhighlighted.com> wrote:
>> __
>> Hi Flavio, Dumitru
>>
>> See inline for replies
>>
>>
>> On Tue, Dec 15, 2020, at 12:37, Flavio Fernandes wrote:
>>>
>>>
>>>> On Dec 15, 2020, at 6:20 AM, Dumitru Ceara <dceara at redhat.com> wrote:
>>>>
>>>> On 12/15/20 12:02 PM, Krzysztof Klimonda wrote:
>>>>> Hi Dumitru,
>>>>>
>>>>> Thanks for checking it out.
>>>>>
>>>>> On Tue, Dec 15, 2020, at 10:45, Dumitru Ceara wrote:
>>>>>> Hi Krzysztof,
>>>>>>
>>>>>> Thanks for the DBs and all the details.
>>>>>>
>>>>>> I gave it a try on my local setup, using your DBs. The behavior below
>>>>>> is identical on v20.06.2, branch-20.12 and current master.
>>>>>
>>>>> I have to admit I'm not sure why in your tests the behaviour is the same in 20.06.2 - unfortunately I no longer have access to the environment with that version, so I can't re-verify that on my end and I based it on the fact that this SG was created by magnum for kubernetes cluster deployed on top of openstack, and the cluster was being tested successfully. The earliest I can try recreating this with 20.06.2 release is next year, so I'm not sure if this will be of any help.
>>>>>
>>>>
>>>> I just rechecked with 20.06.2 and I'm 100% sure that with the DBs you
>>>> provided and the behavior is exactly the same as with newer branches
>>>> (including latest master).
>>>>
>>>>> [...]
>>>>>>
>>>>>> For TCP traffic 172.16.0.11 -> 172.16.0.10 we don't hit any of the
>>>>>> allow-related ACLs. 172.16.0.11 is not set as a port address on port
>>>>>> 81d23182-37ac-4d3d-815e-4c25d26fe154 so it will not be included in the
>>>>>> auto-generated address_set pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4.
>>>>>>
>>>>>> On the other hand, for TCP traffic 10.0.0.11 -> 172.16.0.10 we hit the
>>>>>> 4th allow-related ACL:
>>>>>> to-lport 1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed &&
>>>>>> ip4 && ip4.src == $pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4 && tcp)
>>>>>> allow-related
>>>>>>
>>>>>> ICMP traffic matches the 5th allow-related ACL which doesn't check
>>>>>> source IPs:
>>>>>> to-lport 1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed &&
>>>>>> ip4 && ip4.src == 0.0.0.0/0 && icmp4) allow-related
>>>>>>
>>>>>> So, unless I'm missing something, OVN is doing what it's supposed to do
>>>>>> based on the current configuration. To avoid adding a new ACL, one
>>>>>> option would be to add the 172.16.0.10 and 172.16.0.11 IPs to the
>>>>>> corresponding logical_switch_ports "addresses" column, i.e., something like:
>>>>>
>>>>> The problem with adding those addresses to LSP is that they are outside of OVN/neutron control - this is a kubernetes cluster deployed with magnum, where CNI plugin (calico) doesn't use any tunneling between k8s nodes, each pod has address in 172.16.0.0/16 subnet and uses addresses from 172.16.0.0/16 to communicate witch other pods. To accomodate that ports have 172.16.0.0/16 added to their allowed addresses.
>>>>>
>>>>> I'm assuming that, if OVN is doing what it's supposed to be doing based on the configuration[1], then there is a mismatch between neutron and OVN behaviour in regards to SG with allowed address pairs? I guess someone from neutron team would have to comment whether it's
>>>>
>>>> I agree, some input from neutron would be great. (cc-ing Daniel).
>>>
>>> Hi folks. I'm caching up on this thread and have a follow up question/request.
>>>
>>> @Krzysztof: can you tell me a little more about how the ips 172.16.0.10 and 172.16.0.11 are added as
>>> secondary on the neutron ports? Are you using multiple fixed ip addresses? Do you have 2 neutron subnets configured
>>> on the same neutron network?
>>>
>>> The reason I ask is because a neutron network and its associated neutron subnet(s) are "aggregated" into a single logical switch
>>> in OVN. In fact, neutron subnets have no corresponding row in OVN. So I wonder if what you really want are 2 separate neutron
>>> ports for each vm, instead of a single port with overlapping subnets.
>>
>> Not sure what do you mean by overlapping subnets - the subnets in question are 10.0.0.0/8 and 172.16.0.0/16.
>>
>> The IP addresses are added either as secondary IP addresses set on the interface inside VM (in my manual testing), or are result of calico CNI using "physical" interface (as opposed to IPIP tunnel) for pod-to-pod communication - in that case each pod gets IP address from 172.16.0.0/16 and Linux just pushes packets based on routing rules (internally 172.16.0.0/16 is split into smaller networks allocated to each k8s node, and each network receives an entry in routing table that push packets to a specific node).
>>
>> Whether it should be a separate neutron port is an excellent question - this security group configuration comes from magnum, which configures port with Pod/service(?) network added to allowed address on all kubernetes nodes, and security group being configured to allow for all TCP/UDP traffic between ports with that SG attached. There is entire separate kuryr-kubernetes project that tries to marry kubernetes CNI with neutron directly, but in our very limited testing it requires more work to be stable in our deployments.
>>
>> See https://github.com/openstack/magnum/blob/c556b8964fab129f33e766b1c33908b2eb001df4/magnum/drivers/k8s_fedora_coreos_v1/templates/kubeminion.yaml for where magnum sets up the allowed_address_pairs attribute on the port, and https://github.com/openstack/magnum/blob/c556b8964fab129f33e766b1c33908b2eb001df4/magnum/drivers/k8s_fedora_coreos_v1/templates/kubecluster.yaml#L1038 for where security group is defined.
>>
>>>
>>> Thanks,
>>>
>>> -- flaviof
>>>
>>>>
>>>>>
>>>>> [1] I'm not entirely convinced that's the case given that ICMP traffic is being forwarded - I see how it's doing what programmed flows are telling it to do, but that doesn't seem to be expected result.
>>>>
>>>> I'm confused, why wouldn't this be expected?
>>
>> From (openstack) user's perspective if there are two security groups:
>> - allow for ICMP traffic between ports in the same security group
>> - allow for TCP traffic between ports in the same security group
>>
>> and one of the rules is working, but the other one is not, then this is not an expected behaviour (again, from openstack user perspective). It's definitely possible that there is a disconnect between what's expected in that case between neutron and ovn - perhaps neutron should add additional entries to OVN's northbound db to support that specific security group feature?
>
>
> I agree with everybody here but... :)
>
> - Does it work with ML2/OVS? Are addresses in the allowed-address pairs accounted for remote groups (for example, Floating IPs are not).
> - If the answer to the above is yes, then I'd agree that we need to fix it in OpenStack by:
> a) Adding the allowed-address pairs to the 'addresses' column of the Logical_Switch_Port row assuming that this doesn't break DHCP
> b) Adding more conditions to the ACL to match not only on the $pg_xxx Address_Set but also on each and every IP address of the allowed-addresses.
> - If the answer to the above is yes, we need to add coverage in OpenStack as CI is clearly not testing this path
> - If the answer to the above is no, then you need to add explicit Neutron Security Group rules to allow traffic originated from the allowed-address pairs (you can do this for now as a workaround)
>
>
> Chris, are you able to test this with ML2/OVS and report back the results?
>
> Either way I don't think there's anything wrong with OVN and we can move this discussion to the openstack-discuss ML :)
>
> Thanks,
> daniel
>>
>>
>>>>
>>>> The ICMP ACL explicitly allows traffic to any port in the port group if
>>>> it's ICMP and comes from any IP source:
>>>>
>>>> to-lport 1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed &&
>>>> ip4 && ip4.src == 0.0.0.0/0 && icmp4) allow-related
>>>>
>>>> For TCP the match is more restrictive:
>>>>
>>>> The TCP ACL allows traffic to any port in the port group if the IP
>>>> source is part of the addresses defined on the logical ports that are
>>>> part of the port group:
>>>>
>>>> to-lport 1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed
>>>> &&ip4 && ip4.src == $pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4 && tcp)
>>>> allow-related
>>>>
>>>> But the traffic comes from 172.16.0.11 which is not an address on the
>>>> logical switch port. So traffic doesn't match the allow ACL and instead
>>>> matches the deny-all ACL.
>>>>
>>>> OVN cannot automatically deduce anything from the fact that
>>>> port_security includes 172.16.0.0/16 and without help from the CMS it
>>>> cannot decide to allow traffic from 172.16.0.11.
>>>>
>>>> Regards,
>>>> Dumitru
>>>>
>>>>> Best Regards,
>>>>> Chris
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Dumitru
>>>>>>
>>>>>>
>>>>>> On 12/14/20 4:43 PM, Krzysztof Klimonda wrote:
>>>>>>> Hi Numan,
>>>>>>>
>>>>>>> https://signal.klimonda.com/ovnnb.db-broken.txt - this is the "initial" state where TCP is not being established (but ping works)
>>>>>>> https://signal.klimonda.com/ovnnb.db-working.txt - this is after I create a separate IP-based rule to allow TCP traffic
>>>>>>>
>>>>>>> In both examples, security group in question is ed081ef3-754a-492f-80b2-fb73cd2dceed which is mapped to pg_ed081ef3_754a_492f_80b2_fb73cd2dceed port group.
>>>>>>>
>>>>>>> In the second ovnnb.db (ovnnb.db-working.txt), there is an extra ACL fb464efc-f63b-494b-b59b-6c2860dcecba added from CLI via:
>>>>>>>
>>>>>>> `openstack security group rule create --ingress --protocol tcp --remote-ip 172.16.0.0/24 default`
>>>>>>>
>>>>>>> -- Krzysztof Klimonda kklimonda at syntaxhighlighted.com On Mon, Dec 14,
>>>>>>> 2020, at 14:14, Numan Siddique wrote:
>>>>>>>> On Mon, Dec 14, 2020 at 4:01 PM Krzysztof Klimonda
>>>>>>>> <kklimonda at syntaxhighlighted.com> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> After upgrading to OVN 20.12.0 snapshot d8bc0377c I've noticed a problem in communication between VMs that use allowed address pairs and remote group id in security groups. I believe it has worked properly with OVN 20.06.2 release (although I have no way of verifying it right now).
>>>>>>>> Thanks for reporting the issue.
>>>>>>>>
>>>>>>>> Is it possible for you to share the OVN NB DB somewhere ?
>>>>>>>>
>>>>>>>> It would be easier to reproduce the issue with the DB.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Numan
>>>>>>>>
>>>>>>>>> Given the following scenario:
>>>>>>>>>
>>>>>>>>> - 2 VMs with IP addresses: vm-a with IP addresses 10.0.0.10 and 172.16.0.10 and vm-b with IP addresses 10.0.0.11 and 172.16.0.11 where 10.0.0.0/8 addresses are set on ports, and 172.16.0.0/16 addresses are set in allowed-address for on ports
>>>>>>>>> - There is single security group attached to both ports allowing for ingress tcp traffic coming from the same security group (remote-group)
>>>>>>>>> - There is a TCP service listening on 10.0.0.10 on port 8000
>>>>>>>>>
>>>>>>>>> When I try accessing service from vm-b using 10.0.0.10 address, ovn forwards traffic properly. However, when I try accessing same service via 172.16.0.10 traffic is dropped.
>>>>>>>>>
>>>>>>>>> When I trace packets between VMs using ovn-trace, for first scenario the last step is:
>>>>>>>>>
>>>>>>>>> ----8<----8<----
>>>>>>>>> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
>>>>>>>>> ---------------------------------------------------------------
>>>>>>>>> 4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid ab5a233e
>>>>>>>>> reg0[8] = 1;
>>>>>>>>> reg0[10] = 1;
>>>>>>>>> next;
>>>>>>>>> 5. ls_out_acl (ovn-northd.c:5498): reg0[8] == 1 && (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed && ip4 && ip4.src == $pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4 && tcp), priority 2002, uuid d92706d4
>>>>>>>>> next;
>>>>>>>>> 9. ls_out_port_sec_ip (ovn-northd.c:4525): outport == "864929" && eth.dst == fa:16:3e:bc:20:10 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.0.0.10, 172.16.0.0/16}, priority 90, uuid ff3390b1
>>>>>>>>> next;
>>>>>>>>> 10. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "864929" && eth.dst == {fa:16:3e:bc:20:10}, priority 50, uuid af91c05c
>>>>>>>>> output;
>>>>>>>>> /* output to "864929", type "" */
>>>>>>>>> ----8<----8<----
>>>>>>>>>
>>>>>>>>> However, when I use 172.16.0.0/24 addresses, the last step changes to:
>>>>>>>>>
>>>>>>>>> ----8<----8<----
>>>>>>>>> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
>>>>>>>>> ---------------------------------------------------------------
>>>>>>>>> 4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid ab5a233e
>>>>>>>>> reg0[8] = 1;
>>>>>>>>> reg0[10] = 1;
>>>>>>>>> next;
>>>>>>>>> 5. ls_out_acl (ovn-northd.c:5553): reg0[10] == 1 && (outport == @neutron_pg_drop && ip), priority 2001, uuid e36c0840
>>>>>>>>> ct_commit { ct_label.blocked = 1; };
>>>>>>>>> ----8<----8<----
>>>>>>>>>
>>>>>>>>> Further notes:
>>>>>>>>>
>>>>>>>>> - ICMP traffic between 172.16.0.0/24 addresses is forwarded correctly, with last step of ovn-trace being:
>>>>>>>>>
>>>>>>>>> ----8<----8<----
>>>>>>>>> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
>>>>>>>>> ---------------------------------------------------------------
>>>>>>>>> 4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid ab5a233e
>>>>>>>>> reg0[8] = 1;
>>>>>>>>> reg0[10] = 1;
>>>>>>>>> next;
>>>>>>>>> 5. ls_out_acl (ovn-northd.c:5498): reg0[8] == 1 && (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed && ip4 && ip4.src == 0.0.0.0/0 && icmp4), priority 2002, uuid cd1705d8
>>>>>>>>> next;
>>>>>>>>> 9. ls_out_port_sec_ip (ovn-northd.c:4525): outport == "864929" && eth.dst == fa:16:3e:bc:20:10 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.0.0.10, 172.16.0.0/16}, priority 90, uuid ff3390b1
>>>>>>>>> next;
>>>>>>>>> 10. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "864929" && eth.dst == {fa:16:3e:bc:20:10}, priority 50, uuid af91c05c
>>>>>>>>> output;
>>>>>>>>> /* output to "864929", type "" */
>>>>>>>>> ----8<----8<----
>>>>>>>>>
>>>>>>>>> - If I replace security group rule, changing remote group to remote ip, traffic is forwarded correctly and last step in ovn-trace is:
>>>>>>>>>
>>>>>>>>> ----8<----8<----
>>>>>>>>> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
>>>>>>>>> ---------------------------------------------------------------
>>>>>>>>> 4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl && ct_label.blocked == 0, priority 4, uuid ab5a233e
>>>>>>>>> reg0[8] = 1;
>>>>>>>>> reg0[10] = 1;
>>>>>>>>> next;
>>>>>>>>> 5. ls_out_acl (ovn-northd.c:5498): reg0[8] == 1 && (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed && ip4 && ip4.src == 172.16.0.0/24 && tcp), priority 2002, uuid a0871ca2
>>>>>>>>> next;
>>>>>>>>> 9. ls_out_port_sec_ip (ovn-northd.c:4525): outport == "864929" && eth.dst == fa:16:3e:bc:20:10 && ip4.dst == {255.255.255.255, 224.0.0.0/4, 10.0.0.10, 172.16.0.0/16}, priority 90, uuid ff3390b1
>>>>>>>>> next;
>>>>>>>>> 10. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "864929" && eth.dst == {fa:16:3e:bc:20:10}, priority 50, uuid af91c05c
>>>>>>>>> output;
>>>>>>>>> /* output to "864929", type "" */
>>>>>>>>> ----8<----8<----
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Krzysztof Klimonda
>>>>>>>>> kklimonda at syntaxhighlighted.com
>>>>>>>>> _______________________________________________
>>>>>>>>> discuss mailing list
>>>>>>>>> discuss at openvswitch.org
>>>>>>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>>>>>> _______________________________________________
>>>>>>> discuss mailing list
>>>>>>> discuss at openvswitch.org
>>>>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> discuss mailing list
>>>> discuss at openvswitch.org
>>>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>>>
>>>
>>> *Attachments:*
>>> * signature.asc
>>
>> _______________________________________________
>> discuss mailing list
>> discuss at openvswitch.org
>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20201215/2ee57da4/attachment-0001.html>
More information about the discuss
mailing list