[ovs-discuss] [ovn] TCP/UDP traffic dropped between IP addresses defined in allowed address pairs when remote security group is used

Daniel Alvarez Sanchez dalvarez at redhat.com
Tue Dec 15 14:42:50 UTC 2020


On Tue, Dec 15, 2020 at 2:58 PM Krzysztof Klimonda <
kklimonda at syntaxhighlighted.com> wrote:

> Hi Flavio, Dumitru
>
> See inline for replies
>
>
> On Tue, Dec 15, 2020, at 12:37, Flavio Fernandes wrote:
>
>
>
> On Dec 15, 2020, at 6:20 AM, Dumitru Ceara <dceara at redhat.com> wrote:
>
> On 12/15/20 12:02 PM, Krzysztof Klimonda wrote:
>
> Hi Dumitru,
>
> Thanks for checking it out.
>
> On Tue, Dec 15, 2020, at 10:45, Dumitru Ceara wrote:
>
> Hi Krzysztof,
>
> Thanks for the DBs and all the details.
>
> I gave it a try on my local setup, using your DBs.  The behavior below
> is identical on v20.06.2, branch-20.12 and current master.
>
>
> I have to admit I'm not sure why in your tests the behaviour is the same
> in 20.06.2 - unfortunately I no longer have access to the environment with
> that version, so I can't re-verify that on my end and I based it on the
> fact that this SG was created by magnum for kubernetes cluster deployed on
> top of openstack, and the cluster was being tested successfully. The
> earliest I can try recreating this with 20.06.2 release is next year, so
> I'm not sure if this will be of any help.
>
>
> I just rechecked with 20.06.2 and I'm 100% sure that with the DBs you
> provided and the behavior is exactly the same as with newer branches
> (including latest master).
>
> [...]
>
>
> For TCP traffic 172.16.0.11 -> 172.16.0.10 we don't hit any of the
> allow-related ACLs.  172.16.0.11 is not set as a port address on port
> 81d23182-37ac-4d3d-815e-4c25d26fe154 so it will not be included in the
> auto-generated address_set pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4.
>
> On the other hand, for TCP traffic 10.0.0.11 -> 172.16.0.10 we hit the
> 4th allow-related ACL:
> to-lport  1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed &&
> ip4 && ip4.src == $pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4 && tcp)
> allow-related
>
> ICMP traffic matches the 5th allow-related ACL which doesn't check
> source IPs:
> to-lport  1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed &&
> ip4 && ip4.src == 0.0.0.0/0 && icmp4) allow-related
>
> So, unless I'm missing something, OVN is doing what it's supposed to do
> based on the current configuration.  To avoid adding a new ACL, one
> option would be to add the 172.16.0.10 and 172.16.0.11 IPs to the
> corresponding logical_switch_ports "addresses" column, i.e., something
> like:
>
>
> The problem with adding those addresses to LSP is that they are outside of
> OVN/neutron control - this is a kubernetes cluster deployed with magnum,
> where CNI plugin (calico) doesn't use any tunneling between k8s nodes, each
> pod has address in 172.16.0.0/16 subnet and uses addresses from
> 172.16.0.0/16 to communicate witch other pods. To accomodate that ports
> have 172.16.0.0/16 added to their allowed addresses.
>
> I'm assuming that, if OVN is doing what it's supposed to be doing based on
> the configuration[1], then there is a mismatch between neutron and OVN
> behaviour in regards to SG with allowed address pairs? I guess someone from
> neutron team would have to comment whether it's
>
>
> I agree, some input from neutron would be great. (cc-ing Daniel).
>
>
> Hi folks. I'm caching up on this thread and have a follow up
> question/request.
>
> @Krzysztof: can you tell me a little more about how the ips 172.16.0.10
> and 172.16.0.11 are added as
> secondary on the neutron ports? Are you using multiple fixed ip addresses?
> Do you have 2 neutron subnets configured
> on the same neutron network?
>
> The reason I ask is because a neutron network and its associated neutron
> subnet(s) are "aggregated" into a single logical switch
> in OVN. In fact, neutron subnets have no corresponding row in OVN. So I
> wonder if what you really want are 2 separate neutron
> ports for each vm, instead of a single port with overlapping subnets.
>
>
> Not sure what do you mean by overlapping subnets - the subnets in question
> are 10.0.0.0/8 and 172.16.0.0/16.
>
> The IP addresses are added either as secondary IP addresses set on the
> interface inside VM (in my manual testing), or are result of calico CNI
> using "physical" interface (as opposed to IPIP tunnel) for pod-to-pod
> communication - in that case each pod gets IP address from 172.16.0.0/16
> and Linux just pushes packets based on routing rules (internally
> 172.16.0.0/16 is split into smaller networks allocated to each k8s node,
> and each network receives an entry in routing table that push packets to a
> specific node).
>
> Whether it should be a separate neutron port is an excellent question -
> this security group configuration comes from magnum, which configures port
> with Pod/service(?) network added to allowed address on all kubernetes
> nodes, and security group being configured to allow for all TCP/UDP traffic
> between ports with that SG attached. There is entire separate
> kuryr-kubernetes project that tries to marry kubernetes CNI with neutron
> directly, but in our very limited testing it requires more work to be
> stable in our deployments.
>
> See
> https://github.com/openstack/magnum/blob/c556b8964fab129f33e766b1c33908b2eb001df4/magnum/drivers/k8s_fedora_coreos_v1/templates/kubeminion.yaml for
> where magnum sets up the allowed_address_pairs attribute on the port, and
> https://github.com/openstack/magnum/blob/c556b8964fab129f33e766b1c33908b2eb001df4/magnum/drivers/k8s_fedora_coreos_v1/templates/kubecluster.yaml#L1038 for
> where security group is defined.
>
>
> Thanks,
>
> -- flaviof
>
>
>
> [1] I'm not entirely convinced that's the case given that ICMP traffic is
> being forwarded - I see how it's doing what programmed flows are telling it
> to do, but that doesn't seem to be expected result.
>
>
> I'm confused, why wouldn't this be expected?
>
>
> From (openstack) user's perspective if there are two security groups:
> - allow for ICMP traffic between ports in the same security group
> - allow for TCP traffic between ports in the same security group
>
> and one of the rules is working, but the other one is not, then this is
> not an expected behaviour (again, from openstack user perspective). It's
> definitely possible that there is a disconnect between what's expected in
> that case between neutron and ovn - perhaps neutron should add additional
> entries to OVN's northbound db to support that specific security group
> feature?
>


I agree with everybody here but... :)

- Does it work with ML2/OVS? Are addresses in the allowed-address pairs
accounted for remote groups (for example, Floating IPs are not).
- If the answer to the above is yes, then I'd agree that we need to fix it
in OpenStack by:
a) Adding the allowed-address pairs to the 'addresses' column of the
Logical_Switch_Port row assuming that this doesn't break DHCP
b) Adding more conditions to the ACL to match not only on the $pg_xxx
Address_Set but also on each and every IP address of the allowed-addresses.
- If the answer to the above is yes, we need to add coverage in OpenStack
as CI is clearly not testing this path
- If the answer to the above is no, then you need to add explicit Neutron
Security Group rules to allow traffic originated from the allowed-address
pairs (you can do this for now as a workaround)


Chris, are you able to test this with ML2/OVS and report back the results?

Either way I don't think there's anything wrong with OVN and we can move
this discussion to the openstack-discuss ML :)

Thanks,
daniel

>
>
> The ICMP ACL explicitly allows traffic to any port in the port group if
> it's ICMP and comes from any IP source:
>
> to-lport  1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed &&
> ip4 && ip4.src == 0.0.0.0/0 && icmp4) allow-related
>
> For TCP the match is more restrictive:
>
> The TCP ACL allows traffic to any port in the port group if the IP
> source is part of the addresses defined on the logical ports that are
> part of the port group:
>
> to-lport  1002 (outport == @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed
> &&ip4 && ip4.src == $pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4 && tcp)
> allow-related
>
> But the traffic comes from 172.16.0.11 which is not an address on the
> logical switch port.  So traffic doesn't match the allow ACL and instead
> matches the deny-all ACL.
>
> OVN cannot automatically deduce anything from the fact that
> port_security includes 172.16.0.0/16 and without help from the CMS it
> cannot decide to allow traffic from 172.16.0.11.
>
> Regards,
> Dumitru
>
> Best Regards,
>  Chris
>
>
> Thanks,
> Dumitru
>
>
> On 12/14/20 4:43 PM, Krzysztof Klimonda wrote:
>
> Hi Numan,
>
> https://signal.klimonda.com/ovnnb.db-broken.txt - this is the "initial"
> state where TCP is not being established (but ping works)
> https://signal.klimonda.com/ovnnb.db-working.txt - this is after I create
> a separate IP-based rule to allow TCP traffic
>
> In both examples, security group in question is
> ed081ef3-754a-492f-80b2-fb73cd2dceed which is mapped to
> pg_ed081ef3_754a_492f_80b2_fb73cd2dceed port group.
>
> In the second ovnnb.db (ovnnb.db-working.txt), there is an extra ACL
> fb464efc-f63b-494b-b59b-6c2860dcecba added from CLI via:
>
> `openstack security group rule create --ingress --protocol tcp --remote-ip
> 172.16.0.0/24 default`
>
> -- Krzysztof Klimonda kklimonda at syntaxhighlighted.com On Mon, Dec 14,
> 2020, at 14:14, Numan Siddique wrote:
>
> On Mon, Dec 14, 2020 at 4:01 PM Krzysztof Klimonda
> <kklimonda at syntaxhighlighted.com> wrote:
>
> Hi,
>
> After upgrading to OVN 20.12.0 snapshot d8bc0377c I've noticed a problem
> in communication between VMs that use allowed address pairs and remote
> group id in security groups. I believe it has worked properly with OVN
> 20.06.2 release (although I have no way of verifying it right now).
>
> Thanks for reporting the issue.
>
> Is it possible for you to share the OVN NB DB somewhere ?
>
> It would be easier to reproduce the issue with the DB.
>
> Thanks
> Numan
>
> Given the following scenario:
>
> - 2 VMs with IP addresses: vm-a with IP addresses 10.0.0.10 and
> 172.16.0.10 and vm-b with IP addresses 10.0.0.11 and 172.16.0.11 where
> 10.0.0.0/8 addresses are set on ports, and 172.16.0.0/16 addresses are
> set in allowed-address for on ports
> - There is single security group attached to both ports allowing for
> ingress tcp traffic coming from the same security group (remote-group)
> - There is a TCP service listening on 10.0.0.10 on port 8000
>
> When I try accessing service from vm-b using 10.0.0.10 address, ovn
> forwards traffic properly. However, when I try accessing same service via
> 172.16.0.10 traffic is dropped.
>
> When I trace packets between VMs using ovn-trace, for first scenario the
> last step is:
>
> ----8<----8<----
> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
> ---------------------------------------------------------------
> 4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl &&
> ct_label.blocked == 0, priority 4, uuid ab5a233e
>    reg0[8] = 1;
>    reg0[10] = 1;
>    next;
> 5. ls_out_acl (ovn-northd.c:5498): reg0[8] == 1 && (outport ==
> @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed && ip4 && ip4.src ==
> $pg_ed081ef3_754a_492f_80b2_fb73cd2dceed_ip4 && tcp), priority 2002, uuid
> d92706d4
>    next;
> 9. ls_out_port_sec_ip (ovn-northd.c:4525): outport == "864929" && eth.dst
> == fa:16:3e:bc:20:10 && ip4.dst == {255.255.255.255, 224.0.0.0/4,
> 10.0.0.10, 172.16.0.0/16}, priority 90, uuid ff3390b1
>    next;
> 10. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "864929" && eth.dst
> == {fa:16:3e:bc:20:10}, priority 50, uuid af91c05c
>    output;
>    /* output to "864929", type "" */
> ----8<----8<----
>
> However, when I use 172.16.0.0/24 addresses, the last step changes to:
>
> ----8<----8<----
> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
> ---------------------------------------------------------------
> 4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl &&
> ct_label.blocked == 0, priority 4, uuid ab5a233e
>    reg0[8] = 1;
>    reg0[10] = 1;
>    next;
> 5. ls_out_acl (ovn-northd.c:5553): reg0[10] == 1 && (outport ==
> @neutron_pg_drop && ip), priority 2001, uuid e36c0840
>    ct_commit { ct_label.blocked = 1; };
> ----8<----8<----
>
> Further notes:
>
> - ICMP traffic between 172.16.0.0/24 addresses is forwarded correctly,
> with last step of ovn-trace being:
>
> ----8<----8<----
> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
> ---------------------------------------------------------------
> 4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl &&
> ct_label.blocked == 0, priority 4, uuid ab5a233e
>    reg0[8] = 1;
>    reg0[10] = 1;
>    next;
> 5. ls_out_acl (ovn-northd.c:5498): reg0[8] == 1 && (outport ==
> @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed && ip4 && ip4.src == 0.0.0.0/0
> && icmp4), priority 2002, uuid cd1705d8
>    next;
> 9. ls_out_port_sec_ip (ovn-northd.c:4525): outport == "864929" && eth.dst
> == fa:16:3e:bc:20:10 && ip4.dst == {255.255.255.255, 224.0.0.0/4,
> 10.0.0.10, 172.16.0.0/16}, priority 90, uuid ff3390b1
>    next;
> 10. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "864929" && eth.dst
> == {fa:16:3e:bc:20:10}, priority 50, uuid af91c05c
>    output;
>    /* output to "864929", type "" */
> ----8<----8<----
>
> - If I replace security group rule, changing remote group to remote ip,
> traffic is forwarded correctly and last step in ovn-trace is:
>
> ----8<----8<----
> ct_next(ct_state=est|trk /* default (use --ct to customize) */)
> ---------------------------------------------------------------
> 4. ls_out_acl_hint (ovn-northd.c:5292): !ct.new && ct.est && !ct.rpl &&
> ct_label.blocked == 0, priority 4, uuid ab5a233e
>    reg0[8] = 1;
>    reg0[10] = 1;
>    next;
> 5. ls_out_acl (ovn-northd.c:5498): reg0[8] == 1 && (outport ==
> @pg_ed081ef3_754a_492f_80b2_fb73cd2dceed && ip4 && ip4.src ==
> 172.16.0.0/24 && tcp), priority 2002, uuid a0871ca2
>    next;
> 9. ls_out_port_sec_ip (ovn-northd.c:4525): outport == "864929" && eth.dst
> == fa:16:3e:bc:20:10 && ip4.dst == {255.255.255.255, 224.0.0.0/4,
> 10.0.0.10, 172.16.0.0/16}, priority 90, uuid ff3390b1
>    next;
> 10. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "864929" && eth.dst
> == {fa:16:3e:bc:20:10}, priority 50, uuid af91c05c
>    output;
>    /* output to "864929", type "" */
> ----8<----8<----
>
> --
>  Krzysztof Klimonda
>  kklimonda at syntaxhighlighted.com
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
>
>
>
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
>
>
> *Attachments:*
>
>    - signature.asc
>
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20201215/951cf7c8/attachment-0001.html>


More information about the discuss mailing list