[ovs-dev] HA: HA: SNAT on OVN logical_router in userspace works for ICMP but not TCP or UDP

Rostyslav Fridman Rostyslav_Fridman at epam.com
Wed Feb 13 15:25:49 UTC 2019


Okay, I have managed to make it work, however I do not know if this is how it is supposed to work.

Basically, I reduced OVN scheme from
container – internal switch – (container subnet gateway) internal router – interconnect switch – external router (NAT IP address) – external switch
to
container – internal switch – (container subnet gateway) external router (NAT IP address) – external switch
The difference being that local subnet for NAT resides on the same external router as an external NAT IP address. In this case both ICMP and TCP/UDP NAT translations work. It did not work for TCP when local subnet and external IP resided on different OVN routers.

This behavior seems strange, am I missing something here?

От: Darrell Ball [mailto:dlu998 at gmail.com]
Отправлено: 13 февраля 2019 г. 17:11
Кому: Rostyslav Fridman <Rostyslav_Fridman at epam.com>
Копия: Darrell Ball <dball at vmware.com>; Ben Pfaff <blp at ovn.org>; ovs-dev at openvswitch.org; Vasyl Samoilov <Vasyl_Samoilov at epam.com>
Тема: Re: [ovs-dev] HA: SNAT on OVN logical_router in userspace works for ICMP but not TCP or UDP



On Wed, Feb 13, 2019 at 5:16 AM Rostyslav Fridman via dev <ovs-dev at openvswitch.org<mailto:ovs-dev at openvswitch.org>> wrote:
Hi Darrel,

I've reduced the amount of traffic. Here is dump-flows for ICMP traffic:
ovs-appctl  dpif/dump-flows br-int
ct_state(-new+est-rel-inv+trk),recirc_id(0x303b),in_port(8),packet_type(ns=0,id=0),eth(src=00:00:00:61:f0:c0,dst=00:25:90:e7:23:94),eth_type(0x0800),ipv4(frag=no), packets:6, bytes:588, used:0.994s, actions:ct_clear,push_vlan(vid=111,pcp=0),hash(l4(0)),recirc(0x1)
ct_state(-new-est-rel-inv-trk),recirc_id(0),in_port(8),packet_type(ns=0,id=0),eth(src=0a:00:00:00:00:03,dst=00:00:00:6b:83:b1),eth_type(0x0806),arp(sip=10.0.0.2,tip=10.0.3.253,op=1/0xff,sha=0a:00:00:00:00:03,tha=00:00:00:00:00:00), packets:0, bytes:0, used:never, actions:userspace(pid=0,slow_path(action))
recirc_id(0x1),dp_hash(0x9eeb76ae/0xff),in_port(8),packet_type(ns=0,id=0),eth_type(0x8100),vlan(vid=111,pcp=0),encap(eth_type(0x0800),ipv4(frag=no)), packets:7, bytes:714, used:0.994s, actions:4
ct_state(+new-est-rel-inv+trk),recirc_id(0x303b),in_port(8),packet_type(ns=0,id=0),eth(src=00:00:00:61:f0:c0,dst=00:25:90:e7:23:94),eth_type(0x0800),ipv4(frag=no), packets:0, bytes:0, used:never, actions:ct_clear,push_vlan(vid=111,pcp=0),hash(l4(0)),recirc(0x1)
ct_state(+new-est-rel-inv+trk),recirc_id(0x303a),in_port(8),packet_type(ns=0,id=0),eth(src=00:00:00:73:a8:30,dst=00:00:00:da:6b:85),eth_type(0x0800),ipv4(src=10.0.0.2/255.0.0.0,dst=216.58.215.110/128.0.0.0,ttl=63,frag=no<http://10.0.0.2/255.0.0.0,dst=216.58.215.110/128.0.0.0,ttl=63,frag=no>), packets:7, bytes:686, used:0.994s, actions:set(eth(src=00:00:00:61:f0:c0,dst=00:25:90:e7:23:94)),set(ipv4(src=10.0.0.0/255.0.0.0,dst=128.0.0.0/128.0.0.0,ttl=62)<http://10.0.0.0/255.0.0.0,dst=128.0.0.0/128.0.0.0,ttl=62)>),ct(commit,zone=3,nat(src=10.250.111.40)),recirc(0x303b)
ct_state(+new-est-rel-inv+trk),recirc_id(0x3039),in_port(8),packet_type(ns=0,id=0),eth(src=0a:00:00:00:00:03,dst=00:00:00:6b:83:b1),eth_type(0x0800),ipv4(src=10.0.0.2/255.255.254.0,dst=216.58.215.110/224.0.0.0,ttl=64,frag=no<http://10.0.0.2/255.255.254.0,dst=216.58.215.110/224.0.0.0,ttl=64,frag=no>), packets:7, bytes:686, used:0.994s, actions:ct_clear,ct_clear,set(eth(src=00:00:00:73:a8:30,dst=00:00:00:da:6b:85)),set(ipv4(src=10.0.0.0/255.255.254.0,dst=192.0.0.0/224.0.0.0,ttl=63)<http://10.0.0.0/255.255.254.0,dst=192.0.0.0/224.0.0.0,ttl=63)>),ct(zone=2,nat),recirc(0x303a)
ct_state(-new-est-rel-inv-trk),recirc_id(0),in_port(8),packet_type(ns=0,id=0),eth(src=0a:00:00:00:00:03/01:00:00:00:00:00,dst=00:00:00:6b:83:b1),eth_type(0x0800),ipv4(src=10.0.0.2/255.255.254.0,dst=216.58.215.110/224.0.0.0,ttl=64,frag=no<http://10.0.0.2/255.255.254.0,dst=216.58.215.110/224.0.0.0,ttl=64,frag=no>), packets:7, bytes:686, used:0.994s, actions:ct_clear,ct(zone=5,nat),recirc(0x3039)


Here is dump-flows for TCP traffic:
ovs-appctl  dpif/dump-flows br-int
ct_state(-new-est-rel+inv+trk),recirc_id(0x3038),in_port(8),packet_type(ns=0,id=0),eth(src=00:00:00:61:f0:c0,dst=00:25:90:e7:23:94),eth_type(0x0800),ipv4(frag=no), packets:2, bytes:148, used:3.064s, flags:S, actions:ct_clear,push_vlan(vid=111,pcp=0),hash(l4(0)),recirc(0x1)
ct_state(-new-est-rel-inv-trk),recirc_id(0),in_port(8),packet_type(ns=0,id=0),eth(src=0a:00:00:00:00:03,dst=00:00:00:6b:83:b1),eth_type(0x0806),arp(sip=10.0.0.2,tip=10.0.3.253,op=1/0xff,sha=0a:00:00:00:00:03,tha=00:00:00:00:00:00), packets:0, bytes:0, used:never, actions:userspace(pid=0,slow_path(action))
recirc_id(0x1),dp_hash(0x5be46ea1/0xff),in_port(8),packet_type(ns=0,id=0),eth_type(0x8100),vlan(vid=111,pcp=0),encap(eth_type(0x0800),ipv4(frag=no)), packets:2, bytes:156, used:3.064s, flags:S, actions:5
ct_state(-new-est-rel+inv+trk),recirc_id(0x3037),in_port(8),packet_type(ns=0,id=0),eth(src=00:00:00:73:a8:30,dst=00:00:00:da:6b:85),eth_type(0x0800),ipv4(src=10.0.0.2/255.0.0.0,dst=216.58.215.110/128.0.0.0,ttl=63,frag=no<http://10.0.0.2/255.0.0.0,dst=216.58.215.110/128.0.0.0,ttl=63,frag=no>), packets:2, bytes:148, used:3.064s, flags:S, actions:set(eth(src=00:00:00:61:f0:c0,dst=00:25:90:e7:23:94)),set(ipv4(src=10.0.0.0/255.0.0.0,dst=128.0.0.0/128.0.0.0,ttl=62)<http://10.0.0.0/255.0.0.0,dst=128.0.0.0/128.0.0.0,ttl=62)>),ct(commit,zone=3,nat(src=10.250.111.40)),recirc(0x3038)
ct_state(-new-est-rel+inv+trk),recirc_id(0x3036),in_port(8),packet_type(ns=0,id=0),eth(src=0a:00:00:00:00:03,dst=00:00:00:6b:83:b1),eth_type(0x0800),ipv4(src=10.0.0.2/255.255.254.0,dst=216.58.215.110/224.0.0.0,ttl=64,frag=no<http://10.0.0.2/255.255.254.0,dst=216.58.215.110/224.0.0.0,ttl=64,frag=no>), packets:2, bytes:148, used:3.064s, flags:S, actions:ct_clear,ct_clear,set(eth(src=00:00:00:73:a8:30,dst=00:00:00:da:6b:85)),set(ipv4(src=10.0.0.0/255.255.254.0,dst=192.0.0.0/224.0.0.0,ttl=63)<http://10.0.0.0/255.255.254.0,dst=192.0.0.0/224.0.0.0,ttl=63)>),ct(zone=2,nat),recirc(0x3037)
ct_state(-new-est-rel-inv-trk),recirc_id(0),in_port(8),packet_type(ns=0,id=0),eth(src=0a:00:00:00:00:03/01:00:00:00:00:00,dst=00:00:00:6b:83:b1),eth_type(0x0800),ipv4(src=10.0.0.2/255.255.254.0,dst=216.58.215.110/224.0.0.0,ttl=64,frag=no<http://10.0.0.2/255.255.254.0,dst=216.58.215.110/224.0.0.0,ttl=64,frag=no>), packets:2, bytes:148, used:3.064s, flags:S, actions:ct_clear,ct(zone=5,nat),recirc(0x3036)

This flow looks particularly wrong; it would appear the flow generation has a problem.

ct_state(-new-est-rel+inv+trk),recirc_id(0x3037),in_port(8),packet_type(ns=0,id=0),eth(src=00:00:00:73:a8:30,dst=00:00:00:da:6b:85),eth_type(0x0800),ipv4(src=10.0.0.2/255.0.0.0,dst=216.58.215.110/128.0.0.0,ttl=63,frag=no<http://10.0.0.2/255.0.0.0,dst=216.58.215.110/128.0.0.0,ttl=63,frag=no>), packets:2, bytes:148, used:3.064s, flags:S, actions:set(eth(src=00:00:00:61:f0:c0,dst=00:25:90:e7:23:94)),set(ipv4(src=10.0.0.0/255.0.0.0,dst=128.0.0.0/128.0.0.0,ttl=62)<http://10.0.0.0/255.0.0.0,dst=128.0.0.0/128.0.0.0,ttl=62)>),ct(commit,zone=3,nat(src=10.250.111.40)),recirc(0x3038)

btw; I don't see something corresponding to this in previous OF flow dump link you sent.

Probably, you need to dump both OVN Southbound flows and Openflow flows to see where these flows originate.



As can be seen from output no conntrack entry is created for new state.


-----Исходное сообщение-----
От: Darrell Ball [mailto:dball at vmware.com<mailto:dball at vmware.com>]
Отправлено: 9 февраля 2019 г. 0:29
Кому: Rostyslav Fridman <Rostyslav_Fridman at epam.com<mailto:Rostyslav_Fridman at epam.com>>; Ben Pfaff <blp at ovn.org<mailto:blp at ovn.org>>
Копия: ovs-dev at openvswitch.org<mailto:ovs-dev at openvswitch.org>; Vasyl Samoilov <Vasyl_Samoilov at epam.com<mailto:Vasyl_Samoilov at epam.com>>
Тема: Re: [ovs-dev] SNAT on OVN logical_router in userspace works for ICMP but not TCP or UDP

If TCP packets do not go thru conntrack, then that would explain why the TCP traffic is not natted (since you don't have any other rules that could do that)

You need to find out where the TCP packets are going.
Try making the rules L4 protocol specific (i.e. look for TCP and also do something different for ICMP) Maybe add some other debug rules to trace the TCP packets otherwise



On 2/8/19, 1:47 PM, "Rostyslav Fridman" <Rostyslav_Fridman at epam.com<mailto:Rostyslav_Fridman at epam.com>> wrote:

    I have sent TCP traffic. It does not show up in dump-conntrack for some reason. However, I see it on the external server.

    -----Исходное сообщение-----
    От: Darrell Ball [mailto:dball at vmware.com<mailto:dball at vmware.com>]
    Отправлено: 8 февраля 2019 г. 23:29
    Кому: Rostyslav Fridman <Rostyslav_Fridman at epam.com<mailto:Rostyslav_Fridman at epam.com>>; Ben Pfaff <blp at ovn.org<mailto:blp at ovn.org>>
    Копия: ovs-dev at openvswitch.org<mailto:ovs-dev at openvswitch.org>; Vasyl Samoilov <Vasyl_Samoilov at epam.com<mailto:Vasyl_Samoilov at epam.com>>
    Тема: Re: [ovs-dev] SNAT on OVN logical_router in userspace works for ICMP but not TCP or UDP

    I thought the problem was with TCP/UDP traffic ?
    Did you send TCP traffic for this test ?; if not, can you run the test with TCP ?



    On 2/8/19, 12:53 PM, "Rostyslav Fridman" <Rostyslav_Fridman at epam.com<mailto:Rostyslav_Fridman at epam.com>> wrote:

        # ovs-appctl dpif/dump-flows br-int
        recirc_id(0x1),dp_hash(0x9eeb76ae/0xff),in_port(8),packet_type(ns=0,id=0),eth_type(0x8100),vlan(vid=111,pcp=0),encap(eth_type(0x0800),ipv4(frag=no)), packets:20, bytes:2040, used:0.942s, actions:4
        ct_state(-new-est-rel-inv-trk),recirc_id(0),in_port(8),packet_type(ns=0,id=0),eth(src=0a:00:00:00:00:03/01:00:00:00:00:00,dst=00:00:00:6b:83:b1),eth_type(0x0800),ipv4(src=10.0.0.2/255.255.254.0,dst=216.58.215.110/224.0.0.0,ttl=64,frag=no<http://10.0.0.2/255.255.254.0,dst=216.58.215.110/224.0.0.0,ttl=64,frag=no>), packets:25, bytes:2354, used:0.942s, flags:S, actions:ct_clear,ct(zone=5,nat),recirc(0xb1)
        ct_state(+new-est-rel-inv+trk),recirc_id(0xb2),in_port(8),packet_type(ns=0,id=0),eth(src=00:00:00:73:a8:30,dst=00:00:00:da:6b:85),eth_type(0x0800),ipv4(src=10.0.0.2/255.0.0.0,dst=216.58.215.110/128.0.0.0,ttl=63,frag=no<http://10.0.0.2/255.0.0.0,dst=216.58.215.110/128.0.0.0,ttl=63,frag=no>), packets:20, bytes:1960, used:0.942s, actions:set(eth(src=00:00:00:61:f0:c0,dst=00:25:90:e7:23:94)),set(ipv4(src=10.0.0.0/255.0.0.0,dst=128.0.0.0/128.0.0.0,ttl=62)<http://10.0.0.0/255.0.0.0,dst=128.0.0.0/128.0.0.0,ttl=62)>),ct(commit,zone=3,nat(src=10.250.111.40)),recirc(0xb3)
        ct_state(+new-est-rel-inv+trk),recirc_id(0xb1),in_port(8),packet_type(ns=0,id=0),eth(src=0a:00:00:00:00:03,dst=00:00:00:6b:83:b1),eth_type(0x0800),ipv4(src=10.0.0.2/255.255.254.0,dst=216.58.215.110/224.0.0.0,ttl=64,frag=no<http://10.0.0.2/255.255.254.0,dst=216.58.215.110/224.0.0.0,ttl=64,frag=no>), packets:20, bytes:1960, used:0.942s, actions:ct_clear,ct_clear,set(eth(src=00:00:00:73:a8:30,dst=00:00:00:da:6b:85)),set(ipv4(src=10.0.0.0/255.255.254.0,dst=192.0.0.0/224.0.0.0,ttl=63)<http://10.0.0.0/255.255.254.0,dst=192.0.0.0/224.0.0.0,ttl=63)>),ct(zone=2,nat),recirc(0xb2)
        ct_state(-new+est-rel-inv+trk),recirc_id(0xb3),in_port(8),packet_type(ns=0,id=0),eth(src=00:00:00:61:f0:c0,dst=00:25:90:e7:23:94),eth_type(0x0800),ipv4(frag=no), packets:19, bytes:1862, used:0.942s, actions:ct_clear,push_vlan(vid=111,pcp=0),hash(l4(0)),recirc(0x1)

        ==================================

        # ovs-appctl dpctl/dump-conntrack
        icmp,orig=(src=10.0.0.2,dst=216.58.215.110,id=246,type=8,code=0),reply=(src=216.58.215.110,dst=10.250.111.40,id=246,type=0,code=0),zone=3



_______________________________________________
dev mailing list
dev at openvswitch.org<mailto:dev at openvswitch.org>
https://mail.openvswitch.org/mailman/listinfo/ovs-dev


More information about the dev mailing list