[ovs-discuss] Router not allowing traffic in reverse

John Carew johnwcarew at gmail.com
Fri Mar 1 14:19:11 UTC 2019


Thanks Ben! I will try to walk through this today and see how much miles I can get out of it.

John

-----Original Message-----
From: Ben Pfaff <blp at ovn.org> 
Sent: Thursday, February 28, 2019 9:02 PM
To: John Carew <johnwcarew at gmail.com>
Cc: ovs-discuss at openvswitch.org
Subject: Re: [ovs-discuss] Router not allowing traffic in reverse

OK.

The OVS FAQ has some advice:

Q: I have a sophisticated network setup involving Open vSwitch, VMs or multiple hosts, and other components.  The behavior isn't what I expect.  Help!

    A: To debug network behavior problems, trace the path of a packet,
    hop-by-hop, from its origin in one host to a remote host.  If that's
    correct, then trace the path of the response packet back to the origin.

    The open source tool called ``plotnetcfg`` can help to understand the
    relationship between the networking devices on a single host.

    Usually a simple ICMP echo request and reply (``ping``) packet is good
    enough.  Start by initiating an ongoing ``ping`` from the origin host to a
    remote host.  If you are tracking down a connectivity problem, the "ping"
    will not display any successful output, but packets are still being sent.
    (In this case the packets being sent are likely ARP rather than ICMP.)

    Tools available for tracing include the following:

    - ``tcpdump`` and ``wireshark`` for observing hops across network devices,
      such as Open vSwitch internal devices and physical wires.

    - ``ovs-appctl dpif/dump-flows <br>`` in Open vSwitch 1.10 and later or
      ``ovs-dpctl dump-flows <br>`` in earlier versions.  These tools allow one
      to observe the actions being taken on packets in ongoing flows.

      See ovs-vswitchd(8) for ``ovs-appctl dpif/dump-flows`` documentation,
      ovs-dpctl(8) for ``ovs-dpctl dump-flows`` documentation, and "Why are
      there so many different ways to dump flows?" above for some background.

    - ``ovs-appctl ofproto/trace`` to observe the logic behind how ovs-vswitchd
      treats packets.  See ovs-vswitchd(8) for documentation.  You can out more
      details about a given flow that ``ovs-dpctl dump-flows`` displays, by
      cutting and pasting a flow from the output into an ``ovs-appctl
      ofproto/trace`` command.

    - SPAN, RSPAN, and ERSPAN features of physical switches, to observe what
      goes on at these physical hops.

    Starting at the origin of a given packet, observe the packet at each hop in
    turn.  For example, in one plausible scenario, you might:

    1. ``tcpdump`` the ``eth`` interface through which an ARP egresses a VM,
       from inside the VM.

    2. ``tcpdump`` the ``vif`` or ``tap`` interface through which the ARP
       ingresses the host machine.

    3. Use ``ovs-dpctl dump-flows`` to spot the ARP flow and observe the host
       interface through which the ARP egresses the physical machine.  You may
       need to use ``ovs-dpctl show`` to interpret the port numbers.  If the
       output seems surprising, you can use ``ovs-appctl ofproto/trace`` to
       observe details of how ovs-vswitchd determined the actions in the
       ``ovs-dpctl dump-flows`` output.

    4. ``tcpdump`` the ``eth`` interface through which the ARP egresses the
       physical machine.

    5. ``tcpdump`` the ``eth`` interface through which the ARP ingresses the
       physical machine, at the remote host that receives the ARP.

    6. Use ``ovs-dpctl dump-flows`` to spot the ARP flow on the remote host
       remote host that receives the ARP and observe the VM ``vif`` or ``tap``
       interface to which the flow is directed.  Again, ``ovs-dpctl show`` and
       ``ovs-appctl ofproto/trace`` might help.

    7. ``tcpdump`` the ``vif`` or ``tap`` interface to which the ARP is
       directed.

    8. ``tcpdump`` the ``eth`` interface through which the ARP ingresses a VM,
       from inside the VM.

    It is likely that during one of these steps you will figure out the
    problem.  If not, then follow the ARP reply back to the origin, in reverse.


On Thu, Feb 28, 2019 at 06:34:17PM -0600, John Carew wrote:
> I think there is some misunderstanding, I don't think OVS is working incorrectly or has a problem. I believe it is working exactly how it is configured. I believe OpenStack is just configuring OVS, and it has put in a flow rule or something that is allowing outbound for a network/subnet but not inbound. I'm asking how to figure out where in OVS it is stopping at.
> 
> > On Feb 28, 2019, at 5:51 PM, Ben Pfaff <blp at ovn.org> wrote:
> > 
> > Do you have a reason to believe that it's a problem in OVS, rather 
> > than a problem in what OpenStack is telling OVS to do?
> > 
> > On Thu, Feb 28, 2019 at 04:48:44PM -0600, John Carew wrote:
> >> I agree, but I want to diagnose what is causing it in ovs first before I go to them.
> >> 
> >> John
> >> 
> >>>> On Feb 28, 2019, at 4:33 PM, Ben Pfaff <blp at ovn.org> wrote:
> >>>> 
> >>>> On Wed, Feb 27, 2019 at 09:26:16PM -0600, John Carew wrote:
> >>>> I have setup OpenStack with OVS. I have a single Hyper-V server 
> >>>> running the controller and three CentOS instances(10.0.0.x) on a 
> >>>> private subnet. I created a router in OpenStack with SNAT 
> >>>> disabled, as I only want it to route traffic between the private 
> >>>> subnet(10.0.0.x) and the external subnet(172.16.1.x)/internet. 
> >>>> All of the instances can ping each other along with the external 
> >>>> network(172.16.1.x). From the external network, I can ping the 
> >>>> interface of the ovs router on the external network. I can not 
> >>>> though ping inside the private network. A trace route stops at 
> >>>> the IP of the OVS router. With wireshark, I do not see anything 
> >>>> coming from the external pc’s IP. If I trace route it, I see 
> >>>> packets making all the way to the OVS router and then stop. Since 
> >>>> I can ping one way, and not the other; I believe there is 
> >>>> something in the router/OVS that is stopping the packets to route 
> >>>> into the private subnet. What do I need to look at? (I have 
> >>>> disabled all firewalls on all OSes involved.)
> >>> 
> >>> An OpenStack mailing list might be a better place to track this down.
> >>> OpenStack programs and configures OVS; OVS by itself doesn't have 
> >>> much to do with what's going on.



More information about the discuss mailing list