[ovs-discuss] millions of packets going to a flow?

Gabe Black Gabe.Black at viavisolutions.com
Fri Aug 28 15:40:02 UTC 2015


Maybe a little premature again... Maybe some of the commands might help, but from what I understand, the appctl datapath commands won't work for datapath_type=netdev, which is what the vhost-user/ovs-dpdk sets the bridges to.  Correct me if I'm wrong, but I guess that means none of the appctl commands in the FAQ will work for me.  Bummer.

Gabe

> -----Original Message-----
> From: Gabe Black
> Sent: Friday, August 28, 2015 9:32 AM
> To: 'Ben Pfaff'
> Cc: Mooney, Sean K; bugs at openvswitch.org
> Subject: RE: [ovs-discuss] millions of packets going to a flow?
> 
> I saw that FAQ prior to posting.  I probably should have mentioned that.
> Unfortunately the ovs-appctl does not work and complains of not finding
> /usr/var/run/openvswitch/ovs-vswitchd.pid.  I tried both coping the pidfile
> and symlinking it to where ovs-appctl is looking, but it still failed with the
> same error message.
> 
> Running strace when the pid file is there shows that it does finde the file, but
> it performs a fcntl( ..., F_GETLK, {type=F_UNLCK,...} ...) which to me makes
> me believe it might be seeing if the file/process is locked and erroring out
> because it finds it can't unlock it.
> 
> The error message ovs-appctl displays is the same for both cases though:
> _cannot read pidfile "/usr/var/run/openvswitch/ovs-vswitchd.pid"_.
> 
> I just now out of curiousity tried adding the --pidfile to the ovs-dpdk init
> script (where it invokes vswitchd daemon, and found that now ovs-appctl
> works!
> 
> I'll follow that faq again now armed with a working ovs-appctl. I wonder if a
> bug should be opened against this to add the --pidfile to the ovs-dpdk-init
> script.
> 
> Thanks for taking your time to help me out!   I'll post back with anything else I
> find that might be an defect.
> Gabe
> 
> 
> 
> > -----Original Message-----
> > From: Ben Pfaff [mailto:blp at nicira.com]
> > Sent: Friday, August 28, 2015 9:08 AM
> > To: Gabe Black
> > Cc: Mooney, Sean K; bugs at openvswitch.org
> > Subject: Re: [ovs-discuss] millions of packets going to a flow?
> >
> > On Fri, Aug 28, 2015 at 03:01:52PM +0000, Gabe Black wrote:
> > > However, all the commands I show in this inquiry are ovs-xyz commands.
> > > My questions are around how to debug/troubleshoot where the packets
> > > are going.  I am new to all of this, but to me this did seem like
> > > the most relevant list to post that question.
> >
> > Maybe you want this FAQ.
> >
> > ### Q: I have a sophisticated network setup involving Open vSwitch, VMs
> or
> >    multiple hosts, and other components.  The behavior isn't what I
> >    expect.  Help!
> >
> > A: To debug network behavior problems, trace the path of a packet,
> >    hop-by-hop, from its origin in one host to a remote host.  If
> >    that's correct, then trace the path of the response packet back to
> >    the origin.
> >
> >    Usually a simple ICMP echo request and reply ("ping") packet is
> >    good enough.  Start by initiating an ongoing "ping" from the origin
> >    host to a remote host.  If you are tracking down a connectivity
> >    problem, the "ping" will not display any successful output, but
> >    packets are still being sent.  (In this case the packets being sent
> >    are likely ARP rather than ICMP.)
> >
> >    Tools available for tracing include the following:
> >
> >    - "tcpdump" and "wireshark" for observing hops across network
> >      devices, such as Open vSwitch internal devices and physical
> >      wires.
> >
> >    - "ovs-appctl dpif/dump-flows <br>" in Open vSwitch 1.10 and
> >      later or "ovs-dpctl dump-flows <br>" in earlier versions.
> >      These tools allow one to observe the actions being taken on
> >      packets in ongoing flows.
> >
> >      See ovs-vswitchd(8) for "ovs-appctl dpif/dump-flows"
> >      documentation, ovs-dpctl(8) for "ovs-dpctl dump-flows"
> >      documentation, and "Why are there so many different ways to
> >      dump flows?" above for some background.
> >
> >    - "ovs-appctl ofproto/trace" to observe the logic behind how
> >      ovs-vswitchd treats packets.  See ovs-vswitchd(8) for
> >      documentation.  You can out more details about a given flow
> >      that "ovs-dpctl dump-flows" displays, by cutting and pasting
> >      a flow from the output into an "ovs-appctl ofproto/trace"
> >      command.
> >
> >    - SPAN, RSPAN, and ERSPAN features of physical switches, to
> >      observe what goes on at these physical hops.
> >
> >    Starting at the origin of a given packet, observe the packet at
> >    each hop in turn.  For example, in one plausible scenario, you
> >    might:
> >
> >    1. "tcpdump" the "eth" interface through which an ARP egresses
> >       a VM, from inside the VM.
> >
> >    2. "tcpdump" the "vif" or "tap" interface through which the ARP
> >       ingresses the host machine.
> >
> >    3. Use "ovs-dpctl dump-flows" to spot the ARP flow and observe
> >       the host interface through which the ARP egresses the
> >       physical machine.  You may need to use "ovs-dpctl show" to
> >       interpret the port numbers.  If the output seems surprising,
> >       you can use "ovs-appctl ofproto/trace" to observe details of
> >       how ovs-vswitchd determined the actions in the "ovs-dpctl
> >       dump-flows" output.
> >
> >    4. "tcpdump" the "eth" interface through which the ARP egresses
> >       the physical machine.
> >
> >    5. "tcpdump" the "eth" interface through which the ARP
> >       ingresses the physical machine, at the remote host that
> >       receives the ARP.
> >
> >    6. Use "ovs-dpctl dump-flows" to spot the ARP flow on the
> >       remote host that receives the ARP and observe the VM "vif"
> >       or "tap" interface to which the flow is directed.  Again,
> >       "ovs-dpctl show" and "ovs-appctl ofproto/trace" might help.
> >
> >    7. "tcpdump" the "vif" or "tap" interface to which the ARP is
> >       directed.
> >
> >    8. "tcpdump" the "eth" interface through which the ARP
> >       ingresses a VM, from inside the VM.
> >
> >    It is likely that during one of these steps you will figure out the
> >    problem.  If not, then follow the ARP reply back to the origin, in
> >    reverse.



More information about the discuss mailing list