[ovs-discuss] RFC - OVN end to end packet tracing - ovn-global-trace

Tim Rozet trozet at redhat.com
Tue Jun 9 13:47:37 UTC 2020

Hi Dumitru,
Thanks for the detailed explanation. It makes sense and would like to
comment on a few things you touched on:
1. I do think we need to somehow functionally trigger conntrack when we do
ofproto-trace. It's the only way to know what the real session state ends
up being, and we need to be able to follow that for some of the complex
bugs where packets are getting dropped after they enter a CT based flow.
2. For your ovn-global-trace, it would be great if that could return a json
or other parsable format, so that we could build on top of it with a tool +
GUI to graphically show where the problem is in the network.
3. We really need better user guides on this stuff. Your email is the best
tutorial I've seen yet :) I didn't even know about the
ovs-tcpundump command, or ovn-detrace (until you told me previously). It
would be great to add an ovn troubleshooting guide or something to the docs.

As an administrator I would like to have GUI showing all of the logical
switch ports (skydive as an example, already does this) and then click on a
specific port that someone has reported an issue on. At that point I can
click on the port and ask it to tcpdump me the traffic coming out of it.
>From there, I can select which packet I care about and attempt to do an
ovn-global-trace on it, which will then show me where the packet is getting
dropped and why. I think this would be the ideal behavior.

Tim Rozet
Red Hat CTO Networking Team

On Mon, Jun 8, 2020 at 7:53 AM Dumitru Ceara <dceara at redhat.com> wrote:

> Hi everyone,
> CC-ing ovn-kubernetes mailing list as I know there's interest about this
> there too.
> OVN currently has a couple of tools that help
> tracing/tracking/simulating what would happen to packets within OVN,
> some examples:
> 1. ovn-trace
> 2. ovs-appctl ofproto/trace ... | ovn-detrace
> They're both really useful and provide lots of information but with both
> of them quite it's hard to get an overview of the end-to-end packet
> processing in OVN for a given packet. Therefore both solutions have
> disadvantages when trying to troubleshoot production deployments. Some
> examples:
> a. ovn-trace will not take into account any potential issues with
> translating logical flows to openflow so if there's a bug in the
> translation we'll not be able to detect it by looking at ovn-trace
> output. There is the --ovs switch but the user would have to somehow
> determine on which hypervisor to query for the openflows corresponding
> to logical flows/SB entities.
> b. "ovs-appctl ofproto/trace ... | ovn-detrace" works quite well when
> used on a single node but as soon as traffic gets tunneled to a
> different hypervisor the user has to figure out the changes that were
> performed on the packet on the source hypervisor and adapt the
> packet/flow to include the tunnel information to be used when running
> ofproto/trace on the destination hypervisor.
> c. both ovn-trace and ofproto/trace support minimal hints to specify the
> new conntrack state after conntrack recirculation but that turns out to
> be not enough even in simple scenarios when NAT is involved [0].
> In a production deployment one of the scenarios one would have to
> troubleshoot is:
> "Given this OVN deployment on X nodes why isn't this specific
> packet/traffic that is received on logical port P1 doesn't reach/reach
> port P2."
> Assuming that point "c" above is addressed somehow (there are a few
> suggestions on how to do that [1]) it's still quite a lot of work for
> the engineer doing the troubleshooting to gather all the interesting
> information. One would probably do something like:
> 1. connect to the node running the southbound database and get the
> chassis where the logical port is bound:
> chassis=$(ovn-sbctl --bare --columns chassis list port_binding P1)
> hostname=$(ovn-sbctl --bare --columns hostname list chassis $chassis)
> 2. connect to $hostname and determine the OVS ofport id of the interface
> corresponding to P1:
> in_port=$(ovs-vsctl --bare --columns ofport find interface
> external_ids:iface-id=P1)
> iface=$(ovs-vsctl --bare --columns name find interface
> external_ids:iface-id=P1)
> 3. get a hexdump of the packet to be traced (or the flow), for example,
> on $hostname:
> flow=$(tcpdump -xx -c 1 -i $iface $pkt_filter | ovs-tcpundump)
> 3. run ofproto/trace on $hostname (potentially piping output to
> ovn-detrace):
> ovs-appctl ofproto/trace br-int in_port=$in_port $flow | ovn-detrace
> --ovnnb=$NB_CONN --ovnsb=$SB_CONN
> 4. In the best case the packet is fully processed on the current node
> (e.g., is dropped or forwarded out a local VIF).
> 5. In the worst case the packet needs to be tunneled to a remote
> hypervisor for egress on a remote VIF. The engineer needs to identify in
> the ofproto/trace output the metadata that would be passed through the
> tunnel along with the packet and also the changes that would happen to
> the packet payload (e.g. NAT) on the local hypervisor.
> 6. Determine the hostname of the chassis hosting the remote tunnel
> destination based on "tun_dst" from the ofproto/trace output at point 3
> above:
> chassis_name=$(ovn-sbctl --bare --columns chassis_name find encap
> ip=$tun_dst)
> hostname=$(ovn-sbctl --bare --columns hostname find chassis
> name=$chassis_name)
> 7. Rerun the ofproto/trace on the remote chassis (basically go back to
> step #3 above).
> My initial thought was that all the work above can be automated as all
> the information we need is either in the Southbound DB or in OVS DB on
> the hypervisors and the output of ofproto/trace contains all the packet
> modifications and tunnel information we need. I had started working on a
> tool, "ovn-global-trace", that would do all the work above but I hit a
> few blocking issues:
> - point "c" above, i.e., conntrack related packet modifications: this
> will require some work in OVS ofproto/trace to either support additional
> conntrack hints or to actually run the trace against conntrack on the node.
> - if we choose to query conntrack during ofproto/trace we'd probably
> need a way to also update the conntrack records the trace is run
> against. This would turn out useful for cases when we troubleshoot
> session establishment, e.g., with TCP: first run a trace for the SYN
> packet, then run a a trace for the SYN-ACK packet in the other direction
> but for this second trace we'd need the conntrack entry to have been
> created by the initial trace.
> - ofproto/trace output is plain text: while a tool could parse the
> information from the text output it would probably be easier if
> ofproto/trace would dump the trace information in a structured way
> (e.g., json).
> It would be great to get some feedback from the community about other
> aspects that I might have missed regarding end-to-end packet tracing and
> how we could aggregate current utilities into a single easier to use
> tool like I was hoping "ovn-global-trace" would end up.
> Thanks,
> Dumitru
> [0]
> https://patchwork.ozlabs.org/project/openvswitch/patch/1578648883-1145-1-git-send-email-dceara@redhat.com/
> [1]
> https://mail.openvswitch.org/pipermail/ovs-dev/2020-January/366571.html
> --
> You received this message because you are subscribed to the Google Groups
> "ovn-kubernetes" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ovn-kubernetes+unsubscribe at googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ovn-kubernetes/543bf38d-0578-7f6f-2eef-206d84026a3e%40redhat.com
> .
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200609/962e56a3/attachment.html>

More information about the discuss mailing list