[ovs-discuss] RFC - OVN end to end packet tracing - ovn-global-trace

Dumitru Ceara dceara at redhat.com
Mon Jun 8 11:52:58 UTC 2020

Hi everyone,

CC-ing ovn-kubernetes mailing list as I know there's interest about this
there too.

OVN currently has a couple of tools that help
tracing/tracking/simulating what would happen to packets within OVN,
some examples:

1. ovn-trace
2. ovs-appctl ofproto/trace ... | ovn-detrace

They're both really useful and provide lots of information but with both
of them quite it's hard to get an overview of the end-to-end packet
processing in OVN for a given packet. Therefore both solutions have
disadvantages when trying to troubleshoot production deployments. Some

a. ovn-trace will not take into account any potential issues with
translating logical flows to openflow so if there's a bug in the
translation we'll not be able to detect it by looking at ovn-trace
output. There is the --ovs switch but the user would have to somehow
determine on which hypervisor to query for the openflows corresponding
to logical flows/SB entities.

b. "ovs-appctl ofproto/trace ... | ovn-detrace" works quite well when
used on a single node but as soon as traffic gets tunneled to a
different hypervisor the user has to figure out the changes that were
performed on the packet on the source hypervisor and adapt the
packet/flow to include the tunnel information to be used when running
ofproto/trace on the destination hypervisor.

c. both ovn-trace and ofproto/trace support minimal hints to specify the
new conntrack state after conntrack recirculation but that turns out to
be not enough even in simple scenarios when NAT is involved [0].

In a production deployment one of the scenarios one would have to
troubleshoot is:

"Given this OVN deployment on X nodes why isn't this specific
packet/traffic that is received on logical port P1 doesn't reach/reach
port P2."

Assuming that point "c" above is addressed somehow (there are a few
suggestions on how to do that [1]) it's still quite a lot of work for
the engineer doing the troubleshooting to gather all the interesting
information. One would probably do something like:

1. connect to the node running the southbound database and get the
chassis where the logical port is bound:

chassis=$(ovn-sbctl --bare --columns chassis list port_binding P1)
hostname=$(ovn-sbctl --bare --columns hostname list chassis $chassis)

2. connect to $hostname and determine the OVS ofport id of the interface
corresponding to P1:

in_port=$(ovs-vsctl --bare --columns ofport find interface
iface=$(ovs-vsctl --bare --columns name find interface

3. get a hexdump of the packet to be traced (or the flow), for example,
on $hostname:
flow=$(tcpdump -xx -c 1 -i $iface $pkt_filter | ovs-tcpundump)

3. run ofproto/trace on $hostname (potentially piping output to

ovs-appctl ofproto/trace br-int in_port=$in_port $flow | ovn-detrace
--ovnnb=$NB_CONN --ovnsb=$SB_CONN

4. In the best case the packet is fully processed on the current node
(e.g., is dropped or forwarded out a local VIF).

5. In the worst case the packet needs to be tunneled to a remote
hypervisor for egress on a remote VIF. The engineer needs to identify in
the ofproto/trace output the metadata that would be passed through the
tunnel along with the packet and also the changes that would happen to
the packet payload (e.g. NAT) on the local hypervisor.

6. Determine the hostname of the chassis hosting the remote tunnel
destination based on "tun_dst" from the ofproto/trace output at point 3

chassis_name=$(ovn-sbctl --bare --columns chassis_name find encap
hostname=$(ovn-sbctl --bare --columns hostname find chassis

7. Rerun the ofproto/trace on the remote chassis (basically go back to
step #3 above).

My initial thought was that all the work above can be automated as all
the information we need is either in the Southbound DB or in OVS DB on
the hypervisors and the output of ofproto/trace contains all the packet
modifications and tunnel information we need. I had started working on a
tool, "ovn-global-trace", that would do all the work above but I hit a
few blocking issues:

- point "c" above, i.e., conntrack related packet modifications: this
will require some work in OVS ofproto/trace to either support additional
conntrack hints or to actually run the trace against conntrack on the node.

- if we choose to query conntrack during ofproto/trace we'd probably
need a way to also update the conntrack records the trace is run
against. This would turn out useful for cases when we troubleshoot
session establishment, e.g., with TCP: first run a trace for the SYN
packet, then run a a trace for the SYN-ACK packet in the other direction
but for this second trace we'd need the conntrack entry to have been
created by the initial trace.

- ofproto/trace output is plain text: while a tool could parse the
information from the text output it would probably be easier if
ofproto/trace would dump the trace information in a structured way
(e.g., json).

It would be great to get some feedback from the community about other
aspects that I might have missed regarding end-to-end packet tracing and
how we could aggregate current utilities into a single easier to use
tool like I was hoping "ovn-global-trace" would end up.


[1] https://mail.openvswitch.org/pipermail/ovs-dev/2020-January/366571.html

More information about the discuss mailing list