<div dir="ltr"><div dir="ltr"><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 10, 2020 at 3:36 AM Dumitru Ceara <<a href="mailto:dceara@redhat.com">dceara@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 6/9/20 3:47 PM, Tim Rozet wrote:<br>
> Hi Dumitru,<br>
<br>
Hi Tim,<br>
<br>
> Thanks for the detailed explanation. It makes sense and would like to<br>
> comment on a few things you touched on:<br>
> 1. I do think we need to somehow functionally trigger conntrack when we<br>
> do ofproto-trace. It's the only way to know what the real session state<br>
> ends up being, and we need to be able to follow that for some of the<br>
> complex bugs where packets are getting dropped after they enter a CT<br>
> based flow.<br>
> 2. For your ovn-global-trace, it would be great if that could return a<br>
> json or other parsable format, so that we could build on top of it with<br>
> a tool + GUI to graphically show where the problem is in the network.<br>
<br>
Ack.<br>
<br>
> 3. We really need better user guides on this stuff. Your email is the<br>
> best tutorial I've seen yet :) I didn't even know about the<br>
> ovs-tcpundump command, or ovn-detrace (until you told me previously). It<br>
> would be great to add an ovn troubleshooting guide or something to the docs.<br>
> <br>
<br>
I was planning on sending a patch to update the OVN docs but didn't get<br>
the chance to do it yet.<br>
<br>
> As an administrator I would like to have GUI showing all of the logical<br>
> switch ports (skydive as an example, already does this) and then click<br>
> on a specific port that someone has reported an issue on. At that point<br>
> I can click on the port and ask it to tcpdump me the traffic coming out<br>
> of it. From there, I can select which packet I care about and attempt to<br>
> do an ovn-global-trace on it, which will then show me where the packet<br>
> is getting dropped and why. I think this would be the ideal behavior.<br>
> <br>
<br>
That would be cool. Using your example (skydive) though, I guess one<br>
could also come up with a solution that directly uses the tools already<br>
existing in OVS/OVN essentially performing the steps that something like<br>
ovn-global-trace would do.<br></blockquote><div><br></div><div>They could, but I think it would be better off living in OVN and then consumed by something above it.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thanks,<br>
Dumitru<br>
<br>
> Tim Rozet<br>
> Red Hat CTO Networking Team<br>
> <br>
> <br>
> On Mon, Jun 8, 2020 at 7:53 AM Dumitru Ceara <<a href="mailto:dceara@redhat.com" target="_blank">dceara@redhat.com</a><br>
> <mailto:<a href="mailto:dceara@redhat.com" target="_blank">dceara@redhat.com</a>>> wrote:<br>
> <br>
> Hi everyone,<br>
> <br>
> CC-ing ovn-kubernetes mailing list as I know there's interest about this<br>
> there too.<br>
> <br>
> OVN currently has a couple of tools that help<br>
> tracing/tracking/simulating what would happen to packets within OVN,<br>
> some examples:<br>
> <br>
> 1. ovn-trace<br>
> 2. ovs-appctl ofproto/trace ... | ovn-detrace<br>
> <br>
> They're both really useful and provide lots of information but with both<br>
> of them quite it's hard to get an overview of the end-to-end packet<br>
> processing in OVN for a given packet. Therefore both solutions have<br>
> disadvantages when trying to troubleshoot production deployments. Some<br>
> examples:<br>
> <br>
> a. ovn-trace will not take into account any potential issues with<br>
> translating logical flows to openflow so if there's a bug in the<br>
> translation we'll not be able to detect it by looking at ovn-trace<br>
> output. There is the --ovs switch but the user would have to somehow<br>
> determine on which hypervisor to query for the openflows corresponding<br>
> to logical flows/SB entities.<br>
> <br>
> b. "ovs-appctl ofproto/trace ... | ovn-detrace" works quite well when<br>
> used on a single node but as soon as traffic gets tunneled to a<br>
> different hypervisor the user has to figure out the changes that were<br>
> performed on the packet on the source hypervisor and adapt the<br>
> packet/flow to include the tunnel information to be used when running<br>
> ofproto/trace on the destination hypervisor.<br>
> <br>
> c. both ovn-trace and ofproto/trace support minimal hints to specify the<br>
> new conntrack state after conntrack recirculation but that turns out to<br>
> be not enough even in simple scenarios when NAT is involved [0].<br>
> <br>
> In a production deployment one of the scenarios one would have to<br>
> troubleshoot is:<br>
> <br>
> "Given this OVN deployment on X nodes why isn't this specific<br>
> packet/traffic that is received on logical port P1 doesn't reach/reach<br>
> port P2."<br>
> <br>
> Assuming that point "c" above is addressed somehow (there are a few<br>
> suggestions on how to do that [1]) it's still quite a lot of work for<br>
> the engineer doing the troubleshooting to gather all the interesting<br>
> information. One would probably do something like:<br>
> <br>
> 1. connect to the node running the southbound database and get the<br>
> chassis where the logical port is bound:<br>
> <br>
> chassis=$(ovn-sbctl --bare --columns chassis list port_binding P1)<br>
> hostname=$(ovn-sbctl --bare --columns hostname list chassis $chassis)<br>
> <br>
> 2. connect to $hostname and determine the OVS ofport id of the interface<br>
> corresponding to P1:<br>
> <br>
> in_port=$(ovs-vsctl --bare --columns ofport find interface<br>
> external_ids:iface-id=P1)<br>
> iface=$(ovs-vsctl --bare --columns name find interface<br>
> external_ids:iface-id=P1)<br>
> <br>
> 3. get a hexdump of the packet to be traced (or the flow), for example,<br>
> on $hostname:<br>
> flow=$(tcpdump -xx -c 1 -i $iface $pkt_filter | ovs-tcpundump)<br>
> <br>
> 3. run ofproto/trace on $hostname (potentially piping output to<br>
> ovn-detrace):<br>
> <br>
> ovs-appctl ofproto/trace br-int in_port=$in_port $flow | ovn-detrace<br>
> --ovnnb=$NB_CONN --ovnsb=$SB_CONN<br>
> <br>
> 4. In the best case the packet is fully processed on the current node<br>
> (e.g., is dropped or forwarded out a local VIF).<br>
> <br>
> 5. In the worst case the packet needs to be tunneled to a remote<br>
> hypervisor for egress on a remote VIF. The engineer needs to identify in<br>
> the ofproto/trace output the metadata that would be passed through the<br>
> tunnel along with the packet and also the changes that would happen to<br>
> the packet payload (e.g. NAT) on the local hypervisor.<br>
> <br>
> 6. Determine the hostname of the chassis hosting the remote tunnel<br>
> destination based on "tun_dst" from the ofproto/trace output at point 3<br>
> above:<br>
> <br>
> chassis_name=$(ovn-sbctl --bare --columns chassis_name find encap<br>
> ip=$tun_dst)<br>
> hostname=$(ovn-sbctl --bare --columns hostname find chassis<br>
> name=$chassis_name)<br>
> <br>
> 7. Rerun the ofproto/trace on the remote chassis (basically go back to<br>
> step #3 above).<br>
> <br>
> My initial thought was that all the work above can be automated as all<br>
> the information we need is either in the Southbound DB or in OVS DB on<br>
> the hypervisors and the output of ofproto/trace contains all the packet<br>
> modifications and tunnel information we need. I had started working on a<br>
> tool, "ovn-global-trace", that would do all the work above but I hit a<br>
> few blocking issues:<br>
> <br>
> - point "c" above, i.e., conntrack related packet modifications: this<br>
> will require some work in OVS ofproto/trace to either support additional<br>
> conntrack hints or to actually run the trace against conntrack on<br>
> the node.<br>
> <br>
> - if we choose to query conntrack during ofproto/trace we'd probably<br>
> need a way to also update the conntrack records the trace is run<br>
> against. This would turn out useful for cases when we troubleshoot<br>
> session establishment, e.g., with TCP: first run a trace for the SYN<br>
> packet, then run a a trace for the SYN-ACK packet in the other direction<br>
> but for this second trace we'd need the conntrack entry to have been<br>
> created by the initial trace.<br>
> <br>
> - ofproto/trace output is plain text: while a tool could parse the<br>
> information from the text output it would probably be easier if<br>
> ofproto/trace would dump the trace information in a structured way<br>
> (e.g., json).<br>
> <br>
> It would be great to get some feedback from the community about other<br>
> aspects that I might have missed regarding end-to-end packet tracing and<br>
> how we could aggregate current utilities into a single easier to use<br>
> tool like I was hoping "ovn-global-trace" would end up.<br>
> <br>
> Thanks,<br>
> Dumitru<br>
> <br>
> [0]<br>
> <a href="https://patchwork.ozlabs.org/project/openvswitch/patch/1578648883-1145-1-git-send-email-dceara@redhat.com/" rel="noreferrer" target="_blank">https://patchwork.ozlabs.org/project/openvswitch/patch/1578648883-1145-1-git-send-email-dceara@redhat.com/</a><br>
> [1]<br>
> <a href="https://mail.openvswitch.org/pipermail/ovs-dev/2020-January/366571.html" rel="noreferrer" target="_blank">https://mail.openvswitch.org/pipermail/ovs-dev/2020-January/366571.html</a><br>
> <br>
> -- <br>
> You received this message because you are subscribed to the Google<br>
> Groups "ovn-kubernetes" group.<br>
> To unsubscribe from this group and stop receiving emails from it,<br>
> send an email to <a href="mailto:ovn-kubernetes%2Bunsubscribe@googlegroups.com" target="_blank">ovn-kubernetes+unsubscribe@googlegroups.com</a><br>
> <mailto:<a href="mailto:ovn-kubernetes%252Bunsubscribe@googlegroups.com" target="_blank">ovn-kubernetes%2Bunsubscribe@googlegroups.com</a>>.<br>
> To view this discussion on the web visit<br>
> <a href="https://groups.google.com/d/msgid/ovn-kubernetes/543bf38d-0578-7f6f-2eef-206d84026a3e%40redhat.com" rel="noreferrer" target="_blank">https://groups.google.com/d/msgid/ovn-kubernetes/543bf38d-0578-7f6f-2eef-206d84026a3e%40redhat.com</a>.<br>
> <br>
<br>
</blockquote></div></div>