[ovs-discuss] OVS-bridges Troubleshooting

Flavio Leitner fbl at sysclose.org
Thu Nov 14 17:28:02 UTC 2019


On Thu, 14 Nov 2019 14:38:03 +0200
aeris <aeris at ytechteam.com> wrote:

> about flows - maybe this output would be better -
> 
>   br-int(ovs-bridge3)
>   cookie=0x9f5f4658300163fb, duration=255211.898s, table=71, 
> n_packets=221, n_bytes=9282, 
> priority=95,arp,reg5=0x46,in_port="tapafe5cc97-46",dl_src=b2:ac:73:a2:4e:20,arp_spa=*.*.*.16 
> actions=resubmit(,94)
>   cookie=0x9f5f4658300163fb, duration=255211.898s, table=71, 
> n_packets=68274, n_bytes=6379501, 
> priority=65,ip,reg5=0x46,in_port="tapafe5cc97-46",dl_src=b2:ac:73:a2:4e:20,nw_src=*.*.*.16 
> actions=ct(table=72,zone=NXM_NX_REG6[0..15])
>   br-ex(ovs-bridge2)
>   cookie=0x539c250217577d59, duration=3197928.791s, table=0, 
> n_packets=717237943, n_bytes=120156920973, 
> priority=4,in_port="phy-br-ex",dl_vlan=3 actions=strip_vlan,NORMAL
>   cookie=0x539c250217577d59, duration=3197935.391s, table=0, 
> n_packets=1919206, n_bytes=278077793, priority=2,in_port="phy-br-ex" 
> actions=drop
>   cookie=0x539c250217577d59, duration=3197935.397s, table=0, 
> n_packets=850526618, n_bytes=408960488324, priority=0 actions=NORMAL
>   br-phy(ovs-bridge1)
>   cookie=0x0, duration=3599826.112s, table=0, n_packets=1574126232, 
> n_bytes=530825513146, priority=0 actions=NORMAL

It looks like you have many tables with many flows, which makes really
hard to follow what is happening over email.

Maybe you can spot the bad packet at a point and save it on a pcap.
Then use ovs-pcap <pcap> to convert that packet to a hex string.
Then use that string with ovs-appctl ofproto/trace to simulate what
would happen next?

Go the sources and grep for 'ofproto/trace' in the tests/ subdir and
you will find some practical examples.

HTH,
fbl


> 
> On 11/14/2019 11:58 AM, aeris wrote:
> > We aren't using DPDK now, ports seems to be patch ports as of some 
> > output of ovs-*ctl commands
> >
> > About flow table what I can show ? To be more exact - which out of
> > 300 flows you're interested in (as output of ovs-vsctl dump-flows)?
> >
> > If it is about something about "problematic" VM then -
> > recirc_id(0),in_port(18),ct_state(-trk),eth(src=b2:ac:73:a2:4e:20),eth_type(0x0800),ipv4(src=*.*.*.16,proto=6,frag=no), 
> > packets:46, bytes:3404, used:0.573s, flags:PR., 
> > actions:ct(zone=3),recirc(0xaf01f)
> >
> > recirc_id(0xaf01f),in_port(18),ct_state(-new+est-rel+rpl-inv+trk),ct_zone(0x3),ct_mark(0),eth(src=b2:ac:73:a2:4e:20,dst=64:c3:d6:bb:05:4a),eth_type(0x0800),ipv4(frag=no), 
> > packets:46, bytes:3404, used:0.570s, flags:PR., 
> > actions:push_vlan(vid=500,pcp=0),11
> >
> >
> > Also I've tried to use "ovs-appctl ofproto/trace <bridge-name>" and 
> > none of my bridges accepted as an argument(Previously listed them
> > via "ovs-appctl ofproto/list")
> >
> > stderr told me : "ovs-appctl: ovs-vswitchd: server returned an
> > error"
> >
> >
> > On 11/13/2019 3:29 PM, Flavio Leitner wrote:  
> >> On Wed, 13 Nov 2019 14:02:33 +0200
> >> aeris <aeris at ytechteam.com> wrote:
> >>  
> >>> Hello,
> >>>
> >>> I'm new here, so sorry if i missed something or writing to wrong
> >>> maillist
> >>>
> >>> A bit of foreword -
> >>>
> >>> We had an installation of ovs in our cloud environment and faced
> >>> networking issue with vms at random time, but mostly 3-7 minutes
> >>> after reboot(it just stops answering on every request(typical
> >>> ICMP) during time from a few seconds to a few minutes)
> >>>
> >>> Our internal network for such kind of requests looks like this -
> >>>
> >>> Physical-switch-->linux
> >>> bond-->ovs-bridge1-->ovs-bridge2-->ovs-bridge3-->VM
> >>>
> >>> With ovs-tcpdump i found that there is two-sided communication on
> >>> link to VM and between ovs-bridge1 and linux-bond, but only
> >>> one-sided communication(only icmp-replies/requests depending on
> >>> from which side ping was issued) on link between ovs-bridge2 and
> >>> ovs-bridge3 and no packet on any other interface.
> >>>
> >>> So, actually, my main question is - what instruments do u use when
> >>> troubleshooting such issues, which I in turn can use in my
> >>> implementation. And did you faced similar problems in your
> >>> environments?  
> >> I see you're using ovs-tcpdump, but you haven't told us if you're
> >> using DPDK or not. Also, how are the bridges connected to each
> >> other? veth? patch ports? and last, what do you have in the flow
> >> table?
> >>
> >> One thing that you might find useful is ofproto/trace:
> >> https://developers.redhat.com/blog/2016/10/12/tracing-packets-inside-open-vswitch/ 
> >>
> >>
> >> fbl  
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss  



More information about the discuss mailing list