[ovs-discuss] 回复: ovn-ic POC problem

Han Zhou hzhou at ovn.org
Mon May 4 16:46:39 UTC 2020


On Mon, May 4, 2020 at 4:01 AM <fangtian at ruijie.com.cn> wrote:
>
>
>
>
>
> On Sun, May 3, 2020 at 4:18 AM <fangtian at ruijie.com.cn> wrote:
> >
> >
> >
> >
> >
> > On Wed, Apr 29, 2020 at 9:34 PM <fangtian at ruijie.com.cn> wrote:
> > >
> > >
> > >
> > > Hi all,
> > >
> > > I am a developer of Kube-ovn project. Currently I’m doing an
OVN-interconnect POC, the setup script is attached at the bottom.
> > >
> > > My problem is :as everything seems OK, but I can’t ping the Central
AZ from the East AZ.
> > >
> > > I did diagnose this with the ovn-trace tool ,which prompt me with the
message at the end :
> > >
> > >
> > >
> > > “egress(dp="join", inport="join-ovn-cluster", outport="node-east-1")
> > >
> > > -------------------------------------------------------------------
> > >
> > > 1. ls_out_pre_acl (ovn-northd.c:4690): ip, priority 100, uuid 7a78dffb
> > >
> > >     reg0[0] = 1;
> > >
> > >     next;
> > >
> > > 2. ls_out_pre_stateful (ovn-northd.c:4879): reg0[0] == 1, priority
100, uuid e724b591
> > >
> > >     ct_next;
> > >
> > >
> > >
> > > ct_next(ct_state=est|trk /* default (use --ct to customize) */)
> > >
> > > ---------------------------------------------------------------
> > >
> > > 9. ls_out_port_sec_l2 (ovn-northd.c:4564): outport == "node-east-1",
priority 50, uuid 7c98b562
> > >
> > >     output;
> > >
> > >     /* output to "node-east-1", type "" */
> > >
> > > “
> > >
> > > Seems the ping package was sent out from the outport "node-east-1"
,but no response there.
> > >
> > >
> > >
> > >
> > >
> > > Will someone please to tell me what’s wrong with my POC setup ?
> > >
> > > Thanks!
> > >
> > > Tian
> > >
> >
> > >
> >
> >
> >
> > >>It is hard to tell without the full topology. For example, what is
the path between the source and destination?
> >
> > >>Also, what's the command used to generate the ovn-trace output? What
the above trace shows is that the packet is sent out to "node-east-1", not
from. But you mentioned you were pinging from east to central, so I am
confused.
> >
> > >>In addition, did you check the content of IC-SB, with ovn-ic-sbctl
show? Were the ports and GWs populated in IC-SB?
> >
> >
> >
> > >>Thanks,
> >
> > >>Han
> >
> >
> >
> > Thanks for reply !
> >
> >
> >
> > I did trace the ping data path from the east-1 node,the message is from
the trace command。
> >
> > My configuration is quite simple,2 AZes with 1 node each.
> >
> >
> >
> > On the central node broker-1
> >
> >
> >
> > # ovn-ic-nbctl show
> >
> > Transit_Switch ts-aliyun
> >
> >
> >
> > # ovn-ic-sbctl show
> >
> > availability-zone central
> >
> >     gateway 785b3dec-0af0-4906-8018-50f511bf4e4f
> >
> >         hostname: broker-1
> >
> >         type: geneve
> >
> >             ip: 172.17.88.1
> >
> >         port lsp-ovn-cluster-ts-aliyun-central
> >
> >             transit switch: ts-aliyun
> >
> >             address: ["1a:aa:aa:aa:aa:01 192.168.100.1/24"]
> >
> > availability-zone east
> >
> >     gateway d68ac809-292e-4df0-9a45-0254750f2376
> >
> >         hostname: east-1
> >
> >         type: geneve
> >
> >             ip: 172.17.88.0
> >
> >         port lsp-ovn-cluster-ts-aliyun-east
> >
> >             transit switch: ts-aliyun
> >
> >             address: ["1a:aa:aa:aa:aa:02 192.168.100.2/24"]
> >
> > # ovn-sbctl show
> >
> > Chassis "d68ac809-292e-4df0-9a45-0254750f2376"
> >
> >     hostname: east-1
> >
> >     Encap geneve
> >
> >         ip: "172.17.88.0"
> >
> >         options: {csum="true"}
> >
> >     Port_Binding lsp-ovn-cluster-ts-aliyun-east
> >
> > Chassis "785b3dec-0af0-4906-8018-50f511bf4e4f"
> >
> >     hostname: broker-1
> >
> >     Encap geneve
> >
> >         ip: "172.17.88.1"
> >
> >         options: {csum="true"}
> >
> >     Port_Binding default-http-backend-67cf578fc4-66284.ingress-nginx
> >
> >     Port_Binding tiller-deploy-67cd845dff-j7jqx.kube-system
> >
> >     Port_Binding coredns-autoscaler-65bfc8d47d-gvvgh.kube-system
> >
> >     Port_Binding cr-lrp-ovn-cluster-ts-aliyun-central
> >
> >     Port_Binding kube-ovn-pinger-4hq29.kube-system
> >
> >     Port_Binding node-broker-1
> >
> >     Port_Binding metrics-server-6b55c64f86-l5cq5.kube-system
> >
> > Port_Binding coredns-7c5566588d-kgjj9.kube-system
> >
> >
> >
> > ovn-sbctl show
> >
> > Chassis "d68ac809-292e-4df0-9a45-0254750f2376"
> >
> >     hostname: east-1
> >
> >     Encap geneve
> >
> >         ip: "172.17.88.0"
> >
> >         options: {csum="true"}
> >
> >     Port_Binding lsp-ovn-cluster-ts-aliyun-east
> >
> > Chassis "785b3dec-0af0-4906-8018-50f511bf4e4f"
> >
> >     hostname: broker-1
> >
> >     Encap geneve
> >
> >         ip: "172.17.88.1"
> >
> >         options: {csum="true"}
> >
> >     Port_Binding default-http-backend-67cf578fc4-66284.ingress-nginx
> >
> >     Port_Binding tiller-deploy-67cd845dff-j7jqx.kube-system
> >
> >     Port_Binding coredns-autoscaler-65bfc8d47d-gvvgh.kube-system
> >
> >     Port_Binding cr-lrp-ovn-cluster-ts-aliyun-central
> >
> >     Port_Binding kube-ovn-pinger-4hq29.kube-system
> >
> >     Port_Binding node-broker-1
> >
> >     Port_Binding metrics-server-6b55c64f86-l5cq5.kube-system
> >
> > Port_Binding coredns-7c5566588d-kgjj9.kube-system
> >
> >
> >
> >
> >
> >
> >
> > On the east-1 node , things are similar.
> >
> >
> >
> > sh-4.4# ovn-nbctl show
> >
> > switch 1f157fe4-8576-400e-85fb-7f1fad7daf44 (ovn-default)
> >
> >     port metrics-server-6b55c64f86-l9mhb.kube-system
> >
> >         addresses: ["00:00:00:A0:C4:B0 10.44.0.6"]
> >
> >     port coredns-7c5566588d-bn4fh.kube-system
> >
> >         addresses: ["00:00:00:B1:40:ED 10.44.0.3"]
> >
> >     port coredns-autoscaler-65bfc8d47d-xk9md.kube-system
> >
> >         addresses: ["00:00:00:9C:49:B9 10.44.0.4"]
> >
> >     port kube-ovn-pinger-nfcjk.kube-system
> >
> >         addresses: ["00:00:00:4A:20:7A 10.44.0.5"]
> >
> >     port ovn-default-ovn-cluster
> >
> >         type: router
> >
> >         addresses: ["00:00:00:DE:08:3C"]
> >
> >         router-port: ovn-cluster-ovn-default
> >
> >     port default-http-backend-67cf578fc4-wn5mj.ingress-nginx
> >
> >         addresses: ["00:00:00:C4:22:11 10.44.0.2"]
> >
> > switch 0cbefdd1-8067-4db2-8a2d-3487dc16c3ea (ts-aliyun)
> >
> >     port lsp-ovn-cluster-ts-aliyun-central
> >
> >         type: remote
> >
> >         addresses: ["1a:aa:aa:aa:aa:01 192.168.100.1/24"]
> >
> >     port lsp-ovn-cluster-ts-aliyun-east
> >
> >         type: router
> >
> >         router-port: lrp-ovn-cluster-ts-aliyun-east
> >
> > switch 61a36d09-0b39-46f1-9da1-967ac195905b (join)
> >
> >     port join-ovn-cluster
> >
> >         type: router
> >
> >         addresses: ["00:00:00:17:31:7A"]
> >
> >         router-port: ovn-cluster-join
> >
> >     port node-east-1
> >
> >         addresses: ["00:00:00:5F:D7:E6 100.64.0.2"]
> >
> > router b7ef9172-ccf2-4489-8802-a03f1c441a51 (ovn-cluster)
> >
> >     port ovn-cluster-join
> >
> >         mac: "00:00:00:17:31:7A"
> >
> >         networks: ["100.64.0.1/16"]
> >
> >     port lrp-ovn-cluster-ts-aliyun-east
> >
> >         mac: "1a:aa:aa:aa:aa:02"
> >
> >         networks: ["192.168.100.2/24"]
> >
> >         gateway chassis: [d68ac809-292e-4df0-9a45-0254750f2376]
> >
> >     port ovn-cluster-ovn-default
> >
> >         mac: "00:00:00:DE:08:3C"
> >
> >         networks: ["10.44.0.1/16"]
> >
> > sh-4.4# ovn-sbctl show
> >
> > Chassis "785b3dec-0af0-4906-8018-50f511bf4e4f"
> >
> >     hostname: broker-1
> >
> >     Encap geneve
> >
> >         ip: "172.17.88.1"
> >
>
> >         options: {csum="true"}
>
>
>
> Here it doesn't show the port-binding for port
lsp-ovn-cluster-ts-aliyun-central, which should be learned from IC-SB DB.
However, the port is shown in NB DB, and the chassis binding is also shown
in IC-SB. So it means the chassis information of the port-binding is not
learned from IC-SB to SB of the east AZ. Could you check if there is any
error log in ovn-ic of east AZ? Are the connections to SB and IC-SB both
working well? Could you share the output of "ovn-sbctl list port_binding"
in east AZ as well?
>
>
>
>
>
> ##East ovn-ic log
>
>
>
> 2020-05-04T10:28:58.129Z|00006|reconnect|INFO|tcp:172.17.88.0:6641:
connected
>
> 2020-05-04T10:28:58.129Z|00007|reconnect|INFO|tcp:172.17.88.0:6642:
connected
>
> 2020-05-04T10:28:58.129Z|00008|reconnect|INFO|tcp:172.17.88.1:6645:
connected
>
> 2020-05-04T10:28:58.129Z|00009|reconnect|INFO|tcp:172.17.88.1:6646:
connected
>
> 2020-05-04T10:28:58.129Z|00010|ovn_ic|INFO|ovn-ic lock acquired. This
ovn-ic instance is now active.
>
> 2020-05-04T10:28:58.130Z|00011|ovn_ic|INFO|NB Global not exist.
>
> 2020-05-04T10:28:58.131Z|00012|ovn_ic|INFO|NB Global not exist.
>
> 2020-05-04T10:28:58.133Z|00013|ovn_ic|WARN|Route sync ignores port
lsp-ovn-cluster-ts-aliyun-east on ts ts-aliyun because logical router port
is not found in NB.
>
>
>
> sh-4.4# ovn-sbctl show
>
> Chassis "785b3dec-0af0-4906-8018-50f511bf4e4f"
>
>     hostname: broker-1
>
>     Encap geneve
>
>         ip: "172.17.88.1"
>
>         options: {csum="true"}
>
> Port_Binding lsp-ovn-cluster-ts-aliyun-central
>
>
>
> ## this time it has port binding,but still not work.

It is a little strange that the port-binding wasn't there last time. It
could be a transient state or it could be a bug. Could you confirm if the
current DB status is stable? i.e. does every ovn-sbctl show output shows
the same port-bindings?

If it is stable now, the root cause of not pinging could be something else.
Please do tcpdump to firstly identify on which part of the path was the
packet dropped.
If the ping is from node-east-1 to some node in central AZ, did central
side node receive ping request and sent out reply? Could you do tcpdump on
that node to verify? If not, you can check if the packet arrived iinterconn
GW on each side, by doing tcpdump on the genev_sys_6081interface on the
gateway nodes? If ping is received and reply is sent out by the central AZ
node, you could do similar tcpdump on the GWs to identify which node
dropped the packet. Then the debug can continue there.

>
> Chassis "d68ac809-292e-4df0-9a45-0254750f2376"
>
>     hostname: east-1
>
>     Encap geneve
>
>         ip: "172.17.88.0"
>
>         options: {csum="true"}
>
>     Port_Binding coredns-7c5566588d-bn4fh.kube-system
>
>     Port_Binding coredns-autoscaler-65bfc8d47d-xk9md.kube-system
>
>     Port_Binding node-east-1
>
>     Port_Binding cr-lrp-ovn-cluster-ts-aliyun-east
>
>     Port_Binding metrics-server-6b55c64f86-l9mhb.kube-system
>
>     Port_Binding default-http-backend-67cf578fc4-wn5mj.ingress-nginx
>
>     Port_Binding kube-ovn-pinger-nfcjk.kube-system
>
>
>
>
>
> sh-4.4# ovn-sbctl list port_binding
>
> _uuid               : c9193648-752d-475c-955c-27bdb51bf946
>
> chassis             : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath            : 325496e7-112f-40f3-b612-233152629d3c
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : coredns-7c5566588d-bn4fh.kube-system
>
> mac                 : ["00:00:00:B1:40:ED 10.44.0.3"]
>
> nat_addresses       : []
>
> options             : {}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 3
>
> type                : ""
>
> virtual_parent      : []
>
>
>
> _uuid               : c5f6c879-d954-48a2-99b1-8f377c948dde
>
> chassis             : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath            : 325496e7-112f-40f3-b612-233152629d3c
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : coredns-autoscaler-65bfc8d47d-xk9md.kube-system
>
> mac                 : ["00:00:00:9C:49:B9 10.44.0.4"]
>
> nat_addresses       : []
>
> options             : {}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 4
>
> type                : ""
>
> virtual_parent      : []
>
>
>
> _uuid               : 5d1f8b41-829b-41b1-acd3-00d20a3e27f3
>
> chassis             : []
>
> datapath            : 325496e7-112f-40f3-b612-233152629d3c
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : ovn-default-ovn-cluster
>
> mac                 : ["00:00:00:DE:08:3C"]
>
> nat_addresses       : []
>
> options             : {peer=ovn-cluster-ovn-default}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 1
>
> type                : patch
>
> virtual_parent      : []
>
>
>
> _uuid               : 5e861ab9-5665-4a5e-be91-23e2b3fd52bb
>
> chassis             : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath            : 34bc4c58-707c-41e7-a5a9-ce9047a0f904
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : node-east-1
>
> mac                 : ["00:00:00:5F:D7:E6 100.64.0.2"]
>
> nat_addresses       : []
>
> options             : {}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 2
>
> type                : ""
>
> virtual_parent      : []
>
>
>
> _uuid               : 44857ea4-3be1-4020-b237-51a8894e2db9
>
> chassis             : []
>
> datapath            : fab3717b-6a7c-428f-87b8-a82ddc355f7d
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : ovn-cluster-join
>
> mac                 : ["00:00:00:17:31:7A 100.64.0.1/16"]
>
> nat_addresses       : []
>
> options             : {peer=join-ovn-cluster}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 1
>
> type                : patch
>
> virtual_parent      : []
>
>
>
> _uuid               : 3b30c80d-985c-42ff-93f9-e350eb8192c9
>
> chassis             : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath            : fab3717b-6a7c-428f-87b8-a82ddc355f7d
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : a3957ffe-9bcc-4aac-a0c0-2f3f2082cfc2
>
> logical_port        : cr-lrp-ovn-cluster-ts-aliyun-east
>
> mac                 : ["1a:aa:aa:aa:aa:02 192.168.100.2/24"]
>
> nat_addresses       : []
>
> options             : {distributed-port=lrp-ovn-cluster-ts-aliyun-east}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 4
>
> type                : chassisredirect
>
> virtual_parent      : []
>
>
>
> _uuid               : 829efda5-ddd8-4e72-bef7-8189ae855184
>
> chassis             : 406366bc-64d6-4e68-97d6-25d254009440
>
> datapath            : 44ecebd4-0d9a-44e3-bf25-24213373f503
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : lsp-ovn-cluster-ts-aliyun-central
>
> mac                 : ["1a:aa:aa:aa:aa:01 192.168.100.1/24"]
>
> nat_addresses       : []
>
> options             : {requested-tnl-key="1"}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 1
>
> type                : remote
>
> virtual_parent      : []
>
>
>
> _uuid               : 1799546d-e41a-4e82-9d1e-038dba468011
>
> chassis             : []
>
> datapath            : fab3717b-6a7c-428f-87b8-a82ddc355f7d
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : ovn-cluster-ovn-default
>
> mac                 : ["00:00:00:DE:08:3C 10.44.0.1/16"]
>
> nat_addresses       : []
>
> options             : {peer=ovn-default-ovn-cluster}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 2
>
> type                : patch
>
> virtual_parent      : []
>
>
>
> _uuid               : d3980235-cd54-43fb-996f-1dc5defaaa02
>
> chassis             : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath            : 325496e7-112f-40f3-b612-233152629d3c
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : metrics-server-6b55c64f86-l9mhb.kube-system
>
> mac                 : ["00:00:00:A0:C4:B0 10.44.0.6"]
>
> nat_addresses       : []
>
> options             : {}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 6
>
> type                : ""
>
> virtual_parent      : []
>
>
>
> _uuid               : df777935-63ff-44c4-92d6-1874767cf28b
>
> chassis             : []
>
> datapath            : 44ecebd4-0d9a-44e3-bf25-24213373f503
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : lsp-ovn-cluster-ts-aliyun-east
>
> mac                 : [router]
>
> nat_addresses       : ["1a:aa:aa:aa:aa:02 192.168.100.2
is_chassis_resident(\"cr-lrp-ovn-cluster-ts-aliyun-east\")"]
>
> options             : {peer=lrp-ovn-cluster-ts-aliyun-east}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 2
>
> type                : patch
>
> virtual_parent      : []
>
>
>
> _uuid               : 8b1a6c4e-76b3-4993-8074-b2f1a46968d2
>
> chassis             : []
>
> datapath            : fab3717b-6a7c-428f-87b8-a82ddc355f7d
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : lrp-ovn-cluster-ts-aliyun-east
>
> mac                 : ["1a:aa:aa:aa:aa:02 192.168.100.2/24"]
>
> nat_addresses       : []
>
> options             : {peer=lsp-ovn-cluster-ts-aliyun-east}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 3
>
> type                : patch
>
> virtual_parent      : []
>
>
>
> _uuid               : a163cf6f-c139-4b98-abcd-9b1a4c102e00
>
> chassis             : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath            : 325496e7-112f-40f3-b612-233152629d3c
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : default-http-backend-67cf578fc4-wn5mj.ingress-nginx
>
> mac                 : ["00:00:00:C4:22:11 10.44.0.2"]
>
> nat_addresses       : []
>
> options             : {}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 2
>
> type                : ""
>
> virtual_parent      : []
>
>
>
> _uuid               : 473a316f-d4f7-426e-b95b-7381e30e7b79
>
> chassis             : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath            : 325496e7-112f-40f3-b612-233152629d3c
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : kube-ovn-pinger-nfcjk.kube-system
>
> mac                 : ["00:00:00:4A:20:7A 10.44.0.5"]
>
> nat_addresses       : []
>
> options             : {}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 5
>
> type                : ""
>
> virtual_parent      : []
>
>
>
> _uuid               : dbafda07-5021-4fb3-9542-fa619e9e2224
>
> chassis             : []
>
> datapath            : 34bc4c58-707c-41e7-a5a9-ce9047a0f904
>
> encap               : []
>
> external_ids        : {}
>
> gateway_chassis     : []
>
> ha_chassis_group    : []
>
> logical_port        : join-ovn-cluster
>
> mac                 : ["00:00:00:17:31:7A"]
>
> nat_addresses       : []
>
> options             : {peer=ovn-cluster-join}
>
> parent_port         : []
>
> tag                 : []
>
> tunnel_key          : 1
>
> type                : patch
>
> virtual_parent      : []
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200504/7ffba54a/attachment-0001.html>


More information about the discuss mailing list