[ovs-discuss] 回复: 回复: ovn-ic POC problem
fangtian at ruijie.com.cn
fangtian at ruijie.com.cn
Wed May 6 02:38:13 UTC 2020
Han，
I am really sorry for wasting your time，someone of us set a source IP route rule higher than the ovn-ic route ones，which let the pinging going to the wrong
destination.as I deleted the src-ip rules,it works.
Appreciate your help.
Tian
发件人: Han Zhou <hzhou at ovn.org>
发送时间: 2020年5月5日 0:47
收件人: 方田(云桌面 北京) <fangtian at ruijie.com.cn>
抄送: Han Zhou <hzhou at ovn.org>; ovs-discuss <ovs-discuss at openvswitch.org>
主题: Re: [ovs-discuss] 回复: ovn-ic POC problem
On Mon, May 4, 2020 at 4:01 AM <fangtian at ruijie.com.cn<mailto:fangtian at ruijie.com.cn>> wrote:
>
>
>
>
>
> On Sun, May 3, 2020 at 4:18 AM <fangtian at ruijie.com.cn<mailto:fangtian at ruijie.com.cn>> wrote:
> >
> >
> >
> >
> >
> > On Wed, Apr 29, 2020 at 9:34 PM <fangtian at ruijie.com.cn<mailto:fangtian at ruijie.com.cn>> wrote:
> > >
> > >
> > >
> > > Hi all，
> > >
> > > I am a developer of Kube-ovn project. Currently I’m doing an OVN-interconnect POC, the setup script is attached at the bottom.
> > >
> > > My problem is :as everything seems OK, but I can’t ping the Central AZ from the East AZ.
> > >
> > > I did diagnose this with the ovn-trace tool ,which prompt me with the message at the end :
> > >
> > >
> > >
> > > “egress(dp="join", inport="join-ovn-cluster", outport="node-east-1")
> > >
> > > -------------------------------------------------------------------
> > >
> > > 1. ls_out_pre_acl (ovn-northd.c:4690): ip, priority 100, uuid 7a78dffb
> > >
> > > reg0[0] = 1;
> > >
> > > next;
> > >
> > > 2. ls_out_pre_stateful (ovn-northd.c:4879): reg0[0] == 1, priority 100, uuid e724b591
> > >
> > > ct_next;
> > >
> > >
> > >
> > > ct_next(ct_state=est|trk /* default (use --ct to customize) */)
> > >
> > > ---------------------------------------------------------------
> > >
> > > 9. ls_out_port_sec_l2 (ovn-northd.c:4564): outport == "node-east-1", priority 50, uuid 7c98b562
> > >
> > > output;
> > >
> > > /* output to "node-east-1", type "" */
> > >
> > > “
> > >
> > > Seems the ping package was sent out from the outport "node-east-1" ,but no response there.
> > >
> > >
> > >
> > >
> > >
> > > Will someone please to tell me what’s wrong with my POC setup ?
> > >
> > > Thanks!
> > >
> > > Tian
> > >
> >
> > >
> >
> >
> >
> > >>It is hard to tell without the full topology. For example, what is the path between the source and destination?
> >
> > >>Also, what's the command used to generate the ovn-trace output? What the above trace shows is that the packet is sent out to "node-east-1", not from. But you mentioned you were pinging from east to central, so I am confused.
> >
> > >>In addition, did you check the content of IC-SB, with ovn-ic-sbctl show? Were the ports and GWs populated in IC-SB?
> >
> >
> >
> > >>Thanks,
> >
> > >>Han
> >
> >
> >
> > Thanks for reply ！
> >
> >
> >
> > I did trace the ping data path from the east-1 node，the message is from the trace command。
> >
> > My configuration is quite simple，2 AZes with 1 node each.
> >
> >
> >
> > On the central node broker-1
> >
> >
> >
> > # ovn-ic-nbctl show
> >
> > Transit_Switch ts-aliyun
> >
> >
> >
> > # ovn-ic-sbctl show
> >
> > availability-zone central
> >
> > gateway 785b3dec-0af0-4906-8018-50f511bf4e4f
> >
> > hostname: broker-1
> >
> > type: geneve
> >
> > ip: 172.17.88.1
> >
> > port lsp-ovn-cluster-ts-aliyun-central
> >
> > transit switch: ts-aliyun
> >
> > address: ["1a:aa:aa:aa:aa:01 192.168.100.1/24<http://192.168.100.1/24>"]
> >
> > availability-zone east
> >
> > gateway d68ac809-292e-4df0-9a45-0254750f2376
> >
> > hostname: east-1
> >
> > type: geneve
> >
> > ip: 172.17.88.0
> >
> > port lsp-ovn-cluster-ts-aliyun-east
> >
> > transit switch: ts-aliyun
> >
> > address: ["1a:aa:aa:aa:aa:02 192.168.100.2/24<http://192.168.100.2/24>"]
> >
> > # ovn-sbctl show
> >
> > Chassis "d68ac809-292e-4df0-9a45-0254750f2376"
> >
> > hostname: east-1
> >
> > Encap geneve
> >
> > ip: "172.17.88.0"
> >
> > options: {csum="true"}
> >
> > Port_Binding lsp-ovn-cluster-ts-aliyun-east
> >
> > Chassis "785b3dec-0af0-4906-8018-50f511bf4e4f"
> >
> > hostname: broker-1
> >
> > Encap geneve
> >
> > ip: "172.17.88.1"
> >
> > options: {csum="true"}
> >
> > Port_Binding default-http-backend-67cf578fc4-66284.ingress-nginx
> >
> > Port_Binding tiller-deploy-67cd845dff-j7jqx.kube-system
> >
> > Port_Binding coredns-autoscaler-65bfc8d47d-gvvgh.kube-system
> >
> > Port_Binding cr-lrp-ovn-cluster-ts-aliyun-central
> >
> > Port_Binding kube-ovn-pinger-4hq29.kube-system
> >
> > Port_Binding node-broker-1
> >
> > Port_Binding metrics-server-6b55c64f86-l5cq5.kube-system
> >
> > Port_Binding coredns-7c5566588d-kgjj9.kube-system
> >
> >
> >
> > ovn-sbctl show
> >
> > Chassis "d68ac809-292e-4df0-9a45-0254750f2376"
> >
> > hostname: east-1
> >
> > Encap geneve
> >
> > ip: "172.17.88.0"
> >
> > options: {csum="true"}
> >
> > Port_Binding lsp-ovn-cluster-ts-aliyun-east
> >
> > Chassis "785b3dec-0af0-4906-8018-50f511bf4e4f"
> >
> > hostname: broker-1
> >
> > Encap geneve
> >
> > ip: "172.17.88.1"
> >
> > options: {csum="true"}
> >
> > Port_Binding default-http-backend-67cf578fc4-66284.ingress-nginx
> >
> > Port_Binding tiller-deploy-67cd845dff-j7jqx.kube-system
> >
> > Port_Binding coredns-autoscaler-65bfc8d47d-gvvgh.kube-system
> >
> > Port_Binding cr-lrp-ovn-cluster-ts-aliyun-central
> >
> > Port_Binding kube-ovn-pinger-4hq29.kube-system
> >
> > Port_Binding node-broker-1
> >
> > Port_Binding metrics-server-6b55c64f86-l5cq5.kube-system
> >
> > Port_Binding coredns-7c5566588d-kgjj9.kube-system
> >
> >
> >
> >
> >
> >
> >
> > On the east-1 node , things are similar.
> >
> >
> >
> > sh-4.4# ovn-nbctl show
> >
> > switch 1f157fe4-8576-400e-85fb-7f1fad7daf44 (ovn-default)
> >
> > port metrics-server-6b55c64f86-l9mhb.kube-system
> >
> > addresses: ["00:00:00:A0:C4:B0 10.44.0.6"]
> >
> > port coredns-7c5566588d-bn4fh.kube-system
> >
> > addresses: ["00:00:00:B1:40:ED 10.44.0.3"]
> >
> > port coredns-autoscaler-65bfc8d47d-xk9md.kube-system
> >
> > addresses: ["00:00:00:9C:49:B9 10.44.0.4"]
> >
> > port kube-ovn-pinger-nfcjk.kube-system
> >
> > addresses: ["00:00:00:4A:20:7A 10.44.0.5"]
> >
> > port ovn-default-ovn-cluster
> >
> > type: router
> >
> > addresses: ["00:00:00:DE:08:3C"]
> >
> > router-port: ovn-cluster-ovn-default
> >
> > port default-http-backend-67cf578fc4-wn5mj.ingress-nginx
> >
> > addresses: ["00:00:00:C4:22:11 10.44.0.2"]
> >
> > switch 0cbefdd1-8067-4db2-8a2d-3487dc16c3ea (ts-aliyun)
> >
> > port lsp-ovn-cluster-ts-aliyun-central
> >
> > type: remote
> >
> > addresses: ["1a:aa:aa:aa:aa:01 192.168.100.1/24<http://192.168.100.1/24>"]
> >
> > port lsp-ovn-cluster-ts-aliyun-east
> >
> > type: router
> >
> > router-port: lrp-ovn-cluster-ts-aliyun-east
> >
> > switch 61a36d09-0b39-46f1-9da1-967ac195905b (join)
> >
> > port join-ovn-cluster
> >
> > type: router
> >
> > addresses: ["00:00:00:17:31:7A"]
> >
> > router-port: ovn-cluster-join
> >
> > port node-east-1
> >
> > addresses: ["00:00:00:5F:D7:E6 100.64.0.2"]
> >
> > router b7ef9172-ccf2-4489-8802-a03f1c441a51 (ovn-cluster)
> >
> > port ovn-cluster-join
> >
> > mac: "00:00:00:17:31:7A"
> >
> > networks: ["100.64.0.1/16<http://100.64.0.1/16>"]
> >
> > port lrp-ovn-cluster-ts-aliyun-east
> >
> > mac: "1a:aa:aa:aa:aa:02"
> >
> > networks: ["192.168.100.2/24<http://192.168.100.2/24>"]
> >
> > gateway chassis: [d68ac809-292e-4df0-9a45-0254750f2376]
> >
> > port ovn-cluster-ovn-default
> >
> > mac: "00:00:00:DE:08:3C"
> >
> > networks: ["10.44.0.1/16<http://10.44.0.1/16>"]
> >
> > sh-4.4# ovn-sbctl show
> >
> > Chassis "785b3dec-0af0-4906-8018-50f511bf4e4f"
> >
> > hostname: broker-1
> >
> > Encap geneve
> >
> > ip: "172.17.88.1"
> >
>
> > options: {csum="true"}
>
>
>
> Here it doesn't show the port-binding for port lsp-ovn-cluster-ts-aliyun-central, which should be learned from IC-SB DB. However, the port is shown in NB DB, and the chassis binding is also shown in IC-SB. So it means the chassis information of the port-binding is not learned from IC-SB to SB of the east AZ. Could you check if there is any error log in ovn-ic of east AZ? Are the connections to SB and IC-SB both working well? Could you share the output of "ovn-sbctl list port_binding" in east AZ as well?
>
>
>
>
>
> ##East ovn-ic log
>
>
>
> 2020-05-04T10:28:58.129Z|00006|reconnect|INFO|tcp:172.17.88.0:6641<http://172.17.88.0:6641>: connected
>
> 2020-05-04T10:28:58.129Z|00007|reconnect|INFO|tcp:172.17.88.0:6642<http://172.17.88.0:6642>: connected
>
> 2020-05-04T10:28:58.129Z|00008|reconnect|INFO|tcp:172.17.88.1:6645<http://172.17.88.1:6645>: connected
>
> 2020-05-04T10:28:58.129Z|00009|reconnect|INFO|tcp:172.17.88.1:6646<http://172.17.88.1:6646>: connected
>
> 2020-05-04T10:28:58.129Z|00010|ovn_ic|INFO|ovn-ic lock acquired. This ovn-ic instance is now active.
>
> 2020-05-04T10:28:58.130Z|00011|ovn_ic|INFO|NB Global not exist.
>
> 2020-05-04T10:28:58.131Z|00012|ovn_ic|INFO|NB Global not exist.
>
> 2020-05-04T10:28:58.133Z|00013|ovn_ic|WARN|Route sync ignores port lsp-ovn-cluster-ts-aliyun-east on ts ts-aliyun because logical router port is not found in NB.
>
>
>
> sh-4.4# ovn-sbctl show
>
> Chassis "785b3dec-0af0-4906-8018-50f511bf4e4f"
>
> hostname: broker-1
>
> Encap geneve
>
> ip: "172.17.88.1"
>
> options: {csum="true"}
>
> Port_Binding lsp-ovn-cluster-ts-aliyun-central
>
>
>
> ## this time it has port binding，but still not work.
It is a little strange that the port-binding wasn't there last time. It could be a transient state or it could be a bug. Could you confirm if the current DB status is stable? i.e. does every ovn-sbctl show output shows the same port-bindings?
If it is stable now, the root cause of not pinging could be something else. Please do tcpdump to firstly identify on which part of the path was the packet dropped.
If the ping is from node-east-1 to some node in central AZ, did central side node receive ping request and sent out reply? Could you do tcpdump on that node to verify? If not, you can check if the packet arrived iinterconn GW on each side, by doing tcpdump on the genev_sys_6081interface on the gateway nodes? If ping is received and reply is sent out by the central AZ node, you could do similar tcpdump on the GWs to identify which node dropped the packet. Then the debug can continue there.
>
> Chassis "d68ac809-292e-4df0-9a45-0254750f2376"
>
> hostname: east-1
>
> Encap geneve
>
> ip: "172.17.88.0"
>
> options: {csum="true"}
>
> Port_Binding coredns-7c5566588d-bn4fh.kube-system
>
> Port_Binding coredns-autoscaler-65bfc8d47d-xk9md.kube-system
>
> Port_Binding node-east-1
>
> Port_Binding cr-lrp-ovn-cluster-ts-aliyun-east
>
> Port_Binding metrics-server-6b55c64f86-l9mhb.kube-system
>
> Port_Binding default-http-backend-67cf578fc4-wn5mj.ingress-nginx
>
> Port_Binding kube-ovn-pinger-nfcjk.kube-system
>
>
>
>
>
> sh-4.4# ovn-sbctl list port_binding
>
> _uuid : c9193648-752d-475c-955c-27bdb51bf946
>
> chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath : 325496e7-112f-40f3-b612-233152629d3c
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : coredns-7c5566588d-bn4fh.kube-system
>
> mac : ["00:00:00:B1:40:ED 10.44.0.3"]
>
> nat_addresses : []
>
> options : {}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 3
>
> type : ""
>
> virtual_parent : []
>
>
>
> _uuid : c5f6c879-d954-48a2-99b1-8f377c948dde
>
> chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath : 325496e7-112f-40f3-b612-233152629d3c
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : coredns-autoscaler-65bfc8d47d-xk9md.kube-system
>
> mac : ["00:00:00:9C:49:B9 10.44.0.4"]
>
> nat_addresses : []
>
> options : {}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 4
>
> type : ""
>
> virtual_parent : []
>
>
>
> _uuid : 5d1f8b41-829b-41b1-acd3-00d20a3e27f3
>
> chassis : []
>
> datapath : 325496e7-112f-40f3-b612-233152629d3c
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : ovn-default-ovn-cluster
>
> mac : ["00:00:00:DE:08:3C"]
>
> nat_addresses : []
>
> options : {peer=ovn-cluster-ovn-default}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 1
>
> type : patch
>
> virtual_parent : []
>
>
>
> _uuid : 5e861ab9-5665-4a5e-be91-23e2b3fd52bb
>
> chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath : 34bc4c58-707c-41e7-a5a9-ce9047a0f904
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : node-east-1
>
> mac : ["00:00:00:5F:D7:E6 100.64.0.2"]
>
> nat_addresses : []
>
> options : {}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 2
>
> type : ""
>
> virtual_parent : []
>
>
>
> _uuid : 44857ea4-3be1-4020-b237-51a8894e2db9
>
> chassis : []
>
> datapath : fab3717b-6a7c-428f-87b8-a82ddc355f7d
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : ovn-cluster-join
>
> mac : ["00:00:00:17:31:7A 100.64.0.1/16<http://100.64.0.1/16>"]
>
> nat_addresses : []
>
> options : {peer=join-ovn-cluster}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 1
>
> type : patch
>
> virtual_parent : []
>
>
>
> _uuid : 3b30c80d-985c-42ff-93f9-e350eb8192c9
>
> chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath : fab3717b-6a7c-428f-87b8-a82ddc355f7d
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : a3957ffe-9bcc-4aac-a0c0-2f3f2082cfc2
>
> logical_port : cr-lrp-ovn-cluster-ts-aliyun-east
>
> mac : ["1a:aa:aa:aa:aa:02 192.168.100.2/24<http://192.168.100.2/24>"]
>
> nat_addresses : []
>
> options : {distributed-port=lrp-ovn-cluster-ts-aliyun-east}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 4
>
> type : chassisredirect
>
> virtual_parent : []
>
>
>
> _uuid : 829efda5-ddd8-4e72-bef7-8189ae855184
>
> chassis : 406366bc-64d6-4e68-97d6-25d254009440
>
> datapath : 44ecebd4-0d9a-44e3-bf25-24213373f503
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : lsp-ovn-cluster-ts-aliyun-central
>
> mac : ["1a:aa:aa:aa:aa:01 192.168.100.1/24<http://192.168.100.1/24>"]
>
> nat_addresses : []
>
> options : {requested-tnl-key="1"}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 1
>
> type : remote
>
> virtual_parent : []
>
>
>
> _uuid : 1799546d-e41a-4e82-9d1e-038dba468011
>
> chassis : []
>
> datapath : fab3717b-6a7c-428f-87b8-a82ddc355f7d
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : ovn-cluster-ovn-default
>
> mac : ["00:00:00:DE:08:3C 10.44.0.1/16<http://10.44.0.1/16>"]
>
> nat_addresses : []
>
> options : {peer=ovn-default-ovn-cluster}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 2
>
> type : patch
>
> virtual_parent : []
>
>
>
> _uuid : d3980235-cd54-43fb-996f-1dc5defaaa02
>
> chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath : 325496e7-112f-40f3-b612-233152629d3c
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : metrics-server-6b55c64f86-l9mhb.kube-system
>
> mac : ["00:00:00:A0:C4:B0 10.44.0.6"]
>
> nat_addresses : []
>
> options : {}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 6
>
> type : ""
>
> virtual_parent : []
>
>
>
> _uuid : df777935-63ff-44c4-92d6-1874767cf28b
>
> chassis : []
>
> datapath : 44ecebd4-0d9a-44e3-bf25-24213373f503
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : lsp-ovn-cluster-ts-aliyun-east
>
> mac : [router]
>
> nat_addresses : ["1a:aa:aa:aa:aa:02 192.168.100.2 is_chassis_resident(\"cr-lrp-ovn-cluster-ts-aliyun-east\")"]
>
> options : {peer=lrp-ovn-cluster-ts-aliyun-east}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 2
>
> type : patch
>
> virtual_parent : []
>
>
>
> _uuid : 8b1a6c4e-76b3-4993-8074-b2f1a46968d2
>
> chassis : []
>
> datapath : fab3717b-6a7c-428f-87b8-a82ddc355f7d
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : lrp-ovn-cluster-ts-aliyun-east
>
> mac : ["1a:aa:aa:aa:aa:02 192.168.100.2/24<http://192.168.100.2/24>"]
>
> nat_addresses : []
>
> options : {peer=lsp-ovn-cluster-ts-aliyun-east}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 3
>
> type : patch
>
> virtual_parent : []
>
>
>
> _uuid : a163cf6f-c139-4b98-abcd-9b1a4c102e00
>
> chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath : 325496e7-112f-40f3-b612-233152629d3c
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : default-http-backend-67cf578fc4-wn5mj.ingress-nginx
>
> mac : ["00:00:00:C4:22:11 10.44.0.2"]
>
> nat_addresses : []
>
> options : {}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 2
>
> type : ""
>
> virtual_parent : []
>
>
>
> _uuid : 473a316f-d4f7-426e-b95b-7381e30e7b79
>
> chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e
>
> datapath : 325496e7-112f-40f3-b612-233152629d3c
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : kube-ovn-pinger-nfcjk.kube-system
>
> mac : ["00:00:00:4A:20:7A 10.44.0.5"]
>
> nat_addresses : []
>
> options : {}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 5
>
> type : ""
>
> virtual_parent : []
>
>
>
> _uuid : dbafda07-5021-4fb3-9542-fa619e9e2224
>
> chassis : []
>
> datapath : 34bc4c58-707c-41e7-a5a9-ce9047a0f904
>
> encap : []
>
> external_ids : {}
>
> gateway_chassis : []
>
> ha_chassis_group : []
>
> logical_port : join-ovn-cluster
>
> mac : ["00:00:00:17:31:7A"]
>
> nat_addresses : []
>
> options : {peer=ovn-cluster-join}
>
> parent_port : []
>
> tag : []
>
> tunnel_key : 1
>
> type : patch
>
> virtual_parent : []
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org<mailto:discuss at openvswitch.org>
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200506/3ff0c943/attachment-0001.html>
More information about the discuss
mailing list