[ovs-discuss] No connectivity due to missing ARP reply

Dumitru Ceara dceara at redhat.com
Mon Mar 23 12:27:47 UTC 2020


On 3/21/20 7:04 PM, Plato, Michael wrote:
> 
> Hi all,
> 
> we use OVN with Openstack and have a problem with the following setup:
> 
> 
> 				|					|
> 	-------		|	  10.176.0.156	|	-------
> 	| VM1 |-----   |	 192.168.0.1	        |---| VM2 |
> 	-------		|	   --------		        |	-------
> 10.176.0.3.123	|------|  R1  |-------------|	192.168.0.201 / GW: 192.168.0.1
> GW:10.176.0.1	|	  |(test)|		|	FIP: 10.176.2.19
> 				|	   --------		        |
> 			 Outside		                      test
> 		 (10.176.0.0/16)  	             (192.168.0.0/24)
> 			 (VLAN)			 	 (GENEVE)
> 
> 
> Versions:
> - OVN (20.03)
> - OVS (2.13)
> - networking-ovn (7.1.0)
> 			 
> Problem: 
> - no connectivity due to missing ARP reply for FIP 10.176.2.19 from VM1 (if VM1 is not on GW Chassis for R1 -> is_chassis_resident rules not applied)
> - after moving VM1 to chassis hosting R1 ARP reply appears (due to local "is_chassis_resident" ARP responder rules)
> - temporarily removing priority 75 rules (inserted by commit [0]) restores functionality (even on non gateway chassis), because ARP requests were flooded to complete L2 domain (but this creates a scaling issue)
> 
> 
> Analysis:
> - according to ovs-detrace the ARP requests were dropped instead of being forwarded to remote chassis hosting R1 (as intended by [0])
> 
> 
> Flow: arp,in_port=61,vlan_tci=0x0000,dl_src=fa:16:3e:5e:79:d9,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.176.3.123,arp_tpa=10.176.2.19,arp_op=1,arp_sha=fa:16:3e:5e:79:d9,arp_tha=00:00:00:00:00:00
> 
> 
> bridge("br-int")
> ----------------
> 0. in_port=61, priority 100, cookie 0x862b95fc
> set_field:0x1->reg13
> set_field:0x7->reg11
> set_field:0x5->reg12
> set_field:0x1a->metadata
> set_field:0x4->reg14
> resubmit(,8)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d)
>   *  Port Binding: logical_port "b19ceab1-c7fe-4c3b-8733-d88cabaa0a23", tunnel_key 4, chassis-name "383eb44a-de85-485a-9606-2fc649a9cbb9", chassis-str "os-compute-01"
> 8. reg14=0x4,metadata=0x1a,dl_src=fa:16:3e:5e:79:d9, priority 50, cookie 0x9a357820
> resubmit(,9)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=0 (ls_in_port_sec_l2), priority=50, match=(inport == "b19ceab1-c7fe-4c3b-8733-d88cabaa0a23" && eth.src == {fa:16:3e:5e:79:d9}), actions=(next;)
>    *  Logical Switch Port: b19ceab1-c7fe-4c3b-8733-d88cabaa0a23 type  (addresses ['fa:16:3e:5e:79:d9 10.176.3.123'], dynamic addresses [], security ['fa:16:3e:5e:79:d9 10.176.3.123']
> 9. metadata=0x1a, priority 0, cookie 0x1a478ee1
> resubmit(,10)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=1 (ls_in_port_sec_ip), priority=0, match=(1), actions=(next;)
> 10. arp,reg14=0x4,metadata=0x1a,dl_src=fa:16:3e:5e:79:d9,arp_spa=10.176.3.123,arp_sha=fa:16:3e:5e:79:d9, priority 90, cookie 0x8c5af8ff
> resubmit(,11)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=2 (ls_in_port_sec_nd), priority=90, match=(inport == "b19ceab1-c7fe-4c3b-8733-d88cabaa0a23" && eth.src == fa:16:3e:5e:79:d9 && arp.sha == fa:16:3e:5e:79:d9 && arp.spa == {10.176.3.123}), actions=(next;)
>    *  Logical Switch Port: b19ceab1-c7fe-4c3b-8733-d88cabaa0a23 type  (addresses ['fa:16:3e:5e:79:d9 10.176.3.123'], dynamic addresses [], security ['fa:16:3e:5e:79:d9 10.176.3.123']
> 11. metadata=0x1a, priority 0, cookie 0x13f72632
> resubmit(,12)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=3 (ls_in_pre_acl), priority=0, match=(1), actions=(next;)
> 12. metadata=0x1a, priority 0, cookie 0xe38d6752
> resubmit(,13)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=4 (ls_in_pre_lb), priority=0, match=(1), actions=(next;)
> 13. metadata=0x1a, priority 0, cookie 0xa9a6ed5
> resubmit(,14)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=5 (ls_in_pre_stateful), priority=0, match=(1), actions=(next;)
> 14. metadata=0x1a, priority 0, cookie 0xcf9951d4
> resubmit(,15)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=6 (ls_in_acl), priority=0, match=(1), actions=(next;)
> 15. metadata=0x1a, priority 0, cookie 0xcc08c09e
> resubmit(,16)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=7 (ls_in_qos_mark), priority=0, match=(1), actions=(next;)
> 16. metadata=0x1a, priority 0, cookie 0x918349d8
> resubmit(,17)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=8 (ls_in_qos_meter), priority=0, match=(1), actions=(next;)
> 17. metadata=0x1a, priority 0, cookie 0x944ba2a
> resubmit(,18)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=9 (ls_in_lb), priority=0, match=(1), actions=(next;)
> 18. metadata=0x1a, priority 0, cookie 0xcbae6cab
> resubmit(,19)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=10 (ls_in_stateful), priority=0, match=(1), actions=(next;)
> 19. metadata=0x1a, priority 0, cookie 0xf96fbcc8
> resubmit(,20)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
> 20. metadata=0x1a, priority 0, cookie 0x5b9711bf
> resubmit(,21)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=12 (ls_in_hairpin), priority=0, match=(1), actions=(next;)
> 21. metadata=0x1a, priority 0, cookie 0x120d1c68
> resubmit(,22)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=13 (ls_in_arp_rsp), priority=0, match=(1), actions=(next;)
> 22. metadata=0x1a, priority 0, cookie 0xd446226f
> resubmit(,23)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=14 (ls_in_dhcp_options), priority=0, match=(1), actions=(next;)
> 23. metadata=0x1a, priority 0, cookie 0x31b45717
> resubmit(,24)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=15 (ls_in_dhcp_response), priority=0, match=(1), actions=(next;)
> 24. metadata=0x1a, priority 0, cookie 0x715db0f1
> resubmit(,25)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=16 (ls_in_dns_lookup), priority=0, match=(1), actions=(next;)
> 25. metadata=0x1a, priority 0, cookie 0xd12f2910
> resubmit(,26)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=17 (ls_in_dns_response), priority=0, match=(1), actions=(next;)
> 26. metadata=0x1a, priority 0, cookie 0x97f5a6b7
> resubmit(,27)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=18 (ls_in_external_port), priority=0, match=(1), actions=(next;)
> 27. arp,reg10=0/0x2,metadata=0x1a,arp_tpa=10.176.2.19,arp_op=1, priority 75, cookie 0x4a641791
> set_field:0x5->reg15
> resubmit(,32)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [ingress]
>   *  Logical flow: table=19 (ls_in_l2_lkup), priority=75, match=(flags[1] == 0 && arp.op == 1 && arp.tpa == { 10.176.2.19, 10.176.0.156}), actions=(outport = "9f1c631b-ea6c-4f50-85f0-a5b0bb6e2e8e"; output;)
>    *  Logical Switch Port: 9f1c631b-ea6c-4f50-85f0-a5b0bb6e2e8e type router (addresses ['router'], dynamic addresses [], security []
> 32. priority 0
> resubmit(,33)
> 33. reg15=0x5,metadata=0x1a, priority 100
> set_field:0x7->reg11
> set_field:0x5->reg12
> resubmit(,34)
> 34. priority 0
> set_field:0->reg0
> set_field:0->reg1
> set_field:0->reg2
> set_field:0->reg3
> set_field:0->reg4
> set_field:0->reg5
> set_field:0->reg6
> set_field:0->reg7
> set_field:0->reg8
> set_field:0->reg9
> resubmit(,40)
> 40. metadata=0x1a, priority 0, cookie 0xf960a9ea
> resubmit(,41)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [egress]
>   *  Logical flow: table=0 (ls_out_pre_lb), priority=0, match=(1), actions=(next;)
> 41. metadata=0x1a, priority 0, cookie 0x31e15a58
> resubmit(,42)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [egress]
>   *  Logical flow: table=1 (ls_out_pre_acl), priority=0, match=(1), actions=(next;)
> 42. metadata=0x1a, priority 0, cookie 0x7089b16e
> resubmit(,43)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [egress]
>   *  Logical flow: table=2 (ls_out_pre_stateful), priority=0, match=(1), actions=(next;)
> 43. metadata=0x1a, priority 0, cookie 0x8ab997b2
> resubmit(,44)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [egress]
>   *  Logical flow: table=3 (ls_out_lb), priority=0, match=(1), actions=(next;)
> 44. metadata=0x1a, priority 0, cookie 0x62e08a84
> resubmit(,45)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [egress]
>   *  Logical flow: table=4 (ls_out_acl), priority=0, match=(1), actions=(next;)
> 45. metadata=0x1a, priority 0, cookie 0x19ac76fa
> resubmit(,46)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [egress]
>   *  Logical flow: table=5 (ls_out_qos_mark), priority=0, match=(1), actions=(next;)
> 46. metadata=0x1a, priority 0, cookie 0x826c6009
> resubmit(,47)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [egress]
>   *  Logical flow: table=6 (ls_out_qos_meter), priority=0, match=(1), actions=(next;)
> 47. metadata=0x1a, priority 0, cookie 0xbaa04ed8
> resubmit(,48)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [egress]
>   *  Logical flow: table=7 (ls_out_stateful), priority=0, match=(1), actions=(next;)
> 48. metadata=0x1a, priority 0, cookie 0x7c014dc6
> resubmit(,49)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [egress]
>   *  Logical flow: table=8 (ls_out_port_sec_ip), priority=0, match=(1), actions=(next;)
> 49. metadata=0x1a,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00, priority 100, cookie 0x3a1562c4
> resubmit(,64)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d) [egress]
>   *  Logical flow: table=9 (ls_out_port_sec_l2), priority=100, match=(eth.mcast), actions=(output;)
> 64. priority 0
> resubmit(,65)
> 65. reg15=0x5,metadata=0x1a, priority 100, cookie 0x99c3e9ca
> clone(ct_clear,set_field:0->reg11,set_field:0->reg12,set_field:0->reg13,set_field:0x3->reg11,set_field:0x4->reg12,set_field:0x26->metadata,set_field:0x1->reg14,set_field:0->reg10,set_field:0->reg15,set_field:0->reg0,set_field:0->reg1,set_field:0->reg2,set_field:0->reg3,set_field:0->reg4,set_field:0->reg5,set_field:0->reg6,set_field:0->reg7,set_field:0->reg8,set_field:0->reg9,resubmit(,8))
> ct_clear
> set_field:0->reg11
> set_field:0->reg12
> set_field:0->reg13
> set_field:0x3->reg11
> set_field:0x4->reg12
> set_field:0x26->metadata
> set_field:0x1->reg14
> set_field:0->reg10
> set_field:0->reg15
> set_field:0->reg0
> set_field:0->reg1
> set_field:0->reg2
> set_field:0->reg3
> set_field:0->reg4
> set_field:0->reg5
> set_field:0->reg6
> set_field:0->reg7
> set_field:0->reg8
> set_field:0->reg9
> resubmit(,8)
>   *  Logical datapath: "neutron-c2a82a31-632b-4d24-8f35-8a79e2a207a7" (d516056b-19a6-4613-9838-8c62452fe31d)
>   *  Port Binding: logical_port "9f1c631b-ea6c-4f50-85f0-a5b0bb6e2e8e", tunnel_key 5,
> 8. reg14=0x1,metadata=0x26,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00, priority 50, cookie 0x12ef5598
> resubmit(,9)
>   *  Logical datapath: "neutron-673c043b-5c3a-4eae-90d9-bf6c87c1fc37" (f7fad4e5-71bc-42cd-8a1b-048e25137dc0) [ingress]
>   *  Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.mcast && inport == "lrp-9f1c631b-ea6c-4f50-85f0-a5b0bb6e2e8e), actions=(next;)
>    *  Logical Router Port: lrp-9f1c631b-ea6c-4f50-85f0-a5b0bb6e2e8e mac fa:16:3e:58:84:8c networks ['10.176.0.156/16'] ipv6_ra_configs {}
> 9. metadata=0x26, priority 0, cookie 0xab1b8863
> load:0x1->OXM_OF_PKT_REG4[3]
> resubmit(,10)
>   *  Logical datapath: "neutron-673c043b-5c3a-4eae-90d9-bf6c87c1fc37" (f7fad4e5-71bc-42cd-8a1b-048e25137dc0) [ingress]
>   *  Logical flow: table=1 (lr_in_lookup_neighbor), priority=0, match=(1), actions=(reg9[3] = 1; next;)
> 10. reg9=0x8/0x8,metadata=0x26, priority 100, cookie 0x742e0523
> resubmit(,11)
>   *  Logical datapath: "neutron-673c043b-5c3a-4eae-90d9-bf6c87c1fc37" (f7fad4e5-71bc-42cd-8a1b-048e25137dc0) [ingress]
>   *  Logical flow: table=2 (lr_in_learn_neighbor), priority=100, match=(reg9[3] == 1 || reg9[2] == 1), actions=(next;)
> 11. arp,metadata=0x26, priority 85, cookie 0xb1c400fe
> drop
>   *  Logical datapath: "neutron-673c043b-5c3a-4eae-90d9-bf6c87c1fc37" (f7fad4e5-71bc-42cd-8a1b-048e25137dc0) [ingress]
>   *  Logical flow: table=3 (lr_in_ip_input), priority=85, match=(arp || nd), actions=(drop;)
> 
> 
> Final flow: arp,reg11=0x7,reg12=0x5,reg13=0x1,reg14=0x4,reg15=0x5,metadata=0x1a,in_port=61,vlan_tci=0x0000,dl_src=fa:16:3e:5e:79:d9,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.176.3.123,arp_tpa=10.176.2.19,arp_op=1,arp_sha=fa:16:3e:5e:79:d9,arp_tha=00:00:00:00:00:00
> Megaflow: recirc_id=0,ct_state=-new-est-rel-rpl-inv-trk,ct_label=0/0x1,eth,arp,in_port=61,dl_src=fa:16:3e:5e:79:d9,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=10.176.3.123,arp_tpa=10.176.2.19,arp_op=1,arp_sha=fa:16:3e:5e:79:d9
> Datapath actions: ct_clear
> 
> 
> Looks like theres a rule missing for tunneling ARP/ND to remote chassis in case of distributed router?
> 
> 
> Thanks a lot,
> 
> 
> Michael
> 
> 
> [0] https://github.com/ovn-org/ovn/commit/32f5ebb06226e3433e53e05bdc75d16752859a0e​
> 

Hi Michael,

Thanks for reporting this. Could you please try this patch?
https://patchwork.ozlabs.org/patch/1259982/

>From what I understand it should fix the problem you're seeing.

Thanks,
Dumitru



More information about the discuss mailing list