[ovs-discuss] OVN usage with VMs

Gerhard Stenzel gstenzel at linux.vnet.ibm.com
Fri Jul 8 11:24:38 UTC 2016


On 07/07/2016 09:18 PM, Russell Bryant wrote:
...
> I don't know of any docs that show this in detail.  The closest I'm
> aware of is documentation on the OpenStack integration, which sets up
> VMs with OVN.
>
> http://docs.openstack.org/developer/networking-ovn/testing.html
>
> http://docs.openstack.org/developer/networking-ovn/refarch.html
>
>
> --
> Russell Bryant
>

I also spent hours searching for a tutorial describing this and was 
surprised to find none. After same trial and error, I came up with the 
following for a logical switch with two logical ports and the mapping to 
one OVS and two ports connected to actual VMs ...


Using OVN without Sandbox

Start two VMs with a NIC attached to a linux bridge or an OVS. Remove 
the two VNET devices:

ovs-vsctl del-port bridge vnet2
ovs-vsctl del-port bridge vnet3

At this point, the VMs are without network connection. Now, create OVS 
and add VNETs with iface-id.

[tutorial]# ovs-vsctl add-br br-int
[tutorial]# ovs-vsctl add-port br-int vnet2 -- set Interface vnet2 
external_ids:iface-id=sw0-port1
[tutorial]# ovs-vsctl add-port br-int vnet3 -- set Interface vnet3 
external_ids:iface-id=sw0-port2
[tutorial]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
  cookie=0x0, duration=24.117s, table=0, n_packets=43, n_bytes=3598, 
idle_age=0, priority=0 actions=NORMAL

Ping VM2 from VM1. It should work:

[vm1 ~]# ping 10.0.0.2 -c2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.362 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.391 ms

--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.362/0.376/0.391/0.024 ms

So far, this is plain OVS. Now, bring OVN into the picture:

[tutorial]# service  ovn-northd start
Redirecting to /bin/systemctl start  ovn-northd.service
[tutorial]# service  ovn-controller start
Redirecting to /bin/systemctl start  ovn-controller.service

The ping between the two VMs does not work anymore.

[tutorial]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
  cookie=0x0, duration=6.544s, table=32, n_packets=0, n_bytes=0, 
idle_age=6, priority=0 actions=resubmit(,33)
  cookie=0x0, duration=6.544s, table=34, n_packets=0, n_bytes=0, 
idle_age=6, priority=0 
actions=load:0->NXM_NX_REG0[],load:0->NXM_NX_REG1[],load:0->NXM_NX_REG2[],load:0->NXM_NX_REG3[],load:0->NXM_NX_REG4[],resubmit(,48)

Add the logical switch and ports:

[tutorial]# ovn-nbctl lswitch-add sw0
[tutorial]# ovn-nbctl show
     lswitch 73c496d3-7d76-41ac-9553-b8861237988c (sw0)
[tutorial]# ovn-sbctl show
Chassis "247f358e-8d8f-41d7-8cb5-fa30c28cfb24"
     Encap geneve
         ip: "127.0.0.1"
[tutorial]# ovn-nbctl lport-add sw0 sw0-port2
[tutorial]# ovn-nbctl lport-add sw0 sw0-port1

Find out the MAC addresses for the logical ports:

[tutorial]# ip a sh vnet2
841: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
master ovs-system state UNKNOWN qlen 500
     link/ether fe:54:00:69:57:15 brd ff:ff:ff:ff:ff:ff
     inet6 fe80::fc54:ff:fe69:5715/64 scope link
        valid_lft forever preferred_lft forever
[tutorial]# ip a sh vnet3
840: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
master ovs-system state UNKNOWN qlen 500
     link/ether fe:54:00:ea:62:56 brd ff:ff:ff:ff:ff:ff
     inet6 fe80::fc54:ff:feea:6256/64 scope link
        valid_lft forever preferred_lft forever

Beware, libvirtd changes the first byte of the VMs MAC address to set 
the VNET devices MAC address. The real MAC is in the VMs:

[vm1]# ip a sh  eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
state UP qlen 1000
     link/ether 52:54:00:69:57:15 brd ff:ff:ff:ff:ff:ff
     inet 10.0.0.1/24 brd 10.0.0.255 scope global eth0
        valid_lft forever preferred_lft forever
     inet6 fe80::5054:ff:fe69:5715/64 scope link
        valid_lft forever preferred_lft forever

[vm2]# ip a sh eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
state UP qlen 1000
     link/ether 52:54:00:ea:62:56 brd ff:ff:ff:ff:ff:ff
     inet 10.0.0.2/24 brd 10.0.0.255 scope global eth0
        valid_lft forever preferred_lft forever
     inet6 fe80::5054:ff:feea:6256/64 scope link
        valid_lft forever preferred_lft forever

Use those to set the MAC addresses for the logical ports:

[tutorial]# ovn-nbctl lport-set-addresses sw0-port1 52:54:00:69:57:15
[tutorial]# ovn-nbctl lport-set-addresses sw0-port2 52:54:00:ea:62:56
[tutorial]# ovn-nbctl show
     lswitch 73c496d3-7d76-41ac-9553-b8861237988c (sw0)
         lport sw0-port2
             addresses: ["52:54:00:ea:62:56"]
         lport sw0-port1
             addresses: ["52:54:00:69:57:15"]
[tutorial]# ovn-sbctl show
Chassis "247f358e-8d8f-41d7-8cb5-fa30c28cfb24"
     Encap geneve
         ip: "127.0.0.1"
     Port_Binding "sw0-port1"
     Port_Binding "sw0-port2"

[tutorial]#  ovn-sbctl lflow-list
Datapath: dcd23188-70c7-46d6-a5cb-c8df8f75959c  Pipeline: ingress
   table=0(  ls_in_port_sec), priority=  100, match=(eth.src[40]), 
action=(drop;)
   table=0(  ls_in_port_sec), priority=  100, match=(vlan.present), 
action=(drop;)
   table=0(  ls_in_port_sec), priority=   50, match=(inport == 
"sw0-port1"), action=(next;)
   table=0(  ls_in_port_sec), priority=   50, match=(inport == 
"sw0-port2"), action=(next;)
   table=1(   ls_in_pre_acl), priority=    0, match=(1), action=(next;)
   table=2(       ls_in_acl), priority=    0, match=(1), action=(next;)
   table=3(   ls_in_l2_lkup), priority=  100, match=(eth.mcast), 
action=(outport = "_MC_flood"; output;)
   table=3(   ls_in_l2_lkup), priority=   50, match=(eth.dst == 
52:54:00:69:57:15), action=(outport = "sw0-port1"; output;)
   table=3(   ls_in_l2_lkup), priority=   50, match=(eth.dst == 
52:54:00:ea:62:56), action=(outport = "sw0-port2"; output;)
Datapath: dcd23188-70c7-46d6-a5cb-c8df8f75959c  Pipeline: egress
   table=0(  ls_out_pre_acl), priority=    0, match=(1), action=(next;)
   table=1(      ls_out_acl), priority=    0, match=(1), action=(next;)
   table=2( ls_out_port_sec), priority=  100, match=(eth.mcast), 
action=(output;)
   table=2( ls_out_port_sec), priority=   50, match=(outport == 
"sw0-port1"), action=(output;)
   table=2( ls_out_port_sec), priority=   50, match=(outport == 
"sw0-port2"), action=(output;)

[tutorial]# ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
  cookie=0x0, duration=74.133s, table=0, n_packets=138, n_bytes=12852, 
idle_age=0, priority=100,in_port=1 
actions=load:0x2->NXM_NX_REG5[],load:0x1->OXM_OF_METADATA[],load:0x1->NXM_NX_REG6[],resubmit(,16)
  cookie=0x0, duration=69.124s, table=0, n_packets=142, n_bytes=13524, 
idle_age=0, priority=100,in_port=2 
actions=load:0x1->NXM_NX_REG5[],load:0x1->OXM_OF_METADATA[],load:0x2->NXM_NX_REG6[],resubmit(,16)
  cookie=0x0, duration=74.133s, table=16, n_packets=0, n_bytes=0, 
idle_age=74, priority=100,metadata=0x1,vlan_tci=0x1000/0x1000 actions=drop
  cookie=0x0, duration=74.133s, table=16, n_packets=0, n_bytes=0, 
idle_age=74, 
priority=100,metadata=0x1,dl_src=01:00:00:00:00:00/01:00:00:00:00:00 
actions=drop
  cookie=0x0, duration=74.133s, table=16, n_packets=138, n_bytes=12852, 
idle_age=0, priority=50,reg6=0x1,metadata=0x1 actions=resubmit(,17)
  cookie=0x0, duration=69.124s, table=16, n_packets=142, n_bytes=13524, 
idle_age=0, priority=50,reg6=0x2,metadata=0x1 actions=resubmit(,17)
  cookie=0x0, duration=74.133s, table=17, n_packets=280, n_bytes=26376, 
idle_age=0, priority=0,metadata=0x1 actions=resubmit(,18)
  cookie=0x0, duration=74.133s, table=18, n_packets=280, n_bytes=26376, 
idle_age=0, priority=0,metadata=0x1 actions=resubmit(,19)
  cookie=0x0, duration=74.133s, table=19, n_packets=12, n_bytes=504, 
idle_age=62, 
priority=100,metadata=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 
actions=load:0xffff->NXM_NX_REG7[],resubmit(,32)
  cookie=0x0, duration=66.013s, table=19, n_packets=126, n_bytes=12348, 
idle_age=0, priority=50,metadata=0x1,dl_dst=52:54:00:69:57:15 
actions=load:0x2->NXM_NX_REG7[],resubmit(,32)
  cookie=0x0, duration=63.028s, table=19, n_packets=128, n_bytes=12488, 
idle_age=0, priority=50,metadata=0x1,dl_dst=52:54:00:ea:62:56 
actions=load:0x1->NXM_NX_REG7[],resubmit(,32)
  cookie=0x0, duration=131.334s, table=32, n_packets=266, n_bytes=25340, 
idle_age=0, priority=0 actions=resubmit(,33)
  cookie=0x0, duration=74.133s, table=33, n_packets=128, n_bytes=12488, 
idle_age=0, priority=100,reg7=0x1,metadata=0x1 
actions=load:0x2->NXM_NX_REG5[],resubmit(,34)
  cookie=0x0, duration=74.134s, table=33, n_packets=12, n_bytes=504, 
idle_age=62, hard_age=69, priority=100,reg7=0xffff,metadata=0x1 
actions=load:0x2->NXM_NX_REG5[],load:0x1->NXM_NX_REG7[],resubmit(,34),load:0x1->NXM_NX_REG5[],load:0x2->NXM_NX_REG7[],resubmit(,34),load:0xffff->NXM_NX_REG7[]
  cookie=0x0, duration=69.125s, table=33, n_packets=126, n_bytes=12348, 
idle_age=0, priority=100,reg7=0x2,metadata=0x1 
actions=load:0x1->NXM_NX_REG5[],resubmit(,34)
  cookie=0x0, duration=74.134s, table=34, n_packets=12, n_bytes=504, 
idle_age=62, priority=100,reg6=0x1,reg7=0x1,metadata=0x1 actions=drop
  cookie=0x0, duration=69.125s, table=34, n_packets=0, n_bytes=0, 
idle_age=69, priority=100,reg6=0x2,reg7=0x2,metadata=0x1 actions=drop
  cookie=0x0, duration=131.335s, table=34, n_packets=261, n_bytes=25130, 
idle_age=0, priority=0 
actions=load:0->NXM_NX_REG0[],load:0->NXM_NX_REG1[],load:0->NXM_NX_REG2[],load:0->NXM_NX_REG3[],load:0->NXM_NX_REG4[],resubmit(,48)
  cookie=0x0, duration=74.134s, table=48, n_packets=261, n_bytes=25130, 
idle_age=0, priority=0,metadata=0x1 actions=resubmit(,49)
  cookie=0x0, duration=74.134s, table=49, n_packets=261, n_bytes=25130, 
idle_age=0, priority=0,metadata=0x1 actions=resubmit(,50)
  cookie=0x0, duration=74.134s, table=50, n_packets=7, n_bytes=294, 
idle_age=62, 
priority=100,metadata=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 
actions=resubmit(,64)
  cookie=0x0, duration=74.134s, table=50, n_packets=128, n_bytes=12488, 
idle_age=0, priority=50,reg7=0x1,metadata=0x1 actions=resubmit(,64)
  cookie=0x0, duration=69.125s, table=50, n_packets=126, n_bytes=12348, 
idle_age=0, priority=50,reg7=0x2,metadata=0x1 actions=resubmit(,64)
  cookie=0x0, duration=74.134s, table=64, n_packets=128, n_bytes=12488, 
idle_age=0, priority=100,reg7=0x1,metadata=0x1 actions=output:1
  cookie=0x0, duration=69.125s, table=64, n_packets=133, n_bytes=12642, 
idle_age=0, priority=100,reg7=0x2,metadata=0x1 actions=output:2

The ping between the VMs should work again .. and this:

[tutorial]# ovs-appctl ofproto/trace br-int 
in_port=1,dl_src=52:54:00:69:57:15,dl_dst=52:54:00:ea:62:56 -generate
Bridge: br-int
Flow: 
in_port=1,vlan_tci=0x0000,dl_src=52:54:00:69:57:15,dl_dst=52:54:00:ea:62:56,dl_type=0x0000

-- lots of stuff deleted --

Final flow: 
reg5=0x1,reg6=0x1,reg7=0x2,metadata=0x1,in_port=1,vlan_tci=0x0000,dl_src=52:54:00:69:57:15,dl_dst=52:54:00:ea:62:56,dl_type=0x0000
Megaflow: 
recirc_id=0,in_port=1,vlan_tci=0x0000/0x1000,dl_src=00:00:00:00:00:00/01:00:00:00:00:00,dl_dst=52:54:00:ea:62:56,dl_type=0x0000
Datapath actions: 3






More information about the discuss mailing list