[ovs-discuss] Network issue with spanning-tree

Daniel Dehennin daniel.dehennin at baby-gnu.org
Fri Apr 24 20:31:14 UTC 2015


Hello,

We are experimenting network issues each time we start a new VM.

The other VMs have their network blocked until I see “a topology change
is detected” in Open vSwitch logs.

Do you have any advice for such configuration?

Here is our setup:

We are using Open vSwitch for networking in our OpenNebula setup[1]:

- 4 physical servers (corosync+pacemaker+GFS2 => STONITH)

- Ubuntu Trusty Tahr 14.04.2 (Open vSwitch 2.0.2-0ubuntu0.14.04.1)

- internal bridges “in-br” connected between nodes with vxlan tunnels,
  with a loop and spanning tree

- User VM on their own VLAN (1 VLAN per user)

- OpenNebula manages flows with ovs-ofctl, here are the logs when
  starting a VM:
  #+begin_src text
  Sat Mar  7 15:21:37 2015 [Z0][DiM][I]: New VM state is ACTIVE.
  Sat Mar  7 15:21:37 2015 [Z0][LCM][I]: New VM state is PROLOG.
  Sat Mar  7 15:21:37 2015 [Z0][LCM][I]: New VM state is BOOT
  Sat Mar  7 15:21:37 2015 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/14/deployment.0
  Sat Mar  7 15:21:37 2015 [Z0][VMM][I]: ExitCode: 0
  Sat Mar  7 15:21:37 2015 [Z0][VMM][I]: Successfully execute network driver operation: pre.
  Sat Mar  7 15:21:38 2015 [Z0][VMM][I]: ExitCode: 0
  Sat Mar  7 15:21:38 2015 [Z0][VMM][I]: Successfully execute virtualization driver operation: deploy.
  Sat Mar  7 15:21:38 2015 [Z0][VMM][I]: post: Executed "sudo ovs-vsctl set Port vnet18 tag=20".
  Sat Mar  7 15:21:38 2015 [Z0][VMM][I]: post: Executed "sudo ovs-ofctl add-flow in-br in_port=17,dl_src=02:03:00:00:00:00,priority=40000,actions=normal".
  Sat Mar  7 15:21:38 2015 [Z0][VMM][I]: post: Executed "sudo ovs-ofctl add-flow in-br in_port=17,priority=39000,actions=drop".
  Sat Mar  7 15:21:38 2015 [Z0][VMM][I]: ExitCode: 0
  Sat Mar  7 15:21:38 2015 [Z0][VMM][I]: Successfully execute network driver operation: post.
  Sat Mar  7 15:21:38 2015 [Z0][LCM][I]: New VM state is RUNNING
  #+end_src

A picture is better than thousand words:

#+begin_src picture
   ------+---------------------+---------------------+---------------------+------- physical network (192.168.1.0/24)
         |.1                   |.2                   |.3                   |.4
  +------+------+       +------+------+       +------+------+       +------+------+
  |      |      |       |      |      |       |      |      |       |      |      |
  | +----+----+ |       | +----+----+ |       | +----+----+ |       | +----+----+ |
  | |         | |       | |         | |       | |         | |       | |         | |
  | |  eth0   | |       | |  eth0   | |       | |  eth0   | |       | |  eth0   | |
  | |         | |       | |         | |       | |         | |       | |         | |
  | +---------+ |       | +---------+ |       | +---------+ |       | +---------+ |
  |             |       |             |       |             |       |             |
  |             |       |             |       |             |       |             |
  |             |       |             |       |             |       |             |
  |             |       |             |       |             |       |             |
  | +---------+ |       | +---------+ |       | +---------+ |       | +---------+ |
  | |         | | vxlan | |         | | vxlan | |         | | vxlan | |         | |
  | |  in-br  +-+-------+-+  in-br  +-+-------+-+  in-br  +-+-------+-+  in-br  +-+---+
  | |         | |       | |         | |       | |         | |       | |         | |   |
  | +----+----+ |       | +---------+ |       | +---------+ |       | +---------+ |   |
  |      |      |       |             |       |             |       |             |   |
  +------+------+       +-------------+       +-------------+       +-------------+   |
         |                                                                            |
         |                              vxlan                                         |
         +----------------------------------------------------------------------------+
#+end_src

I setup the isolated bridges and the VXLAN tunnels between hosts simply with:

#+begin_src sh
  #!/bin/bash

  [[ "${DEBUG}" =~ ^[tT]([rR][uU][eE])?$ ]] && DRY=echo
  [[ "${DRY_RUN}" =~ ^[tT]([rR][uU][eE])?$ ]] && DRY=:

  NET=192.168.1
  for hostip in {1..4}
  do
      echo "Creating isolated bridge"
      ${DRY} ssh root@${NET}.${hostip} \
             ovs-vsctl add-br in-br

      # Enable STP first
      echo "Enable spanning tree on node ${hostip}"
      ${DRY} ssh root@${NET}.${hostip} ovs-vsctl set Bridge in-br stp_enable=true

      PREV_NODE_ID=$(( hostip - 1 ))
      # Loop on node4
      [ ${PREV_NODE_ID} -eq 0 ] && PREV_NODE_ID=4

      NEXT_NODE_ID=$(( (hostip+1) % 5 ))
      # Loop on node1
      [ ${NEXT_NODE_ID} -eq 0 ] && NEXT_NODE_ID=1


      INTERFACE_TO_PREV="vx-to-node${PREV_NODE_ID}"
      INTERFACE_TO_NEXT="vx-to-node${NEXT_NODE_ID}"

      # Link to previous node switch
      echo "Connect node ${hostip} to node ${PREV_NODE_ID}"
      ${DRY} ssh root@${NET}.${hostip} \
          ovs-vsctl add-port in-br ${INTERFACE_TO_PREV} -- \
              set interface ${INTERFACE_TO_PREV} type=vxlan \
              options:local_ip=${hostip} \
              remote_ip=${NET}.${PREV_NODE_ID}

      # Link to next node switch
      echo "Connect node ${hostip} to node ${NEXT_NODE_ID}"
      ${DRY} ssh root@${NET}.${hostip} \
          ovs-vsctl add-port in-br ${INTERFACE_TO_NEXT} -- \
              set interface ${INTERFACE_TO_NEXT} type=vxlan \
              options:local_ip=${hostip} \
              options:remote_ip=${NET}.${NEXT_NODE_ID}

      echo
  done
#+end_src

Regards.

Footnotes: 
[1]  http://docs.opennebula.org/4.12/administration/networking/nm.html

-- 
Daniel Dehennin
Récupérer ma clef GPG: gpg --recv-keys 0xCC1E9E5B7A6FE2DF
Fingerprint: 3E69 014E 5C23 50E8 9ED6  2AAD CC1E 9E5B 7A6F E2DF
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 342 bytes
Desc: not available
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20150424/8888cff7/attachment-0002.sig>


More information about the discuss mailing list