[ovs-dev] ib0 as a port on a bridge?

christopher barry cbarry at rjmetrics.com
Wed Sep 19 21:57:41 UTC 2012


On Thu, 2012-09-20 at 05:51 +1000, Joseph Glanville wrote:
> Hi
> 
> On 20 September 2012 04:01, Stephen Hemminger <shemminger at vyatta.com> wrote:
> > On Wed, 19 Sep 2012 13:57:39 -0400
> > christopher barry <cbarry at rjmetrics.com> wrote:
> >
> >> On Tue, 2012-09-18 at 23:49 -0400, christopher barry wrote:
> >> > if this is not the best list for user questions, please send me to
> >> > wherever the best place is for that.
> >> >
> >> > I'm building an opennebula cloud, and I have openvswitch master running
> >> > on squeeze kvm host nodes with 3.5.4 as bridge br0 with a single port
> >> > (eth0 at the moment). Each host node also has a Mellanox QDR HCA for
> >> > storage connectivity. I've configured an ipoib interface (ib0) as well
> >> > on each node. eth0 and ib0 are on separate subnets.
> >> >
> >> > Thinking it would be faster to use ib0 as the bridge port, rather than
> >> > eth0, I reconfigured all the bridges in /etc/network/interfaces to use
> >> > ib0, removing eth0 as a port in the file, and bounced the cluster.
> >> >
> >> > couple of things happened:
> >> > * ib0 did get added as a port, but eth0 remained unexpectedly (I had to
> >> > del-port it) (that may be by design though, dunno)
> >> > * I could not pass traffic through br0 when ib0 was a port.
> >> > * I could ping the bridge with ib0 from it's owning host, but not from
> >> > outside. (assuming it's just br0 answering, and ib0 is non-functional)
> >> >
> >> > I'm confused about what is happening. By itself, the ib0 is functioning
> >> > fine, and the bridge works fine with eth0. Is it because ib0 is not a
> >> > real ethernet nic, per se, that I can't make this work? Was it dumb to
> >> > even try it? Or, am I just missing something obvious?
> >> >
> >> > Thanks for any insight,
> >> > -C
> >> >
> >>
> >> No takers?
> >
> > Isn't Infiniband and Ethernet addressing different?
> > _______________________________________________
> > dev mailing list
> > dev at openvswitch.org
> > http://openvswitch.org/mailman/listinfo/dev
> 
> Your issue here is ib0 is an IPoIB device, it can only pass IP packets.
> OVS is a Layer 2 virtual switch (similar to the linux bridge module
> PS: Thanks Stephen. :P) that only forwards Ethernet frames.
> 
> This leaves you with a few options, listed in basically order of
> difficulty and closeness to Ethernet.
> Almost all of them require writing your own network hooks for OpenNebula.
> 
> a) Use routed interfaces
> b) Build a brouter (look up ebtables/iptables for how to do this)
> c) Tunnel Ethernet over IP (using GRE, L2TP etc) and bridge that with OVS
> d) Use Mellanox BridgeX devices + the eoib module from Mellanox OFED package
> d) Implement your own Ethernet on IB encapsulation
> 
> Sadly there is currently not much OSS work going on to support this
> and the state of converged Infiniband clouds is really hurting for it.
> Mellanox has proposed an eIPoIB device driver to LKML but in it's
> current state/design it has no chance of being merged.
> 
> Joseph.
> 

Excellent explanation. Thank You.

-C




More information about the dev mailing list