[ovs-discuss] ipoib with open vswitch

[Jesse Gross]

On Tue, Apr 20, 2010 at 11:22 AM, Trygve Sanne Hardersen <
trygve at hypobytes.com> wrote:
>
> I'm running XCP on 2 physical machines configured as a pool. Each box
> has two normal Ethernet NICs (with 6-byte HW addresses) and a single
> Infiniband NIC (with a 20-byte HW address).
>
> My original plan was to run the inter-vm network over the IB link, but
> this requires the OVS or Linux bridge to work. Instead I'm now running
> the XCP management interface on the IB link. This makes XCP create a
> bridge interface "brib0" with a broken OVS connection to "ib0". When I
> manually configure ib0 and route traffic through this interface using
> the Linux routing table communication works and I have very fast
> communication between the hosts in the pool.
>

Getting the VMs in a bridge with an Infiniband device requires more than
just supporting multiple MAC address lengths.  The VMs are speaking Ethernet
and what you want would require bidirectional translation between Ethernet
and Infiniband protocols.

Ethernet and Infiniband may appear to be the same to an application because
in this case they are both running IP and the application is using IP or
some protocol on top of IP.  However, at the L2 protocol level they are not
the same and any piece of software that operates at L2, such as OVS or the
virtual Ethernet devices of the guest VMs, will not just work.


>
> The problem I'm now facing is that XCP runs the VM consoles on the
> brib0 interface but I'm unable to route traffic from one of the
> Ethernet NICs to this interface. I've tried to do this using iptables
> but it seems the lib_REDIRECT kernel module is unavailable on XCP. The
> simplest solution would probably be to run the XCP consoles on a
> different NIC than the management interface but I've been unable to
> configure this so far. Routing traffic to the interface is probably
> also possible but I've not achieved this yet.
>

Questions about configuring XCP are really better suited to their mailing
lists, though I wouldn't be surprised if dividing up the management
interface in the way that you are looking for isn't supported.

Personally, I wouldn't try to put the Infiniband device in a bridge at all -
it won't work so there isn't much point.  Just configure the device
manually, as Ben suggested.  You can create static routing entries in either
dom0 or the domUs so that the traffic you do want to do go over the
Infiniband device is directed there.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20100420/5077168f/attachment-0001.html>