[ovs-discuss] ipoib with open vswitch

[Trygve Sanne Hardersen]

Sorry for not getting back to you earlier.

I've now configured XCP without the Infiniband device as you suggested, and
I'm manually routing traffic over this NIC. It seems to work fine.

Many thanks for your help!

Trygve

On Tue, Apr 20, 2010 at 7:48 PM, Jesse Gross <jesse at nicira.com> wrote:

> On Tue, Apr 20, 2010 at 11:22 AM, Trygve Sanne Hardersen <
> trygve at hypobytes.com> wrote:
>>
>> I'm running XCP on 2 physical machines configured as a pool. Each box
>> has two normal Ethernet NICs (with 6-byte HW addresses) and a single
>> Infiniband NIC (with a 20-byte HW address).
>>
>> My original plan was to run the inter-vm network over the IB link, but
>> this requires the OVS or Linux bridge to work. Instead I'm now running
>> the XCP management interface on the IB link. This makes XCP create a
>> bridge interface "brib0" with a broken OVS connection to "ib0". When I
>> manually configure ib0 and route traffic through this interface using
>> the Linux routing table communication works and I have very fast
>> communication between the hosts in the pool.
>>
>
> Getting the VMs in a bridge with an Infiniband device requires more than
> just supporting multiple MAC address lengths.  The VMs are speaking Ethernet
> and what you want would require bidirectional translation between Ethernet
> and Infiniband protocols.
>
> Ethernet and Infiniband may appear to be the same to an application because
> in this case they are both running IP and the application is using IP or
> some protocol on top of IP.  However, at the L2 protocol level they are not
> the same and any piece of software that operates at L2, such as OVS or the
> virtual Ethernet devices of the guest VMs, will not just work.
>
>
>>
>> The problem I'm now facing is that XCP runs the VM consoles on the
>> brib0 interface but I'm unable to route traffic from one of the
>> Ethernet NICs to this interface. I've tried to do this using iptables
>> but it seems the lib_REDIRECT kernel module is unavailable on XCP. The
>> simplest solution would probably be to run the XCP consoles on a
>> different NIC than the management interface but I've been unable to
>> configure this so far. Routing traffic to the interface is probably
>> also possible but I've not achieved this yet.
>>
>
> Questions about configuring XCP are really better suited to their mailing
> lists, though I wouldn't be surprised if dividing up the management
> interface in the way that you are looking for isn't supported.
>
> Personally, I wouldn't try to put the Infiniband device in a bridge at all
> - it won't work so there isn't much point.  Just configure the device
> manually, as Ben suggested.  You can create static routing entries in either
> dom0 or the domUs so that the traffic you do want to do go over the
> Infiniband device is directed there.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20100427/83c39da1/attachment-0001.html>