[PVE-User] Infiniband Mellanox cards and DDR

[Fabrizio Cuseo]

Hallo.

I am testing a Mellanox card (IBM Voltaire HCA 400Ex-D) with a cisco 4x DDR infiniband switch.

The problem I have is that the cards (that seems to be 4X DDR) have a 10Gbit (SDR) link.

---
03:00.0 InfiniBand: Mellanox Technologies MT25208 [InfiniHost III Ex] (rev 20)
---

root at pve1:/etc/apt/sources.list.d# ibstat
CA 'mthca0'
        CA type: MT25208
        Number of ports: 2
        Firmware version: 5.3.0
        Hardware version: 20
        Node GUID: 0x0008f104039814d0
        System image GUID: 0x0008f104039814d3
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 10
                Base lid: 6
                LMC: 0
                SM lid: 2
                Capability mask: 0x02510a68
                Port GUID: 0x0008f104039814d1
                Link layer: InfiniBand

--------

root at pve1:/etc/apt/sources.list.d# ibportstate 6 0
CA PortInfo:
# Port info: Lid 6 port 0
LinkState:.......................Active
PhysLinkState:...................LinkUp
Lid:.............................6
SMLid:...........................2
LMC:.............................0
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps
LinkSpeedEnabled:................2.5 Gbps
LinkSpeedActive:.................2.5 Gbps
Mkey:............................<not displayed>
MkeyLeasePeriod:.................15
ProtectBits:.....................0

-----


If I try to set the speed at DDR, I have this error:

root at pve1:/etc/apt/sources.list.d# ibportstate 6 0 speed 2
Initial CA PortInfo:
# Port info: Lid 6 port 0
LinkState:.......................Active
PhysLinkState:...................LinkUp
Lid:.............................6
SMLid:...........................2
LMC:.............................0
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps
LinkSpeedEnabled:................2.5 Gbps
LinkSpeedActive:.................2.5 Gbps
Mkey:............................<not displayed>
MkeyLeasePeriod:.................15
ProtectBits:.....................0
ibportstate: iberror: failed: smp set portinfo failed


----

The switch reported this card in infiniband topology:

MT25218 InfiniHostEx Mellanox Technologies


Someone have tested the cards ? 

Fabrizio Cuseo








-- 
---
Fabrizio Cuseo - mailto:f.cuseo at panservice.it
Direzione Generale - Panservice InterNetWorking
Servizi Professionali per Internet ed il Networking
Panservice e' associata AIIP - RIPE Local Registry
Phone: +39 0773 410020 - Fax: +39 0773 470219
http://www.panservice.it  mailto:info at panservice.it
Numero verde nazionale: 800 901492