[PVE-User] Infiniband Mellanox cards and DDR

[Fabrizio Cuseo]

I have found that the 400EX is a SDR card, while the 400EX-D is a DDR card. I don't know if is a firmware problem.


http://sup.xenya.si/sup/info/voltaire/HCAwebfinal.pdf

HCA 400Ex/Ex-F
• Dual port 4X (10 Gbps) InfiniBand PCI-Express low 
profile host channel adapter
HCA 400Ex-D
• Dual port 4X DDR (20 Gbps) InfiniBand PCI-Express 
low profile host channel adapter
HCA 400
• Dual port 4X (10 Gbps) InfiniBand PCI/PCI-X low 
profile host channel adapter




----- Messaggio originale -----
Da: "Michael Rasmussen" <mir at miras.org>
A: "Fabrizio Cuseo" <f.cuseo at panservice.it>, "pve-user" <pve-user at pve.proxmox.com>
Inviato: Martedì, 7 luglio 2015 12:01:03
Oggetto: Re: [PVE-User] Infiniband Mellanox cards and DDR

AFAIK this card is a SDR card so DDR is not possible. 


On July 7, 2015 11:46:41 AM CEST, Fabrizio Cuseo <f.cuseo at panservice.it> wrote: 

Hallo. 

I am testing a Mellanox card (IBM Voltaire HCA 400Ex-D) with a cisco 4x DDR infiniband switch. 

The problem I have is that the cards (that seems to be 4X DDR) have a 10Gbit (SDR) link. 

--- 
03:00.0 InfiniBand: Mellanox Technologies MT25208 [InfiniHost III Ex] (rev 20) 
--- 

root at pve1:/etc/apt/sources.list.d# ibstat 
CA 'mthca0' 
CA type: MT25208 
Number of ports: 2 
Firmware version: 5.3.0 
Hardware version: 20 
Node GUID: 0x0008f104039814d0 
System image GUID: 0x0008f104039814d3 
Port 1: 
State: Active 
Physical state: LinkUp 
Rate: 10 
Base lid: 6 
LMC: 0 
SM lid: 2 
Capability mask: 0x02510a68 
Port GUID: 0x0008f104039814d1 
Link laye
 r:
InfiniBand 

-------- 

root at pve1:/etc/apt/sources.list.d# ibportstate 6 0 
CA PortInfo: 
# Port info: Lid 6 port 0 
LinkState:.......................Active 
PhysLinkState:...................LinkUp 
Lid:.............................6 
SMLid:...........................2 
LMC:.............................0 
LinkWidthSupported:..............1X or 4X 
LinkWidthEnabled:................1X or 4X 
LinkWidthActive:.................4X 
LinkSpeedSupported:..............2.5 Gbps 
LinkSpeedEnabled:................2.5 Gbps 
LinkSpeedActive:.................2.5 Gbps 
Mkey:............................<not displayed> 
MkeyLeasePeriod:.................15 
ProtectBits:.....................0 

----- 


If I try to set the speed at DDR, I have this error: 

root at pve1:/etc/apt/sources.list.d# ibportstate 6 0 speed 2 
Initial CA PortInfo: 
# Port info: Lid 6 port 0 
LinkState:.......................Active 
PhysLinkState:...................LinkUp 
Lid:.............................6 
SMLid:...........................2 
LMC:.............................0 
LinkWidthSupported:..............1X or 4X 
LinkWidthEnabled:................1X or 4X 
LinkWidthActive:.................4X 
LinkSpeedSupported:..............2.5 Gbps 
LinkSpeedEnabled:................2.5 Gbps 
LinkSpeedActive:.................2.5 Gbps 
Mkey:............................<not displayed> 
MkeyLeasePeriod:.................15 
ProtectBits:.....................0 
ibportstate: iberror: failed: smp set portinfo failed 


---- 

The switch reported this card in infiniband topology: 

MT25218 InfiniHostEx Mellanox Technologies 


Someone have tested the cards ? 

Fabrizio Cuseo 








-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity. 
---- This mail was virus scanned and spam checked before delivery. This mail is also DKIM signed. See header dkim-signature. 

-- 
---
Fabrizio Cuseo - mailto:f.cuseo at panservice.it
Direzione Generale - Panservice InterNetWorking
Servizi Professionali per Internet ed il Networking
Panservice e' associata AIIP - RIPE Local Registry
Phone: +39 0773 410020 - Fax: +39 0773 470219
http://www.panservice.it  mailto:info at panservice.it
Numero verde nazionale: 800 901492