Difference between revisions of "Raid controller"

From Proxmox VE
Jump to navigation Jump to search
m (→‎Smart Array (using in-kernel cciss driver): adopt new GPG key, as per https://forum.proxmox.com/threads/editing-the-wiki.77176/#post-348125)
 
(30 intermediate revisions by 10 users not shown)
Line 1: Line 1:
 
work in progress.
 
work in progress.
  
−
=Introduction=
+
== Introduction ==
−
A fast and reliable storage controller is one of the most important parts of a Proxmox VE server. This article lists some good working hardware raid controllers and some information about the used configurations. If you wonder how to figure out which Controller you have, log into your Proxmox server and type 'lspci' on the commandline. Then look for a line with RAID Controller or similar.
+
A fast and reliable storage controller is one of the most important parts of a Proxmox VE server. This article lists some hardware raid controllers that are known to work well and some information configuring them. You can use the lspci command in the Proxmox command line. Look for a line with RAID Controller or similar. For a RAID controller to be supported, it must be a "real" hardware controller rather than an embedded or "fake" RAID. Embedded controllers are not supported in Proxmox, and if they do work, you are doing so at your own risk.
 +
 
 +
== Performance ==
 +
Paramount feature for real raid controller if you want high performance is an on board cache and "write back" mode enabled. This way the OS doesn't have to wait until data is physically written to disk, since they are immediately written to the cache and the controller will take care of finishing the subsequent write to the disk(s). If you don't want to loose your data in case of black-out, you need a battery backup (BBU). That ensures that any pending writes during a blackout can finish being saved as soon as power is restored. Some controllers support using an SSD so that a BBU isn't so important, but some SSDs do not deal with power loss well, do your research before relying on this solution.
 +
 
 +
A RAID controller using write-through instead of write-back behaves very poorly. You can use the command pveperf to have a general idea of performance. Look at the FSYNCS/SECOND value.
 +
Here is a real world example of the performance you can expect.
  
−
We are talking about "real" hardware controller, not the so called "fake" ones, the ones that usually are "embedded" in the motherboard. Such controller in fact don't have dedicated hardware but are software - bios/firmware based, so not reliable and not good performer (and are NOT supported by Proxmox, if they work is only on your own risk).
 
−
Paramount feature for real raid controller if you want high performance is a on board cache and "write back" mode enabled. This way the OS had not to wait until writes are physically done on hard disk, they are immediately wrote on the cache and the controller will take care of subsequent write.
 
−
If you don't want to loose your data in case of black-out, you need the so called "BBU", a battery backup on the controller that keeps data that are not yet saved on disk available until power comes back.
 
−
In short, you need a RAID controller with cache and write-back, and you can safely enable write-back only if you have BBU (or similar solution, like a SSD cache instead of RAM one).
 
−
Remember that a RAID controller with write-trough instead of write-back behaves really really badly
 
−
With the command pveverf you can have a rough idea of the performance.
 
−
Here some values just to have an idea (the FSYNCS/SECOND value of pveperf)
 
 
:Single SATA WD 400GB: 1360.17
 
:Single SATA WD 400GB: 1360.17
−
:3 x 15K rpm sas RAID5 with write-through: 159.03
+
:3 x 15K rpm sas RAID5 with write-through: 159.03 (YES, only 159!)
 
:same as above but with write-back enabled: 3133.45
 
:same as above but with write-back enabled: 3133.45
  
 +
The configuration of the RAID array can also have a major impact on performance.  RAID 5 should not be used with modern hard drives because rebuild times are long enough that a second drive could fail causing the entire array to be lost.  If you were planning on using RAID 5, consider using RAID 6 instead. That being said, RAID 6 should be considered slow. If you are looking for performance, look for something that stripes you disks.  RAID 10 or 60 is generally considered a good balance of performance to redundancy with RAID 60 providing additional layers of redundancy.
 +
 +
== General info ==
 +
http://hwraid.le-vert.net/ has info about some vendors raid controllers
 +
* 3Ware cards
 +
* LSI cards
 +
* Adaptec cards
 +
* HP/Compaq SmartArray
  
−
=3Ware=
+
the site also has repositores for debian/ubuntu based systems (http://hwraid.le-vert.net/wiki/DebianPackages) where one can find tools and packages not available directly from vendors, id needed.
 +
 
 +
== 3Ware ==
  
 
'''9690SA SAS/SATA-II RAID PCIe (rev 01)'''
 
'''9690SA SAS/SATA-II RAID PCIe (rev 01)'''
Line 22: Line 31:
 
This Adapter is working well under Proxmox. '''Attention:''' make sure you have write cache enabled and use a BBU! Without write cache the vms seem to 'hang' or 'freeze' sometimes. Performance give about 50MB/s on a Raid 5 with 3 HDD's a 1TB
 
This Adapter is working well under Proxmox. '''Attention:''' make sure you have write cache enabled and use a BBU! Without write cache the vms seem to 'hang' or 'freeze' sometimes. Performance give about 50MB/s on a Raid 5 with 3 HDD's a 1TB
  
−
== 3dm2 Management tool ==
+
=== 3dm2 Management tool ===
 
:debs available from http://jonas.genannt.name
 
:debs available from http://jonas.genannt.name
  
Line 45: Line 54:
 
: Change password ,  allow remote access,  change default port . add NAT access thru firewall. ...
 
: Change password ,  allow remote access,  change default port . add NAT access thru firewall. ...
  
−
=Adaptec=
+
== Adaptec ==
−
All Adaptec controllers with BBU unit are known to perform well. If possible, take one with "Zero-Maintenance Cache Protection", e.g. Adaptec 5805Z.
+
All Adaptec controllers with BBU unit are known to perform well. If possible, take one with "Zero-Maintenance Cache Protection", e.g. Adaptec 5805Z or 6405/6805 with Adaptec Flash Module 600 (AFM-600)
  
−
== Confirmed==
+
IMPORTANT NOTE:
 +
Adaptec 5405/5805 does NOT work with newer UEFI-BIOS Boards!! (This is true for almost all P67,H67,Z68 Boards)
 +
(Official Adaptec Statement: http://ask.adaptec.com/scripts/adaptec_tic.cfg/php.exe/enduser/std_adp.php?p_faqid=17087&p_created=1305289854&p_topview=1)
 +
 
 +
=== Confirmed ===
 
'''Adaptec 2405'''
 
'''Adaptec 2405'''
 
Controller completely working (RAID10), out of the box. But be aware, it has no possibility to extend it with an BBU (Battery Backup Unit).
 
Controller completely working (RAID10), out of the box. But be aware, it has no possibility to extend it with an BBU (Battery Backup Unit).
  
−
==Configuration==
+
=== Management tools ===
−
Raid 10
+
See [[Adaptec_Storage_Manager]] and [[Adaptec maxView Storage Manager]]
  
−
==Management tools==
+
== Areca ==
−
See [[Adaptec_Storage_Manager]]
 
−
 
 
−
=Areca=
 
 
ARC-1210, ARC-1212 and ARC-1222 works very well with good speed  
 
ARC-1210, ARC-1212 and ARC-1222 works very well with good speed  
  
−
=HighPoint=
+
== HighPoint ==
 
HighPoint 3510 - RAID 10. Works very well.
 
HighPoint 3510 - RAID 10. Works very well.
  
  
−
=Compaq / HP=
+
== Compaq / HP ==
 
==Smart Array (using in-kernel cciss driver)==
 
==Smart Array (using in-kernel cciss driver)==
 +
 
* 6i
 
* 6i
 
* 400i
 
* 400i
 +
* P400 PCI-E
 +
* P410 PCI-E
 +
* P600 PCI-X
 +
* P800 PCI-E
 +
* P812 PCI-E
 +
 +
The P600 and P800 RAID controllers work well with ProxMox 3.x and 4.x.  Both controllers are capable of accessing external SAS storage using, for example, HP MSA50 enclosures.
 +
 +
HP SmaryArray controllers can be managed using the CLI tool (hpssacli) available from HP.
 +
 +
To install hpssacli, run the commands below:
 +
wget https://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub -O /tmp/proliant.gpg
 +
apt-key add /tmp/proliant.gpg
 +
echo -e "deb http://downloads.linux.hpe.com/SDR/repo/mcp/ buster/current non-free" > /etc/apt/sources.list.d/proliant.sources.list
 +
apt update && apt install ssacli
 +
 +
== Dell ==
 +
 +
== PERC/LSI (native in-kernel driver) ==
 +
 +
* PERC 5/i
 +
* PERC 5/E
 +
* PERC 6/i
 +
* PERC 6/E
 +
 +
Above cards work well in PROXMOX 2.3, 3.0 and 3.1.
 +
 +
=== Monitoring SMART status of disks in array ===
 +
Monitoring your disks is important so that you can know to replace any disks that may be failing.  The following is how you can manually fetch the status of a disk in array. The first time you want to monitor the smart status of your disks, you must install the smartmontools package.
 +
 +
:install the smartmon tools :
 +
apt-get install smartmontools
 +
 +
After that, you can use smartctl to fetch the status of any particular disk in the array.  Just change the number after the comma (,) to indicate which drive in the array you care about.  The following example checks the second disk. Changing the 1 to a 0 would check the first disk.
 +
 +
:fetch the status of a disk :
 +
smartctl -a -d megaraid,1 /dev/sda
 +
 +
== LSI ==
 +
 +
== Other Vendors ==
 +
Some server vendor can use LSI (and other vendors) RAID controllers in their servers, renamed, examples:
 +
* IBM ServeRAID M5015 SAS/SATA Controller (should be some kind of MegaRAID SAS 2108, ie: MegaRAID SAS 9260-8i)
 +
 +
See also http://www.redbooks.ibm.com/abstracts/tips0054.html
 +
and http://www.lsi.com/channel/products/storagecomponents/Pages/MegaRAIDSAS9260-8i.aspx
 +
 +
In those cases, LSI tools should work anyway.
 +
 +
=== HP Branded ===
 +
* 3041E SAS/SATA 3GB 4-port RAID Card (EH417AA) works.  Note: This controller's feature set does not include cache and write-back or battery/flash back up.  Therefore performance is not improved.
 +
 +
=== Dell Branded ===
 +
Cards listed below are supported with native kernel drivers.
 +
 +
* SAS2208 SAS/SATA 6GB - PERC H710, H710P and H810
 +
* SAS2008 SAS/SATA 6GB - PERC H310
  
 +
== Management/Info Tools ==
 +
see: http://hwraid.le-vert.net/wiki/LSI
  
 +
* SAS/SATA controllers
 +
*: Working tools from http://hwraid.le-vert.net repositories (see at the '''General info''' section):
 +
** megacli
 +
**: Tool to read and setup LSI Logic MegaRAID SAS HW RAID HBAs.
 +
**: Homepage: http://www.lsi.com/channel/support/products/Pages/MegaRAIDSAS9285-8e.aspx
 +
** megactl
 +
**: This package contains both megactl and megasasctl tools that can be used to query LSI's MegaRAID adapters status.
 +
** megaclisas-status
 +
**: Get RAID status out of LSI MegaRAID SAS HW RAID controllers
 +
**: The megacli-status software is a query tool to access the running configuration and status of LSI MegaRAID SAS HBAs.
 +
**: It uses LSI MegaCli proprietary command line tool.
  
 
[[Category: HOWTO]] [[Category: Hardware]]
 
[[Category: HOWTO]] [[Category: Hardware]]

Latest revision as of 13:35, 5 November 2020

work in progress.

Introduction

A fast and reliable storage controller is one of the most important parts of a Proxmox VE server. This article lists some hardware raid controllers that are known to work well and some information configuring them. You can use the lspci command in the Proxmox command line. Look for a line with RAID Controller or similar. For a RAID controller to be supported, it must be a "real" hardware controller rather than an embedded or "fake" RAID. Embedded controllers are not supported in Proxmox, and if they do work, you are doing so at your own risk.

Performance

Paramount feature for real raid controller if you want high performance is an on board cache and "write back" mode enabled. This way the OS doesn't have to wait until data is physically written to disk, since they are immediately written to the cache and the controller will take care of finishing the subsequent write to the disk(s). If you don't want to loose your data in case of black-out, you need a battery backup (BBU). That ensures that any pending writes during a blackout can finish being saved as soon as power is restored. Some controllers support using an SSD so that a BBU isn't so important, but some SSDs do not deal with power loss well, do your research before relying on this solution.

A RAID controller using write-through instead of write-back behaves very poorly. You can use the command pveperf to have a general idea of performance. Look at the FSYNCS/SECOND value. Here is a real world example of the performance you can expect.

Single SATA WD 400GB: 1360.17
3 x 15K rpm sas RAID5 with write-through: 159.03 (YES, only 159!)
same as above but with write-back enabled: 3133.45

The configuration of the RAID array can also have a major impact on performance. RAID 5 should not be used with modern hard drives because rebuild times are long enough that a second drive could fail causing the entire array to be lost. If you were planning on using RAID 5, consider using RAID 6 instead. That being said, RAID 6 should be considered slow. If you are looking for performance, look for something that stripes you disks. RAID 10 or 60 is generally considered a good balance of performance to redundancy with RAID 60 providing additional layers of redundancy.

General info

http://hwraid.le-vert.net/ has info about some vendors raid controllers

  • 3Ware cards
  • LSI cards
  • Adaptec cards
  • HP/Compaq SmartArray

the site also has repositores for debian/ubuntu based systems (http://hwraid.le-vert.net/wiki/DebianPackages) where one can find tools and packages not available directly from vendors, id needed.

3Ware

9690SA SAS/SATA-II RAID PCIe (rev 01)

This Adapter is working well under Proxmox. Attention: make sure you have write cache enabled and use a BBU! Without write cache the vms seem to 'hang' or 'freeze' sometimes. Performance give about 50MB/s on a Raid 5 with 3 HDD's a 1TB

3dm2 Management tool

debs available from http://jonas.genannt.name
install key
 wget -O - http://jonas.genannt.name/debian/jonas_genannt.pub | apt-key add -
add to /etc/apt/sources.list :
deb http://jonas.genannt.name/debian lenny restricted
install
aptitude update
aptitude install 3ware-3dm2-binary 3ware-cli-binary
start service
/etc/init.d/3dm2 start
note firefox does not display all of the screen correctly [for anyways for last 3-4 years ]. so i usually use konqueror
connect - login as admin , default password is 3ware
https://ipaddress:888
configure
Change password , allow remote access, change default port . add NAT access thru firewall. ...

Adaptec

All Adaptec controllers with BBU unit are known to perform well. If possible, take one with "Zero-Maintenance Cache Protection", e.g. Adaptec 5805Z or 6405/6805 with Adaptec Flash Module 600 (AFM-600)

IMPORTANT NOTE: Adaptec 5405/5805 does NOT work with newer UEFI-BIOS Boards!! (This is true for almost all P67,H67,Z68 Boards) (Official Adaptec Statement: http://ask.adaptec.com/scripts/adaptec_tic.cfg/php.exe/enduser/std_adp.php?p_faqid=17087&p_created=1305289854&p_topview=1)

Confirmed

Adaptec 2405 Controller completely working (RAID10), out of the box. But be aware, it has no possibility to extend it with an BBU (Battery Backup Unit).

Management tools

See Adaptec_Storage_Manager and Adaptec maxView Storage Manager

Areca

ARC-1210, ARC-1212 and ARC-1222 works very well with good speed

HighPoint

HighPoint 3510 - RAID 10. Works very well.


Compaq / HP

Smart Array (using in-kernel cciss driver)

  • 6i
  • 400i
  • P400 PCI-E
  • P410 PCI-E
  • P600 PCI-X
  • P800 PCI-E
  • P812 PCI-E

The P600 and P800 RAID controllers work well with ProxMox 3.x and 4.x. Both controllers are capable of accessing external SAS storage using, for example, HP MSA50 enclosures.

HP SmaryArray controllers can be managed using the CLI tool (hpssacli) available from HP.

To install hpssacli, run the commands below: 

wget https://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub -O /tmp/proliant.gpg
apt-key add /tmp/proliant.gpg
echo -e "deb http://downloads.linux.hpe.com/SDR/repo/mcp/ buster/current non-free" > /etc/apt/sources.list.d/proliant.sources.list
apt update && apt install ssacli

Dell

PERC/LSI (native in-kernel driver)

  • PERC 5/i
  • PERC 5/E
  • PERC 6/i
  • PERC 6/E

Above cards work well in PROXMOX 2.3, 3.0 and 3.1.

Monitoring SMART status of disks in array

Monitoring your disks is important so that you can know to replace any disks that may be failing. The following is how you can manually fetch the status of a disk in array. The first time you want to monitor the smart status of your disks, you must install the smartmontools package.

install the smartmon tools :
apt-get install smartmontools

After that, you can use smartctl to fetch the status of any particular disk in the array. Just change the number after the comma (,) to indicate which drive in the array you care about. The following example checks the second disk. Changing the 1 to a 0 would check the first disk.

fetch the status of a disk :
smartctl -a -d megaraid,1 /dev/sda

LSI

Other Vendors

Some server vendor can use LSI (and other vendors) RAID controllers in their servers, renamed, examples:

  • IBM ServeRAID M5015 SAS/SATA Controller (should be some kind of MegaRAID SAS 2108, ie: MegaRAID SAS 9260-8i)

See also http://www.redbooks.ibm.com/abstracts/tips0054.html and http://www.lsi.com/channel/products/storagecomponents/Pages/MegaRAIDSAS9260-8i.aspx

In those cases, LSI tools should work anyway.

HP Branded

  • 3041E SAS/SATA 3GB 4-port RAID Card (EH417AA) works. Note: This controller's feature set does not include cache and write-back or battery/flash back up. Therefore performance is not improved.

Dell Branded

Cards listed below are supported with native kernel drivers.

  • SAS2208 SAS/SATA 6GB - PERC H710, H710P and H810
  • SAS2008 SAS/SATA 6GB - PERC H310

Management/Info Tools

see: http://hwraid.le-vert.net/wiki/LSI

  • SAS/SATA controllers
    Working tools from http://hwraid.le-vert.net repositories (see at the General info section):
    • megacli
      Tool to read and setup LSI Logic MegaRAID SAS HW RAID HBAs.
      Homepage: http://www.lsi.com/channel/support/products/Pages/MegaRAIDSAS9285-8e.aspx
    • megactl
      This package contains both megactl and megasasctl tools that can be used to query LSI's MegaRAID adapters status.
    • megaclisas-status
      Get RAID status out of LSI MegaRAID SAS HW RAID controllers
      The megacli-status software is a query tool to access the running configuration and status of LSI MegaRAID SAS HBAs.
      It uses LSI MegaCli proprietary command line tool.
Retrieved from "https://pve.proxmox.com/mediawiki/index.php?title=Raid_controller&oldid=10862"