https://pve.proxmox.com/mediawiki/api.php?action=feedcontributions&user=Eyal+Traitel&feedformat=atomProxmox VE - User contributions [en]2024-03-28T14:46:57ZUser contributionsMediaWiki 1.35.11https://pve.proxmox.com/mediawiki/index.php?title=ISCSI_Multipath&diff=8085ISCSI Multipath2016-01-07T12:05:41Z<p>Eyal Traitel: /* Dell */ Added Reduxio HX550 product configuration instructions. Additional details to follow...</p>
<hr />
<div>== Introduction ==<br />
Main purpose of multipath connectivity is to provide redundant access to the storage devices, i.e to have access to the storage device when one or more of the components in a path fail. Another advantage of multipathing is the increased throughput by way of load balancing.<br />
Common example for the use of multipathing is a iSCSI SAN connected storage device. You have redundancy and maximum performance.<br />
<br />
If you use iSCSI, multipath is recommended - this works without configurations on the switches. (If you use NFS or CIFS, use bonding, e.g. 802.ad)<br />
<br />
The connection from the Proxmox VE host through the iSCSI SAN is referred as a path. When multiple paths exists to a storage device (LUN) on a storage subsystem, it is referred as multipath connectivity.<br />
Therefore you need to make sure that you got at least two NICs dedicated for iSCSI, using separate networks (and switches to be protected against switch failures).<br />
<br />
This is a generic how-to. Please consult the storage vendor documentation for vendor specific settings.<br />
<br />
== Update your iSCSI configuration ==<br />
<br />
It is important to start all required iSCSI connections at boot time. You can do that by<br />
setting 'node.startup' to 'automatic'. <br />
<br />
The default 'node.session.timeo.replacement_timeout' is 120 seconds. We recommend using a<br />
much smaller value of 15 seconds instead.<br />
<br />
You can set those values in '/etc/iscsi/iscsid.conf' (defaults). If you are already connected to<br />
the iSCSI target, you need to modify the target specific defaults in '/etc/iscsi/nodes/<TARGET>/<PORTAL>/default'<br />
<br />
A modified 'iscsid.conf' file contains the following lines:<br />
<br />
node.startup = automatic<br />
node.session.timeo.replacement_timeout = 15<br />
<br />
Please configure your iSCSI storage on the GUI if you have not done that already ("Datacenter/Storage: Add iSCSI target").<br />
<br />
== Install multipath tools ==<br />
<br />
The default installation does not include this package, so you first need to install the multipath-tools package:<br />
<br />
# aptitude update<br />
# aptitude install multipath-tools<br />
<br />
== Multipath configuration ==<br />
<br />
Then you need to create the multipath configuration file '/etc/multipath.conf'. You can find details about all setting on the manual page:<br />
<br />
# man multipath.conf<br />
<br />
We recommend to use 'wwid' to identify disks (World Wide Identification). You can use the 'scsi_id' command to get the 'wwid' for a specific device. For example, the following command returns the 'wwid' for device '/dev/sda'<br />
<br />
# /lib/udev/scsi_id -g -u -d /dev/sda<br />
<br />
We normally blacklist all devices, and only allow specific devices using 'blacklist_exceptions':<br />
<br />
<pre><br />
blacklist {<br />
wwid .*<br />
}<br />
<br />
blacklist_exceptions {<br />
wwid "3600144f028f88a0000005037a95d0001"<br />
wwid "3600144f028f88a0000005037a95d0002"<br />
}<br />
</pre><br />
<br />
We also use the 'alias' directive to name the device, but this is optional:<br />
<br />
<pre><br />
multipaths {<br />
multipath {<br />
wwid "3600144f028f88a0000005037a95d0001"<br />
alias mpath0<br />
}<br />
multipath {<br />
wwid "3600144f028f88a0000005037a95d0002"<br />
alias mpath1<br />
}<br />
}<br />
</pre><br />
<br />
And finally you need reasonable defaults. We normally use the following multibus configuration:<br />
<br />
<pre><br />
defaults {<br />
polling_interval 2<br />
path_selector "round-robin 0"<br />
path_grouping_policy multibus<br />
getuid_callout "/lib/udev/scsi_id -g -u -d /dev/%n"<br />
rr_min_io 100<br />
failback immediate<br />
no_path_retry queue<br />
}<br />
</pre><br />
<br />
'''Note''' If you run multipath on V2.3 and before, you need to adapt your multipath.conf - 'path_selector' was called 'selector' called <br />
<br />
Also check your SAN vendor documentation.<br />
<br />
To activate those settings you need to restart they multipath daemon with:<br />
<br />
# service multipath-tools restart<br />
<br />
=== Example multipath.conf ===<br />
# cat /etc/mulitpath.conf<br />
<pre><br />
defaults {<br />
polling_interval 2<br />
path_selector "round-robin 0"<br />
path_grouping_policy multibus<br />
getuid_callout "/lib/udev/scsi_id -g -u -d /dev/%n"<br />
rr_min_io 100<br />
failback immediate<br />
no_path_retry queue<br />
}<br />
blacklist {<br />
wwid *<br />
}<br />
<br />
blacklist_exceptions {<br />
wwid "3600144f028f88a0000005037a95d0001"<br />
}<br />
<br />
multipaths {<br />
multipath {<br />
wwid "3600144f028f88a0000005037a95d0001"<br />
alias nexenta0<br />
}<br />
}<br />
</pre><br />
<br />
== Query device status ==<br />
You can view the status with:<br />
<br />
# multipath -ll<br />
<br />
mpath0 (3600144f028f88a0000005037a95d0001) dm-3 NEXENTA,NEXENTASTOR<br />
size=64G features='1 queue_if_no_path' hwhandler='0' wp=rw<br />
`-+- policy='round-robin 0' prio=2 status=active<br />
|- 5:0:0:0 sdb 8:16 active ready running<br />
`- 6:0:0:0 sdc 8:32 active ready running<br />
<br />
<br />
To get more information about used devices use:<br />
<br />
# multipath -v3<br />
<br />
<br />
== Performance test with fio ==<br />
In order to check the performance, you can use fio.<br />
<br />
Example read test:<br />
fio --filename=/dev/mapper/mpath0 --direct=1 --rw=read --bs=1m --size=20G --numjobs=200 --runtime=60 --group_reporting --name=file1<br />
<br />
== Vendor specific settings ==<br />
Please add vendor specific recommendations here.<br />
<br />
=== Dell ===<br />
<br />
You need to load a Dell specific module scsi_dh_rdac permanently, in order to do this, just edit:<br />
nano /etc/modules<br />
<br />
# /etc/modules: kernel modules to load at boot time.<br />
#<br />
# This file contains the names of kernel modules that should be loaded<br />
# at boot time, one per line. Lines beginning with "#" are ignored.<br />
# Parameters can be specified after the module name.<br />
scsi_dh_rdac<br />
<br />
<br />
<pre><br />
defaults {<br />
polling_interval 2<br />
path_selector "round-robin 0"<br />
path_grouping_policy multibus<br />
getuid_callout "/lib/udev/scsi_id -g -u -d /dev/%n"<br />
rr_min_io 100<br />
failback immediate<br />
no_path_retry queue<br />
}<br />
blacklist {<br />
wwid *<br />
}<br />
<br />
blacklist_exceptions {<br />
wwid 3690b22c00008da2c000008a35098b0dc<br />
<br />
}<br />
<br />
devices {<br />
device {<br />
vendor "DELL"<br />
product "MD32xxi"<br />
path_grouping_policy group_by_prio<br />
prio rdac<br />
polling_interval 5<br />
path_checker rdac<br />
path_selector "round-robin 0"<br />
hardware_handler "1 rdac"<br />
failback immediate<br />
features "2 pg_init_retries 50"<br />
no_path_retry 30<br />
rr_min_io 100<br />
}<br />
}<br />
<br />
multipaths {<br />
multipath {<br />
wwid 3690b22c00008da2c000008a35098b0dc<br />
alias md3200i<br />
}<br />
<br />
}</pre><br />
<br />
And you need to configure a suitable filter in /etc/lvm/lvm.conf in order to avoid error messages.<br />
<br />
See also:<br />
*[http://www.dell.com/downloads/global/products/pvaul/en/powervault-md32x0-md32x0i-linux-dm-installation-en.pdf Dell Linux DM Installation Details]<br />
*[http://www.dell.com/downloads/global/products/pvaul/en/powervault-md3200i-performance-tuning-white-paper.pdf Dell Array Tuning Best Practices]<br />
<br />
=== Reduxio HX550 ===<br />
<br />
==== Configuration ====<br />
You need to configure the following:<br />
* Update /etc/iscsi/iscsid.conf<br />
* Create/update /etc/multipath.conf<br />
* Create /etc/udev/rules.d/99-reduxio.rules<br />
<br />
===== /etc/iscsi/iscsid.conf =====<br />
Add or update the following parameters:<br />
node.startup = automatic<br />
# The length of time to wait before retrying a failed IO . Can be reduced to a minimum since multipath detects the failure and immediately fails to another path. The value is in seconds and the default is typically 120.<br />
node.session.timeo.replacement_timeout = 5<br />
<br />
# The time to wait for an iSCSI login to complete. The value is in seconds and the default is 15.<br />
node.conn[0].timeo.login_timeout = 15<br />
<br />
# To specify the time to wait for logout to complete, edit the line.<br />
# The value is in seconds and the default is 15 seconds.<br />
node.conn[0].timeo.logout_timeout = 15<br />
<br />
# Time interval to wait for on connection before sending a ping.<br />
node.conn[0].timeo.noop_out_interval = 5<br />
<br />
# To specify the time to wait for a Nop-out response before failing<br />
# the connection, edit this line. Failing the connection will<br />
# cause IO to be failed back to the SCSI layer. If using dm-multipath<br />
# this will cause the IO to be failed to the multipath layer.<br />
node.conn[0].timeo.noop_out_timeout = 5<br />
<br />
#<br />
# This retry count along with node.conn[0].timeo.login_timeout<br />
# determines the maximum amount of time iscsid will try to<br />
# establish the initial login. node.session.initial_login_retry_max is<br />
# multiplied by the node.conn[0].timeo.login_timeout to determine the<br />
# maximum amount.<br />
node.session.initial_login_retry_max 8<br />
<br />
===== /etc/udev/rules.d/99-reduxio.rules =====<br />
Create the following file:<br />
<br />
# /etc/udev/rules.d/99-reduxio.rules SUBSYSTEM=="block" , ACTION=="change", ATTRS{model}=="TCAS", ATTRS{vendor}=="REDUXIO", RUN+="/bin/sh -c '/usr/sbin/iscsiadm -m session -R '"<br />
SUBSYSTEM=="block" , ACTION=="change", ATTRS{model}=="TCAS", ATTRS{vendor}=="REDUXIO", ATTR{size}=="0", RUN+="/bin/sh -c 'echo 1 > /sys$DEVPATH/../../delete '"<br />
SUBSYSTEM=="block" , ACTION=="change", ATTRS{model}=="TCAS", ATTRS{vendor}=="REDUXIO", RUN+="/bin/sh -c 'service multipathd reload || service multipath-tools reload ' "<br />
SUBSYSTEM=="block" , ACTION=="change", ATTRS{model}=="TCAS", ATTRS{vendor}=="REDUXIO", RUN+="/bin/sh -c '/usr/sbin/multipath -r $DEVNAME '"<br />
<br />
===== /etc/multipath.conf =====<br />
Create or update /etc/multipath.conf. This is required for correct high-availability<br />
<br />
devices {<br />
device {<br />
vendor "REDUXIO"<br />
product "TCAS"<br />
revision "2300"<br />
path_grouping_policy "group_by_prio"<br />
path_checker "tur"<br />
hardware_handler "1 alua"<br />
path_selector "round-robin 0"<br />
prio "alua"<br />
failback "immediate"<br />
features "0"<br />
rr_weight "uniform"<br />
no_path_retry "72"<br />
queue_without_daemon "no"<br />
rr_min_io_rq 10<br />
rr_min_io 10<br />
user_friendly_names "yes"<br />
fast_io_fail_tmo "10"<br />
}<br />
}<br />
blacklist {<br />
# Note: it is highly recommended to blacklist by wwid or vendor instead of device name<br />
devnode "^sd[a]$"<br />
}<br />
<br />
==== See also ====<br />
*[http://www.reduxio.com Reduxio's website]<br />
<br />
== External links ==<br />
<br />
[[Category: HOWTO]]</div>Eyal Traitel