Tape Drives

Note: With Proxmox Backup Server there's now a native tape backup solution available: https://pbs.proxmox.com/docs/tape-backup.html

Note: This article was written for Proxmox VE 4.x releases.

Introduction

In some situations its needed to have access to a directly attached tape drive in a KVM guest. Main reason is to run a backup software in a KVM guest using the tape drive physically connected to the Proxmox VE host.

• Note 1: this guide is focused primarily to run a tape on PVE version 4.x
• Note 2: for this guide, we used a IBM TS3100 Tape Library (Drive: ULTRIUM 5 HH 3580 and Library/Changer: 3573-TL)

Ideally it would have a host that does not belong to the PVE cluster (considering that the idea would -always it were possible- that nodes mainly focus on virtualization tasks). But sometimes we have a device physically connected to one of the PVE hosts (such as a magnetic tape library) and becomes necessary to expose it to the VMs; then this guide is useful for such purposes. NOTE: really there is no problem with give a PVE host the iSCSI Target role, you just have to pay attention to certain minor details, subtleties which is described below.

About iSCSI Targets on Linux

There are three iSCSI target solutions available for Linux: LIO, IET and SCST; this last is which we use to perform the tasks described in this guide. This is because SCST is the best option at this moment, and have the PCI Pass-through capabilities that we need and that we will use. -

Installing SCST on Debian Jessie

First, you are running PVE 4.x, so we assume that the base system is a Debian 8.x (Jessie). As there are no native debian packages to install SCST, we have two options:

1. Download the sources via SVN from SourceForge, build and install the binaries.
2. Get the SCST-deb packages for Debian Jessie 3.1.x adapted from the ppa:scst/3.1.x for Ubuntu (Xenial)

Build and install the binaries from the scst sources (If you like the hard way)

This way has the advantage that will be using the latest version available through the SVN repository. But had a very big disadvantage: you will not have the DKMS features, so, your SCST modules will not automatically built each time you install a new kernel.

First, install all necessary package for the building process, then go to the /usr/src/ for building, an then install the binaries:

aptitude install subversion build-essential module-assistant pve-headers-`uname -r` 
cd /usr/src/
svn co https://scst.svn.sourceforge.net/svnroot/scst/trunk scst
cd scst/
make scst iscsi usr
make scst_install scstadm_install iscsi_install usr_install

Now, we have the scst ready. Only left make a config file in /etc/scst.conf that is we will see below, in the next section Notice that you can see a warning about the new make target and make target_install kernel versions change

Installing from an adapted deb packages of scst 3.1.x for Debian Jessie 8.x (Preferred method and recommended)

The big pro of this method is that you have the advantages of DKMS, so, when a kernel version change is produced, the scst modules are automatically build to match the new kernel version. But as cons, there is no packages or apt repositories for Debian, so you have to build manually. But luckily, exist a PPA of scst 3.1.x for Ubuntu. So, the trick is get the sources from here to make a deb packages for Debian Jessie. For that add you must add to /etc/apt/sources.d/scst-ppa.list and then run aptitude update. As follow:

echo "deb-src http://ppa.launchpad.net/scst/3.1.x/ubuntu xenial main" >/etc/apt/sources.list.d/scst-ppa.list
apt-get update
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A1A5DDAF3B9D5EF7
apt-get update
cd /usr/src
apt-get source scst
cd scst-3.1.1~160517+git0ee4aa8/

change the first line of the file debian/changelog from:

scst (3.1.1~160517+git0ee4aa8-ppa1~xenial) xenial; urgency=high

to:

scst (3.1.1~160517+git0ee4aa8-bpo~jessie) jessie; urgency=high

and build the package (but before, you need get some extra packages for building):

aptitude install dpkg-dev quilt debhelper build-essential module-assistant fakeroot pve-headers-`uname -r`
dpkg-buildpackage

If all was successful, surely you have the deb packages and are ready to install. As follow:

 dpkg -i scst-3.1.1*.deb scst-fileio-tgt_3.1.1*.deb scst-dkms_3.1.1*.deb scstadmin_3.1.1*.deb
 depmod  #only for doubts...

Optional, remove the scst sources from your apt lists:

rm /etc/apt/sources.list.d/scst-ppa.list

If you do not want to build the deb packages, or have troubles to build, and still prefer to have the scst installed via deb, you can get the builded debs here:

1. iscsi-scst_3.1.1~160517+git0ee4aa8-bpo~jessie_amd64.deb
2. scst-dkms_3.1.1~160517+git0ee4aa8-bpo~jessie_amd64.deb
3. scst-fileio-tgt_3.1.1~160517+git0ee4aa8-bpo~jessie_amd64.deb
4. scstadmin_3.1.1~160517+git0ee4aa8-bpo~jessie_amd64.deb

and install them after download in the same way as explained behind.

Configuring the scst target

In the past (SCST versions <2.0), one of the methods used to set SCST was through manipulation by procfs, but these days it has become obsolete. The two methods available today to configure stsc, are: or through low-level manipulation using sysfs, or using the command line scstadmin. This booth methods are well described in the scst README file of the scst sources. But to make it short and simple, we set up a configuration file quickly as follows: First, install the lspci utility (if don't have it), and then lsscsi for identify the tape devices:

aptitude install lsscsi
lscsi
[0:0:0:0]    disk    ATA      ST4000DM000-1F21 CC54  /dev/sda 
[0:0:1:0]    tape    IBM      ULT3580-HH5      BBNF  /dev/st0 
[0:0:1:1]    mediumx IBM      3573-TL          A.40  /dev/sch0
[0:0:2:0]    disk    ATA      WDC WD30EFRX-68E 0A82  /dev/sdb 
[1:0:0:0]    disk    ATA      WDC WD5000AAKX-6 1H18  /dev/sdc 

So, in the above example, we can see that the PCI Devices corresponding to the tape are [0:0:1:0] and [0:0:1:1]. So, our configuration file should see more or less as follows:

nano /etc/scst.conf

# Config file for SCST v3.1.0-pre1.

HANDLER dev_tape {
	DEVICE 0:0:1:0
}

HANDLER dev_changer {
	DEVICE 0:0:1:1
}

TARGET_DRIVER iscsi {
	enabled 1

	TARGET iqn.2010-04.urgencias.local:hu-ds2:tape {
 		enabled 1
		#rel_tgt_id 2
                IncomingUser "username password-of-at-least-12-chars"
                IncomingUser "another_user another_long_password"
		LUN 0 0:0:1:0
		LUN 1 0:0:1:1
	}

A final detail, is the fact that, if you use the deb packages whether compiled by yourself, or downloaded via the links provided, you will notice that there is a file /etc/defaults/scst that you must modify to look like:

# iscsi-scstd command-line options. See also man iscsi-scstd.
# ISCSID_OPTIONS="-u0 -g0 -p3260"

# When using the procfs interface, specify all SCST target drivers in
# SCST_TARGET_MODULES. An example:
# SCST_TARGET_MODULES="scst iscsi-scst scst_tape scst_changer ib_srpt"

Now, you should restart the PVE host on which you installed all this. Yes, restart. Because it is likely to do a systemctl restart scst.service not work properly. What is certain is a reboot. Then you can control the state of scst running systemctl status scst.service. You must get something like:

dmesg |grep -i scst
[...]
[   22.206710] scst: Device 0:0:1:1 doesn't support MODE SENSE or control mode page
[   22.206713] scst: ***WARNING***: Unable to get device's 0:0:1:1 control mode page, using existing values/defaults: TST 0, TMF_ONLY 0, QUEUE ALG 1, QErr 0, SWP 0, TAS 0, D_SENSE 0, DPICZ 0, has_own_order_mgmt 0
'''[   22.206720] scst: Device 0:0:1:1 assigned to dev handler dev_changer'''
[   22.212441] scst: Device 0:0:1:0: TST 0, TMF_ONLY 0, QUEUE ALG 0, QErr 0, SWP 0, TAS 0, D_SENSE 0, DPICZ 0, has_own_order_mgmt 1
'''[   22.212445] scst: Device 0:0:1:0 assigned to dev handler dev_tape'''
[   22.212699] scst: Target iqn.2010-04.urgencias.local:hu-ds2:tape for template iscsi registered successfully
[   22.215947] scst: Added device 0:0:1:0 to group iqn.2010-04.urgencias.local:hu-ds2:tape (LUN 0, flags 0x2) to target iqn.2010-04.urgencias.local:hu-ds2:tape
[   22.216008] scst: Added device 0:0:1:1 to group iqn.2010-04.urgencias.local:hu-ds2:tape (LUN 1, flags 0x2) to target iqn.2010-04.urgencias.local:hu-ds2:tape
[   22.217249] scst: Using autogenerated relative target id 1 for target iqn.2010-04.urgencias.local:hu-ds2:tape
[  137.980152] scst: Using security group "iqn.2010-04.urgencias.local:hu-ds2:tape" for initiator "iqn.2010-04.urgencias.local:hu-fs1" (target iqn.2010-04.urgencias.local:hu-ds2:tape)
[  137.980503] iscsi-scst: Negotiated parameters: InitialR2T No, ImmediateData Yes, MaxConnections 1, MaxRecvDataSegmentLength 1048576, MaxXmitDataSegmentLength 65536, 
[  137.980510] iscsi-scst:     MaxBurstLength 262144, FirstBurstLength 65536, DefaultTime2Wait 0, DefaultTime2Retain 0, 
[  137.980514] iscsi-scst:     MaxOutstandingR2T 16, DataPDUInOrder Yes, DataSequenceInOrder Yes, ErrorRecoveryLevel 0, 
[  137.980518] iscsi-scst:     HeaderDigest None, DataDigest None, OFMarker No, IFMarker No, OFMarkInt 2048, IFMarkInt 2048, RDMAExtensions No
[  137.980524] iscsi-scst: Target parameters set for session 1000037010040: QueuedCommands 32, Response timeout 90, Nop-In interval 30, Nop-In timeout 30

You can try with systemctl status too:

systemctl status scst.service
● scst.service - LSB: SCST - A Generic SCSI Target Subsystem
   Loaded: loaded (/etc/init.d/scst)
   Active: active (running) since Wed 2016-07-27 03:34:26 ART; 2 days ago
   CGroup: /system.slice/scst.service
           └─5952 /usr/sbin/iscsi-scstd

Jul 27 03:34:26 hu-ds2 scst[5785]: Loading and configuring SCST.
Jul 27 03:34:50 hu-ds2 iscsi-scstd[5952]: Connect from 172.16.XXX.XXX:61159 to 172.16.XXX.XXX:3260

Using Tape Drives as iSCSI target (tested with Windows 2012 R2 and Linux Debian Jessie KVM guests)

Now, if all are good, you are ready to connect to tape from any system with capable iSCSI Initiator software.

Working with Windows 2012 R2 Server

For the test, we will use a QEMU VM running Windows 2012 R2 and the backup software Symantec Backup Exec 15 ver.14.2 (Trial). Steps for making the tape (or another iSCSI device) work with this version of Windows:

Procedures	Screenshots
Start the iSCSI Initiator: if Windows says something about that need to start the iSCSI Service respond Ok. Then, in the iSCSI Initiator Properties window, put the IP in the "Target:" TextBox and click in "Connect" button. You must see the target id in the "Discovered targets" area. If prompt to put the discovered target in the Favorites responding Yes, you get the device(s) automatically connected when Windows starts.
Go to "Volumes and Devices" tab, and click in the "Auto Configure" button. You must see the autoconfigured devices of each LUN defined in the iSCSI target in the "Volume List" area.
At this point, Windows says that new hardware was detected and surely is tring to installing drivers. I recommend download the drivers of the manufactured and then, in the Device Manager, Update the drivers.
Finally, you have a working hardware and and ready to use it with any software on the VM. In the above example, we use Backup Exec 15 v14.2 (Trial), and in the Storage View, note that the hardware was autodetected (no need to add manually). Is ready to use. If you have troubles with Backup Exec, try to restart the Backup Exec Device & Media Service from the Services Console of Windows... or directly, restart your VM.

And that is all. In others versions of Windows, the procedure is very similar or equal.

Working with Debian-based Linux Distro (from QEMU VM, not LXC)

For the test, I will use a VM (this is, full virtualization) and not a LXC; the VM has a recently fresh installed Deepin Linux Server 15 Kui (is a nice Chinese distro based on Debian Jessie). Well, first, install the iSCSI Initiator software. In Debian systems, by default is 'open-iscsi'. Also, install the 'lsscsi' too (a utility for view/listing yours SCSI devices).

aptitude install open-iscsi lsscsi

Configure the iSCSI initiator. This is done by touching two files: 'initiatorname.iscsi', responsible of the initiator name, and 'iscsid.conf', responsible of all the rest. The file '/etc/iscsi/initiatorname.iscsi' should content something like this:

## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator.  The InitiatorName must be unique
## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.2010-04.local.urgencias:hu-gts1

And the file '/etc/iscsi/iscsid.conf', should content the CHAP user & pass (if apply), and the startup related settings:

node.startup = automatic
node.leading_login = Yes

node.session.auth.authmethod = CHAP
node.session.auth.username = tape_user
node.session.auth.password = user_passwd_of_the_target
discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = tape_user
discovery.sendtargets.auth.password = user_passwd_of_the_target

node.session.timeo.replacement_timeout = 15

Restart the iSCSI Initiator service, and check that is running:

root@hu-gts1:~# systemctl restart open-iscsi.service
root@hu-gts1:~# systemctl status open-iscsi.service
● open-iscsi.service - LSB: Starts and stops the iSCSI initiator services and logs in to default targets
   Loaded: loaded (/etc/init.d/open-iscsi)
  Drop-In: /lib/systemd/system/open-iscsi.service.d
           └─fix-systemd-deps.conf
   Active: active (running) since Sun 2016-07-31 12:57:29 ART; 17min ago
  Process: 16065 ExecStop=/etc/init.d/open-iscsi stop (code=exited, status=0/SUCCESS)
  Process: 16058 ExecStop=/etc/init.d/umountiscsi.sh stop (code=exited, status=0/SUCCESS)
  Process: 16104 ExecStart=/etc/init.d/open-iscsi start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/open-iscsi.service
           ├─16119 /usr/sbin/iscsid
           └─16120 /usr/sbin/iscsid

Jul 31 12:57:29 hu-gts1 iscsid[16118]: iSCSI logger with pid=16119 started!
Jul 31 12:57:29 hu-gts1 open-iscsi[16104]: Starting iSCSI initiator service: iscsidln: failed to create symbolic link ‘/run/sendsigs.omit…ile exists
Jul 31 12:57:29 hu-gts1 open-iscsi[16104]: .
Jul 31 12:57:29 hu-gts1 open-iscsi[16104]: Setting up iSCSI targets:
Jul 31 12:57:29 hu-gts1 open-iscsi[16104]: iscsiadm: No records found
Jul 31 12:57:29 hu-gts1 open-iscsi[16104]: .
Jul 31 12:57:29 hu-gts1 open-iscsi[16104]: Mounting network filesystems:.
Jul 31 12:57:29 hu-gts1 open-iscsi[16104]: Enabling network swap devices:.
Jul 31 12:57:30 hu-gts1 iscsid[16119]: iSCSI daemon with pid=16120 started!

Discover the target:

root@hu-gts1:~# iscsiadm  -m discovery -t st -p hu-ds2.urgencias.local

172.16.XXX.XXX:3260,1 iqn.2010-04.urgencias.local:hu-ds2:tape

Next let's open a session to a given Target:

root@hu-gts1:~# iscsiadm  -m node  --targetname "iqn.2010-04.urgencias.local:hu-ds2:tape" --portal "172.16.XXX.XXX:3260" --login

Logging in to [iface: default, target: iqn.2010-04.urgencias.local:hu-ds2:tape, portal: 172.16.XXX.XXX,3260] (multiple)
Login to [iface: default, target: iqn.2010-04.urgencias.local:hu-ds2:tape, portal: 172.16.XXX.XXX,3260] successful.

Now, you can see that your tape is attached by iSCSI, use lsscsi or dmesg:

root@hu-gts1:~# lsscsi

[1:0:0:0]    cd/dvd  QEMU     QEMU DVD-ROM     2.5+  /dev/sr0 
[9:0:0:0]    tape    IBM      ULT3580-HH5      BBNF  /dev/st0 
[9:0:0:1]    mediumx IBM      3573-TL          A.40  /dev/sch0

root@hu-gts1:~# dmesg
[...]
[83809.221906] iscsi: registered transport (tcp)
[83809.228893] iscsi: registered transport (iser)
[83819.442280] scsi9 : iSCSI Initiator over TCP/IP
[83819.699400] scsi 9:0:0:0: Sequential-Access IBM      ULT3580-HH5      BBNF PQ: 0 ANSI: 6
[83819.702682] scsi 9:0:0:0: Attached scsi generic sg1 type 1
[83819.704788] scsi 9:0:0:1: Medium Changer    IBM      3573-TL          A.40 PQ: 0 ANSI: 5
[83819.707513] scsi 9:0:0:1: Attached scsi generic sg2 type 8
[83819.721408] st: Version 20101219, fixed bufsize 32768, s/g segs 256
[83819.721829] st 9:0:0:0: Attached scsi tape st0
[83819.721832] st 9:0:0:0: st0: try direct i/o: yes (alignment 1 B)
[83819.723081] SCSI Media Changer driver v0.25 
[83819.741741] ch0: type #1 (mt): 0x1+1 [medium transport]
[83819.741744] ch0: type #2 (st): 0x1000+23 [storage]
[83819.741745] ch0: type #3 (ie): 0x10+1 [import/export]
[83819.741746] ch0: type #4 (dt): 0x100+1 [data transfer]
[83819.763501] ch0: dt 0x100: ch0: ID/LUN unknown
[83819.763504] ch0: INITIALIZE ELEMENT STATUS, may take some time ...
[83819.773365] osst :I: Tape driver with OnStream support version 0.99.4
osst :I: $Id: osst.c,v 1.73 2005/01/01 21:13:34 wriede Exp $
[83819.776784] ch0: ... finished
[83819.776788] ch 9:0:0:1: Attached scsi changer ch0

And is done! You can use the well known Bacula, or Amanda, or any backup software tool for Linux... (but not recommend 'tar' for tape drives with changers, because of the implicit complexity).

Final Notes / Comments

FINAL NOTE: Be very, very careful not to try to use your tape drive for two iSCSI initiators simultaneously or concurrently. This could be potentially dangerous (even if the tape library support multiple connections)

Using Tape Drives via PCI pass through with extra SCSI controller

No one implementation that you can take done directly from Proxmox-VE only using QEMU-KVM. Therefore if direct hardware access is desired, it should be as described in this wiki... (which is just that what you end up doing in some way at the end).