Iscsi/nappit

draft

• nappit is a http interface for setting up zfs and iscsi on a few different operating systems.
• this page is about setting up napp-it omnios to serve iscsi for pve to use as storage.
• for mission critical data drbd,ceph,sheepdog, and others should be used so that the data is always written to multiple systems. unplug the nappit hardware and data is off line.
• per comments on forum omnios is a good choice for iscsi nas. I plan to use for 90% of vm's and backups.

links

• http://www.napp-it.org/index_en.html
• http://www.napp-it.org/manuals/
• install & initial set up: http://www.napp-it.org/doc/downloads/napp-it.pdf
• tuning http://www.napp-it.org/manuals/tuning.html
• http://omnios.omniti.com/wiki.php/GeneralAdministration
• https://www.joyent.com/blog/documentation-for-smf
• https://forum.proxmox.com/threads/iscsi-san-presented-as-nfs-using-freenas.26679/

• HardForum: http://hardforum.com/showthread.php?t=1573272
• ServetheHome: https://forums.servethehome.com/index.php?forums/solaris-nexenta-openindiana-and-napp-it.26/

operating system install

• see http://www.napp-it.org/doc/downloads/napp-it.pdf

network mtu 9000

• do this before setting up network. it is hard to do after.
• from http://www.saqr.org/index.php?option=com_content&view=article&id=100:change-default-mtu&catid=51:solaris&Itemid=72

Note use <other command > to know which drv file to edit. see napp-it.pdf

vi /kernel/drv/ixgbe.conf
default_mtu = 9000;

then reboot

check result

root@sys4:/root#  dladm show-linkprop -p mtu ixgbe0
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
ixgbe0       mtu             rw   9000           1500           1500-15500 

access web page

• change the IP: http://10.10.10.254:81/cgi-bin/admin.pl

Info: First login: admin, no password; delete napp-it.cfg to get defaults

Please do not disable Javascript. For realtime monitoring you need a HTML5 websocket capable browser like a newer Firefox or Google Chrome. Logins are valid for current day on all appliances with the same password. It is suggested to use a secure SAN or management network.

pool and zfs

• see napp-it pdf. create pool and zfs
• noatime , from cli;

zfs set atime=off tank

ssh keys at pve for iscsi

• see https://pve.proxmox.com/wiki/Storage:_ZFS_over_iSCSI

comstar

• see Enable iSCSI via menu ZFS Filesystems

see answered questions at: https://forum.proxmox.com/threads/omnios-napp-it-for-zfs-questions

volumes and logical units are created by Proxmox as needed when you create a disk for a VM.

iSCSI target service

I used cli, could not see how to do on gui. tbd this may not be needed. try not doing the following, and if not deeded delete this .

# svcadm enable -r svc:/network/iscsi/target:default
svcadm: svc:/milestone/network depends on svc:/network/physical, which has multiple instances.

check:

# svcs -x svc:/network/iscsi/target:default
svc:/network/iscsi/target:default (iscsi target)
 State: online since Tue Apr  5 17:23:49 2016
   See: itadm(1M)
   See: /var/svc/log/network-iscsi-target:default.log

comstar for kvm

We want to make certain that iscsi is served only on one network. If two paths exists to your storage and this is intentional you must install multipath on every proxmox host otherwise chances are eminent that you will mess-up your storage. Alternative you can create a bond to your storage. If the bond is to provide real HA it must span over two switches and to be able to do this it will require stackable switches.

here ' >' means item to click next .

1. Create target: comstar> targets >CREATE ISCSI Target
2. Create target portal group: comstar > target portal groups > create target portal-group
3. Connect the target portal group to the target: comstar > target portal groups > add target
4. portal group to ISCSI target Without portal group, ISCSI target are published on every valid IP interface Comstar > Target Portal Groups > add member .

TBD recheck last step. the items to click may be fiferent.

from cli run this:

itadm list-target -v

check 'tpg-tags:' . if tpg-tags: default , that means multipath is set and we do not wat that. means every interface will server isci.

comstar/iscsi can now be used for KVM.

comstar for lxc

• note for pve-zsync, one volume per lxc is needed ?
• for LXC: Manually create a volume

1. disks > volumes > create volume . name lxc-vol , size xxG , uncheck thin provisioned.
2. Comstar > Logical Units > create volume LU ** Choose the volume created in 1a to create a LUN from.
3. Comstar > Views > add view

• • Choose the LUN created in 1b to add a view to. **

force lun: i put auto to autoassign a high number

that way you'll get a number for LUN hard to confure with existing luns later on .

Your LUN should now be visible from proxmox for usage as base volume

iscsi setup at pve

http://pve.proxmox.com/wiki/Storage:_ZFS_over_iSCSI

https://forum.proxmox.com/threads/shared-storage-suggestion-for-a-5-node-cluster.26727/page-2#post-134576  :

make this more clear:

My recommendation would be to create two different storages in proxmox,

- one using zfs_over_iscsi for KVM which provides all zfs features like (linked) clones, life snapshots, etc.

- and one using LVM with network backing as explained here: https://pve.proxmox.com/wiki/Storage_Model#LVM_Groups_with_Network_Backing for lxc.

All the above can be done from a single zfs pool.

Manually create a volume and share this volume through an iscsi target.  ?????? where

Using this target for the iscsi plugin to create a shared LUN for proxmox on which you create a lvm storage with network backing.

Use the same zfs pool for configuration of the zfs_over_iscsi storage to use for KVM's.

The zfs_over_iscsi plugin will not overwrite the zvol used for your iscsi target for lvm storage.

In this way you have options to running cluster wide VM's using both kvm and lxc which can be life migrated across the cluster either manually or through HA.

• make sure multipath is not happening.

From pve check all physical connections at pve to ensure that there is just one target: This is correct set up :

sys5  ~ # iscsiadm -m discovery -t st -p 10.1.10.41:3260
iscsiadm: No portals found
sys5  ~ # iscsiadm -m discovery -t st -p 10.2.2.41:3260
10.2.2.41:3260,2 iqn.2010-09.org.napp-it:1459891666

We want only the 10G network IP to accept icsi connections

TBD: how to make it so there is not multipath .

iscsi for kvm

• kvm: per Mir on forum: If disks is running on top of ZFS you get best performance by using cache = nocache (the default).
• using zfs_over_iscsi for KVM which provides all zfs features like (linked) clones, life snapshots, etc.

• pve > storage > add zfs over iscsi

ID: iscsi-sys4

Portal: 10.2.2.41

Pool: data

Block Size: 8k [ 8k gives better performance but slightly more space is wasted ]

Target: iqn.2010-09.org.napp-it:1459891666 [ one way to get target name to copy and paste is this. ].

iscsiadm -m discovery -t st -p 10.2.2.41:3260

Group: leave blank

iSCSI provider: comstar

Thin Provision: Optional but allows to over provision your storage

Write Cache: Enable. [If pools option 'sync' is standard or always it is safe to enable write cache for improved performance]

iscsi for lxc

• https://forum.proxmox.com/threads/shared-storage-suggestion-for-a-5-node-cluster.26727/page-2#post-134706
• https://pve.proxmox.com/wiki/Storage_Model#LVM_Groups_with_Network_Backing for lxc.

Pay attention to these to important recommendations:

1. disable 'use LUNs direcly'
2. Enable shared use (recommended)

TBD: better organize this.

1- pve storage > add iscsi >

• ID sys4-lxc-vol ,
• Portal 10.2.2.41
• Target: pick target you made for lxc in comstar above.
• disable/un check use LUNs direcly

2- Using this target for the iscsi plugin to create a shared LUN for proxmox on which you create a lvm storage with network backing.

• pve > storage > add LVM
• storage name: iscsi-for-lxc
• For 'Base Storage', use the drop down menu to select the previously defined iSCSI target. sys4-lxc-vol (iSCSI)
• For 'Base Volume' select The correct Lun. Picking wrong LUN will cause bad issues.
• For 'Volume Group Name' give a unique name (this name cannot be changed later). lxc-iscsi-vg
• Enable shared use (recommended)
• Click save

storage.cfg result example

zfs: iscsi-sys4
        target iqn.2010-09.org.napp-it:1459891666
        pool data
        portal 10.2.2.41
        iscsiprovider comstar
        blocksize 8k
        content images
        nowritecache

iscsi: sys4-lxc-vol
        portal 10.2.2.41
        target iqn.2010-09.org.napp-it:1459891666
        content none

lvm: iscsi-for-lxc
        vgname lxc-iscsi-vg
        shared
        base sys4-lxc-vol:0.0.3.scsi-3600144f0000008080000570c0bb60001
        content rootdir

kvm notes

If you use thin provisioned Zvols you should choose scsi disks and virtio-scsi controller because comstar (the iscsi daemon in Omnios) supports the scsi unmap command. This means the trim command is honored by comstar and therefore trimmed blocks will be released to the pool from the Zvol.

• https://forum.proxmox.com/threads/suggestions-for-san-config.26216/page-2#post-131751

nfs for backups

• create a zfs to store dump files: create zfs tank/bkup
• click ZFS Filesystems. set NFS to ON for zfs.
• enable nfs services > nfs . make sure service is online.
• at pve add nfs storage
• permissions : see https://forum.proxmox.com/threads/nappit-omnios-how-to-set-up-for-backup-storage.26921/

• In Omnios 'ZFS Filesystems' click the NFS on your datasets which has NFS enabled and add the following after 'sharenfs='

note the nfs has to be turned off , then click nfs again and put in the setting

I used this:

rw=@10.2.2.0/24:@10.1.0.0/16,root=@10.2.2.0/24:10.1.0.0/16

backup from omnios

• TBD add more.
• http://napp-it.org/extensions/replication_en.html
• https://github.com/oetiker/znapzend/blob/master/README.md

support

omnios has commercial support and modules. if your company will use napp-it / omnios then consider supporting the project. coders do need food and drink.