How to Create RDMs from Local SATA Disks

From vSphere forwards, VMware has included fully working and supported access to SATA based hardware. This includes using that hardware for allowing a virtual machine to see those drives directly through a RDM (Raw Device Mapping). This means that it is possible to share cheap and plentiful local storage directly with virtual machines. Why is this of interest to the corporate environment? Well, perhaps it isn't really that useful in the 'real' world. But it's bloody handy at home! In this article, we'll look at creating RDMs on local SATA storage devices and then sharing these with a Solaris or FreeBSD virtual machine for use in a ZFS pool.

The hardware I'm using to make this work is a white box home-build server suitable for vSphere. Until recently, this server natively ran Community Edition OpenSolaris snv 115 (a mistake I now realise) on a 3.6Ghz Pentium 4 and shared it's 5TB of disks with the other ESX server over iSCSI. That came to an end when we had a series of power cuts a week ago. The up and down power trashed the Solaris disks. I mean trashed one of the mirrored physical system drives so that it was showing up ok, but doing odd things.

I used the downtime as an opportunity to add some more hardware. It's now got a Quad Core processor and 4GB RAM. And this is now the baby ESXi server - my other one has a bigger spec, but is also a similar white box self-build. I realised that there was no need to run a dedicated OS on the box as long as I could allow my ZFS aware VM to 'see' the raw disks. This is a ZFS requirement - you need it to handle all the RAID and disk management stuff.

It runs ESXi just fine and I installed an OpenSolaris VM to read the ZFS drives. The only remaining task was to link the VM to the physical drives. This is normally achieved using a Raw Device Mapping (RDM) when the underlying disk is a SCSI volume. This is true when the volume is a local SCSI disk, iSCSI LUN or Fibre Channel LUN. But what about SATA? When I tried to add an RDM, I was disappointed to note that while the SATA drives would be valid targets for formatting as a VMFS, there is no GUI method to create an RDM. As so often with VMware, this does not mean that there is no way to do it however!

The usual tool for manipulating VMDK files is vmkfstools. So it here that I turned to find the answer to this problem. The manual page states:


-r, --createrdm /vmfs/devices/disks/...
Map a raw disk to a file on a VMFS file system. Once the mapping
is established, it can be used to access the raw disk like a
normal VMFS virtual disk. The ´file length´ of the mapping is
the same as the size of the raw disk that it points to.


From this I played around a little with the syntax until I worked out that it is a case of linking the correct entry in /vmfs/devices/disks to a .VMDK filename. Like a standard UNIX operating system, the disks are listed in /dev/disks. This system has five SAMSUNG 1TB disks and two Western Digital 250GB disks. The symbolic links are important because they allow you to work out which drive has which unique ID. (The :1 :2 :3 etc entries are partitions...)


~ # ls -l /dev/disks
-rw------- 1 root root 1000204886016 Jan 24 21:23 t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562274______
-rw------- 1 root root 1000193768960 Jan 24 21:23 t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562274______:1
-rw------- 1 root root 1000204886016 Jan 24 21:23 t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562289______
-rw------- 1 root root 1000193768960 Jan 24 21:23 t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562289______:1
-rw------- 1 root root 1000204886016 Jan 24 21:23 t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562290______
-rw------- 1 root root 1000193768960 Jan 24 21:23 t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562290______:1
-rw------- 1 root root 1000204886016 Jan 24 21:23 t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562291______
-rw------- 1 root root 1000193768960 Jan 24 21:23 t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562291______:1
-rw------- 1 root root 1000204886016 Jan 24 21:23 t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562302______
-rw------- 1 root root 1000193768960 Jan 24 21:23 t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562302______:1
-rw------- 1 root root 250058268160 Jan 24 21:23 t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76016080
-rw------- 1 root root 250048479744 Jan 24 21:23 t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76016080:1
-rw------- 1 root root 250059350016 Jan 24 21:23 t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450
-rw------- 1 root root 939524096 Jan 24 21:23 t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:1
-rw------- 1 root root 4293918720 Jan 24 21:23 t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:2
-rw------- 1 root root 244821712896 Jan 24 21:23 t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:3
-rw------- 1 root root 4177920 Jan 24 21:23 t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:4
-rw------- 1 root root 262127616 Jan 24 21:23 t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:5
-rw------- 1 root root 262127616 Jan 24 21:23 t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:6
-rw------- 1 root root 115326976 Jan 24 21:23 t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:7
-rw------- 1 root root 299876352 Jan 24 21:23 t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:8
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000202020202057442d5743414c3736303136303830574443205744 -> t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76016080
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000202020202057442d5743414c3736303136303830574443205744:1 -> t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76016080:1
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000202020202057442d5743414c3736353439343530574443205744 -> t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000202020202057442d5743414c3736353439343530574443205744:1 -> t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:1
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000202020202057442d5743414c3736353439343530574443205744:2 -> t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:2
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000202020202057442d5743414c3736353439343530574443205744:3 -> t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:3
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000202020202057442d5743414c3736353439343530574443205744:4 -> t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:4
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000202020202057442d5743414c3736353439343530574443205744:5 -> t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:5
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000202020202057442d5743414c3736353439343530574443205744:6 -> t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:6
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000202020202057442d5743414c3736353439343530574443205744:7 -> t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:7
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000202020202057442d5743414c3736353439343530574443205744:8 -> t10.ATA_____WDC_WD2500JD2D22HBC0__________________________WD2DWCAL76549450:8
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000533156534a39305335363232373420202020202053414d53554e -> t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562274______
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000533156534a39305335363232373420202020202053414d53554e:1 -> t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562274______:1
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000533156534a39305335363232383920202020202053414d53554e -> t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562289______
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000533156534a39305335363232383920202020202053414d53554e:1 -> t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562289______:1
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000533156534a39305335363232393020202020202053414d53554e -> t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562290______
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000533156534a39305335363232393020202020202053414d53554e:1 -> t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562290______:1
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000533156534a39305335363232393120202020202053414d53554e -> t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562291______
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000533156534a39305335363232393120202020202053414d53554e:1 -> t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562291______:1
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000533156534a39305335363233303220202020202053414d53554e -> t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562302______
l--------- 0 root root 1984 Jan 1 1970 vml.0100000000533156534a39305335363233303220202020202053414d53554e:1 -> t10.ATA_____SAMSUNG_HD103SI_________________________S1VSJ90S562302______:1


I created a place to store the RDM files. These could easily go inside the VM you're trying to work with of course:


mkdir /vmfs/volumes/datastore01/rdms


Inside /vmfs/devices/disks you will find a replica of the /dev/disks information. It is this reference that you need to use to actually create the RDM using vmkfstools. So, given the information above I was able to create five RDMs for my existing SAMSUNG drives. These are in a ZFS zpool already, so I need to share all five with my VM:


~ # vmkfstools -r /vmfs/devices/disks/vml.0100000000533156534a39305335363232373420202020202053414d53554e /vmfs/volumes/datastore1/rdms/1tb01.vmdk -a lsilogic
~ # vmkfstools -r /vmfs/devices/disks/vml.0100000000533156534a39305335363232383920202020202053414d53554e /vmfs/volumes/datastore1/rdms/1tb02.vmdk -a lsilogic
~ # vmkfstools -r /vmfs/devices/disks/vml.0100000000533156534a39305335363232393020202020202053414d53554e /vmfs/volumes/datastore1/rdms/1tb03.vmdk -a lsilogic
~ # vmkfstools -r /vmfs/devices/disks/vml.0100000000533156534a39305335363232393120202020202053414d53554e /vmfs/volumes/datastore1/rdms/1tb04.vmdk -a lsilogic
~ # vmkfstools -r /vmfs/devices/disks/vml.0100000000533156534a39305335363233303220202020202053414d53554e /vmfs/volumes/datastore1/rdms/1tb05.vmdk -a lsilogic


The sharp eyes will have noted the change of SCSI controller in each case to be an LSILogic controller. This suits OpenSolaris better than the BusLogic version. When we look inside the RDM folder there should be five VMDK files which report to be 1TB in size. Of course, they're just a pointer to the raw disk. You'll note the spare 250GB drive also got an RDM created for testing purposes before I risked my ZFS zpool to this technique!


~ # ls -la /vmfs/volumes/datastore1/rdms/
drwxr-xr-x 1 root root 1960 Jan 24 21:38 .
drwxr-xr-t 1 root root 1680 Jan 22 21:24 ..
-rw------- 1 root root 1000204886016 Jan 24 21:37 1tb01-rdm.vmdk
-rw------- 1 root root 462 Jan 24 21:37 1tb01.vmdk
-rw------- 1 root root 1000204886016 Jan 24 21:38 1tb02-rdm.vmdk
-rw------- 1 root root 462 Jan 24 21:38 1tb02.vmdk
-rw------- 1 root root 1000204886016 Jan 24 21:38 1tb03-rdm.vmdk
-rw------- 1 root root 462 Jan 24 21:38 1tb03.vmdk
-rw------- 1 root root 1000204886016 Jan 24 21:38 1tb04-rdm.vmdk
-rw------- 1 root root 462 Jan 24 21:38 1tb04.vmdk
-rw------- 1 root root 1000204886016 Jan 24 21:38 1tb05-rdm.vmdk
-rw------- 1 root root 462 Jan 24 21:38 1tb05.vmdk
-rw------- 1 root root 250058268160 Jan 22 21:24 250sata-rdm.vmdk
-rw------- 1 root root 462 Jan 22 21:24 250sata.vmdk


Now we can use these VMDK files with the virtual machine. It is as simple as adding an existing hard disk file to the VM. Firstly, edit the settings of the VM and choose to add new hardware:

Once this VM is powered on, it is able to work directly with the local SATA drives. This does tie the VM to that specific ESX server of course. The following shows how the OpenSolaris VM is able to view these disks directly:


# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@0,0/pci15ad,1976@10/sd@0,0
1. c1t1d0
/pci@0,0/pci15ad,1976@10/sd@1,0
2. c1t2d0
/pci@0,0/pci15ad,1976@10/sd@2,0
3. c1t3d0
/pci@0,0/pci15ad,1976@10/sd@3,0
4. c1t4d0
/pci@0,0/pci15ad,1976@10/sd@4,0
5. c1t5d0
/pci@0,0/pci15ad,1976@10/sd@5,0
Specify disk (enter its number): q

# zpool import
pool: data
id: 3666680638615083755
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
the '-f' flag.
see: http://www.sun.com/msg/ZFS-8000-EY
config:

data ONLINE
raidz1 ONLINE
c1t3d0 ONLINE
c1t5d0 ONLINE
c1t2d0 ONLINE
c1t1d0 ONLINE
c1t4d0 ONLINE

And we're done. :-)