EMC ScaleIO installation and optimization with Redhat 7.3
By Vince
What is ScaleIO? It is a storage solution designed for small businesses and enterprises that use standard servers and disk inside of them to provide storage over ethernet. So if you are a Dell shop, you can install a few servers with a bunch of disks, install some software, and have a storage solution with automatic redundancy. ScaleIO needs direct access to the disk, so it runs best on bare metal servers using an OS such as Redhat Enterprise Linux (RHEL) or Ubuntu. In this guide, we will walk through a quick Redhat installation and optimization. A side note, running it completely in Vmware ESXi is supported, but access to the disk can get difficult, and you cannot VMotion or move the virtual machines anywhere. This is called in the “marketecture “world, hyper-converged. You are running everything off of one common server architecture.
To start with, you’re going to need an EMC support account at support.emc.com (it’s free), and download the ScaleIO complete software package. It is a free download as well, and no license required to run inside a lab/non-prod environment. Once you have the software packages (we are using 2.0.1.3 for this example), we can start.
In this setup, we will use three physical Dell servers (if you just want to play around, create three virtual machines instead) all running RHEL 7.3. Ubuntu works fine as well if that is your Linux flavor of choice. Redhat also allows free enterprise software subscriptions for developers and lab testing, so create an account at https://access.redhat.com if you want to use Redhat and access their package servers.
Terminology
A few terms so we are on the same page:
- MDM - Meta Data Manager - the brains of the operation, this keeps track of where you data is stored and how to access it. These are setup in odd numbers (3 or 5) so there is a tie breaker if one of them fails. All interaction with the system is done through the MDM’s.
- SDS - ScaleIO Data Server - this is the server/service that does all the data work, clients connect to these IP addresses
- SDC - ScaleIO Data Client - this is a light block storage driver installed on each server that needs storage (can be your ESXi server, or directly on a Linux/Windows box). They connect to the MDM and SDS. The SDC stores data location in memory, but uses very little (around 2MB max).
For further reading on the components: https://www.emc.com/collateral/white-papers/h14344-emc-scaleio-basic-architecture.pdf
Architecture
Let’s talk about how to set up your ScaleIO servers. I recommend at least three separate networks connecting your servers together. In a home lab environment, this doesn’t matter (you can just use one for all communication). In a real lab or production environment, you need to split the traffic so there is less of a chance for over subscription on the network (you don’t want packet loss if you can avoid it). There are three types of traffic:
- Management Traffic - this is how each client talks to the servers and you access the CLI of each ScaleIO server
- Server to Server (SDS to SDS) - this is how they re-balance the disk storage - high bandwidth needed, but you can tune this to a certain Mbps to avoid overconsuming
- Client to Server (SDS to SDC) - this is the data path from your client to the server, so how the block storage flows - highest bandwidth available, 40G if you have it (use Intel cards NOT Mellanox for server 40G).
Step one - Install RHEL on all your servers and get them updated with the latest packages. Get the physical cabling for the network plugged in
This can be the hardest part in some environments since you may not have internet access (crazy right?). Run “yum update” and get your servers up to date with the latest kernels and packages. NOTE: as of this writing (Sept 2017), kernel 3.10 is having issues with the ScaleIO client. I suggest changing the default boot kernel to the first installed one.
Step two - copy over the ScaleIO installer package (EMC-ScaleIO-gateway-2.0-13000.211.x86_64.rpm)
Do this however you like, I use sftp which is always turned on by default for RHEL. The installer is called a gateway by EMC.
ScaleIO is installed through a web-installer which is just a RHEL server that has access to the new ScaleIO servers and their network interfaces. I was able to use one of the three servers we are configuring now as the installation server without issue. The installer is a very lightweight application/service. The same installer is used for upgrades down the road (it must have access to every server network interface) so might as well use one of the servers.
Step three - verify network reachability
We need to make sure all of our servers are able to communicated on each network.
An example IP scheme is:
Management: 192.168.50.x/24
SDS - SDS: 192.168.100.x/24
SDS - SDC: 192.168.150.x/24 - this is the client network, so make sure there are enough IP’s for servers you are plugging in. EMC suggests a layer 2 network, but in my testing layer 3 routing works fine if your router passes traffic in hardware (low latency).
ScaleIO-1: 192.168.50.11, 192.168.100.11, 192.168.150.11
ScaleIO-2: 192.168.50.12, 192.168.100.12, 192.168.150.12
ScaleIO-3: 192.168.50.13, 192.168.100.13, 192.168.150.13
The command to configure IP addresses in RHEL is:
nmcli con show
# Once you get a list of the interface names available,
# you can assign an IP address to them
nmcli con mod ens192 ipv4.address 192.168.50.11/24
nmcli con mod ens192 ipv4.gateway 192.168.50.1
nmcli con mod ens192 ipv4.method manual
nmcli con mod ens192 ipv6.method ignore
# Repeat for the other interfaces
With this in mind, we need to ping each server, so login to one of them (scaleio-1 in this instance):
~# ping -c 2 192.168.50.12
PING 192.168.50.12 (192.168.50.12) 56(84) bytes of data.
64 bytes from 192.168.50.12 (192.168.50.12): icmp_seq=1 ttl=58 time=2.23 ms
64 bytes from 192.168.50.12 (192.168.50.12): icmp_seq=2 ttl=58 time=2.30 ms
...and continue on
ping 192.168.100.12
ping 192.168.150.12
ping 192.168.50.13
ping 192.168.100.13
ping 192.168.150.13
Once we have verified the network is reachable, we can get the installation started.
Note: if you are installing this in a small lab, you don’t need 3 networks, just use one IP per server.
Step four - install the gateway package on ScaleIO-1
The gateway service needs the java JDK to run, so this is available in the java-1.8.0-openjdk RHEL package. We also specify the admin password as an environment variable, so change PASSWORD to something else below.
# Install Java
yum install java-1.8.0-openjdk
# Set an admin password to login to our installer
export GATEWAY_ADMIN_PASSWORD=PASSWORD
# Install the installation program
rpm -i EMC-ScaleIO-gateway-2.0-13000.211.x86_64.rpm
If that succeeded, you can go to https://192.168.50.11 and see our installation portal. You will get an invalid certificate error since it is a self-signed cert.
Login with Admin/PASSWORD with the password from above.
Step five - Run through the installer wizard
The installer is a wizard that needs a couple things:
- The installation zip files from the complete installer package we downloaded above. Extract all of the files in the “ScaleIO_2.0.1.3_RHEL_OEL7_Download.zip” file. There will be a bunch of .rpm files. There are packages for each different OS you can install to. If you are using Ubuntu, pick those packages instead. We are going to upload the redhat files.
- A Command Separated Value (CSV) spreadsheet with the details of our installation, including passwords, ip addresses, names, server functions, and clients if any. We will cover the details of this file next.
The CSV File:
In the installation wizard there are two templates available to download, minimal and complete. I recommend taking a look at both of them so you can understand the options available.
The minimal options we need are:
Username
Password
Operating System - Linux
Is MDM/TB
MDM IPs
MDM Name
Once we have those things:
- Click “Get Started”
- On the packages screen, click browse and upload the files we selected above (a bunch of .rpm files)
- Once uploaded, click “Proceed to install”
- Upload the CSV file we created
- If the file checks out, “Click Start Installation”, otherwise fix any errors. You should see 3 servers, a Master, Slave, and Tie Breaker.
- Click the “Monitor” tab to see how things are going. It should be relatively quick (less than a couple minutes). If something hangs in the install it will let you know and fail out. You can repeat this process again or install the packages manually on each server.
# Commands: (might have to reboot the primary mdm after install fails)
# this might not work, run it on each MDM until one works
scli -login -username admin -password Cisco1977!
ALL COMMANDS MIGHT NEED this at the end, only if you are not on the MDM itself that is primary: -mdm_ip 172.31.13.11
# see if you are on the primary or not
scli -query_cluster
# check state of the rest of scaleio (sds)
scli -query_all
1. See if fault sets were created or not
a. scli -query\_all\_fault\_sets -protection\_domain_name amexpd
b. Should be fs1, fs2, fs3
2. If not, add them
a. scli -add\_fault\_set -protection\_domain\_name amexpd -fault\_set\_name fs1
b. scli -add\_fault\_set -protection\_domain\_name amexpd -fault\_set\_name fs2
c. scli -add\_fault\_set -protection\_domain\_name amexpd -fault\_set\_name fs3
3. See which SDS’s are working
a. scli -query\_all\_sds
4. Add the ones that are missing
a. scli -add\_sds -sds\_ip 172.31.13.11 -fault\_set\_name fs1 -protection\_domain\_name amexpd -storage\_pool\_name amexsp -device\_path /dev/sdb -sds\_name scaleio-sds1 -force_clean
b. scli -add\_sds -sds\_ip 172.31.13.12 -fault\_set\_name fs2 -protection\_domain\_name amexpd -storage\_pool\_name amexsp -device\_path /dev/sdb -sds\_name scaleio-sds2 -force_clean
c. scli -add\_sds -sds\_ip 172.31.13.13 -fault\_set\_name fs3 -protection\_domain\_name amexpd -storage\_pool\_name amexsp -device\_path /dev/sdb -sds\_name scaleio-sds3 -force_clean
5. Verify they are all there now
a. scli -query\_all\_sds
6. Verify the cluster looks good
a. scli -query_all
i. Should be all happy, 3 SDS and volume space available
7. Add a client (SDC)
a. First verify there isn’t one already
i. scli -query\_all\_sdc
b. Add the SDC
i. scli -add\_sdc -sdc\_ip 172.31.13.16 -sdc_name scaleio-sdc
8. Create a volume for a client!
a. scli -add\_volume -protection\_domain\_name amexpd -storage\_pool\_name amexsp -size\_gb 104 -volume\_name testvol -thin\_provisioned -use_rmcache
9. List all volumes
a. scli -query\_all\_volumes
10. Map the volume so the client can use it (SDC)
a. scli -map\_volume\_to\_sdc -volume\_name testvol -sdc_ip 172.31.13.16
11. Approve MDM client certs
a. scli -approve\_all\_mdm_certificates
Settings up an SDC - scale io client who will consume storage:
12. Add the interface:
a. nmcli connection add type ethernet con-name scaleio ifname ens224
b. Edit config
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=no
PEERDNS=no
PEERROUTES=no
IPV4\_FAILURE\_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6\_FAILURE\_FATAL=no
IPV6\_ADDR\_GEN_MODE=stable-privacy
NAME=scaleio
UUID=87157ec2-92bf-449d-a3d8-d48e8ed9adbb
DEVICE=ens224
ONBOOT=yes
IPADDR=192.168.200.16
PREFIX=24
13. Specify the servers
a. export MDM_IP=192.168.200.7,192.168.200.8,192.168.200.9
14. Install the client
# redhat
rpm -i EMC-ScaleIO-sdc-2.0-12000.122.el7.x86_64.rpm
# ubuntu
dpkg -i EMC-ScaleIO-sdc-2.0-12000.122.Ubuntu.16.04.x86_64.deb
3. Configure the block storage device and set to auto mount
/opt/emc/scaleio/sdc/bin/drv_cfg -rescan
ls -l /dev/disk/by-id/ | grep scini
a. Pick the device ID you added and build a file system on it
mkfs.xfs /dev/disk/by-id/emc-vol-01b120d04c385346-d754edef00000000
mount -t xfs /dev/disk/by-id/emc-vol-01b120d04c385346-d754edef00000000 /mnt/sio
echo "/dev/disk/by-id/emc-vol-01b120d04c385346-d754edef00000000 /mnt/sio xfs defaults,_netdev 0 0" >> /etc/fstab
b. Verify it works
i. df -h
4. Run some file testing to check for speed
a. Most accurate:
i. time dd if=/dev/zero of=./3gigfile bs=4k iflag=fullblock,count_bytes count=3G
b. Check if it alloc’s quick
i. fallocate -l 10G 10gigfile.img
c. Just make a file (latest way)
i. xfs_mkfile 10240m 10Gigfile
Install the GUI next, you will need the latest java install (1.8 release 121 currently)
-Install it in a separate folder
-Run the GUI with this command:
C:\Users\vtocc1\Documents\Java8.0-121\bin\javaw.exe -Xmx2048m -jar "C:\Program Files\EMC\scaleio\GUI\ui-1.0-SNAPSHOT.jar"
Misc commands:
5. update spare policy percentage if there is an alert in the system
a. scli -modify\_spare\_policy -protection\_domain\_name amexpd -storage\_pool\_name amexsp -spare\_percentage 34 -i\_am_sure
6. update mdm ip address for an SDC in an existing setup (or just uninstall and re-install the client)
a. /opt/emc/scaleio/sdc/bin/drv\_cfg -mod\_mdm\_ip -ip 172.31.13.11 -new\_mdm_ip
Management of system:
7. Login first to MDM, if this fails, try the secondary IP
a. scli -login -username admin -password Cisco1977!
8. Search for commands you need to run
a. scli -help -all | grep -i volume
9. Check the volumes
a. scli -query\_all\_volumes -protection\_domain\_name amexpd
10. See if the SDC has any limits installed on it
a. scli -query\_sdc\_volume\_limits -volume\_name testvol -sdc_ip 172.31.13.16
11. See storage pool usage: shows client use and free space
a. scli -query\_storage\_pool -protection\_domain\_name amexpd -storage\_pool\_name amexsp
12. Check how the SDCs are doing
a. scli -query\_all\_sdc
b. scli -query\_sdc -sdc\_ip 172.31.13.16 -show\_oscillating\_failures
# add new pool and disks (nvme)
scli -add\_storage\_pool -protection\_domain\_name amexpd -storage\_pool\_name amexspnvme
scli -add\_sds\_device -sds\_name scaleio-sds1 -device\_path /dev/nvme0n1 -storage\_pool\_name amexspnvme
scli -modify\_spare\_policy -protection\_domain\_name amexpd -storage\_pool\_name amexspnvme-spare\_percentage 34 -i\_am_sure
# SDS health
scli -query\_sds -sds\_ip 172.31.13.11 -show\_oscillating\_failures
scli -query\_sds\_connectivity\_status -protection\_domain_name amexpd
scli -start\_sds\_network\_test -sds\_ip 172.31.13.11 -parallel\_messages 5 -network\_test\_size\_gb 3 -network\_test\_length_secs 300
scli -query\_sds\_network\_test\_results -sds_ip 172.31.13.11
# Check hard disk latency on SDS
scli -query\_device\_latency\_meters -sds\_ip 172.31.13.11
scli -query\_all\_device\_latency\_meters -protection\_domain\_name amexpd
scli -modify\_volume\_capacity -volume\_name -size\_gb
Then you need to modify the volume on the client
xfs_growfs /dev/scinia
# enable ram read cache on entire domain
scli -enable\_sds\_rmcache -protection\_domain\_name amexpd
scli -disable\_sds\_rmcache -protection\_domain\_name amexpd
scli -set\_rmcache\_usage -protection\_domain\_name amexpd -storage\_pool\_name amexsp -use_rmcache
scli -set\_sds\_rmcache\_size -protection\_domain\_name amexpd -rmcache\_size_mb 16000
scli -query\_all\_fault\_sets -protection\_domain_name amexpd
scli -start\_device\_test -sds_ip 172.31.13.11
scli -start\_device\_test -protection\_domain\_name amexpd
scli -query\_device\_test -sds_ip 172.31.13.11
scli -query\_device\_test -protection\_domain\_name amexpd
scli -query\_system\_limits
Uninstall all scaleIO stuff to re-do it
# ubuntu
dpkg -r emc-scaleio-lia emc-scaleio-mdm emc-scaleio-sdc emc-scaleio-sds emc\_scaleio\_gui
rm -r /opt/emc /var/tmp/scaleio/ /bin/emc ~/EMC
# redhat
rpm -qa | grep -i emc
rpm -e EMC-ScaleIO-lia-2.0-12000.122.el7 EMC-ScaleIO-mdm-2.0-12000.122.el7 EMC-ScaleIO-sds-2.0-12000.122.el7
NVME as disk cache:
dd if=/dev/zero of=/dev/nvme0n1 bs=102400 count=1
scli -add\_sds\_rfcache\_device -sds\_ip 192.168.100.7 -rfcache\_device\_path /dev/nvme0n1
scli -enable\_sds\_rfcache -protection\_domain\_name amexpd
scli -set\_rfcache\_usage -protection\_domain\_name amexpd -storage\_pool\_name amexsp -use_rfcache