Skip to content

Ceph for Cinder and Glance in OpenStack Juno Part 1 – Building the Ceph Cluster

The goal of the ceph / OpenStack integration post series is to give you a step by step guide on how you can build a ceph cluster and integrate it with your openstack environment for a POC as well as demonstrate how you can upload qcow2 images into rbd, transform them into raw and import the raw image into glance. This setup is by no means a production setup as I do not have SSD for the ceph journals. Meaning that the journals will reside on the osd’s them self.

Part 1 of these post series is going to walk you through how to get your RHEL 7.1 hosts ready for ceph 1.2.3 and install the calamari web interface on the admin / ceph-mgmt node.

For this post series I have the following hardware/ software available:

For the OpenStack environment
1x Dell r720 (Controller)
6x Dell r720xd (Compute)
I am using RHEL OSP 6 on RHEL 7.1

For the Ceph environment
1x Dell r720 (ceph-mgmt/admin)
3x Dell r720xd (ceph-mon / This could also be a VM in the real world)
6x Dell r720xd (ceph-osd) a 24 disks
I am using RHEL 7.1 with CEPH 1.2.3

1. My POC environment does not need a host based firewall. Therefore I will turn it off.

# systemctl disable firewalld
# systemctl stop firewalld

2. Create the correct network config for your 10GBE interface in /etc/sysconfig/network-script/ifcfg-Your_interface_name. It is very important that your ceph nodes are connected with at least 1 10GBE Nic which is available across the cluster and MTU 9000 is set.

NAME="System p2p1"
MTU=9000  #####> Essential for ceph cluster

3. Disable NetworkManager and set the network service as a default.

# systemctl disable NetworkManager.service
# systemctl stop NetworkManager.service
# systemctl start network.service
# systemctl enable network.service

4. Disable selinux as ceph doesn’t like it yet.

# sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

5. If you do not have a DNS server make sure your ceph server has the correct host entries

cat > /etc/hosts << EOF compute1 compute2 compute3 compute4 compute5 compute6 compute7 compute8 ceph-mgmt ceph-osd01 ceph-osd02 ceph-osd03 ceph-osd04 ceph-osd05 ceph-mon01 ceph-mon02 ceph-mon03

6. Register your host to rhn with the correct subscription and attach the following repos.

# subscription-manager register --username your_rhn_username --password your_rhn_password
# subscription-manager attach --pool=your_pool_id
# subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-rhceph-1.2-calamari-rpms --enable=rhel-7-server-rhceph-1.2-installer-rpms --enable=rhel-7-server-rhceph-1.2-mon-rpms --enable=rhel-7-server-rhceph-1.2-osd-rpms

7. Update and restart the system

# yum -y update
# init 6

8. Create the root ssh key on the ceph-mgmt host (will also act as admin node) and distribute the keys to the other monitors and osd's in the cluster

[root@ceph-mgmt ~]# ssh-keygen
[root@ceph-mgmt ~]#  for i in ceph-mgmt ceph-mon01 ceph-mon02 ceph-mon03 ceph-osd01 ceph-osd02 ceph-osd03 ceph-osd04 ceph-osd05;do 
    ssh-copy-id $i;

9. Install chrony on all hosts and enable it

[root@ceph-mgmt ~]# for i in ceph-mgmt ceph-mon01 ceph-mon02 ceph-mon03 ceph-osd01 ceph-osd02 ceph-osd03 ceph-osd04 ceph-osd05; do 
   ssh $i "yum -y install chrony ; systemctl enable chronyd ;  systemctl start chronyd"; 

10. Adjust the pid_max count on all the osd servers. As I have more than 20 disks I have to do that.

[root@ceph-mgmt ~]# for i in ceph-osd01 ceph-osd02 ceph-osd03 ceph-osd04 ceph-osd05; do 
 ssh $i "cat 'kernel.pid_max = 4194303' > /etc/sysctl.d/99-pid_max.conf ; sysctl -p /etc/sysctl.d/99-pid_max.conf";

11. Now its time to download the Ceph components from RHN (rhceph-1.2.3-rhel-7-x86_64.iso). Once downloaded mount the iso on your ceph-mgmt host and copy the following files:

[root@ceph-mgmt ~]# mount -o loop rhceph-1.2.3-rhel-7-x86_64.iso /mnt/
[root@ceph-mgmt ~]#  cp /mnt/RHCeph-Calamari-1.2-x86_64-c1e8ca3b6c57-285.pem /etc/pki/product/285.pem
[root@ceph-mgmt ~]#  cp /mnt/RHCeph-Installer-1.2-x86_64-8ad6befe003d-281.pem /etc/pki/product/281.pem
[root@ceph-mgmt ~]#  cp /mnt/RHCeph-MON-1.2-x86_64-d8afd76a547b-286.pem /etc/pki/product/286.pem
[root@ceph-mgmt ~]#  cp /mnt/RHCeph-OSD-1.2-x86_64-25019bf09fe9-288.pem /etc/pki/product/288.pem

12. Install the ice_setup-*.rpm's

[root@ceph-mgmt ~]# yum install /mnt/ice_setup-*.rpm

13. Before we start configuring or installing any further ceph components we need to create a ceph-config folder which will store future ceph configs

[root@ceph-mgmt ~]# mkdir ~/ceph-config
[root@ceph-mgmt ~]# cd ~/ceph-config

14. We now launch the ice server setup which creates a cephdeploy.conf for us which is needed for the ceph deployment later on.

[root@ceph-mgmt ~]# ice_setup -d /mnt

15. Update the ice_setup packages on the calamari admin node.

[root@ceph-mgmt ~]# yum -y update &&  ice_setup update all

16. Initialize the calamari server. During the setup it will ask you for an admin email address and password for user root (weblogin)

[root@ceph-mgmt ~]# calamari-ctl initialize

17. After a successful calamari install you should be able to login to the calamari webserver with root/your_password (You won't see much yet)