{"id":1033,"date":"2015-05-25T22:08:24","date_gmt":"2015-05-26T02:08:24","guid":{"rendered":"http:\/\/blog.domb.net\/?p=1033"},"modified":"2015-05-25T22:26:30","modified_gmt":"2015-05-26T02:26:30","slug":"ceph-for-cinder-and-glance-in-openstack-juno-part-3-building-the-ceph-cluster","status":"publish","type":"post","link":"https:\/\/blog.domb.net\/?p=1033","title":{"rendered":"Ceph for Cinder and Glance in OpenStack Juno Part 3 \u2013 Building the Ceph Cluster"},"content":{"rendered":"

Part 3<\/strong> of the post series is going to walk you through how to integrate cinder and glance with ceph on RHEL OSP 6. Most of the instructions on what you need to do can be found here Block Devices and OpenStack<\/a><\/p>\n

1. Delete the default pools as you won’t need them<\/p>\n

\r\n[root@ceph-mgmt ~]# cd ~\/ceph-config\r\n[root@ceph-mgmt ~]# ceph osd pool delete data data --yes-i-really-really-mean-it\r\n[root@ceph-mgmt ~]# ceph osd pool delete metadata metadata --yes-i-really-really-mean-it\r\n[root@ceph-mgmt ~]# ceph osd pool delete rbd rbd --yes-i-really-really-mean-it\r\n<\/pre>\n
2. Create new pools matching the upstream documentation. <\/p>\n
\r\n[root@ceph-mgmt ~]# ceph osd pool create volumes 4096\r\n[root@ceph-mgmt ~]# ceph osd pool create images 128\r\n[root@ceph-mgmt ~]# ceph osd pool create backups 128\r\n[root@ceph-mgmt ~]# ceph osd pool create vms 4096\r\n<\/pre>\n
3. As I am using a packstack installed OpenStack my controller runs glance-api, cinder-volume and cinder backup. This menas I need to install the following packages on my controller<\/p>\n
\r\n[root@controller ~]# yum -y install python-rbd ceph \r\n<\/pre>\n
4. On the compute nodes we need to install the ceph package<\/p>\n
\r\n[root@controller ~]# for i in controller compute1 compute2 compute3 compute4 compute5 compute6 compute7 compute8;do \r\n  ssh $i yum -y install python-rbd ceph; \r\n  done\r\n<\/pre>\n
5. As we have cephx enabled we need to create the 3 new users for Nova and Cinder<\/p>\n
\r\n[root@ceph-mgmt ~]# ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'\r\n[root@ceph-mgmt ~]# ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'\r\n[root@ceph-mgmt ~]# ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'\r\n<\/pre>\n
6. The controller needs the following keys (ceph.client.glance.keyring, ceph.client.cinder.keyring, ceph.client.cinder-backup.keyring)<\/p>\n
\r\n[root@ceph-mgmt ~]# ceph auth get-or-create client.glance | ssh controller sudo tee \/etc\/ceph\/ceph.client.glance.keyring\r\n[root@ceph-mgmt ~]# ssh controller sudo chown glance:glance \/etc\/ceph\/ceph.client.glance.keyring\r\n[root@ceph-mgmt ~]# ceph auth get-or-create client.cinder | ssh controller sudo tee \/etc\/ceph\/ceph.client.cinder.keyring\r\n[root@ceph-mgmt ~]# ssh controller sudo chown cinder:cinder \/etc\/ceph\/ceph.client.cinder.keyring\r\n[root@ceph-mgmt ~]# ceph auth get-or-create client.cinder-backup | ssh controller sudo tee \/etc\/ceph\/ceph.client.cinder-backup.keyring\r\n[root@ceph-mgmt ~]# ssh controller sudo chown cinder:cinder \/etc\/ceph\/ceph.client.cinder-backup.keyring\r\n<\/pre>\n
7. For nova compute you will only need the client.cinder.key <\/p>\n
\r\n[root@ceph-mgmt ~]# for i in compute1 compute2 compute3 compute4 compute5 compute6 compute7 compute8;do \r\n    ceph auth get-or-create client.cinder | ssh $i sudo tee \/etc\/ceph\/ceph.client.cinder.keyring    \r\n    ceph auth get-key client.cinder | ssh $i tee client.cinder.key ; \r\n  done\r\n<\/pre>\n
8. Ceph was complaining on my installation that the ceph.client.admin.keyring<\/strong> could not be found on the controller. So I had to add it to get it working<\/p>\n
\r\n[root@ceph-mgmt ~]# scp ceph.client.admin.keyring controller:\/etc\/ceph\/\r\n<\/pre>\n
9. Create the following \/etc\/ceph\/ceph.conf for the controller. the line rbd default format = 2 <\/strong> is essential if you want to create layered images. Layered images are needed if you need to convert qcow2 to raw in rbd. <\/p>\n
\r\n[global]\r\nfsid = cfdf7859-08a5-4fe4-9942-e19c6b522945\r\nmon_initial_members = ceph-mon01, ceph-mon02, ceph-mon03\r\nmon_host = 192.168.1.231,192.168.1.242,192.168.1.238\r\nauth_cluster_required = cephx\r\nauth_service_required = cephx\r\nauth_client_required = cephx\r\nfilestore_xattr_use_omap = true\r\nosd_journal_size = 1000\r\nosd_pool_default_size = 3\r\nosd_pool_default_min_size = 2\r\nosd_pool_default_pg_num = 128\r\nosd_pool_default_pgp_num = 128\r\n\r\n[client]\r\nrbd cache = true\r\nrbd cache writethrough until flush = true\r\nrbd default format = 2\r\n\r\n\r\n[client.glance]\r\nkeyring = \/etc\/ceph\/ceph.client.glance.keyring\r\n\r\n[client.cinder]\r\nkeyring = \/etc\/ceph\/ceph.client.cinder.keyring\r\n\r\n[client.cinder-backup]\r\nkeyring = \/etc\/ceph\/ceph.client.cinder-backup.keyring\r\n<\/pre>\n
10. Copy the ceph config from the controller to the compute nodes and remove the lines client.glance and client.cinder-backup.<\/p>\n
\r\n[root@controller ~]# for i in compute1 compute2 compute3 compute4 compute5 compute6 compute7 compute8;do \r\n   scp \/etc\/ceph\/ceph.conf $i:\/etc\/ceph\/\r\n  done\r\n<\/pre>\n
11. Add the following line in the ceph.conf client section for the compute nodes. You might need to create the \/var\/run ceph directory.<\/p>\n
\r\n[client]\r\nrbd cache = true\r\nrbd cache writethrough until flush = true\r\nadmin socket = \/var\/run\/ceph\/$cluster-$type.$id.$pid.$cctid.asok\r\n<\/pre>\n
12. Next we need to add a secret key to libvirt. The uuid only needs to be created on the first compute host and then copied to the others. the rest of the commands need to run on all hosts. <\/p>\n
\r\n[root@compute1 ~]# uuidgen\r\n5b2a0b64-983d-4c5b-9d53-834a69c21489\r\n<\/pre>\n

\ncat > secret.xml <
\n  5b2a0b64-983d-4c5b-9d53-834a69c21489<\/uuid>
\n  
\n    client.cinder secret<\/name>
\n  <\/usage>
\n<\/secret>
\nEOF
\n<\/code><\/p>\n
\r\nsudo virsh secret-define --file secret.xml\r\nSecret 5b2a0b64-983d-4c5b-9d53-834a69c21489 created\r\nsudo virsh secret-set-value --secret 5b2a0b64-983d-4c5b-9d53-834a69c21489 --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml\r\n<\/pre>\n
13. On the controller edit \/etc\/glance\/glance-api.conf and add the following lines. A complete glance-api.conf can be found here glance-api.conf<\/a><\/p>\n
\r\n[DEFAULT]\r\n...\r\ndefault_store = rbd\r\nshow_image_direct_url = True\r\n...\r\n[glance_store]\r\nstores = rbd\r\nrbd_store_pool = images\r\nrbd_store_user = glance\r\nrbd_store_ceph_conf = \/etc\/ceph\/ceph.conf\r\nrbd_store_chunk_size = 8\r\n<\/pre>\n
14. Now we can configure cinder for ceph\/rbd. Add the following lines under the ceph section. A complete cinder.conf can be found here cinder.conf<\/a><\/p>\n
\r\nvolume_driver = cinder.volume.drivers.rbd.RBDDriver\r\nrbd_pool = volumes\r\nrbd_ceph_conf = \/etc\/ceph\/ceph.conf\r\nrbd_flatten_volume_from_snapshot = false\r\nrbd_max_clone_depth = 5\r\nrbd_store_chunk_size = 4\r\nrados_connect_timeout = -1\r\nglance_api_version = 2\r\nrbd_user = cinder\r\nrbd_secret_uuid = 5b2a0b64-983d-4c5b-9d53-834a69c21489\r\n\r\nbackup_driver = cinder.backup.drivers.ceph\r\nbackup_ceph_conf = \/etc\/ceph\/ceph.conf\r\nbackup_ceph_user = cinder-backup\r\nbackup_ceph_chunk_size = 134217728\r\nbackup_ceph_pool = backups\r\nbackup_ceph_stripe_unit = 0\r\nbackup_ceph_stripe_count = 0\r\nrestore_discard_excess_bytes = true\r\nrbd_user = cinder\r\nrbd_secret_uuid = 5b2a0b64-983d-4c5b-9d53-834a69c21489\r\n<\/pre>\n
15. Also make sure that you enable ceph as a backend in cinder.conf<\/p>\n
\r\nenabled_backends=ceph\r\n<\/pre>\n
16. Add the following config to nova.conf on the compute nodes<\/p>\n
\r\n[libvirt]\r\nimages_type = rbd\r\nimages_rbd_pool = vms\r\nimages_rbd_ceph_conf = \/etc\/ceph\/ceph.conf\r\nrbd_user = cinder\r\nrbd_secret_uuid = 5b2a0b64-983d-4c5b-9d53-834a69c21489\r\ninject_password = false\r\ninject_key = false\r\ninject_partition = -2\r\n<\/pre>\n
17. And ensure that the live migration options are set correctly<\/p>\n
\r\nlive_migration_flag=\"VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST\"\r\n<\/pre>\n
18. Restart the openstack services<\/p>\n
\r\n[root@controller ~]# service openstack-glance-api restart\r\n[root@controller ~]# for i in compute1 compute2 compute3 compute4 compute5 compute6 compute7 compute8;do \r\n     ssh $i service openstack-nova-compute restart; \r\ndone\r\n[root@controller ~]# openstack-cinder-volume restart\r\n[root@controller ~]# openstack-cinder-backup restart\r\n<\/pre>\n
19. Create a new cinder type ceph<\/p>\n
\r\n[root@controller ~(keystone_admin)]# cinder type-create ceph\r\n[root@controller ~(keystone_admin)]# cinder type-key ceph set volume_backend_name=ceph\r\n[root@controller ~(keystone_admin)]# cinder service-list\r\n+------------------+------------------------------+------+---------+-------+----------------------------+-----------------+\r\n|      Binary      |             Host             | Zone |  Status | State |         Updated_at         | Disabled Reason |\r\n+------------------+------------------------------+------+---------+-------+----------------------------+-----------------+\r\n|  cinder-backup   |   controller.fidelity.com    | nova | enabled |   up  | 2015-05-26T01:56:45.000000 |       None      |\r\n| cinder-scheduler |   controller.fidelity.com    | nova | enabled |   up  | 2015-05-26T01:56:41.000000 |       None      |\r\n|  cinder-volume   | controller.fidelity.com@ceph | nova | enabled |   up  | 2015-05-26T01:56:47.000000 |       None      |\r\n|  cinder-volume   | controller.fidelity.com@lvm  | nova | enabled |   down| 2015-05-26T01:56:43.000000 |       None      |\r\n+------------------+------------------------------+------+---------+-------+----------------------------+-----------------+\r\n<\/pre>\n
20. Test if you can create a cinder volume in rbd<\/p>\n
\r\n[root@controller ~(keystone_admin)]#  cinder create --volume-type ceph --display-name ceph-test 1\r\n[root@controller ~(keystone_admin)]#  cinder list\r\n[root@controller ~(keystone_admin)]#  rbd -p volumes ls\r\n<\/pre>\n
If you see your volume here your all good and ready to go to part 4 uploading images to glance via rbd. <\/p>\n","protected":false},"excerpt":{"rendered":"
Part 3 of the post series is going to walk you through how to integrate cinder and glance with ceph on RHEL OSP 6. Most of the instructions on what you need to do can be found here Block Devices and OpenStack 1. Delete the default pools as you won’t need them [root@ceph-mgmt ~]# cd…<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_themeisle_gutenberg_block_has_review":false,"footnotes":""},"categories":[3,36],"tags":[38],"class_list":["post-1033","post","type-post","status-publish","format-standard","hentry","category-linux","category-openstack","tag-ceph-openstack-juno-cinder-glance"],"_links":{"self":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts\/1033","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1033"}],"version-history":[{"count":27,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts\/1033\/revisions"}],"predecessor-version":[{"id":1067,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts\/1033\/revisions\/1067"}],"wp:attachment":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1033"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1033"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1033"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}