{"id":1015,"date":"2015-05-25T11:54:46","date_gmt":"2015-05-25T15:54:46","guid":{"rendered":"http:\/\/blog.domb.net\/?p=1015"},"modified":"2015-05-25T12:17:30","modified_gmt":"2015-05-25T16:17:30","slug":"ceph-for-cinder-and-glance-in-openstack-juno-part-2-building-the-ceph-cluster","status":"publish","type":"post","link":"https:\/\/blog.domb.net\/?p=1015","title":{"rendered":"Ceph for Cinder and Glance in OpenStack Juno Part 2 \u2013 Building the Ceph Cluster"},"content":{"rendered":"

Part 2<\/strong> of the post series is going to walk you through how to build your ceph cluster after you performed the initial configuration<\/a> for ceph. <\/p>\n

NOTE<\/strong>: Ceph 1.2.3 fails to install the Diamond package on minion nodes during state.highstate. You can find the bug report and fix here Diamond bug<\/a> . Thanks to my co worker James for pointing that out. <\/p>\n

1. We can now go ahead and create the cluster configurations. It is critical that all commands will be run from that folder to get a successful installation.<\/p>\n

\r\n[root@ceph-mgmt ~]# cd ~\/ceph-config\r\n<\/pre>\n
2. Deploy the ceph monitors<\/p>\n
\r\n[root@ceph-mgmt ~]# ceph-deploy new ceph-mon01 ceph-mon02 ceph-mon03\r\n<\/pre>\n
3. As your ceph-mgmt node will be the admin node we need ceph and ceph-common installed<\/p>\n
\r\n[root@ceph-mgmt ~]# yum -y install ceph ceph-common\r\n<\/pre>\n
4. Next we can install the ceph software on the nodes. Make sure you followed the initial installation of part 1!<\/p>\n
\r\n[root@ceph-mgmt ~]# ceph-deploy -v install ceph-mgmt ceph-mon01 ceph-mon02 ceph-mon03 ceph-osd01 ceph-osd02 ceph-osd03 ceph-osd4 ceph-osd05\r\n<\/pre>\n
5. We now need to add the initial monitors and gather the keys. This will place the keys into your ceph-deploy directory.<\/p>\n
\r\n[root@ceph-mgmt ~]# ceph-deploy mon create-initial\r\n<\/pre>\n
6. As we installed all the nodes and initialized the monitors we can connect them to calamari. Your ceph cluster will still be in an error state as you need at least 3 active osd to get the cluster into a healthy state. <\/p>\n
\r\n[root@ceph-mgmt ~]# ceph-deploy -v calamari connect ceph-mgmt ceph-mon01 ceph-mon02 ceph-mon03 ceph-osd01 ceph-osd02 ceph-osd03 ceph-osd04 ceph-osd05\r\n<\/pre>\n
7. Define the ceph-mgmt node as admin node<\/p>\n
\r\n[root@ceph-mgmt ~]# ceph-deploy admin ceph-mgmt ceph-mon01 ceph-mon02 ceph-mon03 ceph-osd01 ceph-osd02 ceph-osd03 ceph-osd04 ceph-osd05\r\n<\/pre>\n
8. Prepare the disks and add the osd’s to the cluster. As I do not have SSD for a cache pool the journals will reside on the OSD’s which is not optimal from a performance point of view but enough for a poc setup. It is recommended to “dd” out your osd disks before you run the next stage. This step can run for a while depending on the amount of disks you will add to the cluster (It is not necessary to partition your disks as the “–zap” option will do that for you).   <\/p>\n
\r\n[root@ceph-mgmt ~]# for i in osd01 osd02 osd03 osd04 osd05; do\r\nceph-deploy -v osd create ceph-$i:sdb --zap\r\nceph-deploy -v osd create ceph-$i:sdc --zap\r\nceph-deploy -v osd create ceph-$i:sdd --zap\r\nceph-deploy -v osd create ceph-$i:sde --zap\r\nceph-deploy -v osd create ceph-$i:sdf --zap\r\nceph-deploy -v osd create ceph-$i:sdg --zap\r\nceph-deploy -v osd create ceph-$i:sdh --zap\r\nceph-deploy -v osd create ceph-$i:sdi --zap\r\nceph-deploy -v osd create ceph-$i:sdj --zap\r\n.........\r\n.........\r\n.........\r\n.........;\r\ndone\r\n<\/pre>\n
9. After all your OSD are added successfully to your cluster you should be able to run the following command<\/p>\n
\r\n[root@ceph-mgmt ~]# ceph -s \r\n    cluster cfdf7859-08a5-4fe4-9942-e19c6b522945\r\n     health HEALTH_OK\r\n     monmap e1: 3 mons at {ceph-mon01=172.16.200.231:6789\/0,ceph-mon02=172.16.200.242:6789\/0,ceph-mon03=172.16.200.238:6789\/0}, election epoch 10, quorum 0,1,2 ceph-mon01,ceph-mon03,ceph-mon02\r\n     osdmap e513: 115 osds: 115 up, 115 in\r\n      pgmap v7615: 13312 pgs, 4 pools, 2780 MB data, 641 objects\r\n            15632 MB used, 53349 GB \/ 53365 GB avail\r\n               13312 active+clean\r\n\r\n<\/pre>\n
or login into calamari and see an active healthy cluster. <\/p>\n
\"cephhealth\"<\/a><\/p>\n
Congratulations you have now a running ceph cluster!<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"
Part 2 of the post series is going to walk you through how to build your ceph cluster after you performed the initial configuration for ceph. NOTE: Ceph 1.2.3 fails to install the Diamond package on minion nodes during state.highstate. You can find the bug report and fix here Diamond bug . Thanks to my…<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_themeisle_gutenberg_block_has_review":false,"footnotes":""},"categories":[3,36],"tags":[38],"class_list":["post-1015","post","type-post","status-publish","format-standard","hentry","category-linux","category-openstack","tag-ceph-openstack-juno-cinder-glance"],"_links":{"self":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts\/1015","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1015"}],"version-history":[{"count":16,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts\/1015\/revisions"}],"predecessor-version":[{"id":1032,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts\/1015\/revisions\/1032"}],"wp:attachment":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1015"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1015"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1015"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}