{"id":810,"date":"2015-02-25T18:10:18","date_gmt":"2015-02-25T23:10:18","guid":{"rendered":"http:\/\/blog.domb.net\/?p=810"},"modified":"2015-03-21T20:35:59","modified_gmt":"2015-03-22T00:35:59","slug":"cloudforms-manageiq-automation-and-centralized-logging-with-logstash-kibana-and-elasticsearch","status":"publish","type":"post","link":"https:\/\/blog.domb.net\/?p=810","title":{"rendered":"Cloudforms \/ ManageIQ automation and centralized logging with logstash kibana and elasticsearch"},"content":{"rendered":"<p>Cloudforms \/ ManageIQ has a very powerful automation engine. There is basically nothing you can&#8217;t do with it. You could even trigger an api call to make coffee during a provisioning run. But what if your provisioning fails?<\/p>\n<p>Every step which is executed by the cloudforms automation engine gets logged into \/var\/www\/miq\/vmdb\/log\/evm.log or \/var\/www\/miq\/vmdb\/log\/automation.log. As you can imagine the log is very verbose which in one way is awesome (you have all the data) but in the another way all this data makes it difficult to troubleshoot.<\/p>\n<p><a href=\"http:\/\/blog.domb.net\/wp-content\/uploads\/miqlog.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-854\" src=\"http:\/\/blog.domb.net\/wp-content\/uploads\/miqlog.png\" alt=\"miqlog\" width=\"1930\" height=\"954\" srcset=\"https:\/\/blog.domb.net\/wp-content\/uploads\/miqlog.png 1930w, https:\/\/blog.domb.net\/wp-content\/uploads\/miqlog-300x148.png 300w, https:\/\/blog.domb.net\/wp-content\/uploads\/miqlog-1024x506.png 1024w\" sizes=\"auto, (max-width: 1930px) 100vw, 1930px\" \/><\/a><\/p>\n<p>If you do not have a centralized logging in place today you would have ssh to cloudforms and tail the logs to troubleshoot a failed run.<\/p>\n<p>This post will help you build a logstash server on rhel7 with a log filter for the cloudforms evm and automation log. This will empower you to filter the logs based on taskid \/ Next Stage etc&#8230; which give you a complete history on what exactly happened during provisioning run. So for example you could go check the clone_options during provisioning which would look like this:<\/p>\n<p><a href=\"http:\/\/blog.domb.net\/wp-content\/uploads\/logstash1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-848\" src=\"http:\/\/blog.domb.net\/wp-content\/uploads\/logstash1.png\" alt=\"logstash\" width=\"1885\" height=\"913\" srcset=\"https:\/\/blog.domb.net\/wp-content\/uploads\/logstash1.png 1885w, https:\/\/blog.domb.net\/wp-content\/uploads\/logstash1-300x145.png 300w, https:\/\/blog.domb.net\/wp-content\/uploads\/logstash1-1024x496.png 1024w\" sizes=\"auto, (max-width: 1885px) 100vw, 1885px\" \/><\/a><\/p>\n<p>You can find all the code snippets as an install script here http:\/\/github.com\/ldomb\/elkformiqonrhel7<br \/>\n1. Add the hostname to \/etc\/hosts (this is important as the hostname needs to resolve)<\/p>\n<pre lang=\"bash\">\r\necho -e \"\\n ## Adding entry to \/etc\/hosts\\n\"\r\ncat >> \/etc\/hosts << EOF\r\n192.168.1.58  elk.local.domb.com elk\r\nEOF\r\n<\/pre>\n<p>2. Install a rhel7 server and register it to rhn. You will need the following repos:<br \/>\nrhel-7-server-rpms<br \/>\nrhel-7-server-rh-common-rpms<br \/>\nrhel-7-server-optional-beta-rpms (for golang to create a working ssl cert)<\/p>\n<pre lang=\"bash\">subscription-manager register\r\nsubscription-manager attach --pool=$POOL_ID\r\nsubscription-manager repos --disable=*\r\nsubscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-7-server-optional-beta-rpms\r\n<\/pre>\n<p>3. Disable firewalld and ipv6<\/p>\n<pre lang=\"bash\"># systemctl stop firewalld\r\n# systemctl disable firewalld\r\n\r\n# cat >> \/etc\/sysctl.d\/99-disableipv6.conf << EOF\r\nnet.ipv6.conf.all.disable_ipv6=1\r\nEOF\r\nsysctl -p \/etc\/sysctl.d\/99-disableipv6.conf\r\n<\/pre>\n<p>4. Set Selinux to premissive mode<\/p>\n<pre lang=\"bash\"># sed -i s\/SELINUX=enforcing\/SELINUX=permissve\/g \/etc\/selinux\/config\r\n# setenforce 0\r\n<\/pre>\n<p>5. Add the elastic search repo<\/p>\n<pre lang=\"bash\"># rpm --import http:\/\/packages.elasticsearch.org\/GPG-KEY-elasticsearch\r\n# cat &gt; \/etc\/yum.repos.d\/elasticsearch.repo &lt;&lt; EOF\r\n[elasticsearch-1.3]\r\nname=Elasticsearch repository for 1.3.x packages\r\nbaseurl=http:\/\/packages.elasticsearch.org\/elasticsearch\/1.3\/centos\r\ngpgcheck=1\r\ngpgkey=http:\/\/packages.elasticsearch.org\/GPG-KEY-elasticsearch\r\nenabled=1\r\nEOF\r\n<\/pre>\n<p>6. Install elasticsearch, java and apache<\/p>\n<pre lang=\"bash\"># yum install -y elasticsearch java-1.7.0-openjdk httpd\r\n<\/pre>\n<p>7. Disable dynamic scripts for elasticsearch<\/p>\n<pre lang=\"bash\"># cat &gt;&gt; \/etc\/elasticsearch\/elasticsearch.yml &lt;&lt; EOF\r\n### Custom config parameters\r\nscript.disable_dynamic: true\r\nEOF\r\n<\/pre>\n<p>8. Configure Systemd to start elastic search on boot<\/p>\n<pre lang=\"bash\"># systemctl daemon-reload\r\n# systemctl enable elasticsearch.service\r\n# systemctl start elasticsearch.service\r\n<\/pre>\n<p>9. Now that we have elastic search running we can install kibana. Go download and unpack the kibana tarball and make sure to create a \/var\/www\/kibana3\/pub dir which we will serve as the directory for the client certs and scripts.<\/p>\n<pre lang=\"bash\"># wget -P \/var\/www\/html\/ https:\/\/download.elasticsearch.org\/kibana\/kibana\/kibana-3.1.0.tar.gz\r\n# tar -xzvf \/var\/www\/html\/kibana-3.1.0.tar.gz -C \/var\/www\/html\r\n# mv \/var\/www\/html\/kibana-3.1.0 \/var\/www\/html\/kibana3\r\n# rm -f \/var\/www\/html\/kibana-3.1.0.tar.gz\r\n# mkdir -p \/var\/www\/html\/kibana3\/pub\r\n<\/pre>\n<p>10. Change the listen port for kibana in kibana-3.0.1\/config.js from 9200 to 80 as we will proxy all traffic through port 80 in apache<\/p>\n<pre lang=\"bash\"># sed -i \"s@elasticsearch: \\\"http:\/\/\\\"+window.location.hostname+\\\":9200\\\"@elasticsearch: \\\"http:\/\/\\\"+window.location.hostname+\\\":80\\\"@g\" config.js\r\n<\/pre>\n<p>11. All the config for kibana is now done. Let-s create the apache config<\/p>\n<pre lang=\"bash\">\r\nServerName elk.local.domb.com\r\n\r\nDocumentRoot \/var\/www\/html\/kibana3\r\n\r\nAllow from all\r\nOptions -Multiviews\r\n\r\nLogLevel debug\r\nErrorLog \/var\/log\/httpd\/error_log\r\nCustomLog \/var\/log\/httpd\/access_log combined\r\n\r\n# Proxy for _aliases and .*\/_search\r\n\r\nProxyPassMatch http:\/\/127.0.0.1:9200\/$1\r\nProxyPassReverse http:\/\/127.0.0.1:9200\/$1\r\n\r\n# Proxy for kibana-int\/{dashboard,temp} stuff (if you don't want auth on \/, then you will want these to be protected)\r\n\r\nProxyPassMatch http:\/\/127.0.0.1:9200\/$1$2\r\nProxyPassReverse http:\/\/127.0.0.1:9200\/$1$2\r\n\r\nAuthType Basic\r\nAuthBasicProvider file\r\nAuthName \"Restricted\"\r\nAuthUserFile \/etc\/httpd\/conf.d\/kibana-htpasswd\r\nRequire valid-user\r\n\r\n<\/pre>\n<p>12. set a user name and password for the web interface<\/p>\n<pre lang=\"bash\"># htpasswd -c \/etc\/httpd\/conf.d\/kibana-htpasswd admin\r\n<\/pre>\n<p>13. Start apache and make sure it's started on boot<\/p>\n<pre lang=\"bash\"># systemctl enable httpd\r\n# systemctl start httpd\r\n<\/pre>\n<p>14. Add the logstash repo<\/p>\n<pre lang=\"bash\">\r\n# cat &gt; \/etc\/yum.repos.d\/logstash.repo &lt;&lt; EOF\r\n[logstash-1.4]\r\nname=logstash repository for 1.4.x packages\r\nbaseurl=http:\/\/packages.elasticsearch.org\/logstash\/1.4\/centos\r\ngpgcheck=1\r\ngpgkey=http:\/\/packages.elasticsearch.org\/GPG-KEY-elasticsearch\r\nenabled=1\r\nEOF\r\n<\/pre>\n<p>15. Install logstash<\/p>\n<pre lang=\"bash\">\r\n# yum install -y logstash\r\n<\/pre>\n<p>16. As we are going to use logstash-forwarder we want to have a secure connection. to make this happen and to not run into any ssl issues we need to install the golang package from the beta channel so that we can build the cert generator.<\/p>\n<pre lang=\"bash\"># yum install golang -y\r\n# wget https:\/\/raw.githubusercontent.com\/driskell\/log-courier\/develop\/src\/lc-tlscert\/lc-tlscert.go\r\n# go build lc-tlscert.go\r\n<\/pre>\n<p>17. Once you build the lc-tlscert you can execute it and answer all the questions to create an ssl cert.<\/p>\n<pre lang=\"bash\"># .\/lc-tlscert\r\nSpecify the Common Name for the certificate. The common name\r\ncan be anything, but is usually set to the server's primary\r\nDNS name. Even if you plan to connect via IP address you\r\nshould specify the DNS name here.\r\n\r\nCommon name: elk.local.domb.com\r\n\r\nThe next step is to add any additional DNS names and IP\r\naddresses that clients may use to connect to the server. If\r\nyou plan to connect to the server via IP address and not DNS\r\nthen you must specify those IP addresses here.\r\nWhen you are finished, just press enter.\r\n\r\nDNS or IP address 1: 192.168.1.58\r\nDNS or IP address 2:\r\n\r\nHow long should the certificate be valid for? A year (365\r\ndays) is usual but requires the certificate to be regenerated\r\nwithin a year or the certificate will cease working.\r\n\r\nNumber of days: 1200\r\nCommon name: elk.local.domb.com\r\nDNS SANs:\r\nNone\r\nIP SANs:\r\n192.168.1.58\r\n\r\nThe certificate can now be generated\r\nPress any key to begin generating the self-signed certificate.\r\n\r\nSuccessfully generated certificate\r\nCertificate: selfsigned.crt\r\nPrivate Key: selfsigned.key\r\n\r\nCopy and paste the following into your Log Courier\r\nconfiguration, adjusting paths as necessary:\r\n\"transport\": \"tls\",\r\n\"ssl ca\": \"path\/to\/selfsigned.crt\",\r\n\r\nCopy and paste the following into your LogStash configuration,\r\nadjusting paths as necessary:\r\nssl_certificate =&gt; \"path\/to\/selfsigned.crt\",\r\nssl_key =&gt; \"path\/to\/selfsigned.key\",\r\n\r\n<\/pre>\n<p>18. Create an ssl directory in \/etc\/logstash and copy the 2 keys into the newly created dir.<\/p>\n<pre lang=\"bash\"># mkdir -p \/etc\/logstash\/ssl\/\r\n# mv \/root\/selfsigned.crt \/etc\/logstash\/logstash-forwarder.crt; chmod 666 \/etc\/logstash\/logstash-forwarder.crt\r\n# mv \/root\/selfsigned.key \/etc\/logstash\/logstash-forwarder.key; chmod 666 \/etc\/logstash\/logstash-forwarder.key\r\n<\/pre>\n<p>19. Copy the created crt to the kibana3 pub folder<\/p>\n<pre lang=\"bash\"># cp \/etc\/logstash\/logstash-forwarder.crt \/var\/www\/kibana3\/pub\/\r\n<\/pre>\n<p>20. Next we need to create the input configuration for lumberjack. We will be listening on port 5000<\/p>\n<pre lang=\"yaml\">\r\n# cat > \/etc\/logstash\/conf.d\/01-lumberjack-input.conf << EOF\r\ninput {\r\n  lumberjack {\r\n    port => 5000\r\n    type => \"logs\"\r\n    ssl_certificate => \"\/etc\/logstash\/ssl\/logstash-forwarder.crt\"\r\n    ssl_key => \"\/etc\/logstash\/ssl\/logstash-forwarder.key\"\r\n  }\r\n}\r\nEOF\r\n<\/pre>\n<p>21. Finally lets write the filter for the evm and automate.log. You will have a tag task_id and miq_msg<\/p>\n<pre lang=\"yaml\">\r\n# cat > \/etc\/logstash\/conf.d\/11-miq.conf << EOF\r\nfilter {\r\n  if [type] == \"miqautomation\" {\r\n    grok {\r\n      patterns_dir => \"\/opt\/logstash\/pattern\" \r\n      match => { \"message\" => \"%{DATE}T%{TIME}%{SPACE}#%{WORD}:%{WORD}%{NOTSPACE}%{SPACE}%{RUBY_LOGLEVEL}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{WORD:task_id}%{NOTSPACE}%{GREEDYDATA:miq_msg}\" }\r\n    }\r\n  }\r\n}\r\nEOF\r\n<\/pre>\n<p>22. Last we need to tell logstash where to send its data too.<\/p>\n<pre lang=\"yaml\">\r\n# cat > \/etc\/logstash\/conf.d\/30-lumberjack-output.conf << EOF\r\noutput {\r\n  elasticsearch { host => localhost }\r\n  stdout { codec => rubydebug }\r\n}\r\nEOF\r\n<\/pre>\n<p>23. Restart and enable logstash to start on boot<\/p>\n<pre lang=\"bash\"># chkconfig logstash on\r\n#service logstash restart\r\n<\/pre>\n<p>24. We can now prepare the client configs. Cd to \/var\/www\/html\/kibana3\/pub\/ and download the following files<\/p>\n<pre lang=\"bash\"># cd \/var\/www\/html\/kibana3\/pub\r\n# wget http:\/\/download.elasticsearch.org\/logstash-forwarder\/packages\/logstash-forwarder-0.3.1-1.x86_64.rpm\r\n# wget http:\/\/logstashbook.com\/code\/4\/logstash_forwarder_redhat_init\r\n# wget http:\/\/logstashbook.com\/code\/4\/logstash_forwarder_redhat_sysconfig\r\n<\/pre>\n<p>25. In the same directory add this script<\/p>\n<pre lang=\"yaml\">\r\n# cat > \/var\/www\/html\/kibana3\/pub\/logstash-forwarder-installer.sh << EOF2\r\n#!\/bin\/bash\r\nwget -P \/tmp\/ --user=admin --password=$HT_PASS http:\/\/$LG_SERVER\/pub\/logstash-forwarder-0.3.1-1.x86_64.rpm\r\nyum -y localinstall \/tmp\/logstash-forwarder-0.3.1-1.x86_64.rpm\r\nrm -f \/tmp\/logstash-forwarder-0.3.1-1.x86_64.rpm\r\nwget -O \/etc\/init.d\/logstash-forwarder --user=$HT_PASS --password=yourpassword http:\/\/$LG_SERVER\/pub\/logstash_forwarder_redhat_init\r\nchmod +x \/etc\/init.d\/logstash-forwarder\r\nwget -O \/etc\/sysconfig\/logstash-forwarder --user=admin --password=$HT_PASS dhttp:\/\/$LG_SERVER\/pub\/logstash_forwarder_redhat_sysconfig\r\nwget -P \/etc\/pki\/tls\/certs\/ --user=admin --password=$HT_PASS http:\/\/$LG_SERVER\/pub\/logstash-forwarder.crt\r\nmkdir -p \/etc\/logstash-forwarder\r\ncat > \/etc\/logstash-forwarder\/logstash-forwarder.conf << EOF\r\n{\r\n  \"network\": {\r\n    \"servers\": [ \"$LG_SERVER:5000\" ],\r\n    \"timeout\": 15,\r\n    \"ssl ca\": \"\/etc\/pki\/tls\/certs\/logstash-forwarder.crt\"\r\n  },\r\n  \"files\": [\r\n    {\r\n      \"paths\": [\r\n        \"\/var\/www\/miq\/vmdb\/log\/evm.log\"\r\n      ],\r\n      \"fields\": { \"type\": \"miqautomation\" }\r\n    }\r\n   ]\r\n}\r\nEOF\r\n\r\nchkconfig --add logstash-forwarder\r\nservice logstash-forwarder start\r\nEOF2\r\n\r\n\r\nchkconfig --add logstash-forwarder\r\nservice logstash-forwarder start\r\nEOF2\r\n<\/pre>\n<p>26. Login to the cloudforms appliance, download and run the logstash-forwarder-installer.sh<\/p>\n<pre lang=\"bash\"># wget --user=admin --password=yourpassword http:\/\/elk.local.domb.com\/pub\/logstash-forwarder-installer.sh\r\n# chmod +x logstash-forwarder-installer.sh\r\n# .\/logstash-forwarder-installer.sh\r\n<\/pre>\n<p>27. Start the logstash forwarder<\/p>\n<pre lang=\"bash\"># service logstash-forwarder start\r\n<\/pre>\n<p>28. You just finished the setup. Login to your elk server. You should already have data which you can work with on the logstash server.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Cloudforms \/ ManageIQ has a very powerful automation engine. There is basically nothing you can&#8217;t do with it. You could even trigger an api call to make coffee during a provisioning run. But what if your provisioning fails? Every step which is executed by the cloudforms automation engine gets logged into \/var\/www\/miq\/vmdb\/log\/evm.log or \/var\/www\/miq\/vmdb\/log\/automation.log. As&#8230;<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_themeisle_gutenberg_block_has_review":false,"footnotes":""},"categories":[37],"tags":[29,27,30,31,28,26],"class_list":["post-810","post","type-post","status-publish","format-standard","hentry","category-cloudforms","tag-cloudforms","tag-cloudforms-manageiq","tag-elasticsearch","tag-kibana","tag-logstash","tag-rhel7"],"_links":{"self":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts\/810","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=810"}],"version-history":[{"count":66,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts\/810\/revisions"}],"predecessor-version":[{"id":883,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts\/810\/revisions\/883"}],"wp:attachment":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=810"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=810"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=810"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}