Container Metrics And Introspection With CloudForms 4.0 And OpenShift 3.1 – Updated for OpenShift 3.2 and CF 4.1

If you were wondering how CloudForms 4.0 and OpenShift 3.1 work together then you are at the right place. This post is about the integration of CloudForms 4.0 and OpenShift 3.1. I will describe how-to install OSE 3.1 and how-to configure it so that you can connect CloudForms with OpenShift 3.1. My steps here are for a small POC and by no means production. The goal is that CloudForms will discover the OpenShift environment and collect metrics of the containers as well as package information through smart state analysis/introspection. For introspection to work properly you CloudForms appliance needs to have the smart-proxy role enabled.

I wrote 2 scripts which do the whole work for you. If everything works fine you should be able to do the same as I show in the following video

If you don’t wont to do this step by step here is the github repo to it.

https://github.com/ldomb/buildoseforcfme

Step 1. This will prepare the master and nodes for the ose install.

Step 2. Execute prepforeose.sh. You will have to enter the root password during ssh-copy-id.

Step 3. The buildoseforcfme.sh script runs ansible to install OSE v3 and creates all the user/service accounts so that you can connect to OpenShift from CloudForms.

Step 4. Execute the buildoseforcfme.sh script. Make sure to visit the hawkular URL in the browser and accept the cert for https://$HAWKULARFQDN/hawkular/metrics and https://$HAWKULARFQDN:5000 . After the install you will find the needed token for CloudForms in /root/cfme4token.txt. You can add it to your CloudForms OpenShift Provider.

Screen Shot 2016-07-20 at 8.35.52 AM

Step 5. For Hawkular add the hawkular URL. If this is not your master like in this example you will have to point to the node where hawkular is running or in a HA setup to the load balancer which forwards the port 443.
Screen Shot 2016-07-20 at 8.36.06 AM

Step 6. In CloudForms make sure that all the metrics collection check-boxes are enabled under Configure->Configuration->Server-Server Control.
Screen Shot 2016-07-22 at 10.58.26 AM

7 thoughts on “Container Metrics And Introspection With CloudForms 4.0 And OpenShift 3.1 – Updated for OpenShift 3.2 and CF 4.1

  • As of OSE-3.1.1.6, the mount-host defined (when creating registry) should have write privs for group and other.
    $ chmod a+rwx /images

  • Thanks for this write-up! I had a couple issues with this, but I got everything up and running now. I had missed the check boxes in the CloudForms configuration before I saw your post here.

    Quick question though. My OSE container provider has been added and is working fine, but I still don’t see utilization metrics. I have verified that I can get to the hawkular metrics page by visiting the master at https://master.fake.home/hawkular/metrics, but nothing shows up in CloudForms. Any suggestions?

    Thanks.

    • Hi,

      The metrics on the overview page take about a half a day to actually show any data. Are you able to get to your hawkular interface via https://$MASTERFQDN:5000/? This is currently necessary for CloudForms to be able to get the metrics.

      Also when you go into oc project openshift-infra and do a oc get pods, do you see?
      [root@osemaster ~]# oc get pods
      NAME READY STATUS RESTARTS AGE
      hawkular-cassandra-1-f2qx9 1/1 Running 1 3d
      hawkular-metrics-160ne 1/1 Running 1 3d
      heapster-eszkv 1/1 Running 10 3d
      metrics-deployer-pr2jb 0/1 Completed 0 3d

      • Hi Laurent,

        Yes, I accidentally left out the “:5000” in my comment, but I was able to get to hawkular metrics using the master url with port 5000 (https://master.fake.home:5000/hawkular/metrics).

        I didn’t use your script to install, and I was trying to gather metrics via cloudforms on my existing OSE setup, and I finally figured out that the issue was due to a certificate problem. I think your install script probably fixes this issue since you’re specifying certificate files from a CA that’s already trusted by your system. I had to go to the hawkular metrics page, export the certificate, and import it to my system as a trusted CA. Once I did that the metrics started working.

        As a note to anyone else who might encounter this issue (cloudforms not displaying metrics), first go to a pod in the OSE gui and check if the metrics tab exists. If it does, click on it. If you are presented with an error, you can begin troubleshooting from there:
        -No errors – probably an issue with the metrics not being presented on port 5000 of the master
        -Error stating that metrics cannot be gathered – probably an issue with certificates
        -Error stating that data can’t be collected for a time later than the current time – time configuration issue on OSE masters/nodes

        Anyway, thanks for this post! It helped point me in the right direction to figuring out my issue. I ran across it when googling to try to troubleshoot why the metrics wouldn’t show in cloudforms. Eventually I found that the metrics weren’t showing in OSE either.

        Also, for others who have this issue, the metricsPublicURL must be specified with the port number, and you must restart the atomic-openshift-master-api service before it will recognize the changes to the master config. Also for multi-master setups, ensure that you point cloudforms to a specific master (not the ha-proxy url), and point all masters’ “metricsPublicURL” to the master with the router running on it.

        And finally, to ensure that other pods don’t run on my master, I bring it up as unschedulable, allow all pods to be placed other places, make it schedulable to allow the router to start, then label it again as unschedulable. This prevents other services from placing their pods on the master.

        Thanks again Laurent. I’ll probably be following your blog from now on to see what other OSE info you provide. ๐Ÿ™‚

        • I saw this one a few times already “Error stating that data canโ€™t be collected for a time later than the current time โ€“ time configuration issue on OSE masters/nodes”

          Most of the time it was because NTP on CloudForms and NTP on OSE was not in sync. OSE was ahead in time.

  • Hello Laurent,

    Thanks for the great post and the scripts. I am running the OSE v3.2 and cloudforms v4.1. All the pods under the project openshift-infra are running fine & even i have made an entry in the master-config.yaml for “metricsPublicURL: https://$MASTERFQDN/hawkular/metrics” in /etc/origin/master/master-config.yaml. Once it was done i have restarted the atomic-openshift-master and it went successfully. i have used the token to authenticate cloudforms with openshift and that too worked well and all the pods , projects, routes information are shown on the cloudforms console.

    As per your instruction i have not deployed “router management-metrics” as i am using the OSEv3.2 and CFv4.1″

    Issue which i am facing are :-

    I am not able to open the Hawkular console: https://$MASTERFQDN/hawkular/metrics due to which i am not able see any graphical data.

    Even in the cloudforms console i am not able to see the Utilization graphical data.

    Can you please help to fix this issue.

    [root@ ~]# oc get pods
    NAME READY STATUS RESTARTS AGE
    hawkular-cassandra-1-gc3p7 1/1 Running 0 49m
    hawkular-metrics-zf9qw 1/1 Running 0 49m
    heapster-i96gc 1/1 Running 0 49m
    metrics-deployer-15q6c 0/1 Completed 0 50m

  • can you do an oc describe pod hawkular-metrics-zf9qw ? Where is this running? You should be able to get to the url based on what I see.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.


Hit Counter provided by laptop reviews