How CloudForms Performs OpenScap Image Scans

In May 2015 banyan published that they found over 30% of Official Images in Docker Hub Contain High Priority Security Vulnerabilities.

In August 2015, FlawCheck surveyed enterprises asking which piece of the security equation was their top concern about running containers in production environments. At 42%, Vulnerabilities & Malware in container workloads was the top container security concern among those surveyed.

In December 2016 a major vulnerability CVE-2016-9962 (“on-entry vulnerability.”) was found in the Docker Engines which allowed local root users in a container to gain privileges by using ptrace to access file-descriptors of a process launched or moved into the container from another namespace.

At Red Hat we are continuously improving our view on security and introduced a new container scanning feature with CloudForms 4.2 and OpenShift 3.4 which allows CloudForms to flag an image in the docker registry after it found vulnerabilities and OpenShift will deny its execution next time someone will try to run that image.
CloudForms has multiple capabilities on how a docker scan can be initiated

    • scheduled scan of the registry
    • scan based on newly discovered images in the Registry
    • manual execution of the scan via smart state analysis

Having this unique scanning feature with native integration in OpenShift is milestone in perspective of container security as you will have near real time monitoring of your images within the OpenShift environment.

This diagram is showing the flow as well as how we perform a scan on containers in the Red Hat OpenShift environment

  1. CloudForms monitors the Openshift Provider and checks if it finds new images in the registry. If it finds a new image Cloudforms Triggers a scan
  2. CloudForms makes an encrypted call to OpenShift via bearer token (which was created during the initial configuration) and requests a manageiq-img-scan container to be scheduled in the management-infra project
  3. OpenShift schedules a new pod on an available node
  4. The manageiq-img-scan gets started
  5. The manageiq-img-scan scanner check if the image to scan is available in the registry and pull is down
  6. The image to scan is unpacked and its rpm content registered and sent to cloudforms
  7. As we enabled the OpenScap policy CloudForms also initiates the OpenScap scan
  8. After the OpenScap scan finishes the results get uploaded and a new report can be generated from the CloudForms UI
  9. If the scan found any vulnerabilities the CloudForms calls OpenShift to annotate the OpenShift image with images.openshift.io/deny-execution=true and prevents future images from running

The next time someone will try to start the vulnerable image OpenShift will alert the user that the image execution was blocked based on the policy annotation set by CloudForms.

 

Posted in CloudForms, OpenShift | Tagged , , , , , , | Leave a comment

Ansible for Juniper EX2200-c

In my home lab I have 4 Juniper in a virtual chassis. I currently use fedora 24 as my host system so I had to install some binaries and fix some code before I got this running. First you have to install the latest stable ansible version.

Then you need junos-eznc. For fedora run the following command

Once I’ve installed the so called pre req I run the following ansible playbook.

After ansible run I got the following error message:

Next step was to install jxmlease

After that I run the playbook again and ran into another hurdle.

A quick google search for “msg: CTR mode needs counter parameter, not IV)” revealed that the file /usr/lib/python2.7/site-packages/paramiko/transport.py needs patching as the value of IV should be an empty string for this to work. So on line 1649 patch the following:

After that I was successful in running the playbook.

Posted in ansible | Tagged , , | Leave a comment

OpenShift 3.4 Standalone Registry (Atomic Registry)

I realized today that the current OpenShift 3.4 documentation around installing the standalone registry is missing an important parameter. Here is the link to the bug openshift_master_default_subdomain missing . Its a small error but has some impact as the route registry-console-default.yourdomain will not be exposed externally.

Remember the standalone OpenShift registry is a full OpenShift installation thus it will need a DNS wildcard entry pointing to it for the subdomain defined in openshift_master_default_subdomain.

I used the following ansible hosts file to install the registry

If you followed the redhat documentation around requirements you should be good to go. You can know run the

Once the install is done you can check with oc get routes if your route was created correctly

You can know browse to https://registry-console-default.reg.local.domb.com and you should see the following UI (if you did not configure any users yet you will not be able to login!)

If you chose htpasswd_auth create a user with htpasswd /etc/origin/master/htpasswd admin and login. Once logged in you will see the atomic registry.

Then you can create a project and upload an image and you should see something like this

Posted in OpenShift | Tagged , , , , | Leave a comment

Deny container image execution via CloudForms 4.2 / OpenShift 3.4

I’ve been waiting for this feature since quite a while and its finally here and working. CloudForms 4.2 and OpenShift 3.4 have the ability combined of scanning docker images and define if the images are compliant or not. If the image is not compliant CloudForms annotates the image in OpenShift with images.openshift.io/deny-execution: true and if OpenShift is configured correctly it will deny execution the next time someone tries to run the container image.

Having this feature is really awesome as you can prevent someone from building a vulnerable image and deploying it multiple times. Also CloudForms can either have a policy set on the provider which auto scans newly discovered pods/images, schedule image registry scans or your check images on demand.

So here are the steps to get this working (a detailed description on what an image policy is can be found here: https://docs.openshift.com/container-platform/3.4/admin_guide/image_policy.html)

1. First  login to the OpenShift Master and edit /etc/origin/master/master-config.yaml Add the following lines above apiLevels

2. Restart the OpenShift masters so that the policy will take effect.

3. Deploy a vulnerable image in OpenShift. In my case I named it testme.

4. Go to CloudForms and run a Smart State Analysis. Login to CloudForms and go to Compute -> Containers -> Container Images and choose the image you’ve just deployed via OpenShift.

5. Click on the image.  Then on the top left press “Perform Smart StateAnalysis”

6. If your admin in OpenShift you can go to the management-infra project and see that a new pod manageiq-img-scan is started (oc get pods). The image will pull your testme image down and check it for vulnerabilities. Once scanned (remember we did not add any policies yet) you will see the following:

7. Go to policy -> manage policy and check the box for OpenScap Profile Once checked go back to the policies and run “Check Compliance of last known configuration.

8. As I used a image which has known vulnerabilities the compliance status will show Non-Compliant.
What happens in the background as well is that CloudForms tells OpenShift to annotate the image with images.openshift.io/deny-execution: true

9. Go to your image id and copy your sha

10. Now in OpenShift check if the annotation exist. You can either check in the UI in OpenShift under image -> annotations or on the command line

11. Lets see if the policy catches and build a new container based of the above. As you can see below OpenShift denies running the above image!

12. If you want to remove the restriction your can use

Happy OpenShifting

Posted in Cloud, OpenShift | Tagged , , , , | Leave a comment

Red Hat Cloud Suite In A Box

Today I finally finished my Ikea Helmer rack project. The goal of the project was to have enough resources to build a full Red Hat Cloud Suite lab. I am in IT now since 1996 and learned  that the only way to stay on top of the game is by playing with the technology. I have to say it took quite a lot of dremeling to cut the Ikea Helmer rack into shape so that I could fit motherboards, fans, disks and power supplies etc. The cool thing about the Helmer rack is that it fits a mini itx mother board perfectly and if you add switches on the bottom you have a self contained rack. With two fans blowing air in and one fan blowing air out you also achieve a good air flow.

Here is a list of the components used in my lab:
2 x APC Smart UPS 1500W
9 x Antec Earthwatts 380
27 x Gelid FN-PX08-20 silent 8 PWN 80mm
3 x SuperMicro X10SDV-6C+-TLN4F-O for Red Hat Virtualization Hyper-converged with 64 GB RAM which will host
Satellite 6, Ansible Tower, CloudForms and OpenShift
6 x SuperMicro X10SDV-4C-TLN4F-O for Red Hat OpenStack (3 controller 2 compute 1 ceph) or (3 controller 3 Compute hyper converged)
1 x Asustor 608t (NFS and ISCSI storage)
4 x Juniper ex2200-c (fanless) (virtual chassis)
1 x Netgear XS708E-200nes
15 x SSD in different sizes
1 x Raspberry PI as a DNS and pxe server for the main deployment

In the next few weeks I will start building out the RHCS components and write blog posts on how to deploy the different products via Red Hat Management Suite automatically.

Posted in ansible, Cloud, CloudForms, Linux, OpenShift, Openstack, Uncategorized | Tagged , , , , | Leave a comment

Red Hat Summit 2017 S102320: Button Push Deployments With Integrated Red Hat Open Management

My 2017 redhat summit talk Button Push Deployments With Integrated Red Hat Open Management got accepted. I have the big honor of sharing the talk not just with my last years co presenter Michael Dahlgren but also with Red Hat EMEA’S configuration management extraordinaire Maxim Burgerhout.

As we combine our talks you can expect an awesome mixture of different demos and showcases around Red Hat Open Management.

Posted in CloudForms, Linux, OpenShift, Puppet | Tagged | Leave a comment

CloudForms AWS IAM policies

To enable Cloudforms to connect to the AWS provider you need to following policies:

  • AmazonEC2FullAccess
  • AWSConfigUserAccess
  • IAMReadOnlyAccess
  • IAMUserSSHKeys
  • AWSCloudFormationReadOnlyAccess

As well as two custom policies

CFMESNS

CFMESQS

Posted in Uncategorized | 1 Comment

Deploy and build containers on Red Hat OpenShift Container Platform 3.4 via CloudForms 4.2 self service

I had multiple requests if CloudForms is able to provision containers via self service to the Red Hat OpenShift container platform. I liked the idea as you can not expect from every developer in your company to know how to create templates or build configs. I will walk you through, step by step how this can be achieved with CloudForms and it’s native integration into Ansible Tower.

Step 1 Create a new project in Ansible Tower and add https://github.com/ldomb/ansible_tower_roles as the giturl this will download the role / playbook which I wrote to prove the above case.

newproject

Step 2 I assume that you already have an inventory added to your tower instance. Let’s create the Job Template so we can launch containers / pod in OpenShift. Add the following values to the job template

selection_071

Step 3 For extra_vars add the following
projectname: test
template: nodejs-example
apphostname: mynodejsapp
templatename: phptemplate
createtemplate: false
quickapp: false
sti: false
git_url: https://github.com/dudash/openshift-workshops.git
url_to_template: https://github.com/dudash/openshift-workshops.git
scaleme: false
autoscale: false
username: laurent
pass: redhat
contextdir: .
number_of_instances: 1
ochost: https://osemaster.rdu.salab.redhat.com
ocport: 8443
appdomain: apps.rdu.salab.redhat.com

Do not forget to check to box Prompt on launch and save tho job template

Step 4 Login to CloudForms and go to the tab configuration management -> Ansible Job Templates. You should see the following. Generate a catalog item from the octools template and name it ose_sti.
selection_072

Step 5 Then go to Automate -> Customization -> Service Dialogs and click on the ose_sti catalog item. For this example we can delete some of the fields as they are not needed for source to image. Make sure your catalog item has the same fields as I am showing in the picture below.

selection_073

Step 6 Set the STI value to true as I am filtering for it in the ansible playbook. This field is read only.
selection_074

Step 7 For autoscale create a radio button with the values true or false. If you set it to true it will automatically create a horizontal pod autoscaler based on cpu load.
selection_076

Step 8 The Limit for the job needs to be read only as well. I choose it to be the master as the master has the oc tools installed.
selection_077

Step 9 For my OpenShift installation I am using basic auth. For this demo to work you need create a user in OpenShift matching the user loggedin in CloudForms.

Login to your openshift master and add a user to the htpasswd file

$ sudo htpasswd /etc/origin/htpasswd username

Remember the password.

Step 10 Go back to CloudForms. For the username field I’ve created a new method based on Kevin Moreys CloudForms Dropdown list https://github.com/ramrexx/CloudForms_Essentials in CloudForms_Essentials -> Integration -> RedHat -> CloudForms -> Dynamic dialog -> get_loggedin_user

selection_078

dialog_hash = {}

users = $evm.vmdb(:user).all

users.each do |u|
if u.id == $evm.root[‘user’].id
dialog_hash[u.userid] = “#{u.userid}”
end
end

$evm.object[“values”] = dialog_hash
$evm.log(:info, “$evm.object[‘values’]: #{$evm.object[‘values’].inspect}”)

Step 11 Wire the field to be a dynamic dropdown and add the following values.
selection_079

Step 12 The password field needs to be protected and match the one you’ve create with htpasswd on OpenShift. In reality you would have connected your CloudForms appliance and OpenShift to LDAP and grabbed the encrypted password from CloudForms instead of entering it.
selection_089

Step 13 For the email address I used another method in the CloudForms essentials domain as I want to send an email to the developer once his application is deployed. Create new method called get_loggedin_user_email

selection_082

dialog_hash = {}

users = $evm.vmdb(:user).all

users.each do |u|
if u.id == $evm.root[‘user’].id
dialog_hash[u.email] = “#{u.email}” if u.email
end
end

$evm.object[“values”] = dialog_hash
$evm.log(:info, “$evm.object[‘values’]: #{$evm.object[‘values’].inspect}”)

 

The field email address should look like this

selection_081

Step 14 To be able to send a user a customized email message I copied the ConfigurationManagement->AnsibleTower->Service->Provisioning->Email->ServiceProvision_Complete instance and its method ServiceProvision_Complete and renamed the method to ServiceProvision_Complete_OSE. Your instance should look like this.
selection_083

Step 15 Create the email method ServiceProvision_Complete_OSE and add the following content
selection_085

#
# Description: Email the user once provisioning is done.
#

# Setup variables from dialog
user_name = $evm.root[‘dialog_param_username’]
user_mail = $evm.root[‘dialog_user_mail’]
apphostname = $evm.root[‘dialog_param_apphostname’]
giturl = $evm.root[‘dialog_param_git_url’]
projectname = $evm.root[‘dialog_param_projectname’]

# Set up Static variables
from = $evm.object[‘from_email_address’]

#form the messsage
subject = “Container Provisioning Request Complete”

body = “Hello #{user_name}”
body += ”

Your Application #{apphostname} is up and running.”
body += ”

We used the following git_url to build the app #{giturl}”
body += ”

You can reach your app at http://#{apphostname}-#{projectname}.apps.yourdomain.com
body += ”

Go back to work!
body += ”

Please be aware that you get billed for this application”
body += ”

Thank you,”
body += ”

The Cloud Team”

# Send email
$evm.log(“info”, “Sending email to from subject: “)
$evm.execute(:send_email, user_mail, from, subject, body)

Step 16 Wire up your email field and we are done with configuring the catalog item
selection_086

Step 17 Create a new StateMachine named default_ose for Email Owner add

/ConfigurationManagement/AnsibleTower/Service/Provisioning/Email/ServiceProvision_complete_Ose?event=service_provisioned

This will send off the email once you deployed the container

Step 18 Login to your cloudforms self service portal https:///self_service and click on the ose_sti tile which will lead you to the catalog item.
selection_087

Fill out the fields in the catalog item, add it to the shopping cart and order it.
selection_088

You can know check the Ansible Tower and see that your job got kicked off
selection_091

Step 19 After a few seconds you will see that OpenShift starts up the pods with your container in it.
selection_092

Step 20 If your pod started successfully you should see this
selection_093

Step 21 Go back to CloudForms and verify that you also see the newly created pod.
selection_094

and its services

selection_095

Step 22 In the meantime you also should have received an email saying that the container provisioning was complete.
selection_096

Posted in ansible, CloudForms, OpenShift | Tagged , , | Leave a comment

Register hosts to Satellite6 via CloudForms and Ansible Tower part 3

In the first two parts of the blog series you’ve learned how you can create an inventory synchronizing Ansible Tower with CloudForms, create a Job Template in Ansible Tower and auto generate a service catalog item in CloudForms from the Job Template under the configuration management tab. You’ve also learned how you can create a new catalog and add the catalog item to it. Part 3 in the blog series is going to show you how you can create a custom button under the VM instances tab.

Step 1 Click on Automate
selection_026
Customization
selection_027
Buttons
selection_028

Step 2 Select VM and Instances
selection_029
Configuration -> Add new button Group
selection_030
and add the following values
selection_037
click add on the bottom right
addcfme

Step 3 Scroll to your new button group and select it.
selection_035
Go to Configuration Add new Button tab
selection_033
and enter the following values
selection_034

Once done press add on the bottom right and your done.
addcfme

Step 4 I will now show you where you can find the button. Go to Compute
selection_038
Infrastructure
selection_039
Virtual Machines
selection_040

Step 5 Select any of your VM instances
selection_041

Step 6 In the menu bar you will see your button group AnsibleTowerJob and your button registersat6
selection_042
Press the registersat6 button which will load the sat6register service dialog.
selection_043

In part 2 we create a service dialog which can be ordered from the self service ui. There is one big difference between both dialogs. If you submit the dialog from a button, the Limit (‘hostname’) of the select host is automatically passed as a variable to the CloudForms automate engine. Therefore the limit can be left empty while ordering from a button. This is not the case when ordering from a self service dialog.

Posted in ansible, CloudForms, Puppet | Tagged , , , | Leave a comment

Register hosts to Satellite6 via CloudForms and Ansible Tower part 4

This post will walk you through how to customize the service dialog which we created in the previous post so that we pull the following values via api calls from Satellite6:
– Activation Keys
– Organization
– Location
– Hostgroups

The end goal will look like this
selection_045
Step 1 First you will need to pull my automation model which contains the functions to connect to satellite6 and get the necessary information.

Step 1 Download the automate model for satellite6 from the managiq depot so you get access to the api calls to Red Hat Satellite 6 for the dynamic drop downs.
https://depot.manageiq.org/extensions/ldomb/miq-redhat-sat6
Here also a direct link to the download
https://depot.manageiq.org/extensions/ldomb/miq-redhat-sat6/download

Follow the instructions in the README.md to upload the automate domain into CloudForms.

Step 2 As the domain will be imported as disabled
selection_047
We will have to go and enable it.
Click on Configuration -> Edit this Domain
selection_048
and check the box Enabled
selection_049
Save the Domain name
screen-shot-2016-09-20-at-5-30-44-pm

Step 3
You should now see the following
selection_046

Step 4 Edit the Schema to fit your configuration. Select DynamicDropDowns -> and the Schema tab.
selection_050

Click on Configuration -> Edit selected Schema
selection_051

Step 5 Edit the following fields
username: your admin user
password: your admin password
sat6url: https://yoursat6fqdn/api
katellourl: https://yoursat6fqdn/katello/api
selection_052

Click the save button
screen-shot-2016-09-20-at-5-30-44-pm

Step 6 You are now ready to start pointing your service dialog fields to the automate methods. Go to Automate
selection_026
Customization
selection_027
Service Dialogs
selection_053

Select your service catalog item which we created in the previous blog posts.
selection_054

Step 7 Click the Configuration -> Edit this Dialog
selection_055
We want to re arrange the order of the fields a little bit. Click the Extra_Vars value on the left
selection_063
Drag and drop the fields on the right so they will look like mine.

Do not hit save yet!

Step 8
Select the org field
selection_056
We will switch the Type of the filed from Text Box to Drop Down List. Check the Dynamic box which will give you the ability to select the Entry point for that field ( /Integration/RedHat/Satellite6/DynamicDropDowns/List_Organizations )
selection_057

We also want the Show Refresh Button box checked and the values loaded on init. As the organization defines the activation keys we want to Auto Refresh other fields when modified as well.
selection_059

Step 9 Select activationkey and change the values to the following
selection_064

Step 10 Select location and change the fields to match the picture
selection_065

Step 11 Select hostgroup and change the values to reflect the picture below.
selection_066

Step 12 We will update the updatehost filed from a text field to radio buttons.
selection_067

Now you can hit the save button.
screen-shot-2016-09-20-at-5-30-44-pm

Step 13 Congratulations. Your done. when you go know to your service dialog or custom button you will see the following
selection_045

Posted in ansible, CloudForms | Tagged , , | Leave a comment