CloudForms in AWS part 3

This will be the last part of the blog series but probably the most interesting one. When you launch a smartstate analysis you will see the following in you evm log files.

[----] I, [2018-03-11T16:12:10.941498 #3004:4f7130]  INFO -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Vm#raw_scan) NAME [Scan from Vm vmtosmartstate] SCAN [nil] [NilClass]
[----] I, [2018-03-11T16:12:11.068516 #3004:4f7130]  INFO -- : Job created: guid: [b0c7c318-2844-436c-bc44-21a29dfa3cce], userid: [admin], name: [Scan from Vm vmtosmartstate], target class: [VmOrTemplate], target id: [99000000000166], process type: [VmScan], server id: [], zone: [default]
[----] I, [2018-03-11T16:12:15.868847 #3014:4f7130]  INFO -- : MIQ(MiqQueue.put) Message id: [99000000000427],  id: [], Zone: [default], Role: [smartstate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: [600], Priority: [20], State: [ready], Deliver On: [], Data: [], Args: []
[----] I, [2018-03-11T16:12:21.133242 #3004:4f7130]  INFO -- : MIQ(MiqPriorityWorker::Runner#get_message_via_drb) Message id: [99000000000427], MiqWorker id: [99000000000004], Zone: [default], Role: [smartstate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: [600], Priority: [20], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [5.266320951] seconds
[----] I, [2018-03-11T16:13:05.374585 #3424:4f7130]  INFO -- : MIQ(ManageIQ::Providers::Amazon::AgentCoordinator#find_or_create_keypair) KeyPair smartstate-2ae1670a-2fcc-4bbb-abcc-3fa962895096 will be created!
[----] I, [2018-03-11T16:13:05.877668 #3424:4f7130]  INFO -- : MIQ(ManageIQ::Providers::Amazon::AgentCoordinator#deploy_agent) Smartstate agent will be deployed in vpc: [vpc-076ad862], zone: [us-east-1a] subnet: [subnet-3cc6154b]
[----] I, [2018-03-11T16:13:11.145451 #3424:4f7130]  INFO -- : MIQ(ManageIQ::Providers::Amazon::AgentCoordinator#get_agent_image_id) AMI Image: RHEL-Atomic_7.4_HVM_GA-20180104-x86_64-1-Access2-GP2 [ami-d97120a3] is used to launch smartstate agent.
[----] I, [2018-03-11T16:15:55.976224 #2716:4f7130]  INFO -- : MiqServer: local=Y, master=Y, status= started, id=99000000000001, pid=02716, guid=c5b77187-7fc9-426e-b3cb-6f0ce7860c8e, name=EVM, zone=default, hostname=ip-172-31-44-62.ec2.internal, ipaddress=172.31.44.62, version=5.9.0.22, build=20180221205805_f93a675, active roles=automate:database_operations:datab

There are a few interesting parts here. What is the AgentCoordinator? The code can be found here:

/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-9620e26f4381/app/models/manageiq/providers/amazon/agent_coordinator.rb . Its out of scope to discuss the entire ruby code, but it will tell you, how we do it. 

So when you look at the evm log, you see that it says “find_or_create_keypair KeyPair smartstate-2ae1670a-2fcc-4bbb-abcc-3fa962895096″

This means that the key used for the ssh login to the RHEL-Atomic_7.4_HVM_GA-20180104-x86_64-1-Access2-GP2 (ami-d97120a3) is generated on the fly.

Let’s explain how you can get that key so you can SSH into the atomic instance and see whats going on.

vmdb
bin/rails c
$evm = MiqAeMethodService::MiqAeService.new(MiqAeEngine::MiqAeWorkspaceRuntime.new)
$evm.vmdb(:ManageIQ_Providers_CloudManager_AuthKeyPair).all.first
$ssa=$evm.vmdb(:ManageIQ_Providers_CloudManager_AuthKeyPair).find_by_resource_id(99000000000001)
$ssa[:auth_key]
irb(main):007:0> $ssa[:auth_key]
=>
"v2:{UOjuc9OSrBm2AcOU5HmnLi1+7XVumlIlfX2SO7fbe1FyAwt9RGqEqyJD5VSRD4QG8RhoFt0yBe0fk5LCq0RGxgdiuIClv50mePsgTyx7f4cOCT94MKM5s3NkfeEq3BdNCzL+CWji+WSXZZn7zE9l/4ezFW7eesfJzsW6npMlH/SNUBdAgiVZzbsKrHZuP4n5agPBYSX+APf+1KvAOGLYaMREGfw4E7OytKGIfOAD/dSKXRx/bhEadCOvnf6sG6XKYGzi82aa+AMzgexszx/93Ce+Pp6fbODxxXJmoDNyzJEoKyuavDiWBEK2SHMV5t2G8U/pb0mjYvavNCl86gtqOGJQ26bZNuUkJlRoSyErOMyrOOhreq0/2RMJvI4Uy6ekCBggyPFS3iPDduQTNM1tPhZPVAW2H+RB8UKZOtf+Oo+eqfkGvQYlnsJt2a4XET+alDxpDT6TEVRCrMpAUUiJt2B1AG3uQ8wL3RDRMvj54uR/uRsvlFMkO9HYrlcKKrQ4myBa3UbVETNcwT844sYwXEP8thmIVMFwIp03oY5ZQOXoPikD1wJJSHQON2bmIbh/xZ3e2ggPwuLkc9ROSBUxkRK4zXhdezwjI3UfP3LVwujHJVmsIBpUDR7oXw2CUwr0be72MFKoWpmvbXf7E8KZaJz1FNZUoOGpMPz5j3eDetlCEwgjJF2UVElT8+2dLosxN2NeLpFY6vpZXm3erOG6WVLM/v5kiX3V2S2JHaUOZdVRrHvk9yoSF5YhQgn72LlpG+Y4DhT3QOf9HMwzg4A4D6EgkRmZfWdr8ABMMoEFonIkPX08ZwJnr3iNAtam8HCvnGGcs49eTPrAzNG/9zjn6ViS4tvGk9EaYYUuHcWgmOySl2WsezmEdE9oHEC7NkX/MbPdSlr1HxfNAkphjdsvmwSABLhOzFeE25zrftTW3Hf1ui7+tsJztDDio00qtXcT+NY8ySupb2siZcywFZFf3WgRzbOX9yjZv4QG8xPzQ80v4+EbKUuDJJkAHd85kAxZYryHQFi/dQO8O39lhJWog6T58gCteMBnYCvLqnYxr40qwXJ6lp44yoi+EBSH7TpLmsY/AM+YvQH1rqUTbi/1o9PvzwkY3C52WpMFeRlh9XDdg/hWWwcHCJc07dNP2kOCtsR1Te5dhTejsJ9ul2MzAGdRnPwuUz9RrMj0Xj/5X8we+4RPgF+/Z3EpjMOFFzrBbykj003n/pRJE0651a5FbdUkO9QeE3jjj4QGaWAb+3dTbVH3OGVIQ0Ebm31po6G1+9SPyfUSVOmmlHAK4XZybPGNQzsW7uorOAtet9++GAR11mBvuO5g9GSOBGw6EEqN76X1CJ3WFaKwfpoQzDXDC3hOMjahB9cZaWu7zXebJPlKUiZPuIfbB93WCOVjS5TU01McrBsjt0rgsSAgPwVzEAyGVInx1knlQ34SiPvgOZYZuvlr9ln2dAdzfmecSR4na4LwjXJd4B3LhBCDofEtTu21ArdxLu5nHjYg1JiaLm1jL2MJeElFkMtJjH8dKFO/e7rbF6Gtq5aMMP1GNFle2EqW53jJsOF7FLZAa5rTNP2qGKBJyWw6dmOPnszIWrMHrSLEJPTpIr4F0V27rwyknCijIDZG/XENDMRtJHLfpjqajVfnlMDp0+1gHX7GhepBN+1b7l9LaRGavJ9iyyBaiw6kHgp7mseRvceDbTkkOUQHEcNL1CYzxt2QHap7noKtfYa0K3GPCa41pXHwaMrG6jeM3Oi0KaoXTBmlfqeDKtqJ1YOzqQ5+qoVd/Um8nWq/HWvjhtoEK7K4zXaEsaVBlPeqrDqUY+sppbvKENXRRfotim9QG+b5ujINRCLl95HSNIQ5J6zDjYPqDToQuRF72L2SdoDHNSotcSzMDf3m91Idoec5oV8M+jPwqjm7BgzCK7RxW8aAAgh0Ht6j+QHYr8Isg00sCDFBrMG9P6nxbZsipeIklTjyCUWFA8rFPBMDfETFhNCkcpZ7TQqZsQtzBgc3gbgIemXw2ORSKvO6RB++b3s81Xj4OPgwG6cH/rvNGvf1ZsE3ONQFXCoVx0ZCXhdiLcvqovSrMctJKFzZdAVTSLKBARc7nc2JQx3Tam+29nn3cTvHH++zx52roTmpO2eI3gWkqpM8TEYAWQVLK8KaK33WCbu7Q6/iCOEKuHPz9v2L1oJBt/7zpU8wxbfafcxOzYuUTbw74EykGzUVaDUuSCbh14oZO9Z1vuBxWO8U}"

In my case its the first key as I did not create another one yet.

Then go ahead and decrypt that key.

MiqPassword.decrypt("v2:{UOjuc9OSrBm2AcOU5HmnLi1+7XVumlIlfX2SO7fbe1FyAwt9RGqEqyJD5VSRD4QG8RhoFt0yBe0fk5LCq0RGxgdiuIClv50mePsgTyx7f4cOCT94MKM5s3NkfeEq3BdNCzL+CWji+WSXZZn7zE9l/4ezFW7eesfJzsW6npMlH/SNUBdAgiVZzbsKrHZuP4n5agPBYSX+APf+1KvAOGLYaMREGfw4E7OytKGIfOAD/dSKXRx/bhEadCOvnf6sG6XKYGzi82aa+AMzgexszx/93Ce+Pp6fbODxxXJmoDNyzJEoKyuavDiWBEK2SHMV5t2G8U/pb0mjYvavNCl86gtqOGJQ26bZNuUkJlRoSyErOMyrOOhreq0/2RMJvI4Uy6ekCBggyPFS3iPDduQTNM1tPhZPVAW2H+RB8UKZOtf+Oo+eqfkGvQYlnsJt2a4XET+alDxpDT6TEVRCrMpAUUiJt2B1AG3uQ8wL3RDRMvj54uR/uRsvlFMkO9HYrlcKKrQ4myBa3UbVETNcwT844sYwXEP8thmIVMFwIp03oY5ZQOXoPikD1wJJSHQON2bmIbh/xZ3e2ggPwuLkc9ROSBUxkRK4zXhdezwjI3UfP3LVwujHJVmsIBpUDR7oXw2CUwr0be72MFKoWpmvbXf7E8KZaJz1FNZUoOGpMPz5j3eDetlCEwgjJF2UVElT8+2dLosxN2NeLpFY6vpZXm3erOG6WVLM/v5kiX3V2S2JHaUOZdVRrHvk9yoSF5YhQgn72LlpG+Y4DhT3QOf9HMwzg4A4D6EgkRmZfWdr8ABMMoEFonIkPX08ZwJnr3iNAtam8HCvnGGcs49eTPrAzNG/9zjn6ViS4tvGk9EaYYUuHcWgmOySl2WsezmEdE9oHEC7NkX/MbPdSlr1HxfNAkphjdsvmwSABLhOzFeE25zrftTW3Hf1ui7+tsJztDDio00qtXcT+NY8ySupb2siZcywFZFf3WgRzbOX9yjZv4QG8xPzQ80v4+EbKUuDJJkAHd85kAxZYryHQFi/dQO8O39lhJWog6T58gCteMBnYCvLqnYxr40qwXJ6lp44yoi+EBSH7TpLmsY/AM+YvQH1rqUTbi/1o9PvzwkY3C52WpMFeRlh9XDdg/hWWwcHCJc07dNP2kOCtsR1Te5dhTejsJ9ul2MzAGdRnPwuUz9RrMj0Xj/5X8we+4RPgF+/Z3EpjMOFFzrBbykj003n/pRJE0651a5FbdUkO9QeE3jjj4QGaWAb+3dTbVH3OGVIQ0Ebm31po6G1+9SPyfUSVOmmlHAK4XZybPGNQzsW7uorOAtet9++GAR11mBvuO5g9GSOBGw6EEqN76X1CJ3WFaKwfpoQzDXDC3hOMjahB9cZaWu7zXebJPlKUiZPuIfbB93WCOVjS5TU01McrBsjt0rgsSAgPwVzEAyGVInx1knlQ34SiPvgOZYZuvlr9ln2dAdzfmecSR4na4LwjXJd4B3LhBCDofEtTu21ArdxLu5nHjYg1JiaLm1jL2MJeElFkMtJjH8dKFO/e7rbF6Gtq5aMMP1GNFle2EqW53jJsOF7FLZAa5rTNP2qGKBJyWw6dmOPnszIWrMHrSLEJPTpIr4F0V27rwyknCijIDZG/XENDMRtJHLfpjqajVfnlMDp0+1gHX7GhepBN+1b7l9LaRGavJ9iyyBaiw6kHgp7mseRvceDbTkkOUQHEcNL1CYzxt2QHap7noKtfYa0K3GPCa41pXHwaMrG6jeM3Oi0KaoXTBmlfqeDKtqJ1YOzqQ5+qoVd/Um8nWq/HWvjhtoEK7K4zXaEsaVBlPeqrDqUY+sppbvKENXRRfotim9QG+b5ujINRCLl95HSNIQ5J6zDjYPqDToQuRF72L2SdoDHNSotcSzMDf3m91Idoec5oV8M+jPwqjm7BgzCK7RxW8aAAgh0Ht6j+QHYr8Isg00sCDFBrMG9P6nxbZsipeIklTjyCUWFA8rFPBMDfETFhNCkcpZ7TQqZsQtzBgc3gbgIemXw2ORSKvO6RB++b3s81Xj4OPgwG6cH/rvNGvf1ZsE3ONQFXCoVx0ZCXhdiLcvqovSrMctJKFzZdAVTSLKBARc7nc2JQx3Tam+29nn3cTvHH++zx52roTmpO2eI3gWkqpM8TEYAWQVLK8KaK33WCbu7Q6/iCOEKuHPz9v2L1oJBt/7zpU8wxbfafcxOzYuUTbw74EykGzUVaDUuSCbh14oZO9Z1vuBxWO8U}")
=> "-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEA4VajXvftedVzvG .......

Copy your RSA private key to cfme.pem and format it correctly:

sed -i 's/\\n/\n/g' cfme.pem

Next you need to fix perms

chmod 400 cfme.pem

If you want you can also compare fingerprints

openssl  pkcs8 -in smartstate-1c6df722-bf49-4e32-ad6c-e0aea84a842a.pem -nocrypt -topk8 -outform DER | openssl sha1 -c
(stdin)= 70:6f:cc:40:10:ea:97:43:26:6e:f5:73:4c:e8:0d:15:e8:59:5a:c1

This should match what you see in the Compute> Cloud -> KeyPairs.

As I assume that you already launched the smartstate analysis. 3 SQS queues were created in the meantime for you as well as an s3 bucket for the heartbeat where it will store the instance id for the agent. If you go to SQS in the AWS console you will see.

Here is the s3 bucket

When running smartstate you will actually have a message in the request queue. This will the CFME to launch the smartstate instance.

As you probably noticed a security group named smartstate was created for you as well.

Now I was wondering if everything on the atomic host is actually kosher. So that’s where our ssh-key comes in

Lets ssh into the atomic host

ssh -i "cfme.pem" ec2-user@ec2-57-88-166-140.compute-1.amazonaws.com

If everything is working correctly you should see the that the cfme-amazon-smartstate image gets pulled.

 

[root@ip-172-31-22-123 ~]# atomic images list

  REPOSITORY                                                       TAG IMAGE ID CREATED VIRTUAL SIZE TYPE   

>  registry.access.redhat.com/cloudforms46/cfme-amazon-smartstate   latest 53c8509ae76a 2018-02-22 16:20 311.44 MB docker

The next step, of course, is that CFME will launch it.

[root@ip-172-31-22-123 ~]# atomic containers list

  CONTAINER ID IMAGE                NAME COMMAND CREATED     STATE BACKEND RUNTIME

  60cb45e8e0d4 registry.access.redh silly_nort /bin/sh -c 2018-03-11 20:18 running    docker docker

If the scan completes successfully you will see vm_scan_complete

[----] I, [2018-03-11T16:20:46.085250 #2978:4f7130]  INFO -- : Q-task_id([b0c7c318-2844-436c-bc44-21a29dfa3cce]) MIQ(MiqEvent#process_evm_event) target = [#<ManageIQ::Providers::Amazon::CloudManager::Vm id: 99000000000166, vendor: "amazon", format: nil, version: nil, name: "vmtosmartstate", description: nil, location: "ec2-54-160-15-44.compute-1.amazonaws.com", config_xml: nil, autostart: nil, host_id: nil, last_sync_on: "2018-03-11 20:20:08", created_on: "2018-03-11 20:08:19", updated_on: "2018-03-11 20:20:42", storage_id: nil, guid: "13f6cdd0-69db-4ce6-b7eb-3f04f38ce739", ems_id: 99000000000001, last_scan_on: "2018-03-11 20:18:45", last_scan_attempt_on: "2018-03-11 20:12:10", uid_ems: "i-0de0a2338f41c40a4", retires_on: nil, retired: nil, boot_time: "2018-03-11 20:03:10", tools_status: nil, standby_action: nil, power_state: "on", state_changed_on: "2018-03-11 20:08:19", previous_state: nil, connection_state: nil, last_perf_capture_on: nil, registered: nil, busy: nil, smart: nil, memory_reserve: nil, memory_reserve_expand: nil, memory_limit: nil, memory_shares: nil, memory_shares_level: nil, cpu_reserve: nil, cpu_reserve_expand: nil, cpu_limit: nil, cpu_shares: nil, cpu_shares_level: nil, cpu_affinity: nil, ems_created_on: nil, template: false, evm_owner_id: nil, ems_ref_obj: nil, miq_group_id: 99000000000001, linked_clone: nil, fault_tolerance: nil, type: "ManageIQ::Providers::Amazon::CloudManager::Vm", ems_ref: "i-0de0a2338f41c40a4", ems_cluster_id: nil, retirement_warn: nil, retirement_last_warn: nil, vnc_port: nil, flavor_id: 99000000000002, availability_zone_id: 99000000000002, cloud: true, retirement_state: nil, cloud_network_id: nil, cloud_subnet_id: nil, cloud_tenant_id: nil, raw_power_state: "running", publicly_available: nil, orchestration_stack_id: nil, retirement_requester: nil, tenant_id: 99000000000001, resource_group_id: nil, deprecated: nil, storage_profile_id: nil, cpu_hot_add_enabled: nil, cpu_hot_remove_enabled: nil, memory_hot_add_enabled: nil, memory_hot_add_limit: nil, memory_hot_add_increment: nil>]
[----] I, [2018-03-11T16:20:46.098293 #2978:4f7130]  INFO -- : Q-task_id([b0c7c318-2844-436c-bc44-21a29dfa3cce]) MIQ(MiqAlert.evaluate_alerts) [vm_scan_complete] Target: ManageIQ::Providers::Amazon::CloudManager::Vm Name: [vmtosmartstate], Id: [99000000000166]
[----] I, [2018-03-11T16:21:01.555276 #2716:4f7130]  INFO -- : MiqServer: local=Y, master=Y, status= started, id=99000000000001, pid=02716, guid=c5b77187-7fc9-426e-b3cb-6f0ce7860c8e, name=EVM, zone=default, hostname=ip-172-31-44-62.ec2.internal, ipaddress=172.31.44.62, version=5.9.0.22, build=20180221205805_f93a675, active roles=automate:database_operations:database_owner:ems_inventory:ems_operations:event:reporting:scheduler:smartproxy:smartstate:user_interface:web_services:websocket

Now you probably wondering, What happens during the smartstate analysis? I’ve created the following diagram for you to understand the magic: