{"id":2818,"date":"2023-11-23T16:38:03","date_gmt":"2023-11-23T21:38:03","guid":{"rendered":"https:\/\/blog.domb.net\/?p=2818"},"modified":"2023-11-24T15:01:12","modified_gmt":"2023-11-24T20:01:12","slug":"reboot-msk-broker-with-aws-fault-injection-service","status":"publish","type":"post","link":"https:\/\/blog.domb.net\/?p=2818","title":{"rendered":"Reboot MSK Broker with AWS Fault Injection Service"},"content":{"rendered":"<p>Customers have asked me how they can use Fault Injection Service to impact Amazon Managed Streaming for apache Kafka by rebooting brokers. In this blog post, I&#8217;ll demonstrate how to do that.<\/p>\n<p><iframe loading=\"lazy\" title=\"Fault Injection Service custom SSM action for Amazon MSK Broker reboot\" width=\"640\" height=\"360\" src=\"https:\/\/www.youtube.com\/embed\/kWXWaBBDukk?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<p>I will give you two flows on how you can reboot the MSK broker via FIS.<\/p>\n<p>Flow 1: <strong>FIS =&gt; SSM =&gt; MSK reboot broker<\/strong><br \/>Flow 2: <strong>FIS =&gt; SSM =&gt; Lambda =&gt; MSK reboot broker<\/strong><\/p>\n<p>Lets create some IAM roles first that are needed for FIS, SSM and Lambda<\/p>\n<p>1. If you do not have created a FIS role yet, you&#8217;ll have to create one. We will start with the fis trust policy.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"json\">aws iam create-role \\\n  --role-name ssm-fis-role \\\n  --assume-role-policy-document '{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n            \"Principal\": {\n                \"Service\": [\n                  \"fis.amazonaws.com\"\n                ]\n            },\n            \"Action\": \"sts:AssumeRole\"\n        }\n    ]\n}'<\/pre>\n<p>2. Having insights into experiments is key. Therefore, we need to ensure that FIS can write to CloudWatch Logs.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"json\">aws iam put-role-policy \\\n  --role-name ssm-fis-role \\\n  --policy-name fis-cloudwatch \\\n  --policy-document '{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n      {\n        \"Effect\": \"Allow\",\n        \"Action\": [\n          \"logs:CreateLogDelivery\",\n          \"logs:PutResourcePolicy\",\n          \"logs:DescribeResourcePolicies\",\n          \"logs:DescribeLogGroups\"\n        ],\n        \"Resource\": \"*\"\n      }\n    ]\n  }'<\/pre>\n<p>3. Next we will add SSM permissions to the FIS role so that it can call SSM Documents. We can attach the <strong>AWSFaultInjectionSimulatorSSMAccess<\/strong> role policy.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\">aws iam attach-role-policy --role-name ssm-fis-role --policy-arn arn:aws:iam::aws:policy\/service-role\/AWSFaultInjectionSimulatorSSMAccess\n<\/pre>\n<p><strong>&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;-<\/strong><br \/>If you intend to use <em><strong>flow 1<\/strong><\/em>, the execution of an MSK broker reboot through FIS =&gt; SSM without <strong>Lambda<\/strong> in the path follow step <strong>4a<\/strong>. For <em><strong>flow 2 <\/strong><\/em>use <strong>4b<\/strong>.<br \/><strong>&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;-<\/strong><\/p>\n<p>4a. Create the SSM role to execute the <strong>MSK broker<\/strong> reboot. <strong>Once created, follow step 9a.<\/strong><\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"json\">aws iam create-role \\\n  --role-name SSMDocumentMskRole \\\n  --assume-role-policy-document '{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n      {\n        \"Effect\": \"Allow\",\n        \"Action\": [\n          \"kafka:DescribeCluster\",\n          \"kafka:RebootBroker\",\n          \"kafka:DescribeClusterOperation\"\n        ],\n        \"Resource\": \"*\"\n          \"arn:aws:kafka:us-east-1:123456789101:cluster\/demo-cluster-1\/85751407-8d09-48b7-af7b-ab25de795414-3\",\n          \"arn:aws:kafka:us-east-1:123456789101:cluster\/demo-cluster-2\/7e173d6b-fc7c-4504-b401-3813ebc6d495-3\"\n      }\n    ]\n  }'<\/pre>\n<p>4b. This role is needed for <strong>flow 2<\/strong> to trigger the broker reboot, we will call lambda via SSM. This role will be used for the SSM Document of the <strong>SSM to Lambda role<\/strong>.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">aws iam create-role \\\n  --role-name SSMDocumentLambdaRole \\\n  --assume-role-policy-document '{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n      {\n        \"Effect\": \"Allow\",\n        \"Action\": [\n          \"lambda:InvokeFunction\",\n          \"lambda:DeleteFunction\"\n        ],\n        \"Resource\": \"*\"\n      }\n    ]\n  }'<\/pre>\n<p>5. To finish role creations for this experiment we will create the <strong>Lambda execution role<\/strong> and add the trust policy.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"json\">aws iam create-role \\\n  --role-name LambdaMskRebootBrokerexecutionRole \\\n  --assume-role-policy-document '{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n            \"Principal\": {\n                \"Service\": [\n                  \"lambda.amazonaws.com\"\n                ]\n            },\n            \"Action\": \"sts:AssumeRole\"\n        }\n    ]\n}'<\/pre>\n<p>6. Next we will add permission to the role so it can make calls to Amazon MSK and Amazon CloudWatch from Lambda.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"json\">aws iam put-role-policy \\\n  --role-name LambdaMskRebootBrokerexecutionRole \\\n  --policy-name msk-fis-cloudwatch \\\n  --policy-document '{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": \"logs:CreateLogGroup\",\n            \"Resource\": \"*\"\n        },\n        {\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"logs:CreateLogStream\",\n                \"logs:PutLogEvents\"\n            ],\n            \"Resource\": [\n                \"*\"\n            ]\n        },\n        {\n            \"Sid\": \"Statement1\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"kafka:DescribeCluster\",\n                \"kafka:RebootBroker\",\n                \"kafka:DescribeClusterOperation\"\n            ],\n            \"Resource\": [\n                \"arn:aws:kafka:us-east-1:123456789101:cluster\/demo-cluster-1\/85751407-8d09-48b7-af7b-ab25de795414-3\",\n                \"arn:aws:kafka:us-east-1:123456789101:cluster\/demo-cluster-2\/7e173d6b-fc7c-4504-b401-3813ebc6d495-3\"\n            ]\n        }\n    ]\n}'\n<\/pre>\n<p>7. Let&#8217;s create our Lambda function that will trigger the MSK cluster reboot<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\">cat &gt; reboot-msk-broker.py &lt;&lt; EOF\nimport json\nimport boto3\n\ndef lambda_handler(event, context):\n    # Extract the cluster ARN and broker ID from the event\n    response = event\n    client = boto3.client('kafka')\n\n    #convert string to  object\n    response_in_json = json.loads(event)\n\n    #access first_name in dictionary\n    print(response_in_json[\"clusterArn\"])\n    print(response_in_json['brokerId'])\n\n    cluster_arn = response_in_json[\"clusterArn\"]\n    broker_id = response_in_json['brokerId']\n\n    # Reboot the broker\n    response_mskcluster = client.reboot_broker(\n    BrokerIds=[\n        broker_id,\n    ],\n    ClusterArn=cluster_arn\n    )\n\n    cluster_arn = response_mskcluster[\"ClusterArn\"]\n    cluster_operation_arn = response_mskcluster[\"ClusterOperationArn\"]\n   \n    response_operation = client.describe_cluster_operation(ClusterOperationArn=cluster_operation_arn)\n    cluster_operation_state = response_operation[\"ClusterOperationInfo\"]['OperationState']\n    cluster_operation_type = response_operation[\"ClusterOperationInfo\"]['OperationType']\n   \n    return {\n        'brokerId': broker_id,\n        \"cluster_operation_state\": cluster_operation_state,\n        \"cluster_operation_type\": cluster_operation_type,\n        \"cluster_operation_arn\" : cluster_operation_arn\n    }\nEOF<\/pre>\n<p>8. Zip up the reboot-msk-broker.py so we can upload it to lambda in the next step.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\">zip reboot-msk-broker.zip reboot-msk-broker.py\n<\/pre>\n<p>We will now create the lambda function<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\">aws lambda create-function \\\n  --function-name reboot-msk-broker \\\n  --runtime python3.11 \\\n  --role arn:aws:iam::123456789101:role\/LambdaMskRebootBrokerexecutionRole \\\n  --handler reboot-msk-broker.lambda_handler \\\n  --zip-file fileb:\/\/reboot-msk-broker.zip<\/pre>\n<p>9a. Flow 1: Create the SSM Document that we will trigger from FIS. Make sure that you replace the <strong>assumeRole ARN<\/strong> with the ARN for the <strong>SSMDocumentMskRole<\/strong>. Next go to <strong>step 10<\/strong>.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"yaml\">cat &gt; RebootMskBroker.yml &lt;&lt; EOF\ndescription: |\n  ---\n  # Reboot MSK Broker\n  This SSM Document executes the MSK reboot-broker API via Lambda\n  ## Input paramters\n\n  1. The MSK **ClusterARN**\n  2. The **BrokerID** to reboot\n\n  To read more about [msk-reboot-broker](https:\/\/docs.aws.amazon.com\/msk\/1.0\/apireference\/clusters-clusterarn-reboot-broker.html#clusters-clusterarn-reboot-broker-prop-rebootbrokerrequest-brokerids) consult the AWS Documentation. \n\n  ## Output parameters example\n  * \"brokerId\": \"3\"\n  *  \"cluster_operation_state\": \"PENDING\"\n  * \"cluster_operation_type\": \"REBOOT_NODE\"\n  * \"cluster_operation_arn\": \"arn:aws:kafka:us-east-1:123456789012:cluster-operation\/demo-cluster-2\/7e173d6b-fc7c-4504-b401-3813ebc6d495-3\/ee9294d5-bd9a-42c2-a951-f62115092407\"\n\n  You can use the cluster_operation_arn to query the brokers state during its recovery via AWS CLI\n\n  &gt;aws kafka describe-cluster-operation --cluster-operation-arn **arn:aws:kafka:us-east-1:123456789012:cluster-operation\/demo-cluster-1\/85751407-8d09-48b7-af7b-ab25de795414-3\/c21d622d-caf9-4df5-80fe-80f04ddd8816**\nschemaVersion: '0.3'\nassumeRole: 'arn:aws:iam::671061056857:role\/SSMDocumentMSKRole'\nparameters:\n  clusterArn:\n    type: String\n    description: The ARN of the MSK cluster to reboot\n  brokerId:\n    type: String\n    description: The ID of the MSK broker to reboot\nmainSteps:\n  - name: RebootBroker\n    action: 'aws:executeScript'\n    inputs:\n      Runtime: python3.8\n      Handler: reboot_msk_broker\n      Script: |-\n        import json\n        import boto3\n\n        def reboot_msk_broker(event, context):\n            # Extract the cluster ARN and broker ID from the event\n            client = boto3.client('kafka')\n           \n            #convert string to  object\n           \n\n            #access first_name in dictionary\n            print(f\"ClusterARN: event['clusterArn']\")\n            print(f\"BrokerId: event['brokerId']\")\n\n            cluster_arn = event['clusterArn']\n            broker_id = event['brokerId']\n\n            # Reboot the broker\n            response_mskcluster = client.reboot_broker(\n            BrokerIds=[\n                broker_id,\n            ],\n            ClusterArn=cluster_arn\n            )\n\n            cluster_arn = response_mskcluster['ClusterArn']\n            cluster_operation_arn = response_mskcluster['ClusterOperationArn']\n           \n            response_operation = client.describe_cluster_operation(ClusterOperationArn=cluster_operation_arn)\n            cluster_operation_state = response_operation['ClusterOperationInfo']['OperationState']\n            cluster_operation_type = response_operation['ClusterOperationInfo']['OperationType']\n           \n            return {\n                'brokerId': broker_id,\n                \"cluster_operation_state\": cluster_operation_state,\n                \"cluster_operation_type\": cluster_operation_type,\n                \"cluster_operation_arn\" : cluster_operation_arn\n            }\n      InputPayload:\n        clusterArn: '{{ clusterArn }}'\n        brokerId: '{{ brokerId }}'\nEOF<\/pre>\n<p>9b. Flow 2: Create the SSM Document that we will trigger from FIS. Make sure that you replace the <strong>assumeRole ARN<\/strong> with the ARN for the <strong>SSMDocumentLambdaRole<\/strong><\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"yaml\">cat &gt; RebootMskBroker.yml &lt;&lt; EOF\ndescription: |\n  ---\n  # Reboot MSK Broker\n  This SSM Document executes the MSK reboot-broker API via Lambda\n  ## Input paramters\n\n  1. The MSK **ClusterARN**\n  2. The **BrokerID** to reboot\n\n  To read more about [msk-reboot-broker](https:\/\/docs.aws.amazon.com\/msk\/1.0\/apireference\/clusters-clusterarn-reboot-broker.html#clusters-clusterarn-reboot-broker-prop-rebootbrokerrequest-brokerids) consult the AWS Documentation. \n\n  ## Output parameters example\n  * \"brokerId\": \"3\"\n  *  \"cluster_operation_state\": \"PENDING\"\n  * \"cluster_operation_type\": \"REBOOT_NODE\"\n  * \"cluster_operation_arn\": \"arn:aws:kafka:us-east-1:123456789012:cluster-operation\/demo-cluster-2\/7e173d6b-fc7c-4504-b401-3813ebc6d495-3\/ee9294d5-bd9a-42c2-a951-f62115092407\"\n\n  You can use the cluster_operation_arn to query the brokers state during its recovery via AWS CLI\n\n  &gt;aws kafka describe-cluster-operation --cluster-operation-arn **arn:aws:kafka:us-east-1:123456789012:cluster-operation\/demo-cluster-1\/85751407-8d09-48b7-af7b-ab25de795414-3\/c21d622d-caf9-4df5-80fe-80f04ddd8816**\nschemaVersion: '0.3'\nassumeRole: 'arn:aws:iam::671061056857:role\/SSMDocumentLambdaRole'\nparameters:\n  clusterArn:\n    type: String\n    description: The ARN of the MSK cluster to reboot\n  brokerId:\n    type: String\n    description: The ID of the MSK broker to reboot\nmainSteps:\n  - name: RebootBroker\n    action: 'aws:invokeLambdaFunction'\n    inputs:\n      InvocationType: RequestResponse\n      Payload: '\"{\\\"clusterArn\\\": \\\"{{clusterArn}}\\\", \\\"brokerId\\\":\\\"{{brokerId}}\\\"}\"'\n      FunctionName: 'arn:aws:lambda:us-east-1:671061056857:function:rebootMSK'\n    outputs:\n      - Name: brokerId\n        Selector: $.Payload.brokerid\n        Type: String\n      - Name: cluster_operation_state\n        Selector: $.Payload.cluster_operation_state\n        Type: String\n      - Name: cluster_operation_type\n        Selector: $.Payload.cluster_operation_type\n        Type: String\n      - Name: cluster_operation_arn\n        Selector: $.Payload.cluster_operation_arn\n        Type: String\nEOF<\/pre>\n<p>10. Create the SSM document that takes the input for clusterArn and brokerId. This document will be called by the Fault Injection Service experiment.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\">aws ssm create-document \\\n    --content file:\/\/RebootMskBroker.yml \\\n    --name \"RebootMskBrokerNow\" \\\n    --document-type \"Automation\" \\\n    --document-format YAML<\/pre>\n<p>11. Create the FIS template. Note the Template ID that you are getting back as we will need it in the next step.<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"json\">aws fis create-experiment-template \\\n    --cli-input-json '{\n    \"description\": \"RebootMSKBroker\",\n        \"targets\": {},\n        \"actions\": {\n            \"RebootMSK\": {\n                \"actionId\": \"aws:ssm:start-automation-execution\",\n                \"parameters\": {\n                    \"documentArn\": \"arn:aws:ssm:us-east-1:123456789101:document\/RebootMskBrokerNow\",\n                    \"documentParameters\": \"{\"clusterArn\": \"arn:aws:kafka:us-east-1:671061056857:cluster\/demo-cluster-1\/85751407-8d09-48b7-af7b-ab25de795414-3\",\"brokerId\": \"3\" }\",\n                    \"maxDuration\": \"PT1M\"\n                }\n            }\n        },\n        \"stopConditions\": [\n            {\n                \"source\": \"none\"\n            }\n        ],\n        \"roleArn\": \"arn:aws:iam::671061056857:role\/ssm-fis-role\",\n        \"tags\": {\n            \"Name\": \"RebootMskBroker1\"\n        },\n        \"logConfiguration\": {\n            \"cloudWatchLogsConfiguration\": {\n                \"logGroupArn\": \"arn:aws:logs:us-east-1:123456789101:log-group:\/aws\/fis\/experiments:*\"\n            },\n            \"logSchemaVersion\": 2\n        }\n}'<\/pre>\n<p>12. Start the experiment from the command line<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"bash\">aws fis start-experiment \\\n    --experiment-template-id EXTWTgEXTuhavDn<\/pre>\n<p>You can now go to the MSK console and will see your broker rebooting<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2841\" src=\"http:\/\/blog.domb.net\/wp-content\/uploads\/RebootBroker.png\" alt=\"\" width=\"2115\" height=\"446\" srcset=\"https:\/\/blog.domb.net\/wp-content\/uploads\/RebootBroker.png 2115w, https:\/\/blog.domb.net\/wp-content\/uploads\/RebootBroker-300x63.png 300w, https:\/\/blog.domb.net\/wp-content\/uploads\/RebootBroker-1024x216.png 1024w, https:\/\/blog.domb.net\/wp-content\/uploads\/RebootBroker-768x162.png 768w, https:\/\/blog.domb.net\/wp-content\/uploads\/RebootBroker-1536x324.png 1536w, https:\/\/blog.domb.net\/wp-content\/uploads\/RebootBroker-2048x432.png 2048w\" sizes=\"auto, (max-width: 2115px) 100vw, 2115px\" \/><\/p>\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"json\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"true\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"trust policy\" data-enlighter-group=\"\"><\/pre>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"json\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\"><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Customers have asked me how they can use Fault Injection Service to impact Amazon Managed Streaming for apache Kafka by rebooting brokers. In this blog post, I&#8217;ll demonstrate how to do that. I will give you two flows on how you can reboot the MSK broker via FIS. Flow 1: FIS =&gt; SSM =&gt; MSK&#8230;<\/p>\n","protected":false},"author":2,"featured_media":2847,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_themeisle_gutenberg_block_has_review":false,"footnotes":""},"categories":[173],"tags":[194,176,193,195],"class_list":["post-2818","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-chaos-engineering","tag-fault-injection-service","tag-fis","tag-msk","tag-reboot-broker"],"_links":{"self":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts\/2818","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2818"}],"version-history":[{"count":42,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts\/2818\/revisions"}],"predecessor-version":[{"id":2879,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/posts\/2818\/revisions\/2879"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=\/wp\/v2\/media\/2847"}],"wp:attachment":[{"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2818"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2818"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.domb.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2818"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}