x-kinesis_firehose ¶

          x-kinesis_firehose:
  stream-logical-name:
    Properties: {}
    MacroParameters: {}
    Lookup: {}
    Services: {}

         

Services ¶

           x-kinesis_firehose:
  stream-logical-name:
    Services:
      my-app:
        Access: Producer
        ReturnValues: {}

services:
  my-app: {}

          

ReturnValues ¶

The available return values are as defined in AWS Firehose CloudFormation ReturnValues

IAM Permissions ¶

The only value for Access is Producer which allows the clients to publish records to the Delivery Stream.

            {
  "Producer": {
    "Effect": "Allow",
    "Action": [
      "firehose:PutRecord",
      "firehose:PutRecordBatch",
      "firehose:UpdateDestination"
    ],
    "Resource": [
      "${ARN}"
    ]
  }
}

           

Properties ¶

Refer to AWS::KinesisFirehose::DeliveryStream documentation page for all the properties you can set. They are all supported, and been tested with slightly modified versions of the examples.

When creating a new Firehose DeliveryStream, ECS Compose-X wil also automatically create a new IAM role that will be used to grant the service role access to your other AWS Resources, such as S3/Kinesis and others.

See below for information on how to change the behaviour.

Modified properties ¶

The following properties are updated automatically. See MacroParameters to disable the automatic change of these properties.

KinesisStreamSourceConfiguration.RoleARN
AmazonopensearchserviceDestinationConfiguration.RoleARN
S3DestinationConfiguration.RoleARN
ElasticsearchDestinationConfiguration.RoleARN
ExtendedS3DestinationConfiguration.RoleARN
RedshiftDestinationConfiguration.RoleARN

MacroParameters ¶

DoNotOverrideIamRole ¶

This can be either set as a boolean (True|False) or as a list of string, representing the Destination/Source for which you do not want ECS Compose-X to replace with a new IAM Role .

So in the Properties, if you defined RoleARN for one of these properties, it will be automatically updated and replaced with a new IAM Role that is managed by ECS Compose-X.

Setting to true or as a list, will disable the replacement for all/for the properties listed.

Warning

This means that IAM polices will not be created to allow the appropriate API calls to work with the other x-<resource> defined in your file. Use at your own risks, if you know what you are doing.

            x-kinesis_firehose:
  stream-logical-name:
    MacroParameters:
      DoNotOverrideIamRole: true

  another-stream-logical-name:
    MacroParameters:
      DoNotOverrideIamRole:
        - ElasticsearchDestinationConfiguration
        - AmazonopensearchserviceDestinationConfiguration

           

x-iam ¶

This allows to define additional IAM properties manually.

PermissionsBoundary ¶

Allows to define a PermissionsBoundary to link to the IAM Role. By default, there is none.

Link to other AWS Resources ¶

Hint

ECS Compose-X will automatically update the IAM permissions of the new IAM Role associated with the DeliveryStream

Note

ECS Compose-X will NOT update the IAM Role permissions if you disabled it for the source/destination of the delivery stream.

The following resources are supported to be defined in the compose file, and be interpolated with the resource properties

S3 Buckets with x-s3
Kinesis Streams with x-kinesis
KMS keys with x-kms

Tip

Adding OpenSearch is in the to-do list. Feel free to open a Feature Request to see it added with priority.

S3 Buckets ¶

You can use x-s3::<bucket-name> for the following properties

S3DestinationConfiguration:: BucketARN
ExtendedS3DestinationConfiguration:: BucketARN
ExtendedS3DestinationConfiguration::S3BackupConfiguration:: BucketARN
RedshiftDestinationConfiguration::S3BackupConfiguration:: BucketARN
ElasticsearchDestinationConfiguration::S3BackupConfiguration:: BucketARN
AmazonopensearchserviceDestinationConfiguration::S3BackupConfiguration:: BucketARN
SplunkDestinationConfiguration::S3BackupConfiguration:: BucketARN
HttpEndpointDestinationConfiguration::S3BackupConfiguration:: BucketARN

Example for S3 Extended configuration ¶

             x-s3:
  delivery-stream-output-bucket:
    Properties: {}

x-kinesis_firehose:
  stream-to-s3-direct-put:
    Properties:
      DeliveryStreamName: tester-partitioning-delimiter
      DeliveryStreamType: DirectPut
      ExtendedS3DestinationConfiguration:
        BucketARN: x-s3::delivery-stream-output-bucket

            

It will grant the corresponding IAM permissions to the IAM Role linked to the Firehose DeliveryStream

              "kinesis_firehose": {
    "s3destination": {
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": [
        "${ARN}",
        "${ARN}/*"
      ]
    },
    "s3keyaccess": {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
      ],
      "Resource": [
        "${ARN}"
      ]
    }
  }
}

           

Kinesis Streams ¶

You can use x-kinesis::<stream-logical-name> to update the value for `KinesisStreamSourceConfiguration.KinesisStreamARN`_ This will automatically set the right value for it and DeliveryStreamType .

It will grant the corresponding IAM permissions to the IAM Role linked to the Firehose DeliveryStream

                "NotAction": [
      "kinesis:CreateStream",
      "kinesis:DeleteStream"
    ],
    "Resource": [
      "${ARN}"
    ]
  },
  "kinesis_firehose": {
    "kinesisSource": {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:GetShardIterator",

           

KMS Keys ¶

This will allow to update the value for DeliveryStreamEncryptionConfigurationInput.KeyARN and where applicable, EncryptionConfiguration.KMSEncryptionConfig.AWSKMSKeyARN

Note that if the key is imported via Lookup, it must be a Customer CMK.

It will grant the corresponding IAM permissions to the IAM Role linked to the Firehose DeliveryStream

              "kinesis_firehose": {
    "Direct": {
      "Effect": "Allow",
      "Resource": [
        "${ARN}"
      ],
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ]
    },
    "s3": {
      "Effect": "Allow",
      "Resource": [
        "${ARN}"
      ],
      "Action": [
        "kms:Decrypt",
        "kms:GenerateDataKey"
      ]
    }

           

JSON Schema ¶

Model ¶

x-kinesis_firehose ¶

x-kinesis_firehose.spec.json
x-kinesis_firehose
type	object
properties
Lookup	x-resources.common.spec.json#/definitions/Lookup
Properties	https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kinesis-stream.html
	type	object
Settings	x-resources.common.spec.json#/definitions/Settings
Services	x-resources.common.spec.json#/definitions/Services
MacroParameters	type	object
	properties
	x-iam	type	object
		properties
		PermissionsBoundary	type	string
		additionalProperties	False
	DoNotOverrideIamRole	oneOf	type	boolean
			default	False
			type	array
			items	type	string
				enum	S3DestinationConfiguration, RedshiftDestinationConfiguration, KinesisStreamSourceConfiguration, ExtendedS3DestinationConfiguration, ElasticsearchDestinationConfiguration, AmazonopensearchserviceDestinationConfiguration
additionalProperties	False

Definition ¶

            {
  "$schema": "http://json-schema.org/draft-07/schema#",
  "id": "x-kinesis_firehose.spec.json",
  "$id": "x-kinesis_firehose.spec.json",
  "title": "x-kinesis_firehose",
  "description": "x-kinesis_firehose",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "Lookup": {
      "$ref": "x-resources.common.spec.json#/definitions/Lookup"
    },
    "Properties": {
      "type": "object",
      "description": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kinesis-stream.html"
    },
    "Settings": {
      "$ref": "x-resources.common.spec.json#/definitions/Settings"
    },
    "Services": {
      "$ref": "x-resources.common.spec.json#/definitions/Services"
    },
    "MacroParameters": {
      "type": "object",
      "properties": {
        "x-iam": {
          "type": "object",
          "additionalProperties": false,
          "properties": {
            "PermissionsBoundary": {
              "type": "string"
            }
          }
        },
        "DoNotOverrideIamRole": {
          "oneOf": [
            {
              "type": "boolean",
              "default": false
            },
            {
              "type": "array",
              "items": {
                "type": "string",
                "enum": [
                  "S3DestinationConfiguration",
                  "RedshiftDestinationConfiguration",
                  "KinesisStreamSourceConfiguration",
                  "ExtendedS3DestinationConfiguration",
                  "ElasticsearchDestinationConfiguration",
                  "AmazonopensearchserviceDestinationConfiguration"
                ]
              }
            }
          ]
        }
      }
    }
  },
  "oneOf": [
    {
      "required": [
        "Lookup"
      ]
    },
    {
      "required": [
        "Properties"
      ]
    }
  ]
}

           

Test files ¶

You can find the test files here to use as reference for your use-case.