Attention
For production workloads, we recommend sing Lookup you can use existing OpenSearch clusters with your new services. This will avoid accidental deletions or rollback situations where both your DB and services have to rollback.
x-opensearch ¶
x-opensearch:
opensearch-01:
Properties: {}
Settings: {}
Services: {}
Lookup: {}
MacroParameters: {}
Allows you to create / lookup OpenSearch Domains to use with your ECS Services.
Services ¶
The syntax for listing the services remains the same as the other x- resources.
Services:
<service/family name>
Access:
Http: [RO|RW]
DBCluster: [RO|RW]
ReturnValues: {}
Access types ¶
Http ¶
This defines the Http permissions that you wish to grant your service to perform queries to the OpenSearch domain
DBCluster ¶
This defines the IAM permissions you wish to grant to the service that will allow to describe and discover the OpenSearch domain
{
"Http": {
"RW": {
"Effect": "Allow",
"Action": [
"es:ESHttpDelete",
"es:ESHttpGet",
"es:ESHttpHead",
"es:ESHttpPost",
"es:ESHttpPut",
"es:ESHttpPatch"
],
"Resource": [
"${ARN}/*"
]
},
"RO": {
"Effect": "Allow",
"Action": [
"es:ESHttpGet",
"es:ESHttpHead",
"es:ESHttpPost",
"es:ESHttpPut",
"es:ESHttpPatch"
],
"Resource": [
"${ARN}/*"
]
}
},
"DBCluster": {
"RO": {
"Effect": "Allow",
"Action": [
"es:DeleteDomain",
"es:DescribeDomain",
"es:DescribeDomainAutoTunes",
"es:DescribeDomainConfig",
"es:ESCrossClusterGet",
"es:GetCompatibleVersions",
"es:GetUpgradeHistory",
"es:GetUpgradeStatus",
"es:ListPackagesForDomain",
"es:ListTags"
],
"Resource": [
"${ARN}"
]
}
}
}
ReturnValues ¶
See AWS OpenSearch Domain return values from the AWS Documentation pages.
Properties ¶
Refer to the AWS OpenSearch Domain CFN Properties .
Attention
The OpenSearch properties are very tedious, not all instance types support the same settings (EBSOptions etc.). Therefore, ECS Compose-X will try to autocorrect the settings based on the instance types set . However, if it cannot solve the issue due to too incompatible settings contradicting each other, it will fail.
Refer to the Instance Types supported by OpenSearch to check out these settings.
Hint
Given the combinations of settings that only work together, we have implemented validations to verify those early. If we missed something, please report an issue in GitHub
Hint
ECS Compose-X does an instance-types verification to ensure you are not mixing normal and Graviton instance types.
Hint
ECS Compose-X will try to autocorrect the following properties based on the Instance Type
-
OpenSearch domain version
-
EBSOptions
-
EncryptionAtRestOptions
-
AdvancedSecurityOptions
-
ClusterConfig.WarmEnabled
When possible, it will just void / set to False the offensive setting to the instance type.
VPCOptions ¶
Whenever you set the VPCOptions, Compose-X will create a new AWS EC2 Security Group in order to manage the Ingress defined from the other services. When these are set, ECS Compose-X will deal with setting Security Group ingress.
Attention
When setting the VPCOptions, you cannot set more Subnets than you set the total number of instances for the domain. For example, if you set only 1 Instance (i.e. for the master) then you can only set 1 Subnet in the SubnetIds.
MacroParameters ¶
These parameters will allow you to define extra parameters to define your cluster successfully.
Instances: []
DBClusterParameterGroup: {} # AWS DocDB::DBClusterParameterGroup properties
CreateLogGroupsResourcePolicy ¶
When set to True, it will create a new AWS::Logs::ResourcePolicy (10 per accounts limit) that will grant the AWS OpenSearch Service to write to AWS CloudWatch Logs.
If you never had set this up, turn set it to True. However, if you already have your own policy in place, you can use that, simply ensure that the permissions will allow your domain to write to these.
Tip
If no resource policy is set to allow the domain to write to the logs, or is disabled, creation of the domain will fail.
CreateLogGroups ¶
When set to True, it will automatically create all log groups for the given domain to write to. When set as a list, each value it must be one of
-
“SEARCH_SLOW_LOGS”
-
“ES_APPLICATION_LOGS”
-
“INDEX_SLOW_LOGS”
-
“AUDIT_LOGS”
For each of the value enabled above, it will create a new Log Group.
Attention
When settings AUDIT_LOGS make sure to enable the appropriate security options.
RetentionInDays ¶
Allow to define the retention in days value for the new log groups getting created.
GenerateMasterUserSecret ¶
This allows to automatically create a new AWS SecretsManager secret, with a username and password, that you can use to then authenticate and administer the domain with.
CreateMasterUserRole ¶
When set to True, allow to create an IAM role (instead of secret) that your account can assume in order then to administer the domain
Hint
You can set GenerateMasterUserSecret or CreateMasterUserRole, but not both, to True
MasterUserRolePermissionsBoundary ¶
Allows you to define IAM Permissions Boundary when creating the new AWS IAM MasterRole
Settings ¶
Subnets ¶
This parameter allows you to define which subnets group you wish to deploy OpensSearch to. If the VpcOptions were not set in the Properties then they automatically get added.
Lookup ¶
Lookup for OpenSearch domains is available. However not used / tested with legacy ElasticSearch domains.
Examples ¶
version: '3.8'
x-opensearch:
domain-01:
Properties:
AccessPolicies:
Statement:
- Action: es:*
Effect: Allow
Principal:
AWS: arn:aws:iam::123456789012:user/opensearch-user
Resource: arn:aws:es:us-east-1:846973539254:domain/test/*
Version: '2012-10-17'
AdvancedOptions:
override_main_response_version: true
rest.action.multi.allow_explicit_index: true
ClusterConfig:
DedicatedMasterCount: '3'
DedicatedMasterEnabled: true
DedicatedMasterType: m3.medium.search
InstanceCount: '2'
InstanceType: m3.medium.search
ZoneAwarenessEnabled: true
DomainName: test
EBSOptions:
EBSEnabled: true
Iops: '0'
VolumeSize: '20'
VolumeType: gp2
EngineVersion: OpenSearch_1.0
LogPublishingOptions:
ES_APPLICATION_LOGS:
CloudWatchLogsLogGroupArn: arn:aws:logs:us-east-1:123456789012:log-group:/aws/opensearch/domains/opensearch-application-logs
Enabled: true
INDEX_SLOW_LOGS:
CloudWatchLogsLogGroupArn: arn:aws:logs:us-east-1:123456789012:log-group:/aws/opensearch/domains/opensearch-index-slow-logs
Enabled: true
SEARCH_SLOW_LOGS:
CloudWatchLogsLogGroupArn: arn:aws:logs:us-east-1:123456789012:log-group:/aws/opensearch/domains/opensearch-slow-logs
Enabled: true
domain-02:
Properties:
AccessPolicies:
Statement:
- Action: es:*
Effect: Deny
Principal:
AWS: '*'
Resource: '*'
Version: '2012-10-17'
AdvancedOptions:
override_main_response_version: true
rest.action.multi.allow_explicit_index: true
ClusterConfig:
InstanceCount: '1'
InstanceType: m3.medium.search
DomainName: test2
EBSOptions:
EBSEnabled: true
Iops: '0'
VolumeSize: '10'
VolumeType: standard
EngineVersion: OpenSearch_1.0
Tags:
- Key: foo
Value: bar
VPCOptions:
SecurityGroupIds:
- sg-012344rsto
SubnetIds:
- subnet-abdc818l
domain-03:
Properties:
AccessPolicies:
Statement:
- Action: es:*
Effect: Deny
Principal:
AWS: '*'
Resource: '*'
Version: '2012-10-17'
AdvancedOptions:
override_main_response_version: true
rest.action.multi.allow_explicit_index: true
ClusterConfig:
InstanceCount: '1'
InstanceType: m3.medium.search
DomainName: test2
EBSOptions:
EBSEnabled: true
Iops: '0'
VolumeSize: '10'
VolumeType: standard
EngineVersion: OpenSearch_1.0
Tags:
- Key: foo
Value: bar
Services:
app03:
Access:
DBCluster: RO
Http: RW
Settings:
Subnets: StorageSubnets
domain-04:
MacroParameters:
CreateLogGroups:
- SEARCH_SLOW_LOGS
- INDEX_SLOW_LOGS
CreateMasterUserRole: true
RetentionInDays: 120
Properties:
AdvancedOptions:
override_main_response_version: true
rest.action.multi.allow_explicit_index: true
ClusterConfig:
InstanceCount: '1'
InstanceType: r5.medium.search
EBSOptions:
EBSEnabled: true
Iops: '0'
VolumeSize: '10'
VolumeType: gp2
EngineVersion: OpenSearch_1.0
Tags:
- Key: foo
Value: bar
Settings:
Subnets: AppSubnets
domain-05:
MacroParameters:
CreateLogGroups: true
GenerateMasterUserSecret: true
Properties:
AdvancedOptions:
override_main_response_version: true
rest.action.multi.allow_explicit_index: true
AdvancedSecurityOptions:
InternalUserDatabaseEnabled: true
ClusterConfig:
DedicatedMasterType: m6g.large.search
InstanceCount: '1'
InstanceType: r6gd.large.search
EngineVersion: OpenSearch_1.0
Tags:
- Key: foo
Value: bar
Services:
app03:
Access:
DBCluster: RO
Http: RW
version: '3.8'
x-opensearch:
domain-01:
Lookup:
RoleArn: ${RO_ROLE_ARN}
Tags:
- CreatedByComposeX: 'true'
- ComposeXName: domain-01
Services:
app03:
Access:
Http: RO
IAM: RO
domain-02:
Lookup:
RoleArn: ${RO_ROLE_ARN}
Tags:
- CreatedByComposeX: 'true'
- ComposeXName: domain-02
Services:
app03:
Access:
Http: RO
IAM: RO
JSON Schema ¶
Model ¶
x-opensearch ¶
|
x-opensearch.spec.json |
|||||
|
x-opensearch specification |
|||||
|
type |
object |
||||
|
properties |
|||||
|
x-resources.common.spec.json#/definitions/Lookup |
||||
|
type |
string |
|||
|
|||||
|
type |
object |
||||
|
x-resources.common.spec.json#/definitions/Settings |
||||
|
x-resources.common.spec.json#/definitions/Services |
||||
|
#/definitions/MacroParameters |
||||
|
additionalProperties |
False |
||||
|
definitions |
|||||
|
type |
object |
|||
|
properties |
|||||
|
Whether you want ComposeX to created a Log Group Resource Policy. Only 10 per AWS Account |
||||
|
type |
boolean |
||||
|
default |
False |
||||
|
oneOf |
type |
array |
||
|
items |
Creates the CW Log Groups for AWS::OpenSearchService::Domain LogPublishingOption. |
||||
|
type |
string |
||||
|
enum |
SEARCH_SLOW_LOGS, ES_APPLICATION_LOGS, INDEX_SLOW_LOGS, AUDIT_LOGS |
||||
|
Automatically creates a new CW Log group for each option |
|||||
|
type |
boolean |
||||
|
RetentionInDays for the new CW Log Groups |
||||
|
type |
number |
||||
|
enum |
1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653 |
||||
|
Automatically create a AWS Secret to use for the MasterUser AWS::OpenSearchService::Domain MasterUserOptions |
||||
|
type |
boolean |
||||
|
default |
False |
||||
|
Automatically creates a new IAM Role that you can later assume |
||||
|
type |
boolean |
||||
|
default |
False |
||||
|
IAM Policy to use as the Permissions Boundary for the MasterUser Role |
||||
|
type |
string |
||||
|
dependencies |
|||||
|
CreateMasterUserRole |
||||
|
CreateLogGroups |
||||
|
CreateLogGroups |
||||
Definition ¶
{
"$schema": "http://json-schema.org/draft-07/schema#",
"id": "x-opensearch.spec.json",
"$id": "x-opensearch.spec.json",
"title": "x-opensearch",
"description": "x-opensearch specification",
"type": "object",
"additionalProperties": false,
"properties": {
"Lookup": {
"$ref": "x-resources.common.spec.json#/definitions/Lookup"
},
"Use": {
"type": "string"
},
"Properties": {
"type": "object",
"description": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-opensearch-dbcluster.html"
},
"Settings": {
"$ref": "x-resources.common.spec.json#/definitions/Settings"
},
"Services": {
"$ref": "x-resources.common.spec.json#/definitions/Services"
},
"MacroParameters": {
"$ref": "#/definitions/MacroParameters"
}
},
"definitions": {
"MacroParameters": {
"type": "object",
"additionalItems": false,
"properties": {
"CreateLogGroupsResourcePolicy": {
"type": "boolean",
"description": "Whether you want ComposeX to created a Log Group Resource Policy. Only 10 per AWS Account",
"default": false
},
"CreateLogGroups": {
"oneOf": [
{
"type": "array",
"items": {
"type": "string",
"description": "Creates the CW Log Groups for AWS::OpenSearchService::Domain LogPublishingOption.",
"enum": [
"SEARCH_SLOW_LOGS",
"ES_APPLICATION_LOGS",
"INDEX_SLOW_LOGS",
"AUDIT_LOGS"
]
}
},
{
"type": "boolean",
"description": "Automatically creates a new CW Log group for each option"
}
]
},
"RetentionInDays": {
"type": "number",
"enum": [
1,
3,
5,
7,
14,
30,
60,
90,
120,
150,
180,
365,
400,
545,
731,
1827,
3653
],
"description": "RetentionInDays for the new CW Log Groups"
},
"GenerateMasterUserSecret": {
"type": "boolean",
"description": "Automatically create a AWS Secret to use for the MasterUser AWS::OpenSearchService::Domain MasterUserOptions",
"default": false
},
"CreateMasterUserRole": {
"type": "boolean",
"description": "Automatically creates a new IAM Role that you can later assume",
"default": false
},
"MasterUserRolePermissionsBoundary": {
"type": "string",
"description": "IAM Policy to use as the Permissions Boundary for the MasterUser Role"
}
},
"dependencies": {
"MasterUserRolePermissionsBoundary": [
"CreateMasterUserRole"
],
"RetentionInDays": [
"CreateLogGroups"
],
"CreateLogGroupsResourcePolicy": [
"CreateLogGroups"
]
}
}
}
}
Test files ¶
You can find the test files here to use as reference for your use-case.