ecs_composex package

Subpackages

Submodules

ecs_composex.cli module

Console script for ecs_composex.

class ecs_composex.cli. ArgparseHelper ( option_strings , dest = '==SUPPRESS==' , default = '==SUPPRESS==' , help = None ) [source]

Bases: _HelpAction

Used to help print top level ‘–help’ arguments from argparse when used with subparsers

ecs_composex.cli. main ( ) [source]

Main entry point for CLI :return: status code

ecs_composex.cli. main_parser ( ) [source]

Console script for ecs_composex.

ecs_composex.ecs_composex module

Main module to generate a full stack with VPC, Cluster, Compute, Services and all X- AWS resources.

ecs_composex.ecs_composex. add_x_resources ( settings ) [source]

Processes the modules / resources that are defining the environment settings

Return type :

None

ecs_composex.ecs_composex. apply_x_configs_to_ecs ( settings , root_stack , modules ) [source]

Function that evaluates only the x- resources of the root template and iterates over the resources. If there is an implemented module in ECS ComposeX for that resource_stack to map to the ECS Services, it will execute the function available in the module to apply defined settings to the services stack.

The root_stack is used as the parent stack to the services.

Parameters :
Return type :

None

ecs_composex.ecs_composex. apply_x_resource_to_x ( settings , root_stack , vpc_stack , env_resources_only = False ) [source]

Goes over each x resource in the execution and execute logical association between the resources. If env_resources_only is true, only invokes handle_x_dependencies only for the AwsEnvironmentResource resources defined.

Parameters :
Return type :

None

ecs_composex.ecs_composex. create_root_stack ( settings ) [source]

Initializes the root stack template and ComposeXStack

Parameters :

settings ( ecs_composex.common.settings.ComposeXSettings ) – The settings for the execution

Return type :

ecs_composex.common.stacks.ComposeXStack

ecs_composex.ecs_composex. generate_full_template ( settings ) [source]

Function to generate the root template and associate services, x-resources to each other.

  • Checks that the docker images and settings are correct before proceeding further

  • Create the root template / stack

  • Create/Find ECS Cluster

  • Create IAM Stack (services Roles and some policies)

  • Create/Find x-resources

  • Link services and x-resources

  • Associates services/family to root stack

Parameters :

settings ( ecs_composex.common.settings.ComposeXSettings ) – The settings for the execution

Return root_template :

Template, params

Return type :

root_template, list

ecs_composex.ecs_composex. get_mod_function ( module_name , function_name ) [source]

Function to get function in a given module name from function_name

Parameters :
  • module_name ( str ) – the name of the module in ecs_composex to find and try to import

  • function_name ( str ) – name of the function to try to get

Returns :

function, if found, from the module

Return type :

function

ecs_composex.ecs_composex. invoke_x_to_ecs ( module_name , services_stack , resource , settings ) [source]

Function to associate X resources to Services

Parameters :
Return type :

None

Returns :

ecs_composex.ecs_composex. set_all_mappings_to_root_stack ( settings ) [source]

Adds all the mappings to the root stack1

ecs_composex.exceptions module

Custom exceptions for compose-x

exception ecs_composex.exceptions. ComposeBaseException ( msg , * args ) [source]

Bases: Exception

Top class for Compose-X Exceptions

exception ecs_composex.exceptions. IncompatibleOptions ( msg , * args ) [source]

Bases: ComposeBaseException

Exception when two x-resources conflict, i.e. when you try to use Lookup on x-cloudmap and create a new VPC

ecs_composex.ingress_settings module

Module to help with defining the network settings for the ECS Service based on the family services definitions.

class ecs_composex.ingress_settings. Ingress ( definition , ports ) [source]

Bases: object

Class to group the configuration for Service network settings

associate_aws_ingress_rules ( template ) [source]

Method to associate AWS ingress rules to a specific template

Parameters :

template ( troposphere.Template ) –

Returns :

associate_ext_ingress_rules ( template ) [source]

Method to associate External ingress rules to a specific template

Parameters :

template ( troposphere.Template ) –

Returns :

aws_sources_key = 'AwsSources'
create_ext_sources_ingress_rule ( destination_title , allowed_source , security_group , ** props ) [source]

Creates the Security Ingress rule for a CIDR based rule

Parameters :
  • destination_title ( str ) –

  • allowed_source ( dict ) –

  • security_group

  • props ( dict ) –

Return type :

None

defined = True
ext_sources_key = 'ExtSources'
ipv4_key = 'IPv4'
ipv6_key = 'IPv6'
master_key = 'Ingress'
network_settings = ['Ingress', 'UseCloudmap', 'IsPublic']
services_key = 'Services'
set_aws_sources_ingress ( settings , destination_title , sg_ref ) [source]

Method to define AWS Sources ingresses

Parameters :
  • settings

  • destination_title

  • sg_ref

Return type :

None

set_ext_sources_ingress ( destination_tile , security_group ) [source]

Method to add ingress rules from external sources to a given Security Group (ie. ALB Security Group). If a list of IPs is found in the config[‘ext_sources’] part of the network section of configs for the service, then it will use that. If no IPv4 source is indicated, it will by default allow traffic from 0.0.0.0/0

Parameters :
  • destination_tile ( str ) – The name of the destination for description

  • security_group ( str or troposphere.ec2.SecurityGroup or troposphere.Ref or Troposphere.GetAtt ) – security group (object or title string) to add the rules to

ecs_composex.ingress_settings. define_protocol ( port_string ) [source]

Function to define the port protocol. Defaults to TCP if not specified otherwise

Parameters :

port_string ( str ) – the port string to parse from the ports list in the compose file

Returns :

protocol, ie. udp or tcp

Return type :

str

ecs_composex.ingress_settings. flatten_ip ( ip_str ) [source]

Function to remove all non alphanum characters from IP CIDR notation

Parameters :

ip_str

Return type :

str

ecs_composex.ingress_settings. generate_security_group_props ( allowed_source ) [source]

Function to parse the allowed source and create the SG Opening options accordingly.

Parameters :

allowed_source ( dict ) – The allowed source defined in configs

Returns :

security group ingress properties

Return type :

dict

ecs_composex.ingress_settings. lookup_security_group ( settings , lookup ) [source]

Function to fetch the security group ID based on lookup details

Parameters :
Returns :

ecs_composex.ingress_settings. set_port_from_str ( port ) [source]

Function to filter out port string and define published port, target port and protocol

Parameters :

port ( str ) –

Returns :

the ports parameters

Return type :

tuple

ecs_composex.ingress_settings. set_service_ports ( ports ) [source]

Function to define common structure to ports

Returns :

list of ports the ecs_service uses formatted according to dict

Return type :

list

ecs_composex.mods_manager module

This module allows managing import of x-resources modules into ecs-composex dynamically and order the resources processing based on the type of resource this is.

Priority order goes

  • AWS Environment resources

  • AWS API based resources (purely serverless resources)

  • AWS Networking based resources (resources that require VPC)

class ecs_composex.mods_manager. ModManager ( settings ) [source]

Bases: object

Class to manage the modules

add_module_from_module_def ( res_key , mod_key , module_name ) [source]
import_resource_modules ( res_key , module_path ) [source]
init_mods_resources ( settings ) [source]
load_module ( res_key , res_def ) [source]
Return type :

XResourceModule | None

modules_repr ( ) [source]
class ecs_composex.mods_manager. XResourceModule ( res_key , x_class , posix_path , resource_class = None , definition = None ) [source]

Bases: object

property definition : dict
property iam_policies : dict
import_json_schema ( ) [source]
import_perms_definition ( ) [source]
property json_schema
property lookup_resources : list
Returns :

list of resources to import from Lookup

Return type :

list[ XResource ] x_resources:

property mapping_key
property mappings : dict
property mod_key
property new_resources : list

Function to create a list of new resources. Check if empty resource is supported

Returns :

list of resources to create

Return type :

list[ XResource ] x_resources:

property path
property res_key
property resource_class : XResource | ServicesXResource | ApiXResource | AwsEnvironmentResource | NetworkXResource | DatabaseXResource
property resources : dict [ str , XResource | ServicesXResource | ApiXResource | AwsEnvironmentResource | NetworkXResource | DatabaseXResource ]
property resources_list : list [ XResource | ServicesXResource | ApiXResource | AwsEnvironmentResource | NetworkXResource | DatabaseXResource ]
set_resources ( settings ) [source]

Method to define the ComposeXResource for each service. First updates the resources dict

Parameters :

settings ( ecs_composex.common.settings.ComposeXSettings ) –

property stack_class
ecs_composex.mods_manager. get_module ( module_name ) [source]

Function to get the XResourceModule if it has been defined.

Return type :

tuple

Returns :

the_class, maps to the main class for the given x-module

ecs_composex.rds_resources_settings module

Module of functions factorizing common patterns for TCP based access such as RDS, DocumentDB

ecs_composex.rds_resources_settings. add_secret_arn_env_var ( family , secret_env_key , secret_definition ) [source]

Adds environment variable to service, using the Name/ARN of the service as value

Parameters :
Returns :

ecs_composex.rds_resources_settings. add_secret_to_container ( db , secret_import , service , target ) [source]

Function to add DB secret to container

Parameters :
  • db ( ecs_composex.common.compose_resources.Rds ) – the RDS DB object

  • service – The target service definition

  • secret_import ( str , AWSHelper ) – secret arn

  • target ( tuple ) –

ecs_composex.rds_resources_settings. add_secrets_access_policy ( service_family , secret_import , db , use_task_role = False ) [source]

Function to add or append policy to access DB Secret for the Execution Role If the use_task_role true, also allows the task role access to the secret.

Return type :

None

ecs_composex.rds_resources_settings. add_security_group_ingress ( service_stack , db_name , sg_id , port ) [source]

Function to add a SecurityGroupIngress rule into the ECS Service template

Parameters :
  • service_stack ( ecs_composex.ecs.ServicesStack ) – The root stack for the services

  • db_name ( str ) – the name of the database to use for imports

  • sg_id – The security group Id to use for ingress. DB Security group, not service’s

  • port – The port for Ingress to the DB.

ecs_composex.rds_resources_settings. define_db_prefix ( db , mappings_definition ) [source]
ecs_composex.rds_resources_settings. define_db_secrets ( db , secret_import , target ) [source]

Function to return the list of env vars set for the DB to use as env vars for the Secret.

Returns :

list of names to use.

Return type :

list

ecs_composex.rds_resources_settings. define_secrets_keys_mappings ( mappings_definition ) [source]

Function to analyze the secrets mapping provided

Parameters :

mappings_definition

Returns :

ecs_composex.rds_resources_settings. filter_out_tag_resources ( lookup_attributes , rds_resource , tagging_api_id ) [source]

Function to return the ClusterARN to use out of multiple found when using GroupTaggingAPI :param dict lookup_attributes: :type rds_resource: :param rds_resource: :param str tagging_api_id: :return: The cluster ARN to use :rtype: str :raises: LookupError

ecs_composex.rds_resources_settings. generate_rds_secrets_permissions ( resources , db_name ) [source]

Function to generate the IAM policy to use for the ECS Execution role to get access to the RDS secrets :rtype: dict :return:

ecs_composex.rds_resources_settings. generate_secret_string ( secret_var_name , secret_import , db , family ) [source]

Generates an additional secret that will put together the connection string that some services require in order to connect to the DB. Generally, not recommended.

Return type :

list

ecs_composex.rds_resources_settings. generate_secrets_from_secrets_mappings ( db , secrets_list , secret_definition , mappings_definition ) [source]

Function to generate a list of EcsSecrets

Parameters :
  • db ( ecs_composex.common.compose_resources.Rds ) – the RDS DB object

  • secrets_list ( list ) –

  • secret_definition

  • mappings_definition

Returns :

ecs_composex.rds_resources_settings. handle_db_secret_to_services ( db , secret_import , target ) [source]

Maps DB Secret to ECS Service containers. It however won’t expose the secret to an AWS SideCar (i.e. x-ray).

Parameters :
Return type :

None

ecs_composex.rds_resources_settings. handle_import_dbs_to_services ( db , target ) [source]

Function to map the Looked up DBs (DocDB and RDS) to the services.

Return type :

None

ecs_composex.rds_resources_settings. handle_new_tcp_resource ( resource , port_parameter , sg_parameter , settings , secret_parameter = None ) [source]

Funnction to standardize TCP services access from services.

ecs_composex.rds_resources_settings. handle_task_role_access ( use_task_role , policy , secret_import , task_role , db , family ) [source]
Return type :

None

ecs_composex.rds_resources_settings. import_dbs ( db , settings ) [source]

Function to go over each service defined in the DB and assign found DB settings to service

Return type :

None

ecs_composex.rds_resources_settings. lookup_rds_resource ( rds_resource , arn_re , native_lookup_function , cfn_resource_type , tagging_api_id , subattribute_key = None ) [source]
Parameters :
  • rds_resource

  • arn_re

  • native_lookup_function

  • cfn_resource_type

  • tagging_api_id

  • subattribute_key

Returns :

ecs_composex.rds_resources_settings. lookup_rds_secret ( rds_resource , secret_lookup ) [source]

Lookup RDS DB Secret specified

Parameters :
Returns :

ecs_composex.resource_settings module

Module to handle resource settings definition to containers.

ecs_composex.resource_settings. add_dependency ( resource , family , settings ) [source]

Add dependency across the resource stack and the ECS Service stack

Parameters :
Return type :

None

ecs_composex.resource_settings. add_new_arns_to_statement_resource ( existing_arns , new_arns ) [source]

Identifies if an ARN in the new arns to add already is set. Returns the list of non duplicate ARNs to add to statement

if type of input is unmanaged, add to the list anyway…

Parameters :
  • existing_arns ( list ) –

  • new_arns ( list ) –

ecs_composex.resource_settings. define_iam_permissions ( resource_mapping_key , dest_resource , dest_resource_template , policy_title , access_type_policy_model , access_definition , resource_arns , access_subkey = None , roles = None , sid_override = None ) [source]

If a policy already exists to manage resources of the same AWS Service, imports the policy, else, creates one. The SID of the policy allows grouping resources that have a similar access pattern together in the same statement policy, reducing the policy length (later, might allow for managed policies). If there were no SID set already in a statement, adds it.

Parameters :
  • resource_mapping_key

  • dest_resource

  • dest_resource_template ( troposphere.Template ) –

  • policy_title ( str ) –

  • access_type_policy_model ( dict ) –

  • access_definition ( str , dict ) –

  • resource_arns ( list ) –

  • access_subkey ( str ) –

  • roles ( list ) – List of Role pointers to use as Policy targets

Return type :

None

ecs_composex.resource_settings. determine_arns ( arn , policy_doc , ignore_missing_primary = False ) [source]

Function allowing to detect whether the resource permissions has a defined override for resources ARN. This allows to extend the ARN syntax.

The policy skeleton must have Resource as a list, and contain ${ARN} into it.

Parameters :
  • arn ( str , list , AWSHelperFn ) –

  • policy_doc ( dict ) – The policy document content

  • ignore_missing_primary ( bool ) – Whether the policy should contain ${ARN} at least

Returns :

The list or Resource to put in to the IAM policy

Return type :

list

ecs_composex.resource_settings. generate_resource_permissions ( resource_name , policies , arn , ignore_missing_primary = False ) [source]

Function to generate IAM permissions for a given x-resource. Returns the mapping of these for the given resource. Suffix takes the values and reduces to the first 118 characters to ensure policy length is below 128 Short prefix ensures the uniqueness of the policy name but allows to be a constant throughout the life of the CFN Stack. It is 8 chars long, leaving a 2 chars margin

Parameters :
  • resource_name ( str ) – The name of the resource

  • policies ( dict ) – the policies associated with the x-resource type.

  • arn ( str , AWSHelper ) – The ARN of the resource if already looked up.

  • ignore_missing_primary ( bool ) – Whether the policy should contain ${ARN} at least

Returns :

dict of the IAM policies associated with the resource.

Rtype dict :

ecs_composex.resource_settings. get_access_type_policy_model ( access_type , policies_models , access_subkey = None ) [source]
Parameters :
  • access_type ( str | dict ) –

  • policies_models ( dict ) –

  • access_subkey ( str ) –

Return type :

dict

Returns :

ecs_composex.resource_settings. get_att_already_set ( existing_arns , cfn_type , get_object , get_prop ) [source]

Compares a list of :type existing_arns: list :param existing_arns: :type get_object: :param get_object: :type get_prop: str :param get_prop: :rtype: bool :return:

ecs_composex.resource_settings. get_parameter_settings ( resource , parameter ) [source]

Function to define a set of values for the purpose of exposing resources settings from their stack to another.

Parameters :
Returns :

Ordered combination of settings

Return type :

tuple

ecs_composex.resource_settings. handle_kms_access ( settings , resource , target ) [source]

Function to map KMS permissions for the services which need access to a resource using a KMS Key

Parameters :
ecs_composex.resource_settings. handle_resource_to_services ( settings , x_resource , arn_parameter , nested = False , access_subkeys = None , targets_overrides = None ) [source]

Function to evaluate the type of resource coming in and pass on the settings and parameters for IAM and otherwise assignment

Parameters :
  • x_resource ( ecs_composex.common.compose_resource.XResource ) –

  • settings ( ecs_composex.common.settings.ComposeXSettings ) –

  • arn_parameter

  • nested ( bool ) –

  • targets_overrides ( list ) – overrides the list of targets to process for the resource

ecs_composex.resource_settings. import_resource_into_service_stack ( settings , resource , family , params_to_add , params_values ) [source]

Function to either add parameters to the services stack or mapping for a given resource

Parameters :
Return type :

None

Links the KMS key of a given resource (if necessary) to the service in order to use that key Avoids having to do x-kms.Lookup to a service

Parameters :
Return type :

None

Returns :

Function to assign the new resource to the service/family using it.

Parameters :
  • settings ( ecs_composex.common.settings.ComposeXSettings ) –

  • resource ( ecs_composex.common.compose_resources.ServicesXResource ) – The resource

  • arn_parameter ( ecs_composex.common.cfn_parameter.Parameter ) – The parameter mapping to the ARN attribute

  • access_subkeys ( list [ str ] ) – Allows to access subkeys from the resource policies

Return type :

None

ecs_composex.resource_settings. map_resource_env_vars_to_family_service_environment ( target , resource ) [source]

Function to expose environment variables to a specific container service from a given x-resource based on the environment defined in the service definition.

These take precedence ReturnValues settings which would be applied to all containers in the family

Parameters :
Return type :

None

ecs_composex.resource_settings. map_resource_env_vars_to_family_services ( target , resource ) [source]

Function to deal with the env vars to add to the family stack based on the resource Services definition

Parameters :
Return type :

None

ecs_composex.resource_settings. map_resource_return_value_to_services_command ( family , settings ) [source]

Checks if their is a x-<res_key>::<name>::<return_value>

Return type :

None

ecs_composex.resource_settings. map_service_perms_to_resource ( family , target , arn_value , resource = None , resource_policies = None , resource_mapping_key = None , access_definition = None , access_subkey = None , ignore_missing_primary = False ) [source]

Maps the resource to the services / target family. Sets up IAM and environment variables

Parameters :
Return type :

None

ecs_composex.resource_settings. map_x_resource_perms_to_resource ( dest_resource , arn_value , access_definition , resource = None , resource_policies = None , resource_mapping_key = None , access_subkey = None , ignore_missing_primary = False ) [source]

Maps an x-resource to another’s IAM Manager

Parameters :
Return type :

None

ecs_composex.resource_settings. set_arn_att_value ( resource , arn_settings , arn_parameter ) [source]
Parameters :
Return type :

troposphere.AWSHelperFn

Returns :

Sets IAM Permissions to the ECS Service to access the resource

Parameters :
  • resource

  • target

  • arn_attr_value ( troposphere.AWSHelperFn ) –

  • access_subkeys ( typing.Optional [ list ]) –

Return type :

None

Returns :

ecs_composex.resource_settings. set_sid_name ( access_definition , access_subkey ) [source]

Defines the name of the SID to use for the policy. Defines access_type

Parameters :
  • access_definition ( dict , str ) –

  • access_subkey ( str ) –

Returns :

access_type

Return type :

str

ecs_composex.resource_settings. set_update_container_env_vars_from_resource_attribute ( svc_container_environment , svc , resource , target ) [source]

For each environment variable set, if it is a string matching the regular expression, replace environment variable value with resource attribute ReturnValue

Parameters :
Return type :

None

Returns :

ecs_composex.resources_import module

Module to import CFN Resources defined by their properties

ecs_composex.resources_import. find_aws_properties_in_aws_resource ( property_type_to_find , resource_properties , found_properties = None ) [source]
Parameters :
  • property_type_to_find

  • resource_properties ( dict ) –

  • found_properties ( list ) –

Return type :

list

Returns :

ecs_composex.resources_import. find_aws_resources_in_template_resources ( root_stack , resource_types ) [source]

Function looking for resources in the stack template that are of the type we are looking for.

Parameters :
  • root_stack ( ComposeXStack ) –

  • resource_types ( tuple ( AWSObject ) ) – the AWSObject resources types we are looking for.

Returns :

List of resources of the given type

Return type :

list

ecs_composex.resources_import. get_dest_resource_nested_property ( properties_path , dest_resource ) [source]

Function that will return the :type properties_path: str :param properties_path: :type dest_resource: AWSObject | AWSProperty :param dest_resource: :rtype: tuple | None :return:

ecs_composex.resources_import. handle_list ( properties , property_class ) [source]

Function to handle list properties

Parameters :
  • property_class

  • properties

Returns :

ecs_composex.resources_import. import_non_functions ( props , prop_name , top_class , properties , set_to_novalue , ignore_missing ) [source]

Function to set property for flat object or recursive to sub properties

Parameters :
  • props ( dict ) –

  • prop_name ( str ) –

  • top_class

  • properties ( dict ) –

  • set_to_novalue ( bool ) –

  • ignore_missing ( bool ) –

ecs_composex.resources_import. import_record_properties ( properties , top_class , set_to_novalue = False , ignore_missing_required = True , ignore_missing_sub_required = False ) [source]

Generic function importing the RecordSet properties. If the property was not defined, it is either left empty or set to AWS::NoValue For inner recursive, we enforce check on required properties.

Parameters :
  • properties ( dict ) –

  • top_class – The class we are going to import properties for

  • set_to_novalue ( bool ) – Instead of skipping the property, actively set to AWS::NoValue

  • ignore_missing_required ( bool ) – Whether raise an error when missing an essential key.

  • ignore_missing_sub_required ( bool ) – Whether raise an error when missing an essential key in sub properties

Returns :

The properties for the RecordSet

Return type :

dict

ecs_composex.resources_import. skip_if ( resource , prop_attr ) [source]

Helper function to skip when conditions are not met to link one resource to another. :type resource: :param resource: :type prop_attr: :param prop_attr: :rtype: bool :return:

Module contents

Top-level package for ECS ComposeX.