ecs_composex.common package

Submodules

ecs_composex.common.aws module

Common functions and variables fetched from AWS.

ecs_composex.common.aws. assert_can_create_stack ( client , name ) [source]

Checks whether a stack already exists or not

ecs_composex.common.aws. assert_can_update_stack ( client , name ) [source]

Checks whether a stack already exists or not

ecs_composex.common.aws. define_lookup_role_from_info ( info , session ) [source]

Function to override ComposeXSettings session to specific session for Lookup

Parameters
  • info

  • session

Returns

boto3 session from lookup settings

Return type

boto3.session.Session

ecs_composex.common.aws. define_tagsgroups_filter_tags ( tags ) [source]

Function to create the filters out of tags list

Parameters

tags ( list ) – list of Key/Value dict

Returns

filters

Return type

list

ecs_composex.common.aws. deploy ( settings , root_stack ) [source]

Function to deploy (create or update) the stack to CFN. :param ComposeXSettings settings: :param ComposeXStack root_stack: :return:

ecs_composex.common.aws. find_aws_resource_arn_from_tags_api ( info , session , aws_resource_search , types = None , allow_multi = False ) [source]

Function to find the RDS DB based on info

Parameters
  • info ( dict ) –

  • session ( boto3.session.Session ) – Boto3 session for clients

  • aws_resource_search ( str ) – Resource type we are after within the AWS Service, ie. cluster, instance

  • types ( dict ) – Additional types to match.

Returns

ecs_composex.common.aws. get_account_id ( session ) [source]

Function to get the current session account ID

Parameters

session ( boto3.session.Session ) – Boto3 Session to make the API call.

Returns

account ID

Return type

str

ecs_composex.common.aws. get_change_set_status ( client , change_set_name , settings ) [source]
ecs_composex.common.aws. get_cross_role_session ( session , arn , session_name = None ) [source]

Function to override ComposeXSettings session to specific session for Lookup

Parameters
  • session ( boto3.session.Session ) – The original session fetching the credentials for X-Role

  • arn ( str ) –

  • session_name ( str ) – Override name of the session

Returns

boto3 session from lookup settings

Return type

boto3.session.Session

ecs_composex.common.aws. get_region_azs ( session ) [source]

Function to return the AZ from a given region. Uses default region for this

Parameters

session ( boto3.session.Session ) – Boto3 session

Returns

list of AZs in the given region

Return type

list

ecs_composex.common.aws. get_resources_from_tags ( session , aws_resource_search , search_tags ) [source]
Parameters
  • session ( boto3.session.Session ) – The boto3 session for API calls

  • aws_resource_search ( str ) – AWS Service short code, ie. rds, ec2

  • search_tags ( list ) – The tags to search the resource with.

Returns

ecs_composex.common.aws. handle_multi_results ( arns , name , res_type , regexp , allow_multi = False ) [source]

Function to evaluate more than one result to see if we can match an unique name.

Parameters
  • arns ( list ) –

  • name ( str ) –

  • res_type ( str ) –

  • regexp ( str ) –

Raises

LookupError

Returns

The ARN of the resource matching the name.

ecs_composex.common.aws. handle_search_results ( arns , name , res_types , aws_resource_search , allow_multi = False ) [source]

Function to parse tag resource search results

Parameters
  • arns ( list ) –

  • name ( str ) –

  • res_types ( dict ) –

  • aws_resource_search ( str ) –

Returns

ecs_composex.common.aws. plan ( settings , root_stack ) [source]

Function to create a recursive change-set and return diffs :param ComposeXSettings settings: :param ComposeXStack root_stack: :return:

ecs_composex.common.aws. validate_search_input ( res_types , res_type ) [source]

Function to validate the search query

Parameters
  • res_types ( dict ) –

  • res_type ( str ) –

Returns

ecs_composex.common.aws. validate_stack_availability ( settings , root_stack ) [source]

Function to check that the stack can be updated :param settings: :param root_stack: :return:

ecs_composex.common.cfn_conditions module

Common Conditions across the templates

ecs_composex.common.cfn_conditions. define_stack_name ( template = None ) [source]

Function to return Stack name contstruct. Adds the conditions and parameters if template is given.

Parameters

template ( troposphere.Template ) – the template to add it to.

Returns

ecs_composex.common.cfn_conditions. pass_root_stack_name ( ) [source]

Function to add root_stack to a stack parameters

Returns

rootstack name value based on condition

ecs_composex.common.cfn_params module

” Common parameters for CFN This is a crucial part as all the titles, marked _T are string which are then used the same way across all imports, which gives consistency for CFN to use the same names, which it heavily relies onto.

You can change the names values so you like so long as you keep it Alphanumerical [a-zA-Z0-9]

class ecs_composex.common.cfn_params. Parameter ( title , return_value = None , ** kwargs ) [source]

Bases: troposphere.Parameter

Class to extend the default Parameter behaviour

ecs_composex.common.compose_networks module

Class and functions to interact with the networks: defined in compose files.

class ecs_composex.common.compose_networks. ComposeNetwork ( name , definition , subnets_list ) [source]

Bases: object

Class to keep track of the Docker-compose Volumes

driver_opts_key = 'driver'
main_key = 'networks'
ecs_composex.common.compose_networks. match_networks_services_config ( service , net_config , networks ) [source]

Function to map network config in services and top-level networks

Parameters
  • service

  • net_config ( dict ) –

  • networks ( list ) –

Raises

LookupError

ecs_composex.common.compose_resources module

Module to define the ComposeX Resources into a simple object to make it easier to navigate through.

class ecs_composex.common.compose_resources. XResource ( name , definition , module_name , settings ) [source]

Bases: object

Class to represent each defined resource in the template

Variables
  • name ( str ) – The name of the resource as defined in compose file

  • definition ( dict ) – The definition of the resource as defined in compose file

  • logical_name ( str ) – Name of the resource to use in CFN template as for export/import

debug_families_targets ( ) [source]

Method to troubleshoot family and service mapping

define_export_name ( output_definition , attribute_parameter ) [source]

Method to define the export name for the resource :return:

define_ref_env_vars ( env_name , parameter ) [source]

Method to define construct parameters for Environment Variable for default Ref value of resource

Parameters
Returns

dict with the Name and Value for environment variable

Return type

dict

define_return_value_env_vars ( env_name , parameter ) [source]

Method to define construct parameters for Environment Variable for parameters with specific return_value

Parameters
Returns

dict with the Name and Value for environment variable

Return type

dict

generate_outputs ( ) [source]

Method to create the outputs for XResources

generate_resource_envvars ( ) [source]

Method to define all the env var of a resource based on its own defined output attributes

handle_families_targets_expansion ( service , settings ) [source]

Method to list all families and services that are targets of the resource. Allows to implement family and service level association to resource

Parameters
handle_family_scaling_expansion ( service , settings ) [source]

Method to search for the families of given service and add it if not already present

Parameters
Returns

init_env_names ( ) [source]

Method to define the environment variables for the resource

Returns

list of environment variable names

Return type

list

policies_scaffolds = {}
set_attributes_from_mapping ( attribute_parameter ) [source]

Method to define the attribute outputs for lookup resources, which use FindInMap or Ref

Parameters

attribute_parameter ( ecs_composex.common.cfn_params.Parameter ) – The parameter mapped to the resource attribute

Returns

The FindInMap setting for mapped resource

set_new_resource_outputs ( output_definition , attribute_parameter ) [source]

Method to define the outputs for the resource when new

set_override_subnets ( ) [source]
set_services_scaling ( settings ) [source]

Method to map services and families targets of the services defined.

Parameters

settings ( ecs_composex.common.settings.ComposeXSettings ) –

Returns

set_services_targets ( settings ) [source]

Method to map services and families targets of the services defined. TargetStructure: (family, family_wide, services[], access)

Parameters

settings ( ecs_composex.common.settings.ComposeXSettings ) –

Returns

ecs_composex.common.compose_resources. get_parameter_settings ( resource , parameter ) [source]

Function to define a set of values for the purpose of exposing resources settings from their stack to another.

Parameters
  • resource – The XResource we want to extract the outputs from

  • parameter – The parameter we want to extract the outputs for

Returns

Ordered combination of settings

Return type

tuple

ecs_composex.common.compose_resources. get_setting_key ( name , settings_dict ) [source]
ecs_composex.common.compose_resources. set_resources ( settings , resource_class , res_key , mod_key = None ) [source]

Method to define the ComposeXResource for each service.

Parameters

ecs_composex.common.compose_services module

class ecs_composex.common.compose_services. ComposeFamily ( services , family_name ) [source]

Bases: object

Class to group services logically to create the final ECS Service

add_container_level_log_group ( service , log_group_title , expiry ) [source]

Method to add a new log group for a specific container/service defined when awslogs-group has been set.

Parameters
  • service

  • log_group_title ( str ) –

  • expiry

add_service ( service ) [source]
apply_services_params ( ) [source]
assign_iam_managed_policies ( role , prop ) [source]

Method to assign managed policies to IAM role

Parameters
  • role

  • prop

Returns

assign_iam_policies ( role , prop ) [source]

Method to handle assignment of IAM policies defined from compose file.

Parameters
  • role

  • prop

Returns

assign_policies ( role_name = None ) [source]

Method to assign IAM configuration (policies, boundary etc.) to the Task Role. Role can be overriden

Parameters

role_name ( str ) – The role LogicalName as defined in the template

assign_predefined_alerts ( service_predefined , valid_predefined , settings_key ) [source]
define_default_alarm_settings ( key , value , settings_key , valid_predefined ) [source]
define_predefined_alarm_settings ( new_settings ) [source]

Method to define the predefined alarm settings based on the alarm characteristics

Parameters

new_settings

Returns

define_predefined_alarms ( ) [source]

Method to define which predefined alarms are available :return: dict of the alarms :rtype: dict

define_shared_volumes ( ) [source]

Method to create a list of shared volumes within the task family and set the volume to shared = True if not.

Returns

list of shared volumes within the task definition

Return type

list

handle_alarms ( ) [source]

Method to define the alarms for the services.

handle_iam ( ) [source]
handle_logging ( ) [source]

Method to go over each service logging configuration and accordingly define the IAM permissions needed for the exec role

handle_permission_boundary ( prop_key ) [source]
init_task_definition ( ) [source]
merge_alarm_settings ( key , value , settings_key , valid_predefined ) [source]

Method to merge multiple services alarms definitions

Parameters
  • key ( str ) –

  • value ( dict ) –

  • settings_key ( str ) –

Returns

refresh ( ) [source]
refresh_container_logging_definition ( ) [source]
set_initial_services_dependencies ( ) [source]

Method to iterate over each depends_on service set in the family services and add them up

Returns

set_merge_alarm_topics ( key , value ) [source]
set_repository_credentials ( settings ) [source]

Method to go over each service and identify which ones have credentials to pull the Docker image from a private repository

Parameters

settings ( ecs_composex.common.settings.ComposeXSettings ) –

Returns

set_secrets_access ( ) [source]

Method to handle secrets permissions access

set_service_update_config ( ) [source]

Method to determine the update_config for the service. When a family has multiple containers, this applies to all tasks.

set_services_mount_points ( ) [source]

Method to set the mount points to the Container Definition of the defined service

set_task_compute_parameter ( ) [source]

Method to update task parameter for CPU/RAM profile

set_task_definition ( ) [source]

Function to set or update the task definition

Parameters

self – the self of services

set_volumes ( ) [source]

Method to create the volumes definition to the Task Definition

Returns

set_xray ( ) [source]
sort_container_configs ( ) [source]

Method to sort out the containers dependencies and create the containers definitions based on the configs. :return:

sort_iam_settings ( key , setting ) [source]

Method to sort out iam configuration

Parameters
  • key ( tuple ) –

  • setting ( dict ) –

Returns

update_family_subnets ( settings ) [source]

Method to update the stack parameters

Parameters

settings ( ecs_composex.common.settings.ComposeXSettings ) –

upload_services_env_files ( settings ) [source]

Method to go over each service and if settings are to upload files to S3, will create objects and update the container definition for env_files accordingly.

Parameters

settings ( ecs_composex.common.settings.ComposeXSettings ) –

Returns

validate_service_predefined_alarms ( valid_predefined , service_predefined ) [source]
class ecs_composex.common.compose_services. ComposeService ( name , definition , volumes = None , secrets = None ) [source]

Bases: object

Class to represent a service

Variables

container_name ( str ) – name of the container to use in definitions

define_compose_logging ( ) [source]

Method to define logging for service.

define_essential ( deployment ) [source]

Method to define whether the container is essential. :param dict deployment:

define_families ( ) [source]

Function to assign the service to a family / families :param deployment: :return:

define_kernel_options ( ) [source]

Method to define and return the kernel option settings for cap_add and cap_drop :return:

define_logging ( ) [source]

Method to define logging properties

define_port_mappings ( ) [source]

Method to determine the list of port mappings to use for either AWS VPC deployments or else (bridge etc). Not in use atm as AWS VPC is made mandatory

define_shm_size ( ) [source]

Method to import and determine SHM SIZE

define_start_condition ( deployment ) [source]

Method to define the start condition success for the container

Parameters

deployment

Returns

define_sysctls ( ) [source]

Method to define the SystemControls

define_tmpfs ( ) [source]

Method to define the tmpfs settings

define_ulimits ( ) [source]

Method to set the ulimits

ecs_plugin_aws_keys = [('x-aws-role', <class 'dict'>), ('x-aws-policies', <class 'list'>), ('x-aws-autoscaling', <class 'dict'>), ('x-aws-pull_credentials', <class 'str'>), ('x-aws-logs_retention', <class 'int'>), ('x-aws-min_percent', <class 'int'>), ('x-aws-max_percent', <class 'int'>)]
import_x_aws_settings ( ) [source]
is_in_family ( family_name ) [source]

Method to check whether this service is part of a given family

Parameters

family_name ( str ) –

Returns

True/False

Return type

bool

keys = [('build', (<class 'str'>, <class 'dict'>)), ('cap_add', <class 'list'>), ('cap_drop', <class 'list'>), ('cgroup_parent', <class 'str'>), ('command', (<class 'list'>, <class 'str'>)), ('configs', <class 'dict'>), ('container_name', <class 'str'>), ('credential_spec', <class 'str'>), ('depends_on', <class 'list'>), ('deploy', <class 'dict'>), ('devices', <class 'list'>), ('dns', (<class 'list'>, <class 'str'>)), ('dns_search', <class 'list'>), ('entrypoint', (<class 'str'>, <class 'list'>)), ('environment', (<class 'list'>, <class 'dict'>)), ('env_file', (<class 'list'>, <class 'str'>)), ('expose', <class 'list'>), ('external_links', <class 'list'>), ('extra_hosts', <class 'list'>), ('healthcheck', <class 'dict'>), ('hostname', <class 'str'>), ('labels', <class 'dict'>), ('labels', (<class 'dict'>, <class 'list'>)), ('logging', <class 'dict'>), ('links', <class 'list'>), ('network_mode', <class 'str'>), ('networks', (<class 'list'>, <class 'dict'>)), ('image', <class 'str'>), ('init', <class 'bool'>), ('isolation', <class 'str'>), ('pid', <class 'str'>), ('ports', <class 'list'>), ('privileged', <class 'bool'>), ('read_only', <class 'bool'>), ('restart', <class 'str'>), ('shm_size', <class 'str'>), ('security_opt', <class 'str'>), ('secrets', <class 'list'>), ('stop_signal', <class 'str'>), ('sysctls', (<class 'list'>, <class 'dict'>)), ('tmpfs', (<class 'str'>, <class 'list'>)), ('ulimits', <class 'dict'>), ('user', (<class 'int'>, <class 'str'>)), ('userns_mode', <class 'str'>), ('volumes', <class 'list'>), ('x-configs', <class 'dict'>), ('x-logging', <class 'dict'>), ('x-iam', <class 'dict'>), ('x-xray', <class 'bool'>), ('x-scaling', <class 'dict'>), ('x-network', <class 'dict'>), ('x-alarms', <class 'dict'>)]
main_key = 'services'
map_secrets ( secrets ) [source]
map_volumes ( volumes = None ) [source]

Method to apply mapping of volumes to the service and define the mapping configuration

Parameters

volumes ( list ) –

Returns

merge_x_aws_role ( key ) [source]

Method to update the service definition with the x-aws-role information if NOT defined in the composex definition.

Parameters

key ( str ) –

merge_x_policies ( key ) [source]

Method to merge policies

Parameters

key ( str ) –

set_add_capacities ( add_key , valid , cap_adds , all_adds , fargate ) [source]

Method to set the kernel capacities to add

Parameters
  • add_key ( str ) –

  • valid ( list ) –

  • cap_adds ( list ) –

  • all_adds ( list ) –

  • fargate ( list ) –

set_compute_resources ( deployment ) [source]

Function to analyze the Docker Compose deploy attribute and set settings accordingly. deployment keys: replicas, mode, resources

Parameters

deployment ( dict ) – definition[‘deploy’]

set_container_definition ( ) [source]

Function to define the container definition matching the service definition

set_drop_capacities ( drop_key , valid , cap_adds , all_adds , all_drops , fargate ) [source]

Method to set the drop kernel capacities

Parameters
  • drop_key ( str ) –

  • valid ( list ) –

  • cap_adds ( list ) –

  • all_adds ( list ) –

  • all_drops ( list ) –

  • fargate ( list ) –

set_ecs_healthcheck ( ) [source]

Function to set healtcheck configuration :return:

set_env_files ( ) [source]

Method to list all the env files and check the files are found and available.

set_networks ( ) [source]
set_replicas ( deployment ) [source]

Function to set the service deployment settings.

set_service_deploy ( ) [source]

Function to setup the service configuration from the deploy section of the service in compose file.

set_update_config ( deployment ) [source]

Method to set the update_config from the deploy service keys

Parameters

deployment ( dict ) –

Returns

set_user_group ( ) [source]

Method to assign the user / group IDs for the container

set_x_credentials_secret ( key ) [source]

Method that will set the secret associated to the service to retrieve the docker image if defined through x-aws-pull_credentials

ecs_composex.common.compose_services. add_policies ( config , key , new_policies ) [source]
ecs_composex.common.compose_services. assign_policy_to_role ( role_secrets , role ) [source]

Function to assign the policy to role Policies :param list role_secrets: :param troposphere.iam.Role role: :return:

ecs_composex.common.compose_services. assign_secrets_to_roles ( secrets , exec_role , task_role ) [source]

Function to assign secrets access policies to exec_role and/or task_role

Parameters
  • secrets

  • exec_role

  • task_role

Returns

ecs_composex.common.compose_services. handle_iam_boundary ( config , key , new_value ) [source]
Parameters
  • config – the IAM Config

  • key – The key, here, boundary

  • new_value

ecs_composex.common.compose_services. handle_same_task_services_dependencies ( services_config ) [source]

Function to define inter-tasks dependencies

Parameters

services_config ( list ) –

Returns

ecs_composex.common.compose_services. identify_repo_credentials_secret ( settings , task , secret_name ) [source]

Function to identify the secret_arn :param settings: :param ComposeFamily task: :param secret_name: :return:

ecs_composex.common.compose_volumes module

Class and functions to interact with the volumes: defined in compose files.

class ecs_composex.common.compose_volumes. ComposeVolume ( name , definition ) [source]

Bases: object

Class to keep track of the Docker-compose Volumes

When properties are defined, the priority in evaluation goes * x-efs * driver * driver_opts

Assumed local when none else defined.

driver_key = 'driver'
driver_opts_key = 'driver_opts'
efs_defaults = {'Encrypted': True, 'LifecyclePolicies': [{'TransitionToIA': 'AFTER_14_DAYS'}], 'PerformanceMode': 'generalPurpose'}
main_key = 'volumes'
ecs_composex.common.compose_volumes. evaluate_plugin_efs_properties ( definition ) [source]

Function to parse the definition in case user uses the docker cli definition for EFS

Returns

ecs_composex.common.compose_volumes. handle_volume_dict_config ( service , config , volumes ) [source]
Parameters
ecs_composex.common.compose_volumes. handle_volume_str_config ( service , config , volumes ) [source]

Function to return the volume configuration (long) :param ComposeService service: :param str config: :param list volumes:

ecs_composex.common.compose_volumes. is_tmpfs ( config ) [source]

Function to identify whether the volume defined is tmpfs

Parameters

config ( dict ) –

Returns

whether the volume defined is tmpfs

Return type

bool

ecs_composex.common.compose_volumes. match_volumes_services_config ( service , vol_config , volumes ) [source]

Function to map volume config in services and top-level volumes

Parameters
  • service

  • vol_config

  • volumes

Raises

LookupError

ecs_composex.common.config module

class ecs_composex.common.config. ComposeXConfig ( settings ) [source]

Bases: object

Class to parse and define configuration settings for ECS ComposeX

composex_key = 'composex'
master_key = 'x-configs'
class ecs_composex.common.config. ComputeConfig ( settings ) [source]

Bases: ecs_composex.common.config.ComposeXConfig

Class to determine the compute settings to use when deploying on top of EC2.

default_spot_config = {'bid_price': 0.42, 'spot_instance_types': {'m5a.2xlarge': {'weight': 7}, 'm5a.4xlarge': {'weight': 15}, 'm5a.xlarge': {'weight': 3}}, 'use_spot': True}
spot_key = 'spot_config'

ecs_composex.common.ecs_composex module

Global settings and variables re-used across the project

ecs_composex.common.envsubst module

Module to do a better env variables handling.

ecs_composex.common.envsubst. expandvars ( path , default = None , skip_escaped = True ) [source]
Expand environment variables of form $var and ${var}.

If parameter ‘skip_escaped’ is True, all escaped variable references (i.e. preceded by backslashes) are skipped. Unknown variables are set to ‘default’. If ‘default’ is None, they are left unchanged.

ecs_composex.common.files module

Functions to manage a template and wheter it should be stored in S3

class ecs_composex.common.files. FileArtifact ( file_name , settings , file_format = None , template = None , content = None ) [source]

Bases: object

Class to handle files artifacts, such as configuration files or templates. It will allow to upload the content to S3 or write to local filesystem. It also handles CloudFormation templates validation.

Variables
  • url ( str ) – The URL in S3 where the file will be uploaded to or available from.

  • body ( str ) – The content of the FileArtifact

  • template ( troposphere.Template ) – the CFN template

  • file_name ( str ) – the base name of the file

  • mime ( str ) – MIME-type of the file

  • session ( boto3.session.Session ) – session for clients to make API calls to AWS

  • can_upload ( bool ) – Indicate whether or not config allows for upload to S3.

  • no_upload ( bool ) – Turns off upload if True

  • validate ( bool ) – Indicates whether the template is validated.

  • output_dir ( str ) – Path to the local director to output the file to.

  • file_path ( str ) – Output file path for the FileArtifact

define_body ( ) [source]

Method to define the body of the file artifact. Sets the mime type that will be used for upload into S3.

define_file_specs ( file_name , file_format , settings ) [source]

Method to set the file body from template if self.template is Template

Parameters
  • file_name – name of the file

  • file_format – format to use for the file.

  • settings – The settings for execution

Returns

file_path = None
mime = 'text/plain'
upload ( settings ) [source]

Method to handle uploading the files to S3.

validate ( settings ) [source]

Method to validate the CloudFormation template, either via URL once uploaded to S3 or via TemplateBody

write ( settings ) [source]

Method to write the files to local filesystem based on parameters (directory name etc.)

ecs_composex.common.files. upload_file ( body , bucket_name , file_name , settings , prefix = None , mime = None ) [source]

Upload template_body to a file in s3 with given prefix and bucket_name

Parameters
  • body ( str ) – Template body, would come from troposphere template to_json() or to_yaml()

  • bucket_name ( str ) – name of the bucket to upload the file to

  • file_name ( str ) – Name of the file

  • prefix ( str , optional ) – override default prefix for the file in S3

Returns

url_path, the https://s3.amazonaws.com/ URL to the file

Return type

str

ecs_composex.common.outputs module

Functions to format CFN template Outputs

class ecs_composex.common.outputs. ComposeXOutput ( resource , values , export = True , duplicate_attr = False ) [source]

Bases: object

Class to make the output easier.

delim = '::'
root_string_base = '${RootStackName}::'
stack_string_base = '${AWS::StackName}::'
validate_input ( resource , values , export = True , duplicate_attr = False ) [source]
ecs_composex.common.outputs. get_import_value ( title , attribute_name , delimiter = None ) [source]

Wrapper function to define ImportValue for defined resource name

Parameters
  • title – name of the resource exported

  • attribute_name – attribute exported

  • delimiter – delimiter between stack name, resource name and attribute

Returns

ecs_composex.common.outputs. validate ( value ) [source]

Method to validate the input :raises: ValueError

ecs_composex.common.services_helpers module

ecs_composex.common.services_helpers. define_ingress_mappings ( service_ports ) [source]

Function to create a mapping of sources for a common target

ecs_composex.common.services_helpers. define_string_interpolation ( var_value ) [source]

Function to determine whether an env variable string should use Sub.

Parameters

var_value ( str ) – The env var string as defined in compose file

Returns

String as is or Sub for interpolation

Return type

str

ecs_composex.common.services_helpers. extend_container_envvars ( container , env_vars ) [source]
ecs_composex.common.services_helpers. extend_container_secrets ( container , secret ) [source]

Function to add secrets to a Container definition

Parameters
  • container ( troposphere.ecs.ContainerDefinition ) – container definition

  • secret ( troposphere.ecs.Secret ) – secret to add

ecs_composex.common.services_helpers. import_env_variables ( environment ) [source]

Function to import Docker compose env variables into ECS Env Variables

Parameters

environment ( dict ) – Environment variables as defined on the ecs_service definition

Returns

list of Environment

Return type

list<troposphere.ecs.Environment>

ecs_composex.common.services_helpers. import_secrets ( template , service , container , settings ) [source]

Function to import secrets from composex mapping to AWS Secrets in Secrets Manager

Parameters
Returns

ecs_composex.common.services_helpers. set_else_none ( key , props , alt_value = None , eval_bool = False ) [source]

Function to serialize if not keyisset () set other value

Parameters
  • key ( str ) –

  • props ( dict ) –

  • alt_value

  • eval_bool ( bool ) – Allows to gets booleans properties

Returns

ecs_composex.common.services_helpers. set_logging_expiry ( service ) [source]

Method to reset the logging retention period to the closest valid value.

Parameters

service ( ecs_composex.common.compose_services.ComposeService ) –

Returns

ecs_composex.common.services_helpers. validate_healthcheck ( healthcheck , valid_keys , required_keys ) [source]

Healthcheck definition validation

Parameters
  • healthcheck ( dict ) –

  • valid_keys ( list ) –

  • required_keys ( list ) –

ecs_composex.common.settings module

Module for the ComposeXSettings class

class ecs_composex.common.settings. ComposeXSettings ( content = None , profile_name = None , session = None , for_macro = False , ** kwargs ) [source]

Bases: object

Class to handle the settings to use for ECS ComposeX.

active_commands = [{'name': 'up', 'help': 'Generates & Validates the CFN templates, Creates/Updates stack in CFN'}, {'name': 'render', 'help': 'Generates & Validates the CFN templates locally. No upload to S3'}, {'name': 'create', 'help': 'Generates & Validates the CFN templates locally. Uploads files to S3'}, {'name': 'plan', 'help': 'Creates a recursive change-set to show the diff prior to an update'}]
add_new_family ( family_name , service , assigned_services ) [source]
all_commands = [{'name': 'up', 'help': 'Generates & Validates the CFN templates, Creates/Updates stack in CFN'}, {'name': 'render', 'help': 'Generates & Validates the CFN templates locally. No upload to S3'}, {'name': 'create', 'help': 'Generates & Validates the CFN templates locally. Uploads files to S3'}, {'name': 'plan', 'help': 'Creates a recursive change-set to show the diff prior to an update'}, {'name': 'config', 'help': 'Merges docker-compose files to provide with the final compose content version'}, {'name': 'init', 'help': 'Initializes your AWS Account with prerequisites settings for ECS'}, {'name': 'version', 'help': 'ECS ComposeX Version'}]
allowed_formats = ['json', 'yaml', 'text']
arn_arg = 'RoleArn'
bucket_arg = 'BucketName'
cluster_name_arg = 'ClusterName'
command_arg = 'command'
config_render_arg = 'config'
create_arg = 'create'
create_ec2_arg = 'AddComputeResources'
create_spotfleet_arg = 'UseSpotFleetHosts'
create_vpc_arg = 'CreateVpc'
default_azs = ['eu-west-1a', 'eu-west-1b']
default_format = 'json'
default_output_dir = '/tmp/1623920812'
default_vpc_cidr = '100.127.254.0/24'
deploy_arg = 'up'
evaluate_private_namespace ( ) [source]

Method to go over all services and figure out if any of them requires cloudmap. If so it will also expect x-dns.PrivateNamespace to be set.

format_arg = 'TemplateFormat'
handle_assigned_existing_service ( family_name , service , assigned_services ) [source]
init_s3 ( ) [source]

Method to initialize S3 settings

Returns

input_file_arg = 'DockerComposeXFile'
name_arg = 'Name'
neutral_commands = [{'name': 'init', 'help': 'Initializes your AWS Account with prerequisites settings for ECS'}, {'name': 'version', 'help': 'ECS ComposeX Version'}]
output_dir_arg = 'OutputDirectory'
override_session ( session , profile_name , kwargs ) [source]

Method to set the session based on input params

Parameters
  • session ( boto3.session.Session ) – The session to override the API calls with

  • profile_name ( str ) – Name of a profile configured in .aws/config

  • kwargs ( dict ) – CLI kwargs

parse_command ( kwargs , content = None ) [source]

Method to analyze the command and set execution settings accordingly.

Parameters
  • kwargs ( dict ) –

  • content ( dict ) –

Returns

plan_arg = 'plan'
region_arg = 'RegionName'
render_arg = 'render'
set_azs_from_api ( ) [source]

Method to set the AWS Azs based on DescribeAvailabilityZones :return:

set_azs_from_vpc_import ( subnets , session = None ) [source]

Function to get the list of AZs for a given set of subnets :param dict subnets: :param session: The Session used to find the EC2 subnets (useful for lookup). :return:

set_bucket_name_from_account_id ( ) [source]
set_content ( kwargs , content = None , fully_load = True ) [source]

Method to initialize the compose content

Parameters
  • kwargs ( dict ) –

  • content ( dict ) –

  • fully_load ( bool ) –

set_efs ( ) [source]

Method to add a x-efs definition to the compose-x definition when a volume is flagged as using NFS/EFS

set_families ( ) [source]

Method to define the list of families :return:

set_networks ( vpc_stack , root_stack ) [source]

Method configuring the networks defined at root level :return:

set_output_settings ( kwargs ) [source]

Method to set the output settings based on kwargs

set_secrets ( ) [source]

Function to parse the settings compose content and define the secrets.

Parameters

settings ( ecs_composex.common.settings.ComposeXSettings ) –

Returns

set_services ( ) [source]

Method to define the ComposeXResource for each service. :return:

set_volumes ( ) [source]

Method configuring the volumes at root level :return:

single_nat_arg = 'SingleNat'
validation_commands = [{'name': 'config', 'help': 'Merges docker-compose files to provide with the final compose content version'}]
vpc_cidr_arg = 'VpcCidr'
zones_arg = 'Zones'

ecs_composex.common.stacks module

Module to handle Root stacks and substacks in ECS composeX. Allows to treat everything in memory before uploading files into S3 and on disk.

class ecs_composex.common.stacks. ComposeXStack ( name , stack_template , stack_parameters = None , file_name = None , ** kwargs ) [source]

Bases: troposphere.cloudformation.Stack , object

Class to define a CFN Stack as a composition of its template object, parameters, tags etc.

Variables

template_file ( ecs_composex.common.files.FileArtifact ) – The FileArtifact associated with the stack.

add_dependencies ( dependencies ) [source]

Function to add dependencies to DependsOn :return:

add_parameter ( parameter ) [source]

Function to add a parameter or set of parameters to the stack :param parameter: :return:

attributes = ['Condition', 'CreationPolicy', 'DeletionPolicy', 'DependsOn', 'Metadata', 'UpdatePolicy', 'UpdateReplacePolicy']
get_from_vpc_stack ( vpc_stack , * parameters ) [source]
get_top_root_stack ( ) [source]
is_void = False
mark_nested_stacks ( ) [source]

Method to go over the stack resources, identify the nested stacks, and set a marker of the parent to them

no_vpc_parameters ( settings ) [source]

Method to set the stack parameters when we are not creating a VPC.

render ( settings ) [source]

Function to use when the template is finalized and can be uploaded to S3.

render_parameters_list_cfn ( ) [source]

Renders parameters in a CFN parameters config file format

Returns

params

Return type

list

write_config_file ( settings ) [source]

Method to write the parameters file for the stack. Only uses manual input.

ecs_composex.common.stacks. process_stacks ( root_stack , settings , is_root = True ) [source]

Function to go through all stacks of a given template and update the template It will recursively render sub stacks defined.

Parameters
ecs_composex.common.stacks. render_codepipeline_config_file ( parameters ) [source]

Method to write all the parameters in the AWS CFN Config format for Codepipeline :param list parameters: :return:

ecs_composex.common.tagging module

From the docker ComposeX definition file, allows to add generic tags to all objects supporting AWS Tags from CFN

Obviously as per AWS CFN API, when you create a stack with Tags, these tags propagate to all resources that support tagging via CFN. Which is great, but very little people use that. And given that the AWS Stack itself has no cost, the FinOps people usually only are able to track the resources that generate actual billing.

By adding the tags to the resources wherever supported and have these as parameters, this enforces the presence of some tags, even though their values might differ. Using parameters to set the values also allows to copy-paste templates within the same organization and simply change the values at the time of creating the CFN Stack.

You might have IAM policies in place to try to force tagging. I find this doesn’t use a lot of parameters unless you had an insane number of tags.

When defining the tags in ComposeX as a list, the names of your tags can contain some special characters which otherwise you could not, i.e., vpc::usage::ecsapps

ecs_composex.common.tagging. add_all_tags ( root_template , settings , params = None , xtags = None ) [source]

Function to go through all stacks of a given template and update the template It will recursively render sub stacks defined. If there are no substacks, it will go over the resources of the template add the tags.

Parameters
  • root_template ( troposphere.Template ) – the root template to iterate over the resources.

  • settings ( ecs_composex.common.settings.ComposeXSettings ) – Execution settings

  • params ( list ) – Parameters to add to template if any

  • xtags ( troposphere.Tags ) – List of Tags to add to the resources.

ecs_composex.common.tagging. add_object_tags ( obj , tags ) [source]

Function to add tags to the object if the object supports it

Parameters
  • obj – Troposphere object to add the tags to

  • tags ( troposphere.Tags ) – list of tags as defined in Docker composeX file

ecs_composex.common.tagging. apply_tags_to_resources ( settings , resource , params , xtags ) [source]
Parameters
  • settings ( ecs_composex.common.settings.ComposeXSettings ) – Execution settings

  • resource – The resource to add the tags to

  • params ( list ) – Parameters to add to template if any

  • xtags ( troposphere.Tags ) – List of Tags to add to the resources.

Returns

ecs_composex.common.tagging. default_tags ( ) [source]
ecs_composex.common.tagging. define_extended_tags ( tags ) [source]

Function to generate the tags to be added to objects from x-tags

Parameters

tags ( list or dict ) – tags as defined in composex file

Returns

Tags() or None

Return type

troposphere.Tags or None

ecs_composex.common.tagging. define_tag_parameter_title ( tag_name ) [source]

Returns the formatted name title for a given tag

Parameters

tag_name ( str ) – name of the tag as defined in the ComposeX file

Returns

reformatted tag name to work on CFN

Return type

str

ecs_composex.common.tagging. expand_launch_template_tags_specs ( lt , tags ) [source]

Function to expand the LaunchTemplate TagSpecifications with defined x-tags.

Parameters
  • lt – the LaunchTemplate object

  • tags ( troposphere.Tags ) – the Tags as built from x-tags

Type

troposphere.ec2.LaunchTemplate

ecs_composex.common.tagging. generate_tags_parameters ( tags ) [source]

Function to generate a list of parameters used for the tags values

Returns

list of parameters and tags to add to objects

Return type

tuple

ecs_composex.common.tagging. merge_tags_lists ( x_data , y_data ) [source]

Module contents

Most commonly used functions shared across all modules.

ecs_composex.common. add_defaults ( template ) [source]

Function to CFN parameters and conditions to the template whhich are used across ECS ComposeX

Parameters

template ( Template ) – source template to add the params and conditions to

ecs_composex.common. add_outputs ( template , outputs ) [source]

Function to add parameters to the template

Parameters
  • template ( troposphere.Template ) – the template to add the parameters to

  • outputs ( list<troposphere.Output> ) – list of parameters to add to the template

ecs_composex.common. add_parameters ( template , parameters ) [source]

Function to add parameters to the template

Parameters
  • template ( troposphere.Template ) – the template to add the parameters to

  • parameters ( list<troposphere.Parmeter> ) – list of parameters to add to the template

ecs_composex.common. build_default_stack_parameters ( stack_params , ** kwargs ) [source]

Function to check and define default parameters for the root stack from the CLI options

Parameters
  • stack_params ( list ) – list of parameters to add to to use for the root stack

  • kwargs ( dict ) – extended arguments

ecs_composex.common. build_parameters_file ( params , parameter_name , parameter_value ) [source]

Function to build arguments file to pass onto CFN. Adds the parameter key/value so it can be written to file afterwards

Parameters
  • params ( list ) – list of parameters

  • parameter_name ( str ) – key of the parameter

  • parameter_value ( str||list ) – value of the parameter

ecs_composex.common. build_template ( description = None , * parameters ) [source]

Entry point function to creating the template for ECS ComposeX resources

Parameters
  • description ( str , optional ) – Optional custom description for the CFN template

  • parameters ( List<troposphere.Parameters> , optional ) – List of optional parameters to add to the template.

Returns template

the troposphere template

Return type

Template

ecs_composex.common. init_template ( description = None ) [source]

Function to initialize the troposphere base template

Parameters

description ( str ) – Description used for the CFN

Returns

template

Return type

Template

ecs_composex.common. keyisset ( x , y ) [source]

Macro to figure if the the dictionary contains a key and that the key is not empty

Parameters
  • x ( str ) – The key to check presence in the dictionary

  • y ( dict ) – The dictionary to check for

Returns

True/False

Return type

bool

ecs_composex.common. keypresent ( x , y ) [source]

Macro to figure if the the dictionary contains a key and that the key is not empty

Parameters
  • x ( str ) – The key to check presence in the dictionary

  • y ( dict ) – The dictionary to check for

Returns

True/False

Return type

bool

ecs_composex.common. load_composex_file ( file_path ) [source]

File to load and read the docker compose file

Parameters

file_path ( str ) – path to the docker compose file

Returns

content of the docker file

Return type

dict

ecs_composex.common. no_value_if_not_set ( props , key , is_bool = False ) [source]

Function to simplify setting value if the key is in the dict and else Ref(AWS_NO_VALUE) for resource properties

Parameters
  • props ( dict ) –

  • key ( str ) –

  • is_bool ( bool ) –

Returns

ecs_composex.common. setup_logging ( ) [source]

Function to setup logging for ECS ComposeX. In case this is used in a Lambda function, removes the AWS Lambda default log handler

Returns

the_logger

Return type

Logger

ecs_composex.common. validate_input ( compose_content , res_key ) [source]

Function to validate the resources names in ComposeX File for a given resource key

Parameters
  • compose_content ( dict ) – the docker/ComposeX content

  • res_key ( str ) – key of the category in docker compose to look for

Returns

True/False if all keys are valid

Return type

bool

ecs_composex.common. validate_kwargs ( required_keys , kwargs , caller = None ) [source]

Function to ensure minimum keys in kwargs are present.

Parameters
  • required_keys ( list ) – the list of keys that have to be present

  • kwargs ( dict or set ) – the arguments to verify

  • caller ( str ) – optional argument to help understand what’s missing

Returns

True/False if all keys are valid

Return type

bool

Raises

KeyError if key is missing from kwargs

ecs_composex.common. validate_resource_title ( resource_name , resource_type = None ) [source]

Function to validate the key for the resource is valid

Parameters
  • resource_name ( str ) – Name of the resource to evaluate

  • resource_type ( str ) – category of the resource, optional

Returns

True/False

Return type

bool