Designing your manifest¶

Purpose of the manifest file¶

The manifest file is there to describe what you want to provision and into which accounts you want to provision products into. It is possible to use AWS Organizations to make your manifest file more concise and easier to work with but the premise is the same - it is just a list of accounts and AWS Service Catalog products.

Sections of the manifest file¶

There are three sections to a manifest file - the global parameters, the accounts list and the launches. Each of the three are described in the following sections.

Parameters¶

It is possible to specify global parameters that should be used when provisioning your AWS Service Catalog Products. You can set the value to an explicit value or you can set the value to the result of a function call - using funcation calls to set parameter values is known as using a macro.

Here is an example of a simple global parameter:

schema: puppet-2019-04-01

parameters:
    CloudTrailLoggingBucketName:
      default: cloudtrail-logs-for-aws

It is possible to also specify a parameter at the account level:

accounts:
  - account_id: '<YOUR_ACCOUNT_ID>'
    name: '<YOUR_ACCOUNT_NAME>'
    default_region: eu-west-1
    regions_enabled:
      - eu-west-1
      - eu-west-1
    tags:
      - type:prod
      - partition:eu
      - scope:pci
    parameters:
      RoleName:
        default: DevAdmin
      Path:
        default: /human-roles/

And finally you specify parameters at the launch level:

launches:
  account-iam-for-prod:
    portfolio: demo-central-it-team-portfolio
    product: account-iam
    version: v1
    parameters:
      RoleName:
        default: DevAdmin
      Path:
        default: /human-roles/
    deploy_to:
      tags:
        - tag: type:prod
          regions: default_region

Whenever Puppet provisions a product it checks the parameters for the product. If it sees the name match one of the parameter values it will use it. In order to avoid clashes with parameter names we recommend using descriptive names like in the example - using the parameter names like BucketName will lead you into trouble pretty quickly.

The order of precedence for parameters is account level parameters override all others and launch level parameters override global.

Retrieving AWS SSM Parameters¶

Note

This was added in version 0.0.33

You can retrieve parameter values from SSM in the puppet account - these parameters do not belong to the spoke account. Here is an an example:

schema: puppet-2019-04-01

parameters:
    CentralLoggingBucketName:
      ssm:
        name: central-logging-bucket-name

You can get a different value for each region:

schema: puppet-2019-04-01

parameters:
    CentralLoggingBucketName:
      ssm:
        name: central-logging-bucket-name
        region: eu-west-1

Note

Since 0.94.0 you can use intrinsic functions within the parameter name

You can use intrinsic functions for AWS AccountId and Region within the ssm parameter name:

schema: puppet-2019-04-01

parameters:
    VPCCidrRange:
      ssm:
        name: /vpcs/${AWS::AccountId}/${AWS::Region}/cidr

${AWS::AccountId} and ${AWS::Region} will be replaced with the spoke account id and the region where the provisioning will occur. This allows you to build out account specific parameters using SSM parameter store parameters in the hub account.

You can use your account vending machine to provision the account specific ssm parameters in the hub account and then use this to read them.

Setting AWS SSM Parameters¶

Note

This was added in version 0.0.34

You can set the value of an SSM Parameter to the output of a CloudFormation stack output:

account-iam-sysops:
  portfolio: demo-central-it-team-portfolio
  product: account-iam
  version: v1
  parameters:
    Path:
      default: /human-roles/
    RoleName:
      default: SysOps
  deploy_to:
    tags:
    - regions: default_region
      tag: type:prod
  outputs:
    ssm:
      -  param_name: account-iam-sysops-role-arn
         stack_output: RoleArn

The example above will provision the product account-iam into an account. Once the stack has been completed it will get the value of the output named RoleArn of the CloudFormation stack and insert it into SSM within the default region using a parameter name of account-iam-sysops-role-arn

You can also set override which region the output is read from and which region the SSM parameter is written to:

account-iam-sysops:
  portfolio: demo-central-it-team-portfolio
  product: account-iam
  version: v1
  parameters:
    Path:
      default: /human-roles/
    RoleName:
      default: SysOps
  deploy_to:
    tags:
    - regions: default_region
      tag: type:prod
  outputs:
    ssm:
      -  param_name: account-iam-sysops-role-arn
         stack_output: RoleArn
         region: us-east-1

Note

There is currently no capability of reading a value from a CloudFormation stack from one region and setting an SSM param in another.

Macros¶

You can also use a macro to set the value of a parameter. It works in the same way as a normal parameter except it executes a function to get the value first. Here is an an example:

schema: puppet-2019-04-01

parameters:
    AllAccountIds:
      macro:
        method: get_accounts_for_path
        args: /

At the moment there are the following macros supported:

+------------------------+------------------------------+----------------------------------------------+
| macro method name      | args                         | description                                  |
+========================+==============================+==============================================+
| get_accounts_for_path  | ou path to get accounts for  | Returns a comma seperated list of account ids|
+------------------------+------------------------------+----------------------------------------------+

Mappings¶

Note

This was added in version 0.92.0

Within the mappings section you can define mappings that can be used to set the value of parameters.

Here a mapping is defined:

schema: puppet-2019-04-01

mappings:
  WebProxyDevice:
    us-east-1:
      "ami": "ami-15f77f867"
    us-west-1:
      "ami": "ami-0bdb82235"
    eu-west-1:
      "ami": "ami-16506cd98"

Here a mapping is used:

schema: puppet-2019-04-01

launches:
  DeployProxy:
    deploy_to:
      tags:
      - regions: enabled_regions
        tag: ou:prod
    parameters:
      AMI:
        mapping:
        - WebProxyDevice
        - AWS::Region
        - ami
    portfolio: networking
    product: proxy
    version: v3

When DeployProxy is provisioned the parameter named AMI will have its value determined. It’s value will be taken from the mappings section. The framework will look through the mappings for one named WebProxyDevice. The framework will then look within the WebProxyDevice dictionary for an object with the name of the current region and then within that for an item with the key ami.

It is also possible to use AWS::AccountId:

schema: puppet-2019-04-01

mappings:
  AccountDetails:
    0123456789010:
      "owner": "appteam1@example.com"

You can also combine them:

schema: puppet-2019-04-01

mappings:
  Networking:
    0123456789010:
      eu-west-1:
        cidr: "10.0.0.0/24"

You can use the special value of default as a catch all when you do you not specify a value:

schema: puppet-2019-04-01

mappings:
  AccountDetails:
    owners:
        0123456789010: "appteam1@example.com"
        0098765432102: "appteam2@example.com"
        default: "cloudteam@example.com"

Accounts¶

With the accounts section, you can describe your AWS accounts. You can set a default region, the enabled regions and you can tag your accounts. This metadata describing your account is used to determine which packages get deployed into your accounts.

Setting a default region¶

Within your account you may have a _home_ or a default region. This may be the closest region to the team using the account. You use default_region when describing your account and then you can use default_region again as a target when you specify your product launches - the product will be provisioned into the region specified.

Here is an example with a default_region set to us-east-1:

schema: puppet-2019-04-01

accounts:
  - account_id: '<YOUR_ACCOUNT_ID>'
    name: '<YOUR_ACCOUNT_NAME>'
    default_region: us-east-1
    regions_enabled:
      - us-east-1
      - us-west-2
    tags:
      - type:prod
      - partition:us
      - scope:pci

Note

Please note default_region can only be a string - not a list.

Setting enabled regions¶

You may chose not to use every region within your AWS Account. When describing an AWS account you can specify which regions are enabled for an account using regions_enabled.

Here is an example with regions_enabled set to us-east-1 and us-west-2:

schema: puppet-2019-04-01

accounts:
  - account_id: '<YOUR_ACCOUNT_ID>'
    name: '<YOUR_ACCOUNT_NAME>'
    default_region: us-east-1
    regions_enabled:
      - us-east-1
      - us-west-2
    tags:
      - type:prod
      - partition:us
      - scope:pci

Note

Please note regions_enabled can only be a list of strings - not a single string

Setting tags¶

You can describe your account using tags. Tags are specified using a list of strings. We recommend using namespaces for your tags, adding an extra dimension to them. If you choose to do this you can use a colon to split name and values.

Here is an example with namespaced tags:

schema: puppet-2019-04-01

accounts:
  - account_id: '<YOUR_ACCOUNT_ID>'
    name: '<YOUR_ACCOUNT_NAME>'
    default_region: us-east-1
    regions_enabled:
      - us-east-1
      - us-west-2
    tags:
      - type:prod
      - partition:us
      - scope:pci

In this example there the following tags: - namespace of type and value of prod - namespace of partition and value of us - namespace of scope and value of pci.

The goal of tags is to provide a classification for your accounts that can be used to a deployment time.

Using an OU id or path (integration with AWS Organizations)¶

Note

This was added in version 0.0.18

When specifying an account you can use short hand notation of ou instead of account_id to build out a list of accounts with the same properties.

For example you can use an AWS Organizations path:

schema: puppet-2019-04-01

accounts:
  - ou: /prod
    name: '<CHOOSE A NAME FOR YOUR ACCOUNTS LIST>'
    default_region: us-east-1
    regions_enabled:
      - us-east-1
      - us-west-2
    tags:
      - type:prod
      - partition:us
      - scope:pci

The framework will get a list of all AWS accounts within the /prod Organizational unit and expand your manifest to look like the following (assuming accounts 0123456789010 and 0109876543210 are the only accountss within /prod):

schema: puppet-2019-04-01

accounts:
  - account_id: 0123456789010
    name: '<YOUR_ACCOUNT_NAME>'
    default_region: us-east-1
    regions_enabled:
      - us-east-1
      - us-west-2
    tags:
      - type:prod
      - partition:us
      - scope:pci
  - account_id: 0109876543210
    name: '<YOUR_ACCOUNT_NAME>'
    default_region: us-east-1
    regions_enabled:
      - us-east-1
      - us-west-2
    tags:
      - type:prod
      - partition:us
      - scope:pci

Launches¶

Launches allow you to decide which products get provisioned into each account. You link product launches to accounts using tags or explicit account ids and you can set which regions the products are launched into.

Timeouts¶

Note

This was added in version 0.1.14

If you are worried that a launch may fail and take a long time to fail you can set a timeout timeoutInSeconds:

schema: puppet-2019-04-01

launches:
  account-iam-for-prod:
    portfolio: example-simple-central-it-team-portfolio
    product: account-iam
    timeoutInSeconds: 10
    version: v1
    deploy_to:
      tags:
        - tag: type:prod
          regions: default_region

Tag based launches¶

You can specify a launch to occur using tags in the deploy_to section of a launch.

Here is an example, it deploys a v1 of a product named account-iam from the portfolio example-simple-central-it-team-portfolio into into the default_region of all accounts tagged type:prod:

schema: puppet-2019-04-01

launches:
  account-iam-for-prod:
    portfolio: example-simple-central-it-team-portfolio
    product: account-iam
    version: v1
    deploy_to:
      tags:
        - tag: type:prod
          regions: default_region

When you specify more than one tag entry in deploy_to->tags the framework will interpret this as an or so the following snippet will provision v1 of account-iam to all accounts tagged type:prod or type:dev:

schema: puppet-2019-04-01

launches:
  account-iam-for-prod:
    portfolio: example-simple-central-it-team-portfolio
    product: account-iam
    version: v1
    deploy_to:
      tags:
        - tag: type:prod
          regions: default_region
        - tag: type:dev
          regions: default_region

Account based launches¶

You can also specify a launch to occur explicity in an account by using the accounts section in the deploy_to section of a launch.

Here is an example, it deploys a v1 of a product named account-iam from the portfolio example-simple-central-it-team-portfolio into into the default_region of the accounts 0123456789010:

schema: puppet-2019-04-01

launches:
  account-iam-for-prod:
    portfolio: example-simple-central-it-team-portfolio
    product: account-iam
    version: v1
    deploy_to:
      accounts:
        - account_id: '0123456789010'
          regions: default_region

Choosing which regions to provision into¶

When writing your launches you can choose which regions you provision into.

The valid values for regions are:

enabled - this will deploy to each enabled region for the account
regions_enabled - this will deploy to each enabled region for the account
default_region - this will deploy to the default region specified for the account
all - this will deploy to all regions enabled in your config (whilst setting up Puppet)
list of AWS regions - you can type in a list of AWS regions (each region selected should be present in your config)

Dependencies between launches¶

Where possible we recommend building launches to be independent. However, there are cases where you may need to setup a hub account before setting up a spoke or there may be times you are using AWS Lambda to back AWS CloudFormation custom resources. In these examples it would be beneficial to be able to say deploy launch x and then launch y. To achieve this You can use depends_on within your launch like so:

launches:
  account-vending-account-creation:
    portfolio: demo-central-it-team-portfolio
    product: account-vending-account-creation
    version: v1
    depends_on:
      - account-vending-account-bootstrap-shared
      - account-vending-account-creation-shared
    deploy_to:
      tags:
        - tag: scope:puppet-hub
          regions: default_region

  account-vending-account-bootstrap-shared:
    portfolio: demo-central-it-team-portfolio
    product: account-vending-account-bootstrap-shared
    version: v1
    deploy_to:
      tags:
        - tag: scope:puppet-hub
          regions: default_region

  account-vending-account-creation-shared:
    portfolio: demo-central-it-team-portfolio
    product: account-vending-account-creation-shared
    version: v1
    deploy_to:
      tags:
        - tag: scope:puppet-hub
          regions: default_region

In this example the framework will deploy account-vending-account-creation only when account-vending-account-bootstrap-shared and account-vending-account-creation-shared have been attempted.

Termination of products¶

Note

This was added in version 0.1.11

To terminate the provisioned product from a spoke account (which will delete the resources deployed) you can change the status of the launch using the status keyword:

launches:
  account-vending-account-creation:
    portfolio: demo-central-it-team-portfolio
    product: account-vending-account-creation
    version: v1
    status: terminated
    deploy_to:
      tags:
        - tag: scope:puppet-hub
          regions: default_region

When you mark a launch as terminated and run your pipeline the resources will be deleted and you can then remove the launch from your manifest. Leaving it in will not cause any errors but will result in your pipeline running time to be longer than it needs to be.

Please note, when mark your launch as terminated it cannot have dependencies, parameters or outputs. Leaving these in will cause the termination action to fail.

Note

When you set status to terminated you must remove your depends_on and parameters for it to work.

Warning

Since 0.1.16, terminating a product will also remove any SSM Parameters you created for it via the manifest.yaml

Managing large manifests or working in teams (multiple manifest files)¶

Note

This was added in version 0.71.0

If you have a large manifest file or are working in a team you may find it difficult managing changes occurring to your manifest file. You may find yourself having a lot of merge conflicts. To resolve this you can split your manifest file into smaller pieces. You can specify launches in a launch directory within your ServiceCatalogPuppet repository:

✗ tree ServiceCatalogPuppet
ServiceCatalogPuppet
├── launches
│   └── launches-for-team-a.yaml
├── manifest.yaml

The file (in this example launches-for-team-a.yaml) should be a list of launches:

✗ cat launches-for-team-a.yaml
account-vending-account-creation:
    portfolio: demo-central-it-team-portfolio
    product: account-vending-account-creation
    version: v1
    depends_on:
      - account-vending-account-bootstrap-shared
      - account-vending-account-creation-shared
    deploy_to:
      tags:
        - tag: scope:puppet-hub
          regions: default_region

account-vending-account-bootstrap-shared:
    portfolio: demo-central-it-team-portfolio
    product: account-vending-account-bootstrap-shared
    version: v1
    deploy_to:
      tags:
        - tag: scope:puppet-hub
          regions: default_region

The framework will load the manifest.yaml and overwrite any launches with ones defined in files from the launches directory. The framework will not warn you of any overrides.

You can also specify parameters and spoke-local-portfolios in directories too. When doing so, the files should contain lists of parameters or spoke-local-portfolios and should not be a dictionary.

✗ tree ServiceCatalogPuppet
ServiceCatalogPuppet
├── parameters
│   └── parameters-for-team-a.yaml
├── spoke-local-portfolios
│   └── spoke-local-portfolios-for-team-a.yaml
├── manifest.yaml

The names of the file within the launches, parameters and spoke-local-portfolios are ignored.

You can also declare other manifest files in a manifests directory:

✗ tree ServiceCatalogPuppet
ServiceCatalogPuppet
├── manifests
│   └── manifest-for-team-a.yaml
│   └── manifest-for-networking.yaml
│   └── manifest-for-governance.yaml

When you write a manifest file in the manifests directory the accounts section is ignored - you can only specify launches, parameters and spoke-local-portfolios.

Managing large manifests or working across multiple environments (external versions / properties files)¶

Note

This was added in version 0.76.0

If you are using puppet to manage multiple environments you may find it easier to keep the versions of your launches in properties files instead of the manifest.yaml files. To do this you create a file named manifest.properties in the same directory as your manifest.yaml file. Within this file you can specify the following:

[launches]
IAM-1.version = v50

This will set the version for the launch with the name IAM-1 to v50.

Please note this will overwrite the values specified in the manifest.yaml files with no warning.

If you are using multiple instances of puppet you can also create a file named manifest-<puppet-account-id>.properties. Values in this file will overwrite all other values making the order of reading:

manifest.yaml
files in manifests/
manifest.properties
manifest-<puppet-account-id>.properties

Lambda Invocations¶

Note

This was added in version 0.83.0

If you are migrating to puppet from your own AWS Lambda and AWS Step Functions solution you may want to reuse some of your Lambda functions to orchestrate activities like the removal of default VPCs or other actions in your accounts where using AWS Service Catalog + AWS CloudFormation may be cumbersome. To do this you can use lambda-invocation in your manifest file:

lambda-invocations:
  remove-default-vpc:
    function_name: remove-default-vpc
    qualifier: $LATEST
    invocation_type: Event
    invoke_for:
      tags:
        - regions: enabled_regions
          tag: scope:all

The above example will build a list by walking through each enabled_region for all accounts tagged scope:all. It will then invoke the $LATEST version of the remove-default-vpc in your puppet account for each item in the list, setting the parameters in the event object of the designated lambda to include account_id and region properties so you can implement whatever you want.

lambda-invocations:
  remove-default-vpc:
    function_name: remove-default-vpc
    qualifier: $LATEST
    invocation_type: Event
    depends_on:
      - name: remove-default-vpc-lambda
        type: launch
    invoke_for:
      tags:
        - regions: enabled_regions
          tag: scope:all

The lambda-invocations section includes support for depends_on where you can depend on another lambda-invocations or a launch. Using the depends_on you can provision the AWS Lambda function before executing using puppet as your complete solution for configuration.

The properties for function_name, qualifier and invocation_type are passed as is to the AWS Boto3 Lambda invoke function.

You can use parameters as you can for launches:

lambda-invocations:
  remove-default-vpc:
    function_name: remove-default-vpc
    qualifier: $LATEST
    invocation_type: Event
    parameters:
      RoleName:
        default: DevAdmin
      CentralLoggingBucketName:
        ssm:
          name: central-logging-bucket-name
          region: eu-west-1
    depends_on:
      - name: remove-default-vpc-lambda
        type: launch
    invoke_for:
      tags:
        - regions: enabled_regions
          tag: scope:all

If you set the invocation_type to Event puppet will not check if the Lambda function completed successfully. If you set the invocation_type to RequestResponse then it will wait for completion and error should the function not exit successfully.

Designing your manifest¶

Purpose of the manifest file¶

Sections of the manifest file¶

Parameters¶

Retrieving AWS SSM Parameters¶

Setting AWS SSM Parameters¶

Macros¶

Mappings¶

Accounts¶

Setting a default region¶

Setting enabled regions¶

Setting tags¶

Using an OU id or path (integration with AWS Organizations)¶

Launches¶

Timeouts¶

Tag based launches¶

Account based launches¶

Choosing which regions to provision into¶

Dependencies between launches¶

Termination of products¶

Managing large manifests or working in teams (multiple manifest files)¶

Managing large manifests or working across multiple environments (external versions / properties files)¶

Lambda Invocations¶

Table of Contents

Previous topic

Next topic

This Page

Designing your manifest¶

Purpose of the manifest file¶

Sections of the manifest file¶

Parameters¶

Retrieving AWS SSM Parameters¶

Setting AWS SSM Parameters¶

Macros¶

Mappings¶

Accounts¶

Setting a default region¶

Setting enabled regions¶

Setting tags¶

Using an OU id or path (integration with AWS Organizations)¶

Launches¶

Timeouts¶

Tag based launches¶

Account based launches¶

Choosing which regions to provision into¶

Dependencies between launches¶

Termination of products¶

Managing large manifests or working in teams (multiple manifest files)¶

Managing large manifests or working across multiple environments (external versions / properties files)¶

Sharing mode¶

Lambda Invocations¶