copilot env deploy fails after adding environment to existing project

[Epiphero]

I'm trying to add a demo environment to an existing project.

To do so, I used copilot env init --name demo followed by copilot env deploy --name demo, but I'm presented with:

    No export named atom-project-demo-VpcId found. Rollback reque
    sted by user.

Re-running the deployment with the --diff flag, I get a different error:

No export named atom-project-demo-PrivateSubnets found. Rollback reque
    sted by user.

Intrigued by the difference, I run again with the --diff flag and get:

No export named atom-project-demo-EnvironmentSecurityGroup found. Roll
    back requested by user.

Continuing on with the retries, I get the same EnvironmentSecurityGroup error for two subsequent attempts, then back to PrivateSubnets. Here is the stack that it's trying to deploy (with a few redactions for things that are trademarked or look sensitive):

copilot env deploy --name demo --diff
+ Conditions:
+     CreateALB: !Not [!Equals [!Ref ALBWorkloads, ""]]
+     CreateInternalALB: !Not [!Equals [!Ref InternalALBWorkloads, ""]]
+     DelegateDNS: !Not [!Equals [!Ref AppDNSName, ""]]
+     ExportHTTPSListener: !And
+         - !Condition CreateALB
+         - !Equals [!Ref CreateHTTPSListener, true]
+     ExportInternalHTTPSListener: !And
+         - !Condition CreateInternalALB
+         - !Equals [!Ref CreateInternalHTTPSListener, true]
+     CreateEFS: !Not [!Equals [!Ref EFSWorkloads, ""]]
+     CreateNATGateways: !Not [!Equals [!Ref NATWorkloads, ""]]
+     CreateAppRunnerVPCEndpoint: !Not [!Equals [!Ref AppRunnerPrivateWorkloads, ""]]
+     ManagedAliases: !And
+         - !Condition DelegateDNS
+         - !Not [!Equals [!Ref Aliases, ""]]
~ Description: CloudFormation environment bootstrap template with the necessary roles to create envs and workloads. -> CloudFormation environment template for infrastructure shared among Copilot workloads.
~ Metadata:
    ~ Version: bootstrap -> v1.13.0
~ Outputs:
    + AppRunnerVpcEndpointId:
    +     Condition: CreateAppRunnerVPCEndpoint
    +     Value: !Ref AppRunnerVpcEndpoint
    +     Description: VPC Endpoint to App Runner for private services
    +     Export:
    +         Name: !Sub ${AWS::StackName}-AppRunnerVpcEndpointId
    + ClusterId:
    +     Value: !Ref Cluster
    +     Export:
    +         Name: !Sub ${AWS::StackName}-ClusterId
    + DefaultHTTPTargetGroupArn:
    +     Condition: CreateALB
    +     Value: !Ref DefaultHTTPTargetGroup
    +     Export:
    +         Name: !Sub ${AWS::StackName}-DefaultHTTPTargetGroup
    + EnabledFeatures:
    +     Value: !Sub '${ALBWorkloads},${InternalALBWorkloads},${EFSWorkloads},${NATWorkloads},${Aliases},${AppRunnerPrivateWorkloads}'
    +     Description: Required output to force the stack to update if mutating feature params, like ALBWorkloads, does not change the template.
    + EnvironmentSecurityGroup:
    +     Value: !Ref EnvironmentSecurityGroup
    +     Export:
    +         Name: !Sub ${AWS::StackName}-EnvironmentSecurityGroup
    + HTTPListenerArn:
    +     Condition: CreateALB
    +     Value: !Ref HTTPListener
    +     Export:
    +         Name: !Sub ${AWS::StackName}-HTTPListenerArn
    + HTTPSListenerArn:
    +     Condition: ExportHTTPSListener
    +     Value: !Ref HTTPSListener
    +     Export:
    +         Name: !Sub ${AWS::StackName}-HTTPSListenerArn
    + InternalHTTPListenerArn:
    +     Condition: CreateInternalALB
    +     Value: !Ref InternalHTTPListener
    +     Export:
    +         Name: !Sub ${AWS::StackName}-InternalHTTPListenerArn
    + InternalHTTPSListenerArn:
    +     Condition: ExportInternalHTTPSListener
    +     Value: !Ref InternalHTTPSListener
    +     Export:
    +         Name: !Sub ${AWS::StackName}-InternalHTTPSListenerArn
    + InternalLoadBalancerDNSName:
    +     Condition: CreateInternalALB
    +     Value: !GetAtt InternalLoadBalancer.DNSName
    +     Export:
    +         Name: !Sub ${AWS::StackName}-InternalLoadBalancerDNS
    + InternalLoadBalancerFullName:
    +     Condition: CreateInternalALB
    +     Value: !GetAtt InternalLoadBalancer.LoadBalancerFullName
    +     Export:
    +         Name: !Sub ${AWS::StackName}-InternalLoadBalancerFullName
    + InternalLoadBalancerHostedZone:
    +     Condition: CreateInternalALB
    +     Value: !GetAtt InternalLoadBalancer.CanonicalHostedZoneID
    +     Export:
    +         Name: !Sub ${AWS::StackName}-InternalLoadBalancerCanonicalHostedZoneID
    + InternalLoadBalancerSecurityGroup:
    +     Condition: CreateInternalALB
    +     Value: !Ref InternalLoadBalancerSecurityGroup
    +     Export:
    +         Name: !Sub ${AWS::StackName}-InternalLoadBalancerSecurityGroup
    + InternalWorkloadsHostedZone:
    +     Condition: CreateInternalALB
    +     Value: !Ref InternalWorkloadsHostedZone
    +     Export:
    +         Name: !Sub ${AWS::StackName}-InternalWorkloadsHostedZoneID
    + InternalWorkloadsHostedZoneName:
    +     Condition: CreateInternalALB
    +     Value: !Sub ${EnvironmentName}.${AppName}.internal
    +     Export:
    +         Name: !Sub ${AWS::StackName}-InternalWorkloadsHostedZoneName
    + InternetGatewayID:
    +     Value: !Ref InternetGateway
    +     Export:
    +         Name: !Sub ${AWS::StackName}-InternetGatewayID
    + LastForceDeployID:
    +     Value: ""
    +     Description: Optionally force the template to update when no immediate resource change is present.
    + ManagedFileSystemID:
    +     Condition: CreateEFS
    +     Value: !Ref FileSystem
    +     Description: The ID of the Copilot-managed EFS filesystem.
    +     Export:
    +         Name: !Sub ${AWS::StackName}-FilesystemID
    + PrivateRouteTableIDs:
    +     Condition: CreateNATGateways
    +     Value: !Join [',', [!Ref PrivateRouteTable1, !Ref PrivateRouteTable2]]
    +     Export:
    +         Name: !Sub ${AWS::StackName}-PrivateRouteTableIDs
    + PrivateSubnets:
    +     Value: !Join [',', [!Ref PrivateSubnet1, !Ref PrivateSubnet2]]
    +     Export:
    +         Name: !Sub ${AWS::StackName}-PrivateSubnets
    + PublicALBAccessible:
    +     Condition: CreateALB
    +     Value: true
    + PublicLoadBalancerDNSName:
    +     Condition: CreateALB
    +     Value: !GetAtt PublicLoadBalancer.DNSName
    +     Export:
    +         Name: !Sub ${AWS::StackName}-PublicLoadBalancerDNS
    + PublicLoadBalancerFullName:
    +     Condition: CreateALB
    +     Value: !GetAtt PublicLoadBalancer.LoadBalancerFullName
    +     Export:
    +         Name: !Sub ${AWS::StackName}-PublicLoadBalancerFullName
    + PublicLoadBalancerHostedZone:
    +     Condition: CreateALB
    +     Value: !GetAtt PublicLoadBalancer.CanonicalHostedZoneID
    +     Export:
    +         Name: !Sub ${AWS::StackName}-CanonicalHostedZoneID
    + PublicRouteTableID:
    +     Value: !Ref PublicRouteTable
    +     Export:
    +         Name: !Sub ${AWS::StackName}-PublicRouteTableID
    + PublicSubnets:
    +     Value: !Join [',', [!Ref PublicSubnet1, !Ref PublicSubnet2]]
    +     Export:
    +         Name: !Sub ${AWS::StackName}-PublicSubnets
    + ServiceDiscoveryNamespaceID:
    +     Value: !GetAtt ServiceDiscoveryNamespace.Id
    +     Export:
    +         Name: !Sub ${AWS::StackName}-ServiceDiscoveryNamespaceID
    + VpcId:
    +     Value: !Ref VPC
    +     Export:
    +         Name: !Sub ${AWS::StackName}-VpcId
~ Parameters:
    + ALBWorkloads:
    +     Type: String
    + Aliases:
    +     Type: String
    + AppDNSDelegationRole:
    +     Type: String
    + AppDNSName:
    +     Type: String
    + AppRunnerPrivateWorkloads:
    +     Type: String
    + CreateHTTPSListener:
    +     Type: String
    +     AllowedValues: [true, false]
    + CreateInternalHTTPSListener:
    +     Type: String
    +     AllowedValues: [true, false]
    + EFSWorkloads:
    +     Type: String
    + InternalALBWorkloads:
    +     Type: String
    + NATWorkloads:
    +     Type: String
    + ServiceDiscoveryEndpoint:
    +     Type: String
~ Resources:
    + AddonsStack:
    +     Metadata:
    +         'aws:copilot:description': 'A CloudFormation nested stack for your additional AWS resources'
    +     Type: AWS::CloudFormation::Stack
    +     Properties:
    +         Parameters:
    +             App: !Ref AppName
    +             Env: !Ref EnvironmentName
    +         TemplateURL: https://stackset-atom-project-in-pipelinebuiltartifactbuc-redacted.s3.amazonaws.com/manual/addons/environments/***********redacted**********.yml
    + AppRunnerVpcEndpoint:
    +     Metadata:
    +         'aws:copilot:description': 'VPC Endpoint to connect environment to App Runner for private services'
    +     Type: AWS::EC2::VPCEndpoint
    +     Condition: CreateAppRunnerVPCEndpoint
    +     Properties:
    +         VpcEndpointType: Interface
    +         VpcId: !Ref VPC
    +         SecurityGroupIds:
    +             - !Ref AppRunnerVpcEndpointSecurityGroup
    +         ServiceName: !Sub 'com.amazonaws.${AWS::Region}.apprunner.requests'
    +         SubnetIds:
    +             - !Ref PrivateSubnet1
    +             - !Ref PrivateSubnet2
    + AppRunnerVpcEndpointSecurityGroup:
    +     Metadata:
    +         'aws:copilot:description': 'A security group for App Runner private services'
    +     Type: AWS::EC2::SecurityGroup
    +     Condition: CreateAppRunnerVPCEndpoint
    +     Properties:
    +         GroupDescription: atom-project-demo-AppRunnerVpcEndpointSecurityGroup
    +         VpcId: !Ref VPC
    +         Tags:
    +             - Key: Name
    +               Value: copilot-atom-project-demo-app-runner-vpc-endpoint
    + AppRunnerVpcEndpointSecurityGroupIngressFromEnvironment:
    +     Type: AWS::EC2::SecurityGroupIngress
    +     Condition: CreateAppRunnerVPCEndpoint
    +     Properties:
    +         Description: Ingress from services in the environment
    +         GroupId: !Ref AppRunnerVpcEndpointSecurityGroup
    +         IpProtocol: -1
    +         SourceSecurityGroupId: !Ref EnvironmentSecurityGroup
    + Cluster:
    +     Metadata:
    +         'aws:copilot:description': 'An ECS cluster to group your services'
    +     Type: AWS::ECS::Cluster
    +     Properties:
    +         CapacityProviders: ['FARGATE', 'FARGATE_SPOT']
    +         Configuration:
    +             ExecuteCommandConfiguration:
    +                 Logging: DEFAULT
    +         ClusterSettings:
    +             - Name: containerInsights
    +               Value: disabled
    + DefaultHTTPTargetGroup:
    +     Type: AWS::ElasticLoadBalancingV2::TargetGroup
    +     Condition: CreateALB
    +     Properties:
    +         #  Check if your application is healthy within 20 = 10*2 seconds, compared to 2.5 mins = 30*5 seconds.
    +         HealthCheckIntervalSeconds: 10 # Default is 30.
    +         HealthyThresholdCount: 2 # Default is 5.
    +         HealthCheckTimeoutSeconds: 5
    +         Port: 80
    +         Protocol: HTTP
    +         TargetGroupAttributes:
    +             - Key: deregistration_delay.timeout_seconds
    +               Value: 60 # Default is 300.
    +         TargetType: ip
    +         VpcId: !Ref VPC
    + DefaultInternalHTTPTargetGroup:
    +     Type: AWS::ElasticLoadBalancingV2::TargetGroup
    +     Condition: CreateInternalALB
    +     Properties:
    +         #  Check if your application is healthy within 20 = 10*2 seconds, compared to 2.5 mins = 30*5 seconds.
    +         HealthCheckIntervalSeconds: 10 # Default is 30.
    +         HealthyThresholdCount: 2 # Default is 5.
    +         HealthCheckTimeoutSeconds: 5
    +         Port: 80
    +         Protocol: HTTP
    +         TargetGroupAttributes:
    +             - Key: deregistration_delay.timeout_seconds
    +               Value: 60 # Default is 300.
    +         TargetType: ip
    +         VpcId: !Ref VPC
    + DefaultPublicRoute:
    +     Type: AWS::EC2::Route
    +     DependsOn: InternetGatewayAttachment
    +     Properties:
    +         RouteTableId: !Ref PublicRouteTable
    +         DestinationCidrBlock: 0.0.0.0/0
    +         GatewayId: !Ref InternetGateway
    + EFSSecurityGroup:
    +     Metadata:
    +         'aws:copilot:description': 'A security group to allow your containers to talk to EFS storage'
    +     Type: AWS::EC2::SecurityGroup
    +     Condition: CreateEFS
    +     Properties:
    +         GroupDescription: !Join ['', [!Ref AppName, '-', !Ref EnvironmentName, EFSSecurityGroup]]
    +         VpcId: !Ref VPC
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}-efs'
    + EFSSecurityGroupIngressFromEnvironment:
    +     Type: AWS::EC2::SecurityGroupIngress
    +     Condition: CreateEFS
    +     Properties:
    +         Description: Ingress from containers in the Environment Security Group.
    +         GroupId: !Ref EFSSecurityGroup
    +         IpProtocol: -1
    +         SourceSecurityGroupId: !Ref EnvironmentSecurityGroup
    + EnvironmentHTTPSSecurityGroupIngressFromPublicALB:
    +     Type: AWS::EC2::SecurityGroupIngress
    +     Condition: ExportHTTPSListener
    +     Properties:
    +         Description: HTTPS ingress from the public ALB
    +         GroupId: !Ref EnvironmentSecurityGroup
    +         IpProtocol: -1
    +         SourceSecurityGroupId: !Ref PublicHTTPSLoadBalancerSecurityGroup
    + EnvironmentHTTPSecurityGroupIngressFromPublicALB:
    +     Type: AWS::EC2::SecurityGroupIngress
    +     Condition: CreateALB
    +     Properties:
    +         Description: HTTP ingress from the public ALB
    +         GroupId: !Ref EnvironmentSecurityGroup
    +         IpProtocol: -1
    +         SourceSecurityGroupId: !Ref PublicHTTPLoadBalancerSecurityGroup
    ~ EnvironmentManagerRole:
        ~ Properties:
            ~ Policies:
                ~ - (changed item)
                  ~ PolicyDocument:
                      ~ Statement:
                          + - Sid: ImportedCertificates
                          +   Effect: Allow
                          +   Action: ['acm:DescribeCertificate']
                          +   Resource:
                          +     - "arn:aws:acm:us-east-1:redacted:certificate/redacted"
                          (21 unchanged items)
    + EnvironmentSecurityGroup:
    +     Metadata:
    +         'aws:copilot:description': 'A security group to allow your containers to talk to each other'
    +     Type: AWS::EC2::SecurityGroup
    +     Properties:
    +         GroupDescription: !Join ['', [!Ref AppName, '-', !Ref EnvironmentName, EnvironmentSecurityGroup]]
    +         VpcId: !Ref VPC
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}-env'
    + EnvironmentSecurityGroupIngressFromInternalALB:
    +     Type: AWS::EC2::SecurityGroupIngress
    +     Condition: CreateInternalALB
    +     Properties:
    +         Description: Ingress from the internal ALB
    +         GroupId: !Ref EnvironmentSecurityGroup
    +         IpProtocol: -1
    +         SourceSecurityGroupId: !Ref InternalLoadBalancerSecurityGroup
    + EnvironmentSecurityGroupIngressFromSelf:
    +     Type: AWS::EC2::SecurityGroupIngress
    +     Properties:
    +         Description: Ingress from other containers in the same security group
    +         GroupId: !Ref EnvironmentSecurityGroup
    +         IpProtocol: -1
    +         SourceSecurityGroupId: !Ref EnvironmentSecurityGroup
    + FileSystem:
    +     Condition: CreateEFS
    +     Type: AWS::EFS::FileSystem
    +     Metadata:
    +         'aws:copilot:description': 'An EFS filesystem for persistent task storage'
    +     Properties:
    +         BackupPolicy:
    +             Status: ENABLED
    +         Encrypted: true
    +         FileSystemPolicy:
    +             Version: '2012-10-17'
    +             Id: CopilotEFSPolicy
    +             Statement:
    +                 - Sid: AllowIAMFromTaggedRoles
    +                   Effect: Allow
    +                   Principal:
    +                     AWS: '*'
    +                   Action:
    +                     - elasticfilesystem:ClientWrite
    +                     - elasticfilesystem:ClientMount
    +                   Condition:
    +                     Bool:
    +                         'elasticfilesystem:AccessedViaMountTarget': true
    +                     StringEquals:
    +                         'iam:ResourceTag/copilot-application': !Sub '${AppName}'
    +                         'iam:ResourceTag/copilot-environment': !Sub '${EnvironmentName}'
    +                 - Sid: DenyUnencryptedAccess
    +                   Effect: Deny
    +                   Principal: '*'
    +                   Action: 'elasticfilesystem:*'
    +                   Condition:
    +                     Bool:
    +                         'aws:SecureTransport': false
    +         LifecyclePolicies:
    +             - TransitionToIA: AFTER_30_DAYS
    +         PerformanceMode: generalPurpose
    +         ThroughputMode: bursting
    + HTTPListener:
    +     Metadata:
    +         'aws:copilot:description': 'A load balancer listener to route HTTP traffic'
    +     Type: AWS::ElasticLoadBalancingV2::Listener
    +     Condition: CreateALB
    +     Properties:
    +         DefaultActions:
    +             - TargetGroupArn: !Ref DefaultHTTPTargetGroup
    +               Type: forward
    +         LoadBalancerArn: !Ref PublicLoadBalancer
    +         Port: 80
    +         Protocol: HTTP
    + HTTPSListener:
    +     Metadata:
    +         'aws:copilot:description': 'A load balancer listener to route HTTPS traffic'
    +     Type: AWS::ElasticLoadBalancingV2::Listener
    +     Condition: ExportHTTPSListener
    +     Properties:
    +         Certificates:
    +             - CertificateArn: arn:aws:acm:us-east-1:redacted:certificate/redacted
    +         DefaultActions:
    +             - TargetGroupArn: !Ref DefaultHTTPTargetGroup
    +               Type: forward
    +         LoadBalancerArn: !Ref PublicLoadBalancer
    +         Port: 443
    +         Protocol: HTTPS
    + InternalALBIngressFromEnvironmentSecurityGroup:
    +     Type: AWS::EC2::SecurityGroupIngress
    +     Condition: CreateInternalALB
    +     Properties:
    +         Description: Ingress from the env security group
    +         GroupId: !Ref InternalLoadBalancerSecurityGroup
    +         IpProtocol: -1
    +         SourceSecurityGroupId: !Ref EnvironmentSecurityGroup
    + InternalHTTPListener:
    +     Metadata:
    +         'aws:copilot:description': 'An internal load balancer listener to route HTTP traffic'
    +     Type: AWS::ElasticLoadBalancingV2::Listener
    +     Condition: CreateInternalALB
    +     Properties:
    +         DefaultActions:
    +             - TargetGroupArn: !Ref DefaultInternalHTTPTargetGroup
    +               Type: forward
    +         LoadBalancerArn: !Ref InternalLoadBalancer
    +         Port: 80
    +         Protocol: HTTP
    + InternalHTTPSListener:
    +     Metadata:
    +         'aws:copilot:description': 'An internal load balancer listener to route HTTPS traffic'
    +     Type: AWS::ElasticLoadBalancingV2::Listener
    +     Condition: ExportInternalHTTPSListener
    +     Properties:
    +         DefaultActions:
    +             - TargetGroupArn: !Ref DefaultInternalHTTPTargetGroup
    +               Type: forward
    +         LoadBalancerArn: !Ref InternalLoadBalancer
    +         Port: 443
    +         Protocol: HTTPS
    + InternalLoadBalancer:
    +     Metadata:
    +         'aws:copilot:description': 'An internal Application Load Balancer to distribute private traffic from within the VPC to your services'
    +     Condition: CreateInternalALB
    +     Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    +     Properties:
    +         Scheme: internal
    +         SecurityGroups: [!GetAtt InternalLoadBalancerSecurityGroup.GroupId]
    +         Subnets: [!Ref PrivateSubnet1, !Ref PrivateSubnet2]
    +         Type: application
    + InternalLoadBalancerSecurityGroup:
    +     Metadata:
    +         'aws:copilot:description': 'A security group for your internal load balancer allowing HTTP traffic from within the VPC'
    +     Condition: CreateInternalALB
    +     Type: AWS::EC2::SecurityGroup
    +     Properties:
    +         GroupDescription: Access to the internal load balancer
    +         VpcId: !Ref VPC
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}-internal-lb'
    + InternalWorkloadsHostedZone:
    +     Metadata:
    +         'aws:copilot:description': 'A hosted zone named demo.atom-project.internal for backends behind a private load balancer'
    +     Condition: CreateInternalALB
    +     Type: AWS::Route53::HostedZone
    +     Properties:
    +         Name: !Sub ${EnvironmentName}.${AppName}.internal
    +         VPCs:
    +             - VPCId: !Ref VPC
    +               VPCRegion: !Ref AWS::Region
    + InternetGateway:
    +     Metadata:
    +         'aws:copilot:description': 'An Internet Gateway to connect to the public internet'
    +     Type: AWS::EC2::InternetGateway
    +     Properties:
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}'
    + InternetGatewayAttachment:
    +     Type: AWS::EC2::VPCGatewayAttachment
    +     Properties:
    +         InternetGatewayId: !Ref InternetGateway
    +         VpcId: !Ref VPC
    + MountTarget1:
    +     Type: AWS::EFS::MountTarget
    +     Condition: CreateEFS
    +     Properties:
    +         FileSystemId: !Ref FileSystem
    +         SubnetId: !Ref PrivateSubnet1
    +         SecurityGroups:
    +             - !Ref EFSSecurityGroup
    + MountTarget2:
    +     Type: AWS::EFS::MountTarget
    +     Condition: CreateEFS
    +     Properties:
    +         FileSystemId: !Ref FileSystem
    +         SubnetId: !Ref PrivateSubnet2
    +         SecurityGroups:
    +             - !Ref EFSSecurityGroup
    + NatGateway1:
    +     Metadata:
    +         'aws:copilot:description': 'NAT Gateway 1 enabling workloads placed in private subnet 1 to reach the internet'
    +     Type: AWS::EC2::NatGateway
    +     Condition: CreateNATGateways
    +     Properties:
    +         AllocationId: !GetAtt NatGateway1Attachment.AllocationId
    +         SubnetId: !Ref PublicSubnet1
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}-0'
    + NatGateway1Attachment:
    +     Metadata:
    +         'aws:copilot:description': 'An Elastic IP for NAT Gateway 1'
    +     Type: AWS::EC2::EIP
    +     Condition: CreateNATGateways
    +     DependsOn: InternetGatewayAttachment
    +     Properties:
    +         Domain: vpc
    + NatGateway2:
    +     Metadata:
    +         'aws:copilot:description': 'NAT Gateway 2 enabling workloads placed in private subnet 2 to reach the internet'
    +     Type: AWS::EC2::NatGateway
    +     Condition: CreateNATGateways
    +     Properties:
    +         AllocationId: !GetAtt NatGateway2Attachment.AllocationId
    +         SubnetId: !Ref PublicSubnet2
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}-1'
    + NatGateway2Attachment:
    +     Metadata:
    +         'aws:copilot:description': 'An Elastic IP for NAT Gateway 2'
    +     Type: AWS::EC2::EIP
    +     Condition: CreateNATGateways
    +     DependsOn: InternetGatewayAttachment
    +     Properties:
    +         Domain: vpc
    + PrivateRoute1:
    +     Type: AWS::EC2::Route
    +     Condition: CreateNATGateways
    +     Properties:
    +         RouteTableId: !Ref PrivateRouteTable1
    +         DestinationCidrBlock: 0.0.0.0/0
    +         NatGatewayId: !Ref NatGateway1
    + PrivateRoute2:
    +     Type: AWS::EC2::Route
    +     Condition: CreateNATGateways
    +     Properties:
    +         RouteTableId: !Ref PrivateRouteTable2
    +         DestinationCidrBlock: 0.0.0.0/0
    +         NatGatewayId: !Ref NatGateway2
    + PrivateRouteTable1:
    +     Type: AWS::EC2::RouteTable
    +     Condition: CreateNATGateways
    +     Properties:
    +         VpcId: !Ref 'VPC'
    + PrivateRouteTable1Association:
    +     Type: AWS::EC2::SubnetRouteTableAssociation
    +     Condition: CreateNATGateways
    +     Properties:
    +         RouteTableId: !Ref PrivateRouteTable1
    +         SubnetId: !Ref PrivateSubnet1
    + PrivateRouteTable2:
    +     Type: AWS::EC2::RouteTable
    +     Condition: CreateNATGateways
    +     Properties:
    +         VpcId: !Ref 'VPC'
    + PrivateRouteTable2Association:
    +     Type: AWS::EC2::SubnetRouteTableAssociation
    +     Condition: CreateNATGateways
    +     Properties:
    +         RouteTableId: !Ref PrivateRouteTable2
    +         SubnetId: !Ref PrivateSubnet2
    + PrivateSubnet1:
    +     Metadata:
    +         'aws:copilot:description': 'Private subnet 1 for resources with no internet access'
    +     Type: AWS::EC2::Subnet
    +     Properties:
    +         CidrBlock: 10.0.2.0/24
    +         VpcId: !Ref VPC
    +         AvailabilityZone: !Select [0, !GetAZs '']
    +         MapPublicIpOnLaunch: false
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}-priv0'
    + PrivateSubnet2:
    +     Metadata:
    +         'aws:copilot:description': 'Private subnet 2 for resources with no internet access'
    +     Type: AWS::EC2::Subnet
    +     Properties:
    +         CidrBlock: 10.0.3.0/24
    +         VpcId: !Ref VPC
    +         AvailabilityZone: !Select [1, !GetAZs '']
    +         MapPublicIpOnLaunch: false
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}-priv1'
    + PublicHTTPLoadBalancerSecurityGroup:
    +     Metadata:
    +         'aws:copilot:description': 'A security group for your load balancer allowing HTTP traffic'
    +     Condition: CreateALB
    +     Type: AWS::EC2::SecurityGroup
    +     Properties:
    +         GroupDescription: HTTP access to the public facing load balancer
    +         SecurityGroupIngress:
    +             - CidrIp: 0.0.0.0/0
    +               Description: Allow from anyone on port 80
    +               FromPort: 80
    +               IpProtocol: tcp
    +               ToPort: 80
    +         VpcId: !Ref VPC
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}-lb-http'
    + PublicHTTPSLoadBalancerSecurityGroup:
    +     Metadata:
    +         'aws:copilot:description': 'A security group for your load balancer allowing HTTPS traffic'
    +     Condition: ExportHTTPSListener
    +     Type: AWS::EC2::SecurityGroup
    +     Properties:
    +         GroupDescription: HTTPS access to the public facing load balancer
    +         SecurityGroupIngress:
    +             - CidrIp: 0.0.0.0/0
    +               Description: Allow from anyone on port 443
    +               FromPort: 443
    +               IpProtocol: tcp
    +               ToPort: 443
    +         VpcId: !Ref VPC
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}-lb-https'
    + PublicLoadBalancer:
    +     Metadata:
    +         'aws:copilot:description': 'An Application Load Balancer to distribute public traffic to your services'
    +     Condition: CreateALB
    +     Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    +     Properties:
    +         Scheme: internet-facing
    +         SecurityGroups:
    +             - !GetAtt PublicHTTPLoadBalancerSecurityGroup.GroupId
    +             - !If [ExportHTTPSListener, !GetAtt PublicHTTPSLoadBalancerSecurityGroup.GroupId, !Ref "AWS::NoValue"]
    +         Subnets: [!Ref PublicSubnet1, !Ref PublicSubnet2]
    +         Type: application
    + PublicRouteTable:
    +     Metadata:
    +         'aws:copilot:description': "A custom route table that directs network traffic for the public subnets"
    +     Type: AWS::EC2::RouteTable
    +     Properties:
    +         VpcId: !Ref VPC
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}'
    + PublicSubnet1:
    +     Metadata:
    +         'aws:copilot:description': 'Public subnet 1 for resources that can access the internet'
    +     Type: AWS::EC2::Subnet
    +     Properties:
    +         CidrBlock: 10.0.0.0/24
    +         VpcId: !Ref VPC
    +         AvailabilityZone: !Select [0, !GetAZs '']
    +         MapPublicIpOnLaunch: true
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}-pub0'
    + PublicSubnet1RouteTableAssociation:
    +     Type: AWS::EC2::SubnetRouteTableAssociation
    +     Properties:
    +         RouteTableId: !Ref PublicRouteTable
    +         SubnetId: !Ref PublicSubnet1
    + PublicSubnet2:
    +     Metadata:
    +         'aws:copilot:description': 'Public subnet 2 for resources that can access the internet'
    +     Type: AWS::EC2::Subnet
    +     Properties:
    +         CidrBlock: 10.0.1.0/24
    +         VpcId: !Ref VPC
    +         AvailabilityZone: !Select [1, !GetAZs '']
    +         MapPublicIpOnLaunch: true
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}-pub1'
    + PublicSubnet2RouteTableAssociation:
    +     Type: AWS::EC2::SubnetRouteTableAssociation
    +     Properties:
    +         RouteTableId: !Ref PublicRouteTable
    +         SubnetId: !Ref PublicSubnet2
    + ServiceDiscoveryNamespace:
    +     Metadata:
    +         'aws:copilot:description': 'A private DNS namespace for discovering services within the environment'
    +     Type: AWS::ServiceDiscovery::PrivateDnsNamespace
    +     Properties:
    +         Name: !Ref ServiceDiscoveryEndpoint
    +         Vpc: !Ref VPC
    + VPC:
    +     Metadata:
    +         'aws:copilot:description': 'A Virtual Private Cloud to control networking of your AWS resources'
    +     Type: AWS::EC2::VPC
    +     Properties:
    +         CidrBlock: 10.0.0.0/16
    +         EnableDnsHostnames: true
    +         EnableDnsSupport: true
    +         InstanceTenancy: default
    +         Tags:
    +             - Key: Name
    +               Value: !Sub 'copilot-${AppName}-${EnvironmentName}'
Continue with the deployment? Yes
✔ Proposing infrastructure changes for the atom-project-demo environment.
- Creating the infrastructure for the atom-project-demo environment.                 [update rollback complete]  [65.8s]
  The following resource(s) failed to create: [InternetGateway, AddonsSt
  ack, VPC]. The following resource(s) failed to update: [EnvironmentMan
  agerRole].
  - A CloudFormation nested stack for your additional AWS resources                  [rollback complete]         [35.5s]
    No export named atom-project-demo-PrivateSubnets found. Rollback reque
    sted by user.
  - An ECS cluster to group your services                                            [delete complete]           [4.1s]
  - An IAM Role to describe resources in your environment                            [update complete]           [14.1s]
    Resource update cancelled
  - A security group to allow your containers to talk to each other                  [not started]
  - An Internet Gateway to connect to the public internet                            [delete complete]           [11.5s]
    Resource creation cancelled
  - Private subnet 1 for resources with no internet access                           [not started]
  - Private subnet 2 for resources with no internet access                           [not started]
  - A custom route table that directs network traffic for the public subnets         [not started]
  - Public subnet 1 for resources that can access the internet                       [not started]
  - Public subnet 2 for resources that can access the internet                       [not started]
  - A private DNS namespace for discovering services within the environment          [not started]
  - A Virtual Private Cloud to control networking of your AWS resources              [delete complete]           [7.4s]
    Resource creation cancelled
✘ deploy environment demo: stack atom-project-demo did not complete successfully and exited with status UPDATE_ROLLBACK_COMPLETE

The manifest file:

# The manifest for the "dev" environment.
# Read the full specification for the "Environment" type at:
#  https://aws.github.io/copilot-cli/docs/manifest/environment/

# Your environment name will be used in naming your resources like VPC, cluster, etc.
name: dev
type: Environment

# Import your own VPC and subnets or configure how they should be created.
# network:
#   vpc:
#     id:

# Configure the load balancers in your environment, once created.
http:
  public:
    certificates:
      - arn:aws:acm:us-east-1:redacted:certificate/redacted
#   private:

# Configure observability for your environment resources.
observability:
  container_insights: false

(Although it doesn't work with the default file with no certificate either.)

[dannyrandall]

Hey @Epiphero! Sorry you're running into this issue!🥺 Could you share your environment addons template as well? Feel free to redact any information like you did above.

[Epiphero]

Thanks, I hadn't thought to look in there. Here is is:

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.

Conditions:
  IsProduction: !Equals
    - !Ref Env
    - 'prod'

Resources:
  # Subnet group to control where the Redis gets placed
  RedisSubnetGroup:
    Type: AWS::ElastiCache::SubnetGroup
    Properties:
      Description: Group of subnets to place Redis into
      SubnetIds: !Split [ ',', { 'Fn::ImportValue': !Sub '${App}-${Env}-PrivateSubnets' } ]

  # Security group to add the Redis cluster to the VPC,
  # and to allow the Fargate containers to talk to Redis on port 6379
  RedisSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: "Redis Security Group"
      VpcId: { 'Fn::ImportValue': !Sub '${App}-${Env}-VpcId' }

  # Enable ingress from other ECS services created within the environment.
  RedisIngress:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: Ingress from Fargate containers
      GroupId: !Ref 'RedisSecurityGroup'
      IpProtocol: tcp
      FromPort: 6379
      ToPort: 6379
      SourceSecurityGroupId: { 'Fn::ImportValue': !Sub '${App}-${Env}-EnvironmentSecurityGroup' }

  RedisParameterGroup:
    Type: AWS::ElastiCache::ParameterGroup
    Properties:
      CacheParameterGroupFamily: redis7
      Description: Notification parameters
      Properties:
        notify-keyspace-events: 'K$hg'

  # The cluster itself.
  Redis:
    Type: AWS::ElastiCache::CacheCluster
    Properties:
      Engine: redis
      CacheNodeType:
        !If ['IsProduction', 'cache.m4.large', 'cache.t2.micro']
      NumCacheNodes: 1
      CacheParameterGroupName: !Ref 'RedisParameterGroup'
      CacheSubnetGroupName: !Ref 'RedisSubnetGroup'
      VpcSecurityGroupIds:
        - !GetAtt 'RedisSecurityGroup.GroupId'

  # Redis endpoint stored in SSM so that other services can retrieve the endpoint.
  RedisEndpointAddressParam:
    Type: AWS::SSM::Parameter
    Properties:
      Name: !Sub '/${App}/${Env}/redis'   # Other services can retrieve the endpoint from this path.
      Type: String
      Value: !GetAtt 'Redis.RedisEndpoint.Address'

Outputs:
  RedisEndpoint:
    Description: The endpoint of the redis cluster
    Value: !GetAtt 'Redis.RedisEndpoint.Address'
    Export:
      Name: !Sub ${App}-${Env}-RedisEndpoint

[dannyrandall]

Thank you! So sorry to ask for some more info, but I just noticed this section -

  - A Virtual Private Cloud to control networking of your AWS resources              [delete complete]           [7.4s]
    Resource creation cancelled

Could you open the CloudFormation console in a browser and find the atom-project-demo stack? You may have to filter for "Failed" stacks instead of "Active". Once you find it, click on the "Events" tab and share any error messages you see there!😊 I think CloudFormation may remove/hide older failed stacks (I'm not positive!) so if it's not visible in the console, then (1) ensure you're in the region you deployed the env to and (2) redeploy the env if it's disappeared.

Thank you!!

[Epiphero]

Ah, good tip, I see:

Resource handler returned message: "The maximum number of internet gateways has been reached. (Service: Ec2, Status Code: 400, Request ID: redacted)" (RequestToken: redacted, HandlerErrorCode: ServiceLimitExceeded)

I adjusted the "Service Quotas > AWS services > Amazon Virtual Private Cloud (Amazon VPC) > Internet gateways per Region" quota and now I'm back to the No export named atom-genesys-demo-PrivateSubnets found. Rollback reque sted by user message.

Looking back in the stacks on the console:

Embedded stack arn:aws:cloudformation:us-east-1:860793925650:stack/atom-genesys-demo-AddonsStack-MJANC1N1M6MF/0eac3590-da4c-11ed-a596-12f087ff0335 was not successfully created: No export named atom-genesys-demo-PrivateSubnets found. Rollback requested by user.

Looking into the AddonsStack, I see for example atom-project-test-AddonsStack-IDSTRING, but not one for the demo environment yet.

Does this tie back to the addons config above? Thanks for the help!

[dannyrandall]

Ah I see! Ok, so after you raised your Internet Gateway quota, you tried running copilot env deploy --name demo again, and there are no other errors in the stack events other than the "No export..." error?🤔

[Epiphero]

Correct (well, that error, and a few "Resource creation cancelled" errors that seem to be because it gave up after the "No export..." error).

Given that it seems to be related to the addon, I:

1. moved the copilot/environments/addons/redis.yml out of the copilot directory;
2. deployed again, this time successfully;
3. moved redis.yml back;
4. deployed a second time, again successfully

From this I gather I had a combination of a service limit issue and some sort of dependency specification deficiency. So while ideally, I'd figure out how to modify the addon to allow single step deployment, at least I have a workaround now.

Thanks again 🙏

[jryanmcc]

I just ran into this as well... I don't appear of have any service limits. I am however importing existing Vpc and subnets. My resolution was exactly the same as @Epiphero above. It would be nice to see this fixed, though I agree I won't be standing up new environments very often.

[dannyrandall]

Hi @Epiphero and @jryanmcc! Sorry for the super late response - I just came across this issue and reading through it again think I found the root cause! The issue is that your addon template depends on Outputs that don't exist at the time CFN is deploying the stack. From the CloudFormation docs:

Output values are available after the stack operation is complete. Stack output values aren't available when a stack status is in any of the IN_PROGRESS status.

To fix it, you need to remove any Fn::ImportValue's in your Addon stack, and use Ref instead. First, update your addons.parameters.yml to pass values you want referenced from the main environment stack. For @Epiphero's example:

Parameters:
  PrivateSubnet1: !Ref PrivateSubnet1
  PrivateSubnet2: !Ref PrivateSubnet2
  VpcId: !Ref VPC
  EnvironmentSecurityGroup: !Ref EnvironmentSecurityGroup

Then, replace any Fn::ImportValue's in your Addon stack with !Ref to the defined parameter names. For example, change the RedisSecurityGroup resource to:

  RedisSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: "Redis Security Group"
      VpcId: !Ref VpcId

Hope this helps!

[github-actions]

This issue is stale because it has been open 60 days with no response activity. Remove the stale label, add a comment, or this will be closed in 14 days.

[github-actions]

This issue is closed due to inactivity. Feel free to reopen the issue if you have any further questions!

[tim-xergy]

Can we re-open this. I have a new environment where I have chosen the default for creating a new vpc when creating a new environment. But when I try to deploy it says the maximum number of gateways has been reached.

Does copilot try and use the AWS default VPC if none is specified? because if so that'll reach the limit unless you manually add a new VPC

[Lou1415926]

hey @tim-xergy !

Does copilot try and use the AWS default VPC if none is specified?

No, it creates a new one.

I think the message indicates that you have 5 gateways already in that account and region. You can have in total of 5 internet gateway per account per region. The quota can be found here.

You can either delete an existing environment (thus the VPC and the internet gateway) that you don't need, or request for an increase of quota!