Custom AMI creation is failing on Ubuntu 18.04 when retrieving SGE package

[mauri-melato]

Bug description

Beginning on 2020-12-10 the creation of custom AMIs with the Ubuntu 18.04 operating system is failing.

Impacted functionalities include both the building a custom ParallelCluster AMI through the pcluster createami command and the use of a custom AMI at runtime.
This issue does not affect the use of default ParallelCluster AMIs, nor does it affect the creation of custom AMIs by modifying a ParallelCluster AMI. This issue does not affect the creation of custom AMIs for any operating systems other than Ubuntu 18.04.

This issue affects all versions of ParallelCluster supporting Ubuntu 18.04, from 2.5.0 up to 2.10.0.
The root cause of this issue lies in the sge_preinstall.sh script from the ParallelCluster cookbook project that was relying on the Ubuntu Eoan (19.10) repo, which is no longer available.

We will update this issue as we have more information, and we will provide a recommended workaround as soon as one is available. Thank you

[ddeidda]

Hi,

the issue has been fixed in 2.10 branch by patch #835 and will be released with the next patch version 2.10.1.

For 2.10.0 we have provided a patched cookbook archive at the url https://us-east-1-aws-parallelcluster.s3.amazonaws.com/patches/2.10.0/aws-parallelcluster-cookbook-2.10.0.patch-20201211.tgz that can be used to create an Ubuntu 18.04 custom AMI.

When usng the pcluster createami command to create your custom ParallelCluster AMI please specify the above cookbook URL with the -cc parameter like in the example below:

$ pcluster createami -cc https://us-east-1-aws-parallelcluster.s3.amazonaws.com/patches/2.10.0/aws-parallelcluster-cookbook-2.10.0.patch-20201211.tgz -ai <base_ami> -os ubuntu1804 

We don't have yet a solution for the use case "Custom AMI at Runtime"

[demartinofra]

The released 2.10.1 addresses this issue. I will resolve this and leave the wiki for future reference: https://github.com/aws/aws-parallelcluster/wiki/Issue-with-Ubuntu-18.04-Custom-AMI-creation