airflow-ci-infra.git
17 months agoRemove left-over docker containers before fixing permissions (#5)
Ash Berlin-Taylor [Mon, 1 Mar 2021 11:10:58 +0000 (11:10 +0000)] 
Remove left-over docker containers before fixing permissions (#5)

If the docker container is still running and creating files (as might be
the case for the prod image builds) then some files could be left
uncleaned, causing the next job to fail.

17 months agoUser-data script to bootstrap self-hosted runner on ASG (#4)
Ash Berlin-Taylor [Mon, 1 Mar 2021 10:56:24 +0000 (10:56 +0000)] 
User-data script to bootstrap self-hosted runner on ASG (#4)

This runner-supervisor script has been manually uploaded to S3 (it was too big
to include in the userdata)

The cloud init script has been manually uploaded by running, and the ASG
is configured to pick the Latest version already, so new instances will
start using the new script.

```
aws --profile airflow ec2 create-launch-template-version \
    --launch-template-name GithubRunner \
    --launch-template-data UserData="$(base64 -w0 cloud-init.yml)" \
    --source-version='$Latest'
```

17 months agoLambda function to scale ASG based on Github webhooks (#2)
Ash Berlin-Taylor [Thu, 18 Feb 2021 09:55:49 +0000 (09:55 +0000)] 
Lambda function to scale ASG based on Github webhooks (#2)

19 months agoMerge pull request #1 from apache/register-runner-script
Ash Berlin-Taylor [Fri, 15 Jan 2021 14:29:26 +0000 (14:29 +0000)] 
Merge pull request #1 from apache/register-runner-script

Add script to help store self-hosted runner creds in AWS SSM

19 months agofixup! Add script to help store self-hosted runner creds in AWS SSM 1/head
Ash Berlin-Taylor [Fri, 15 Jan 2021 12:47:28 +0000 (12:47 +0000)] 
fixup! Add script to help store self-hosted runner creds in AWS SSM

19 months agoAdd script to help store self-hosted runner creds in AWS SSM
Ash Berlin-Taylor [Tue, 12 Jan 2021 11:54:51 +0000 (11:54 +0000)] 
Add script to help store self-hosted runner creds in AWS SSM

We can't create self-hosted runners "on-demand", so we need to
pre-create a "pool" of them for use by the auto-scaled nodes.

This script automated the process of converting the short-lived token in
to long-lived credentials (by using the runner binaries in a temporary
directory) and then storing the resulting files in AWS's ParameterStore

19 months agoAdd readme
Ash Berlin-Taylor [Mon, 4 Jan 2021 16:10:59 +0000 (16:10 +0000)] 
Add readme