A Guided Tour¶
This page is the web version of the interactive commandline tutorial.
The best way of getting started with handoff is to go through interactively. You should be able to finish it in 30 minutes to an hour.
handoff quick_start make
./projects/start
Below is the web-version of the tutorial.
Running a task locally¶
In this section, we will learn how to run a task with handoff locally, using project 01_word_count.
Each project directory contains:
> ls -l 01_word_count
files
project.yml
project.yml looks like:
> cat 01_word_count/project.yml
commands:
- command: cat
args: "./files/the_great_dictator_speech.txt"
- command: wc
args: "-w"
envs:
- key: TITLE
value: "The Great Dictator"
Here,
commandslists the commands and arguments.envslists the environment varaibles.
The example from 01_word_count runs a command line equivalent of:
cat ./files/the_great_dictator_speech.txt | wc -w
Now let’s run. Try entering this command below:
> handoff --project 01_word_count --workspace workspace run local
INFO - 2020-08-06 03:35:12,691 - handoff.config - Reading configurations from 01_word_count/project.yml
INFO - 2020-08-06 03:35:12,693 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:35:12,771 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:35:13,056 - handoff.config - You have the access to AWS resources.
WARNING - 2020-08-06 03:35:13,123 - handoff.config - Environment variable HO_BUCKET is not set. Remote file read/write will fail.
INFO - 2020-08-06 03:35:13,123 - handoff.config - Writing configuration files in the workspace configuration directory workspace/config
INFO - 2020-08-06 03:35:13,123 - handoff.config - Copying files from the local project directory 01_word_count
INFO - 2020-08-06 03:35:13,124 - handoff.config - Running run local in workspace directory
INFO - 2020-08-06 03:35:13,124 - handoff.config - Job started at 2020-08-06 03:35:13.124542
INFO - 2020-08-06 03:35:13,130 - handoff.config - Job ended at 2020-08-06 03:35:13.130391
If you see the output that looks like:
INFO - 2020-08-03 04:51:01,971 - handoff.config - Reading configurations from 01_word_count/project.yml
...
INFO - 2020-08-03 04:51:02,690 - handoff.config - Processed in 0:00:00.005756
Then great! You just ran the first local test. It created a workspace directory that looks like:
> ls -l workspace
artifacts
config
files
And the word count is stored at workspace/artifacts/state. Here is the content:
> cat workspace/artifacts/state
644
By the way, the example text is from the awesome speech by Charlie Chaplin’s in the movie the Great Dictator.
Here is a link to the famous speech scene. Check out on YouTube: https://www.youtube.com/watch?v=J7GY1Xg6X20
And here is the first few paragraphs of the text:
I’m sorry, but I don’t want to be an emperor. That’s not my business. I don’t want to rule or conquer anyone. I should like to help everyone - if possible - Jew, Gentile - black man - white. We all want to help one another. Human beings are like that. We want to live by each other’s happiness - not by each other’s misery. We don’t want to hate and despise one another. In this world there is room for everyone. And the good earth is rich and can provide for everyone. The way of life can be free and beautiful, but we have lost the way.
Greed has poisoned men’s souls, has barricaded the world with hate, has goose-stepped us into misery and bloodshed. We have developed speed, but we have shut ourselves in. Machinery that gives abundance has left us in want. Our knowledge has made us cynical. Our cleverness, hard and unkind. We think too much and feel too little. More than machinery we need humanity. More than cleverness we need kindness and gentleness. Without these qualities, life will be violent and all will be lost….
Now to the second example. This time project.yml looks like:
> cat 02_collect_stats/project.yml
commands:
- command: cat
args: ./files/the_great_dictator_speech.txt
- command: python files/stats_collector.py
- command: wc
args: -w
…which is shell equivalent to
cat ./files/the_great_dictator_speech.txt | python ./files/stats_collector.py | wc -w
The script for the second command stats_collector.py can be found in 02_collect_stats/files directory and it is a Python script that looks like:
> cat 02_collect_stats/files/stats_collector.py
#!/usr/bin/python
import io, json, logging, sys, os
LOGGER = logging.getLogger()
def collect_stats(outfile):
"""
Read from stdin and count the lines. Output to a file after done.
"""
lines = io.TextIOWrapper(sys.stdin.buffer, encoding="utf-8")
output = {"rows_read": 0}
for line in lines:
try:
o = json.loads(line)
print(json.dumps(o))
if o["type"].lower() == "record":
output["rows_read"] += 1
except json.decoder.JSONDecodeError:
print(line)
output["rows_read"] += 1
with open(outfile, "w") as f:
json.dump(output, f)
f.write("\n")
if __name__ == "__main__":
collect_stats("artifacts/collect_stats.json")
The script reads from stdin and counts the lines while passing the raw input to stdout. The raw text is then processed by the third command (wc -w) and it conts the number of words.
Now let’s run. Try entering this command below:
> handoff --project 02_collect_stats --workspace workspace run local
INFO - 2020-08-06 03:35:13,401 - handoff.config - Reading configurations from 02_collect_stats/project.yml
INFO - 2020-08-06 03:35:13,402 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:35:13,481 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:35:13,765 - handoff.config - You have the access to AWS resources.
WARNING - 2020-08-06 03:35:13,830 - handoff.config - Environment variable HO_BUCKET is not set. Remote file read/write will fail.
INFO - 2020-08-06 03:35:13,830 - handoff.config - Writing configuration files in the workspace configuration directory workspace/config
INFO - 2020-08-06 03:35:13,830 - handoff.config - Copying files from the local project directory 02_collect_stats
INFO - 2020-08-06 03:35:13,831 - handoff.config - Running run local in workspace directory
INFO - 2020-08-06 03:35:13,831 - handoff.config - Job started at 2020-08-06 03:35:13.831683
INFO - 2020-08-06 03:35:13,881 - handoff.config - Job ended at 2020-08-06 03:35:13.881507
INFO - 2020-08-06 03:35:13,881 - handoff.config - Processed in 0:00:00.049824
Let’s check out the contents of the second command:
> cat workspace/artifacts/collect_stats.json
{"rows_read": 15}
In the next section, we will try pullin the currency exchange rate data. You will also learn how to create Python virtual enviroments for each command and pip-install commands.
Virtual environment and install¶
In this section, we will retrieve currency exchange rates and write out to CSV file.
We will install singer.io (https://singer.io), a data collection framework, in Python vitual environment.
We will use 03_exchange_rates project. project.yml looks like:
> cat 03_exchange_rates/project.yml
commands:
- command: "tap-exchangeratesapi"
args: "--config config/tap-config.json"
venv: "proc_01"
installs:
- "pip install tap-exchangeratesapi"
- command: "python files/stats_collector.py"
venv: "proc_01"
- command: "target-csv"
args: "--config config/target-config.json"
venv: "proc_02"
installs:
- "pip install target-csv"
deploy:
provider: "aws"
platform: "fargate"
envs:
resource_group: "handoff-test"
docker_image: "singer_exchange_rates_to_csv"
task: "test-03-exchange-rates"
…which is shell equivalent to
tap-exchangeratesapi | python files/stats_collector.py | target-csv
Before we can run this, we need to install tap-exchangeratesapi and target-csv. The instructions for the install are listed in install section of project.yml.
Notice venv entries for each command. handoff can create Python virtual
enviroment for each command to avoid conflicting dependencies among the
commands.
To install everything, run this command:
> handoff -p 03_exchange_rates -w workspace_03 workspace install
INFO - 2020-08-06 03:35:14,158 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:35:14,240 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:35:14,524 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:35:14,524 - handoff.config - Platform: aws
INFO - 2020-08-06 03:35:19,456 - handoff.config - Running /bin/bash -c "source proc_01/bin/activate && pip install wheel && pip install tap-exchangeratesapi"
Requirement already satisfied: wheel in ./proc_01/lib/python3.6/site-packages (0.34.2)
Processing /home/ubuntu/.cache/pip/wheels/1f/73/f9/xxxxxxxx0dba8423841c1404f319bb/tap_exchangeratesapi-0.1.1-cp36-none-any.whl
Processing /home/ubuntu/.cache/pip/wheels/6e/07/1b/xxxxxxxx6d9ce55c05f67a69127e25/singer_python-5.3.3-cp36-none-any.whl
Processing /home/ubuntu/.cache/pip/wheels/fc/d8/34/xxxxxxxx027b62dfcf922fdf8e396d/backoff-1.3.2-cp36-none-any.whl
Collecting requests==2.21.0
.
.
.
Collecting python-dateutil
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytzdata
Using cached pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
Collecting pytz
Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting six>=1.5
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: jsonschema, simplejson, pytz, tzlocal, six, python-dateutil, pytzdata, pendulum, singer-python, target-csv
Successfully installed jsonschema-2.6.0 pendulum-1.2.0 python-dateutil-2.8.1 pytz-2020.1 pytzdata-2020.1 simplejson-3.11.1 singer-python-2.1.4 six-1.15.0 target-csv-0.3.0 tzlocal-2.1
Now let’s run the task. Try entering this command below:
> handoff -p 03_exchange_rates -w workspace_03 run local
INFO - 2020-08-06 03:35:29,258 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:35:29,339 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:35:29,626 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:35:29,626 - handoff.config - Platform: aws
INFO - 2020-08-06 03:35:29,626 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:35:29,693 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:35:29,693 - handoff.config - Writing configuration files in the workspace configuration directory workspace_03/config
INFO - 2020-08-06 03:35:29,694 - handoff.config - Copying files from the local project directory 03_exchange_rates
INFO - 2020-08-06 03:35:29,695 - handoff.config - Running run local in workspace_03 directory
INFO - 2020-08-06 03:35:29,695 - handoff.config - Job started at 2020-08-06 03:35:29.695732
.
.
.
INFO - 2020-08-06 03:35:33,964 - handoff.config - Job ended at 2020-08-06 03:35:33.964206
INFO - 2020-08-06 03:35:33,964 - handoff.config - Processed in 0:00:04.268474
This process should have created a CSV file in artifacts directory:
exchange_rate-20200806T033530.csv
…which looks like:
CAD,HKD,ISK,PHP,DKK,HUF,CZK,GBP,RON,SEK,IDR,INR,BRL,RUB,HRK,JPY,THB,CHF,EUR,MYR,BGN,TRY,CNY,NOK,NZD,ZAR,USD,MXN,SGD,AUD,ILS,KRW,PLN,date
0.0127290837,0.0725398406,1.3197211155,0.4630976096,0.0618218792,2.9357569721,0.2215388446,0.007434429,0.0401958831,0.0863047809,135.1005146082,0.7041915671,0.050374336,0.6657569721,0.0625373506,1.0,0.29312749,0.0088188911,0.0083001328,0.0399311089,0.0162333997,0.0642571381,0.0655312085,0.0889467131,0.0142670983,0.158440405,0.0093592297,0.2132744024,0.0130336985,0.0134852258,0.032375498,11.244189907,0.0371372842,2020-07-10T00:00:00Z
0.0126573311,0.072330313,1.313014827,0.4612685338,0.061324547,2.9145799012,0.2195057661,0.007408402,0.0399036244,0.085529654,134.613509061,0.7019439868,0.049830313,0.6601894563,0.0620593081,1.0,0.2929324547,0.0088014827,0.0082372323,0.0397907743,0.0161103789,0.0641054366,0.0653286656,0.0878500824,0.0141894563,0.1562817133,0.0093319605,0.209931631,0.0129678748,0.013383855,0.0321466227,11.2139209226,0.0368682043,2020-07-13T00:00:00Z
Now that we know how to run locally, we will gradually thinking about how to deploy this in the cloud severlessly. We will learn how to save and fetch the configurations to the remote storage. Before doing that, we will cover how to set up AWS account and profile in the next section.
Setting up an AWS account and profile¶
handoff helps you to deploy the task in the cloud computing services. Our goal in this tutorial is to deploy the currency exchange rates task to AWS Fargate.
Before we can deploy, we need to set up the AWS account.
Do I need a credit card to use AWS? How much does it cost?
Yes, you need a credit card. But it won’t cost a cup of coffee for this tutorial. (Probably not even a dollar.) Some of the AWS services we use comes with a Free Tier and you won’t be charged for the eligible usage.
Learn more about Free Tier.
Here is the complete list of the services we use:
- Simple Storage Service, free with Free Tier usage:
- https://aws.amazon.com/s3/pricing/#AWS_Free_Tier
- Fargate, almost free:
- Price calculator http://fargate-pricing-calculator.site.s3-website-us-east-1.amazonaws.com/ (We use 0.25vCPU with 0.5GB RAM for less than 5 minutes.)
- Systems Management Parameter Store, free (Standard tier):
- Pricing: https://aws.amazon.com/systems-manager/pricing/#Parameter_Store
- Elastic Container Registry, free for 500MB per month
- Pricing: https://aws.amazon.com/ecr/pricing/
AWS sign up is easy. Just go to:
and complete the sign up process.
The rest of the tutorial assumes that you have an active AWS account.
If you haven’t, install AWS CLI (command line interface):
Also generate access key ID and secret access key by following:
Take note of the values that looks like:
- Access key ID: AKIAIOSFODNN7EXAMPLE
- Secret access key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
The only time that you can view or download the secret access key is when you create the keys. You cannot recover them later. So keep a local copy.
Last step before going back to handoff. From the console, run:
Enter Access key ID and Secret access key you created in the last step.
For region, use one of these keys (for USA users, us-east-1 would do):
ap-northeast-1
ap-northeast-2
ap-south-1
ap-southeast-1
ap-southeast-2
ca-central-1
eu-central-1
eu-north-1
eu-west-1
eu-west-2
eu-west-3
sa-east-1
us-east-1
us-east-2
us-west-1
us-west-2
Learn more about AWS profile here
Now that we have set up the AWS account, let’s store the configurations in the cloud in the next section.
Running task locally with remotely stored configurations¶
We will continue using 03_exchange_rates example. But this time, we will store the configurations to the remote data store.
Let’s review how 03_exchange_rates project directory is structured:
> ls -l 03_exchange_rates
config
files
project.yml
In the config directory, we have a couple of configuration files:
> ls -l 03_exchange_rates/config
tap-config.json
target-config.json
…which look like:
> cat 03_exchange_rates/config/tap-config.json
{ "base": "JPY", "start_date": "2020-07-10" }
> cat 03_exchange_rates/config/target-config.json
{
"delimiter": ",",
"quotechar": "'",
"destination_path": "artifacts"
}
Often times, such configuration files contain sensitive information. So we push the configrations to a secure key store such as AWS Systems Manager (SSM) Parameter Store (In SecuresString format) handoff wraps this proces by a single command. handoff packs all the JSON format files under the configuration directory and send it to the remote key-value storage.
Try running:
> handoff -p 03_exchange_rates config push
INFO - 2020-08-06 03:35:35,362 - handoff.config - Compiling config from 03_exchange_rates
INFO - 2020-08-06 03:35:35,362 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:35:35,443 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:35:35,732 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:35:35,732 - handoff.config - Platform: aws
INFO - 2020-08-06 03:35:35,732 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:35:35,799 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:35:35,799 - handoff.config - Putting the config to AWS SSM Parameter Store with Standard tier
INFO - 2020-08-06 03:35:35,814 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:35:36,184 - handoff.config - See the parameters at https://console.aws.amazon.com/systems-manager/parameters/?region=us-east-1&tab=Table#list_parameter_filters=Name:Contains:xxxxxxxxe-rates-config
Look at the end of the log that says,
See the parameters at https://console.aws.amazon.com/systems-manager/parameters/?region=...
Grab the link and open in the browswer (AWS login required) to confirm that the parameters are uploaded.
We also have some files needed for the task execution:
> ls -l 03_exchange_rates/files
stats_collector.py
We need to store this somewhere accessible. So we will create a cloud data storage (S3 bucket) for that.
Try running:
> handoff -p 03_exchange_rates cloud create_bucket
INFO - 2020-08-06 03:35:36,484 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:35:36,567 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:35:36,854 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:35:36,854 - handoff.config - Platform: aws
INFO - 2020-08-06 03:35:36,855 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:35:36,921 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:35:37,361 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:35:37,364 - handoff.config - Platform: aws
INFO - 2020-08-06 03:35:37,365 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:35:37,381 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
.
.
.
INFO - 2020-08-06 03:35:37,815 - handoff.config - {'StackId': 'arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/handoff-test-bucket/xxxxxxxx861a6983', 'ResponseMetadata': {'RequestId': 'xxxxxxxx36ff08b9', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'xxxxxxxx36ff08b9', 'content-type': 'text/xml', 'content-length': '389', 'date': 'Thu, 06 Aug 2020 03:35:37 GMT'}, 'RetryAttempts': 0}}
INFO - 2020-08-06 03:35:37,815 - handoff.config - Check the progress at https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/stackinfo?viewNested=true&hideStacks=false&stackId=arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/handoff-test-bucket/xxxxxxxx861a6983
Wait for a minute and check here
https://s3.console.aws.amazon.com
to make sure the bucket is created before you proceed.
Now it’s time to push the files to the bucket.
Try running:
> handoff -p 03_exchange_rates files push
INFO - 2020-08-06 03:38:38,087 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:38:38,168 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:38:38,456 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:38:38,456 - handoff.config - Platform: aws
INFO - 2020-08-06 03:38:38,456 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:38:38,523 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:38:38,538 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:38:38,620 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:38:38,914 - handoff.services.cloud.aws.s3 - Uploading 03_exchange_rates/files to bucket xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:38:38,915 - handoff.services.cloud.aws.s3 - Files to be uploaded: ['03_exchange_rates/files/stats_collector.py']
.
.
.
INFO - 2020-08-06 03:38:38,915 - handoff.services.cloud.aws.s3 - Uploading 03_exchange_rates/files/stats_collector.py to Amazon S3 bucket xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/files/stats_collector.py
INFO - 2020-08-06 03:38:39,279 - handoff.config - See the files at https://s3.console.aws.amazon.com/s3/buckets/xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/files/
Look at the end of the log that says,
See the files at https://s3.console.aws.amazon.com/s3/...
Grab the link and open in the browswer (AWS login required) to see the files directory is uploaded.
Install the workspace:
> handoff -p 03_exchange_rates -w workspace workspace install
INFO - 2020-08-06 03:38:39,718 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:38:39,799 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:38:40,082 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:38:40,082 - handoff.config - Platform: aws
INFO - 2020-08-06 03:38:44,491 - handoff.config - Running /bin/bash -c "source proc_01/bin/activate && pip install wheel && pip install tap-exchangeratesapi"
Requirement already satisfied: wheel in ./proc_01/lib/python3.6/site-packages (0.34.2)
Processing /home/ubuntu/.cache/pip/wheels/1f/73/f9/xxxxxxxx0dba8423841c1404f319bb/tap_exchangeratesapi-0.1.1-cp36-none-any.whl
Collecting requests==2.21.0
Using cached requests-2.21.0-py2.py3-none-any.whl (57 kB)
Processing /home/ubuntu/.cache/pip/wheels/fc/d8/34/xxxxxxxx027b62dfcf922fdf8e396d/backoff-1.3.2-cp36-none-any.whl
.
.
.
Collecting tzlocal
Using cached tzlocal-2.1-py2.py3-none-any.whl (16 kB)
Collecting python-dateutil
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytz
Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting six>=1.5
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: pytzdata, pytz, tzlocal, six, python-dateutil, pendulum, simplejson, singer-python, jsonschema, target-csv
Successfully installed jsonschema-2.6.0 pendulum-1.2.0 python-dateutil-2.8.1 pytz-2020.1 pytzdata-2020.1 simplejson-3.11.1 singer-python-2.1.4 six-1.15.0 target-csv-0.3.0 tzlocal-2.1
Now let’s run the command by pulling the configurations and files from remote.
Try running:
> handoff -p 03_exchange_rates -w workspace run remote_config --push-artifacts
INFO - 2020-08-06 03:38:53,883 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:38:53,963 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:38:54,248 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:38:54,248 - handoff.config - Platform: aws
INFO - 2020-08-06 03:38:54,248 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:38:54,315 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:38:54,315 - handoff.config - Writing configuration files in the workspace configuration directory workspace/config
INFO - 2020-08-06 03:38:54,315 - handoff.config - Reading configurations from remote parameter store.
INFO - 2020-08-06 03:38:54,316 - handoff.config - Reading precompiled config from remote.
INFO - 2020-08-06 03:38:54,330 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
.
.
.
INFO - 2020-08-06 03:38:57,479 - handoff.services.cloud.aws.s3 - Files to be uploaded: ['workspace/artifacts/state', 'workspace/artifacts/exchange_rate-20200806T033855.csv', 'workspace/artifacts/collect_stats.json']
INFO - 2020-08-06 03:38:57,479 - handoff.services.cloud.aws.s3 - Uploading workspace/artifacts/state to Amazon S3 bucket xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/artifacts/state
INFO - 2020-08-06 03:38:57,631 - handoff.services.cloud.aws.s3 - Uploading workspace/artifacts/exchange_rate-20200806T033855.csv to Amazon S3 bucket xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/artifacts/exchange_rate-20200806T033855.csv
INFO - 2020-08-06 03:38:57,991 - handoff.services.cloud.aws.s3 - Uploading workspace/artifacts/collect_stats.json to Amazon S3 bucket xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/artifacts/collect_stats.json
INFO - 2020-08-06 03:38:58,139 - handoff.config - See the files at https://s3.console.aws.amazon.com/s3/buckets/xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/artifacts/
INFO - 2020-08-06 03:38:58,139 - handoff.services.cloud.aws.s3 - Copying recursively from s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/* to s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/runs/2020-08-06T03:38:58.139874/*
INFO - 2020-08-06 03:38:58,445 - handoff.services.cloud.aws.s3 - Copied s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/artifacts/collect_stats.json to s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/runs/2020-08-06T03:38:58.139874/artifacts/collect_stats.json
INFO - 2020-08-06 03:38:58,644 - handoff.services.cloud.aws.s3 - Copied s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/artifacts/exchange_rate-20200806T033855.csv to s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/runs/2020-08-06T03:38:58.139874/artifacts/exchange_rate-20200806T033855.csv
INFO - 2020-08-06 03:38:58,822 - handoff.services.cloud.aws.s3 - Copied s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/artifacts/state to s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/runs/2020-08-06T03:38:58.139874/artifacts/state
INFO - 2020-08-06 03:38:58,988 - handoff.services.cloud.aws.s3 - Copied s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/files/stats_collector.py to s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/runs/2020-08-06T03:38:58.139874/files/stats_collector.py
Notice that we used –push-artifacts option in the last command. With this option, we pushed the result to the bucket under
test-03-exchange-rates/last/artifacts
directory.
Also note that artifacts are automatically archived at each run at
test-03-exchange-rates/runs/
directory.
Next step is to prepare a Docker image and test running it locally.
Building, running, and pushing a Docker image¶
We will continue using 03_exchange_rates example. Instead of running the task on the host machine, let’s run on Docker.
Let’s build a Docker image.
Try running the following command. Enter y when prompted at the beginning. The build may take 5~10 minutes.
> handoff -p 03_exchange_rates container build
INFO - 2020-08-06 03:38:59,397 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:38:59,478 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:38:59,766 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:38:59,766 - handoff.config - Platform: aws
INFO - 2020-08-06 03:38:59,766 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:38:59,833 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:38:59,913 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:38:59,913 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:38:59,916 - handoff.config - Platform: aws
INFO - 2020-08-06 03:38:59,916 - handoff.config - Setting environment variables from config.
.
.
.
INFO - 2020-08-06 03:41:40,008 - handoff.config - Step 27/27 : CMD handoff ${COMMAND:-run remote_config} -w workspace -a -d ${DATA:-{}} -a
INFO - 2020-08-06 03:41:40,044 - handoff.config - ---> Running in 21a548c43c0b
INFO - 2020-08-06 03:41:40,148 - handoff.config - ---> 1bbadd87b10d
INFO - 2020-08-06 03:41:41,621 - handoff.config - Successfully built 1bbadd87b10d
INFO - 2020-08-06 03:41:41,652 - handoff.config - Successfully tagged singer_exchange_rates_to_csv:0.5
Now let’s run the code in the Docker container.
> handoff -p 03_exchange_rates container run
INFO - 2020-08-06 03:41:45,336 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:41:45,509 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:41:45,964 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:41:45,965 - handoff.config - Platform: aws
INFO - 2020-08-06 03:41:45,965 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:41:46,029 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:41:46,108 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:41:46,108 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:41:46,111 - handoff.config - Platform: aws
INFO - 2020-08-06 03:41:46,112 - handoff.config - Setting environment variables from config.
.
.
.
INFO - 2020-08-06 03:41:54,603 - handoff.services.cloud.aws.s3 - Copied s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/artifacts/collect_stats.json to s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/runs/2020-08-06T03:41:54.279771/artifacts/collect_stats.json
INFO - 2020-08-06 03:41:54,797 - handoff.services.cloud.aws.s3 - Copied s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/artifacts/exchange_rate-20200806T033855.csv to s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/runs/2020-08-06T03:41:54.279771/artifacts/exchange_rate-20200806T033855.csv
INFO - 2020-08-06 03:41:55,002 - handoff.services.cloud.aws.s3 - Copied s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/artifacts/exchange_rate-20200806T034150.csv to s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/runs/2020-08-06T03:41:54.279771/artifacts/exchange_rate-20200806T034150.csv
INFO - 2020-08-06 03:41:55,172 - handoff.services.cloud.aws.s3 - Copied s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/artifacts/state to s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/runs/2020-08-06T03:41:54.279771/artifacts/state
INFO - 2020-08-06 03:41:55,342 - handoff.services.cloud.aws.s3 - Copied s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/files/stats_collector.py to s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/runs/2020-08-06T03:41:54.279771/files/stats_collector.py
Confirm the run by checking the logs. Also check the artifacts on S3:
xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/artifacts/
directory.
Now that we know the Docker container runs fine, let’s push it to AWS Elastic Container Registry. This may take a few minutes.
> handoff -p 03_exchange_rates container push
INFO - 2020-08-06 03:41:57,295 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:41:57,378 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:41:57,659 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:41:57,659 - handoff.config - Platform: aws
INFO - 2020-08-06 03:41:57,660 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:41:57,725 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:41:57,804 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:41:57,804 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:41:57,807 - handoff.config - Platform: aws
INFO - 2020-08-06 03:41:57,808 - handoff.config - Setting environment variables from config.
.
.
.
INFO - 2020-08-06 03:44:22,758 - handoff.config - id: 6734fdd29c24 [================================================> ] 526.1MB/540.8MB
INFO - 2020-08-06 03:44:23,478 - handoff.config - id: c836a53f9c0b [=============================================> ] 401MB/439.5MB
INFO - 2020-08-06 03:44:24,493 - handoff.config - id: 44aecb8afa22 [=============================================> ] 400.4MB/439.5MB
INFO - 2020-08-06 03:44:29,789 - handoff.config - id: c836a53f9c0b [================================================> ] 426.4MB/439.5MB
INFO - 2020-08-06 03:44:29,849 - handoff.config - id: 6734fdd29c24 status: Pushed
INFO - 2020-08-06 03:44:30,083 - handoff.config - id: 44aecb8afa22 [================================================> ] 426.9MB/439.5MB
INFO - 2020-08-06 03:44:35,306 - handoff.config - id: c836a53f9c0b [==================================================>] 453.1MB
INFO - 2020-08-06 03:44:35,744 - handoff.config - id: 44aecb8afa22 [==================================================>] 453.7MB
INFO - 2020-08-06 03:44:37,333 - handoff.config - id: c836a53f9c0b status: Pushed
INFO - 2020-08-06 03:44:39,716 - handoff.config - id: 44aecb8afa22 status: Pushed
Confirm that the Docker image has been uploaded to:
https://console.aws.amazon.com/ecr/repositories?region=us-east-1
Now that the Docker image is prepared, we will finally deploy the task on AWS Fargate in the next tutorial.
Deploying on AWS Fargate¶
We will finally deploy the task to AWS Fargate so we can automate the recurring task.
Fargate is part of AWS Elastic Container Service (ECS). The Fargate task run on the ECS cluster. We will first need to create a cluster. (You will only be charged for the usage.)
To make this process easy, handoff packed everything up in a command:
> handoff -p 03_exchange_rates cloud create_resources
INFO - 2020-08-06 03:44:42,787 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:44:42,867 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:44:43,152 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:44:43,152 - handoff.config - Platform: aws
INFO - 2020-08-06 03:44:43,152 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:44:43,218 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:44:44,678 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:44:44,686 - handoff.config - Platform: aws
INFO - 2020-08-06 03:44:44,686 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:44:44,709 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
.
.
.
INFO - 2020-08-06 03:44:45,269 - handoff.config - {'StackId': 'arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/handoff-test-resources/xxxxxxxxdd50faa5', 'ResponseMetadata': {'RequestId': 'xxxxxxxx20b2ef81', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'xxxxxxxx20b2ef81', 'content-type': 'text/xml', 'content-length': '392', 'date': 'Thu, 06 Aug 2020 03:44:45 GMT'}, 'RetryAttempts': 0}}
INFO - 2020-08-06 03:44:45,270 - handoff.config - Check the progress at https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/stackinfo?viewNested=true&hideStacks=false&stackId=arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/handoff-test-resources/xxxxxxxxdd50faa5
At the end of the log, you should see a line like:
Check the progress at https://console.aws.amazon.com/cloudformation/home?region=xxxx
Grab the entire link and open in a browser (you need to login in to AWS) to see the progress of the resource creation.
Wait until it says “CREATE_COMPLETE”
Now it’s time to deploy the task and here is the command:
> handoff -p 03_exchange_rates cloud create_task
INFO - 2020-08-06 03:47:45,531 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:47:45,610 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:47:45,897 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:47:45,897 - handoff.config - Platform: aws
INFO - 2020-08-06 03:47:45,898 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:47:45,964 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:47:46,436 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:47:46,439 - handoff.config - Platform: aws
INFO - 2020-08-06 03:47:46,439 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:47:46,521 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
.
.
.
INFO - 2020-08-06 03:47:46,847 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:47:47,363 - handoff.config - {'StackId': 'arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/test-03-exchange-rates/xxxxxxxxa78ee1f9', 'ResponseMetadata': {'RequestId': 'xxxxxxxxdb623334', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'xxxxxxxxdb623334', 'content-type': 'text/xml', 'content-length': '392', 'date': 'Thu, 06 Aug 2020 03:47:46 GMT'}, 'RetryAttempts': 0}}
INFO - 2020-08-06 03:47:47,363 - handoff.config - Check the progress at https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/stackinfo?viewNested=true&hideStacks=false&stackId=arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/test-03-exchange-rates/xxxxxxxxa78ee1f9
Here again, at the end of the log, you should see a line like:
Check the progress at https://console.aws.amazon.com/cloudformation/home?region=xxxx
Grab the entire link and open in a browser.
Wait until it says “CREATE_COMPLETE”
Once the task is created, try running on Fargate. To do so, run this command:
> handoff -p 03_exchange_rates cloud run
INFO - 2020-08-06 03:50:47,634 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:50:47,713 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:50:48,005 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:50:48,005 - handoff.config - Platform: aws
INFO - 2020-08-06 03:50:48,006 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:50:48,073 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:50:48,546 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:50:48,550 - handoff.config - Platform: aws
INFO - 2020-08-06 03:50:48,550 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:50:48,565 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
.
.
.
INFO - 2020-08-06 03:50:48,602 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:50:50,202 - handoff.config - {'tasks': [{'taskArn': 'arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/xxxxxxxx2f7337f3', 'clusterArn': 'arn:aws:ecs:us-east-1:xxxxxxxxxxxx:cluster/xxxxxxxxe-rates', 'taskDefinitionArn': 'arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task-definition/xxxxxxxxe-rates:11', 'overrides': {'containerOverrides': [{'name': 'singer_exchange_rates_to_csv', 'environment': [{'name': 'TASK_TRIGGERED_AT', 'value': '2020-08-06T03:50:48.550977'}]}], 'inferenceAcceleratorOverrides': []}, 'lastStatus': 'PROVISIONING', 'desiredStatus': 'RUNNING', 'cpu': '256', 'memory': '512', 'containers': [{'containerArn': 'arn:aws:ecs:us-east-1:xxxxxxxxxxxx:container/xxxxxxxxd07ec661', 'taskArn': 'arn:aws:ecs:us-east-1:xxxxxxxxxxxx:task/xxxxxxxx2f7337f3', 'name': 'singer_exchange_rates_to_csv', 'image': 'xxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/singer_exchange_rates_to_csv:0.5', 'lastStatus': 'PENDING', 'networkInterfaces': [], 'cpu': '256', 'memory': '512'}], 'version': 1, 'createdAt': datetime.datetime(2020, 8, 6, 3, 50, 50, 108000, tzinfo=tzlocal()), 'group': 'family:xxxxxxxxe-rates', 'launchType': 'FARGATE', 'platformVersion': '1.3.0', 'attachments': [{'id': 'xxxxxxxx028447f7', 'type': 'ElasticNetworkInterface', 'status': 'PRECREATED', 'details': [{'name': 'subnetId', 'value': 'subnet-0c65a427feccde6be'}]}], 'tags': []}], 'failures': [], 'ResponseMetadata': {'RequestId': 'xxxxxxxx2456c334', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'xxxxxxxx2456c334', 'content-type': 'application/x-amz-json-1.1', 'content-length': '1370', 'date': 'Thu, 06 Aug 2020 03:50:49 GMT'}, 'RetryAttempts': 0}}
INFO - 2020-08-06 03:50:50,202 - handoff.config - Check the task at https://us-east-1.console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/xxxxxxxxe-rates/tasks/xxxxxxxx2f7337f3
At the end of the log, you should see a line like:
Check the task at https://us-east-1.console.aws.amazon.com/ecs/home?region=xxxx
Grab the entire link and open in a browser.
- At the top you see:
- Task : 5a232313-e390….
In Details tab,
Cluster xxxxxxxxe-rates Launch type FARGATE … Last status PROVISIONING Desired status RUNNING
At the bottom of the page, you see:
| Name | Container Runtime ID | Status | Image | |
|---|---|---|---|---|
| singer_excha… | 1371…. | PROVISIONING | xxxxx |
Expand the section by clicking on a dark side-way trible “>” by the Task name.
The status should change in this order:
- PROVIONING
- RUNNING
- STOPPED
Once the status becomes RUNNING or STOPPED, you should be able to see the execution log by clicking “View logs in CloudWatch” link at the bottom of the section.
The log should be similar to the output from the local execution.
We confirmed that the task run in the same way as local execution. Now let’s go to the next section to learn how to schedule the task so it runs periodically.
Scheduling a task on Fargate¶
In this section, we will learn how to schedule a task so it runs automatically.
To schedule a task, use schedule command with target_id and
We pass those values to handoff with --data (-d for short) option:
> handoff -p 03_exchange_rates cloud schedule --data '{"target_id": "1", "cron": "55 03 * * ? *"}'
INFO - 2020-08-06 03:50:50,489 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:50:50,570 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:50:50,855 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:50:50,856 - handoff.config - Platform: aws
INFO - 2020-08-06 03:50:50,856 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:50:50,922 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:50:51,364 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:50:51,368 - handoff.config - Platform: aws
INFO - 2020-08-06 03:50:51,373 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:50:51,396 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
.
.
.
INFO - 2020-08-06 03:50:51,760 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:50:52,131 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:50:52,745 - handoff.config - {'FailedEntryCount': 0, 'FailedEntries': [], 'ResponseMetadata': {'RequestId': 'xxxxxxxxef778e23', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'xxxxxxxxef778e23', 'content-type': 'application/x-amz-json-1.1', 'content-length': '41', 'date': 'Thu, 06 Aug 2020 03:50:52 GMT'}, 'RetryAttempts': 0}}
INFO - 2020-08-06 03:50:52,745 - handoff.config - Check the status at https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/xxxxxxxxe-rates/scheduledTasks
At the end of the log, you should see a line like:
Check the progress at Check the status at https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/...
Grab the entire link and open in a browser (you need to login in to AWS) to see it’s scheduled.
We confirmed that the task run in the same way as local execution. Now let’s go to the next module to learn how to schedule the task so it runs periodically.
Cleaning up¶
Let’s clean everything up so we won’t pay a fraction of penny after forgeting about this exercise.
First unschedule the task:
> handoff -l warning -p 03_exchange_rates cloud unschedule -d '{"target_id": 1}'
Then delete the task:
> handoff -l warning -p 03_exchange_rates cloud delete_task
If there is no other task in the same resource group, we can delete it:
> handoff -l warning -p 03_exchange_rates cloud delete_resources
Here is how to delete the configurations from SSM Parameter Store:
> handoff -p 03_exchange_rates config delete
INFO - 2020-08-06 03:34:46,049 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:34:46,131 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:34:46,415 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:34:46,415 - handoff.config - Platform: aws
INFO - 2020-08-06 03:34:46,416 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:34:46,481 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:34:46,496 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
Here is how to delete the files from S3 bucket:
> handoff -p 03_exchange_rates files delete
INFO - 2020-08-06 03:34:47,114 - handoff.config - Reading configurations from 03_exchange_rates/project.yml
INFO - 2020-08-06 03:34:47,195 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:34:47,479 - handoff.config - You have the access to AWS resources.
INFO - 2020-08-06 03:34:47,480 - handoff.config - Platform: aws
INFO - 2020-08-06 03:34:47,480 - handoff.config - Setting environment variables from config.
INFO - 2020-08-06 03:34:47,545 - handoff.config - Environment variable HO_BUCKET was set autoamtically as xxxxxxxxxxxx-handoff-test
INFO - 2020-08-06 03:34:47,560 - botocore.credentials - Found credentials in shared credentials file: ~/.aws/credentials
INFO - 2020-08-06 03:34:47,602 - handoff.services.cloud.aws.s3 - GET s3://xxxxxxxxxxxx-handoff-test/test-03-exchange-rates/last/files
If there is no other task in the same resource group, we can delete the bucket, too:
> handoff -l warning -p 03_exchange_rates cloud delete_bucket
WARNING - 2020-08-06 03:34:49,044 - handoff.config - This will only delete the CloudFormation stack. The bucket xxxxxxxxxxxx-handoff-test will be retained.
The previous command only deleted the CloudFormation stack, but not the bucket itself. Here is how to delete all the files in s3://xxxxxxxxxxxx-handoff-test bucket. This cannot be reversed:
> aws s3 rm --recursive s3://xxxxxxxxxxxx-handoff-test/
Here is how to delete s3://xxxxxxxxxxxx-handoff-test bucket. The bucket must be empty. This cannot be reversed:
> aws s3 rb s3://xxxxxxxxxxxx-handoff-test
Now delete singer_exchange_rates_to_csv repository from ECR. This cannot be reversed. –force option will ignore that we still have images in the repository.
> aws ecr delete-repository --repository-name singer_exchange_rates_to_csv --force
If you had created a role and want to delete it, do:
> handoff -l warning -p 03_exchange_rates cloud delete_role
That’s all!