handoff.cloud¶
Deploy configurable unix pipeline jobs serverlessly.
NEW:
- handoff v0.2.0b4 patch release fixes HttpError 403 on
workspace install
command. Please runpip install -U handoff
- Monitor metrics with Grafana
What is it?¶
handoff (repository) is a framework for executing unix-pipeline processes serverlessly. It separates the code and run-time configurations, making the deployment faster, more flexible, and more secure.
handoff helps the teams who want to:
- Automate the ETL/ELT process (earthquake data ELT example)
- Massively web-crawl and collect data (market & education data example)
- Relieve the pain of managing Apache Airflow Worker nodes.
…and more!
handoff is an open-source (APLv2) project sponsored by ANELEN.
How it works¶
handoff was originally designed to deploy a single-line ETL process like singer.io on AWS Fargate, and it was extended to run any commands and program.
In handoff.cloud framework, the configurations are stored and retrieved from a secure parameter store such as AWS Systems Manager Parameter Store. This avoids storing sensitive information or configurations that changes frequently on the Docker image.
Supporting files for commands can also be fetched from a cloud storage such as AWS S3. The artifacts, the files produced by each command can also be stored in the cloud storage.
This repository also contains AWS Cloudformation templates and deployment scripts to support the workflow from local development to the production deployment.
Get started easily¶
Check out the super quick example
Contents¶
Copyright 2020- Anelen Co., LLC.