handoff.cloud

Deploy configurable unix pipeline jobs serverlessly.

NEW:

• handoff v0.2.0b4 patch release fixes HttpError 403 on workspace install command. Please run pip install -U handoff
• Monitor metrics with Grafana

What is it?

handoff (repository) is a framework for executing unix-pipeline processes serverlessly. It separates the code and run-time configurations, making the deployment faster, more flexible, and more secure.

handoff helps the teams who want to:

• Automate the ETL/ELT process (earthquake data ELT example)
• Massively web-crawl and collect data (market & education data example)
• Relieve the pain of managing Apache Airflow Worker nodes.

…and more!

handoff is an open-source (APLv2) project sponsored by ANELEN.

How it works

handoff was originally designed to deploy a single-line ETL process like singer.io on AWS Fargate, and it was extended to run any commands and program.

In handoff.cloud framework, the configurations are stored and retrieved from a secure parameter store such as AWS Systems Manager Parameter Store. This avoids storing sensitive information or configurations that changes frequently on the Docker image.

Supporting files for commands can also be fetched from a cloud storage such as AWS S3. The artifacts, the files produced by each command can also be stored in the cloud storage.

This repository also contains AWS Cloudformation templates and deployment scripts to support the workflow from local development to the production deployment.

Get started easily

Check out the super quick example

---

Contents

• Quick Start
  • Install
  • A Super Quick Example
  • A Guided Tour
• Essential Commands at a Glance
• A Guided Tour
  • Running a task locally
  • Virtual environment and install
  • Setting up an AWS account and profile
  • Running task locally with remotely stored configurations
  • Building, running, and pushing a Docker image
  • Deploying on AWS Fargate
  • Scheduling a task on Fargate
  • Cleaning up
• Monitorig the process with Grafana
  • Grafana for AWS CloudWatch logs
• FAQs
  • Q. How do I configure project.yml so it installs a Python command from a Github repository?
• Contributing
  • Types of Contributions
  • Submit a pull request through the GitHub website.
  • Deploying
• Contributor Code of Conduct
• History
  • 0.2.0b4 (2020-08-18)
  • 0.2.0b3 (2020-08-10)
  • 0.2.0b2 (2020-08-06)
  • 0.2.0b1 (2020-08-05)
  • 0.2.0b0 (2020-08-05)
  • 0.1.2-alpha (2020-07-17)
  • 0.1.1-alpha (2020-07-16)
  • 0.1.0-alpha (2020-07-16)

---

Copyright 2020- Anelen Co., LLC.