Workflow

Extract data from https://date.nager.at/Api API - A public holiday dataset, transform the data in a readable format and load into a Postgresql database for accessibility, scheduled the flow to run and update table at 09:00, Sunday through Saturday with Airflow.

Environment Setup

1. Install Docker from https://docs.docker.com/compose/install/
2. Deploy Airflow on Docker Compose https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html
3. Setup PgAdmin, Postgres, and Airflow credentials for logins

Initialize Database

$ docker compose up airflow-init

Start Services

$ docker compose up

PgAdmin Server

1. Register server
2. Setup connection and input credentials.

Airflow Server

1. Connect to postgres db in airflow. Admin -> Connections