GitHub - toludoyin/datafam-birthday: ETL pipeline to fetch data from excel and create a calender dashboard in Tableau that tracks birthdays.

toludoyin / datafam-birthday Public

Notifications You must be signed in to change notification settings
Fork 0
Star 1

ETL pipeline to fetch data from excel and create a calender dashboard in Tableau that tracks birthdays.

1 star 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 36 Commits
dags		dags
Dockerfile		Dockerfile
README.md		README.md
docker-compose.yaml		docker-compose.yaml
requirements.txt		requirements.txt

Repository files navigation

Datafam Birthday

Objective

Build an ETL pipeline that process data in batch: periodically(daily)

Using the DataFam birthday data collected from the community.
Create a pipeline to put data in a database
Transform the data
Build a calender dashboard to visualize the data

Technologies

Data Ingestion and Transformation : Python
Workflow Orchestration : Apache Airflow
Database : OLTP(PostgreSQL)
Visualization : Tableau

Reference

Follow doc to set up docker https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html
Gspread doc https://docs.gspread.org/en/v6.0.0/user-guide.html, https://docs.gspread.org/en/latest/oauth2.html

Airflow: port 8282

Pgadmin: port 8001

Postgres db host on Airflow: host.docker.internal

About

ETL pipeline to fetch data from excel and create a calender dashboard in Tableau that tracks birthdays.

Report repository

Releases

No releases published

Packages

No packages published

Languages