The Data Engineering Cookbook
-
Updated
Jul 21, 2025 - Python
#
data-engineer
Here are 80 public repositories matching this topic...
One framework to develop, deploy and operate data workflows with Python and SQL.
python data-science data sql database pipeline etl analytics snowflake data-warehouse data-structures data-engineering dataops warehouse elt data-pipelines data-engineer trino data-lineage data-engineering-pipeline
-
Updated
Aug 19, 2025 - Python
A Data Engineering project. Repository for backend infrastructure and Streamlit app files for a Premier League Dashboard.
python go docker bigquery google-cloud data-visualization data-pipeline data-engineer firestore prefect cloud-run streamlit
-
Updated
May 25, 2024 - Python
😎 A curated list of awesome DataOps tools
-
Updated
Jul 11, 2025 - Python
datacamp Data Engineer with Python course. 73 hours/ 19 Courses /2 Skill Assessments
-
Updated
Nov 29, 2022 - Python
Data Engineering Project with Hadoop HDFS and Kafka
python docker data kafka hadoop docker-compose data-engineering kafka-consumer hdfs kafka-producer hdfs-dfs hadoop-filesystem data-engineer hadoop-hdfs hdfs-client python-hdfs-client pipline data-engineering-pipeline kafka-ui kafkaui
-
Updated
Nov 4, 2023 - Python
Crawls sites, to find new content and scrap it
-
Updated
Apr 20, 2021 - Python
Code, Examples, Templates and Scripts for DataWorksSummit 2017 Sydney Talk
-
Updated
Sep 19, 2017 - Python
-
Updated
May 22, 2023 - Python
A data engineering platform for maintaining a data ecosystem to support self-driving cars research.
-
Updated
Dec 8, 2022 - Python
Wraps the DB by opening a REST API for storing and retrieving documents info & recommendations
-
Updated
Jul 22, 2018 - Python
A comprehensive data pipeline leveraging Airflow, DBT, Google Cloud Platform (GCP), and Docker to extract, transform, and load data seamlessly from a staging layer to a data warehouse and data mart.
-
Updated
Dec 1, 2024 - Python
End-to-end data engineering processes for the NIGERIA Health Facility Registry (HFR). The project leveraged Selenium, Pandas, PySpark, PostgreSQL and Airflow
python data-science airflow data-mining apache-spark pyspark postgresql-database elt data-engineer data-ingestion data-warehousing database-design selenium-python etl-pipeline erdiagram data-engineering-pipeline etl-automation airflow-dags nigerian-data data-warehouse-architecture
-
Updated
Sep 1, 2022 - Python
Repositório dedicado ao desafio do hackathon de engenharia de dados A3 Data Challenge Woman
-
Updated
Sep 30, 2021 - Python
Docker powered starter for geospatial analysis of lightning atmospheric data.
docker jupyter sqlite csv-files machine-learning-algorithms databases pandas python3 data-warehouse orchestrator data-engineer spatialite clustering-analysis data-engineering-pipeline noaa-weather streamlit-dashboard
-
Updated
May 16, 2025 - Python
The Real-time Ecommerce Data Collection and Processing project empowers businesses with real-time insights by efficiently extracting, processing, and storing ecommerce data from multiple sources. Combining Golang and Python, this cutting-edge solution streamlines data handling from diverse ecommerce websites.
python go golang data-science firebase google database storage dependency-injection gcp google-cloud pubsub web-scraping beautifulsoup datastore data-pipeline data-engineer solid-principles firestore
-
Updated
Jan 18, 2025 - Python
python aws lambda aws-lambda dynamodb s3 s3-bucket stream-processing datapipeline iam-policy data-engineer iam-role iam-credentials aws-event
-
Updated
May 28, 2022 - Python
This project designs and implements an ETL pipeline using Apache Airflow (Docker Compose) to ingest, process, and store retail data. AWS S3 acts as the data lake, AWS Redshift as the data warehouse, and Looker Studio for visualization. [Data Engineer]
-
Updated
Jul 7, 2025 - Python
IGTI MBA Engenharida de dados - Bootcamp Engenheiro de Dados Cloud - Desafio final
kubernetes airflow spark pyspark data-engineering data-engineer igti airflow-dags engenharia-de-dados
-
Updated
Apr 1, 2022 - Python
UserInsight-Streaming-Data-Pipeline is a real-time pipeline that ingests API data into Kafka, processes it with Spark, stores it in S3, and uses AWS Lambda to load it into Redshift. The data is then used to create a dashboard in Looker. [Data Engineer]
-
Updated
Jul 7, 2025 - Python
Improve this page
Add a description, image, and links to the data-engineer topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the data-engineer topic, visit your repo's landing page and select "manage topics."