ActiveBig Data
A curated list of awesome ETL frameworks, libraries, and software.
This page lists names, links and short descriptions. The original list on GitHub is the source and belongs to its authors.
"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…
"an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes."
"Dagster is a data orchestrator for machine learning, analytics, and ETL. It lets you define pipelines in terms of the data flow between reusable, logical components, then test locally and run anywhere. With a unified view of pipelines and the assets they produce, Dagster can schedule and…
"a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in."
"a workflow orchestration framework for building resilient data pipelines in Python."
"a scalable and reliable runtime for durable function executions called Temporal Workflow Executions."
"an open-source pure-Python workflow engine that lets people write better pipelines."
"an open source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data."
"A lightweight, comprehensive batch framework designed to enable the development of robust batch applications that are vital for the daily operations of enterprise systems."
"a Python library for pulling data out of HTML and XML files."
"an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well."
A wrapper around SQLAlchemy that simplifies database operations (including upserting).
"enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications."
"an open-source Python library that loads data from various, often messy data sources into well-structured datasets."
"a Python library for validating, documenting, and profiling your data to maintain quality and improve communication between teams about data and data pipelines."
"helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does."
"Iterative JSON parser with Pythonic interfaces."
"a CLI tool to copy data between any databases with a single command seamlessly."
"a set of tools to provide lightweight pipelining in Python."
"the most feature-rich and easy-to-use library for processing XML and HTML in the Python language."
"the declarative code-first data integration engine."
"Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more."
"Parse strings using a specification based on the Python format() syntax."
"a general purpose Python package for extracting, transforming and loading tables of data."
"A jquery-like library for python."
"the Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL."
"a general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything."
"A functional standard library for Python."
"a parallel bulk data loader that helps data transfer between various storages, databases, NoSQL and cloud services."
"lets you define and run high-quality ETL jobs using Ruby."
"a simple, flexible, and powerful SQL database access toolkit for Ruby."
"a cloud asset inventory built for platform teams. Sync your cloud infrastructure metadata into your data warehouse, powering insights and automation."
"provides parallelized processing of multi-stage, language-agnostic pipelines with data versioning and data lineage tracking."
"a declarative data streaming and integration tool with 300+ pre-built connectors, configured via YAML."
"Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases."
"combines data preparation, data blending, and analytics — predictive, statistical, and spatial — in a visual workflow designer."
"enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS."
"a serverless data integration service that makes it easy for analytics users to discover, prepare, move, and integrate data from multiple sources."
"Fully managed, cloud-native data integration platform."
"automates data movement from disparate sources into your destination."
"Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines."
"a no-code data movement platform that is usable by your most technical as well as your non-technical and business users."
"A fully managed, serverless data integration service that helps you visually integrate data sources with more than 90 built-in, maintenance-free connectors."
"Stitch is a cloud-first, open source platform for rapidly moving data. A simple, powerful ETL service, Stitch connects to all your data sources – from databases like MySQL and MongoDB, to SaaS applications like Salesforce and Zendesk – and replicates that data to a destination of your choosing."
"a unified programming model for Batch and Streaming data processing."
"a framework and distributed processing engine for stateful computations over unbounded and bounded data streams."
"Change data capture for a variety of databases."
"a tool for scalably and reliably streaming data between Apache Kafka and other systems. It makes it simple to quickly define connectors that move large collections of data into and out of Kafka."
"a fast and general-purpose cluster computing system. It provides high-level APIs in Scala, Java, and Python that make parallel jobs easy to write, and an optimized engine that supports general computation graphs. It also supports a rich set of higher-level tools including Shark (Hive on Spark),…
"a rich, web-based interface for designing, controlling, and monitoring a dataflow."
"Use Cask Data Application Platform to visually build and manage data applications in hybrid and multi-cloud environments."
An ETL tool for extracting data from source systems, transforming it, and loading it into target systems using a visual mapping and workflow designer.
"a component of the Microsoft SQL Server database software that can be used to perform a broad range of data migration tasks."
"Free and open fair-code licensed node based Workflow Automation Tool. Easily automate tasks across different services."
"a graphical ETL tool for designing data integration workflows using a drag-and-drop interface, also known as Kettle."
Joe Reis & Matt Housley's tool-agnostic overview of the data engineering lifecycle, including the ETL-to-ELT shift (2022).
Chad Sanderson on formalizing schema and quality guarantees between data producers and consumers.
Why cheap cloud warehouse compute flipped the ETL paradigm to ELT.
igorbarinov/awesome-data-engineering
A curated list of data engineering tools for software developers
oxnr/awesome-bigdata
A curated list of awesome big data frameworks, ressources and other awesomeness.
awesomedata/awesome-public-datasets
A topic-centric list of HQ open datasets.
ohenley/awesome-ada
A curated list of awesome resources related to the Ada and SPARK programming language
briatte/awesome-network-analysis
A curated list of awesome network analysis resources.
bytewax/awesome-public-real-time-datasets
A list of publicly available datasets with real-time data maintained by the team at bytewax.io