I have been working on the batch processing aspects lately, and I found many topics on this idea. One of the ways was to use Luigi, a Spotify maintained project. The main problem is that there aren't many good blogs about how to implement batch processing in a simple and easy-to-understand way. So, I thought, why not write one myself that can be helpful for many people out there? Let's buckle up for a ride through Luigi, a workflow orchestration tool. We won't be seeing every feature it has, but we will be looking into the main feature, which is batch processing. I won’t be showing how to implement a batch process in Luigi for now let’s focus on the concepts and features. We will be doing that in the next blog😉.
What is Luigi?
It's a Python-based batch processing framework built and maintained by Spotify. You can check out their GitHub repo. They use it for their workflows, which led to its development and adoption by many companies. It was mainly used to automate the tasks involved in their music streaming service and build reproducible processes. It's open-sourced by Spotify, so you can contribute to it as well. Luigi is a workflow orchestration framework, meaning it gives you full control over the batch process, including a nice dashboard to view and control the tasks. There are many more features available in this; let’s first check the architecture of Luigi.
Architecture
There are some essential architectural designs and concepts everyone should know about before using Luigi. Let’s discuss a few of the main topics here.
This is a simple flow showing how the Luigi architecture looks. Luigi is a workflow orchestration framework, which means it runs as a central server that manages all the client or local servers. There are two tasks for the central server:
There is also a web interface to manage and schedule the job that you want to run.
Tasks
Tasks are the basic building blocks in Luigi. The important thing is that they are defined as Python classes that inherit from Luigi.Task. They encapsulate the logic for a specific job and include three essential methods:
requires(): Specifies dependencies on other tasks, ensuring they are executed before the current task.
run(): Contains the code to execute the task, defining the business logic or operations performed.
output(): Returns one or more target objects, representing the task's output, which is typically a file or database entry.
Here’s a Luigi example that demonstrates a simple ETL pipeline.
We have a pipeline that:
Extracts data from a text file.
Transforms the data by converting all text to uppercase.
Loads the transformed data into a final output file.
In this Luigi pipeline, each step is a task, which is a piece of work that depends on other tasks. The first task, ExtractData, creates a text file (extracted_data.txt) with a list of sample words. The second task, TransformData, depends on ExtractData, so it runs only after the first task is done. It reads the file, changes all words to uppercase, and saves them in transformed_data.txt. The third task, LoadData, depends on TransformData and waits for it to finish before it runs. It reads the transformed file, adds "Loaded:" to each word, and writes the result to final_data.txt. Luigi uses the requires() method to manage dependencies, making sure tasks run in the right order. The output() method shows where each task's result will be saved, and the run() method has the actual steps to process the data. Running the script with luigi.build([LoadData()], local_scheduler=True) starts the whole pipeline, running tasks one after the other while keeping their dependencies in check.
Targets
A target in Luigi represents the output of a task and acts as a checkpoint to determine if the task needs to run. Before executing a task, Luigi checks whether the target already exists. If it does, Luigi skips the task, assuming it has been completed successfully. This prevents unnecessary recomputation and makes pipelines more efficient and fault-tolerant.
Types of Targets in Luigi
LocalTarget – Used for files stored locally.
S3Target – Stores data in Amazon S3.
GCSFileSystemTarget – Saves files in Google Cloud Storage.
PostgresTarget / MySqlTarget – Used for database records.
The example is shown in task section.
Dependencies
In Luigi, dependencies define the relationships between tasks in a data pipeline. They ensure that each task runs only after its required tasks are completed, maintaining the correct execution order. This helps streamline workflows, prevent errors from out-of-order execution, and ensure data consistency.
How Dependencies Work
Dependencies are defined using the requires() method in a Luigi task. A task lists its required tasks, and Luigi automatically ensures that those tasks are completed before running the dependent task.
Dependency Graph
In Luigi, dependency graphs visually represent the relationships between tasks in a pipeline. They help in understanding task execution order, identifying bottlenecks, and debugging failures.
How Luigi Builds Dependency Graphs
Each Luigi task defines dependencies using the requires() method. When a pipeline runs, Luigi constructs a Directed Acyclic Graph (DAG) where:
Luigi ensures that each task executes only after all its dependencies are completed.
Scheduling
Luigi provides flexible scheduling options to automate workflow execution. Tasks can be run manually, on a schedule, or triggered by dependencies.
Running Tasks Manually
You can execute a Luigi task manually using the command line:
luigi --module my_tasks TaskName --local-scheduler
This runs the task immediately without relying on the central scheduler.
Using luigid for Centralized Scheduling
For managing large workflows, Luigi offers a central scheduler (luigid). It tracks task status and ensures dependencies run in the correct order. Start the scheduler with:
luigid --background
Then, run tasks using:
luigi --module my_tasks TaskName --scheduler-host localhost
This enables centralized monitoring and parallel execution of tasks.
Automating with Cron Jobs
Luigi does not have a built-in scheduler like Apache Airflow but can be integrated with cron jobs for periodic execution. Example cron entry for running a Luigi task every day at midnight:
0 0 * * * luigi --module my_tasks TaskName --scheduler-host localhost
Features of Luigi
Luigi is a Python-based workflow management system designed to build and automate complex data pipelines. It ensures tasks run in the correct order and handles dependencies efficiently. Here are some of its key features:
Task Dependency Management
Luigi automatically tracks and resolves dependencies between tasks using a Directed Acyclic Graph (DAG). This ensures that each task runs only when its dependencies are met.
Automatic Checkpoints with Targets
Each task in Luigi produces an output target (like a file or database entry). If a target exists, Luigi skips re-execution, preventing redundant processing and improving efficiency.
Built-in Scheduler & Worker System
Luigi comes with a central scheduler (luigid) that allows tasks to be monitored, retried, and executed in parallel using workers. This makes it great for handling large-scale data pipelines.
Scalability & Parallel Execution
Luigi can run tasks in parallel using multiple workers, making it ideal for big data processing and distributed computing.
Integration with Various Storage & Compute Systems
Luigi supports multiple storage and database systems, including:
Local file system (LocalTarget)
Amazon S3 (S3Target)
Google Cloud Storage (GCSFileSystemTarget)
Hadoop & HDFS (HdfsTarget)
Databases like MySQL & PostgreSQL (PostgresTarget, MySqlTarget)
Visualization & Monitoring
Luigi provides a web-based UI that allows users to monitor running jobs, view logs, and visualize task dependencies. Run the scheduler with:
luigid --background
and access the UI at http://localhost:8082.
Fault Tolerance & Retry Mechanism
Luigi automatically retries failed tasks and provides logging to help diagnose errors. This ensures robustness in long-running workflows.
Extensibility & Customization
Luigi allows you to create custom targets, parameterized tasks, and extend existing functionality, making it highly adaptable to different data engineering needs.
Support for Incremental Processing
By using checkpoints (targets), Luigi only processes new or changed data, reducing computational overhead for large datasets.
Problems with Luigi
No Native Support for Real-Time Streaming
Luigi is designed for batch processing, meaning it lacks support for real-time or event-driven workflows. If you need real-time data processing, tools like Apache Airflow, Apache Kafka, or Flink may be better suited.
Requires Manual Error Handling & Retries
While Luigi provides a retry mechanism, it does not handle failures automatically as efficiently as some other orchestration tools. You must explicitly define retry logic in your tasks to avoid failures stopping the entire pipeline.
Limited Support for Dynamic Workflows
Luigi workflows are static, meaning the dependency graph is defined before execution. Dynamically adding tasks at runtime is difficult compared to Apache Airflow, which supports dynamic DAG creation using Python scripts.
Web UI is Basic & Lacks Advanced Features
The Luigi web UI is minimalistic and does not provide detailed logs, alerting, or advanced visualization. Other tools like Airflow offer more interactive UIs with better monitoring capabilities.
Dependency Management Can Get Complex
Large DAGs with many dependencies can become hard to manage. Unlike Airflow, Luigi does not provide a built-in way to visualize complex DAGs dynamically.
Limited Community & Ecosystem
Smaller community compared to Apache Airflow, Prefect. Fewer pre-built integrations for third-party services, requiring custom implementations for many use cases.
Higher Resource Usage Due to Process-Based Execution
Luigi uses processes instead of threads for task execution, which can be more resource intensive. This approach may lead to higher memory and CPU usage compared to threading, which is more lightweight.
Conclusion
Luigi is a Python-based workflow management system developed by Spotify. The Spotify team had stopped development of the Luigi, and it didn’t get many fixes for a year. But now it has been actively maintained. Although, the Spotify has stopped using it you can find the detailed blog here.
I personally believe Luigi can be useful for small to medium-scale tasks. There are many frameworks and options available, and Luigi may not have a large community like Spring Batch. However, with each fix, Luigi is improving and will continue to get better over time. The main thing for me is that it is the simplest library for batch processing in Python, which is a language used by many beginners.
For large-scale tasks, you might consider using more robust options like Apache Flink or Apache Spark, which are designed to handle big data processing efficiently. If you prefer a lighter solution, Spring Batch could be a good choice, as it offers a more streamlined approach without being too resource-intensive. These alternatives provide greater scalability and performance for handling complex workflows and data processing needs.
References
