What is Batch Processing?

Batch processing is handling a large amount of data in batches. Batches can be time ranges or numbers of records, etc. We use batch processing when the amount of data is finite and known and can be processed with some delay. As per the Spring documentation, batch processing is the processing of a finite amount of data in a manner that does not require external interaction or interruption.

For example, suppose we open a Google Form to retrieve user feedback for your mobile application. Then you download the responses as a CSV file. Now you have the CSV file and want to convert the data from CSV to a database table. Here, you need batch processing to efficiently use the resources without errors or omissions. As you're processing a huge amount of data, it will take time to complete.

Why choose batch processing? One might ask. The main advantage of batch processing is that it handles a finite amount of data, which can be very large, even reaching millions, billions, or trillions of records. Depending on the requirements, you can design an approach that uses resources efficiently.

What is Spring Batch?

Spring Batch is a framework that facilitates batch processing by providing various features such as parallel processing, fault tolerance, horizontal scalability, and ease of use. It was initially developed through a collaboration between Accenture and Interface21 (now SpringSource), the creators of the Spring Framework in 2007. At the end of March 2008, the Spring Batch 1.0.0 release was made available to the public. It was one of the initial concepts for batch processing in Java.

Now, Spring Batch is being developed by VMware, the same company that develops Spring Boot. The latest version, Spring Batch 5, introduces many new features. Development is progressing rapidly, with new releases every six months. Spring Batch provides various features, making it one of the best batch-processing frameworks out there. Let’s look at some of the features.

Features of Spring Batch

• Flexibility: Spring Batch provides excellent flexibility. Whatever batch processing you need to do, Spring Batch can support it. Run batches in parallel on a single device or distribute them across multiple devices; Spring Batch supports it all.

• Fault Tolerance: The spring batch has various types of fault tolerance mechanisms such as the retry, skip, restart ability etc. which are essential for the perfect batch processing framework.

• Transaction Management: Your job needs transactions in a database. Worry not, Spring Batch can automatically start transactions. If it fails at any point, it will automatically roll back, so you don’t have to worry about it.

• Chunk-based processing: Spring Batch supports chunk processing, meaning it doesn’t commit or save until a certain number of records or items are processed. This can significantly improve performance when storing data in a database or file.

• Job Management: Spring Batch can handle batch jobs efficiently. It stores information such as the number of items processed, the current item number, etc., in its database. In case of a crash, it can recover and continue from where it left off.

• Support for various resources like databases, files and more: Since it is part of Spring Boot, which already supports many databases out of the box, Spring Batch inherits that capability. On top of that, it also supports many custom databases, files, and other resources.

Java isn’t your cup of tea? Why not use Python for batch processing with Luigi? It's a Python-based framework for workflow orchestration developed by Spotify. Get concepts of it in my blog link below.

Architecture of Spring Batch

The Spring batch architecture contains 3 high-level components. The architecture looks like this:

The application component contains the custom code and configuration used to build your batch processes. Business logic services and all the configurations of how you organize the job are considered the application.

The Batch Core contains the core runtime classes necessary to launch and control a batch job. It includes implementations for JobLauncher, Job, and Step. Both Application and Core are built on top of a common infrastructure.

The infrastructure contains all the readers, writers, and different services. Their core also uses such elements. The application layer is connected to the infrastructure layer due to the custom implementations that the developer creates for the read and write.

Core Components of Spring Batch

Job

A single execution unit that summarises a series of processes for batch application in Spring Batch. Multiple jobs can exist in a single application. Job is an entity that comprises the details of the entire batch process. From where to start to what comes next, every detail is contained within a Job entity. A job can contain many N number steps.

A batch job in Spring Batch is represented by the Job interface provided by the spring-batch-core dependency:

public interface Job {
    String getName();
    void execute(JobExecution execution);
}

At a fundamental level, the Job interface requires that implementations specify the Job name (the getName() method) and what the Job is supposed to do (the execute method).

The execute method gives a reference to an JobExecution object. The JobExecution represents the actual execution of the Job at runtime. It contains several runtime details, such as the start time, the end time, the execution status, and so on.

In Spring Batch, a job is defined as a Spring bean, allowing Spring Batch to manage everything from creation to termination. The definition kind of looks like this.

@Bean
public Job newsletterJob(JobRepository jobRepository, Step step1) {
    return new JobBuilder("newsletter-job", jobRepository)
            .start(step1)
            .build();
}

Step

A Step is a domain object that encapsulates an independent, sequential phase of a batch Job. It contains all of the information necessary to define a unit of work in a batch Job. The job can contain any number of steps. This step can be individually parallelised, fault-tolerant and restartable.

A Step in the Spring Batch is represented by the Step interface provided by the spring-batch-core dependency:

public interface Step {
  String getName();
  void execute(StepExecution stepExecution) throws JobInterruptedException;
}

Similar to the Job interface, the Step interface requires, at a fundamental level, an implementation to specify the step name (the getName() method) and what the step is supposed to do (the execute method).

The execute method provides a reference to an StepExecution object. The StepExecution represents the actual execution of the step at runtime. It contains several runtime details, such as the start time, the end time, the execution status, and so on. Spring Batch stores this runtime information in the metadata repository, similar to the JobExecution, as we have seen previously.

JobParameters

Job parameters are values passed to the job, allowing the entire job process to access them. One of the key features of Spring Batch is its ability to determine whether a job is new or if it is running a previous job, and this is determined using the values of job parameters passed. The values passed as job parameters will be hashed and stored. When a new job is run, the values of job parameters are compared to check whether they match any old job.

 JobParameters jobParameters = new JobParametersBuilder()
                .addString("filePath", "input/data.csv")
                .addLong("time", System.currentTimeMillis()) // make the job unique every time it runs
                .toJobParameters();

Access to the job parameters is possible both in the JobExecution context and the Stepexecution context.

JobLauncher

As we have created a job with steps now we want some way to invoke the job on startup, this can be done using the JobLauncher interface.

public interface JobLauncher {
   JobExecution run(Job job, JobParameters jobParameters)
          throws
             JobExecutionAlreadyRunningException,
             JobRestartException,
             JobInstanceAlreadyCompleteException,
             JobParametersInvalidException;
}

The run method is designed to launch a given Job with a set of JobParameters.

JobExecutionContext

It stores the different details about the job execution that are used by Spring Batch to keep track of the different parts of the job execution. The JobExecutionContext is a key-value pair storage, where Spring Batch puts some keys, and it is available to the entire job process to access. Even custom data can be added and accessed later in the job execution. It can be used to share data between steps.

JobExecution jobExecution = jobLauncher.run(job, jobParameters);
JobExecutionContext jobExecutionContext = jobExecution.getExecutionContext();
jobExecutionContext.put("jobStartTime", new Date());

StepExecutionContext

It is the same as the JobExecutionContext but only local to a step execution. There you may say each step will have its step execution context. It is also a key value store which spring batch stores keys related to the step execution. Even custom data can be added and accessed later in the step execution. It can be used to share data between different parts of the same steps.

StepExecution stepExecution = jobExecution.getStepExecutions().iterator().next();
StepExecutionContext stepExecutionContext = stepExecution.getExecutionContext();
stepExecutionContext.put("currentItem", currentItem);

JobRepository

Spring batch is stateful, it means that it stores details of the job execution and the step execution. This is denoted by the JobExecution and StepExecution. Spring batch supports different types of databases for storing its state, relational databases such as MySQL, and PostgreSQL and non-relational databases such as MongoDB. The relations between different tables can be seen below.

Each table is crucial for the spring batch to manage running job states. Each of them serves as metadata storage for different components of the spring batch environment. Let’s look into each table.

Job_Instance

The Job_Instance table stores the unique job runs. I have previously mentioned that the job parameters are used to uniquely identify a job. This is the table used for that purpose. A job can be run again if something fails and you want to restart the old job. The job parameters are hashed and stored as a job key in this table. The job name and the job key are together used to identify if the job is new or not. For each unique run of the job, the spring batch creates a new job instance.

Job_Execution

The Job_Execution table stores the details of each job run. Even if a job is restarted, it is stored as a new row, and the details are kept separate. The Job_Execution table consists of various columns that store details about the job execution, such as start_time, end_time, and status. The job execution, along with the job instance, is checked to see if the job is complete or failed. If it has failed, it can be restarted. Completed jobs cannot be restarted by default; you will need to manually configure that.

Job_Execution_Params

The Job_Execution_Params table stores the job parameters. There can be multiple job parameters for a single job. Sometimes you want to pass a value as a job parameter but don’t want that value to uniquely identify the job. In that case, you have the option to set the identifying to false, and it will be reflected in this table.

Job_Execution_Context

Used to store the JobExecutionContext data. The stored data is in Java serialized format, which is then encoded in base64. The base64 encoding is used to prevent character issues that can occur during storage.

Step_Execution

The Step_Execution stores the details about the execution of different steps. Each step execution is stored as a new row. It stores different details including the start_time, end_time, Read_count, Write_count etc. done by the step.

Step_Execution_Context

It stores the StepExecutionContext data. The stored data is in Java serialized format, which is then encoded in base64. The base64 encoding is used to prevent character issues that can occur during storage.

Types of Steps

There are many types of steps available each of them is very useful in many different scenarios. The most used step type is the Tasklet step. Anyway we will look into each of them.

TaskletStep

The Tasklet step is a straightforward step that calls the Tasklet's execute method. The Tasklet is a functional interface with an execute method that the developer must implement. Spring Batch will automatically call the Tasklet's execute method repeatedly until it returns a finished status. By default, Spring Batch runs the Tasklet step inside a transaction, ensuring data integrity and consistency. The Tasklet allows us to implement everything from reading to writing, including fault tolerance mechanisms like retry and skip.

The most famous implementation of the Tasklet is the chunk-oriented Tasklet, which will read, process, and write the data in chunks, which is very neat when it comes to processing huge amounts of data. Only for the chunk-oriented step does Spring Batch provide various components like fault tolerance mechanisms such as retry, skip, and restart ability. Even parallel processing mechanisms are only available for the chunk-oriented step. We will look into this step in detail in future blogs.

Partition Step

The partitioning step divides data into different parts based on specific criteria. It's like cutting a pizza into slices, where each slice is a partition. The criteria are set by the partition handler. Partitioning is done according to the criteria you provide. By default, Spring Batch offers only one implementation out of the box.

Flow Step

The flow step is used to define the order in which steps should be executed. It allows for conditional execution of steps. For example, if the first step is completed successfully, the next step can be executed. If the first step fails, you can skip the second step and execute the third one instead. This can be managed using the flow step.

@Bean
public Job conditionalJob() {
    return jobBuilderFactory.get("conditionalJob")
        .start(step1())
        .on("COMPLETED")
        .to(step2()) // If step1 succeeds, go to step2
        .from(step1())
        .on("FAILED")
        .to(step3()) // If step1 fails, go to step3
        .end() // End the job
        .build();
}

JobStep

It is a step that calls another job to execute. You can have multiple job definitions, and one job can have a step that calls another job. We will look into it in future blogs for now it is all you need to know.

@Bean
public Step childStep() {
    return new StepBuilder("childStep", jobRepository)
        .tasklet(
            (contribution, chunkContext)
                -> {
                System.out.println("Executing Child Job Step");
                return RepeatStatus.FINISHED;
            },
            transactionManager)
        .build();
}

@Bean
public Job childJob() {
    return new JobBuilder("childJob", jobRepository)
                         .start(childStep())
                         .build();
}

// Parent Job: Includes a JobStep to execute the child job
@Bean
public Step parentStep(Job childJob, JobLauncher jobLauncher) {
    JobStep jobStep = new JobStep();
    jobStep.setJobLauncher(jobLauncher);
    jobStep.setName("parentStepExecutingChildJob");
    return jobStep;
}

@Bean
public Job parentJob(Step parentStep) {
    return new JobBuilder("parentJob", jobRepository)
                          .start(parentStep)
                          .build();
}

Basic Job Flow

The JobLauncher will start the job. Before it begins, a new row will be created in the Job_Instance table if it is a new job, and the job parameters will be added to the Job_Execution_Params table. Then, the Job execution will start, and a new row will be added to the Job_Execution table. The job doesn't need to be unique; for every job run, a new row will be added to the Job_Execution table. The job will call the Step, and a new row will be added to the Step_Execution table. Then, the step will execute the tasklet, and so on.

Launching a Job

There are several ways to launch a job, and we'll explore all of them in this section. Each method is used in different scenarios based on your purpose.

Automatic Job Launch on Startup

If we have only one job definition, we can configure Spring Batch to launch it on startup. However, if there are multiple job definitions, this approach can cause issues, so it's better not to use it. To enable this, simply set the property to true in application.properties or application.yml.

spring:
  batch:
    job:
      enabled: true

If you have multiple job definitions then you can provide the name of the job you want to run on startup in application.properties or application.yml as below.

spring:
  batch:
    job:
      enabled: true
      name: newsletter-job

Using CommandLineRunner

It’s a good option if you want to run the job using the command line or a command. For this approach to work you have to know which class you have added the job configuration as you have to pass it to the CommandLineRunner.

$ java CommandLineJobRunner in.akashrchandran.nbt.config.BatchConfig

Using JobLauncher

There might be a need to run a Job based on our needs like we need to schedule it using the Quartz or trigger run when we get an API request etc. Then we can’t set it to run on startup or use the command line to run, the solution here is to use the Job launcher interface. You can autowire it using the spring dependency injection.

jobLauncher.run(job, new JobParameters());

You can use the JobLauncher anywhere and run the job based on your needs. I will show an example of running the job from a web container.

@Controller
public class JobLauncherController {

    @Autowired
    JobLauncher jobLauncher;

    @Autowired
    Job job;

    @PostMapping("/jobLauncher")
    public void handle() throws Exception {
        jobLauncher.run(job, new JobParameters());
    }
}

Code Example

We won't stop with just the basics. Let's try to apply what we've learned and create a simple application that sends newsletters. There will be a text file containing some random email IDs. Since the application is simple, we will implement it using the Tasklet we discussed earlier.

I will be using IntelliJ Ultimate IDE to code this project. However, you can use any IDE you like, such as Vscode, Vim, or even Notepad. It’s your choice.

Getting the dependencies

Head over to the Spring Initializer to get the dependencies for our application. We will require the dependencies I have selected below.

Let me clarify the dependencies needed. We require Spring Batch, of course. The Java Mail Sender is for sending emails, and H2 is for the in-memory database because I don't want to set up an external database for this simple project. I don't recommend using H2 in production. You should provide a database for the JobRepository. It can be almost any database provider, like MySQL, PostgreSQL, or even MongoDB.

Download the zip file and load it into your IDE, and then we can start coding.

Creating a BatchConfig class

We'll start by creating a class BatchConfig to store all the configurations for our batch process, including the Job and Step definitions. There is no need to keep the same name it’s just your choice. You have to annotate the class, to let the spring know that the class contains the configurations for our batch process.

@Configuration
public class BatchConfig {
// Batch config
}

1. Defining the Job

From what we've learned, we know we need to define the entire batch process as a Job. The job definition is simple since we will only have one step, which is sending the emails.

    @Bean
    public Job newsletterJob(JobRepository jobRepository, Step step) {
        return new JobBuilder("newsletter-job", jobRepository)
                .start(step)
                .build();
    }

Since it is a Spring bean, the parameters will be autowired for us, that's the magic of Spring. I won't explain that here, so let's move on to the next part.

2. Defining the Step

Next, we need to define the step and pass the tasklet, which we will write shortly, to execute.

    @Bean
    public Step newsletterStep(JobRepository jobRepository,
                               PlatformTransactionManager transactionManager,
                               Tasklet tasklet) {
        return new StepBuilder("newsletter-step", jobRepository)
                .tasklet(tasklet, transactionManager)
                .build();
    }

As we have only a single tasklet we are going to define, we can autowire that as well.

You can view the entire class code on GitHub, here is the link.

Configuration for Sending Emails

Java Mail Sender needs some setup before you can send emails, like your email address and password. I'll be using Gmail to send my emails, but you can use any other service.

For Gmail, you need to get an App Password. You can't use your regular Google account password for security reasons, so you will need to generate one. I'll provide a link to an article on how to obtain the App Password.

Now we have to add some properties to application.properties or application.yml whichever you use. I prefer the YAML file. So I will show that only, you can just use the same property keys application.properties and it should work.

spring:
  application:
    name: newsletter-batch-tasklet-async
  mail:
    properties:
      mail:
        smtp:
          auth: true
          starttls:
            enable: true
    host: smtp.gmail.com
    port: 587
    username: ${SPRING_MAIL_USERNAME}
    password: ${SPRING_MAIL_PASSWORD}

The ${SPRING_MAIL_USERNAME} and ${SPRING_MAIL_PASSWORD} are used here to get the values from the Environment Variables. You can set the environment variables. You can set the environment variables based on your operating system

# unix or linux based system
$ export SPRING_MAIL_USERNAME=example@example.com

# windows command prompt
$ set SPRING_MAIL_USERNAME=example@example.com

# windows poweshell
> $env:SPRING_MAIL_USERNAME = "example.org"

If you're using the IntelliJ IDE, you can set the environment variables in the Run Configurations.

Creating the Tasklet

To create a Tasklet step, we need to implement the Tasklet interface provided by Spring Batch. First, we will create a class named NewsletterTasklet and make it implement the Tasklet interface. After that, you will likely see an error indicating that you need to implement the methods of that interface, specifically the execute() method.

We need to annotate the class @Component to inform Spring that it is a Spring bean. Additionally, we will autowire JavaMailSender to send the mail. This recipientsFilePath is the path of the file that contains the emails. Using the @Value annotation so that we can define the values in the application.properties or applciation.yml.

@Component
@RequiredArgsConstructor
@Slf4j
public class NewsletterTasklet implements Tasklet {
    @Value("${newsletter.recipients.file}")
    private String recipientsFilePath;

    @Value("${newsletter.max.concurrent:10}")
    private int maxConcurrent;

    private final JavaMailSender mailSender;

    @Override
    public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
        // custom code to send the newsletter
    }
}

1. Reading emails from a text file

We will be reading the emails from a text file named spam.txt which I have already kept in the resource folder. So we need to define a method which reads the text file splits the lines and returns the emails as a list. We can just define a private method for the same.

    private List<String> readRecipientsFromFile() {
        try {
            Resource resource = new ClassPathResource(recipientsFilePath);
            if (resource.exists()) {
                try (Stream<String> lines = new BufferedReader(
                        new InputStreamReader(resource.getInputStream(), StandardCharsets.UTF_8))
                        .lines()) {
                    log.info("Successfully loaded recipients from resources: {}", recipientsFilePath);
                    return lines.filter(line -> line != null && !line.isBlank())
                            .toList();
                }
            } else {
                log.warn("Resource file '{}' does not exist", recipientsFilePath);
            }
        } catch (Exception e) {
            log.error("Error reading recipients from file '{}': {}", recipientsFilePath, e.getMessage(), e);
        }
        return Collections.emptyList();
    }

2. Sending an email to the email ID

Then we need a method to send email to the email ID we passed. We can create another private function for that as well.

    private void sendEmail(String to) {
        SimpleMailMessage message = new SimpleMailMessage();
        message.setFrom("yourEmail@domain.com");
        message.setTo(to);
        message.setSubject("Newsletter");
        message.setText("This is the content of the newsletter.");
        mailSender.send(message);
        log.info("Email sent successfully to {}", to);
    }

3. Implementing Asynchronous Email Sending

Finally, we need to code the execute method which will use the above techniques to read and send emails. We use the ExecutorService to call the send email function asynchronously. We can use many threads to send the message but it is always to limit the number of threads that are concurrently executing. We can do something like this

@Override
public RepeatStatus execute(
    StepContribution contribution, ChunkContext chunkContext) throws Exception {
    log.info("Starting NewsletterTasklet execution with async processing.");
    List<String> recipients = readRecipientsFromFile();
    int totalRecipients = recipients.size();
    ExecutorService executor = Executors.newFixedThreadPool(maxConcurrent);
    List<CompletableFuture<Void>> allFutures = new ArrayList<>();
    // Create async tasks for all recipients
    for (String recipient : recipients) {
        CompletableFuture<Void> future = CompletableFuture.runAsync(() -> {
            try {
                sendEmail(recipient);
            } catch (Exception e) {
                log.error("Failed to send email to {}: {}", recipient,
                    e.getMessage(), e);
            }
        }, executor);
        allFutures.add(future);
    }
    CompletableFuture.allOf(allFutures.toArray(new CompletableFuture[0])).join();
    executor.shutdown();
    try {
        if (!executor.awaitTermination(30, TimeUnit.SECONDS)) {
            executor.shutdownNow();
        }
    } catch (InterruptedException e) {
        executor.shutdownNow();
        Thread.currentThread().interrupt();
    }

    log.info("Finished NewsletterTasklet execution. Total emails sent: {}", totalRecipients);
    return RepeatStatus.FINISHED;
}

Executing the batch process

I have listed about 10 emails in my spam.txt file. In a real scenario, you would read these from a user database, but this example is simpler. Before executing the batch process we need to add some more properties to the application.yml.

I will be showing the job running on application startup. You can use any way to run the job, as I have shown in the launching the jobs part.

spring:
  batch:
    jdbc:
      initialize-schema: always # to create job repository tables.
    job:
      enabled: true # to run the job on applcaiiton startup.

newsletter:
  recipients:
    file: spam.txt # file name which contains the emails.
  max:
    concurrent: 5 # limiting the number of concurrent threads.

Now we can simply run the application. Ensure you have added the environment variables; otherwise, the application will display an error.

As you can see the emails have been sent successfully within 17 seconds. I could have increased the concurrent thread count and may have gotten better results but that is not what this blog is about.

You can view the entire code on my Github repository.

What is Luigi?

It's a Python-based batch processing framework built and maintained by Spotify. You can check out their GitHub repo. They use it for their workflows, which led to its development and adoption by many companies. It was mainly used to automate the tasks involved in their music streaming service and build reproducible processes. It's open-sourced by Spotify, so you can contribute to it as well. Luigi is a workflow orchestration framework, meaning it gives you full control over the batch process, including a nice dashboard to view and control the tasks. There are many more features available in this; let’s first check the architecture of Luigi.

Architecture

There are some essential architectural designs and concepts everyone should know about before using Luigi. Let’s discuss a few of the main topics here.

This is a simple flow showing how the Luigi architecture looks. Luigi is a workflow orchestration framework, which means it runs as a central server that manages all the client or local servers. There are two tasks for the central server:

• Ensure that two instances of the same task are not running simultaneously.

• Show a visualization of the tasks being run.

There is also a web interface to manage and schedule the job that you want to run.

Tasks

Tasks are the basic building blocks in Luigi. The important thing is that they are defined as Python classes that inherit from Luigi.Task. They encapsulate the logic for a specific job and include three essential methods:

• requires(): Specifies dependencies on other tasks, ensuring they are executed before the current task.

• run(): Contains the code to execute the task, defining the business logic or operations performed.

• output(): Returns one or more target objects, representing the task's output, which is typically a file or database entry.

Here’s a Luigi example that demonstrates a simple ETL pipeline.

We have a pipeline that:

1. Extracts data from a text file.

2. Transforms the data by converting all text to uppercase.

3. Loads the transformed data into a final output file.

In this Luigi pipeline, each step is a task, which is a piece of work that depends on other tasks. The first task, ExtractData, creates a text file (extracted_data.txt) with a list of sample words. The second task, TransformData, depends on ExtractData, so it runs only after the first task is done. It reads the file, changes all words to uppercase, and saves them in transformed_data.txt. The third task, LoadData, depends on TransformData and waits for it to finish before it runs. It reads the transformed file, adds "Loaded:" to each word, and writes the result to final_data.txt. Luigi uses the requires() method to manage dependencies, making sure tasks run in the right order. The output() method shows where each task's result will be saved, and the run() method has the actual steps to process the data. Running the script with luigi.build([LoadData()], local_scheduler=True) starts the whole pipeline, running tasks one after the other while keeping their dependencies in check.

Targets

A target in Luigi represents the output of a task and acts as a checkpoint to determine if the task needs to run. Before executing a task, Luigi checks whether the target already exists. If it does, Luigi skips the task, assuming it has been completed successfully. This prevents unnecessary recomputation and makes pipelines more efficient and fault-tolerant.

Types of Targets in Luigi

1. LocalTarget – Used for files stored locally.

2. S3Target – Stores data in Amazon S3.

3. GCSFileSystemTarget – Saves files in Google Cloud Storage.

4. PostgresTarget / MySqlTarget – Used for database records.

The example is shown in task section.

Dependencies

In Luigi, dependencies define the relationships between tasks in a data pipeline. They ensure that each task runs only after its required tasks are completed, maintaining the correct execution order. This helps streamline workflows, prevent errors from out-of-order execution, and ensure data consistency.

How Dependencies Work

Dependencies are defined using the requires() method in a Luigi task. A task lists its required tasks, and Luigi automatically ensures that those tasks are completed before running the dependent task.

Dependency Graph

In Luigi, dependency graphs visually represent the relationships between tasks in a pipeline. They help in understanding task execution order, identifying bottlenecks, and debugging failures.

How Luigi Builds Dependency Graphs

Each Luigi task defines dependencies using the requires() method. When a pipeline runs, Luigi constructs a Directed Acyclic Graph (DAG) where:

• Nodes represent tasks.

• Edges represent dependencies.

Luigi ensures that each task executes only after all its dependencies are completed.

Scheduling

Luigi provides flexible scheduling options to automate workflow execution. Tasks can be run manually, on a schedule, or triggered by dependencies.

1. Running Tasks Manually

   You can execute a Luigi task manually using the command line:

luigi --module my_tasks TaskName --local-scheduler

This runs the task immediately without relying on the central scheduler.

2. Using luigid for Centralized Scheduling

   For managing large workflows, Luigi offers a central scheduler (luigid). It tracks task status and ensures dependencies run in the correct order. Start the scheduler with:

luigid --background

Then, run tasks using:

luigi --module my_tasks TaskName --scheduler-host localhost

This enables centralized monitoring and parallel execution of tasks.

3. Automating with Cron Jobs

   Luigi does not have a built-in scheduler like Apache Airflow but can be integrated with cron jobs for periodic execution. Example cron entry for running a Luigi task every day at midnight:

0 0 * * * luigi --module my_tasks TaskName --scheduler-host localhost

Features of Luigi

Luigi is a Python-based workflow management system designed to build and automate complex data pipelines. It ensures tasks run in the correct order and handles dependencies efficiently. Here are some of its key features:

1. Task Dependency Management
   Luigi automatically tracks and resolves dependencies between tasks using a Directed Acyclic Graph (DAG). This ensures that each task runs only when its dependencies are met.

2. Automatic Checkpoints with Targets
   Each task in Luigi produces an output target (like a file or database entry). If a target exists, Luigi skips re-execution, preventing redundant processing and improving efficiency.

3. Built-in Scheduler & Worker System
   Luigi comes with a central scheduler (luigid) that allows tasks to be monitored, retried, and executed in parallel using workers. This makes it great for handling large-scale data pipelines.

4. Scalability & Parallel Execution
   Luigi can run tasks in parallel using multiple workers, making it ideal for big data processing and distributed computing.

5. Integration with Various Storage & Compute Systems
   Luigi supports multiple storage and database systems, including:

   • Local file system (LocalTarget)

   • Amazon S3 (S3Target)

   • Google Cloud Storage (GCSFileSystemTarget)

   • Hadoop & HDFS (HdfsTarget)

   • Databases like MySQL & PostgreSQL (PostgresTarget, MySqlTarget)

6. Visualization & Monitoring
   Luigi provides a web-based UI that allows users to monitor running jobs, view logs, and visualize task dependencies. Run the scheduler with:

    luigid --background
   

   and access the UI at http://localhost:8082.

7. Fault Tolerance & Retry Mechanism
   Luigi automatically retries failed tasks and provides logging to help diagnose errors. This ensures robustness in long-running workflows.

8. Extensibility & Customization
   Luigi allows you to create custom targets, parameterized tasks, and extend existing functionality, making it highly adaptable to different data engineering needs.

9. Support for Incremental Processing
   By using checkpoints (targets), Luigi only processes new or changed data, reducing computational overhead for large datasets.

Problems with Luigi

1. No Native Support for Real-Time Streaming

   Luigi is designed for batch processing, meaning it lacks support for real-time or event-driven workflows. If you need real-time data processing, tools like Apache Airflow, Apache Kafka, or Flink may be better suited.

2. Requires Manual Error Handling & Retries

   While Luigi provides a retry mechanism, it does not handle failures automatically as efficiently as some other orchestration tools. You must explicitly define retry logic in your tasks to avoid failures stopping the entire pipeline.

3. Limited Support for Dynamic Workflows

   Luigi workflows are static, meaning the dependency graph is defined before execution. Dynamically adding tasks at runtime is difficult compared to Apache Airflow, which supports dynamic DAG creation using Python scripts.

4. Web UI is Basic & Lacks Advanced Features

   The Luigi web UI is minimalistic and does not provide detailed logs, alerting, or advanced visualization. Other tools like Airflow offer more interactive UIs with better monitoring capabilities.

5. Dependency Management Can Get Complex

   Large DAGs with many dependencies can become hard to manage. Unlike Airflow, Luigi does not provide a built-in way to visualize complex DAGs dynamically.

6. Limited Community & Ecosystem

   Smaller community compared to Apache Airflow, Prefect. Fewer pre-built integrations for third-party services, requiring custom implementations for many use cases.

7. Higher Resource Usage Due to Process-Based Execution

   Luigi uses processes instead of threads for task execution, which can be more resource intensive. This approach may lead to higher memory and CPU usage compared to threading, which is more lightweight.

Conclusion

Luigi is a Python-based workflow management system developed by Spotify. The Spotify team had stopped development of the Luigi, and it didn’t get many fixes for a year. But now it has been actively maintained. Although, the Spotify has stopped using it you can find the detailed blog here.

I personally believe Luigi can be useful for small to medium-scale tasks. There are many frameworks and options available, and Luigi may not have a large community like Spring Batch. However, with each fix, Luigi is improving and will continue to get better over time. The main thing for me is that it is the simplest library for batch processing in Python, which is a language used by many beginners.

For large-scale tasks, you might consider using more robust options like Apache Flink or Apache Spark, which are designed to handle big data processing efficiently. If you prefer a lighter solution, Spring Batch could be a good choice, as it offers a more streamlined approach without being too resource-intensive. These alternatives provide greater scalability and performance for handling complex workflows and data processing needs.

References

• Luigi Documentation

• Luigi for Data Orchestration: Building Blocks, Capabilities, Setup

• How to Build a Data Orchestration Pipeline Using Luigi in Python? | Airbyte

• How To Build a Data Processing Pipeline Using Luigi in Python on Ubuntu 20.04 | DigitalOcean

• End-to-End Analysis Automation over Distributed Resources with Luigi Analysis Workflows

• DAGs — Airflow Documentation

• Spring | Batch

So here I am writing a blog about it, so you don't have to waste your precious time by searching and reading documentation { I hate it too }. Let's not waste time and begin the tutorial.

Initialize Your React Project

There are multiple ways you can create a react template. There are many tools available to do the same. I am going to use the same tools that I used for the project. So we are going to use React with TypeScript built using Vite.

Creating Template using Vite

what is Vite?

Vite is a build tool that aims to provide a faster and leaner development experience for modern web projects. Vite is different as it was created by Evan You, who also created Vue.js, as a means to simplify and accelerate the build process. Vite leverages native ES modules in the browser to load your code instantly, no matter how large the app is. It also supports hot module replacement for an extremely fast development workflow.

Setup React + TS using Vite

You need to have Node JS version 14.18+, 16+. I will be using NPM as my package manager. To get started open a folder in which you want to set up the project, I am going with Documents folder and paste the command below:

npm create vite@latest

You may be asked to install Vite if you don't have it already just press continue. It will ask for,

• Project name: Give it any name you like, I prefer to give skeleton-loading as we trying to create skeleton-loading animation here.

• Framework: Choose React of course.

• Variant: For variant choose TypeScript.

Now cd into the folder and install all the required packages. You can use the below commands,

 cd skeleton-loading # change this to your folder name
 npm install

Installing packages may take some, depending mostly on your network speed. After installation, let's run the development server to check if everything is installed correctly. To run the development server use the command below,

npm run dev

So we have successfully set up React and Vite, and now we have to install Chakra UI.

Installing Chakra UI

They have a dedicated section in their documentation dealing installation of Chakra UI specifically for Vite, you can read it here. As per the docs installation command is,

npm i @chakra-ui/react @emotion/react @emotion/styled framer-motion

After installation open the project in your Favourite IDE, I am going with VS Code. And find main.tsx file, you can use Ctrl + P to search file in vs code.

And then wrap the App component with the ChakraProvider component. You have to import ChakraProvider component from the import { ChakraProvider } from '@chakra-ui/react'. The main.tsx file would look like this,

import React from "react";
import ReactDOM from "react-dom/client";
import { ChakraProvider } from "@chakra-ui/react";
import App from "./App.tsx";
import "./index.css";

ReactDOM.createRoot(document.getElementById("root") as HTMLElement).render(
  <React.StrictMode>
    <ChakraProvider>
      <App />
    </ChakraProvider>
  </React.StrictMode>
);

You can delete the code inside the files index.css and App.css and also remove their import from the files main.tsx and App.tsx. As these styles may interfere with the styles provided by Chakra UI, so it's better to remove them.

Now setup for our Chakra UI is complete. To verify if our setup is complete we can try something like placing a button on our website. Go to App.tsx and remove all the code inside the App function or paste the below code after removing the entire code.

import { Button } from '@chakra-ui/react'

function App() {
  return (
    <Button colorScheme='blue'>Button</Button>
  )
}
export default App

Then start the development server using npm run dev command. And you should see something like this on the webpage.

Let's start building!

We don't want to build much of a UI, we just need to learn how to use the skeleton loading animation. This is pretty easy actually, so we directly make it. In Chakra UI, three types of skeleton components can be used to display loading states:

• Skeleton: The Skeleton component is the basic skeleton component in Chakra UI. It is used to create a loading placeholder for the content. It takes the height as a prop and displays a rectangular skeleton with the specified height. This component is useful when you want to display loading placeholders for multiple items or lines of text.

  Example usage:

<Stack>
<Skeleton height="20px"/>
<Skeleton height="20px"/>
</Stack>

• SkeletonCircle: The SkeletonCircle component is used to create a circular loading placeholder. It takes the size props to specify the diameter of the circle. This component is often used to display loading placeholders for avatar images or circular icons. Example usage:

<Box padding='6' boxShadow='lg' bg='white'>
  <SkeletonCircle size='10' />
  <SkeletonText mt='4' noOfLines={4} spacing='4' skeletonHeight='2' />
</Box>

• SkeletonText: The SkeletonText component is used to create a loading placeholder for text content. It takes props such as mt (margin-top), noOfLines (number of lines), spacing (line spacing), and skeletonHeight to define the appearance of the loading text. This component is useful when you want to display loading placeholders for paragraphs or blocks of text.

Example usage:

<SkeletonText mt='4' noOfLines={4} spacing='4' skeletonHeight='2' />

Now we know all the skeleton animation components provided by the Chakra UI. Now let's try to make an animation for a card with an image and text below it.

Building a skeleton loading card

Firstly we start building cards with the card component provided by Chakra UI. I like to create a folder called component in which all components used will be present. There is no square image skeleton component provided by Chakra, so we have to use the Skeleton component and give it some height.

Let's create a file called SkeletonCard.tsx and return a card with Skeleton and SkeletonText components nested inside. The main thing is trying to recreate the component to which you want to apply to load using these 3 components. My SkeletonCard.tsx file looks like this,

import { Card, CardBody, Skeleton, SkeletonText } from "@chakra-ui/react"

const SkeletonCard = () => {
  return (
    <Card>
    <Skeleton height="200px" />
    <CardBody>
      <SkeletonText />
    </CardBody>
  </Card>
  )
}
export default SkeletonCard;

We have adjusted the height to ensure that it appears larger than the width, giving it the appearance of an image. Now we have to open App.tsx file and import this component.

In App.tsx file we can return a grid with a template attribute of {base: `"nav" "main"`}. I have added a navbar just for the heading Example Skeleton Loading Animation. You can remove it if you want to, if you do then change the template. We have to make a Grid component to define the template areas.

We need eight cards therefore we will be using an array called skeletons with 8 elements. And we map each element to a Skeleton card, you can set the array elements as key. The code in App.tsx looks like,

import { Grid, GridItem, Heading, SimpleGrid } from "@chakra-ui/react";
import SkeletonCard from "./components/SkeletonCard";

function App() {
  const skeletons = [1, 2, 3, 4, 5, 6, 7, 8];
  return (
    <Grid
      templateAreas={{
        base: `"nav" "main"`,
      }}
    >
      <GridItem area="nav" justifySelf="center">
        <Heading>Example Skeleton Loading Animation</Heading>
      </GridItem>
      <GridItem>
        <SimpleGrid columns={4} padding={10} spacing={6}>
          {skeletons.map((skeleton) => (
            <SkeletonCard key={skeleton} />
          ))}
        </SimpleGrid>
      </GridItem>
    </Grid>
  );
}

export default App;

Thus we have successfully implemented a skeleton animation using Chakra UI. This is that easy to do. To check the result you can use the development server. To do that just run npm run dev. And you should be able to see the result.

We have implemented only the animation, it's the best case when you're trying to fetch data from an API or database so the user has to wait for a while, so you can use this animation to increase the user experience. You can use the documentation of Chakra to know more examples and use cases.

Conclusion

We have completed implementing the skeleton loading animation on our website using Chakra UI. There are more libraries out there which can implement skeleton loading and you may be interested in them, so I have mentioned some below.

• react-loading-skeleton

• react-content-loader

There may be more I could only find them. User Experience is a crucial so using skeleton animations can give you a plus point atleast it did for me. Thank you for reading patiently and do follow me on social media.

He takes his Banana Book laptop and starts coding. He chooses Django paired with the Django REST framework as we want to make the banana website robust. But he gets get stuck and couldn't figure out what to do next. He reaches burnout and feels like he cannot do it. Then he remembers his bananas and his love for bananas gave him more determination. He remembers an old friend Akash😎, who can help him to complete this task. So he asks the guy and that guy writes a tutorial for him.

Let's help Steve to make a REST API using the Django REST framework.

Let's help Steve

Steve is a great guy and his love for bananas is immeasurable, so let's help Steve by building him a REST API using the Django REST framework. Firstly we require a laptop that has Python3 installed on it. You can download it from Python.org. Different versions of Python are available but be careful and choose any version >= 3.8.

We will be building a Rest API to fetch banana stocks from the database. Let's not connect the database, it may get confusing to call it in one tutorial. Here instead we can use a JSON file which stores data and parse it to get the values of the stock. We are trying to learn how to use the Django rest framework and create a REST API using it. So we won't be learning much about database connection here.

Setting up Django

It's better to use a dedicated environment for each Django project. Even Django team suggests using a virtual environment. Python itself comes with venv for managing environments which we will use for this guide.

Setting up environment

You can give any name to the virtual environment, but here I am going with djangoproject.

Windows

Make sure you have installed Python before continuing to the next instructions.

Next, create a folder in your preferred directory and name it whatever you like I am going with Django REST API framework. Next to set up the virtual environment enter the command opening terminal or CMD.

py -3 -m venv djangoproject
.venv\scripts\activate

Linux or Mac OS

Most of the Linux Distributions come with python3 installed by default, just check if it exists by using the command.

python3 --version

For Mac users just head to python.org and download the .pkg file and start installing.

Next to setting up a virtual environment, on Linux sometimes we need to install the extra library. To do the same, just enter the command in your terminal.

sudo apt-get install python3-venv

Next, to initialize and activate a virtual environment on Mac and Linux, use the commands below.

python3 -m venv djangoproject
source .venv/bin/activate

Installing Django

Next, we need to install Django on our virtual environment. To do that we can simply use the command below. Make sure you have activated the virtual environment before executing this command.

pip install Django

After that, we have to start a project in our directory to get started. Django provides a template for the project and we can use that to create our app. To get the template in our current working directory you can use the command below.

django-admin startproject django_rest_trial .

django_rest_trial is the name of the project which I choose, but you're free to use whatever name you prefer. After running that some files and folders should appear on the file explorer.

Now we need to create an app inside our Django project. The apps are like parts of a website which manage a certain task. for example, we can have an app for handling payment on the website or we can have an app that verifies login and sign-up. This is one of the pros of using Django as different teams can manage different apps yet they can make them work together.

To create a Django app you can use the below command. Here rest_api is my app name as it manages the REST APIs on our website. Feel free to choose a different name if you want to.

py manage.py startapp rest_api #windows

python3 manage.py startapp rest_api # Linux and MAc

After executing this command a new folder should appear with the name of the app that was created using the above command.

Even though we don't require a database, Django sessions and other values like admin, users etc. need a database. By default, Django comes with SQLite, which is a great option as it doesn't need much of any configuration. Django needs to migrate the required tables and values to the database to do that we have to use the below command.

py manage.py migrate # Windows

python manage.py migrate # Linux

We have finished setting up our Django project, you can even try running it by just running the server using the runserver command. Go to http://127.0.0.1:8000/ and you should see the standard Django greeting page.

py manage.py runserver # Windows

python3 manage.py runserver # Linux

Setting up the Django REST framework

Next, we have to install the Django REST framework, to do the same we can use pip. To install it use the command below:

pip install djangorestframework

Then we need to edit the settings.py of our Django project and include rest_framework in our project. You can find the settings file inside the main project folder which is the django_rest_trial for me.

And that's it we have completed the setup of the Django REST framework. We also need to install whitenoise which helps in simplifying static file serving for Python web apps. You may think that we don't have any static files for our API, but Django has it for the admin panel, debug panel etc.

pip install whitenoise[brotli]

You have to set up whitenoise as well, to do that you have to add whitenoise to your middleware in the setting.py file.

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware', #Added
    'django.contrib.sessions.middleware.SessionMiddleware',
    ...
]

You need also setup static files root to that copy and paste the below line after STATIC_URL variable. And add the lines below

if not DEBUG:
    STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
    STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'

Let's start coding

First of all, we need to get the JSON file on which we are going to store the data on. I will provide it below, just copy and paste it into a file named database.json and keep it the in the root folder with the manage.py file.

{
  "bananaTypes": ["Apple Banana", "Cavendish", "Lady Finger", "Red Dacca"],
  "availableStock": [100, 200, 300, 400]
}

Next, we need to configure routes or endpoints which our REST APIs will use. Let the rest_api app uses the route /rest/. That is over API endpoints will be starting with the rest keyword.

To configure this we need to create urls.py file in the rest_api app folder. Keeping it aside we will need it later, open the main project folder and open the urls.py file inside it and add the following element to the urlpatterns list. You may need to import the include function from the django.urls. Finally, the file would look like this,

from django.contrib import admin
from django.urls import include, path # include was also imported

urlpatterns = [
    path('admin/', admin.site.urls),
    path('rest/', include('rest_api.urls')), # added
]

Next, we will start coding the main part of the project. Open the views.py file in the app folder and continue the part.

Coding the main part

Let's start by creating a function that reads our database which is the database.json file. I need to parse JSON so we import json library also. And create a read_database() function. which somewhat looks like,

import json

def read_database():
    with open('database.json', 'r') as f:
        return json.load(f)

After that, we create a function which accepts both GET and POST request and respond with the available stock of each banana variety. To do that we need to import app_view and Response from Django REST framework.

The app_view is a decorator which is used to denote what type of requests it can accept in our case we accept both GET and POST requests. It almost acts the same way as the app.route in Flask, we mention methods in the same if you know.

The Response object allows you to easily serialize data into the appropriate format (such as JSON or XML) and set the appropriate HTTP status code for the response. It also provides several convenient methods for setting headers, cookies, and other response metadata.

The views.py file would be like after all changes,

import json

from rest_framework.decorators import api_view
from rest_framework.response import Response

def read_database():
    with open('daabase.json', 'r') as f:
        return json.load(f)

@api_view(['GET', 'POST'])
def get_stocks(request):
    try:
        database = read_database()
        banana_type = database['bananaTypes']
        avail_stock = database['availableStock']
        response = {}
        for banana,stock in zip(banana_type,avail_stock):
            response[banana] = stock
        return Response({'error':False, 'bananaStock': response}, status=200)
    except Exception as e:
        return Response({'error':True, 'message': str(e)}, status=500)

If you remember we have to still route the function get_stocks to one of the endpoints. To do that open the urls.py in the rest_api or app folder and paste the below code,

from django.urls import path
from . import views


urlpatterns = [
    path('get_stock/', views.get_stocks, name='available stocks'),
]

Now our Django REST API is successfully created. You can try running it on localhost by using the command python manage.py runserver and heading to http://127.0.0.1:8000/rest/get_stocks/ to see the API working in real-time.

You can even use Postman or Hoppscotch to test the API to check if it's working correctly.

Hosting API on render.com

You need an account on render.com. You can signup for one here. And connect your GitHub account to it. You can find all the instructions in detail on my old blog, I will link it below

Creating Requirements.txt

Usually, I use external library pipreqs, but you can just simply type django, djangorestframework, gunicorn and whitenoise[brotli] in the requirements.txt file. When deploying, render will install the latest version of these modules.

Creating a .gitignore file

.gitignore file helps to avoid uploading unwanted or private files to the git clients such as GitHub or GitLab. When sharing your code with others, there are often files or parts of your project, you do not want to share. So these files or folders can be mentioned in the .gitignore file.

For this Django project, we don't want to upload djangoproject, __pycache__ and db.sqlite3to GitHub. So we mention them in the .gitignore file like this.

Upload to GitHub or GitLab

I am using GitHub here, but you can use GitLab as well. I have installed git, and I have connected it to my GitHub account. You'll find many articles on how to do it, just google it. If you're using Visual Studio then it has an built GitHub plugin which makes it way easier. I have successfully uploaded it to GitHub, you can find mine at akashrchandran/Django-REST-API-framework.

Make the app production ready

We will need to change some parts of the settings of Django to make it production ready.

Find the declaration of the SECRET_KEY setting by opening settings.py. We don't want to keep production secrets in source code, so we'll get them via an environment variable we'll set later:

# Add this to begining of the file
import os

Next change SECRET_KEY to be equal to the environmental variable we are going to provide later.

SECRET_KEY = os.environ.get('SECRET_KEY', default='thisisahugesecret')

Also, change DEBUG to dynamically change based on environment variables

DEBUG = 'RENDER' not in os.environ

We need to change allowed hosts also, as we are hosting. Rather than directly giving the host id after deployment we can also make it dynamic

# https://docs.djangoproject.com/en/3.0/ref/settings/#allowed-hosts
ALLOWED_HOSTS = []

RENDER_EXTERNAL_HOSTNAME = os.environ.get('RENDER_EXTERNAL_HOSTNAME') 
if RENDER_EXTERNAL_HOSTNAME:
    ALLOWED_HOSTS.append(RENDER_EXTERNAL_HOSTNAME)

And we are done our project is now production ready. Now let's push changes to GitHub and continue.

Create a Build Script

We need to run a series of commands to build our app. We can accomplish this with a build script. Create a script called build.sh at the root of your repository:

#!/usr/bin/env bash
# exit on error
set -o errexit

pip install -r requirements.txt

python manage.py collectstatic --no-input
python manage.py migrate

Make sure the script is executable before checking it into Git:

chmod a+x build.sh

We completed the build script as well, and now push it to GitHub.

Deploying to Render.com

As we are not deploying a static site, we should choose the option Web Service.

If you have connected your GitHub account, then you can just search the repository name and find it there.

Give a name to your web service, then select the region which is nearest to you, for me in Singapore. Leave all other values default. For the Build command, copy and paste the below:

./build.sh

For the Start command, copy and paste the below:

gunicorn django_rest_trial.wsgi:application

where django_rest_trial is the name of the Django project. Do change this as per your project name.

Next, scroll down and click the Advance button and add the environment variables. Here we only require SECRET_KEY. You can also use the generate button which is provided by render. After adding click the Create Web Service button.

After clicking the button you should see a log with events happening, wait till it finishes deploying.

After the site is live visit the route /rest/get_stocks and you should see the API working.

And here we have done it and we have successfully deployed our REST API to render.com. Now sit back and relax by eating as many bananas as you can. The deployed version is available here.

Steve is partying

Thanks for helping Steve to complete his REST API, he is now partying🥳. You can get more information on how to use the Django REST framework to best way read the documentation. There are many alternatives to the Django REST framework, you can give them a try if you want,

• Django Tastypie

• Django Restless

• django-jsonview

They are good alternatives but some hardly maintained any more. Django REST framework is more powerful with authentication and stuff, you can learn about it in the docs.

That's all we have completed the tutorial. Looking forward to meet you in the next one. Meanwhile you can follow me on hashnode and other platforms,

Thanks for reading patiently.

That's why most of the developers use Vercel, Netlify, Heroku, and Firebase as they handle deployment for you and prevent you from making any big mistakes. Anyways vercel just introduced 3 new databases or storage options. And let's see them in brief.

What is vercel?

Well, Vercel is a cloud platform which enables developers to host their websites and web apps that deploy and scale automatically. Vercel is the favourite choice for the deployment of Next.js apps as they have more efficient support for the framework. That's not that big as vercel also maintains the Next.js.

But the best part is that they support almost every language framework though they specialize in javascript frameworks. I have hosted many Python frameworks like Flask, FastAPI and even Django. They even support PHP, I have hosted my app there and haven't got any issues with speed or rendering.

They have a crazy awesome user interface and experience that makes it so user-friendly. It provides a free tier named Hobby for developers to experiment and try their platform. It’s against Vercel’s TOS to run a commercial app on the free tier. Even though you technically could, they will shut down your site if caught in violation. So you'll have to go with a paid plan and it's a bit expensive when compared to other platforms.

Introduction

Vercel introduced 3 new additions to their storage solutions. They did it on 1st May and you can find the announcement on the website. The new additions are:

• Vercel KV (Beta)

• Vercel Postgres (Beta)

• Vercel Blob (Private Beta)

We will be talking about them individually one by one below. This is a huge thing as they are trying to control users from using any other third-party database solution and managing it themselves. The database was the only thing missing on vercel but now it's kind of complete at least it is in beta for public use. They are not yet available for enterprise usage, mostly they will be available soon.

Vercel KV

It's key-value storage like Redis, in fact exactly like Redis as it is powered by Upstash. If you didn't know Upstash is a Serverless in-memory cloud database from Upstash Inc, a California-based company. It is a Serverless Redis Service. It can be used for caching layer or as a database. You can more details about it here.

For now, I am more interested in the speed at which it works, let's try to create a simple free tier vercel KV storage. The creation is pretty straightforward, they just ask you for the name and the region. I went with Singapore as my region, as it seems to be the nearest one.

They also provide CLI to set and get values directly from the database.

I copied the Template that they provide for the testing, you can get the template source on GitHub. I modified its page.tsx file just to set and get a value from the database.

import kv from "@vercel/kv"

export default async function Home() {
  console.time("res");
  await kv.set("message", "Hello World!");
  const msg = await kv.get("message");
  console.timeEnd("res");
  return <p>{msg}</p>;
}

When I first ran it took more than 200 ms to load, as it was the first request server took time to warm up and respond. Even 2nd and 3rd try still resulted in more than 100 ms, but after that, it came down under to 25 - 50 ms range.

Keep in mind that this was both reading and writing. But when I tested reading from the database alone it was a little above 15 ms.

The storage supports all the operations that can be done on Redis. There are limitations applied on creation, requests, record size etc you can find them here. The pricing seems to be a little above the expected but it may drop in the future.

Vercel Postgres

This was the part for which I was most excited. They introduced Postgres one of the most advanced general-purpose object-relational database management systems. I have been using it for my projects. It's powered by Neon a fully managed serverless PostgreSQL. Neon separates storage and computing to offer modern developer features such as serverless, branching, bottomless storage, and more.

Coincidentally I used Neon for a freelancing project and it was amazing as it provide a web interface to edit, give analytics and view tables which made the development of my Django app super easy. I will have to try it to be sure if it is as good as Neon. The creation of Postgres is simply the same as that of Vercel KV and only requires a name and region ( Singapore is available so I choose that ).

I copied the Template that they provide for the testing, you can get the template source at GitHub. I modified its page.tsx file just to create, insert and select a value from the database.

import { sql } from "@vercel/postgres";

export default async function Home() {
  console.time("create");
  await sql`CREATE TABLE IF NOT EXISTS PERSON (name varchar(255), age int)`;
  console.timeEnd("insert");
  console.time("create");
  await sql`INSERT INTO PERSON (name, age) VALUES ('Akash', 21)`;
  console.timeEnd("insert");
  console.time("select");
  const { rows } = await sql`SELECT * FROM PERSON`;
  console.timeEnd("select");
  return (
    <div>
      {rows.map((row) => (
        <div key={row.age}>
          {row.name} - {row.age}
        </div>
      ))}
    </div>
  );
}

create: 474.285ms
insert: 102.999ms
select: 103.254ms

I have tested it for like 10 times and I think the time averages out nearest to these values. It's pretty good and fast for me at least. Vercel also has a web interface to view tables and execute queries same as Neon.

The pricing is almost the same as that of Vercel KV. And also include the same limitations as of the same. You can view the pricing here.

Vercel Blob

This is something that can be very useful. Vercel Blob allows you to upload and serve files via a global network through unique and unguessable URLs. it's based on Cloudflare R2 which allows you to store large amounts of unstructured data like images and PDF files on the edge in other words it's a replacement for storage buckets like S3 and that could be a game changer if your app does file uploads. It is still in private beta but you can sign up for the waiting list to try it. To signup head to this link.

They say it is users can upload files of any size. But for the private beta phase, the upload size is limited to 4 MB, but soon they will release its full potential and will be available for public beta.

The SDK is pretty simple for developers to use same as the other storage. I don't have access to it so I can't test it for myself but I would say it would be good as the other two. As per the doc, the snippet given is

import { put } from '@vercel/blob';
import { NextResponse } from 'next/server';

export default async function upload(request) {
  const form = await request.formData();
  const file = form.get('file') as File;
  const blob = await put(file.name, file, { access: 'public' });

  return NextResponse.json(blob);
}

The pricing is available here. It will be good for companies looking for storage that has some simplicity yet advanced technologies and have loads of money to waste.

Is it worth it?

Without much of a surprise No, it's not worth it as they are priced insanely high. As it's vercel they have a bit high pricing as always but the products are also somehow astonishingly good and have a good user interface. To phrase it they are Apple of the web industry. But that doesn't justify the pricing,

• Vercel KV is 1.5x the price of Upstash scaling without the way more generous free tier.

• Vercel Postgres is 2x the price of Neon scaling without the way more generous free tier. The free tier of Neon has 3GB of storage compared to 256MB of Vercel, no limit of storage on its pro plans compared to 512MB of storage for Vercel, no limit on read and write data compared to Vercel, no limit on compute time compared to Vercel.

• Vercel Blob is 2 times the price of Cloudflare R2 without the way more generous free tier.

The prices may change later. The VP of Developer Experience at Vercel commented on Reddit that the prices may change with the feedback of the beta users and the pricing of vercel blob isn't final yet.

This is still good for developers who don't want to use other applications just for the database instead they can find all the things in a single place which makes it easier to maintain and use.

Conclusion

Vercel is an amazing platform for hosting web applications without a doubt. And I am sure I am going to use Vercel Postgres for future projects for sure. You should try it for sure as all the storage options are available for the Hobby plan. Vercel is trying new Server Actions an alpha feature in Next.js, built on top of React Actions. They enable server-side data mutations, reduced client-side JavaScript, and progressively enhanced forms. This would be a great feature for web developers.

Lastly thank you for reading this patiently and please follow me on Linkedin, you can find the link at Akash R. Chandran | LinkedIn.

They didn't have a GUI to manage apps only CLI to deploy and manipulate apps, the only thing they provided was the register/login and docs if I remember correctly. I tried deploying some apps but failed for some I don't remember. So let the deta cloud and tried something else.

But yesterday I came across the same site and was amazed that they are shifting to a new better version called Deta Space. This is in still beta but it has everything a modern platform has like GUI to manage apps, view logs, better docs and even a page to find user-build apps that are deployed on Deta Space. So I decided to test it out by deploying a sample FastAPI application. We will use the same repository as I used in my old blog showing the deployment of the FastAPI application on Render.

Creating Deta Space Account

As with any other site, you will need an account to host apps on Deta Space. They don't provide any third-party signup method so you will have to have to use a good old email password. You are required to choose a unique username, I think it's for subdomain assignment not sure yet though. You can sign up using this link.

After login, you should be redirected to the dashboard and you should look something like this

Setting Up CLI

Although the site has more GUI-rich features it still lacks a deployment option. So we have to set up our CLI tool if we want to continue. Setting up the CLI - Space Docs (deta.space) here you can find the instructions to do so. After installation login can be done by using the command

space login

If installed correctly it should show something like this. Next, we need is the authentication token from Deta space.

Getting token from Deta Space

After you log in you will be redirected to the dashboard, use the command palette. Just search for the settings and then click generate a new token.

After that copy the token and come back to the terminal page. And paste it onto the terminal

we have successfully set up CLI. Now let's move on to the next part which is pushing the code to Deta Space.

I am not going to show to how to code or install FastAPI for that you can read my old blog post.

You can also view the code project at my GitHub repository at akashrchandran/simple-api (github.com).

Creating .spaceignore file

To exclude certain files and directories from being uploaded during space push, use the .spaceignore file. It acts similarly to a .gitignore file, where each line in the file specifies a pattern. All files and directories matching the patterns will be excluded from the push.
For example, a .spaceignore file with the following content space push will ignore the test and docs paths.

test
docs

For our project, we will be ignoring the folders

venv
__pycache__

Deploying to Deta Space

After all the hard work we have done here is going to pay off, let's make the app accessible from anywhere on the planet. Open up the code in visual studio code or any other IDE or Notepad your wish. And open the terminal on the side or use the integrated terminal option in VS code.

Make sure you have the Space CLI installed & authenticated, then run the following command to create a new Space project in the current directory.

space new

Editing the Spacefile

This will create a Spacefile in the root directory. And it will only have the version v:0 as text and comment to the docs. This is where we define our runtime engine and other requirements. To get a clear idea about it you can visit the Spacefile - Space Docs (deta.space). To run the FastAPI app we need to define some requirements in the Spacefile. Here's what the file looks like after adding the required lines.

# Spacefile Docs: https://go.deta.dev/docs/spacefile/v0
v: 0
icon: ./icon.png
app_name: "Simple FastAPI APP"
micros:
  - name: simple-fastapi
    src: ./
    engine: python3.9
    dev: .venv/bin/uvicorn main:app --reload

I ran into a few troubles while making the Spacefile, it seems python apps don't support run commands. Use dev to specify a command which starts your Micro in development mode. This command will be used to start your micro when you run.
You can specify the version of python you need using the key engine. There are only two versions available currently python3.9 and python3.8.
src is the directory where the main.py file resides. You can keep it inside another folder and then specify the path to that folder.

for the icon I am using the logo of FastAPI, you can check it out at logo-teal.png (1023×369) (tiangolo.com).

PS: You are required to keep the file main.py as the entry file or it will not work. Make sure you use main.py for any python project.

Deploying the app

To deploy we just need to give the command, and it should automatically deploy and give the output URL.

space push

Congratulations the app is live🥳🥳. And you will be able to see the application in your dashboard. You can find my deployed version at https://simplefastapi-1-h6036723.deta.app.

Conclusion

This was a simple blog explaining the way to deploy the FastAPI application on Deta Space. There are many more things you can add to the project, I am just setting an example. The documentation is pretty amazing and you can find almost all the parts documented, do give it a read at Getting started - Space Docs (deta.space).

Overall I would say it's pretty easy compared to other services, even though some services are popular yet it's hard to get the app deployed on the service. The Deta Space is not that famous but it was easy to deploy apps. I am sure that I will use the service again to host my apps.

The Deta Space is still under construction and it's developed and updated daily. This makes it vulnerable to bugs and other problems. If you are stuck at some place, first read the docs. If you're still stuck after that then you can join their Discord server and post your queries and problems.

The community may be small but they are active and almost answers every single query raised. Giving feedback will help build the platform faster and better, do share your review on Discord.

Finally thank you for reading this blog, it was a good experience to share my view and knowledge about the platform. Have a great day!

Introduction to Python Packages

Python is a widely used high-level interpreted language. Python's design philosophy emphasizes code readability with its notable use of significant whitespace. Its syntax allows programmers to express concepts in fewer lines of code than would be possible in languages such as C++ or Java. The language provides constructs intended to enable clear programs on both a small and large scale.

Python packages are collections of modules that you can import into your scripts to make use of their functionality. In this article, we'll take a look at some of the most popular Python packages that every developer should know about.

What is a Python Package?

Python packages are modules that contain code written in the Python programming language. Packages can be used to extend the functionality of Python or to add new features to existing Python code. Python packages can be installed from a variety of sources, including the Python Package Index (PyPI), or third-party repositories. Once installed, packages can be imported into your Python code using the import statement.

Packages can contain a variety of different types of content, including:

• Modules: A module is a single Python file that contains code. Modules can be imported into other modules, or the main Python interpreter, using the import statement.

• Classes: A class is a template for creating objects. Classes can be defined in modules and then imported into other modules, or the main Python interpreter, using the from...import statement.

• Functions: A function is a block of code that performs a specific task. Functions can be defined in modules and then imported into other modules, or the main Python interpreter, using the from...import statement.

• Data: Data is any information that can be stored in computer memory. Data can be stored in variables, and passed to functions as arguments.

The Packages are:

Pandas

The Python package Pandas is a free and open-source library that provides high-performance data structures and data analysis tools. It is built on the popular Python libraries NumPy and matplotlib, which allow for efficient data manipulation and analysis. Data structures and operations for manipulating numerical tables and time series data are available in Pandas. It is a must-have for data manipulation and data science tasks. Pandas is a highly effective data analysis, manipulation, and visualisation tool. It is used in many industries, including finance, analytics, and data science, for data cleaning, preparation, and analysis. Pandas make it simple to work with large datasets, allowing users to summarise, manipulate, and visualise data quickly and easily.

NumPy

NumPy is a robust and effective module for doing scientific computations. It offers quick mathematical operations on these massive, multi-dimensional arrays and matrices for which it is developed. Many mathematical operations are offered by NumPy, such as Fourier analysis, linear algebra, and random number creation. For manipulating arrays, it also offers sophisticated indexing and slicing features. In disciplines including data science, machine learning, and computational physics, NumPy is extensively employed. Its popularity is a result of the performance and usability it combines, making it a crucial Python tool for scientific computing.

Matplotlib

Matplotlib is a Python package that offers a range of tools to developers for making interactive visualisations, such as graphs and charts, that can be used for data analysis. It is a commonly used library in science, and it works particularly well for making static 2D and 3D plots that may be highly customised and detailed. Also, it offers assistance with story interactivity and animation creation. After you grasp the fundamentals of charting, Matplotlib's simple syntax makes it simple to use. For everyone who wishes to extract amazing images from their data, it offers a necessary tool.

TensorFlow

Popular open-source software called TensorFlow is used to create and train machine learning models. It was created by Google and is one of the most used deep learning frameworks. Because TensorFlow is built on a dataflow graph, users can see how data flows through a neural network model. It offers a versatile and effective language for creating and refining models on huge datasets. Applications supported by this package include natural language processing, picture and speech recognition, and more. You may quickly scale sophisticated models to huge datasets with TensorFlow.

Scikit-learn

Built on top of NumPy, SciPy, and matplotlib, Scikit-learn is a free and open-source machine learning framework. It offers straightforward and effective tools for data analysis and mining. Scikit-learn offers tools for model selection and evaluation, data preprocessing, feature selection, and model visualisation in addition to several algorithms for classification, regression, clustering, and dimensionality reduction. Classification, regression, natural language processing, image processing, and other machine learning tasks are only a few of the many machine learning tasks for which it is frequently employed in both industry and research. Scikit-learn is a vital tool for anyone interested in machine learning because of its simplicity of use and extensive capability.

PyTorch

A well-liked open-source machine-learning library for Python is called Pytorch. Because of its adaptability, clarity, and scalability, it is widely used. Using the capabilities of graphics processing units (GPUs) to speed up training, Pytorch offers a simple interface for creating deep learning algorithms. The software is widely used in research, education, and business for a range of tasks including speech recognition, computer vision, natural language processing, and more. Pytorch has developed into a vital toolkit for machine learning enthusiasts and engineers alike thanks to its comprehensive collection of tools and community support.

Flask

Flask is a lightweight and powerful web development framework for building Python-based web applications. It includes a comprehensive set of tools and libraries for developing a wide range of web applications, from simple static sites to more complex web applications with dynamic data. Flask has an easy-to-learn and-use API, as well as a flexible and extensible design that allows you to tailor it to your specific requirements. It also includes built-in unit testing support, simple error handling, and easy integration with other libraries and services. Because of its simplicity, flexibility, and ease of use, Flask is a popular choice for developing web applications.

Django

Django is a Python web framework that is open source and follows the Model-View-Controller (MVC) architecture. It is widely used to rapidly create high-quality web applications. Django's clean design encourages code reuse and employs object-oriented programming principles. Django includes a powerful URL routing system for creating clean URLs, an ORM for working with databases, and an HTML template engine. It also has an authentication system, security features, and middleware support built in. Django has a vibrant community that is constantly improving and contributing to the framework. Because of its scalability, flexibility, and ease of use, it is a popular choice among developers.

Requests

With the Python library Requests, HTTP requests are made easier. It was developed for usage with API requests and page scraping. Requests offer a straightforward and understandable interface for handling HTTP requests and responses. The package includes functions for managing cookies and sessions, authenticating users, and encoding data in several different forms. Moreover, Requests supports the HTTP/1.1 and HTTP/2 protocols and offers synchronous and asynchronous request handling. The programme is frequently used for tasks including automation, data extraction, and web development.

BeautifulSoup

Beautiful Soup is a popular Python package for parsing HTML and XML documents. This package enables programmers to easily extract data from HTML pages. It makes it simple to use web scraping functions in Python. Beautiful Soup parses HTML documents and allows users to extract the content they require. Beautiful Soup's extensive documentation allows it to work with HTML files in a variety of formats. To efficiently extract content, the package supports a variety of searching and filtering methods. It is the go-to Python package for extracting data from web pages.

Conclusion

In conclusion, the Python programming language is a very strong and flexible tool that can be applied to a wide range of projects, including web development and data analysis. The top 10 Python packages that every developer should be familiar with can help developers use Python to its best ability. These software packages offer strong tools and capabilities that can be utilised to create intricate applications and address intricate issues. These packages make it simple for developers to build dependable and effective apps. We trust that this article has helped you gain a better knowledge of the top 10 Python packages that every developer should be familiar with.

I am a self-taught python developer for more than 3 years now and yet I'm constantly surprised by the new things I learn every day. Each day I find a new operator or function which makes life so much easier. Many operators and functions are inbuilt and knowing them makes a huge difference. It's not possible to learn them all but at least you can get familiar with them. So here's my cheat sheet for some rarely used in python3.

Operators

We can directly say that operator is a symbol or a set of symbols which is used to represent an action or a process. Operators are used to manipulate values and perform calculations, comparisons, and other operations in a program. All most all programming languages have the same set of operators like

• Arithmetic operators ( + , - , * , / )

• Comparison operators ( == , != , < , > )

• Logical operators ( AND , OR , NOT )

• Assignment operators ( = )

• Bitwise operators ( bitwise AND , bitwise OR , bitwise NOT )

These are simple operators that can be found in many programming languages.

But we are not here to talk about them, let's get right into the topic which is some rarely used operators in python. If you are already familiar with those, it suggests that you possess strong development skills.

Floor Division ( // )

In contrast to standard division, floor division returns the largest possible integer. This number is either less than or equal to the outcome of the usual division. We can do the same in python by using the operator //.

# Floor division
print(5 // 2)
# Output: 2

In python, there is no ceiling equivalent operator. But you can use upside-down floor division.

def ceildiv(a, b):
    return -(a // -b)

print(ceildiv(5, 2)

Walrus Operator ( := )

Python 3.8 introduces the walrus operator that lets you assign a value to a variable as part of an expression. You can use it in situations where you want to evaluate an expression and use the result later. By reducing duplication and improving logic flow, the walrus operator makes your code more concise and readable. You don't need to declare the variable first and then assign it, you can directly do that.

a = [1, 2, 3, 4]
if (n := len(a)) > 3:
    print(f"List is too long ({n} elements, expected <= 3)")

Matrix Multiplication Operator ( @ )

Matrix Multiplication Operator was introduced in Python 3.5 and it is used to multiply two-dimensional arrays in python. It's not available to use with the standard data structures of python rather it was specifically introduced for the numpy arrays. Numpy is a library specifically dealing with the operations of arrays or lists.

import numpy as np
A = np.array([[1, 2], [3, 4], [5, 6]])
B = np.array([[7, 8], [9, 10]])
result = A @ B
print(result)

But you can do the same on standard not directly but you can convert it to numpy array and then multiply using the Operator.

import numpy as np
A = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
arr = np.array(A)
B = [[10, 11, 12], [13, 14, 15], [16, 17, 18]]
arr2 = np.array(B)
result = arr @ arr2
print(result)

Set-like Operators

Python's "set-like operators" are operations like union, intersection, difference, and the symmetric difference that behave like set operations but can also be used with other sequence types.

• | (union): returns a new set that is absent of duplicates and contains every element from both sets (or other sequence types).

• & (intersection): returns a new set that only includes the components that are shared by the two sets (or other sequence types).

• - (difference): returns a new set that only includes the items from the first set (or another sequence type) that aren't present in the second set.

• ^ (symmetric difference): returns a new set that only includes the items that are present in one of the two sets (or other sequence types), but not both.

list1 = [1, 2, 3]
list2 = [2, 3, 4]

# Union
result_union = list(set(list1) | set(list2))
print(result_union)
# Output: [1, 2, 3, 4]

# Intersection
result_intersection = list(set(list1) & set(list2))
print(result_intersection)
# Output: [2, 3]

# Difference
result_difference = list(set(list1) - set(list2))
print(result_difference)
# Output: [1]

# Symmetric difference
result_symmetric_difference = list(set(list1) ^ set(list2))
print(result_symmetric_difference)
# Output: [1, 4]

In addition to sets, these operators can be applied to other sequence classes, such as lists, tuples, and strings. The returned output will be of the same type as the first argument when applied to non-set sequence types.

Functions

You can think of a function as a block of code that performs a specific task. Functions enable you to break your code into smaller, more modular pieces, making it easier to read, write, and maintain. Input arguments can be passed into functions, and then they can perform calculations or operations on them.

Many inbuilt functions are available in python and many may know the standard ones like

• len() - returns the number of items in an object such as a string, list, or tuple.

• input() - allows the user to input values from the keyboard and returns the entered value as a string.

• range() - a function that generates a sequence of numbers between the given start, stop and step values.

• sum() - returns the sum of all elements in an iterable such as a list or a tuple.

These are just some of the examples, the list may go far further. But for now, let's go deep down into rarely used python built-in functions.

sorted()

In Python, you can sort a list with list.sort(). Keep in mind that this method modifies the original list in place, so the list gets sorted directly. Instead, you can use the sorted() function to get a sorted copy of the list while maintaining the integrity of the original list. The original list is not changed by this method; instead, it returns a fresh, sorted list.

arr = [5, 7, 3, 6, 2, 9, 1, 4, 8]
sort_arr = sorted(arr)
print(arr)
print(sort_arr)
# Output
# [5, 7, 3, 6, 2, 9, 1, 4, 8]
# [1, 2, 3, 4, 5, 6, 7, 8, 9]

slice()

To specify how to slice a sequence, use a slice object. You can define the beginning and ending points of the slicing. Additionally, you can specify the step, allowing you to, for example, slice only the remaining items.

my_list = [1, 2, 3, 4, 5]
my_slice = slice(1, 4, 2)
print(my_list[my_slice])  # Output: [2, 4]

divmod()

The divmod() is a method in python that takes two numbers as arguments and returns a pair of numbers consisting of their quotient and remainder. This is better when you have both remainder and quotient needed for something.

a = 6
b = 4
print(divmod(a,b))
# Output: (1, 2)

reversed()

The method returns an iterator that accesses the given sequence in reverse order. list.reverse() also does the same task but the original list is modified directly, and no new list is created. And with reversed() it does not modify the original list but instead returns a new list with the elements in reverse order.

my_list = [1, 2, 3, 4, 5]
reversed_list = list(reversed(my_list))
print(reversed_list)
# Output: [5, 4, 3, 2, 1]

callable()

callable() is an internal function that, if the given object may be used as a function, returns True; otherwise, it returns False.

def foo():
    pass

print(callable(foo))
# Output: True

x = 5
print(callable(x))
# Output: False

class Bar:
    def __call__(self):
        pass

b = Bar()
print(callable(b))
# Output: True

lst = [1, 2, 3]
print(callable(lst))
# Output: False

setattr() and getattr()

The value of an attribute on an object is set with the setattr() function. Three parameters are required: the object whose attribute is to be set, the attribute's name as a string, and the value to set.

class Person:
    pass

p = Person()
setattr(p, 'name', 'John')
print(p.name)
# Output: John

The value of an attribute of an object is retrieved by the getattr() function. It requires two arguments: the object whose attribute is to be obtained and the attribute's name as a string. AttributeError is raised if the attribute is missing.

class Person:
    name = 'John'

p = Person()
print(getattr(p, 'name'))
# Output: John

vars()

A built-in function called vars() gives the value of an object's __dict__ attribute. It gives back a dictionary that contains the object's (local) symbol table, or all the attributes that were defined for the object.

class MyClass:
    def __init__(self, a, b):
        self.a = a
        self.b = b
obj = MyClass(1, 2)
print(vars(obj))
# output: {'a': 1, 'b': 2}

filter()

It is possible to filter components from an iterable (like a list) based on a function by using the built-in function filter(). The filter() method accepts two arguments: an iterable and a function that returns either True or False (such as a list). Only the components for which the function returns True are included in the resulting iterable after being applied to each element in the iterable.

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

def is_even(n):
    return n % 2 == 0

even_numbers = list(filter(is_even, numbers))

print(even_numbers)
# Output: [2, 4, 6, 8, 10]

next()

To get the next item from an iterator, use the built-in Python method next(). The next element in the sequence is returned when next() is invoked on an iterator. The StopIteration exception is raised if the sequence contains no additional components. To iterate over an iterator's components one by one, use the next() function in a loop.

my_list = [1, 2, 3, 4, 5]
my_iterator = iter(my_list)

print(next(my_iterator)) # Output: 1
print(next(my_iterator)) # Output: 2
print(next(my_iterator)) # Output: 3
print(next(my_iterator)) # Output: 4
print(next(my_iterator)) # Output: 5

# This will raise the StopIteration exception
print(next(my_iterator)) # Output: StopIteration

hash()

Python comes with a built-in function called hash() that gives an object's hash value. The hash value, which is a distinct integer that symbolises the object, can be used for several things, including comparing objects for equality and storing them in hash tables.

# hash a string
my_string = "hello world"
hash_value = hash(my_string)
print(hash_value)
# Output: -7156188717690076749

zip()

Python's zip() method accepts iterable as input and produces an iterator of tuples as a result. The matching elements from each iterable are contained in each tuple.

numbers = [1, 2, 3]
colors = ['red', 'green', 'blue']
result = zip(numbers, colors)
print(list(result))
# Output: [(1, 'red'), (2, 'green'), (3, 'blue')]

locals() and globals()

Built-in functions called locals() and globals() return dictionaries with the current local and global symbol tables, respectively.

x = 10  # global variable

def my_func():
    y = 20  # local variable
    print("locals:", locals())
    print("globals:", globals())

my_func()
# Output:
# locals: {'y': 20}
# globals: {..., 'x': 10, 'my_func': <function my_func at 0x7f5de5b72560>}

dis()

Python bytecodes can be disassembled using the dis module. It is primarily used to comprehend how the Python interpreter runs your code. A list of instructions containing the disassembled code is written to the console.

import dis

def my_func(a, b):
    c = a + b
    return c

dis.dis(my_func)

Output:

  4           0 LOAD_FAST                0 (a)
              2 LOAD_FAST                1 (b)
              4 BINARY_ADD
              6 STORE_FAST               2 (c)

  5           8 LOAD_FAST                2 (c)
             10 RETURN_VALUE

eval()

eval() is a Python function that returns the result of evaluating a string as a Python expression. The syntax for the eval() function is:

eval(expression, globals=None, locals=None)

where expression is the string to be evaluated as a Python expression and globals and locals are optional dictionaries defining the global and local variable namespaces.

Here's an example of how to use eval():

x = 10
y = 20
expr = "x + y"
result = eval(expr)
print(result)

Conclusion

There are several built-in operators and functions in Python, some of which are well-known while others are less well-known. In addition to performing numerous tasks like sorting, filtering, and data transformation, these functions and operators can assist developers in writing more concise and effective code. Certain lesser-known Python operators and functions, like slice(), dis(), reduce(), and chain(), as well as some frequently used ones, likesorted(), filter(), and len(), have been addressed in this blog. Even though it's impossible to go over every function and operator in Python, the examples we've given should aid developers in bettering their knowledge and comprehension of the language.

Introduction

If you're a programmer or a member of a development team, you probably use GitHub to collaborate on code and manage projects. GitHub is a popular platform that allows developers to share and review code, track changes, and work together on projects.

You may find it difficult to keep track of changes and updates to your codebase when you are working on multiple projects simultaneously. Notifications can help in this situation. Notifications allow you to stay informed about new pull requests, issues, and other changes to your code without constantly checking GitHub.

One way to get GitHub notifications is to use Discord, a popular chat platform for gamers that has become increasingly popular among developers. You can set up a webhook in Discord to receive notifications from GitHub and post them to a particular channel there. You can easily keep track of changes to your code and work more productively with your team if you do this. We'll show you how to set up a webhook on Discord to receive GitHub notifications in this blog post. Everything will be covered, including setting up a webhook on Discord, customising your notifications, and troubleshooting frequent problems.

Set Up a Discord Webhook

You must set up a webhook on Discord to begin receiving GitHub notifications on your channel. Sending automated messages from one app to another is possible using webhooks. In this instance, we'll make use of a webhook to send Discord notifications from GitHub. Follow these steps:-

• Open Discord and select the server where you want to receive GitHub notifications. Better to create a channel for only receiving GitHub notifications.

• Click on the server settings and select the Integrations tab. Click on the Create Webhook button and follow the prompts to create a new webhook. Give It a good name, anything you prefer.

• You'll receive a webhook URL once you've finished creating the webhook. You'll use this URL to link your GitHub repository to your Discord channel; it's kind of like a secret code. Keep it secure because anyone with access to the URL can notify your Discord channel.

That's it for setting up the webhook on Discord! In the next section, we'll show you how to configure your GitHub repository to send notifications to your Discord webhook.

Connect Your GitHub Repository to Your Discord Webhook

Now that you've set up your webhook on Discord, you need to connect it to your GitHub repository so that you can receive notifications on your Discord channel. Follow these steps to do the same:-

• Select any one of your GitHub repositories and go to the settings tab and select

  Webhook tab.

• Click on Add Webhook and enter the webhook URL from discord to payload URL space. And most important step add /github to the end of that URL.

  for example: https://discord.com/api/webhooks/104353453497/fmerooergkmrekgmergoemgergmergo-efoerjnrej/github

• Configure the webhook to listen for the events you want to receive notifications for, such as pull requests, issues, or commits. I like to receive everything, but you can customize it.

• Save the webhook and test it by making a change to your codebase and verifying that you receive a notification in your Discord channel.

That completes the setup of the GitHub repository and Discord webhook connection! You can use these steps to enable notifications for any activity on your GitHub repository on Discord. We'll go over some suggestions for personalising your notifications to make them more useful in the following section.

Customize Your GitHub Notifications

Now that you're receiving notifications on your Discord channel for activity on your GitHub repository, you may want to customize them to make them more useful. Some of the tips are given below:-

• Modify the webhook settings to receive notifications for only the events you care about. To do that, go to your GitHub repository's Settings page and select the "Webhooks" tab to change the webhook settings. Click the Edit button next to the Discord webhook you generated. The webhook can then be set up to listen to particular events from there.

  

Conclusion

We've demonstrated how to configure GitHub notifications on your Discord channel in this article. You won't need to frequently check GitHub if you follow the instructions in this guide to keep up with activity in your GitHub repositories.

To deliver notifications to your Discord channel, we first created a webhook on Discord, which offers a special Address. We then instructed GitHub to deliver notifications to the webhook URL by adding the webhook URL to our GitHub repository's settings. The notification messages' format was then altered to make them easier to read. We can add the details we wish to see in our notifications by altering the JSON payload that is provided to the Discord webhook.

In case you encounter problems along the road, we have included some troubleshooting advice. These pointers ought to help you fix any problems and enable GitHub alerts for your Discord channel. You can expedite your development workflow and maintain project updates by utilising GitHub and Discord together. We sincerely hope that this guide was useful to you, and we strongly advise you to experiment with various webhook settings to discover the configuration that best suits your needs.

Thank you for reading, and happy coding!

Creating Render Account

To host on Render, you'll need to sign up or create an account. It's free and supports direct login using google, GitHub and GitLab account. Or you can use email and password to signup. You need to verify your email to get full access to the platform. Just head to render/register.

Need for GitHub or GitLab account

It's easier to deploy an application that has its source code hosted on GitHub or GitLab. Render connects with GitHub or GitLab to deploy your apps and websites automatically on every push to your project. This is very useful, as you don't have to manually deploy every time you make a change. You can either connect GitHub or GitLab or both.

Let's make a simple API using FastAPI

We need an application to deploy to the Render. Let's build a simple API app which echos Hello World.

Setting up environment

You'll need to download and install Python3 to your local machine. It's better to set up a virtual environment for the application.

Windows

For windows, just head to python.org and download the latest EXE file and install. Installation is pretty well interactive, you just need to follow the instructions shown.

Next, create a folder in your preferred directory and name simple-api. Next to set up virtual environment enter the command opening terminal or CMD.

py -3 -m venv .venv
.venv\scripts\activate

Linux or Mac OS

Most of the Linux Distributions come with python3 be installed by default, just check if it exists by using command

python3 --version

For mac users just head to python.org and download the .pkg file and start installing.

Next to set up virtual environment, on Linux sometimes we need to install extra library. To do the same, just enter the command in your terminal.

sudo apt-get install python3-venv

Next, to initialize and activate a virtual environment on Mac and Linux, use the commands below.

python3 -m venv .venv
source .venv/bin/activate

Installing requirements

Make sure your package manager that is pip is up-to-date.

python -m pip install --upgrade pip

Now install FastAPI, using pip

pip install fastapi

FastAPI also requires an ASGI server, for production, such as Uvicorn or Hypercorn. I personally just use Uvicorn.

pip install "uvicorn[standard]"

Coding the API

Open the folder in any text editors, you can use notepad or advanced IDE's like VS Code or Sublime Text 4. Make a new file called main.py and we use this file to write the code in.

Just copy and paste the below lines to the file. If you're good with python or professional, then you can add or modify, making the API better.

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
def read_root():
    return {"Hello": "World"}

Save the file. Let's try out the simple-api using the command:

uvicorn main:app --reload

Head to localhost:8000 or http://127.0.0.1:8000. You should see the Hello World.

All done. Now we just need to create some extra files which are need by Render to identify it as a python application.

Creating Requirements.txt

Usually I use external library pipreqs, but you can just simply type fastapi and uvicorn in the requirements.txt file. When deploying, render will install the latest version of these modules.

Creating a .gitignore file

.gitignore file helps to avoid uploading of unwanted or private files to the git clients such as GitHub or GitLab. When sharing your code with others, there are often files or parts of your project, you do not want to share. So these files or folders can be mentioned in the .gitignore file.

For this simple-api we don't want to upload .venv and __pycache__ to the GitHub. So we mention them in the .gitignore file like this.

Upload to GitHub or GitLab

I am using the GitHub here, but you can use GitLab as well. I have installed git, and I have connected it to my GitHub account. You'll find many articles on how to do it, just google it. If you're using Visual Studio then it has in built GitHub plugin which makes it way easier.

I have successfully uploaded to GitHub, you can find mine at akashrchandran/simple-api. You should definitely add a readme.md to the repo, I will do it later.

Deploying to Render.com

As we are not deploying a static site, we should choose the option Web Service.

If you have connected your GitHub account, then you can just search the repository name and find it there.

Give a name to your web-service, then to select the region which is nearest to you, for me is Singapore. Leave all other values default. For the Start command, copy and paste the below:

uvicorn main:app --host 0.0.0.0 --port 10000

Deploying may take some time, so patiently wait. After successful deployment, you should see the message like this below.

Congratulations, you have successfully deployed your FastAPI application to the render platform. You can view the site I deployed here.

Summary

It was so simple to deploy a FastAPI based application to render. If you don't want to code or waste time on making a FastAPI based app, then you can try deploying mine. It is available on GitHub.

Render may the Heroku alternative thing you're looking for, as it gives out almost the same free specifications as Heroku free tier. Do give it a try!

Thanks for reading...

Gathering Requirements

Before we start making our bot, we need two things. It won't take much time, only few minutes. The things we need are:

• Telegram Bot Token

• TMDB API key

TMDB is the site from which we will pull the movie information from. It's free, you just need to register and fill some forms for an API key. So let's start gathering them.

Get Telegram Bot token

It's actually pretty simple to get the token from telegram. But first you need is a telegram account. I assume that you have one. Search for a bot named BotFather or use this link. It should have a verified emoji in its name.

Click on the bot and press start, or if you don't see that option, then type /start. I should greet you with a help message. You can read through that if you want to, it lists all details about what the bot can do. Next is, type /newbot and press send. It will ask for the name of the bot. After that it will ask for bot username which must be unique and should end in bot like mine here is phpmoviesbot.

Now you will receive a message which has a http API token. This was the token we were looking for, save it somewhere because we will need it later.

Get API key from TMDB

Head to TMDB and find register or join now. Then create an account. Then find settings and on the left side you will see an option named API, press that or use this link. You will need to fill a form about what you will be using the API for etc, just fill it yourself. After that you will get an API Key (v3 auth), save it somewhere we need it later.

Now that we have what we wanted, let's start with the programming side.

Setting up webhook

Setting up the webhook through which your Telegram bot will communicate is the next step in the creation process. In order to avoid having to query the API every few minutes (or seconds) to see whether, for example, a new message has been received, APIs use webhooks to notify you that something has occurred.

Telegram only makes use of one form of webhook, which delivers an update object each time something occurs. The webhook configuration is really simple. There are only two things you must be aware of: your API token (which you should already have from step one), and the URL at which your bot will be hosted. It will have a URL similar as https://yourdomain.com/yourbot.php. To ensure that Telegram sends the webhook, insert https at the beginning of the URL.

Go to https://api.telegram.org/<TELEGRAM_BOT_TOKEN>/setwebhook?url=https://yourdomain.com/yourbot.php in a standard web browser now. Do replace <TELEGRAM_BOT_TOKEN> with your telegram bot token.

Coding part

For this, I am assuming you know basic PHP syntax, formattings etc. Let's start then, we know that PHP script start with <?php and ends with ?> and we write the code in between. We are receiving webhook, therefore let's get input from the POST body:

$content = file_get_contents('php://input');
$update = json_decode ($content, true);

The received data is in JSON, to decode it to PHP based array, we have to use json_decode function. Now let's declare some variables which will store some, data from the update like username, chat ID etc.

$chat_id = $update['message']['chat']['id'];
$message = $update['message']['text'];
$username = $update['message']['from']['username'];

We would have to send many messages, so let's define it directly as a function to make it easier to reuse.

function send_message ($chat_id, $message) {
    $apiToken = <TELEGRAM_BOT_TOKEN>;
    $text = urlencode($message);
    file_get_contents("https://api.telegram.org/bot$apiToken/sendMessage?chat_id=$chat_id&text=$text");
}

Replace the <TELEGRAM_BOT_TOKEN> with the telegram bot token you made. Let's test the bot with a /start message, to do that we have to write some more code, just an if statement as below.

if ($message == '/start') 
{
    send_message ($chat_id, "Hey @$username  \nsend me any query to search for the movie.");
}

It seems to be working. Now let's complete it by adding the movie information fetching part. I want to make the bot search everything we text we send except /start. So we just add else to the pre-existing if statement. I am expecting you know how to parse JSON.

else {
    $req = file_get_contents("https://api.themoviedb.org/3/search/movie?api_key=<TMDB_API_KEY>&language=en-US&page=1&query=$message" );
    $movie = json_decode ( $req, true )['results'];
    if ($movie) {
        $movie = $movie[0];
        $title = $movie['title'];
        $overview = $movie['overview'];
        $popularity = $movie['popularity'];
        $release_date = $movie['release_date'];
        $vote_count = $movie['vote_count'];
        send_message($chat_id, "
        Title: $title
        Description: $overview
        Popularity : $popularity
        Release Date : $release_date
        Votes: $vote_count");
    } 
    else {
        send_message($chat_id, "Couldn't find any movie with that name!");
    }

Replace <TMDB_API_KEY> with the TMDB API KEY we gathered. Now let's try the bot, by sending a movie name.

Works perfectly, You can add more functions to it, but for now I am stopping it here.

Summed up

We learnt to create a small bot using webhooks. I will add an article about host the bot, for free, on vercel. The entire code for the bot is available below.

<?php
$content = file_get_contents ('php://input');
$update = json_decode ($content, true);
$chat_id = $update['message']['chat']['id'];
$message = $update['message']['text'];
$username = $update['message']['from']['username'];

if ($message == '/start') {
    send_message ($chat_id, "Hey @$username  \nsend me any query to search for the movie.");
} else {
    $req = file_get_contents("https://api.themoviedb.org/3/search/movie?api_key=<TMDB_API_KEY>&language=en-US&page=1&query=$message" );
    $movie = json_decode ( $req, true )['results'];
    if ($movie) {
        $movie = $movie[0];
        $title = $movie['title'];
        $overview = $movie['overview'];
        $popularity = $movie['popularity'];
        $release_date = $movie['release_date'];
        $vote_count = $movie['vote_count'];
        send_message($chat_id, "
        Title: $title
        Description: $overview
        Popularity : $popularity
        Release Date : $release_date
        Votes: $vote_count");
    } 
    else {
        send_message($chat_id, "Couldn't find any movie with that name!");
    }
}

function send_message ($chat_id, $message) {
    $apiToken = <TELEGRAM_BOT_TOKEN>;
    $text = urlencode($message);
    file_get_contents("https://api.telegram.org/bot$apiToken/sendMessage?chat_id=$chat_id&text=$text");
}
?>

Thanks for reading.

Git is pretty much used by every developer nowadays. Personally, even after using it for approximately two years, I continue to discover new git features. So here are some of the commands I discovered, and seems they aren't popular enough. Without wasting time, let's dive into it.

Check logs but in easier ways

git log a useful tool for looking back and reading the history of every action taken with a repository. A git log's multiple settings can be utilized to narrow down history. The git log typically contains a list of commits. And sometimes the list is so big it's hard to find anything we need. Let me show you an easier way to do the same, but the output is short yet brief.

git log --oneline

It's super helpful to check the latest commits. With the --oneline flag, each commit is reduced to a single line. It just shows the commit ID and the first line of the commit message by default. The output should look like:

1266f06 (origin/develop) updated readme
33bf5fb Merge branch 'develop'
bd99bba fixed error
19389a7 Merge pull request #5 from akashrchandran/develop
5a2258e added exceptions
9dbb9de updated readme

git shortlog

A modified version of the git log command called git shortlog is used mostly to make release notifications. It displays the first line of each commit message and groups each commit according to author. This makes it simple to determine who has been working on what.

Akash R Chandran (4):
      Initial commit
      fixed all errors
      added gitignore
      Added support for heroku deployment

Paulo Martini(3):
      added files via upload
      updated
      fixed requests errors

git log --graph

The branch structure of the commit history is represented via an ASCII graph when the --graph option is used. To make it simpler to determine which commit belongs to which branch, this is frequently used in conjunction with the --oneline and --decorate commands:

*   64f95a3 (HEAD -> main, origin/main, origin/HEAD) Merge pull request #6 from akashrchandran/develop
|\
| * 1266f06 (origin/develop) updated readme
* | 33bf5fb Merge branch 'develop'
|\|
| * bd99bba fixed error
* | 19389a7 Merge pull request #5 from akashrchandran/develop
|\|
| * 5a2258e added exceptions
|/
* 9dbb9de updated readme

To get more info on git log you can visit here.

Switching branches

The git checkout command can be used for a wide range of purposes. Because of this, the Git community has decided to release a new command: git switch. It was designed primarily for the process of switching branches, as the name suggests.

$ git switch develop
Switched to a new branch 'develop'
branch 'develop' set up to track 'origin/develop'.

Switch back and forth between two branches

git switch makes it very easy to switch between the same branches back to back. It doesn't even need the branch name to switch, just remove branch name and put a hyphen(-):

$ git switch -
Switched to branch 'main'
Your branch is up to date with 'origin/main'.

Using the same command again will switch back to develop.

Checking the changes made to file

The function of diffing outputs the differences between two input data sets. A diff operation is performed on Git data sources when the Git command git diff is invoked. Commits, branches, files, and other types of data sources are examples. This document will go over typical git diff commands and diffing work flow patterns. git diff, git status, and git log are frequently used to examine the present state of a Git repository.

$ git diff 
diff --git a/spotify.php b/spotify.php
index 6987fef..bc766cf 100644
--- a/spotify.php
+++ b/spotify.php
@@ -64,6 +64,7 @@ class Spotify
                curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
                curl_setopt($ch, CURLOPT_URL, $formated_url);
                $result = curl_exec($ch);
+               echo $result;
                return $result;
        }
 }

Restore to an older version of the file

Another extremely useful option available with the git restore command is "—source." You can quickly restore any earlier version of a particular file with the use of this option:

git restore --source 6bcf266b index.html

Adding comments to your commits

To preserve a better development history, developers typically write clear and self-explanatory commit notes. Thanks to well-liked commit conventions, some development teams even automatically produce release notes with commit messages. How can I include some extra notes in a commit? Commit message additions will be reflected on remote Git hosting GUIs. How can I record a message for a CI/CD server or add a hidden note to remember about any technical work? We can add extra comments for commits with the help of the built-in Git notes functionality.

To connect a note to the current commit, issue the following command:

$ git notes add -m "This is a test comment"

By omitting the -m option, you can use the editor programme to write a multiline, lengthy note. The git notes show command allows you to view the most recent note. Additionally, the git log command by default displays notes. Git notes were previously supported by GitHub as comments, but they are now deprecated in favour of web-based comments.

Git does not automatically push or pull notes, but you can manage remote notes explicitly using the following commands:

git push <remote> refs/notes/*
git fetch origin refs/notes/*:refs/notes/*

Print the SHA1 hashes

git rev-parse is an ancillary plumbing command primarily used for manipulation. One common usage of git rev-parse is to print the SHA1 hashes given a revision specifier. In addition, it has various options to format this output, such as --short for printing a shorter unique SHA1.

$ git rev-parse HEAD
64f95a3edf8ecaf1f874b9a25fb7312a722f463a

Conclusion

These commands are mostly for better productivity and simplicity. It's good if you know what they do. Git is well known these days, but many of its powers are still unknown to the public. It's true that you can "survive" with a handful of commands like commit, push, and pull.

Thanks for reading 😊.