Export big amount of data using Celery With Django

Export big amount of data using Celery With Django

In this tutorial we are going to build Random Posts Generator app with Django using Celery and RabbitMQ.

Why should we use Celery?

Web applications usually start out simple but can become quite complex, and most of them quickly exceed the responsibility of only responding to HTTP requests. Assume that your web application sends confirmation mail to each registered user. Once your application overloaded with traffic, your web server can only handle certain number of requests and leave the user waiting for way too long.

At a certain point SQLite becomes too "lite" for real-world applications, if you are planning to use heavy tasks in your web application it is better to switch another database backend to avoid any errors.

Celery is the best choice for doing background task processing in the Python/Django ecosystem. It has a simple and clear API, and it integrates beautifully with Django. So, we are using Celery to handle the time-consuming tasks by passing them to queue to be executed in the background and always keep the server ready to respond to new requests.

Random Posts Generator

We are going to build random posts generator app which will generate hundreds of posts at once and let's see how Celery will handle this task.

Celery requires a solution to send and receive messages; usually this comes in the form of a separate service called a message broker. We will be configuring celery to use the RabbitMQ messaging system, as it provides robust, stable performance and interacts well with celery.

We can install RabbitMQ through Ubuntu’s repositories by following command:

sudo apt-get install rabbitmq-server

Then enable and start the RabbitMQ service:

sudo systemctl enable rabbitmq-server
sudo systemctl start rabbitmq-server

Install RabbitMQ on Mac

Create Django project named randompostgenerator with an app named posts and run the following command to install Celery:

pip3 install Celery

Once installation completed, add the CELERY_BROKER_URL configuration to the settings.py file:

CELERY_BROKER_URL = 'amqp://localhost'

Setting up the username, password and vhost.

Then, create celery.py inside your project.

celery celery.py
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'randompostgenerator.settings')

app = Celery('randompostgenerator')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

We are setting the default Django settings module for the 'celery' program and loading task modules from all registered Django app configs.

Now inside your __init__.py import the celery:

from .celery import app as celery_app

__all__ = ['celery_app']

This will make sure our Celery app loaded every time Django starts.

Alright! Let's create our Post model in models.py:

from django.db import models

class Post(models.Model):
    title = models.CharField(max_length=200)
    description = models.TextField()
    author = models.CharField(max_length=200)

    def __str__(self):
        return self.title

We are going to use Python package named Faker which will generate fake data for our model. Install the Faker by following command:

pip3 install Faker

Now, create new file named tasks.py inside posts app and create a Celery task that generates a number of random posts.

from faker import Faker
from .models import Post
from celery import shared_task

@shared_task
def create_random_posts(number_of_posts):
    fake = Faker()
    for i in range(number_of_posts):
        title = fake.sentence()
        description = fake.text()
        author = fake.name()
        Post.objects.create(
            title = title,
            description = description,
            author = author

        )

    return f'{number_of_posts} posts successfully generated!'

@shared_task will create the independent instance of the task for each app, making task reusable. This makes the @shared_task decorator useful for libraries and reusable apps, since they will not have access to the app of the user.

Faker will generate fake data for model fields. Learn more about Faker

Then in out forms.py:

from django import forms
from django.core.validators import MinValueValidator, MaxValueValidator

class RandomPostGeneratorForm(forms.Form):
    number_of_posts = forms.IntegerField(
        validators=[
            MinValueValidator(50),
            MaxValueValidator(500)
        ]
    )

This form will expect a positive integer field between 50 and 500.

celery

Then, in our views.py:

from django.shortcuts import render, redirect
from django.contrib import messages

from .models import Post
from .forms import RandomPostGeneratorForm
from .tasks import create_random_posts

def posts_view(request):
    posts = Post.objects.all()
    return render(request, 'posts.html', {'posts':posts})


def generate_view(request):
    form = RandomPostGeneratorForm(request.POST)
    if form.is_valid():
        number_of_posts = form.cleaned_data.get('number_of_posts')
        #executing this function in the background
        create_random_posts.delay(number_of_posts)
        messages.success(request, 'Random posts generated! Refresh the page after few seconds to see the result.')
        return redirect('posts')
    return render(request, 'generate.html', {'form':form})

By calling create_random_posts with delay() method, we are instructing Celery to execute this function in the background.

Finally, let's see templates:

base.html
<html>
<head>
  <meta charset="utf-8">
  <title>Random</title>
  <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">
</head>
<body>
    <div class="container py-4">
    <h1>Random Post Generator</h1>
    <a href="{% url 'posts' %}">Post List</a> |
    <a href="{% url 'generate' %}">Generate Random Posts</a>
    <hr>
    {% if messages %}
        {% for message in messages %}
        <div class="alert alert-success" role="alert">
            {{message}}
        </div>
        {% endfor %}
        <hr>
    {% endif %}

    {% block content %}
    {% endblock %}
    </div>
</body>
</html>

generate.html
{% extends 'base.html' %}

{% block content %}
  <h2>Generate Random Posts</h2>
  <form method="post">
    {% csrf_token %}
    {% for field in form %}
      <div class="form-group">{{ field }}</div>
    {% endfor %}
    <button class="btn btn-primary" type="submit">Generate</button>
  </form>
{% endblock %}

posts.html
{% extends 'base.html' %}

{% block content %}
  <h2>Posts List</h2>

  <ul>
    {% for post in posts %}
        <h3 class="mt-5">{{post.title}}</h3>
        <p class="lead">{{post.description|truncatewords:25}}</p>
        <p>{{post.author}}</p>
    {% endfor %}
  </ul>
{% endblock %}

Great! Now it is time to start the Celery worker process. Open your terminal and run the following command:

celery -A randompostgenerator worker -l info

Change the randompostgenerator to your project name if it is different. After running command, your terminal should look similar to this:

celery

Also don't forget to run your Django project. Here is the final result:

celery with django

You can clone or download the project on my GitHub

Random Posts Generator GitHub

That's it! Make sure you are following me on social media and please support me by buying me a coffee☕ so I can upload more tutorials like this. See you in next post DEVs!

Instagram

Twitter