Export big amount of data using Celery With Django
In this tutorial we are going to build Random Posts Generator app with Django using Celery and RabbitMQ.
Why should we use Celery?
Web applications usually start out simple but can become quite complex, and most of them quickly exceed the responsibility of only responding to HTTP requests. Assume that your web application sends confirmation mail to each registered user. Once your application overloaded with traffic, your web server can only handle certain number of requests and leave the user waiting for way too long.
At a certain point SQLite becomes too "lite" for real-world applications, if you are planning to use heavy tasks in your web application it is better to switch another database backend to avoid any errors.
Celery is the best choice for doing background task processing in the Python/Django ecosystem. It has a simple and clear API, and it integrates beautifully with Django. So, we are using Celery to handle the time-consuming tasks by passing them to queue to be executed in the background and always keep the server ready to respond to new requests.
Random Posts Generator
We are going to build random posts generator app which will generate hundreds of posts at once and let's see how Celery will handle this task.
Celery requires a solution to send and receive messages; usually this comes in the form of a separate service called a message broker. We will be configuring celery to use the RabbitMQ messaging system, as it provides robust, stable performance and interacts well with celery.
We can install RabbitMQ through Ubuntu’s repositories by following command:
Then enable and start the RabbitMQ service:
Create Django project named randompostgenerator with an app named posts and run the following command to install Celery:
Once installation completed, add the CELERY_BROKER_URL configuration to the settings.py file:
Then, create celery.py inside your project.celery.py
We are setting the default Django settings module for the 'celery' program and loading task modules from all registered Django app configs.
Now inside your __init__.py import the celery:
This will make sure our Celery app loaded every time Django starts.
Alright! Let's create our Post model in models.py:
We are going to use Python package named Faker which will generate fake data for our model. Install the Faker by following command:
Now, create new file named tasks.py inside posts app and create a Celery task that generates a number of random posts.
@shared_task will create the independent instance of the task for each app, making task reusable. This makes the @shared_task decorator useful for libraries and reusable apps, since they will not have access to the app of the user.
Faker will generate fake data for model fields. Learn more about Faker
Then in out forms.py:
This form will expect a positive integer field between 50 and 500.
Then, in our views.py:
By calling create_random_posts with delay() method, we are instructing Celery to execute this function in the background.
Finally, let's see templates:base.html
Great! Now it is time to start the Celery worker process. Open your terminal and run the following command:
Change the randompostgenerator to your project name if it is different. After running command, your terminal should look similar to this:
Also don't forget to run your Django project. Here is the final result:
You can clone or download the project on my GitHub
That's it! Make sure you are following me on social media and please support me by buying me a coffee☕ so I can upload more tutorials like this. See you in next post DEVs!