How to implement datatable server side from this code using django
Question:
Currently I have this code that shows me the normal data table but if I have millions of records it becomes very slow, I understand that serverside of datatable can be used but I don’t know how I can implement it
list.html
<table class="table table-striped table-hover responsive" style="width:100%" id="buyer">
<thead>
<th>#id</th>
<th>Fecha</th>
<th>Cod Cliente</th>
<th>Cliente</th>
<th>Dirección</th>
<th>Comentario</th>
<th>Tipo Retiro</th>
<th>Total</th>
<th>Guias a Generar</th>
</thead>
<tbody>
{% for item in ResulOrdenes %}
<tr>
<td>{{item.DOCNUM}}</td>
<td>{{item.DOC_DATE|date:"Y-m-d"}}</td>
<td>{{ item.CARDCODE }}</td>
<td>{{ item.CARDNAME }}</td>
<td>{{ item.ADDRESS }}</td>
<td>{{ item.COMMENTS }}</td>
<td>{{ item.URETIRO }}</td>
<td>{{ item.DOCTOTAL }}</td>
<td>{{ item.NGUIAS }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% endblock %}
{% block content_wrapper %}
<p></p>
{% endblock content_wrapper %}
{% block js_page %}
<script>
$(document).ready(function() {
$('.table').DataTable({
dom: 'Bfrtip',
});
</script>
{% endblock %}
view.py
def Listado_guia(request):
template_name='app_listguia/lista.html'
conn = dbapi.connect(..)
cursorSol2 = conn.cursor()
sql_command = 'select a."DocNum" as "DOCNUM", a."DocDate" as "DOC_DATE", a."CardCode" as "CARDCODE", a."CardName" as "CARDNAME", a."DocTotal" as "DOCTOTAL", CEILING(a."DocTotal"/20000) as "NGUIAS", a."DocStatus" as "DOCSTATUS", a."Address" as "ADDRESS", a."Address2" as "ADDRESS2", a."Comments" as "COMMENTS", a."U_Retiro_mercaderia" as "URETIRO" '
sql_command = sql_command + 'from "'+ connections.databases[ConfigBaseHana]["NAME"] +'"."ODLN" as a '
cursorSol2.execute(sql_command)
RES_ORDENES = cursorSol2.fetchall()
cursorSol2.close()
conn.close()
dnow = today.strftime("%Y-%m-%d")
contexto = { "ResulOrdenes":RES_ORDENES, "dnow":dnow }
return render(request,template_name,contexto)
urls.py
from django.urls import path
from django.contrib.auth import views as auth_views
from app_guia.views import *
urlpatterns = [
path('guiaList',Listado_guia,name='urlListGuia_list'),
]
Can you please give me some light on how to proceed? Thanks
Answers:
The approach you are using is not suitable for millions of records. First of all, if this is your complete sql, you should stick with normal Django QuerySets and switch your view to use a (generic) class-based view. This gives you a proper paginator, which will reduce the load on the database, django and the resulting html significantly.
If you do not want to have pagination, but rather a continuous list, you need a javascript component to virtually render your table. This way, only a few hundred DOM nodes are present in the clients html, and will be populated from server, on the fly, while scrolling. The term to search for is "virtualized table", an examplatory candidate would be react-window.
On a sidenote: if the code shown is your "real" production code (and not a boiled-down demo version for this post), I highly recommend working through a django tutorial, e.g. here or here, and sticking to the default django way as documented in the djangoproject as much as possible. You will save yourself hours of programming and huge problems with future enhancements.
To improve the performance of your data table, you can use the server-side processing mode of the DataTables library.
- Add the
serverSide option to your DataTable initialization code and
set it to true:
$(document).ready(function() {
$('.table').DataTable({
serverSide: true,
processing: true,
ajax: '/guiaList',
dom: 'Bfrtip',
});
});
- In your view function, you need to handle the AJAX request and return
the data in the expected format. You can use the JsonResponse class
from Django to return the data in JSON format
from django.http import JsonResponse
from django.core.paginator import Paginator
def Listado_guia(request):
draw = int(request.GET.get('draw', 1))
start = int(request.GET.get('start', 0))
length = int(request.GET.get('length', 10))
conn = dbapi.connect(...)
cursorSol2 = conn.cursor()
sql_command = 'SELECT ...'
cursorSol2.execute(sql_command)
total_items = cursorSol2.rowcount
results = cursorSol2.fetchall()
cursorSol2.close()
conn.close()
paginator = Paginator(results, length)
page_results = paginator.get_page(int(start/length)+1)
data = []
for item in page_results:
data.append({
'DOCNUM': item[0],
'DOC_DATE': item[1].strftime("%Y-%m-%d"),
'CARDCODE': item[2],
'CARDNAME': item[3],
'DOCTOTAL': item[4],
'NGUIAS': item[5],
'DOCSTATUS': item[6],
'ADDRESS': item[7],
'ADDRESS2': item[8],
'COMMENTS': item[9],
'URETIRO': item[10]
})
response = {
'draw': draw,
'recordsTotal': total_items,
'recordsFiltered': total_items,
'data': data
}
return JsonResponse(response)
The start, length, and draw parameters are sent automatically by DataTables to the server when requesting data. The start parameter indicates the start index of the current page, the length parameter indicates the number of rows to display on the current page, and the draw parameter is used to ensure that the response matches the current state of the table.
The JsonResponse returns the data in the format expected by DataTables, which includes the draw parameter, the total number of records (recordsTotal), the number of filtered records (recordsFiltered), and the data array.
Currently I have this code that shows me the normal data table but if I have millions of records it becomes very slow, I understand that serverside of datatable can be used but I don’t know how I can implement it
list.html
<table class="table table-striped table-hover responsive" style="width:100%" id="buyer">
<thead>
<th>#id</th>
<th>Fecha</th>
<th>Cod Cliente</th>
<th>Cliente</th>
<th>Dirección</th>
<th>Comentario</th>
<th>Tipo Retiro</th>
<th>Total</th>
<th>Guias a Generar</th>
</thead>
<tbody>
{% for item in ResulOrdenes %}
<tr>
<td>{{item.DOCNUM}}</td>
<td>{{item.DOC_DATE|date:"Y-m-d"}}</td>
<td>{{ item.CARDCODE }}</td>
<td>{{ item.CARDNAME }}</td>
<td>{{ item.ADDRESS }}</td>
<td>{{ item.COMMENTS }}</td>
<td>{{ item.URETIRO }}</td>
<td>{{ item.DOCTOTAL }}</td>
<td>{{ item.NGUIAS }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% endblock %}
{% block content_wrapper %}
<p></p>
{% endblock content_wrapper %}
{% block js_page %}
<script>
$(document).ready(function() {
$('.table').DataTable({
dom: 'Bfrtip',
});
</script>
{% endblock %}
view.py
def Listado_guia(request):
template_name='app_listguia/lista.html'
conn = dbapi.connect(..)
cursorSol2 = conn.cursor()
sql_command = 'select a."DocNum" as "DOCNUM", a."DocDate" as "DOC_DATE", a."CardCode" as "CARDCODE", a."CardName" as "CARDNAME", a."DocTotal" as "DOCTOTAL", CEILING(a."DocTotal"/20000) as "NGUIAS", a."DocStatus" as "DOCSTATUS", a."Address" as "ADDRESS", a."Address2" as "ADDRESS2", a."Comments" as "COMMENTS", a."U_Retiro_mercaderia" as "URETIRO" '
sql_command = sql_command + 'from "'+ connections.databases[ConfigBaseHana]["NAME"] +'"."ODLN" as a '
cursorSol2.execute(sql_command)
RES_ORDENES = cursorSol2.fetchall()
cursorSol2.close()
conn.close()
dnow = today.strftime("%Y-%m-%d")
contexto = { "ResulOrdenes":RES_ORDENES, "dnow":dnow }
return render(request,template_name,contexto)
urls.py
from django.urls import path
from django.contrib.auth import views as auth_views
from app_guia.views import *
urlpatterns = [
path('guiaList',Listado_guia,name='urlListGuia_list'),
]
Can you please give me some light on how to proceed? Thanks
The approach you are using is not suitable for millions of records. First of all, if this is your complete sql, you should stick with normal Django QuerySets and switch your view to use a (generic) class-based view. This gives you a proper paginator, which will reduce the load on the database, django and the resulting html significantly.
If you do not want to have pagination, but rather a continuous list, you need a javascript component to virtually render your table. This way, only a few hundred DOM nodes are present in the clients html, and will be populated from server, on the fly, while scrolling. The term to search for is "virtualized table", an examplatory candidate would be react-window.
On a sidenote: if the code shown is your "real" production code (and not a boiled-down demo version for this post), I highly recommend working through a django tutorial, e.g. here or here, and sticking to the default django way as documented in the djangoproject as much as possible. You will save yourself hours of programming and huge problems with future enhancements.
To improve the performance of your data table, you can use the server-side processing mode of the DataTables library.
- Add the
serverSideoption to your DataTable initialization code and
set it totrue:
$(document).ready(function() {
$('.table').DataTable({
serverSide: true,
processing: true,
ajax: '/guiaList',
dom: 'Bfrtip',
});
});
- In your view function, you need to handle the AJAX request and return
the data in the expected format. You can use theJsonResponseclass
from Django to return the data in JSON format
from django.http import JsonResponse
from django.core.paginator import Paginator
def Listado_guia(request):
draw = int(request.GET.get('draw', 1))
start = int(request.GET.get('start', 0))
length = int(request.GET.get('length', 10))
conn = dbapi.connect(...)
cursorSol2 = conn.cursor()
sql_command = 'SELECT ...'
cursorSol2.execute(sql_command)
total_items = cursorSol2.rowcount
results = cursorSol2.fetchall()
cursorSol2.close()
conn.close()
paginator = Paginator(results, length)
page_results = paginator.get_page(int(start/length)+1)
data = []
for item in page_results:
data.append({
'DOCNUM': item[0],
'DOC_DATE': item[1].strftime("%Y-%m-%d"),
'CARDCODE': item[2],
'CARDNAME': item[3],
'DOCTOTAL': item[4],
'NGUIAS': item[5],
'DOCSTATUS': item[6],
'ADDRESS': item[7],
'ADDRESS2': item[8],
'COMMENTS': item[9],
'URETIRO': item[10]
})
response = {
'draw': draw,
'recordsTotal': total_items,
'recordsFiltered': total_items,
'data': data
}
return JsonResponse(response)
The start, length, and draw parameters are sent automatically by DataTables to the server when requesting data. The start parameter indicates the start index of the current page, the length parameter indicates the number of rows to display on the current page, and the draw parameter is used to ensure that the response matches the current state of the table.
The JsonResponse returns the data in the format expected by DataTables, which includes the draw parameter, the total number of records (recordsTotal), the number of filtered records (recordsFiltered), and the data array.