Pulling data from a PostgreSQL table and rendering to a web page using Django
Question:
I am unable to pull data from a newly populated PostgreSQL table and render it on a web page in my Django application. I have a small application where a user uploads a .json file. The app performs some operations on the json, resulting in a Pandas dataframe. This dataframe then populates a PostgreSQL data table, resulting in one column of data whose length corresponds to the initial data operations. So far, this works correctly.
I have checked the postgres database to verify that the table has been populated, and it has. However, I’m having trouble pulling that column of data from the database and displaying it on the same page that the user uploaded the data from. I’m unsure why I can’t easily retrieve the data from the database, loop over it, and render it to the page. The error message tells me that my app "id" does not exist, despite including an "id" primary key in my models.py.
As you can see, in my views.py file, I have a ProcessedDataListView class that is intended to create the context and send data to my data_display.html file. Here is my error message:
In template
/home/linux/django_app/eef/myapp/templates/upload_json.html, error at
line 12
column myapp_processeddatamodel.id does not exist LINE 1: SELECT
"myapp_processeddatamodel"."id", "myapp_processeddata… ^
PostgreSQL log
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx STATEMENT: SELECT "myapp_processeddatamodel"."id", "myapp_processeddatamodel"."my_list" FROM "myapp_processeddatamodel" LIMIT 21
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx ERROR: column myapp_processeddatamodel.id does not exist at character 8
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx STATEMENT: SELECT "myapp_processeddatamodel"."id", "myapp_processeddatamodel"."my_list" FROM "myapp_processeddatamodel" LIMIT 21
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx ERROR: column myapp_processeddatamodel.id does not exist at character 8
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx STATEMENT: SELECT "myapp_processeddatamodel"."id", "myapp_processeddatamodel"."my_list" FROM "myapp_processeddatamodel" LIMIT 21
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx ERROR: column myapp_processeddatamodel.id does not exist at character 8
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx STATEMENT: SELECT "myapp_processeddatamodel"."id", "myapp_processeddatamodel"."my_list" FROM "myapp_processeddatamodel" LIMIT 21
2023-03-18 11:15:19.783 GMT [9694] linux@mydbx ERROR: column myapp_processeddatamodel.id does not exist at character 8
2023-03-18 11:15:19.783 GMT [9694] linux@mydbx STATEMENT: SELECT "myapp_processeddatamodel"."id", "myapp_processeddatamodel"."my_list" FROM "myapp_processeddatamodel" LIMIT 21
Database structure
from django.db import models
class AuthGroup(models.Model):
name = models.CharField(unique=True, max_length=150)
class Meta:
managed = False
db_table = 'auth_group'
class AuthGroupPermissions(models.Model):
id = models.BigAutoField(primary_key=True)
group = models.ForeignKey(AuthGroup, models.DO_NOTHING)
permission = models.ForeignKey('AuthPermission', models.DO_NOTHING)
class Meta:
managed = False
db_table = 'auth_group_permissions'
unique_together = (('group', 'permission'),)
class AuthPermission(models.Model):
name = models.CharField(max_length=255)
content_type = models.ForeignKey('DjangoContentType', models.DO_NOTHING)
codename = models.CharField(max_length=100)
class Meta:
managed = False
db_table = 'auth_permission'
unique_together = (('content_type', 'codename'),)
class AuthUser(models.Model):
password = models.CharField(max_length=128)
last_login = models.DateTimeField(blank=True, null=True)
is_superuser = models.BooleanField()
username = models.CharField(unique=True, max_length=150)
first_name = models.CharField(max_length=150)
last_name = models.CharField(max_length=150)
email = models.CharField(max_length=254)
is_staff = models.BooleanField()
is_active = models.BooleanField()
date_joined = models.DateTimeField()
class Meta:
managed = False
db_table = 'auth_user'
class AuthUserGroups(models.Model):
id = models.BigAutoField(primary_key=True)
user = models.ForeignKey(AuthUser, models.DO_NOTHING)
group = models.ForeignKey(AuthGroup, models.DO_NOTHING)
class Meta:
managed = False
db_table = 'auth_user_groups'
unique_together = (('user', 'group'),)
class AuthUserUserPermissions(models.Model):
id = models.BigAutoField(primary_key=True)
user = models.ForeignKey(AuthUser, models.DO_NOTHING)
permission = models.ForeignKey(AuthPermission, models.DO_NOTHING)
class Meta:
managed = False
db_table = 'auth_user_user_permissions'
unique_together = (('user', 'permission'),)
class DjangoAdminLog(models.Model):
action_time = models.DateTimeField()
object_id = models.TextField(blank=True, null=True)
object_repr = models.CharField(max_length=200)
action_flag = models.SmallIntegerField()
change_message = models.TextField()
content_type = models.ForeignKey('DjangoContentType', models.DO_NOTHING, blank=True, null=True)
user = models.ForeignKey(AuthUser, models.DO_NOTHING)
class Meta:
managed = False
db_table = 'django_admin_log'
class DjangoContentType(models.Model):
app_label = models.CharField(max_length=100)
model = models.CharField(max_length=100)
class Meta:
managed = False
db_table = 'django_content_type'
unique_together = (('app_label', 'model'),)
class DjangoMigrations(models.Model):
id = models.BigAutoField(primary_key=True)
app = models.CharField(max_length=255)
name = models.CharField(max_length=255)
applied = models.DateTimeField()
class Meta:
managed = False
db_table = 'django_migrations'
class DjangoSession(models.Model):
session_key = models.CharField(primary_key=True, max_length=40)
session_data = models.TextField()
expire_date = models.DateTimeField()
class Meta:
managed = False
db_table = 'django_session'
class MyappProcesseddatamodel(models.Model):
number_0 = models.TextField(db_column='0', blank=True, null=True) # Field renamed because it wasn't a valid Python identifier.
class Meta:
managed = False
db_table = 'myapp_processeddatamodel'
Views.py
from django.shortcuts import render, redirect
from .forms import JsonFileUploadForm
from .models import ProcessedDataModel
from django.db import models
import pandas as pd
from sqlalchemy import create_engine
from django.shortcuts import render
from .models import ProcessedDataModel
import json
publication_type_output = [{
5215227: 'Journal article',
5215228: 'Dissertation or thesis',
5215229: 'Technical report',
5215230: 'Book or book chapter',
5215231: 'Conference paper'
}]
def upload_json(request):
# Clear the database before processing new data
# clear_database()
if request.method == 'POST' and 'upload_button' in request.POST:
form = JsonFileUploadForm(request.POST, request.FILES)
if form.is_valid():
# load the file data from the TemporaryUploadedFile object
data = json.load(request.FILES['json_file'])
processed_data = get_data(data, publication_type_output)
# Replace these values with your actual PostgreSQL credentials
db_url = 'postgresql://linux:mypass@localhost/mydbx'
# Create an SQLAlchemy engine
engine = create_engine(db_url)
# Replace 'your_table_name' with the actual table name you want to use
processed_data.to_sql('myapp_processeddatamodel', engine, if_exists='replace', index=False)
# Fetch all rows from the table
all_data = ProcessedDataModel.objects.all()
context = {'form': form, 'data': all_data}
return render(request, 'upload_json.html', context)
else:
form = JsonFileUploadForm()
context = {'form': form}
return render(request, 'upload_json.html', context)
def processed_data_view(request):
processed_data = ProcessedDataModel.objects.all()
return render(request, 'processed_data.html', {'processed_data': processed_data})
# GET DATA FROM DATABASE AND PASS IT TO A TEMPLATE CONTEXT
from django.views.generic import ListView
from .models import ProcessedDataModel
class ProcessedDataListView(ListView):
model = ProcessedDataModel
template_name = 'upload_json.html'
context_object_name = 'data'
app level urls.py
from django.urls import path
from . import views
from .views import ProcessedDataListView
app_name = 'myapp'
urlpatterns = [
path('upload-json/', views.upload_json, name='upload_json'),
path('processed-data/', views.processed_data_view, name='processed_data_view'),
path('upload-json/', views.ProcessedDataListView.as_view(), name='upload-json'),
]
models.py
from django.db import models
from django.contrib.postgres.fields import ArrayField
class ProcessedDataModel(models.Model):
id = models.AutoField(primary_key=True)
my_list = models.TextField()
def __str__(self):
return self.my_list
upload_json.html
{% block content %}
<h1>Upload JSON File</h1>
<form method="post" enctype="multipart/form-data">
{% csrf_token %}
<div class="form-group">
{{ form.json_file.label_tag }}
{{ form.json_file }}
</div>
<button type="submit" name="upload_button" class="btn btn-primary">Upload</button>
</form>
{% if data %}
<h2>Data:</h2>
<ul>
{% for item in data %}
<li>{{ item.my_list }}</li>
{% endfor %}
</ul>
{% endif %}
{% endblock %}
I have already tried removing the old PostgreSQL table and database and starting them afresh, though this resulted in the same problem.
Answers:
Answering myself as I have now found the problem and the solution.
I had a PostGres table setup with a colunm named "x", ready to be populated. I was attempting to populate this column with a Pandas dataframe, though I was mistakenly not specifying a column name (which should have been "x"). For this reason it was being assigned the name 0, which of course did not match "x". This meant that there was no information for me to pull out of "x" to display on the page, hence why there was no display. I now pass in a dataframe with the same column name as that of my corresponding database column name, and the data is populated and displayed on the page with no problems.
I am unable to pull data from a newly populated PostgreSQL table and render it on a web page in my Django application. I have a small application where a user uploads a .json file. The app performs some operations on the json, resulting in a Pandas dataframe. This dataframe then populates a PostgreSQL data table, resulting in one column of data whose length corresponds to the initial data operations. So far, this works correctly.
I have checked the postgres database to verify that the table has been populated, and it has. However, I’m having trouble pulling that column of data from the database and displaying it on the same page that the user uploaded the data from. I’m unsure why I can’t easily retrieve the data from the database, loop over it, and render it to the page. The error message tells me that my app "id" does not exist, despite including an "id" primary key in my models.py.
As you can see, in my views.py file, I have a ProcessedDataListView class that is intended to create the context and send data to my data_display.html file. Here is my error message:
In template
/home/linux/django_app/eef/myapp/templates/upload_json.html, error at
line 12column myapp_processeddatamodel.id does not exist LINE 1: SELECT
"myapp_processeddatamodel"."id", "myapp_processeddata… ^
PostgreSQL log
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx STATEMENT: SELECT "myapp_processeddatamodel"."id", "myapp_processeddatamodel"."my_list" FROM "myapp_processeddatamodel" LIMIT 21
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx ERROR: column myapp_processeddatamodel.id does not exist at character 8
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx STATEMENT: SELECT "myapp_processeddatamodel"."id", "myapp_processeddatamodel"."my_list" FROM "myapp_processeddatamodel" LIMIT 21
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx ERROR: column myapp_processeddatamodel.id does not exist at character 8
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx STATEMENT: SELECT "myapp_processeddatamodel"."id", "myapp_processeddatamodel"."my_list" FROM "myapp_processeddatamodel" LIMIT 21
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx ERROR: column myapp_processeddatamodel.id does not exist at character 8
2023-03-18 11:15:19.782 GMT [9694] linux@mydbx STATEMENT: SELECT "myapp_processeddatamodel"."id", "myapp_processeddatamodel"."my_list" FROM "myapp_processeddatamodel" LIMIT 21
2023-03-18 11:15:19.783 GMT [9694] linux@mydbx ERROR: column myapp_processeddatamodel.id does not exist at character 8
2023-03-18 11:15:19.783 GMT [9694] linux@mydbx STATEMENT: SELECT "myapp_processeddatamodel"."id", "myapp_processeddatamodel"."my_list" FROM "myapp_processeddatamodel" LIMIT 21
Database structure
from django.db import models
class AuthGroup(models.Model):
name = models.CharField(unique=True, max_length=150)
class Meta:
managed = False
db_table = 'auth_group'
class AuthGroupPermissions(models.Model):
id = models.BigAutoField(primary_key=True)
group = models.ForeignKey(AuthGroup, models.DO_NOTHING)
permission = models.ForeignKey('AuthPermission', models.DO_NOTHING)
class Meta:
managed = False
db_table = 'auth_group_permissions'
unique_together = (('group', 'permission'),)
class AuthPermission(models.Model):
name = models.CharField(max_length=255)
content_type = models.ForeignKey('DjangoContentType', models.DO_NOTHING)
codename = models.CharField(max_length=100)
class Meta:
managed = False
db_table = 'auth_permission'
unique_together = (('content_type', 'codename'),)
class AuthUser(models.Model):
password = models.CharField(max_length=128)
last_login = models.DateTimeField(blank=True, null=True)
is_superuser = models.BooleanField()
username = models.CharField(unique=True, max_length=150)
first_name = models.CharField(max_length=150)
last_name = models.CharField(max_length=150)
email = models.CharField(max_length=254)
is_staff = models.BooleanField()
is_active = models.BooleanField()
date_joined = models.DateTimeField()
class Meta:
managed = False
db_table = 'auth_user'
class AuthUserGroups(models.Model):
id = models.BigAutoField(primary_key=True)
user = models.ForeignKey(AuthUser, models.DO_NOTHING)
group = models.ForeignKey(AuthGroup, models.DO_NOTHING)
class Meta:
managed = False
db_table = 'auth_user_groups'
unique_together = (('user', 'group'),)
class AuthUserUserPermissions(models.Model):
id = models.BigAutoField(primary_key=True)
user = models.ForeignKey(AuthUser, models.DO_NOTHING)
permission = models.ForeignKey(AuthPermission, models.DO_NOTHING)
class Meta:
managed = False
db_table = 'auth_user_user_permissions'
unique_together = (('user', 'permission'),)
class DjangoAdminLog(models.Model):
action_time = models.DateTimeField()
object_id = models.TextField(blank=True, null=True)
object_repr = models.CharField(max_length=200)
action_flag = models.SmallIntegerField()
change_message = models.TextField()
content_type = models.ForeignKey('DjangoContentType', models.DO_NOTHING, blank=True, null=True)
user = models.ForeignKey(AuthUser, models.DO_NOTHING)
class Meta:
managed = False
db_table = 'django_admin_log'
class DjangoContentType(models.Model):
app_label = models.CharField(max_length=100)
model = models.CharField(max_length=100)
class Meta:
managed = False
db_table = 'django_content_type'
unique_together = (('app_label', 'model'),)
class DjangoMigrations(models.Model):
id = models.BigAutoField(primary_key=True)
app = models.CharField(max_length=255)
name = models.CharField(max_length=255)
applied = models.DateTimeField()
class Meta:
managed = False
db_table = 'django_migrations'
class DjangoSession(models.Model):
session_key = models.CharField(primary_key=True, max_length=40)
session_data = models.TextField()
expire_date = models.DateTimeField()
class Meta:
managed = False
db_table = 'django_session'
class MyappProcesseddatamodel(models.Model):
number_0 = models.TextField(db_column='0', blank=True, null=True) # Field renamed because it wasn't a valid Python identifier.
class Meta:
managed = False
db_table = 'myapp_processeddatamodel'
Views.py
from django.shortcuts import render, redirect
from .forms import JsonFileUploadForm
from .models import ProcessedDataModel
from django.db import models
import pandas as pd
from sqlalchemy import create_engine
from django.shortcuts import render
from .models import ProcessedDataModel
import json
publication_type_output = [{
5215227: 'Journal article',
5215228: 'Dissertation or thesis',
5215229: 'Technical report',
5215230: 'Book or book chapter',
5215231: 'Conference paper'
}]
def upload_json(request):
# Clear the database before processing new data
# clear_database()
if request.method == 'POST' and 'upload_button' in request.POST:
form = JsonFileUploadForm(request.POST, request.FILES)
if form.is_valid():
# load the file data from the TemporaryUploadedFile object
data = json.load(request.FILES['json_file'])
processed_data = get_data(data, publication_type_output)
# Replace these values with your actual PostgreSQL credentials
db_url = 'postgresql://linux:mypass@localhost/mydbx'
# Create an SQLAlchemy engine
engine = create_engine(db_url)
# Replace 'your_table_name' with the actual table name you want to use
processed_data.to_sql('myapp_processeddatamodel', engine, if_exists='replace', index=False)
# Fetch all rows from the table
all_data = ProcessedDataModel.objects.all()
context = {'form': form, 'data': all_data}
return render(request, 'upload_json.html', context)
else:
form = JsonFileUploadForm()
context = {'form': form}
return render(request, 'upload_json.html', context)
def processed_data_view(request):
processed_data = ProcessedDataModel.objects.all()
return render(request, 'processed_data.html', {'processed_data': processed_data})
# GET DATA FROM DATABASE AND PASS IT TO A TEMPLATE CONTEXT
from django.views.generic import ListView
from .models import ProcessedDataModel
class ProcessedDataListView(ListView):
model = ProcessedDataModel
template_name = 'upload_json.html'
context_object_name = 'data'
app level urls.py
from django.urls import path
from . import views
from .views import ProcessedDataListView
app_name = 'myapp'
urlpatterns = [
path('upload-json/', views.upload_json, name='upload_json'),
path('processed-data/', views.processed_data_view, name='processed_data_view'),
path('upload-json/', views.ProcessedDataListView.as_view(), name='upload-json'),
]
models.py
from django.db import models
from django.contrib.postgres.fields import ArrayField
class ProcessedDataModel(models.Model):
id = models.AutoField(primary_key=True)
my_list = models.TextField()
def __str__(self):
return self.my_list
upload_json.html
{% block content %}
<h1>Upload JSON File</h1>
<form method="post" enctype="multipart/form-data">
{% csrf_token %}
<div class="form-group">
{{ form.json_file.label_tag }}
{{ form.json_file }}
</div>
<button type="submit" name="upload_button" class="btn btn-primary">Upload</button>
</form>
{% if data %}
<h2>Data:</h2>
<ul>
{% for item in data %}
<li>{{ item.my_list }}</li>
{% endfor %}
</ul>
{% endif %}
{% endblock %}
I have already tried removing the old PostgreSQL table and database and starting them afresh, though this resulted in the same problem.
Answering myself as I have now found the problem and the solution.
I had a PostGres table setup with a colunm named "x", ready to be populated. I was attempting to populate this column with a Pandas dataframe, though I was mistakenly not specifying a column name (which should have been "x"). For this reason it was being assigned the name 0, which of course did not match "x". This meant that there was no information for me to pull out of "x" to display on the page, hence why there was no display. I now pass in a dataframe with the same column name as that of my corresponding database column name, and the data is populated and displayed on the page with no problems.