Sqlite3 DB is locked by an unknown process
Question:
I am trying to read an sqlite3 db into memory for further processing using python like so:
con = sqlite3.connect(':memory:')
print('Reading sqlite file into memory... ', end='', flush=True)
src_con.backup(con) # <---- This line seems to hang, SOMETIMES
print('Completed')
src_con.close()
cur = con.cursor()
While the code works fine most of the times, occasionally I observe a hang in the line src_con.backup(con).
This db is dumped by a process that I dont own on a shared network disk.
Here are my observations based on the advice I have found elsewhere on the internet:
fuser filename.db does not show any processes from my user accountsqlite3 filename.db "pragma integrity_check;" returns Error: database is locked- the md5sum of the
filename.db (the db that hangs) and its copy filename2.db (doesn’t hang) are identical. So is the OS locking the db – because the lock info is not in the DB file itself?
- This locked DB appears to occur when the process creating it did not exit cleanly.
Copying the db out (into say, filename2.db) and reading it is a workaround – but I want to understand whats going on and why – and if there is a way to read the locked DB in read-only mode.
Answers:
You’ve to close the database before you backup or restore. Btw, do you have right permissions to the source (src)? This related link might help:
How to back up a SQLite db?
The stance of the SQLite developers about using a database stored in a networked file system is, essentially, "Don’t. Use a client-server database instead."
This simple, "remote database" approach is usually not the best way to use a single SQLite database from multiple systems, (even if it appears to "work"), as it often leads to various kinds of trouble and grief. Because these problems are inevitable with some usages, but not frequent or repeatable, it behooves application developers to not rely on early testing success to decide that their remote database use will work as desired.
Sounds like you’re running into one of those problems. What I suspect is happening, because of
This locked DB appears to occur when the process creating it did not exit cleanly.
is that the networked file system you’re using isn’t detecting that the creating process’s lock on the database is gone due to the process no longer existing, so the backup is just waiting for the lock to be released so that it can acquire it.
Even if you figure out how to broadcast that the lock is available from the creating process’s computer to others with that file mounted, there’s bound to be other subtle and not-so-subtle problems that pop up now and then. So your best bet is to follow the official advice:
If your data lives on a different machine from your application, then you should consider a client/server database.
I am trying to read an sqlite3 db into memory for further processing using python like so:
con = sqlite3.connect(':memory:')
print('Reading sqlite file into memory... ', end='', flush=True)
src_con.backup(con) # <---- This line seems to hang, SOMETIMES
print('Completed')
src_con.close()
cur = con.cursor()
While the code works fine most of the times, occasionally I observe a hang in the line src_con.backup(con).
This db is dumped by a process that I dont own on a shared network disk.
Here are my observations based on the advice I have found elsewhere on the internet:
fuser filename.dbdoes not show any processes from my user accountsqlite3 filename.db "pragma integrity_check;"returnsError: database is locked- the md5sum of the
filename.db(the db that hangs) and its copyfilename2.db(doesn’t hang) are identical. So is the OS locking the db – because the lock info is not in the DB file itself? - This locked DB appears to occur when the process creating it did not exit cleanly.
Copying the db out (into say, filename2.db) and reading it is a workaround – but I want to understand whats going on and why – and if there is a way to read the locked DB in read-only mode.
You’ve to close the database before you backup or restore. Btw, do you have right permissions to the source (src)? This related link might help:
How to back up a SQLite db?
The stance of the SQLite developers about using a database stored in a networked file system is, essentially, "Don’t. Use a client-server database instead."
This simple, "remote database" approach is usually not the best way to use a single SQLite database from multiple systems, (even if it appears to "work"), as it often leads to various kinds of trouble and grief. Because these problems are inevitable with some usages, but not frequent or repeatable, it behooves application developers to not rely on early testing success to decide that their remote database use will work as desired.
Sounds like you’re running into one of those problems. What I suspect is happening, because of
This locked DB appears to occur when the process creating it did not exit cleanly.
is that the networked file system you’re using isn’t detecting that the creating process’s lock on the database is gone due to the process no longer existing, so the backup is just waiting for the lock to be released so that it can acquire it.
Even if you figure out how to broadcast that the lock is available from the creating process’s computer to others with that file mounted, there’s bound to be other subtle and not-so-subtle problems that pop up now and then. So your best bet is to follow the official advice:
If your data lives on a different machine from your application, then you should consider a client/server database.