Would SQLite for Python be helpful for a file this size?
Question:
I have a file of around 60 million lines. I am trying to constantly query the file to find information for a list of names. Each line in the file contains a name followed by relevant information. I tried to build a dictionary but the file was too big.
Many suggesting SQLite for Python. Would that be helpful for a file this size?
Answers:
you can use Linux Power tools.
Awk can do your work very easily.
Suppose your file looks like this
name1 a b c
name2 e d r t
name3 x y
and you want to find info of name2. then using this command you can find
awk -F" " -v name=name2 '{if(name==$1){$1=""; print $0}}' filename
it will output only info of name2 like this
e d r t
you can pass multiple name and manage your logic
I have a file of around 60 million lines. I am trying to constantly query the file to find information for a list of names. Each line in the file contains a name followed by relevant information. I tried to build a dictionary but the file was too big.
Many suggesting SQLite for Python. Would that be helpful for a file this size?
you can use Linux Power tools.
Awk can do your work very easily.
Suppose your file looks like this
name1 a b c
name2 e d r t
name3 x y
and you want to find info of name2. then using this command you can find
awk -F" " -v name=name2 '{if(name==$1){$1=""; print $0}}' filename
it will output only info of name2 like this
e d r t
you can pass multiple name and manage your logic