Speed up iterating through a file

Question

I’m running the picoCTF PW Crack 5 gym challenge, but it is slow to iterate through the password list they provide for the challenge. Below is the code I’m using in the script to solve the challenge and would appreciate any explanation on how I could have sped up the process.

    correct = False
    with open('dictionary.txt') as pw_list:
      while correct != True:
        for line in pw_list:
          if hash_pw(line) == correct_pw_hash:
            print(line)
            correct = True
          else:
            continue

Asked By: Ryukashin

||

Source

Answer 1

This is a shorted equivalent way:

with open('dictionary.txt') as pw_list:
    for line in pw_list:
        if hash_pw(line) == correct_pw_hash:
            print(line)
            break

To speed this up you can try multi-processing since the hashing is CPU intensive. You can try different numbers on the processes and of course you need to set the hash_pw function and correct_pw_hash variable.

from multiprocessing import Pool

def check(line):
    if hash_pw(line) == correct_pw_hash:
        print(line)
   

if __name__ == "__main__":
    # Try different values on this to get the optimal performance.
    number_of_processes=8

    p = Pool(processes=8)
    file_name  = 'dictionary.txt'
    lines = []
    with open(file_name,'r') as f:
        for line in f:
            lines.append(line.strip())

    result = p.map(check, lines)
    p.close()
    p.join()

Please note that this way the entire file is loaded into the memory. If the file is huge, things become more complex.

More about multiprocessing:
https://docs.python.org/3/library/multiprocessing.html

Answered By: thanos.a

Speed up iterating through a file

Question:

Answers: