Python script to write(+lock) / read a file between 3 processes

Question

Iam new to python programming. I need to write a script which will be executed by 3 newly created processes at different time but very closely. At all time the resources need to check if a file is locked or not, because The first process to execute the python script, will execute a command, and lock a file to write the result of the command in it.
While The other 2 processes will wait for the file to be unlocked in order to read the result id and execute a different command from the first one.

How can i acheive that in python?
thank you

Asked By: YoussHark

||

Source

Answer 1

You can use:

os.open("lockfile", os.O_CREAT | os.O_EXCL)

to create a file that represents a lock. Using open in this way, the call will fail if the file already exists, and it is guaranteed that if the file gets created and the call succeeds, no other process was also able to create the file. So this gives you a locking mechanism represented by this file. The process of successfully creating the file means you acquired the lock. To give up the lock, you delete the file.

Other processes can check for the existence of the file to know that a process has the lock. If the file does not exist, then no process has the lock. It sounds like maybe you’d have processes watch for this condition, and then proceed knowing that the writing process has finished writing because it has given up the lock.

NOTE that this lock file can be the file you actually want to write to, but it doesn’t have to be. You can use a separate file just to handle the locking, and the process that gets the lock can know it can then safely write to one or more other files without some other process also trying to write to them.

Answered By: CryptoFool

Answer 2

If I am not misunderstanding, the task could be achieved by using a Lock for gaining writing access, a Semaphore to notify the processes which, having failed to gain the write Lock, will want to read the file and finally a Barrier to re-align the processes.
An Example:

import multiprocessing as mp
import time
from random import randint


def fun(lock_w, sem_r, barrier, task, filename):
  me = mp.current_process()
  for i in range(3):
    time.sleep(randint(1, 4))
    if(lock_w.acquire(block=False)):
      print(me.pid, "write access")
      task()
      sem_r.release()
      sem_r.release()
    else:
      sem_r.acquire()
      print(me.pid, "read access")
      task()
    if barrier.wait() == 0:
      print(me.pid, "releasing Lock")
      lock_w.release()

def task1():
  print("tPerform Task 1")

def task2():
  print("tPerform Task 2")

def task3():
  print("tPerform Task 3")

lock_w = mp.Lock()
sem_r = mp.Semaphore(0)
bar = mp.Barrier(3)
t1 = mp.Process(target=fun, args=(lock_w, sem_r, bar, task1, "foo.txt", ))
t2 = mp.Process(target=fun, args=(lock_w, sem_r, bar, task2, "foo.txt", ))
t3 = mp.Process(target=fun, args=(lock_w, sem_r, bar, task3, "foo.txt", ))
t1.start()
t2.start()
t3.start()
t1.join()
t2.join()
t3.join()

Example putput:

585 write access
    Perform Task 1
586 read access
    Perform Task 2
587 read access
    Perform Task 3
585 releasing Lock
587 write access
    Perform Task 3
586 read access
    Perform Task 2
585 read access
    Perform Task 1
587 releasing Lock
586 write access
    Perform Task 2
585 read access
    Perform Task 1
587 read access
    Perform Task 3
586 releasing Lock

Answered By: DLM

Answer 3

Alas I had to use file locking as process approach wont work. AAnd I actually have 5 separate applications running.

def lock(article):
    lockfile = os.path.join(lock_folder, article)
    os.open(lockfile, os.O_CREAT | os.O_EXCL)
    return

def unlock(article):
    lockfile = os.path.join(lock_folder, article)
    os.remove(lockfile)
    return

Then to use it

    try:
        try:
            lock(article)
        except FileExistsError as e:
            print('locked. skipping...')
            return

        DO_SOMETHING()

    finally:
        unlock(article)

Now in my implementaton I do’t hve the try-finally. If an error happens I want the file lock to rmain so I have a way to go back and find issues

Answered By: QuentinJS

Python script to write(+lock) / read a file between 3 processes

Question:

Answers: