Python – Mock/trigger file creation event while monitoring it in same script/thread?
Question:
I’m writing a test script and use below function detect_event_and_get_json_data() to monitor for newly Json created event/s in a json_path and parse these data for checking.
But the problem is: inside this while loop for waiting Json event/s, is there any way I can trigger/mock an creation event within this test script? Do I need to run this mock in another thread?
import glob
import time
json_path = "/home/json_events"
def collect_all_files():
"""
Collect all .json files in the given folder
Returned values:
json_file_set: set containing all the .json files in the given folder
"""
json_file_list = glob.glob(json_path + "/*.json")
json_file_set = set(json_file_list)
return json_file_set
def detect_new_json_files(previous_json_files):
"""
Detect newly created .json files in the given json dir
Returned values:
new_json_file_set: set containing all newly created .json files in the given folder
"""
current_json_file_set = collect_all_files(json_path)
newly_created_json_files = current_json_file_set.difference(previous_json_files)
return newly_created_json_files
def retrieve_data_from_json_file(json_file):
"""
Retrieve the target data from the json file
Returned values:
data: retrieved data in str
"""
with open(json_file) as json_file_object:
data = json_file_object.read()
return data
def detect_event_and_get_json_data():
"""
Detect when json event/s occurs(created) and return list of data from json files
"""
previous_json_files = collect_all_files(json_path)
count = 0
poll_time = 1
TIMEOUT = 30
while count < TIMEOUT:
newly_created_json_files = detect_new_json_files(json_path, previous_json_files)
# Not only one json file created in one event
if len(newly_created_json_files) >= 1:
# Init an empty json data list
json_data_list = []
for newly_created_json_file in newly_created_json_files:
# Retrieve data from newly created json file/s
tmp = retrieve_data_from_json_file(newly_created_json_file)
# Append json data files content to list and return
json_data_list += [tmp]
return json_data_list
# Update current json files in json path to previous set
previous_json_files = collect_all_files(json_path)
time.sleep(poll_time)
count += poll_time
# No newly created, return None
return None
def test_json_newly_event():
"""
test case monitor for newly created json files and verify payload
"""
json_data_list = detect_event_and_get_json_data()
# mock an Json creation event: /home/json_events/a.json, b.json
# process checking Json content of a and b by json_data_list[0:1]
Answers:
Yes, you can simulate the creation of a JSON event within your test script’s while loop. You don’t have to run it in another thread, but it’s a good idea to avoid blocking the main thread and to ensure that your script remains responsive.
Here’s an updated version of the code that makes use of some more recent Python features.
import os
import json
import time
from pathlib import Path
from typing import List
import threading
import queue
def detect_event_and_get_json_data(json_path: str, result_queue: queue.Queue):
"""
Monitor for newly created JSON files in a directory and put their data into the result queue.
"""
json_path = Path(json_path)
previous_json_files = set(json_path.glob('*.json'))
count = 0
poll_time = 1
TIMEOUT = 30
while count < TIMEOUT:
current_json_files = set(json_path.glob('*.json'))
newly_created_json_files = current_json_files - previous_json_files
if newly_created_json_files:
json_data_list = []
for newly_created_json_file in newly_created_json_files:
with newly_created_json_file.open() as f:
json_data_list.append(json.load(f))
result_queue.put(json_data_list)
return
previous_json_files = current_json_files
time.sleep(poll_time)
count += poll_time
result_queue.put([])
def test_json_newly_event():
"""
Test case to monitor for newly created JSON files and verify their contents.
"""
json_path = Path(__file__).parent / 'test_data' / 'json_events'
# Create a queue to hold the result
result_queue = queue.Queue()
# Start the monitoring loop in a separate thread
monitor_thread = threading.Thread(target=detect_event_and_get_json_data, args=(json_path, result_queue))
monitor_thread.start()
# Wait a bit to ensure that the monitoring loop has started
time.sleep(1)
# Create some mock JSON files
mock_json_data_1 = {"foo": "bar"}
with (json_path / "a.json").open('w') as f:
json.dump(mock_json_data_1, f)
mock_json_data_2 = {"baz": 42}
with (json_path / "b.json").open('w') as f:
json.dump(mock_json_data_2, f)
# Wait a bit to allow the monitoring loop to detect the events
time.sleep(1)
# Stop the monitoring loop
monitor_thread.join()
# Get the data collected by the monitoring loop
json_data_list = result_queue.get()
# Check that the collected data matches the expected data
expected_json_data_list = [mock_json_data_1, mock_json_data_2]
assert json_data_list == expected_json_data_list
VERSION 1 COMMENTS
Here are the main modifications I made:
1.) Instead of os.path, I used pathlib.Path to handle file paths.
2.) I changed the detect event and get json data function’s return type to List[dict] to make it clear that it returns a list of dictionaries.
3.) Instead of a list comprehension and set, I used a set comprehension to collect the JSON files.
4.) I used the open function to open the files in the with block and json.load to load the JSON data.
5.) For the json path argument, I added a type hint.
6.) I used the parent attribute of file to get the current file’s parent directory, and Path to create the path to the test data directory.
7.) I used the Path object’s join method to create the paths to the JSON files, and the Path object’s open method to open the files.
Instead of None, I made an empty list the default return value for detect event and get json data.
VERSION 2 COMMENTS:
1.) I gave the detect event and get json data function a result queue argument to hold the JSON data that it got.
2.) I changed detect event and get json data so that the JSON data was put into the result queue instead of being returned.
3.) I created a queue.
Queue object to hold the result in the test function.
4.) I gave the result queue as an argument to the function detect event and get json data.
5.) After the monitoring loop was done, I used result queue.get() to get the JSON data that had been collected.
6.) In detect event and get json data, I made the default value for the result queue an empty list.
So you can check and try this out, if this will for your case. hoping solves your issue.
I’m writing a test script and use below function detect_event_and_get_json_data() to monitor for newly Json created event/s in a json_path and parse these data for checking.
But the problem is: inside this while loop for waiting Json event/s, is there any way I can trigger/mock an creation event within this test script? Do I need to run this mock in another thread?
import glob
import time
json_path = "/home/json_events"
def collect_all_files():
"""
Collect all .json files in the given folder
Returned values:
json_file_set: set containing all the .json files in the given folder
"""
json_file_list = glob.glob(json_path + "/*.json")
json_file_set = set(json_file_list)
return json_file_set
def detect_new_json_files(previous_json_files):
"""
Detect newly created .json files in the given json dir
Returned values:
new_json_file_set: set containing all newly created .json files in the given folder
"""
current_json_file_set = collect_all_files(json_path)
newly_created_json_files = current_json_file_set.difference(previous_json_files)
return newly_created_json_files
def retrieve_data_from_json_file(json_file):
"""
Retrieve the target data from the json file
Returned values:
data: retrieved data in str
"""
with open(json_file) as json_file_object:
data = json_file_object.read()
return data
def detect_event_and_get_json_data():
"""
Detect when json event/s occurs(created) and return list of data from json files
"""
previous_json_files = collect_all_files(json_path)
count = 0
poll_time = 1
TIMEOUT = 30
while count < TIMEOUT:
newly_created_json_files = detect_new_json_files(json_path, previous_json_files)
# Not only one json file created in one event
if len(newly_created_json_files) >= 1:
# Init an empty json data list
json_data_list = []
for newly_created_json_file in newly_created_json_files:
# Retrieve data from newly created json file/s
tmp = retrieve_data_from_json_file(newly_created_json_file)
# Append json data files content to list and return
json_data_list += [tmp]
return json_data_list
# Update current json files in json path to previous set
previous_json_files = collect_all_files(json_path)
time.sleep(poll_time)
count += poll_time
# No newly created, return None
return None
def test_json_newly_event():
"""
test case monitor for newly created json files and verify payload
"""
json_data_list = detect_event_and_get_json_data()
# mock an Json creation event: /home/json_events/a.json, b.json
# process checking Json content of a and b by json_data_list[0:1]
Yes, you can simulate the creation of a JSON event within your test script’s while loop. You don’t have to run it in another thread, but it’s a good idea to avoid blocking the main thread and to ensure that your script remains responsive.
Here’s an updated version of the code that makes use of some more recent Python features.
import os
import json
import time
from pathlib import Path
from typing import List
import threading
import queue
def detect_event_and_get_json_data(json_path: str, result_queue: queue.Queue):
"""
Monitor for newly created JSON files in a directory and put their data into the result queue.
"""
json_path = Path(json_path)
previous_json_files = set(json_path.glob('*.json'))
count = 0
poll_time = 1
TIMEOUT = 30
while count < TIMEOUT:
current_json_files = set(json_path.glob('*.json'))
newly_created_json_files = current_json_files - previous_json_files
if newly_created_json_files:
json_data_list = []
for newly_created_json_file in newly_created_json_files:
with newly_created_json_file.open() as f:
json_data_list.append(json.load(f))
result_queue.put(json_data_list)
return
previous_json_files = current_json_files
time.sleep(poll_time)
count += poll_time
result_queue.put([])
def test_json_newly_event():
"""
Test case to monitor for newly created JSON files and verify their contents.
"""
json_path = Path(__file__).parent / 'test_data' / 'json_events'
# Create a queue to hold the result
result_queue = queue.Queue()
# Start the monitoring loop in a separate thread
monitor_thread = threading.Thread(target=detect_event_and_get_json_data, args=(json_path, result_queue))
monitor_thread.start()
# Wait a bit to ensure that the monitoring loop has started
time.sleep(1)
# Create some mock JSON files
mock_json_data_1 = {"foo": "bar"}
with (json_path / "a.json").open('w') as f:
json.dump(mock_json_data_1, f)
mock_json_data_2 = {"baz": 42}
with (json_path / "b.json").open('w') as f:
json.dump(mock_json_data_2, f)
# Wait a bit to allow the monitoring loop to detect the events
time.sleep(1)
# Stop the monitoring loop
monitor_thread.join()
# Get the data collected by the monitoring loop
json_data_list = result_queue.get()
# Check that the collected data matches the expected data
expected_json_data_list = [mock_json_data_1, mock_json_data_2]
assert json_data_list == expected_json_data_list
VERSION 1 COMMENTS
Here are the main modifications I made:
1.) Instead of os.path, I used pathlib.Path to handle file paths.
2.) I changed the detect event and get json data function’s return type to List[dict] to make it clear that it returns a list of dictionaries.
3.) Instead of a list comprehension and set, I used a set comprehension to collect the JSON files.
4.) I used the open function to open the files in the with block and json.load to load the JSON data.
5.) For the json path argument, I added a type hint.
6.) I used the parent attribute of file to get the current file’s parent directory, and Path to create the path to the test data directory.
7.) I used the Path object’s join method to create the paths to the JSON files, and the Path object’s open method to open the files.
Instead of None, I made an empty list the default return value for detect event and get json data.
VERSION 2 COMMENTS:
1.) I gave the detect event and get json data function a result queue argument to hold the JSON data that it got.
2.) I changed detect event and get json data so that the JSON data was put into the result queue instead of being returned.
3.) I created a queue.
Queue object to hold the result in the test function.
4.) I gave the result queue as an argument to the function detect event and get json data.
5.) After the monitoring loop was done, I used result queue.get() to get the JSON data that had been collected.
6.) In detect event and get json data, I made the default value for the result queue an empty list.
So you can check and try this out, if this will for your case. hoping solves your issue.