How to monitor the status of model training running on the server via fast-api

Question

By the following code, I want to get the status of training, just if it is started and when it is finished.

app = FastAPI()

@app.post("/train")
async def train_on_background():
    background_tasks.add_task(train, input_data, output_data)




def train(input_data, output_data):
        dataset = prepare_dataset(input_data, output_data)
         
        model = Trainer(dataset)
        model.auto_train()
        
        
        filename = "Model" 
        filename = filename + ".pkl"
        dump(model, open(filename, 'wb'))

The train() function should train the model by doing several steps. train_on_background() should run the train function on the background.

I want to add a separate command to check the training status. the command should respond started when the training starts (train() function is called), & finished when it train() function is ended.

Asked By: Zaid Ghazal

||

Source

Answer 1

The solution for your problem is by making a temporary file containing the status of the train():

@app.get(“/check”)
def check():
    filename = “Model”
    filename = filename + “.pkl”
    file_exists = os.path.exists(filename)
    if file_exists:
        return {“message”: “Model Already Exists”}
    elif (os.path.exists(f”temp_monitor.txt”)):
         return {“message”: “Training in progress...“}
def monitor_file(text):
    if text is None:
        os.remove(f”temp_monitor.txt”)
    else:
        with open(f’temp_monitor.txt’, ‘w’) as f:
            f.write(text)

These 2 functions works on creating a temporary file containing the current status of the training. The check command looks for the model if exists to respond with finish.

If the model doesn’t exist, it looks for the temporary file, and respond by training in progress to indicate that the training was started.

def train(input_data, output_data):4
        dataset = prepare_dataset(input_data, output_data)
        monitor_file(“started”)
        model = Trainer(dataset)
        model.auto_train()

Modify your train function to save the current status.

Note that you can use this method to save any status you want and check using check command.

Answered By: alibustami

How to monitor the status of model training running on the server via fast-api

Question:

Answers: