In deep learning, can the prediction speed increase as the batch size decreases?
Question:
We are developing a prediction model using deepchem’s GCNModel.
Model learning and performance verification proceeded without problems, but it was confirmed that a lot of time was spent on prediction.
We are trying to predict a total of 1 million data, and the parameters used are as follows.
model = GCNModel(n_tasks=1, mode='regression', number_atom_features=32, learning_rate=0.0001, dropout=0.2, batch_size=32, device=device, model_dir=model_path)
I changed the batch size to improve the performance, and it was confirmed that the time was faster when the value was decreased than when the value was increased.
All models had the same GPU memory usage.
From common sense I know, it is estimated that the larger the batch size, the faster it will be. But can you tell me why it works in reverse?
We would be grateful if you could also let us know how we can further improve the prediction time.
Answers:
let’s clarify some definitions first.
Epoch
Times that your model and learning algorithm will walk through your dataset.
(Complete passes)
BatchSize
The number of samples(every single row of your training data) before updating the internal model. in other words, the number of samples processed before the model is updated.
So Your batch size is something between 1 and your len(training_data)
Generally, more batch size gives more accuracy of training data.
Epoch ↑ Batch Size ↑ Accuracy ↑ Speed ↓
So the short answer to question is more batch size takes more memory and needs more process and obviously takes longer time to learn.
https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network
Here is the link for more details.
There are two components regarding the speed:
- Your batch size and model size
- Your CPU/GPU power in spawning and processing batches
And two of them need to be balanced. For example, if your model finishes prediction of this batch, but the next batch is not yet spawned, you will notice a drop in GPU utilization for a brief moment. Sadly there is no inner metrics that directly tell you this balance – try using time.time() to benchmark your model’s prediction as well as the dataloader speed.
However, I don’t think that’s worth the effort, so you can keep decreasing the batch size up to the point there is no improvement – that’s where to stop.
As I saw, near batch sizes (BS) effect a bit of training and prediction durations. For example, there is no difference between BS of 2 and 5 thanks to GPU which can work asyncly. For my case (it can be changed according to model you will use), I have obtained critical differences for about each 20 BS gap.
I don’t know your BS amount. However even if you set your BS as 10 from 1 or something else, you may not increase prediction fast up to you want. I can suggest a way to decide that: Prepare a benchmark code that predict with random generated tensor by using your model, and just see the prediction durations for your model according to different BS amounts. But generally, a large BS can decrease unit prediction time.
You can see some prediction times I got from my study following. Maybe these measures can figure out somethings clearly to you.
BATCH SIZE: 10 About Initialization Duration: 0.020000
[EXP 1] Input Size: torch.Size([1, 10, 320, 240]) GPU Prediction Duration: 0.004644 (0.000464) CPU Prediction Duration: 0.061246 (0.006125)
Input Size: torch.Size([1, 10, 105, 120]) GPU Prediction Duration: 0.004010 (0.000401) CPU Prediction Duration: 0.013519 (0.001352)
[EXP 2] Input Size: torch.Size([1, 10, 320, 240]) GPU Prediction Duration: 0.006054 (0.000605) CPU Prediction Duration: 0.066053 (0.006605)
Input Size: torch.Size([1, 10, 105, 120]) GPU Prediction Duration: 0.003977 (0.000398) CPU Prediction Duration: 0.013553 (0.001355)
[EXP 3] Input Size: torch.Size([1, 10, 320, 240]) GPU Prediction Duration: 0.004811 (0.000481) CPU Prediction Duration: 0.066356 (0.006636)
Input Size: torch.Size([1, 10, 105, 120]) GPU Prediction Duration: 0.004073 (0.000407) CPU Prediction Duration: 0.013309 (0.001331)
[EXP 4] Input Size: torch.Size([1, 10, 320, 240]) GPU Prediction Duration: 0.004611 (0.000461) CPU Prediction Duration: 0.061540 (0.006154)
Input Size: torch.Size([1, 10, 105, 120]) GPU Prediction Duration: 0.004031 (0.000403) CPU Prediction Duration: 0.018667 (0.001867)
[EXP 5] Input Size: torch.Size([1, 10, 320, 240]) GPU Prediction Duration: 0.006329 (0.000633) CPU Prediction Duration: 0.060230 (0.006023)
Input Size: torch.Size([1, 10, 105, 120]) GPU Prediction Duration: 0.004031 (0.000403) CPU Prediction Duration: 0.013159 (0.001316)
BATCH SIZE: 30 About Initialization Duration: 0.060000
[EXP 1] Input Size: torch.Size([1, 30, 320, 240]) GPU Prediction Duration: 0.005804 (0.000193) CPU Prediction Duration: 0.077307 (0.002577)
Input Size: torch.Size([1, 30, 105, 120]) GPU Prediction Duration: 0.004139 (0.000138) CPU Prediction Duration: 0.014293 (0.000476)
[EXP 2] Input Size: torch.Size([1, 30, 320, 240]) GPU Prediction Duration: 0.007187 (0.000240) CPU Prediction Duration: 0.081585 (0.002719)
Input Size: torch.Size([1, 30, 105, 120]) GPU Prediction Duration: 0.004193 (0.000140) CPU Prediction Duration: 0.014282 (0.000476)
[EXP 3] Input Size: torch.Size([1, 30, 320, 240]) GPU Prediction Duration: 0.006055 (0.000202) CPU Prediction Duration: 0.074875 (0.002496)
Input Size: torch.Size([1, 30, 105, 120]) GPU Prediction Duration: 0.005654 (0.000188) CPU Prediction Duration: 0.016254 (0.000542)
[EXP 4] Input Size: torch.Size([1, 30, 320, 240]) GPU Prediction Duration: 0.005906 (0.000197) CPU Prediction Duration: 0.068274 (0.002276)
Input Size: torch.Size([1, 30, 105, 120]) GPU Prediction Duration: 0.004203 (0.000140) CPU Prediction Duration: 0.016259 (0.000542)
[EXP 5] Input Size: torch.Size([1, 30, 320, 240]) GPU Prediction Duration: 0.007311 (0.000244) CPU Prediction Duration: 0.073374 (0.002446)
Input Size: torch.Size([1, 30, 105, 120]) GPU Prediction Duration: 0.004222 (0.000141) CPU Prediction Duration: 0.014630 (0.000488)
BATCH SIZE: 50 About Initialization Duration: 0.100000
[EXP 1] Input Size: torch.Size([1, 50, 320, 240]) GPU Prediction Duration: 0.007298 (0.000146) CPU Prediction Duration: 0.087110 (0.001742)
Input Size: torch.Size([1, 50, 105, 120]) GPU Prediction Duration: 0.004315 (0.000086) CPU Prediction Duration: 0.015497 (0.000310)
[EXP 2] Input Size: torch.Size([1, 50, 320, 240]) GPU Prediction Duration: 0.007052 (0.000141) CPU Prediction Duration: 0.086180 (0.001724)
Input Size: torch.Size([1, 50, 105, 120]) GPU Prediction Duration: 0.004359 (0.000087) CPU Prediction Duration: 0.014936 (0.000299)
[EXP 3] Input Size: torch.Size([1, 50, 320, 240]) GPU Prediction Duration: 0.006832 (0.000137) CPU Prediction Duration: 0.085362 (0.001707)
Input Size: torch.Size([1, 50, 105, 120]) GPU Prediction Duration: 0.004473 (0.000089) CPU Prediction Duration: 0.015069 (0.000301)
[EXP 4] Input Size: torch.Size([1, 50, 320, 240]) GPU Prediction Duration: 0.007035 (0.000141) CPU Prediction Duration: 0.085918 (0.001718)
Input Size: torch.Size([1, 50, 105, 120]) GPU Prediction Duration: 0.004349 (0.000087) CPU Prediction Duration: 0.014744 (0.000295)
[EXP 5] Input Size: torch.Size([1, 50, 320, 240]) GPU Prediction Duration: 0.006997 (0.000140) CPU Prediction Duration: 0.082447 (0.001649)
Input Size: torch.Size([1, 50, 105, 120]) GPU Prediction Duration: 0.005865 (0.000117) CPU Prediction Duration: 0.016434 (0.000329)
On the other hand, increasing or decreasing BS may cause to accuracy changes in positive or negative means.
We are developing a prediction model using deepchem’s GCNModel.
Model learning and performance verification proceeded without problems, but it was confirmed that a lot of time was spent on prediction.
We are trying to predict a total of 1 million data, and the parameters used are as follows.
model = GCNModel(n_tasks=1, mode='regression', number_atom_features=32, learning_rate=0.0001, dropout=0.2, batch_size=32, device=device, model_dir=model_path)
I changed the batch size to improve the performance, and it was confirmed that the time was faster when the value was decreased than when the value was increased.
All models had the same GPU memory usage.
From common sense I know, it is estimated that the larger the batch size, the faster it will be. But can you tell me why it works in reverse?
We would be grateful if you could also let us know how we can further improve the prediction time.
let’s clarify some definitions first.
Epoch
Times that your model and learning algorithm will walk through your dataset.
(Complete passes)
BatchSize
The number of samples(every single row of your training data) before updating the internal model. in other words, the number of samples processed before the model is updated.
So Your batch size is something between 1 and your len(training_data)
Generally, more batch size gives more accuracy of training data.
Epoch ↑ Batch Size ↑ Accuracy ↑ Speed ↓
So the short answer to question is more batch size takes more memory and needs more process and obviously takes longer time to learn.
https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network
Here is the link for more details.
There are two components regarding the speed:
- Your batch size and model size
- Your CPU/GPU power in spawning and processing batches
And two of them need to be balanced. For example, if your model finishes prediction of this batch, but the next batch is not yet spawned, you will notice a drop in GPU utilization for a brief moment. Sadly there is no inner metrics that directly tell you this balance – try using time.time() to benchmark your model’s prediction as well as the dataloader speed.
However, I don’t think that’s worth the effort, so you can keep decreasing the batch size up to the point there is no improvement – that’s where to stop.
As I saw, near batch sizes (BS) effect a bit of training and prediction durations. For example, there is no difference between BS of 2 and 5 thanks to GPU which can work asyncly. For my case (it can be changed according to model you will use), I have obtained critical differences for about each 20 BS gap.
I don’t know your BS amount. However even if you set your BS as 10 from 1 or something else, you may not increase prediction fast up to you want. I can suggest a way to decide that: Prepare a benchmark code that predict with random generated tensor by using your model, and just see the prediction durations for your model according to different BS amounts. But generally, a large BS can decrease unit prediction time.
You can see some prediction times I got from my study following. Maybe these measures can figure out somethings clearly to you.
BATCH SIZE: 10 About Initialization Duration: 0.020000
[EXP 1] Input Size: torch.Size([1, 10, 320, 240]) GPU Prediction Duration: 0.004644 (0.000464) CPU Prediction Duration: 0.061246 (0.006125)
Input Size: torch.Size([1, 10, 105, 120]) GPU Prediction Duration: 0.004010 (0.000401) CPU Prediction Duration: 0.013519 (0.001352)
[EXP 2] Input Size: torch.Size([1, 10, 320, 240]) GPU Prediction Duration: 0.006054 (0.000605) CPU Prediction Duration: 0.066053 (0.006605)
Input Size: torch.Size([1, 10, 105, 120]) GPU Prediction Duration: 0.003977 (0.000398) CPU Prediction Duration: 0.013553 (0.001355)
[EXP 3] Input Size: torch.Size([1, 10, 320, 240]) GPU Prediction Duration: 0.004811 (0.000481) CPU Prediction Duration: 0.066356 (0.006636)
Input Size: torch.Size([1, 10, 105, 120]) GPU Prediction Duration: 0.004073 (0.000407) CPU Prediction Duration: 0.013309 (0.001331)
[EXP 4] Input Size: torch.Size([1, 10, 320, 240]) GPU Prediction Duration: 0.004611 (0.000461) CPU Prediction Duration: 0.061540 (0.006154)
Input Size: torch.Size([1, 10, 105, 120]) GPU Prediction Duration: 0.004031 (0.000403) CPU Prediction Duration: 0.018667 (0.001867)
[EXP 5] Input Size: torch.Size([1, 10, 320, 240]) GPU Prediction Duration: 0.006329 (0.000633) CPU Prediction Duration: 0.060230 (0.006023)
Input Size: torch.Size([1, 10, 105, 120]) GPU Prediction Duration: 0.004031 (0.000403) CPU Prediction Duration: 0.013159 (0.001316)
BATCH SIZE: 30 About Initialization Duration: 0.060000
[EXP 1] Input Size: torch.Size([1, 30, 320, 240]) GPU Prediction Duration: 0.005804 (0.000193) CPU Prediction Duration: 0.077307 (0.002577)
Input Size: torch.Size([1, 30, 105, 120]) GPU Prediction Duration: 0.004139 (0.000138) CPU Prediction Duration: 0.014293 (0.000476)
[EXP 2] Input Size: torch.Size([1, 30, 320, 240]) GPU Prediction Duration: 0.007187 (0.000240) CPU Prediction Duration: 0.081585 (0.002719)
Input Size: torch.Size([1, 30, 105, 120]) GPU Prediction Duration: 0.004193 (0.000140) CPU Prediction Duration: 0.014282 (0.000476)
[EXP 3] Input Size: torch.Size([1, 30, 320, 240]) GPU Prediction Duration: 0.006055 (0.000202) CPU Prediction Duration: 0.074875 (0.002496)
Input Size: torch.Size([1, 30, 105, 120]) GPU Prediction Duration: 0.005654 (0.000188) CPU Prediction Duration: 0.016254 (0.000542)
[EXP 4] Input Size: torch.Size([1, 30, 320, 240]) GPU Prediction Duration: 0.005906 (0.000197) CPU Prediction Duration: 0.068274 (0.002276)
Input Size: torch.Size([1, 30, 105, 120]) GPU Prediction Duration: 0.004203 (0.000140) CPU Prediction Duration: 0.016259 (0.000542)
[EXP 5] Input Size: torch.Size([1, 30, 320, 240]) GPU Prediction Duration: 0.007311 (0.000244) CPU Prediction Duration: 0.073374 (0.002446)
Input Size: torch.Size([1, 30, 105, 120]) GPU Prediction Duration: 0.004222 (0.000141) CPU Prediction Duration: 0.014630 (0.000488)
BATCH SIZE: 50 About Initialization Duration: 0.100000
[EXP 1] Input Size: torch.Size([1, 50, 320, 240]) GPU Prediction Duration: 0.007298 (0.000146) CPU Prediction Duration: 0.087110 (0.001742)
Input Size: torch.Size([1, 50, 105, 120]) GPU Prediction Duration: 0.004315 (0.000086) CPU Prediction Duration: 0.015497 (0.000310)
[EXP 2] Input Size: torch.Size([1, 50, 320, 240]) GPU Prediction Duration: 0.007052 (0.000141) CPU Prediction Duration: 0.086180 (0.001724)
Input Size: torch.Size([1, 50, 105, 120]) GPU Prediction Duration: 0.004359 (0.000087) CPU Prediction Duration: 0.014936 (0.000299)
[EXP 3] Input Size: torch.Size([1, 50, 320, 240]) GPU Prediction Duration: 0.006832 (0.000137) CPU Prediction Duration: 0.085362 (0.001707)
Input Size: torch.Size([1, 50, 105, 120]) GPU Prediction Duration: 0.004473 (0.000089) CPU Prediction Duration: 0.015069 (0.000301)
[EXP 4] Input Size: torch.Size([1, 50, 320, 240]) GPU Prediction Duration: 0.007035 (0.000141) CPU Prediction Duration: 0.085918 (0.001718)
Input Size: torch.Size([1, 50, 105, 120]) GPU Prediction Duration: 0.004349 (0.000087) CPU Prediction Duration: 0.014744 (0.000295)
[EXP 5] Input Size: torch.Size([1, 50, 320, 240]) GPU Prediction Duration: 0.006997 (0.000140) CPU Prediction Duration: 0.082447 (0.001649)
Input Size: torch.Size([1, 50, 105, 120]) GPU Prediction Duration: 0.005865 (0.000117) CPU Prediction Duration: 0.016434 (0.000329)
On the other hand, increasing or decreasing BS may cause to accuracy changes in positive or negative means.