How to verify if the tensorflow code trains completely in FP16?
Question:
I’m trying to train a TensorFlow (version 2.11.0) code in float16.
I checked that FP16 is supported on the RTX 3090 GPU. So, I followed the below link to train the whole code in reduced precision.
https://www.tensorflow.org/api_docs/python/tf/keras/backend/set_floatx
Then, When I was testing the training time of the model, I observed that the training time is not reduced. Interestingly when I manually convert some precision try to change the precision for some parts of code, the training time gets increased (due to the overhead of converting to FP16).
According to the above link it was mentioned that by default the code will run in float16, but is there any option to check if the underlying hardware really runs in float16?
Furthermore, my network is a 3-layer MLP network (which is quite small), is there any change that FP16 works only with larger networks?
Note:
-
I’m trying to train an Reinforcement Learning algorithm in reduced precision
(link: https://github.com/openai/maddpg)
-
I used CUDA 9.0, cuDNN 7.6.5 to train the algorithm
Answers:
It’s possible that the training time of your model isn’t reduced when using float16 due to the small size of your network. In general, the benefits of using float16 are more noticeable for larger models that require more computations. For small models like yours, the overhead of converting between data types might outweigh the potential benefits of using float16.
To verify that your hardware is running in float16, you can use the following command to print the default data type:
import tensorflow as tf
print(tf.keras.backend.floatx())
This should print ‘float16’ if the default data type has been set to float16.
Additionally, you can use the following command to check if your GPU supports float16 computations:
tf.test.is_gpu_available(cuda_only=True, min_cuda_compute_capability=None)
This should return True if your GPU supports float16 computations.
It’s also worth noting that using float16 can sometimes result in numerical instability, especially for models with very small weights. In such cases, it might be necessary to use float32 for certain parts of the model, even if it comes at the cost of increased computation time.
I’m trying to train a TensorFlow (version 2.11.0) code in float16.
I checked that FP16 is supported on the RTX 3090 GPU. So, I followed the below link to train the whole code in reduced precision.
https://www.tensorflow.org/api_docs/python/tf/keras/backend/set_floatx
Then, When I was testing the training time of the model, I observed that the training time is not reduced. Interestingly when I manually convert some precision try to change the precision for some parts of code, the training time gets increased (due to the overhead of converting to FP16).
According to the above link it was mentioned that by default the code will run in float16, but is there any option to check if the underlying hardware really runs in float16?
Furthermore, my network is a 3-layer MLP network (which is quite small), is there any change that FP16 works only with larger networks?
Note:
-
I’m trying to train an Reinforcement Learning algorithm in reduced precision
(link: https://github.com/openai/maddpg)
-
I used CUDA 9.0, cuDNN 7.6.5 to train the algorithm
It’s possible that the training time of your model isn’t reduced when using float16 due to the small size of your network. In general, the benefits of using float16 are more noticeable for larger models that require more computations. For small models like yours, the overhead of converting between data types might outweigh the potential benefits of using float16.
To verify that your hardware is running in float16, you can use the following command to print the default data type:
import tensorflow as tf
print(tf.keras.backend.floatx())
This should print ‘float16’ if the default data type has been set to float16.
Additionally, you can use the following command to check if your GPU supports float16 computations:
tf.test.is_gpu_available(cuda_only=True, min_cuda_compute_capability=None)
This should return True if your GPU supports float16 computations.
It’s also worth noting that using float16 can sometimes result in numerical instability, especially for models with very small weights. In such cases, it might be necessary to use float32 for certain parts of the model, even if it comes at the cost of increased computation time.