Error when converting DICOM image to pixel_array using tensorflow_io

I am trying to create a TensorFlow Dataset from DICOM images using the tf.data API and tensorflow_io, and I want to perform some pre-processing using Hounsfield units from the images. The DICOM images have a shape of (512,512). I have extracted the PixelData from the image and want to convert it to a numpy array of appropriate shape using the following code:

image_bytes = tf.io.read_file(image_path)
PixelData = tfio.image.decode_dicom_data(image_bytes, tfio.image.dicom_tags.PixelData).numpy()
pixel_array = np.frombuffer(PixelData, dtype=tf.uint16)
pixel_array = np.reshape(pixel_array, (512,512))
print(pixel_array)

This code should be equivalent to

Image = pydicom.dcmread(image_path)
pixel_array = Image.pixel_array
print(pixel_array)

and

Image = pydicom.dcmread(image_path)
PixelData = Image.PixelData
pixel_array = np.frombuffer(PixelData, dtype=np.uint16)
pixel_array = np.reshape(pixel_array, (512,512))
print(pixel_array)

The DICOM tags are the same ones used by pydicom, and are given here. The PixelData should return the raw byte values of the DICOM image. I have confirmed through pydicom that the raw pixel data is stored as np.uint16 values. However, when I try to convert the byte data given by tensorflow into a numpy array using the np.frombuffer function, I get an error of buffer size not being divisible by the element length.

When I run the above scripts, these are the following output shapes

1. Tensorflow: Doesn’t run with tf.uint16, gives output shape (1310719,) when using tf.uint8
2. Pydicom direct pixel_array: Output shape of (512,512)
3. Pydicom PixelData to pixel_array: Output shape of (512,512)

The pydicom examples provide identical outputs in both cases, however the tensorflow DICOM tag seems to provide a completely different result. Please find attached an example DICOM file here. Is there something wrong with the library or my implementation?

Edit:
The DICOM images are actually signed 16 bit integers, not unsigned. Hence, the following three snippets of code produce the same output:

Direct pixel_array from pydicom

import pydicom
import numpy as np

dcm = pydicom.dcmread("ID_000012eaf.dcm")
print(dcm.pixel_array)

Converting PixelData to pixel_array manually

import pydicom
import numpy as np
dcm = pydicom.dcmread("ID_000012eaf.dcm")
PixelData = dcm.PixelData
pixel_array = np.frombuffer(PixelData, dtype=np.int16)
pixel_array = np.reshape(pixel_array, (512,512))
print(pixel_array)

Directly obtaining pixel_array using tensorflow_io

import tensorflow as tf
import tensorflow_io as tfio
import numpy as np
image_bytes = tf.io.read_file("ID_000012eaf.dcm")
pixel_array = tfio.image.decode_dicom_image(image_bytes, on_error='lossy', scale='preserve', dtype=tf.float32).numpy()
pixel_array = pixel_array.astype('int16')
pixel_array /= 2.
pixel_array = np.reshape(pixel_array, (512,512))
print(pixel_array)

However, this final snippet of code still doesn’t work for some reason:

import tensorflow as tf
import tensorflow_io as tfio
import numpy as np

image_bytes = tf.io.read_file("ID_000012eaf.dcm")
PixelData = tfio.image.decode_dicom_data(image_bytes, tfio.image.dicom_tags.PixelData).numpy()
pixel_array = np.frombuffer(PixelData, dtype=np.int16)
print(pixel_array)

Edit 2: These two snippets of code should theoretically work, however they display errors that the length of the byte string is not divisible by the size of int16:

import tensorflow as tf
import tensorflow_io as tfio
import numpy as np
image_bytes = tf.io.read_file("ID_000012eaf.dcm")
PixelData = tfio.image.decode_dicom_data(image_bytes, tfio.image.dicom_tags.PixelData).numpy()
pixel_array =  np.frombuffer(PixelData, dtype=np.int16)
pixel_array = np.reshape(pixel_array, (512,512))
print(pixel_array)

and

import tensorflow as tf
import tensorflow_io as tfio
import numpy as np
image_bytes = tf.io.read_file("ID_000012eaf.dcm")
PixelData = tfio.image.decode_dicom_data(image_bytes, tfio.image.dicom_tags.PixelData)
pixel_array = tf.io.decode_raw(PixelData, tf.int16)
pixel_array = tf.reshape(pixel_array, (512,512))
print(pixel_array)

Edit 3: After getting the hint that the byte string provided by decode_dicom_data contains hexadecimal values, I found a way to convert my data into the desired pixel_array, but I’m curious why PixelData is stored this way:

import tensorflow as tf
import tensorflow_io as tfio
import numpy as np
image_bytes = tf.io.read_file("ID_000012eaf.dcm")
PixelData = tfio.image.decode_dicom_data(image_bytes, tfio.image.dicom_tags.PixelData).numpy()
pixel_array = np.zeros(262144, dtype=np.int16)
start,stop = 0,4
for i in range(262144):
    pixel_array[i] = int(PixelData[start:stop], base=16)
    start+=5
    stop+=5
pixel_array = np.reshape(pixel_array, (512,512))
print(pixel_array)

PixelData from pydicom:

PixelData = b'0xf80xf80xf80...'

PixelData from Tensorflow_io

PixelData = b'f830\f830\f830\...'

Any suggestions for code refactoring and linting would be highly appreciated. I am extremely grateful to @ai2ys for helping me diagnose these issues.