reading header lines generated value error

Question

here is my binary file:
ftp://trmmopen.gsfc.nasa.gov/pub/merged/3B42RT/3B42RT.2014010318.7.bin.gz

I tried to read the header lines as follows:

fname = '3B42RT.2014010318.7.bin'

with open(fname, 'rb') as fi:

    header = np.fromfile(fi,dtype='S',count=2880)
    print header

I got ValueError: The elements are 0-sized.
The IDL file says the number of header lines are 2880, so I put count as 2880.

ftp://trmmopen.gsfc.nasa.gov/pub/merged/software/read_rt_file.pro

Any ideas please.

Asked By: puti

||

Source

Answer 1

I got ValueError: The elements are 0-sized.

As far as I can tell, the S specifier is to read fixed length string. You have to provide a length specifier:

header = np.fromfile(fi,dtype='S50',count=2880)

That being said, from a dump of your file:

00000000  61 6c 67 6f 72 69 74 68  6d 5f 49 44 3d 33 42 34  |algorithm_ID=3B4|
00000010  32 52 54 20 61 6c 67 6f  72 69 74 68 6d 5f 76 65  |2RT algorithm_ve|
00000020  72 73 69 6f 6e 3d 30 37  2e 30 30 20 67 72 61 6e  |rsion=07.00 gran|
[...]
00000480  2e 67 6f 76 20 72 75 6e  5f 6c 61 74 65 6e 63 79  |.gov run_latency|
00000490  3d 4c 41 53 54 20 20 20  20 20 20 20 20 20 20 20  |=LAST           |
000004a0  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |                |
*
00000b40  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00000db0  ff ff 83 01 83 01 83 01  83 01 83 01 83 01 83 01  |................|

The various headers are variable length and delimited by spaces ('x20'). You probably have to find an other way to parse those data.

Answered By: Sylvain Leroux

Answer 2

From closer inspection:

00000040  31 34 30 31 30 33 31 38  2e 37 2e 62 69 6e 20 68  |14010318.7.bin h|
00000050  65 61 64 65 72 5f 62 79  74 65 5f 6c 65 6e 67 74  |eader_byte_lengt|
00000060  68 3d 32 38 38 30 20 66  69 6c 65 5f 62 79 74 65  |h=2880 file_byte|
00000070  5f 6c 65 6e 67 74 68 3d  28 63 68 61 72 32 38 38  |_length=(char288|
00000080  30 29 5f 68 65 61 64 65  72 2b 28 69 6e 74 32 29  |0)_header+(int2)|
00000090  78 31 34 34 30 78 34 38  30 78 32 5f 64 61 74 61  |x1440x480x2_data|
000000a0  2b 28 69 6e 74 31 29 78  31 34 34 30 78 34 38 30  |+(int1)x1440x480|
000000b0  5f 64 61 74 61 2b 28 69  6e 74 32 29 78 31 34 34  |_data+(int2)x144|
000000c0  30 78 34 38 30 5f 64 61  74 61 20 6e 6f 6d 69 6e  |0x480_data nomin|

If appears that the header length is stored in the header_byte_length header field. That leads to that little parser (I let error handling and "beautification" up to you).

fname = '3B42RT.2014010318.7.bin'

with open(fname,'rb') as fi:
    headers = { 'header_byte_length': "999999" } 
    pos = 0

    acc = ""
    while pos < int(headers['header_byte_length']):
        c = fi.read(1)
        pos += 1
        if c == ' ':
            try:
                k,v = acc.split('=')
                headers[k] = v
            except ValueError:
                pass
            acc = ''
        else:
            acc += c

    print headers
    print "Data starts a offset ",headers['header_byte_length']

    fi.seek(int(headers['header_byte_length']))

Answered By: Sylvain Leroux

reading header lines generated value error

Question:

Answers: