Numpy: Joining structured arrays?

Question

Input

I have many numpy structured arrays in a list like this example:

import numpy

a1 = numpy.array([(1, 2), (3, 4), (5, 6)], dtype=[('x', int), ('y', int)])

a2 = numpy.array([(7,10), (8,11), (9,12)], dtype=[('z', int), ('w', float)])

arrays = [a1, a2]

Desired Output

What is the correct way to join them all together to create a unified structured array like the following?

desired_result = numpy.array([(1, 2, 7, 10), (3, 4, 8, 11), (5, 6, 9, 12)],
                             dtype=[('x', int), ('y', int), ('z', int), ('w', float)])

Current Approach

This is what I’m currently using, but it is very slow, so I suspect there must be a more efficent way.

from numpy.lib.recfunctions import append_fields

def join_struct_arrays(arrays):
    for array in arrays:
        try:
            result = append_fields(result, array.dtype.names, [array[name] for name in array.dtype.names], usemask=False)
        except NameError:
            result = array

    return result

Asked By: Jon-Eric

||

Source

Answer 1

Here is an implementation that should be faster. It converts everything to arrays of numpy.uint8 and does not use any temporaries.

def join_struct_arrays(arrays):
    sizes = numpy.array([a.itemsize for a in arrays])
    offsets = numpy.r_[0, sizes.cumsum()]
    n = len(arrays[0])
    joint = numpy.empty((n, offsets[-1]), dtype=numpy.uint8)
    for a, size, offset in zip(arrays, sizes, offsets):
        joint[:,offset:offset+size] = a.view(numpy.uint8).reshape(n,size)
    dtype = sum((a.dtype.descr for a in arrays), [])
    return joint.ravel().view(dtype)

Edit: Simplified the code and avoided the unnecessary as_strided().

Answered By: Sven Marnach

Answer 2

You can also use the function merge_arrays of numpy.lib.recfunctions:

import numpy.lib.recfunctions as rfn
rfn.merge_arrays(arrays, flatten = True, usemask = False)

Out[52]: 
array([(1, 2, 7, 10.0), (3, 4, 8, 11.0), (5, 6, 9, 12.0)], 
     dtype=[('x', '<i4'), ('y', '<i4'), ('z', '<i4'), ('w', '<f8')])

Answered By: joris

Answer 3

and yet another way, a little more readable and also a lot faster I think:

def join_struct_arrays(arrays):
    newdtype = []
    for a in arrays:
        descr = []
        for field in a.dtype.names:
            (typ, _) = a.dtype.fields[field]
            descr.append((field, typ))
        newdtype.extend(tuple(descr))
    newrecarray = np.zeros(len(arrays[0]), dtype = newdtype)
    for a in arrays:
        for name in a.dtype.names:
            newrecarray[name] = a[name]
    return newrecarray

EDIT: with the suggestions of Sven it becomes (a little bit slower, but actually pretty readable):

def join_struct_arrays2(arrays):
    newdtype = sum((a.dtype.descr for a in arrays), [])
    newrecarray = np.empty(len(arrays[0]), dtype = newdtype)
    for a in arrays:
        for name in a.dtype.names:
            newrecarray[name] = a[name]
    return newrecarray

Answered By: joris

Answer 4

def join_struct_arrays(*arrs):
    dtype = [(name, d[0]) for arr in arrs for name, d in arr.dtype.fields.items()]
    r = np.empty(arrs[0].shape, dtype=dtype)
    for a in arrs:
       for name in a.dtype.names:
           r[name] = a[name]
    return r

Answered By: lfjbb

Numpy: Joining structured arrays?

Question:

Input

Desired Output

Current Approach

Answers: