sklearn clustering with custom metric: pairwise_distances throwing error

I would like to cluster sets of spatial data using my own metric. The data comes as pairs of (x,y) values in a dataframe, where each set of pairs has an id. Like in the following example where I have three sets of points:

import pandas as pd 
import numpy as np

df = pd.DataFrame({'id': [1] * 4 + [2] * 5 + [3] * 3, 
                   'x': np.random.random(12),
                   'y': np.random.random(12)}) 
df['xy'] = df[['x','y']].apply(lambda row: [row['x'],row['y']], axis = 1)

Here is the distance function I would like to use:

from scipy.spatial.distance import directed_hausdorff
def some_distance(u, v):
    return max(directed_hausdorff(u, v)[0], directed_hausdorff(v, u)[0])

This function computes the Hausdorff distance, i.e. the distance between two subsets u and v of n-dimensional space. In my case, I would like to use this distance function to cluster subsets of the real plane. In the data above there are three such subsets (ids from 1 to 3) so the resulting distance matrix should be 3×3.

My idea for the clustering step was to use sklearn.cluster.AgglomerativeClustering with a precomputed metric, which in turn I want to compute with sklearn.metrics.pairwise import pairwise_distances.

from sklearn.metrics.pairwise import pairwise_distances
def to_np_array(col):
    return np.array(list(col.values))
X = df.groupby('id')['xy'].apply(to_np_array).as_matrix()
m = pairwise_distances(X, X, metric=some_distance)

However, the last line is giving me an error:

ValueError: setting an array element with a sequence.

What does work fine, however, is calling some_distance(X[1], X[2]).
My hunch is that X needs to be a different format for pairwise_distances to work. Any ideas on how to make this work, or how to compute the matrix myself so I can stick it into sklearn.cluster.AgglomerativeClustering?

The error stack is

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-e34155622595> in <module>
     12 def some_distance(u, v):
     13     return max(directed_hausdorff(u, v)[0], directed_hausdorff(v, u)[0])
---> 14 m = pairwise_distances(X, X, metric=some_distance)

C:ProgramDataAnaconda3libsite-packagessklearnmetricspairwise.py in pairwise_distances(X, Y, metric, n_jobs, **kwds)
   1430         func = partial(distance.cdist, metric=metric, **kwds)
   1431 
-> 1432     return _parallel_pairwise(X, Y, func, n_jobs, **kwds)
   1433 
   1434 

C:ProgramDataAnaconda3libsite-packagessklearnmetricspairwise.py in _parallel_pairwise(X, Y, func, n_jobs, **kwds)
   1065 
   1066     if effective_n_jobs(n_jobs) == 1:
-> 1067         return func(X, Y, **kwds)
   1068 
   1069     # TODO: in some cases, backend='threading' may be appropriate

C:ProgramDataAnaconda3libsite-packagessklearnmetricspairwise.py in _pairwise_callable(X, Y, metric, **kwds)
   1079     """Handle the callable case for pairwise_{distances,kernels}
   1080     """
-> 1081     X, Y = check_pairwise_arrays(X, Y)
   1082 
   1083     if X is Y:

C:ProgramDataAnaconda3libsite-packagessklearnmetricspairwise.py in check_pairwise_arrays(X, Y, precomputed, dtype)
    106     if Y is X or Y is None:
    107         X = Y = check_array(X, accept_sparse='csr', dtype=dtype,
--> 108                             warn_on_dtype=warn_on_dtype, estimator=estimator)
    109     else:
    110         X = check_array(X, accept_sparse='csr', dtype=dtype,

C:ProgramDataAnaconda3libsite-packagessklearnutilsvalidation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)
    525             try:
    526                 warnings.simplefilter('error', ComplexWarning)
--> 527                 array = np.asarray(array, dtype=dtype, order=order)
    528             except ComplexWarning:
    529                 raise ValueError("Complex data not supportedn"

C:ProgramDataAnaconda3libsite-packagesnumpycorenumeric.py in asarray(a, dtype, order)
    536 
    537     """
--> 538     return array(a, dtype, copy=False, order=order)
    539 
    540 

ValueError: setting an array element with a sequence.