How to maximize performance of spaCy on an M1 Mac (currently much slower than Intel)

I’ve observed that nlp.pipe is 30-40% slower on my almost brand new M1 Pro Macbook than on my old Macbook Pro from 2017. Most other functions are faster on the M1 by a similar margin, so this is not the performance I would expect.

For a benchmark, I’m running the following code (with scispacy):

import os
from pathlib import Path
import re
import spacy
import time


with open(Path(os.path.expanduser('~')) / 'Downloads' / 'icd10cm-table and index-April 1 2023' / 'icd10cm-tabular-April 1 2023.xml') as f:
    content = f.read()
matches = re.findall(r'<desc>([^<]*)</desc>', content)

nlp = spacy.load('en_core_sci_lg')
start_time = time.time()
for x in nlp.pipe(matches):
    pass
print('%s seconds elapsed' % (time.time() - start_time))

My M1 Mac takes over 75 seconds to complete the task, while my 2017 Intel Mac can do it in 46 seconds.

I don’t know whether spacy uses numpy, but I installed a fast version of numpy using ABarrier’s answer to this question. That made numpy faster, but made no difference for spacy. I’m assuming that somewhere there is an unoptimized binary being used, but I don’t know how to figure out what it is.

Instructions to replicate my benchmark:

You can get the file I’m using here (cdc.gov); it’s a table of ICD-10 concepts in XML format. If you don’t have scispacy or don’t have the en_core_sci_lg model, create an environment and run

pip install scispacy
pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.1/en_core_sci_lg-0.5.1.tar.gz