using imaplib.IMAP4_SSL in multiple processes

Question:

I want to reuse imaplib.IMAP4_SSL instance over many processes so I don’t have to login multiple times.
Here is some code:

import imaplib
from multiprocessing import Process

def fetch(mail_client):
    mail_client.uid('fetch', b'1', 'BODY[TEXT]')

def main():
    c = imaplib.IMAP4_SSL('imap.gmail.com')
    c.login(user='**', password='***')
    c.select('inbox')

    procs = [Process(target=fetch, args=(c,)) for _ in range(100)]
    for p in procs:
        p.start()

    for p in procs:
        p.join()

if __name__ == '__main__':
    main()

But I get the errors that are related to socket:

imaplib.IMAP4.abort: socket error: [Errno 32] Broken pipe

I thought that this is because processes is writing to the same socket, that imaplib.IMAP4_SSL has, so i tried to add multiprocessing.Lock to prevent simultaneous access:

import imaplib
from multiprocessing import Process, Lock


def fetch(mail_client, lock):
    with lock:
        mail_client.uid('fetch', b'1', 'BODY[TEXT]')


def main():
    c = imaplib.IMAP4_SSL('imap.gmail.com')
    c.login(user='[email protected]', password='Qwe=1dSAzxc+%')
    c.select('inbox')
    lock = Lock()

    procs = [Process(target=fetch, args=(c, lock)) for _ in range(100)]
    for p in procs:
        p.start()

    for p in procs:
        p.join()


if __name__ == '__main__':
    main()

But the error persists.
Some further investigation has shown that the first process calls mail.uid successfully, but second process gets imaplib.IMAP4.abort: command: UID => socket error: EOF anyway.

I am using Ubuntu 16.04.
Any suggestions are highly appreciated.

Update: found another exception in stacktrace, maybe it causes all other exceptions:

ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2217)`

Seems like this issue is related
Python ssl problem with multiprocessing

Asked By: Alexandr Tatarinov

||

Answers:

Seems like is a problem with SSL. SSL-wrapped socket cannot be reused over multiple processes. I am using threads now and everything is okay without Lock.

Answered By: Alexandr Tatarinov

You can’t. The SSL context isn’t shared between processes; without it, the encryption state gets out of sync.

If you need to perform multiple concurrent operations on the same IMAP inbox, you will need to make multiple connections to the server.

Answered By: user149341

Did you ever solve this? I would love to know.

Also, is there any reason why you do not want to initiate multiple connections?

Answered By: דברי