Python multiprocessing EOF

Question:

So i am running into a very odd error. I am making a basic little GUI for a very basic TCP server, but when i spawn the process, it returns the following traceback:

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:Python27liblib-tkTkinter.py", line 1410, in __call__
    return self.func(*args)
  File "C:UsersusernameDesktoplocalsshserverv2.py", line 48, in start_server
    process.start()
  File "C:Python27libmultiprocessingprocess.py", line 130, in start
    self._popen = Popen(self)
  File "C:Python27libmultiprocessingforking.py", line 271, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:Python27libmultiprocessingforking.py", line 193, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:Python27libpickle.py", line 224, in dump
    self.save(obj)
  File "C:Python27libpickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:Python27libpickle.py", line 419, in save_reduce
    save(state)
  File "C:Python27libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python27libpickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:Python27libpickle.py", line 681, in _batch_setitems
    save(v)
  File "C:Python27libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python27libmultiprocessingforking.py", line 66, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "C:Python27libpickle.py", line 401, in save_reduce
    save(args)
  File "C:Python27libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python27libpickle.py", line 548, in save_tuple
    save(element)
  File "C:Python27libpickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:Python27libpickle.py", line 419, in save_reduce
    save(state)
  File "C:Python27libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python27libpickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:Python27libpickle.py", line 681, in _batch_setitems
    save(v)
  File "C:Python27libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python27libpickle.py", line 725, in save_inst
    save(stuff)
  File "C:Python27libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python27libpickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:Python27libpickle.py", line 681, in _batch_setitems
    save(v)
  File "C:Python27libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python27libpickle.py", line 725, in save_inst
    save(stuff)
  File "C:Python27libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python27libpickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:Python27libpickle.py", line 681, in _batch_setitems
    save(v)
  File "C:Python27libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python27libpickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:Python27libpickle.py", line 681, in _batch_setitems
    save(v)
  File "C:Python27libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python27libpickle.py", line 725, in save_inst
    save(stuff)
  File "C:Python27libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python27libpickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:Python27libpickle.py", line 681, in _batch_setitems
    save(v)
  File "C:Python27libpickle.py", line 313, in save
    (t.__name__, obj))
PicklingError: Can't pickle 'tkapp' object: <tkapp object at 0x02A4C4F0>

my code is the following:

import SocketServer
import multiprocessing
from Tkinter import *

class MyTCPHandler(SocketServer.BaseRequestHandler):
    """
    The RequestHandler class for our server.

    It is instantiated once per connection to the server, and must
    override the handle() method to implement communication to the
    client.
    """

    def handle(self):
        # self.request is the TCP socket connected to the client
        self.data = self.request.recv(1024).strip()
        print "{} wrote:".format(self.client_address[0])+str(self.data)
        #print self.data
        # just send back the same data, but upper-cased
        self.request.sendall(self.data.upper())



class Application(object):
    def __init__(self):
        self.root=Tk()
        self.root.resizable(FALSE,FALSE)
        self.root.geometry('500x500')
        self.root.title("Server GUI")
    def set_widgets(self):
        self.start_server_button=Button(self.root,text="Start Server",command=self.start_server)
        self.end_server_button=Button(self.root,text="Stop Server",command=self.stop_server)
        self.logger=Text(self.root,width=50)
    def grid_widgets(self):
        self.start_server_button.grid(row=1,column=0)
        self.end_server_button.grid(row=2,column=0)
        self.logger.grid(row=0,column=0)
    def configure(self):
        pass
    def run(self):
        self.set_widgets()
        self.grid_widgets()
        self.configure()
        self.root.mainloop()
    def start_server(self):
        self.logger.insert(END,"Starting process for client.")
        process=multiprocessing.Process(target=self.start_serving)
        process.start()
    def start_serving(self):
        HOST, PORT = "localhost", 9999
        self.server = SocketServer.TCPServer((HOST, PORT), MyTCPHandler)
        self.server.serve_forever()
    def stop_server(self):
        self.server.shutdown()
if __name__ == "__main__":
    Application().run()
Asked By: IT Ninja

||
Source

Answers:

I am not entirely sure if this will fix your problem, however, another stackoverflow question may have the answer you’re looking for. It appears that you need to add the __getstate__() method into the Application class.

Answered By: acrognale

Based on this error:

PicklingError: Can't pickle 'tkapp' object: <tkapp object at 0x02A4C4F0>

It looks like you might be trying to share/send an object to a subprocess, though implicitly, fundamentally the multiprocessing module by default, follows the "no share" methodology, meaning all objects are copied and sent from parent processes to subprocesses. This achieves a safe and true multithreading system, at the cost of performance among other things.

Here are requirements that make objects sendable to subprocesses:

  • Python uses pickle to serialize objects, so objects must be serializable via pickle.
  • Serialized function objects must be importable.

Given that the following method is not importable:

def start_server(self):
    self.logger.insert(END,"Starting process for client.")
    process=multiprocessing.Process(target=self.start_serving)
    process.start()

One guess is multiprocessing tried serialize all referenced member variables, most of which cannot be serialized, which led to the error.

You could try changing the method to be a top-level function instead:

def start_serving():
    HOST, PORT = "localhost", 9999
    server = SocketServer.TCPServer((HOST, PORT), MyTCPHandler)
    server.serve_forever()

This assumes no interprocess communication is needed.
If you do need interprocess communication, you can use utilities like multiprocessing.Queue.

Use this to start the subprocess:

 def start_server(self):
    self.logger.insert(END, "Starting process for client.")
    self.process = multiprocessing.Process(target=start_serving)
    # Make sure to maintain a reference to the process.
    self.process.start()

And this to kill it:

def stop_server(self):
    self.process.terminate()


if __name__ == "__main__":
    Application().run()

I recommend using ThreadingMixIn over manually creating a subprocess, though I’m not sure if it uses thread or process given that thread doesn’t actually run on multi-core machines due to the GIL while process do.

I’ve tested it, and unfortunately you can’t kill the subprocess. serve_forever seems to be blocking.

Answered By: Samy Vilar

I got the same problem: 

p=multiprocessing.Process(target=self.targetMethod)

I changed it to :

p=multiprocessing.Process(target=self.targetMethod())

I don’t get the EOF error anymore. I do not understand exactly what happens here.
It’s too late, but hope it helps somebody.

Answered By: Athena