Function Initialize And Object Initialization (multiprocessing)

I recently saw a answer/comment about how functions are objects in python. So, I wondering why when I take that example, and create a class around it while initializing a variable

Solution 1:

Instance methods are not automatically picklable. So

p.imap(f.f, jobs)

fails because p.imap tries to pickle the arguments. There is a way to "teach" pickle how to pickle instance methods (see Steven Bethard's answer), but your code has another problem: Passing the queue to the instance leads to a RuntimeError:

RuntimeError: Queue objects should only be shared between processes through inheritance

The error message is a little confusing (at least to me) since you can pass the queue as an argument to p.imap, but you can not pass it to the class F first, and then transfer it to the worker processes through f.f.

Anyway, because of these problems, I'd suggest sticking with the original code instead of trying to wrap the code in a class.

---

Here is an example of how to pickle instance methods:

import multiprocessing as mp
import copy_reg
import types

def_pickle_method(method):
    # Author: Steven Bethard (author of argparse)# http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-# instancemethods
    func_name = method.im_func.__name__
    obj = method.im_self
    cls = method.im_classcls_name = ''if func_name.startswith('__') andnot func_name.endswith('__'):
        cls_name = cls.__name__.lstrip('_')
    if cls_name:
        func_name = '_' + cls_name + func_name
    return _unpickle_method, (func_name, obj, cls)


def_unpickle_method(func_name, obj, cls):
    # Author: Steven Bethard# http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-# instancemethodsfor cls in cls.mro():
        try:
            func = cls.__dict__[func_name]
        except KeyError:
            passelse:
            breakreturn func.__get__(obj, cls)

# This call to copy_reg.pickle allows you to pass methods as the first arg to# mp.Pool methods. If you comment out this line, `pool.map(self.foo, ...)` results in# PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup# __builtin__.instancemethod failed

copy_reg.pickle(types.MethodType, _pickle_method, _unpickle_method)

classF(object):
    deff(self, x):
        fq.put('Doing: ' + str(x))        
        return x*x

deff_init(q):
    # http://stackoverflow.com/a/3843313/190597 (Olson)global fq
    fq = q

defmain():
    jobs = range(1,6)
    q = mp.Queue()
    p = mp.Pool(None, f_init, [q]) 
    f = F()
    results = p.imap(f.f, jobs)
    p.close()

    for r in results:
        print(r, q.get())

if __name__ == '__main__':
    main()

yields

(1, 'Doing: 2')
(4, 'Doing: 3')
(9, 'Doing: 4')
(16, 'Doing: 1')
(25, 'Doing: 5')