Python multiprocess pool模塊報錯pickling error怎么辦

小編給大家分享一下Python multiprocess pool模塊報錯pickling error怎么辦，希望大家閱讀完這篇文章之后都有所收獲，下面讓我們一起去探討吧！

具體如下：

問題

之前在調(diào)用class內(nèi)的函數(shù)用multiprocessing模塊的pool函數(shù)進行多線程處理的時候報了以下下錯誤信息:

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

查了下官方文檔發(fā)現(xiàn)python默認只能pickle以下的類型:

• None, True, and False

• integers, floating point numbers, complex numbers

• strings, bytes, bytearrays

• tuples, lists, sets, and dictionaries containing only picklable objects

• functions defined at the top level of a module (using def, not lambda)

• built-in functions defined at the top level of a module

• classes that are defined at the top level of a module

• instances of such classes whose dict or the result of calling getstate() is picklable (see section -

• Pickling Class Instances for details).

函數(shù)只能pickle在頂層定義的函數(shù)，很明顯的class內(nèi)的函數(shù)無法被pickle因此會報錯。

import multiprocessing
def work():  # top-level 函數(shù)
  print "work!"
class Foo():
  def work(self): # 非top-level函數(shù)
    print "work"
pool1 = multiprocessing.Pool(processes=4)
foo = Foo()
pool1.apply_async(foo.work)
pool1.close()
pool1.join()
# 此時報錯
pool2 = multiprocessing.Pool(processes=4)
pool2.apply_async(work)
pool2.close()
pool2.join()
# 此時工作正常

解決方案

調(diào)用pathos包下的multiprocessing模塊代替原生的multiprocessing。pathos中multiprocessing是用dill包改寫過的，dill包可以將幾乎所有python的類型都serialize，因此都可以被pickle?；蛘咭部梢宰约河胐ill寫一個（有點重復(fù)造輪子之嫌?。?/p>

看完了這篇文章，相信你對“Python multiprocess pool模塊報錯pickling error怎么辦”有了一定的了解，如果想了解更多相關(guān)知識，歡迎關(guān)注億速云行業(yè)資訊頻道，感謝各位的閱讀！