python multiprocessing pool.map用于多个参数

  • Z时代
  • 2024-01-10
  • 分类：问答

在Python多处理库中，是否存在pool.map的变体，它支持多个参数？

text = "test"
def harvester(text, case):
    X = case[0]
    text+ str(X)
if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=6)
    case = RAW_DATASET
    pool.map(harvester(text,case),case, 1)
    pool.close()
    pool.join()

回答：

答案取决于版本和情况。JF Sebastian首先描述了最近版本的Python（从3.3开始）的最一般的答案。1它使用Pool.starmap方法，该方法接受一个参数元组序列。然后，它会自动将每个元组的参数解包，并将其传递给给定的函数：

import multiprocessing
from itertools import product
def merge_names(a, b):
    return '{} & {}'.format(a, b)
if __name__ == '__main__':
    names = ['Brown', 'Wilson', 'Bartlett', 'Rivera', 'Molloy', 'Opie']
    with multiprocessing.Pool(processes=3) as pool:
        results = pool.starmap(merge_names, product(names, repeat=2))
    print(results)
# Output: ['Brown & Brown', 'Brown & Wilson', 'Brown & Bartlett', ...

对于早期版本的Python，您需要编写一个辅助函数来显式解压缩参数。如果要使用with，则还需要编写一个包装器以变为Pool上下文管理器。（感谢muon指出这一点。）

import multiprocessing
from itertools import product
from contextlib import contextmanager
def merge_names(a, b):
    return '{} & {}'.format(a, b)
def merge_names_unpack(args):
    return merge_names(*args)
@contextmanager
def poolcontext(*args, **kwargs):
    pool = multiprocessing.Pool(*args, **kwargs)
    yield pool
    pool.terminate()
if __name__ == '__main__':
    names = ['Brown', 'Wilson', 'Bartlett', 'Rivera', 'Molloy', 'Opie']
    with poolcontext(processes=3) as pool:
        results = pool.map(merge_names_unpack, product(names, repeat=2))
    print(results)
# Output: ['Brown & Brown', 'Brown & Wilson', 'Brown & Bartlett', ...

在更简单的情况下，使用固定的第二个参数，您也可以使用partial，但仅在Python 2.7+中使用。

import multiprocessing
from functools import partial
from contextlib import contextmanager
@contextmanager
def poolcontext(*args, **kwargs):
    pool = multiprocessing.Pool(*args, **kwargs)
    yield pool
    pool.terminate()
def merge_names(a, b):
    return '{} & {}'.format(a, b)
if __name__ == '__main__':
    names = ['Brown', 'Wilson', 'Bartlett', 'Rivera', 'Molloy', 'Opie']
    with poolcontext(processes=3) as pool:
        results = pool.map(partial(merge_names, b='Sons'), names)
    print(results)
# Output: ['Brown & Sons', 'Wilson & Sons', 'Bartlett & Sons', ...

以上是 python multiprocessing pool.map用于多个参数 的全部内容， 来源链接： utcz.com/qa/435804.html

回到顶部