Joblib is a user friendly package enabling reliable local parallel processing on a wide variety of tasks and is the recommended method for sofia_redux.toolkit. Documentation may be found at https://joblib.readthedocs.io.

Usage

multitask() is the high level wrapper function that makes use of joblib for processing tasks in parallel. If multitask() fails a second attempt will be made to process the tasks serially.

Multitask Example (Joblib)

from sofia_redux.toolkit.utilities.multiprocessing import multitask

def multi_add_ten(args, i):
    return args[i] + 10

numbers = [10, 11, 12, 13, 14]
indices = range(len(numbers))
multitask(multi_add_ten, indices, numbers, None, jobs=-1)
# gives [20, 21, 22, 23, 24]

The above code uses joblib to add 10 to a set of numbers in parallel. jobs=-1 tells joblib to perform the processing using all available CPUs. There is virtually no overhead using this method compared with other methods such as dask.