multijob.commandline module

Turn job objects into command line arguments and back again.

The usage of these functions is explained in more detail in the From scripts to GNU Parallel tutorial.

You should use job_from_argv() and argv_from_job() to convert between multijob.job.Job instances and command line parameters. These functions use typemaps and coercions.

A typemap is a dictionary that maps named params to a specific coercion.

A coercion is a function that turns a string into a Python data type or back (see Coercion).

If you need to take more control over parsing, you can get convenient random-access to the command line parameters via UnparsedArguments, or can apply a particular coercion to a string value via value_from_string().

To turn a job into a shell command, use shell_command_from_job(). This uses the shell_word_from_string() function for escaping.

Example: Turning a list of job objects into a command line script:

>>> from multijob.job import JobBuilder
>>> def dummy(): pass
>>>
>>> # create the jobs
>>> builder = JobBuilder()
>>> _ = builder.add('a', 2)
>>> _ = builder.add_linspace('b', 1.0, 3.0, 3)
>>> jobs = builder.build(dummy, repetitions=1)
>>>
>>> # print the jobs
>>> for job in jobs:
...     print(shell_command_from_job('./run-jobs', job))
./run-jobs --id=0 --rep=0 -- a=2 b=1.0
./run-jobs --id=1 --rep=0 -- a=2 b=2.0
./run-jobs --id=2 --rep=0 -- a=2 b=3.0

Example: Decoding arguments inside a script:

>>> # define a worker function and a typemap for it
>>> TYPEMAP = dict(a='int', b='float')
>>> def worker(a, b):
...     return a * b
...
>>> # argv = sys.argv
>>> argv = ['./run-jobs', '--id=2', '--rep=0', '--', 'a=2', 'b=3.0']
>>> # skip the first argument
>>> argv = argv[1:]
>>>
>>> # decode the job
>>> job = job_from_argv(argv, worker, typemap=TYPEMAP)
>>>
>>> # run the job
>>> result = job.run()
>>> result.result
6.0
class multijob.commandline.Coercion(_private, _parser)[source]

Bases: object

Coercion from command line args to Python objects.

Coercions describe how command line arguments are converted to Python data types. This concept is used throughout the module documentation.

Coercions are functions that take a single parameter (the string to be coerced) and return a single Python object. Coercions may also be named. Instead of supplying a callback, the coercion may be one of:

  • 'str'
  • 'int'
  • 'float'
  • 'bool'

Example: coercion function:

>>> Coercion.of(int, paramname='x')('x', '42')
42
>>> Coercion.of(str, paramname='y')('y', '42')
'42'

Example: named coercion:

>>> Coercion.of('bool', paramname='p')('p', 'False')
False
>>> Coercion.of('float', paramname='f')('f', '4.2')
4.2

Example: coercions are coercions:

>>> a = Coercion.of(int, paramname='x')
>>> Coercion.of(a, paramname='y')
<multijob.commandline.Coercion object at 0x...>

Example: coercion must be callable:

>>> Coercion.of(42, paramname='blorp')
Traceback (most recent call last):
TypeError: 'blorp' coercion must be callable: 42

Example: explains error when coercion fails:

>>> def bad_coercion(value):
...     raise ValueError("nope")
...
>>> coercion = Coercion.of(bad_coercion, paramname='x')
>>> coercion('x', '42')
Traceback (most recent call last):
ValueError: Could not coerce 'x'='42':
nope

Example: propagates Nones:

>>> Coercion.of(None, paramname='x') is None
True
static of(name_or_callable, *, paramname, named_coercions=_Default)[source]

Create/check a coercion.

Calling this functions turns a callable or named coercion into a proper coercion object with suitable error handling.

Parameters:
  • name_or_callable (str|Callable|Coercion) – The coercion.
  • paramname (str) – The name of the parameter which we’re going to coerce. This is only used for error messages.
  • named_coercions (dict) – Optional. Available named coercions. Defaults to Coercion.NAMED_COERCIONS.
Returns:

A coercion object with suitable error handling. Returns None if name_or_callable was None.
Return type:

Coercion
class multijob.commandline.JobArgvConfig(*, job_id_key, repetition_id_key)[source]

Bases: object

Specify how job parameters are encoded.

job_id_key

str – Name of the job_id attribute.

repetition_id_key

str – Name of the repetition_id attribute.

class multijob.commandline.UnparsedArguments(args)[source]

Bases: object

A collection of unnamed arguments.

This may be useful if you have to do some parsing manually.

Use from_argv() to construct this object from an argv array.

Parameters:args (dict) – the name-value command line parameters.
static from_argv(argv)[source]

Create UnparsedArguments from an argv array.

read(name, coercion)[source]

Consume and coerce a named argument.

Since the argument is consumed, it cannot be consumed again.

Parameters:
  • name (str) – The name of the argument to consume.
  • coercion (Coercion) – The coercion to apply to this value.
Returns:

the coerced value.
Raises:
  • KeyError – when no such argument name exists.
  • ValueError – when the value can’t be coerced.
multijob.commandline.argv_from_job(job, *, typemap=None, default_coercion=None, job_argv_config=None)[source]

Format a job as command line arguments.

To get a job object back from these arguments, use job_from_argv().

The job.callback will not be stored and will have to be provided explicitly when parsing the arguments.

Parameters:
  • job (multijob.job) – The job to format.
  • typemap (Typemap) – Optional. Controls how individual params are formatted.
  • default_coercion (Coercion) – Optional. Controls how params without a typemap entry are formatted.
  • job_argv_config (JobArgvConfig) – Optional. Controls names of job attributes.
Returns:

The encoded params.
Return type:

list

Example: simple usage:

>>> from multijob.job import Job
>>> def target(): pass
>>> job = Job(42, 3, target, dict(a=42, b=True, c='foo'))
>>> argv_from_job(job)
['--id=42', '--rep=3', '--', 'a=42', 'b=True', 'c=foo']

Example: storing a list:

>>> from multijob.job import Job
>>> import csv
>>> import io
>>> def csv_line(items):
...     out = io.StringIO()
...     csv.writer(out).writerow(items)
...     return out.getvalue().strip('\r\n')
>>> def target():
...     pass
>>> job = Job(3, 0, target, dict(xs=[1, 2, 3]))
>>> argv_from_job(job, typemap=dict(xs=csv_line))
[..., '--', 'xs=1,2,3']

Example: customizing job format:

>>> from multijob.job import Job
>>> conf = JobArgvConfig(
...     job_id_key='--job-id',
...     repetition_id_key='--nexecs')
>>> job = Job(2, 7, lambda x: x, dict(a='b'))
>>> argv_from_job(job, job_argv_config=conf)
['--job-id=2', '--nexecs=7', '--', 'a=b']
multijob.commandline.job_from_argv(argv, callback, *, typemap, default_coercion=None, job_argv_config=None)[source]

Parse command line arguments into a job object.

Parameters:
  • argv (list) – The arguments.
  • callback (callable) – A function to invoke with the params, see multijob.job.Job for details.
  • typemap (Typemap) – Controls how individual params are parsed.
  • default_coercion (Coercion) – Optional. Controls how params without a typemap entry are formatted.
  • job_argv_config (JobArgvConfig) – Optional. Controls names of job attributes.
Returns:

a runnable job with the params from this argv.
Return type:

multijob.job.Job

Example: simple usage:

>>> def target(a, b, c):
...     print("Args: a={!r}, b={!r}, c={!r}".format(a, b, c))
...     return "some value"
...
>>> argv = ['--id=42', '--rep=3', '--', 'a=42', 'b=True', 'c=foo']
>>> TYPEMAP = dict(a='int', b='bool', c='str')
>>> job = job_from_argv(argv, target, typemap=TYPEMAP)
>>> (job.job_id, job.repetition_id)
(42, 3)
>>> result = job.run()
Args: a=42, b=True, c='foo'
>>> result
<multijob.job.JobResult object at 0x...>

Example: reading a list:

>>> import csv
>>> def target():
...     pass
>>> def csv_line(line):
...     return [float(x) for x in list(csv.reader([line]))[0]]
>>> argv = ['--id=3', '--rep=14', '--', 'xs=1.2,3.4,5.6']
>>> job = job_from_argv(argv, target, typemap=dict(xs=csv_line))
>>> job.params['xs']
[1.2, 3.4, 5.6]

Example: customizing job format

>>> def target():
...     pass
>>> conf = JobArgvConfig(
...     job_id_key='--job-id',
...     repetition_id_key='--iteration')
>>> argv = ['--job-id=15', '--iteration=2', '--', 'a=foo']
>>> job = job_from_argv(argv, target,
...                     typemap=dict(a=str),
...                     job_argv_config=conf)
>>> (job.job_id, job.repetition_id, job.params)
(15, 2, {'a': 'foo'})
multijob.commandline.shell_command_from_job(prefix, job, *, typemap=None, default_coercion=None, job_argv_config=None)[source]

Turn a job into a shell command.

Parameters:
Returns:

The job as a shell command.
Return type:

str

Example:

>>> job = multijob.job.Job(40, 2, lambda x: x, dict(y='foo bar'))
>>> print(shell_command_from_job('$RUN_GA', job))
$RUN_GA --id=40 --rep=2 -- 'y=foo bar'
multijob.commandline.shell_word_from_string(word)[source]

Escape arguments for POSIX shells.

Parameters:word (str) – The word to escape.
Returns:The safely escaped word.
Return type:str

Example:

>>> print(shell_word_from_string(""))
''
>>> print(shell_word_from_string("foo"))
foo
>>> print(shell_word_from_string("foo bar"))
'foo bar'
>>> print(shell_word_from_string("foo'bar'baz"))
'foo'\''bar'\''baz'
multijob.commandline.value_from_string(name, value, coercion)[source]

Parse a value from a string with a given coercion.

This function primarily adds error handling, and contains support for named coercions.

Parameters:
  • name (str) – The name of this argument, needed for diagnostics.
  • value (str) – The string to coerce.
  • coercion (Coercion) – a Coercion. See Coercion.of() for details on acceptable values.
Returns:

the coerced value.

Example: normal usage:

>>> value_from_string('x', '42', int)
42

Example: complains when no coercion was provided:

>>> value_from_string('y', '13', None)
Traceback (most recent call last):
TypeError: no coercion found for 'y'='13'