ai2_kit.tool.batch module

class ai2_kit.tool.batch.BatchTool

Bases: object

A toolkit to help generate batch scripts.

gen_batches(*work_dirs: str, out: str, cmd: str | None = None, concurrency: int = 1, header_file: str | None = None, suppress_error: bool = False, checkpoint: bool = True, checkpoint_file: str = 'done.ckpt', rel_path: bool = False)

Generate batch scripts for each work directory.

This command will apply cmd to each work directory and generate batch scripts according to concurrency.

Parameters:

• work_dirs – path or glob of work directories

• out – path to write batch scripts, use {i} to represent the index of concurrent job

• cmd – command to run, if None, will read from stdin, use {word_dir} to represent the word directory,

use {i} to represent the index of concurrent job :param concurrency: number of concurrent jobs, decide the number of batch scripts to generate, if 0, will generate one batch script for each work directory :param header_file: path to header file, will be added to the beginning of each batch script :param suppress_error: if True, will add set -e to the beginning of each batch script :param checkpoint: if True, will add checkpoint to each batch script, and skip the work directory if checkpoint exists :param checkpoint_file: checkpoint file name :param rel_path: if True, will use relative path in batch script

map_path(*sources: str, target: str, copy=False)

Map source files or directory to target path, use link by default.

Parameters:

• sources – path or glob of source files or directories

• target – target path, use {i} to represent the index of the source path,

or {basename} to represent the basename of the source path :param copy: use copy instead of link

run_cmd(*work_dirs: str, cmd: str)

Run command in each work directory.

Parameters:

• work_dirs – path or glob of work directories

• cmd – command to run, use {work_dir} to represent the work directory