| 1 | #!/usr/bin/env perl | ||
| 2 | |||
| 3 | =encoding utf8 | ||
| 4 | |||
| 5 | =head1 NAME | ||
| 6 | |||
| 7 | ppll | ||
| 8 | |||
| 9 | =cut | ||
| 10 | |||
| 11 | 5 | ▶ | use autodie; |
|
Subroutine
BEGIN: called CC=1 SCAR=0.0
| |||
| 12 | 5 | ▶ | use strict; |
|
Subroutine
BEGIN: called CC=1 SCAR=0.0
| |||
| 13 | 5 | ▶ | use utf8::all; |
|
Subroutine
BEGIN: called CC=1 SCAR=0.0
| |||
| 14 | 5 | ▶ | use v5.20; |
|
Subroutine
BEGIN: called CC=1 SCAR=0.0
| |||
| 15 | 5 | ▶ | use warnings; |
|
Subroutine
BEGIN: called CC=1 SCAR=0.0
| |||
| 16 | |||
| 17 | 5 | ▶ | use FindBin; |
|
Subroutine
BEGIN: called CC=1 SCAR=0.0
| |||
| 18 | |||
| 19 | 5 | ▶ | use lib "$FindBin::Bin/../lib"; |
|
Subroutine
BEGIN: called CC=1 SCAR=0.0
| |||
| 20 | |||
| 21 | 5 | require App::ppll; | |
| 22 | |||
| 23 | 0 | exit App::ppll->new->call; | |
| 24 | |||
| 25 | =head1 SYNOPSIS | ||
| 26 | |||
| 27 | ppll [OPTIONS] COMMAND [ARGS] | ||
| 28 | ppll [OPTIONS] -c COMMAND [PARAMETERS] | ||
| 29 | ppll --help | ||
| 30 | ppll --version | ||
| 31 | |||
| 32 | In the first form, all arguments after I<COMMAND> are considered arguments to | ||
| 33 | I<COMMAND>, not options to C<ppll>, even if they begin with a hyphen. | ||
| 34 | |||
| 35 | =head1 DESCRIPTION | ||
| 36 | |||
| 37 | C<ppll> is a tool to control the execution of commands. It can run commands in | ||
| 38 | parallel, construct commands from lists of parameters, and more. | ||
| 39 | |||
| 40 | It handles the output of commands and can prefix lines with which command | ||
| 41 | produced it, print timestamps, etc. | ||
| 42 | |||
| 43 | C<ppll> has functionality similar to C<xargs> and C<parallel>. | ||
| 44 | |||
| 45 | =head2 PARAMETERS AND JOBS | ||
| 46 | |||
| 47 | ‘Parameters’ is what makes C<ppll> run multiple commands. In one way or | ||
| 48 | another C<ppll> can be given a number of parameters, and it will create and run | ||
| 49 | one ‘job’ for each parameter. | ||
| 50 | |||
| 51 | If no parameter-related option is used C<ppll> will read the parameters from | ||
| 52 | STDIN (if it’s not a TTY). Ordinarily, each line read becomes one parameter, but | ||
| 53 | there are options to control this. | ||
| 54 | |||
| 55 | The other important input to C<ppll> is the ‘command’. The command-line | ||
| 56 | arguments after all options make up the command and its arguments. | ||
| 57 | |||
| 58 | Each job will execute the command. All occurrences of C<{}> in any part of the | ||
| 59 | command (and its arguments) will be replaced with the job’s parameter. If there | ||
| 60 | are no such occurrences the parameter will be appended as an additional | ||
| 61 | argument. | ||
| 62 | |||
| 63 | For example: | ||
| 64 | |||
| 65 | ppll echo {} <<EOF | ||
| 66 | cow | ||
| 67 | pig | ||
| 68 | sheep | ||
| 69 | EOF | ||
| 70 | |||
| 71 | Will create jobs that execute C<echo cow>, C<echo pig> and so on. An other way | ||
| 72 | to do the same thing would be: | ||
| 73 | |||
| 74 | ppll echo <<EOF | ||
| 75 | cow | ||
| 76 | pig | ||
| 77 | sheep | ||
| 78 | EOF | ||
| 79 | |||
| 80 | C<{}> becomes useful when it’s not supposed to be the last argument, e.g: | ||
| 81 | |||
| 82 | ppll cp {} farm/ <<EOF | ||
| 83 | cow | ||
| 84 | pig | ||
| 85 | sheep | ||
| 86 | EOF | ||
| 87 | |||
| 88 | Or when it’s supposed to be just a part of an argument, or show up more than | ||
| 89 | once, e.g: | ||
| 90 | |||
| 91 | ppll curl -o photo-of-a-{}.jpg http://example.com/img/{}.jpg <<EOF | ||
| 92 | cow | ||
| 93 | pig | ||
| 94 | sheep | ||
| 95 | EOF | ||
| 96 | |||
| 97 | The above would run | ||
| 98 | C<curl -o photo-of-a-cow.jpg http://example.com/img/cow.jpg>, etc. | ||
| 99 | |||
| 100 | =head3 JOB CONTROL | ||
| 101 | |||
| 102 | The default is to run several such jobs in parallel. Once a job is finished | ||
| 103 | another one is started, until there are no more jobs to run. | ||
| 104 | |||
| 105 | The number of jobs run at the same time is controlled by the B<--jobs> option. | ||
| 106 | The default is one per CPU (but not necessarily one on each CPU). | ||
| 107 | |||
| 108 | If a job fails (exits with a code other than 0), no more jobs will be started. | ||
| 109 | C<ppll> I<attempts to> stop other running jobs. | ||
| 110 | |||
| 111 | The B<--force> option tells C<ppll> to not care about failed jobs and always | ||
| 112 | keep starting new ones. Any failed jobs are still noted and will still affect | ||
| 113 | C<ppll>’s own exit code. | ||
| 114 | |||
| 115 | =head3 SERIAL LINES, PARALLEL FIELDS | ||
| 116 | |||
| 117 | This feature deserves its own section. | ||
| 118 | |||
| 119 | It can be used to parallelise heavy tasks, while also making sure certain tasks | ||
| 120 | are run in a particular order. | ||
| 121 | |||
| 122 | Each line of input will create a ‘group’ of parameters, that will be run before | ||
| 123 | the next group starts. Each line is split up into individual parameters (using | ||
| 124 | the C<IFS> environment variable), that will be run at the same time. | ||
| 125 | |||
| 126 | An example: | ||
| 127 | |||
| 128 | ppll --slpf wake-up.sh <<EOF | ||
| 129 | rooster | ||
| 130 | hens chickens | ||
| 131 | cows pigs sheep | ||
| 132 | cat | ||
| 133 | EOF | ||
| 134 | |||
| 135 | The above would run C<wake-up.sh> first for C<rooster>. I<Then> it would run it | ||
| 136 | for C<hens> and C<chickens>, at the same time, but after the C<rooster> job has | ||
| 137 | completed. Then C<cows>, C<pigs> and C<sheep> in parallel, and so on. | ||
| 138 | |||
| 139 | =head1 OPTIONS | ||
| 140 | |||
| 141 | =head2 GENERAL | ||
| 142 | |||
| 143 | =over | ||
| 144 | |||
| 145 | =item B<--help> | ||
| 146 | |||
| 147 | =item B<-h> | ||
| 148 | |||
| 149 | =item B<-?> | ||
| 150 | |||
| 151 | Print (this) help and exit. | ||
| 152 | |||
| 153 | =item B<--version> | ||
| 154 | |||
| 155 | =item B<-V> | ||
| 156 | |||
| 157 | Print B<ppll>’s version and exit. | ||
| 158 | |||
| 159 | =back | ||
| 160 | |||
| 161 | =head2 PARAMETERS AND JOBS | ||
| 162 | |||
| 163 | =over | ||
| 164 | |||
| 165 | =item B<--command> I<COMMAND> | ||
| 166 | |||
| 167 | =item B<-C> I<COMMAND> | ||
| 168 | |||
| 169 | Run I<COMMAND> as a shell command with each parameter passed as an argument. | ||
| 170 | This allows for shell constructs, such as piping, redirecting, etc. | ||
| 171 | |||
| 172 | If used then the rest of the command-line arguments are used as parameters, | ||
| 173 | instead of a command. | ||
| 174 | |||
| 175 | More or less the same as: | ||
| 176 | |||
| 177 | ppll $SHELL -c <COMMAND> - | ||
| 178 | |||
| 179 | (C<$SHELL> defaults to C</bin/sh>.) | ||
| 180 | |||
| 181 | =item B<--delimiter> I<DELIMITER> | ||
| 182 | |||
| 183 | =item B<-d> I<DELIMITER> | ||
| 184 | |||
| 185 | Use I<DELIMITER>, as a regex, to split lines into fields. Implies B<--fields>. | ||
| 186 | |||
| 187 | =item B<--empty> | ||
| 188 | |||
| 189 | Include empty parameters. These are otherwise discarded. | ||
| 190 | |||
| 191 | =item B<--fields> | ||
| 192 | |||
| 193 | =item B<-f> | ||
| 194 | |||
| 195 | Alias for C<--mode fields>. | ||
| 196 | |||
| 197 | =item B<--force> | ||
| 198 | |||
| 199 | =item B<-k> | ||
| 200 | |||
| 201 | Keep starting new jobs even if a job has failed. | ||
| 202 | |||
| 203 | =item B<--jobs> I<N> | ||
| 204 | |||
| 205 | =item B<-j> I<N> | ||
| 206 | |||
| 207 | Run (up to) this many jobs in parallel. | ||
| 208 | |||
| 209 | =item B<--lines> | ||
| 210 | |||
| 211 | =item B<-l> | ||
| 212 | |||
| 213 | Alias for C<--mode lines>. | ||
| 214 | |||
| 215 | =item B<--mode> I<MODE> | ||
| 216 | |||
| 217 | How input should be turned into parameters. I<MODE> is one of: | ||
| 218 | |||
| 219 | =over | ||
| 220 | |||
| 221 | =item C<auto> (default) | ||
| 222 | |||
| 223 | If there’s only one line of input, behave as if C<fields>, otherwise as if | ||
| 224 | C<lines>. | ||
| 225 | |||
| 226 | =item C<fields> | ||
| 227 | |||
| 228 | Split lines into fields and use each field as a parameter. | ||
| 229 | Splitting is done using a delimiter set with B<--delimiter>, or the C<IFS> | ||
| 230 | environment variable. | ||
| 231 | |||
| 232 | =item C<lines> | ||
| 233 | |||
| 234 | Use each line as a parameter. | ||
| 235 | |||
| 236 | =item C<slpf> | ||
| 237 | |||
| 238 | Split each line into parameters. All parameters from a single line will be run | ||
| 239 | in parallel, but lines will be run serially. | ||
| 240 | |||
| 241 | =back | ||
| 242 | |||
| 243 | See also: B<--fields>, B<--lines>, B<--serial-lines-parallel-fields> | ||
| 244 | |||
| 245 | =item B<--random> | ||
| 246 | |||
| 247 | =item B<-R> | ||
| 248 | |||
| 249 | Shuffle the parameters. | ||
| 250 | |||
| 251 | =item B<--replace-string> I<STRING> | ||
| 252 | |||
| 253 | =item B<-replstr> I<STRING> | ||
| 254 | |||
| 255 | =item B<-I=s> I<STRING> | ||
| 256 | |||
| 257 | Use I<STRING> instead of C<{}> as a placeholder for the parameter. | ||
| 258 | |||
| 259 | =item B<--reverse> | ||
| 260 | |||
| 261 | =item B<-r> | ||
| 262 | |||
| 263 | Use the parameters in reverse order. | ||
| 264 | |||
| 265 | =item B<--sequence> I<SPEC> | ||
| 266 | |||
| 267 | =item B<-s> I<SPEC> | ||
| 268 | |||
| 269 | Use I<SPEC> as parameters. | ||
| 270 | |||
| 271 | I<SPEC> is in one of the forms: | ||
| 272 | |||
| 273 | =over | ||
| 274 | |||
| 275 | =item C<N..M> | ||
| 276 | |||
| 277 | Parameters are from I<N> to I<M>, inclusive. | ||
| 278 | |||
| 279 | =item C<M> | ||
| 280 | |||
| 281 | Parameters are from 1 to I<M>, inclusive. | ||
| 282 | |||
| 283 | =back | ||
| 284 | |||
| 285 | =item B<--serial-lines-parallel-fields> | ||
| 286 | |||
| 287 | =item B<-slpf> | ||
| 288 | |||
| 289 | Alias for C<--mode slpf>. | ||
| 290 | |||
| 291 | =back | ||
| 292 | |||
| 293 | =head2 OUTPUT | ||
| 294 | |||
| 295 | =over | ||
| 296 | |||
| 297 | =item B<--colour> | ||
| 298 | |||
| 299 | =item B<--color> | ||
| 300 | |||
| 301 | =item B<--no-colour> | ||
| 302 | |||
| 303 | =item B<--no-color> | ||
| 304 | |||
| 305 | Turn on or off ANSI colours in the output. | ||
| 306 | |||
| 307 | The default is to enable colours if STDOUT is a TTY. | ||
| 308 | |||
| 309 | =item B<--markers> | ||
| 310 | |||
| 311 | =item B<--no-markers> | ||
| 312 | |||
| 313 | Prefix (the default), or don’t prefix, jobs’ STDOUT and STDERR output with | ||
| 314 | C<E<gt>> or C<≫>, respectively. | ||
| 315 | |||
| 316 | =item B<--quiet> | ||
| 317 | |||
| 318 | =item B<-q> | ||
| 319 | |||
| 320 | Print less. Can be repeated for less and less. | ||
| 321 | |||
| 322 | =item B<--summary> | ||
| 323 | |||
| 324 | =item B<-S> | ||
| 325 | |||
| 326 | =item B<--no-summary> | ||
| 327 | |||
| 328 | Print (the default if there’s more than one parameter), or don’t print, a | ||
| 329 | summary of how many jobs ran and which of them failed. | ||
| 330 | |||
| 331 | =item B<--timestamp-format> I<FORMAT> | ||
| 332 | |||
| 333 | Format timestamps using I<FORMAT>. See L<DateTime::strftime|DateTime::strftime>. | ||
| 334 | |||
| 335 | The default is C<%T.%3N>. | ||
| 336 | |||
| 337 | Implies B<--timestamps>. | ||
| 338 | |||
| 339 | =item B<--timestamps> | ||
| 340 | |||
| 341 | =item B<-t> | ||
| 342 | |||
| 343 | Prefix lines from jobs’ output with a timestamp. The time printed will be the | ||
| 344 | I<approximate> time the line was printed by the job. | ||
| 345 | |||
| 346 | =item B<--verbose> | ||
| 347 | |||
| 348 | =item B<-v> | ||
| 349 | |||
| 350 | Print more. Can be repeated for more and more. | ||
| 351 | |||
| 352 | =back | ||
| 353 | |||
| 354 | =head1 DIAGNOSTICS | ||
| 355 | |||
| 356 | =head1 CONFIGURATION AND ENVIRONMENT | ||
| 357 | |||
| 358 | C<ppll> uses some environment variables in some cases: | ||
| 359 | |||
| 360 | =over | ||
| 361 | |||
| 362 | =item C<COLUMNS> | ||
| 363 | |||
| 364 | If set then used when printing things across the whole width of the terminal. If | ||
| 365 | not set then C<ppll> will try to work out the terminal width in other ways, | ||
| 366 | falling back to 80. | ||
| 367 | |||
| 368 | =item C<IFS> | ||
| 369 | |||
| 370 | Used for splitting lines into fields, e.g. with C<--fields> or | ||
| 371 | C<--serial-lines-parallel-fields>. | ||
| 372 | |||
| 373 | =item C<SHELL> | ||
| 374 | |||
| 375 | Used with C<--commmand> (C<-c>). | ||
| 376 | |||
| 377 | =back | ||
| 378 | |||
| 379 | =head1 EXAMPLES | ||
| 380 | |||
| 381 | ppll -t make | ||
| 382 | |||
| 383 | Run a long-running command, e.g. C<make>, and prefix the output with timestamps. | ||
| 384 | |||
| 385 | ppll -j 10 -s 1..1000 gobbledygook --with-rigmarole --number | ||
| 386 | |||
| 387 | For each number from 1 to 1,000 (inclusive), run the C<gobbledygook> command | ||
| 388 | with some arguments. Add the a number as the last argument, e.g. C<gobbledygook | ||
| 389 | --with-rigmarole --number 42>. | ||
| 390 | |||
| 391 | Run no more than 10 such commands at a time. | ||
| 392 | |||
| 393 | ppll curl -o photo-of-a-{}.jpg http://example.com/img/{}.jpg <<EOF | ||
| 394 | cow | ||
| 395 | pig | ||
| 396 | sheep | ||
| 397 | EOF | ||
| 398 | |||
| 399 | Download three files in parallel. Note the use of C<{}> to use each parameter in | ||
| 400 | two places in each command. | ||
| 401 | |||
| 402 | =head1 SEE ALSO | ||
| 403 | |||
| 404 | =over | ||
| 405 | |||
| 406 | =item | ||
| 407 | L<xargs|https://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html> | ||
| 408 | |||
| 409 | POSIX, and thus available almost everywhere, and can run multiple commands based | ||
| 410 | on a list of parameters, much like C<ppll>. | ||
| 411 | |||
| 412 | =item L<GNU Parallel|https://www.gnu.org/software/parallel/> | ||
| 413 | |||
| 414 | Also runs commands in parallel, but with an interface quite different to | ||
| 415 | C<ppll>’s. | ||
| 416 | |||
| 417 | =item L<moreutil|https://joeyh.name/code/moreutils/>’s C<parallel> and C<ts> | ||
| 418 | |||
| 419 | moreutils contains a C<parallel> tool similar yet different to GNU Parallel. | ||
| 420 | |||
| 421 | It also has a separate tool for adding timestamps to output. | ||
| 422 | |||
| 423 | =back | ||
| 424 | |||
| 425 | =head1 AUTHOR | ||
| 426 | |||
| 427 | Theo -q Willows, C<< <theo@willows.se> >> | ||
| 428 | |||
| 429 | =head1 BUGS AND LIMITATIONS | ||
| 430 | |||
| 431 | Please report any bugs or feature requests through the web interface at | ||
| 432 | L<https://gitlab.com/munkei-software/ppll/issues>. | ||
| 433 | |||
| 434 | =head1 SUPPORT | ||
| 435 | |||
| 436 | You can find documentation for this module with the perldoc command. | ||
| 437 | |||
| 438 | perldoc ppll | ||
| 439 | |||
| 440 | Or: | ||
| 441 | |||
| 442 | ppll --help | ||
| 443 | |||
| 444 | You can also look for information at: | ||
| 445 | |||
| 446 | =over | ||
| 447 | |||
| 448 | =item * MetaCPAN | ||
| 449 | |||
| 450 | L<https://metacpan.org/pod/App::ppll> | ||
| 451 | |||
| 452 | =item * GitLab | ||
| 453 | |||
| 454 | L<https://gitlab.com/munkei-software/ppll> | ||
| 455 | |||
| 456 | =back | ||
| 457 | |||
| 458 | =head1 LICENSE AND COPYRIGHT | ||
| 459 | |||
| 460 | Copyright 2019 Theo Willows. | ||
| 461 | |||
| 462 | This program is free software; you can redistribute it and/or modify it under | ||
| 463 | the terms of either: the GNU General Public License as published by the Free | ||
| 464 | Software Foundation; or the Artistic License. | ||
| 465 | |||
| 466 | See L<http://dev.perl.org/licenses/> for more information. | ||
| 467 | |||
| 468 | =cut | ||