src/sem.pod

   1 #!/usr/bin/perl -w
   2
   3 # SPDX-FileCopyrightText: 2021-2024 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
   4 # SPDX-License-Identifier: GFDL-1.3-or-later
   5 # SPDX-License-Identifier: CC-BY-SA-4.0
   6
   7 =head1 NAME
   8
   9 sem - semaphore for executing shell command lines in parallel
  10
  11 =head1 SYNOPSIS
  12
  13 B<sem> [--fg] [--id <id>] [--semaphoretimeout <secs>] [-j <num>] [--wait] command
  14
  15 =head1 DESCRIPTION
  16
  17 GNU B<sem> is an alias for GNU B<parallel --semaphore>.
  18
  19 GNU B<sem> acts as a counting semaphore. When GNU B<sem> is called
  20 with command it starts the command in the background. When I<num>
  21 number of commands are running in the background, GNU B<sem> waits for
  22 one of these to complete before starting the command.
  23
  24 GNU B<sem> does not read any arguments to build the command (no -a,
  25 :::, and ::::). It simply waits for a semaphore to become available
  26 and then runs the command given.
  27
  28 Before looking at the options you may want to check out the examples
  29 after the list of options. That will give you an idea of what GNU
  30 B<sem> is capable of.
  31
  32 =head1 OPTIONS
  33
  34 =over 9
  35
  36 =item I<command>
  37
  38 Command to execute. The command may be followed by arguments for the
  39 command.
  40
  41
  42 =item B<--bg>
  43
  44 Run command in background thus GNU B<sem> will not wait for
  45 completion of the command before exiting. This is the default.
  46
  47 In toilet analogy: GNU B<sem> waits for a toilet to be available,
  48 gives the toilet to a person, and exits immediately.
  49
  50 See also: B<--fg>
  51
  52
  53 =item B<--jobs> I<N>
  54
  55 =item B<-j> I<N>
  56
  57 =item B<--max-procs> I<N>
  58
  59 =item B<-P> I<N>
  60
  61 Run up to N commands in parallel. Default is 1 thus acting like a
  62 mutex.
  63
  64 In toilet analogy: B<-j> is the number of toilets.
  65
  66
  67 =item B<--jobs> I<+N>
  68
  69 =item B<-j> I<+N>
  70
  71 =item B<--max-procs> I<+N>
  72
  73 =item B<-P> I<+N>
  74
  75 Add N to the number of CPU cores.  Run up to this many jobs in
  76 parallel. For compute intensive jobs B<-j> +0 is useful as it will run
  77 number-of-cpu-cores jobs simultaneously.
  78
  79
  80 =item B<--jobs> I<-N>
  81
  82 =item B<-j> I<-N>
  83
  84 =item B<--max-procs> I<-N>
  85
  86 =item B<-P> I<-N>
  87
  88 Subtract N from the number of CPU cores.  Run up to this many jobs in
  89 parallel.  If the evaluated number is less than 1 then 1 will be used.
  90 See also B<--use-cpus-instead-of-cores>.
  91
  92
  93 =item B<--jobs> I<N>%
  94
  95 =item B<-j> I<N>%
  96
  97 =item B<--max-procs> I<N>%
  98
  99 =item B<-P> I<N>%
 100
 101 Multiply N% with the number of CPU cores.  Run up to this many jobs in
 102 parallel.  If the evaluated number is less than 1 then 1 will be used.
 103 See also B<--use-cpus-instead-of-cores>.
 104
 105
 106 =item B<--jobs> I<procfile>
 107
 108 =item B<-j> I<procfile>
 109
 110 =item B<--max-procs> I<procfile>
 111
 112 =item B<-P> I<procfile>
 113
 114 Read parameter from file. Use the content of I<procfile> as parameter
 115 for I<-j>. E.g. I<procfile> could contain the string 100% or +2 or
 116 10.
 117
 118
 119 =item B<--pipe>
 120
 121 Pass stdin (standard input) to I<command>.
 122
 123 If I<command> read from stdin (standard input), use B<--pipe>.
 124
 125
 126 =item B<--semaphorename> I<name>
 127
 128 =item B<--id> I<name>
 129
 130 Use B<name> as the name of the semaphore. Default is the name of the
 131 controlling tty (output from B<tty>).
 132
 133 The default normally works as expected when used interactively, but
 134 when used in a script I<name> should be set. I<$$> or I<my_task_name>
 135 are often a good value.
 136
 137 The semaphore is stored in ~/.parallel/semaphores/
 138
 139 In toilet analogy the name corresponds to different types of toilets:
 140 e.g. male, female, customer, staff.
 141
 142
 143 =item B<--fg>
 144
 145 Do not put command in background.
 146
 147 In toilet analogy: GNU B<sem> waits for a toilet to be available,
 148 takes a person to the toilet, waits for the person to finish, and
 149 exits.
 150
 151
 152 =item B<--semaphoretimeout> I<secs>
 153
 154 =item B<--st> I<secs>
 155
 156 If I<secs> > 0: If the semaphore is not released within I<secs>
 157 seconds, take it anyway.
 158
 159 If I<secs> < 0: If the semaphore is not released within I<secs>
 160 seconds, exit.
 161
 162 In toilet analogy: I<secs> > 0: If no toilet becomes available within
 163 I<secs> seconds, pee on the floor. I<secs> < 0: If no toilet becomes
 164 available within I<secs> seconds, exit without doing anything.
 165
 166
 167 =item B<--wait>
 168
 169 Wait for all commands to complete.
 170
 171 In toilet analogy: Wait until all toilets are empty, then exit.
 172
 173
 174 =back
 175
 176 =head1 UNDERSTANDING A SEMAPHORE
 177
 178 Try the following example:
 179
 180   sem -j 2 'sleep 1;echo 1 finished';   echo sem 1 exited
 181   sem -j 2 'sleep 2;echo 2 finished';   echo sem 2 exited
 182   sem -j 2 'sleep 3;echo 3 finished';   echo sem 3 exited
 183   sem -j 2 'sleep 4;echo 4 finished';   echo sem 4 exited
 184   sem --wait; echo sem --wait done
 185
 186 In toilet analogy this uses 2 toilets (B<-j 2>). GNU B<sem> takes '1'
 187 to a toilet, and exits immediately. While '1' is sleeping, another GNU
 188 B<sem> takes '2' to a toilet, and exits immediately.
 189
 190 While '1' and '2' are sleeping, another GNU B<sem> waits for a free
 191 toilet. When '1' finishes, a toilet becomes available, and this GNU
 192 B<sem> stops waiting, and takes '3' to a toilet, and exits
 193 immediately.
 194
 195 While '2' and '3' are sleeping, another GNU B<sem> waits for a free
 196 toilet.  When '2' finishes, a toilet becomes available, and this GNU
 197 B<sem> stops waiting, and takes '4' to a toilet, and exits
 198 immediately.
 199
 200 Finally another GNU B<sem> waits for all toilets to become free.
 201
 202
 203 =head1 EXAMPLE: Gzipping *.log
 204
 205 Run one gzip process per CPU core. Block until a CPU core becomes
 206 available.
 207
 208   for i in *.log ; do
 209     echo $i
 210     sem -j+0 gzip $i ";" echo done
 211   done
 212   sem --wait
 213
 214 =head1 EXAMPLE: Protecting pod2html from itself
 215
 216 pod2html creates two files: pod2htmd.tmp and pod2htmi.tmp which it
 217 does not clean up. It uses these two files for a short time. But if
 218 you run multiple pod2html in parallel (e.g. in a Makefile with make
 219 -j) there is a risk that two different instances of pod2html will
 220 write to the files at the same time:
 221
 222   # This may fail due to shared pod2htmd.tmp/pod2htmi.tmp files
 223   foo.html:
 224           pod2html foo.pod --outfile foo.html
 225
 226   bar.html:
 227           pod2html bar.pod --outfile bar.html
 228
 229   $ make -j foo.html bar.html
 230
 231 You need to protect pod2html from running twice at the same time.
 232 B<sem> running as a mutex will make sure only one runs:
 233
 234   foo.html:
 235           sem --id pod2html pod2html foo.pod --outfile foo.html
 236
 237   bar.html:
 238           sem --id pod2html pod2html bar.pod --outfile bar.html
 239
 240   clean: foo.html bar.html
 241           sem --id pod2html --wait
 242           rm -f pod2htmd.tmp pod2htmi.tmp
 243
 244   $ make -j foo.html bar.html clean
 245
 246 =head1 BUGS
 247
 248 None known.
 249
 250
 251 =head1 REPORTING BUGS
 252
 253 Report bugs to <bug-parallel@gnu.org>.
 254
 255
 256 =head1 AUTHOR
 257
 258 Copyright (C) 2010-2024 Ole Tange, http://ole.tange.dk and Free
 259 Software Foundation, Inc.
 260
 261
 262 =head1 LICENSE
 263
 264 This program is free software; you can redistribute it and/or modify
 265 it under the terms of the GNU General Public License as published by
 266 the Free Software Foundation; either version 3 of the License, or
 267 at your option any later version.
 268
 269 This program is distributed in the hope that it will be useful,
 270 but WITHOUT ANY WARRANTY; without even the implied warranty of
 271 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 272 GNU General Public License for more details.
 273
 274 You should have received a copy of the GNU General Public License
 275 along with this program.  If not, see <https://www.gnu.org/licenses/>.
 276
 277 =head2 Documentation license I
 278
 279 Permission is granted to copy, distribute and/or modify this
 280 documentation under the terms of the GNU Free Documentation License,
 281 Version 1.3 or any later version published by the Free Software
 282 Foundation; with no Invariant Sections, with no Front-Cover Texts, and
 283 with no Back-Cover Texts.  A copy of the license is included in the
 284 file LICENSES/GFDL-1.3-or-later.txt.
 285
 286 =head2 Documentation license II
 287
 288 You are free:
 289
 290 =over 9
 291
 292 =item B<to Share>
 293
 294 to copy, distribute and transmit the work
 295
 296 =item B<to Remix>
 297
 298 to adapt the work
 299
 300 =back
 301
 302 Under the following conditions:
 303
 304 =over 9
 305
 306 =item B<Attribution>
 307
 308 You must attribute the work in the manner specified by the author or
 309 licensor (but not in any way that suggests that they endorse you or
 310 your use of the work).
 311
 312 =item B<Share Alike>
 313
 314 If you alter, transform, or build upon this work, you may distribute
 315 the resulting work only under the same, similar or a compatible
 316 license.
 317
 318 =back
 319
 320 With the understanding that:
 321
 322 =over 9
 323
 324 =item B<Waiver>
 325
 326 Any of the above conditions can be waived if you get permission from
 327 the copyright holder.
 328
 329 =item B<Public Domain>
 330
 331 Where the work or any of its elements is in the public domain under
 332 applicable law, that status is in no way affected by the license.
 333
 334 =item B<Other Rights>
 335
 336 In no way are any of the following rights affected by the license:
 337
 338 =over 2
 339
 340 =item *
 341
 342 Your fair dealing or fair use rights, or other applicable
 343 copyright exceptions and limitations;
 344
 345 =item *
 346
 347 The author's moral rights;
 348
 349 =item *
 350
 351 Rights other persons may have either in the work itself or in
 352 how the work is used, such as publicity or privacy rights.
 353
 354 =back
 355
 356 =back
 357
 358 =over 9
 359
 360 =item B<Notice>
 361
 362 For any reuse or distribution, you must make clear to others the
 363 license terms of this work.
 364
 365 =back
 366
 367 A copy of the full license is included in the file as
 368 LICENCES/CC-BY-SA-4.0.txt
 369
 370
 371 =head1 DEPENDENCIES
 372
 373 GNU B<sem> uses Perl, and the Perl modules Getopt::Long,
 374 Symbol, Fcntl.
 375
 376
 377 =head1 SEE ALSO
 378
 379 B<parallel>(1)
 380
 381 =cut