src/sem.pod

   1 #!/usr/bin/perl -w
   2
   3 # SPDX-FileCopyrightText: 2021 Ole Tange, http://ole.tange.dk and Free Software and Foundation, Inc.
   4 # SPDX-License-Identifier: GFDL-1.3-or-later
   5 # SPDX-License-Identifier: CC-BY-SA-4.0
   6
   7 =head1 NAME
   8
   9 sem - semaphore for executing shell command lines in parallel
  10
  11 =head1 SYNOPSIS
  12
  13 B<sem> [--fg] [--id <id>] [--semaphoretimeout <secs>] [-j <num>] [--wait] command
  14
  15 =head1 DESCRIPTION
  16
  17 GNU B<sem> is an alias for GNU B<parallel --semaphore>.
  18
  19 GNU B<sem> acts as a counting semaphore. When GNU B<sem> is called
  20 with command it starts the command in the background. When I<num>
  21 number of commands are running in the background, GNU B<sem> waits for
  22 one of these to complete before starting the command.
  23
  24 GNU B<sem> does not read any arguments to build the command (no -a,
  25 :::, and ::::). It simply waits for a semaphore to become available
  26 and then runs the command given.
  27
  28 Before looking at the options you may want to check out the examples
  29 after the list of options. That will give you an idea of what GNU
  30 B<sem> is capable of.
  31
  32 =head1 OPTIONS
  33
  34 =over 9
  35
  36 =item I<command>
  37
  38 Command to execute. The command may be followed by arguments for the
  39 command.
  40
  41
  42 =item B<--bg>
  43
  44 Run command in background thus GNU B<sem> will not wait for
  45 completion of the command before exiting. This is the default.
  46
  47 In toilet analogy: GNU B<sem> waits for a toilet to be available,
  48 gives the toilet to a person, and exits immediately.
  49
  50 See also: B<--fg>
  51
  52
  53 =item B<--jobs> I<N>
  54
  55 =item B<-j> I<N>
  56
  57 =item B<--max-procs> I<N>
  58
  59 =item B<-P> I<N>
  60
  61 Run up to N commands in parallel. Default is 1 thus acting like a
  62 mutex.
  63
  64 In toilet analogy: B<-j> is the number of toilets.
  65
  66
  67 =item B<--jobs> I<+N>
  68
  69 =item B<-j> I<+N>
  70
  71 =item B<--max-procs> I<+N>
  72
  73 =item B<-P> I<+N>
  74
  75 Add N to the number of CPU cores.  Run up to this many jobs in
  76 parallel. For compute intensive jobs B<-j> +0 is useful as it will run
  77 number-of-cpu-cores jobs simultaneously.
  78
  79
  80 =item B<--jobs> I<-N>
  81
  82 =item B<-j> I<-N>
  83
  84 =item B<--max-procs> I<-N>
  85
  86 =item B<-P> I<-N>
  87
  88 Subtract N from the number of CPU cores.  Run up to this many jobs in
  89 parallel.  If the evaluated number is less than 1 then 1 will be used.
  90 See also B<--use-cpus-instead-of-cores>.
  91
  92
  93 =item B<--jobs> I<N>%
  94
  95 =item B<-j> I<N>%
  96
  97 =item B<--max-procs> I<N>%
  98
  99 =item B<-P> I<N>%
 100
 101 Multiply N% with the number of CPU cores.  Run up to this many jobs in
 102 parallel.  If the evaluated number is less than 1 then 1 will be used.
 103 See also B<--use-cpus-instead-of-cores>.
 104
 105
 106 =item B<--jobs> I<procfile>
 107
 108 =item B<-j> I<procfile>
 109
 110 =item B<--max-procs> I<procfile>
 111
 112 =item B<-P> I<procfile>
 113
 114 Read parameter from file. Use the content of I<procfile> as parameter
 115 for I<-j>. E.g. I<procfile> could contain the string 100% or +2 or
 116 10.
 117
 118
 119 =item B<--semaphorename> I<name>
 120
 121 =item B<--id> I<name>
 122
 123 Use B<name> as the name of the semaphore. Default is the name of the
 124 controlling tty (output from B<tty>).
 125
 126 The default normally works as expected when used interactively, but
 127 when used in a script I<name> should be set. I<$$> or I<my_task_name>
 128 are often a good value.
 129
 130 The semaphore is stored in ~/.parallel/semaphores/
 131
 132 In toilet analogy the name corresponds to different types of toilets:
 133 e.g. male, female, customer, staff.
 134
 135
 136 =item B<--fg>
 137
 138 Do not put command in background.
 139
 140 In toilet analogy: GNU B<sem> waits for a toilet to be available,
 141 takes a person to the toilet, waits for the person to finish, and
 142 exits.
 143
 144
 145 =item B<--semaphoretimeout> I<secs>
 146
 147 =item B<--st> I<secs>
 148
 149 If I<secs> > 0: If the semaphore is not released within I<secs>
 150 seconds, take it anyway.
 151
 152 If I<secs> < 0: If the semaphore is not released within I<secs>
 153 seconds, exit.
 154
 155 In toilet analogy: I<secs> > 0: If no toilet becomes available within
 156 I<secs> seconds, pee on the floor. I<secs> < 0: If no toilet becomes
 157 available within I<secs> seconds, exit without doing anything.
 158
 159
 160 =item B<--wait>
 161
 162 Wait for all commands to complete.
 163
 164 In toilet analogy: Wait until all toilets are empty, then exit.
 165
 166
 167 =back
 168
 169 =head1 UNDERSTANDING A SEMAPHORE
 170
 171 Try the following example:
 172
 173   sem -j 2 'sleep 1;echo 1 finished';   echo sem 1 exited
 174   sem -j 2 'sleep 2;echo 2 finished';   echo sem 2 exited
 175   sem -j 2 'sleep 3;echo 3 finished';   echo sem 3 exited
 176   sem -j 2 'sleep 4;echo 4 finished';   echo sem 4 exited
 177   sem --wait; echo sem --wait done
 178
 179 In toilet analogy this uses 2 toilets (B<-j 2>). GNU B<sem> takes '1'
 180 to a toilet, and exits immediately. While '1' is sleeping, another GNU
 181 B<sem> takes '2' to a toilet, and exits immediately.
 182
 183 While '1' and '2' are sleeping, another GNU B<sem> waits for a free
 184 toilet. When '1' finishes, a toilet becomes available, and this GNU
 185 B<sem> stops waiting, and takes '3' to a toilet, and exits
 186 immediately.
 187
 188 While '2' and '3' are sleeping, another GNU B<sem> waits for a free
 189 toilet.  When '2' finishes, a toilet becomes available, and this GNU
 190 B<sem> stops waiting, and takes '4' to a toilet, and exits
 191 immediately.
 192
 193 Finally another GNU B<sem> waits for all toilets to become free.
 194
 195
 196 =head1 EXAMPLE: Gzipping *.log
 197
 198 Run one gzip process per CPU core. Block until a CPU core becomes
 199 available.
 200
 201   for i in *.log ; do
 202     echo $i
 203     sem -j+0 gzip $i ";" echo done
 204   done
 205   sem --wait
 206
 207 =head1 EXAMPLE: Protecting pod2html from itself
 208
 209 pod2html creates two files: pod2htmd.tmp and pod2htmi.tmp which it
 210 does not clean up. It uses these two files for a short time. But if
 211 you run multiple pod2html in parallel (e.g. in a Makefile with make
 212 -j) there is a risk that two different instances of pod2html will
 213 write to the files at the same time:
 214
 215   # This may fail due to shared pod2htmd.tmp/pod2htmi.tmp files
 216   foo.html:
 217           pod2html foo.pod --outfile foo.html
 218
 219   bar.html:
 220           pod2html bar.pod --outfile bar.html
 221
 222   $ make -j foo.html bar.html
 223
 224 You need to protect pod2html from running twice at the same time.
 225 B<sem> running as a mutex will make sure only one runs:
 226
 227   foo.html:
 228           sem --id pod2html pod2html foo.pod --outfile foo.html
 229
 230   bar.html:
 231           sem --id pod2html pod2html bar.pod --outfile bar.html
 232
 233   clean: foo.html bar.html
 234           sem --id pod2html --wait
 235           rm -f pod2htmd.tmp pod2htmi.tmp
 236
 237   $ make -j foo.html bar.html clean
 238
 239 =head1 BUGS
 240
 241 None known.
 242
 243
 244 =head1 REPORTING BUGS
 245
 246 Report bugs to <bug-parallel@gnu.org>.
 247
 248
 249 =head1 AUTHOR
 250
 251 Copyright (C) 2010-2021 Ole Tange, http://ole.tange.dk and Free
 252 Software Foundation, Inc.
 253
 254
 255 =head1 LICENSE
 256
 257 This program is free software; you can redistribute it and/or modify
 258 it under the terms of the GNU General Public License as published by
 259 the Free Software Foundation; either version 3 of the License, or
 260 at your option any later version.
 261
 262 This program is distributed in the hope that it will be useful,
 263 but WITHOUT ANY WARRANTY; without even the implied warranty of
 264 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 265 GNU General Public License for more details.
 266
 267 You should have received a copy of the GNU General Public License
 268 along with this program.  If not, see <http://www.gnu.org/licenses/>.
 269
 270 =head2 Documentation license I
 271
 272 Permission is granted to copy, distribute and/or modify this
 273 documentation under the terms of the GNU Free Documentation License,
 274 Version 1.3 or any later version published by the Free Software
 275 Foundation; with no Invariant Sections, with no Front-Cover Texts, and
 276 with no Back-Cover Texts.  A copy of the license is included in the
 277 file LICENSES/GFDL-1.3-or-later.txt.
 278
 279 =head2 Documentation license II
 280
 281 You are free:
 282
 283 =over 9
 284
 285 =item B<to Share>
 286
 287 to copy, distribute and transmit the work
 288
 289 =item B<to Remix>
 290
 291 to adapt the work
 292
 293 =back
 294
 295 Under the following conditions:
 296
 297 =over 9
 298
 299 =item B<Attribution>
 300
 301 You must attribute the work in the manner specified by the author or
 302 licensor (but not in any way that suggests that they endorse you or
 303 your use of the work).
 304
 305 =item B<Share Alike>
 306
 307 If you alter, transform, or build upon this work, you may distribute
 308 the resulting work only under the same, similar or a compatible
 309 license.
 310
 311 =back
 312
 313 With the understanding that:
 314
 315 =over 9
 316
 317 =item B<Waiver>
 318
 319 Any of the above conditions can be waived if you get permission from
 320 the copyright holder.
 321
 322 =item B<Public Domain>
 323
 324 Where the work or any of its elements is in the public domain under
 325 applicable law, that status is in no way affected by the license.
 326
 327 =item B<Other Rights>
 328
 329 In no way are any of the following rights affected by the license:
 330
 331 =over 2
 332
 333 =item *
 334
 335 Your fair dealing or fair use rights, or other applicable
 336 copyright exceptions and limitations;
 337
 338 =item *
 339
 340 The author's moral rights;
 341
 342 =item *
 343
 344 Rights other persons may have either in the work itself or in
 345 how the work is used, such as publicity or privacy rights.
 346
 347 =back
 348
 349 =back
 350
 351 =over 9
 352
 353 =item B<Notice>
 354
 355 For any reuse or distribution, you must make clear to others the
 356 license terms of this work.
 357
 358 =back
 359
 360 A copy of the full license is included in the file as
 361 LICENCES/CC-BY-SA-4.0.txt
 362
 363
 364 =head1 DEPENDENCIES
 365
 366 GNU B<sem> uses Perl, and the Perl modules Getopt::Long,
 367 Symbol, Fcntl.
 368
 369
 370 =head1 SEE ALSO
 371
 372 B<parallel>(1)
 373
 374 =cut