3 Getting good performance from mdrun
4 ===================================
5 The |Gromacs| build system and the :ref:`gmx mdrun` tool has a lot of built-in
6 and configurable intelligence to detect your hardware and make pretty
7 effective use of that hardware. For a lot of casual and serious use of
8 :ref:`gmx mdrun`, the automatic machinery works well enough. But to get the
9 most from your hardware to maximize your scientific quality, read on!
11 Hardware background information
12 -------------------------------
13 Modern computer hardware is complex and heterogeneous, so we need to
14 discuss a little bit of background information and set up some
15 definitions. Experienced HPC users can skip this section.
20 A hardware compute unit that actually executes
21 instructions. There is normally more than one core in a
22 processor, often many more.
25 A special kind of memory local to core(s) that is much faster
26 to access than main memory, kind of like the top of a human's
27 desk, compared to their filing cabinet. There are often
28 several layers of caches associated with a core.
31 A group of cores that share some kind of locality, such as a
32 shared cache. This makes it more efficient to spread
33 computational work over cores within a socket than over cores
34 in different sockets. Modern processors often have more than
38 A group of sockets that share coarser-level locality, such as
39 shared access to the same memory without requiring any network
40 hardware. A normal laptop or desktop computer is a node. A
41 node is often the smallest amount of a large compute cluster
42 that a user can request to use.
45 A stream of instructions for a core to execute. There are many
46 different programming abstractions that create and manage
47 spreading computation over multiple threads, such as OpenMP,
48 pthreads, winthreads, CUDA, OpenCL, and OpenACC. Some kinds of
49 hardware can map more than one software thread to a core; on
50 Intel x86 processors this is called "hyper-threading", while
51 the more general concept is often called SMT for
52 "simultaneous multi-threading". IBM Power8 can for instance use
53 up to 8 hardware threads per core.
54 This feature can usually be enabled or disabled either in
55 the hardware bios or through a setting in the Linux operating
56 system. |Gromacs| can typically make use of this, for a moderate
57 free performance boost. In most cases it will be
58 enabled by default e.g. on new x86 processors, but in some cases
59 the system administrators might have disabled it. If that is the
60 case, ask if they can re-enable it for you. If you are not sure
61 if it is enabled, check the output of the CPU information in
62 the log file and compare with CPU specifications you find online.
64 thread affinity (pinning)
65 By default, most operating systems allow software threads to migrate
66 between cores (or hardware threads) to help automatically balance
67 workload. However, the performance of :ref:`gmx mdrun` can deteriorate
68 if this is permitted and will degrade dramatically especially when
69 relying on multi-threading within a rank. To avoid this,
70 :ref:`gmx mdrun` will by default
71 set the affinity of its threads to individual cores/hardware threads,
72 unless the user or software environment has already done so
73 (or not the entire node is used for the run, i.e. there is potential
75 Setting thread affinity is sometimes called thread "pinning".
78 The dominant multi-node parallelization-scheme, which provides
79 a standardized language in which programs can be written that
80 work across more than one node.
83 In MPI, a rank is the smallest grouping of hardware used in
84 the multi-node parallelization scheme. That grouping can be
85 controlled by the user, and might correspond to a core, a
86 socket, a node, or a group of nodes. The best choice varies
87 with the hardware, software and compute task. Sometimes an MPI
88 rank is called an MPI process.
91 A graphics processing unit, which is often faster and more
92 efficient than conventional processors for particular kinds of
93 compute workloads. A GPU is always associated with a
94 particular node, and often a particular socket within that
98 A standardized technique supported by many compilers to share
99 a compute workload over multiple cores. Often combined with
100 MPI to achieve hybrid MPI/OpenMP parallelism.
103 A proprietary parallel computing framework and API developed by NVIDIA
104 that allows targeting their accelerator hardware.
105 |Gromacs| uses CUDA for GPU acceleration support with NVIDIA hardware.
108 An open standard-based parallel computing framework that consists
109 of a C99-based compiler and a programming API for targeting heterogeneous
110 and accelerator hardware. |Gromacs| uses OpenCL for GPU acceleration
111 on AMD devices (both GPUs and APUs); NVIDIA hardware is also supported.
114 Modern CPU cores have instructions that can execute large
115 numbers of floating-point instructions in a single cycle.
118 |Gromacs| background information
119 --------------------------------
120 The algorithms in :ref:`gmx mdrun` and their implementations are most relevant
121 when choosing how to make good use of the hardware. For details,
122 see the Reference Manual. The most important of these are
127 The domain decomposition (DD) algorithm decomposes the
128 (short-ranged) component of the non-bonded interactions into
129 domains that share spatial locality, which permits the use of
130 efficient algorithms. Each domain handles all of the
131 particle-particle (PP) interactions for its members, and is
132 mapped to a single MPI rank. Within a PP rank, OpenMP threads
133 can share the workload, and some work can be off-loaded to a
134 GPU. The PP rank also handles any bonded interactions for the
135 members of its domain. A GPU may perform work for more than
136 one PP rank, but it is normally most efficient to use a single
137 PP rank per GPU and for that rank to have thousands of
138 particles. When the work of a PP rank is done on the CPU, mdrun
139 will make extensive use of the SIMD capabilities of the
140 core. There are various `command-line options
141 <controlling-the-domain-decomposition-algorithm` to control
142 the behaviour of the DD algorithm.
145 The particle-mesh Ewald (PME) algorithm treats the long-ranged
146 components of the non-bonded interactions (Coulomb and/or
147 Lennard-Jones). Either all, or just a subset of ranks may
148 participate in the work for computing long-ranged component
149 (often inaccurately called simple the "PME"
150 component). Because the algorithm uses a 3D FFT that requires
151 global communication, its performance gets worse as more ranks
152 participate, which can mean it is fastest to use just a subset
153 of ranks (e.g. one-quarter to one-half of the ranks). If
154 there are separate PME ranks, then the remaining ranks handle
155 the PP work. Otherwise, all ranks do both PP and PME work.
157 Running mdrun within a single node
158 ----------------------------------
160 :ref:`gmx mdrun` can be configured and compiled in several different ways that
161 are efficient to use within a single :term:`node`. The default configuration
162 using a suitable compiler will deploy a multi-level hybrid parallelism
163 that uses CUDA, OpenMP and the threading platform native to the
164 hardware. For programming convenience, in |Gromacs|, those native
165 threads are used to implement on a single node the same MPI scheme as
166 would be used between nodes, but much more efficient; this is called
167 thread-MPI. From a user's perspective, real MPI and thread-MPI look
168 almost the same, and |Gromacs| refers to MPI ranks to mean either kind,
169 except where noted. A real external MPI can be used for :ref:`gmx mdrun` within
170 a single node, but runs more slowly than the thread-MPI version.
172 By default, :ref:`gmx mdrun` will inspect the hardware available at run time
173 and do its best to make fairly efficient use of the whole node. The
174 log file, stdout and stderr are used to print diagnostics that
175 inform the user about the choices made and possible consequences.
177 A number of command-line parameters are available to modify the default
181 The total number of threads to use. The default, 0, will start as
182 many threads as available cores. Whether the threads are
183 thread-MPI ranks, and/or OpenMP threads within such ranks depends on
187 The total number of thread-MPI ranks to use. The default, 0,
188 will start one rank per GPU (if present), and otherwise one rank
192 The total number of OpenMP threads per rank to start. The
193 default, 0, will start one thread on each available core.
194 Alternatively, mdrun will honor the appropriate system
195 environment variable (e.g. ``OMP_NUM_THREADS``) if set.
198 The total number of ranks to dedicate to the long-ranged
199 component of PME, if used. The default, -1, will dedicate ranks
200 only if the total number of threads is at least 12, and will use
201 around a quarter of the ranks for the long-ranged component.
204 When using PME with separate PME ranks,
205 the total number of OpenMP threads per separate PME ranks.
206 The default, 0, copies the value from ``-ntomp``.
209 A string that specifies the ID numbers of the GPUs to be
210 used by corresponding PP ranks on this node. For example,
211 "0011" specifies that the lowest two PP ranks use GPU 0,
212 and the other two use GPU 1.
215 Can be set to "auto," "on" or "off" to control whether
216 mdrun will attempt to set the affinity of threads to cores.
217 Defaults to "auto," which means that if mdrun detects that all the
218 cores on the node are being used for mdrun, then it should behave
219 like "on," and attempt to set the affinities (unless they are
220 already set by something else).
223 If ``-pin on``, specifies the logical core number to
224 which mdrun should pin the first thread. When running more than
225 one instance of mdrun on a node, use this option to to avoid
226 pinning threads from different mdrun instances to the same core.
229 If ``-pin on``, specifies the stride in logical core
230 numbers for the cores to which mdrun should pin its threads. When
231 running more than one instance of mdrun on a node, use this option
232 to to avoid pinning threads from different mdrun instances to the
233 same core. Use the default, 0, to minimize the number of threads
234 per physical core - this lets mdrun manage the hardware-, OS- and
235 configuration-specific details of how to map logical cores to
239 Can be set to "interleave," "pp_pme" or "cartesian."
240 Defaults to "interleave," which means that any separate PME ranks
241 will be mapped to MPI ranks in an order like PP, PP, PME, PP, PP,
242 PME, ... etc. This generally makes the best use of the available
243 hardware. "pp_pme" maps all PP ranks first, then all PME
244 ranks. "cartesian" is a special-purpose mapping generally useful
245 only on special torus networks with accelerated global
246 communication for Cartesian communicators. Has no effect if there
247 are no separate PME ranks.
250 Used to set where to execute the non-bonded interactions.
251 Can be set to "auto", "cpu", "gpu."
252 Defaults to "auto," which uses a compatible GPU if available.
253 Setting "cpu" requires that no GPU is used. Setting "gpu" requires
254 that a compatible GPU be available and will be used.
256 Examples for mdrun on one node
257 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
263 Starts mdrun using all the available resources. mdrun
264 will automatically choose a fairly efficient division
265 into thread-MPI ranks, OpenMP threads and assign work
266 to compatible GPUs. Details will vary with hardware
267 and the kind of simulation being run.
273 Starts mdrun using 8 threads, which might be thread-MPI
274 or OpenMP threads depending on hardware and the kind
275 of simulation being run.
279 gmx mdrun -ntmpi 2 -ntomp 4
281 Starts mdrun using eight total threads, with four thread-MPI
282 ranks and two OpenMP threads per core. You should only use
283 these options when seeking optimal performance, and
284 must take care that the ranks you create can have
285 all of their OpenMP threads run on the same socket.
286 The number of ranks must be a multiple of the number of
287 sockets, and the number of cores per node must be
288 a multiple of the number of threads per rank.
294 Starts mdrun using GPUs with IDs 1 and 2 (e.g. because
295 GPU 0 is dedicated to running a display). This requires
296 two thread-MPI ranks, and will split the available
297 CPU cores between them using OpenMP threads.
301 gmx mdrun -ntmpi 4 -gpu_id "1122"
303 Starts mdrun using four thread-MPI ranks, and maps them
304 to GPUs with IDs 1 and 2. The CPU cores available will
305 be split evenly between the ranks using OpenMP threads.
309 gmx mdrun -nt 6 -pin on -pinoffset 0
310 gmx mdrun -nt 6 -pin on -pinoffset 3
312 Starts two mdrun processes, each with six total threads.
313 Threads will have their affinities set to particular
314 logical cores, beginning from the logical core
315 with rank 0 or 3, respectively. The above would work
316 well on an Intel CPU with six physical cores and
317 hyper-threading enabled. Use this kind of setup only
318 if restricting mdrun to a subset of cores to share a
319 node with other processes.
323 mpirun -np 2 gmx_mpi mdrun
325 When using an :ref:`gmx mdrun` compiled with external MPI,
326 this will start two ranks and as many OpenMP threads
327 as the hardware and MPI setup will permit. If the
328 MPI setup is restricted to one node, then the resulting
329 :ref:`gmx mdrun` will be local to that node.
331 Running mdrun on more than one node
332 -----------------------------------
333 This requires configuring |Gromacs| to build with an external MPI
334 library. By default, this mdrun executable is run with
335 :ref:`mdrun_mpi`. All of the considerations for running single-node
336 mdrun still apply, except that ``-ntmpi`` and ``-nt`` cause a fatal
337 error, and instead the number of ranks is controlled by the
339 Settings such as ``-npme`` are much more important when
340 using multiple nodes. Configuring the MPI environment to
341 produce one rank per core is generally good until one
342 approaches the strong-scaling limit. At that point, using
343 OpenMP to spread the work of an MPI rank over more than one
344 core is needed to continue to improve absolute performance.
345 The location of the scaling limit depends on the processor,
346 presence of GPUs, network, and simulation algorithm, but
347 it is worth measuring at around ~200 particles/core if you
348 need maximum throughput.
350 There are further command-line parameters that are relevant in these
354 Defaults to "on." If "on," a Verlet-scheme simulation will
355 optimize various aspects of the PME and DD algorithms, shifting
356 load between ranks and/or GPUs to maximize throughput. Some
357 mdrun features are not compatible with this, and these ignore
361 Can be set to "auto," "no," or "yes."
362 Defaults to "auto." Doing Dynamic Load Balancing between MPI ranks
363 is needed to maximize performance. This is particularly important
364 for molecular systems with heterogeneous particle or interaction
365 density. When a certain threshold for performance loss is
366 exceeded, DLB activates and shifts particles between ranks to improve
370 During the simulation :ref:`gmx mdrun` must communicate between all ranks to
371 compute quantities such as kinetic energy. By default, this
372 happens whenever plausible, and is influenced by a lot of :ref:`[.mdp]
373 options. <mdp-general>` The period between communication phases
374 must be a multiple of :mdp:`nstlist`, and defaults to
375 the minimum of :mdp:`nstcalcenergy` and :mdp:`nstlist`.
376 ``mdrun -gcom`` sets the number of steps that must elapse between
377 such communication phases, which can improve performance when
378 running on a lot of ranks. Note that this means that _e.g._
379 temperature coupling algorithms will
380 effectively remain at constant energy until the next
381 communication phase. :ref:`gmx mdrun` will always honor the
382 setting of ``mdrun -gcom``, by changing :mdp:`nstcalcenergy`,
383 :mdp:`nstenergy`, :mdp:`nstlog`, :mdp:`nsttcouple` and/or
384 :mdp:`nstpcouple` if necessary.
386 Note that ``-tunepme`` has more effect when there is more than one
387 :term:`node`, because the cost of communication for the PP and PME
388 ranks differs. It still shifts load between PP and PME ranks, but does
389 not change the number of separate PME ranks in use.
391 Note also that ``-dlb`` and ``-tunepme`` can interfere with each other, so
392 if you experience performance variation that could result from this,
393 you may wish to tune PME separately, and run the result with ``mdrun
394 -notunepme -dlb yes``.
396 The :ref:`gmx tune_pme` utility is available to search a wider
397 range of parameter space, including making safe
398 modifications to the :ref:`tpr` file, and varying ``-npme``.
399 It is only aware of the number of ranks created by
400 the MPI environment, and does not explicitly manage
401 any aspect of OpenMP during the optimization.
403 Examples for mdrun on more than one node
404 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
405 The examples and explanations for for single-node mdrun are
406 still relevant, but ``-nt`` is no longer the way
407 to choose the number of MPI ranks.
411 mpirun -np 16 gmx_mpi mdrun
413 Starts :ref:`mdrun_mpi` with 16 ranks, which are mapped to
414 the hardware by the MPI library, e.g. as specified
415 in an MPI hostfile. The available cores will be
416 automatically split among ranks using OpenMP threads,
417 depending on the hardware and any environment settings
418 such as ``OMP_NUM_THREADS``.
422 mpirun -np 16 gmx_mpi mdrun -npme 5
424 Starts :ref:`mdrun_mpi` with 16 ranks, as above, and
425 require that 5 of them are dedicated to the PME
430 mpirun -np 11 gmx_mpi mdrun -ntomp 2 -npme 6 -ntomp_pme 1
432 Starts :ref:`mdrun_mpi` with 11 ranks, as above, and
433 require that six of them are dedicated to the PME
434 component with one OpenMP thread each. The remaining
435 five do the PP component, with two OpenMP threads
440 mpirun -np 4 gmx mdrun -ntomp 6 -gpu_id 00
442 Starts :ref:`mdrun_mpi` on a machine with two nodes, using
443 four total ranks, each rank with six OpenMP threads,
444 and both ranks on a node sharing GPU with ID 0.
448 mpirun -np 8 gmx mdrun -ntomp 3 -gpu_id 0000
450 Using a same/similar hardware as above,
451 starts :ref:`mdrun_mpi` on a machine with two nodes, using
452 eight total ranks, each rank with three OpenMP threads,
453 and all four ranks on a node sharing GPU with ID 0.
454 This may or may not be faster than the previous setup
455 on the same hardware.
459 mpirun -np 20 gmx_mpi mdrun -ntomp 4 -gpu_id 0
461 Starts :ref:`mdrun_mpi` with 20 ranks, and assigns the CPU cores evenly
462 across ranks each to one OpenMP thread. This setup is likely to be
463 suitable when there are ten nodes, each with one GPU, and each node
468 mpirun -np 20 gmx_mpi mdrun -gpu_id 00
470 Starts :ref:`mdrun_mpi` with 20 ranks, and assigns the CPU cores evenly
471 across ranks each to one OpenMP thread. This setup is likely to be
472 suitable when there are ten nodes, each with one GPU, and each node
477 mpirun -np 20 gmx_mpi mdrun -gpu_id 01
479 Starts :ref:`mdrun_mpi` with 20 ranks. This setup is likely
480 to be suitable when there are ten nodes, each with two
485 mpirun -np 40 gmx_mpi mdrun -gpu_id 0011
487 Starts :ref:`mdrun_mpi` with 40 ranks. This setup is likely
488 to be suitable when there are ten nodes, each with two
489 GPUs, and OpenMP performs poorly on the hardware.
491 Controlling the domain decomposition algorithm
492 ----------------------------------------------
493 This section lists all the options that affect how the domain
494 decomposition algorithm decomposes the workload to the available
498 Can be used to set the required maximum distance for inter
499 charge-group bonded interactions. Communication for two-body
500 bonded interactions below the non-bonded cut-off distance always
501 comes for free with the non-bonded communication. Particles beyond
502 the non-bonded cut-off are only communicated when they have
503 missing bonded interactions; this means that the extra cost is
504 minor and nearly independent of the value of ``-rdd``. With dynamic
505 load balancing, option ``-rdd`` also sets the lower limit for the
506 domain decomposition cell sizes. By default ``-rdd`` is determined
507 by :ref:`gmx mdrun` based on the initial coordinates. The chosen value will
508 be a balance between interaction range and communication cost.
511 On by default. When inter charge-group bonded interactions are
512 beyond the bonded cut-off distance, :ref:`gmx mdrun` terminates with an
513 error message. For pair interactions and tabulated bonds that do
514 not generate exclusions, this check can be turned off with the
515 option ``-noddcheck``.
518 When constraints are present, option ``-rcon`` influences
519 the cell size limit as well.
520 Particles connected by NC constraints, where NC is the LINCS order
521 plus 1, should not be beyond the smallest cell size. A error
522 message is generated when this happens, and the user should change
523 the decomposition or decrease the LINCS order and increase the
524 number of LINCS iterations. By default :ref:`gmx mdrun` estimates the
525 minimum cell size required for P-LINCS in a conservative
526 fashion. For high parallelization, it can be useful to set the
527 distance required for P-LINCS with ``-rcon``.
530 Sets the minimum allowed x, y and/or z scaling of the cells with
531 dynamic load balancing. :ref:`gmx mdrun` will ensure that the cells can
532 scale down by at least this factor. This option is used for the
533 automated spatial decomposition (when not using ``-dd``) as well as
534 for determining the number of grid pulses, which in turn sets the
535 minimum allowed cell size. Under certain circumstances the value
536 of ``-dds`` might need to be adjusted to account for high or low
537 spatial inhomogeneity of the system.
539 Finding out how to run mdrun better
540 -----------------------------------
542 The Wallcycle module is used for runtime performance measurement of :ref:`gmx mdrun`.
543 At the end of the log file of each run, the "Real cycle and time accounting" section
544 provides a table with runtime statistics for different parts of the :ref:`gmx mdrun` code
545 in rows of the table.
546 The table contains colums indicating the number of ranks and threads that
547 executed the respective part of the run, wall-time and cycle
548 count aggregates (across all threads and ranks) averaged over the entire run.
549 The last column also shows what precentage of the total runtime each row represents.
550 Note that the :ref:`gmx mdrun` timer resetting functionalities (`-resethway` and `-resetstep`)
551 reset the performance counters and therefore are useful to avoid startup overhead and
552 performance instability (e.g. due to load balancing) at the beginning of the run.
554 The performance counters are:
556 * Particle-particle during Particle mesh Ewald
557 * Domain decomposition
558 * Domain decomposition communication load
559 * Domain decomposition communication bounds
560 * Virtual site constraints
561 * Send X to Particle mesh Ewald
563 * Launch GPU operations
564 * Communication of coordinates
567 * Waiting + Communication of force
568 * Particle mesh Ewald
572 * PME 3D-FFT Communication
573 * PME solve Lennard-Jones
575 * PME wait for particle-particle
576 * Wait + Receive PME force
579 * Non-bonded position/force buffer operations
580 * Virtual site spread
585 * Communication of energies
587 * Add rotational forces
591 As performance data is collected for every run, they are essential to assessing
592 and tuning the performance of :ref:`gmx mdrun` performance. Therefore, they benefit
593 both code developers as well as users of the program.
594 The counters are an average of the time/cycles different parts of the simulation take,
595 hence can not directly reveal fluctuations during a single run (although comparisons across
596 multiple runs are still very useful).
598 Counters will appear in MD log file only if the related parts of the code were
599 executed during the :ref:`gmx mdrun` run. There is also a special counter called "Rest" which
600 indicated for the amount of time not accounted for by any of the counters above. Theerfore,
601 a significant amount "Rest" time (more than a few percent) will often be an indication of
602 parallelization inefficiency (e.g. serial code) and it is recommended to be reported to the
605 An additional set of subcounters can offer more fine-grained inspection of performance. They are:
607 * Domain decomposition redistribution
608 * DD neighbor search grid + sort
609 * DD setup communication
611 * DD make constraints
613 * Neighbor search grid local
616 * NS search non-local
620 * Listed buffer operations
622 * Ewald force correction
623 * Non-bonded position buffer operations
624 * Non-bonded force buffer operations
626 Subcounters are geared toward developers and have to be enabled during compilation. See
627 :doc:`/dev-manual/build-system` for more information.
629 TODO In future patch:
630 - red flags in log files, how to interpret wallcycle output
631 - hints to devs how to extend wallcycles
633 TODO In future patch: import wiki page stuff on performance checklist; maybe here,
636 .. _gmx-mdrun-on-gpu:
638 Running mdrun with GPUs
639 -----------------------
641 NVIDIA GPUs from the professional line (Tesla or Quadro) starting with
642 the Kepler generation (compute capability 3.5 and later) support changing the
643 processor and memory clock frequency with the help of the applications clocks feature.
644 With many workloads, using higher clock rates than the default provides significant
645 performance improvements.
646 For more information see the `NVIDIA blog article`_ on this topic.
647 For |Gromacs| the highest application clock rates are optimal on all hardware
648 available to date (up to and including Maxwell, compute capability 5.2).
650 Application clocks can be set using the NVIDIA system managemet tool
651 ``nvidia-smi``. If the system permissions allow, :ref:`gmx mdrun` has
652 built-in support to set application clocks if built with :ref:`NVML support<CUDA GPU acceleration>`.
653 Note that application clocks are a global setting, hence affect the
654 performance of all applications that use the respective GPU(s).
655 For this reason, :ref:`gmx mdrun` sets application clocks at initialization
656 to the values optimal for |Gromacs| and it restores them before exiting
657 to the values found at startup, unless it detects that they were altered
660 .. _NVIDIA blog article: https://devblogs.nvidia.com/parallelforall/increase-performance-gpu-boost-k80-autoboost/
662 Reducing overheads in GPU accelerated runs
663 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
665 In order for CPU cores and GPU(s) to execute concurrently, tasks are
666 launched and executed asynchronously on the GPU(s) while the CPU cores
667 execute non-offloaded force computation (like long-range PME electrostatics).
668 Asynchronous task launches are handled by GPU device driver and
669 require CPU involvement. Therefore, the work of scheduling
670 GPU tasks will incur an overhead that can in some cases significantly
671 delay or interfere with the CPU execution.
673 Delays in CPU execution are caused by the latency of launching GPU tasks,
674 an overhead that can become significant as simulation ns/day increases
675 (i.e. with shorter wall-time per step).
676 The overhead is measured by :ref:`gmx mdrun` and reported in the performance
677 summary section of the log file ("Launch GPU ops" row).
678 A few percent of runtime spent in this category is normal,
679 but in fast-iterating and multi-GPU parallel runs 10% or larger overheads can be observed.
680 In general, there a user can do little to avoid such overheads, but there
681 are a few cases where tweaks can give performance benefits.
682 In single-rank runs timing of GPU tasks is by default enabled and,
683 while in most cases its impact is small, in fast runs performance can be affected.
684 The performance impact will be most significant on NVIDIA GPUs with CUDA,
685 less on AMD with OpenCL.
686 In these cases, when more than a few percent of "Launch GPU ops" time is observed,
687 it is recommended turning off timing by setting the ``GMX_DISABLE_GPU_TIMING``
688 environment variable.
689 In parallel runs with with many ranks sharing a GPU
690 launch overheads can also be reduced by staring fewer thread-MPI
691 or MPI ranks per GPU; e.g. most often one rank per thread or core is not optimal.
693 The second type of overhead, interference of the GPU driver with CPU computation,
694 is caused by the scheduling and coordination of GPU tasks.
695 A separate GPU driver thread can require CPU resources
696 which may clash with the concurrently running non-offloaded tasks,
697 potentially degrading the performance of PME or bonded force computation.
698 This effect is most pronounced when using AMD GPUs with OpenCL with
699 all stable driver releases to date (up to and including fglrx 12.15).
700 To minimize the overhead it is recommended to
701 leave a CPU hardware thread unused when launching :ref:`gmx mdrun`,
702 especially on CPUs with high core count and/or HyperThreading enabled.
703 E.g. on a machine with a 4-core CPU and eight threads (via HyperThreading) and an AMD GPU,
704 try ``gmx mdrun -ntomp 7 -pin on``.
705 This will leave free CPU resources for the GPU task scheduling
706 reducing interference with CPU computation.
707 Note that assigning fewer resources to :ref:`gmx mdrun` CPU computation
708 involves a tradeoff which may outweigh the benefits of reduced GPU driver overhead,
709 in particular without HyperThreading and with few CPU cores.
711 TODO In future patch: any tips not covered above
713 Running the OpenCL version of mdrun
714 -----------------------------------
716 The current version works with GCN-based AMD GPUs, and NVIDIA CUDA
717 GPUs. Make sure that you have the latest drivers installed. For AMD GPUs,
718 Mesa version 17.0 or newer with LLVM 4.0 or newer is supported in addition
719 to the proprietary driver. For NVIDIA GPUs, using the proprietary driver is
720 required as the open source nouveau driver (available in Mesa) does not
721 provide the OpenCL support.
722 The minimum OpenCL version required is |REQUIRED_OPENCL_MIN_VERSION|. See
723 also the :ref:`known limitations <opencl-known-limitations>`.
725 Devices from the AMD GCN architectures (all series) and NVIDIA Fermi
726 and later (compute capability 2.0) are known to work, but before
727 doing production runs always make sure that the |Gromacs| tests
728 pass successfully on the hardware.
730 The OpenCL GPU kernels are compiled at run time. Hence,
731 building the OpenCL program can take a few seconds introducing a slight
732 delay in the :ref:`gmx mdrun` startup. This is not normally a
733 problem for long production MD, but you might prefer to do some kinds
734 of work, e.g. that runs very few steps, on just the CPU (e.g. see ``-nb`` above).
736 The same ``-gpu_id`` option (or ``GMX_GPU_ID`` environment variable)
737 used to select CUDA devices, or to define a mapping of GPUs to PP
738 ranks, is used for OpenCL devices.
740 Some other :ref:`OpenCL management <opencl-management>` environment
741 variables may be of interest to developers.
743 .. _opencl-known-limitations:
745 Known limitations of the OpenCL support
746 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
748 Limitations in the current OpenCL support of interest to |Gromacs| users:
750 - No Intel devices (CPUs, GPUs or Xeon Phi) are supported
751 - Due to blocking behavior of some asynchronous task enqueuing functions
752 in the NVIDIA OpenCL runtime, with the affected driver versions there is
753 almost no performance gain when using NVIDIA GPUs.
754 The issue affects NVIDIA driver versions up to 349 series, but it
755 known to be fixed 352 and later driver releases.
756 - On NVIDIA GPUs the OpenCL kernels achieve much lower performance
757 than the equivalent CUDA kernels due to limitations of the NVIDIA OpenCL
759 - The AMD APPSDK version 3.0 ships with OpenCL compiler/runtime components,
760 libamdocl12cl64.so and libamdocl64.so (only in earlier releases),
761 that conflict with newer fglrx GPU drivers which provide the same libraries.
762 This conflict manifests in kernel launch failures as, due to the library path
763 setup, the OpenCL runtime loads the APPSDK version of the aforementioned
764 libraries instead of the ones provided by the driver installer.
765 The recommended workaround is to remove or rename the APPSDK versions of the
768 Limitations of interest to |Gromacs| developers:
770 - The current implementation is not compatible with OpenCL devices that are
771 not using warp/wavefronts or for which the warp/wavefront size is not a
773 - Some Ewald tabulated kernels are known to produce incorrect results, so
774 (correct) analytical kernels are used instead.
776 Performance checklist
777 ---------------------
779 There are many different aspects that affect the performance of simulations in
780 |Gromacs|. Most simulations require a lot of computational resources, therefore
781 it can be worthwhile to optimize the use of those resources. Several issues
782 mentioned in the list below could lead to a performance difference of a factor
783 of 2. So it can be useful go through the checklist.
785 |Gromacs| configuration
786 ^^^^^^^^^^^^^^^^^^^^^^^
788 * Don't use double precision unless you're absolute sure you need it.
789 * Compile the FFTW library (yourself) with the correct flags on x86 (in most
790 cases, the correct flags are automatically configured).
791 * On x86, use gcc or icc as the compiler (not pgi or the Cray compiler).
792 * On POWER, use gcc instead of IBM's xlc.
793 * Use a new compiler version, especially for gcc (e.g. from the version 5 to 6
794 the performance of the compiled code improved a lot).
795 * MPI library: OpenMPI usually has good performance and causes little trouble.
796 * Make sure your compiler supports OpenMP (some versions of Clang don't).
797 * If you have GPUs that support either CUDA or OpenCL, use them.
799 * Configure with ``-DGMX_GPU=ON`` (add ``-DGMX_USE_OPENCL=ON`` for OpenCL).
800 * For CUDA, use the newest CUDA availabe for your GPU to take advantage of the
801 latest performance enhancements.
802 * Use a recent GPU driver.
803 * If compiling on a cluster head node, make sure that ``GMX_CPU_ACCELERATION``
804 is appropriate for the compute nodes.
809 * For an approximately spherical solute, use a rhombic dodecahedron unit cell.
810 * When using a time-step of 2 fs, use :mdp:`cutoff-scheme` = :mdp-value:`constraints=h-bonds`
811 (and not :mdp-value:`constraints=all-bonds`), since this is faster, especially with GPUs,
812 and most force fields have been parametrized with only bonds involving
813 hydrogens constrained.
814 * You can increase the time-step to 4 or 5 fs when using virtual interaction
815 sites (``gmx pdb2gmx -vsite h``).
816 * For massively parallel runs with PME, you might need to try different numbers
817 of PME ranks (``gmx mdrun -npme ???``) to achieve best performance;
818 :ref:`gmx tune_pme` can help automate this search.
819 * For massively parallel runs (also ``gmx mdrun -multidir``), or with a slow
820 network, global communication can become a bottleneck and you can reduce it
821 with ``gmx mdrun -gcom`` (note that this does affect the frequency of
822 temperature and pressure coupling).
824 Checking and improving performance
825 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
827 * Look at the end of the ``md.log`` file to see the performance and the cycle
828 counters and wall-clock time for different parts of the MD calculation. The
829 PP/PME load ratio is also printed, with a warning when a lot of performance is
830 lost due to imbalance.
831 * Adjust the number of PME ranks and/or the cut-off and PME grid-spacing when
832 there is a large PP/PME imbalance. Note that even with a small reported
833 imbalance, the automated PME-tuning might have reduced the initial imbalance.
834 You could still gain performance by changing the mdp parameters or increasing
835 the number of PME ranks.
836 * If the neighbor searching takes a lot of time, increase nstlist (with the
837 Verlet cut-off scheme, this automatically adjusts the size of the neighbour
838 list to do more non-bonded computation to keep energy drift constant).
840 * If ``Comm. energies`` takes a lot of time (a note will be printed in the log
841 file), increase nstcalcenergy or use ``mdrun -gcom``.
842 * If all communication takes a lot of time, you might be running on too many
843 cores, or you could try running combined MPI/OpenMP parallelization with 2
844 or 4 OpenMP threads per MPI process.