3 Copyright (C) 1989-2017 Regents of the University of Illinois
8 Charm++ is a message-passing parallel language and runtime system.
9 It is implemented as a set of libraries for C++, is efficient,
10 and is portable to a wide variety of parallel machines.
11 Source code is provided, and non-commercial use is free.
14 GETTING THE LATEST SOURCE
15 =========================
17 You can use anonymous Git access to obtain the latest Charm++ source
20 git clone http://charm.cs.illinois.edu/gerrit/charm
28 First-time users are encouraged to run the top-level "build" script and follow its lead:
32 ADVANCED BUILD OPTIONS:
33 -----------------------
35 First, you need to decide which version of charm++ to use. The "build"
36 script takes several command line options to compile Charm++. The command line syntax is:
38 build <target> <version> [options ...]
39 [--basedir=dir] [--libdir=dir] [--incdir=dir]
42 For detailed help messages, pass -h or --help to the build script, i.e.
47 <target> specifies the parts of Charm++ to compile. The most often used
48 <target> is "charm++", which will compile the key Charm++ executables and
49 runtime libraries. Other common targets are "AMPI" and "LIBS".
50 <versions> defines the CPU, OS and Communication layer of the machines. See
51 "How to choose a <version>" below for details.
55 <options> defines more detailed information of the compilations, including
56 compilers, features to support, etc. See "How to choose <option>"
58 [--libdir=dir] specify additional lib paths for building Charm++.
59 [--incdir=dir] specify additional include paths for building Charm++.
60 [--basedir=dir] a shortcut to specify additional include and lib paths for
61 building Charm++, the include path is dir/include and lib path
65 Running build script, a directory of the name of combination of version and
66 options like "<version>-<option1>-<option2>-..." will be created and
67 the build script will compile Charm++ under this directory.
69 For example, on an ordinary Linux PC:
71 ./build charm++ netlrts-linux-x86_64
73 will build charm++ in the directory: netlrts-linux-x86_64/. The communication
74 defaults to UDP packets and the compiler to gcc.
76 For a more complex example, consider a shared-memory build with the
77 Intel C++ compiler, where you want the communication to happen over TCP sockets:
79 ./build charm++ netlrts-linux-x86_64 smp icc tcp
81 will build charm++ in the directory: netlrts-linux-x86_64-smp-tcp-icc/.
83 You can specify multiple options, however you can use at most one compiler
84 option. The sequencing of options given to the build script should follow the rules:
85 1. compiler option will be at the end;
86 2. other options are sorted alphabetically.
88 **** How to choose a <version> ****
90 Here is the table for choosing a correct build version. The default setting of compiler
91 in Charm++ is gcc/g++ on Linux and Clang on MacOS. However, one can use <options> to specify other
92 compilers. See the detailed explanation of the <options> below.
93 (Note: this isn't a complete list. Run ./build for a complete listing)
95 Charm version OS Communication Default Compiler
96 ------------- --------- -------------- --------------------
97 netlrts-linux-x86_64 Linux UDP GNU compiler
98 netlrts-darwin-x86_64 MacOS X UDP Clang C++ compiler
99 mpi-linux-x86_64 Linux MPI GNU compiler
100 mpi-darwin-x86_64 MacOS X MPI Clang C++ compiler
101 pamilrts-bluegeneq CNK PAMI XLC
102 gni-crayxe Linux GNI GNU compiler
103 verbs-linux-x86_64 Linux IB Verbs GNU compiler
104 net-win64 Windows UDP MS Visual C++
105 net-cygwin Cygwin UDP GNU compiler
108 To choose <version>, your choice is determined by two options:
110 1.) The way a parallel program written in Charm++ will communicate:
112 "netlrts-" Charm++ communicates using the regular TCP/IP stack
113 (UDP packets), which works everywhere but is fairly slow. Use this
114 option for networks of workstations, clusters, or single-machine
115 development and testing.
117 "gni-", "bluegeneq-", "pamilrts-", "verbs-" Charm++
118 communicates using direct calls to the machine's communication primitives.
119 Use these versions on machines that support them for best performance.
121 "mpi-" Charm++ communicates using MPI calls.
123 "multicore-" Charm++ communicates using shared memory within a single node.
126 2.) Your operating system:
128 "linux-x86_64" Linux with AMD64 64-bit x86 instructions
129 "win64" MS Windows with MS Visual C++ compiler
130 "cygwin" MS Windows with Cygnus' Cygwin Unix layer
131 "darwin-x86_64" Apple Mac OS X
132 "bluegeneq" IBM's Blue Gene/Q
133 "cray{xe/xc}" Cray's X-Series Supercomputer
134 "linux-ppc" POWER/PowerPC
137 Your Charm++ version is made by concatenating the options, e.g.:
139 "netlrts-linux-x86_64" Charm++ for a network of 64-bit Linux workstations,
142 "gni-crayxc" Charm++ for Cray XC systems using the system's compiler.
145 **** How to choose <options> ****
147 <version> above defines the most important OS, CPU and Communication of
148 your machine, and most of time it uses gcc as the default compiler.
149 To use a different compiler or demand additional special feature support, you
150 need to choose <options> from the following list:
152 * icc - Intel C/C++ compiler.
153 * ifort - Intel Fortran compiler
154 * xlc - IBM XLC compiler.
155 * clang - Clang compiler.
156 * mpicxx - Use MPI-wrappers for MPI builds.
157 * pgcc - Portland Group's C++ compiler.
159 * smp - Enable direct SMP support. An "smp" version communicates using
160 shared memory within a machine; but normal message passing across
161 machines. Because of locking, "smp" may slightly impact non-SMP
162 performance. Try your application to decide if enabling smp mode
163 improves performance.
165 * tcp - for netlrts- version, default communication is via UDP. Using option
166 tcp will switch to TCP. TCP version of Charm++ is usually slower
167 than UDP, but it is more reliable.
168 * async - For Blue Gene/Q, this option enables use of hardware communication
169 threads. For applications with significant communication on large
170 scale, this option typically improves performance.
171 * regularpages - on Cray systems, Charm++'s default is to use hugepages. This
172 option disables hugepages, and uses regular malloc for messages.
173 * persistent - on Cray systems, this option enables use of persitent mode for
175 * pxshm - use Posix Shared Memory for communication between Charm++ processes
176 within a shared-memory host.
177 * syncft - enable in-memory fault tolerance support in Charm++.
178 * mlogft - enable message logging fault tolerance support in Charm++.
179 * tsan - compile Charm++ with support for Thread Sanitizer.
180 * papi - enable PAPI performance counters.
181 * ooc - build Charm++ with out-of-core execution features.
182 * bigsim - compile Charm++ as running on the BigSim emulator.
184 * help - show supported options for a version. For example, for netlrts-linux,
186 > ./build charm++ netlrts-linux help
188 Supported compilers: clang craycc gcc icc icc8 iccstatic pgcc xlc xlc64
189 Supported options: bigemulator bigsim causalft clustermatic g95 gfortran ifort lcs local mlogft omp ooc papi persistent pgf90 pxshm scyld smp syncft sysvshm tcp tsan turing
195 If you have downloaded a binary version of Charm++, you can skip
196 this step-- Charm++ should already be compiled. For win64 systems,
197 see README.win; for Cygwin version, see README.cygwin.
199 Once you have decided on a version, unpack Charm++, cd into charm,
202 > ./build <target> <version> <opts>
205 "charm++" The basic Charm++ language
206 "AMPI" An implementation of MPI on top of Charm++
207 "LIBS" Charm++, AMPI, and other libraries built on top of them
208 "Tau" TAU's performance profiling/tracing
210 <version> is described above
212 <opts> are build-time options (such as the compiler or "smp"),
213 or command line options passed to the charmc compiler script.
214 Common compile time options such as -g, -O, -Ipath, -Lpath, -llib are
217 For example, on a Linux machine, you would run
218 > ./build charm++ netlrts-linux-x86_64 -O
220 This will construct a netlrts-linux-x86_64 directory, link over all
221 the Charm++ source code into netlrts-linux-x86_64/tmp, build the entire
222 Charm++ runtime system in netlrts-linux-x86_64/tmp, and link example
223 programs into netlrts-linux-x86_64/examples.
225 Charm++ can be compiled with several optional features enabled or
226 disabled. These include runtime error checking, tracing, interactive
227 debugging, deterministic record-replay, and more. They can be
228 controlled by passing flags of the form --enable-featurename or
229 --disable-featurename to the build command:
230 > ./build charm++ netlrts-linux-x86_64 --disable-tracing
232 The list of optional features available is shown in the output of
235 Production optimizations: Pass the configure option --with-production
236 to ./build to turn on optimizations in Charm++/Converse. This disables
237 most of the run-time checking performed by Converse and Charm++
238 runtime. This option should be used only after the program has been
239 debugged. Also, this option disables Converse/Charm++ tracing
240 mechanisms such as projections and summary.
242 Performance analysis: Pass the configuration option "--with-production
243 --enable-tracing" to enable tracing and generation of logs for analysis with
244 Projections. This is the recommended way to analyze performance of applications.
246 When Charm++ is built successfully, the directory structure under the
247 target directory will look like:
249 netlrts-linux-x86_64/
251 --- bin/ # all executables
253 --- doc/ # documentations
255 --- include/ # header files
259 --- lib_so/ # dynamic libraries
261 --- examples/ # all example programs
263 --- tests/ # all test programs
265 --- tmp/ # Charm++ build directory
271 To make a sample program, cd into examples/charm++/queens/.
272 This program solves the N-queens problem-- find how many ways there
273 are to arrange N queens on an NxN chess board such that none may
276 To build the program, type make. You should get an
277 executable named "pgm".
283 Following the previous example, to run the program on two processors, type
285 > ./charmrun ./pgm 12 6 +p2
287 This should run for a few seconds, and print out:
288 There are 14200 Solutions to 12 queens. Finish time=4.030000
290 Charmrun is used to provide a uniform interface to run charm programs.
291 On some platforms, charmrun is just a shell script which calls the
292 platform-specific start program, such as mpirun on mpi versions.
294 For netlrts- version, charmrun is an executable which invokes ssh to start
295 node programs on remote machines. You should set up a ~/.nodelist that
296 enumerates all the machines you want to run jobs on, otherwise it will
297 create a default ~/.nodelist for you that contains only localhost. Here is a
298 typical .nodelist file:
300 group main ++shell /bin/ssh
303 The default remote shell program is ssh, but you can define a different remote
304 shell to start remote processes using the ++shell option. You should
305 also make sure that ssh or your alternative can connect to these machines without
306 password authentication. Just type following command to verify:
307 > ssh <machinename> date
308 If this gives you current date immediately, your running environment with this
309 node has been setup correctly.
311 For development purposes, the netlrts- version of charmrun comes with an easy-to-use
312 "++local" option. No remote shell invocation is needed in this case. It starts
313 node programs right on your local machine. This could be useful if you just
314 want to run program on only one machine, for example, your laptop. This
315 can save you all the hassle of setting up ssh or charmd daemons.
316 To use this option, just type:
318 > ./charmrun ++local ./pgm 12 100 +p2
320 However, for best performance, you should launch one node program per processor.
323 BUILDING DYNAMIC LIBRARIES
324 ==========================
326 In order to compile Charm++ into dynamic libraries, one needs to specify the
327 "-build-shared" option to the Charm compiler script "charmc"
328 at link time. For example, to compile Charm++ under netlrts-linux/tmp, run
330 make charm++ OPTS='-O -build-shared'
332 Charm++'s dynamic libraries are compiled into the lib_so/ directory.
333 Typically, they are generated with a ".so" suffix.
335 Note, "-build-shared" option is automatically turned on when building
336 Charm++ using the "build" script. So you don't need to pass "-build-shared"
339 One can compile a Charm++ application linking against Charm++ dynamic
340 libraries by linking with charmc's "-charm-shared" option.
343 charmc -o pgm pgm.o -charm-shared
345 You can then run the program as usual.
346 Note, linking against Charm++ dynamic libraries produces much smaller size
347 binaries and takes much less linking time.
353 The Charm++ web page, with documentation, more programs,
354 and the latest version of Charm++, is at
355 http://charm.cs.illinois.edu/
357 The Charm++ mailing list, for questions, comments, suggestions,
358 improvements, or bug reports is
359 charm@cs.illinois.edu
365 Charm++ was created and is maintained by the Parallel Programming Lab,
366 in the Computer Science department at the University of Illinois at
367 Urbana-Champaign. Our managing professor is Dr. L.V. Kale; students
368 and staff have included (in rough time order) Wennie Shu, Kevin Nomura, Wayne
369 Fenton, Balkrishna Ramkumar, Vikram Saletore, Amitabh B. Sinha, Manish
370 Gupta, Attila Gursoy, Nimish Shah, Sanjeev Krishnan, Jayant DeSouza,
371 Parthasarathy Ramachandran, Jeff Wright, Michael Lang, Jackie Wang,
372 Fang Hu, Michael Denardo, Joshua Yelon, Narain Jagathesan, Zehra Sura,
373 Krishnan Varadarajan, Sameer Paranjpye, Milind Bhandarkar, Robert Brunner,
374 Terry Wilmarth, Gengbin Zheng, Orion Lawlor, Celso Mendes, Karthik Mahesh,
375 Neelam Saboo, Greg Koenig, Sameer Kumar, Sayantan Chakravorty, Chao Huang,
376 Chee Lee, Fillipo Gioachin, Isaac Dooley, Abhinav Bhatele, Aaron Becker,
377 Ryan Mokos, Ramprasad Venkataraman, Gagan Gupta, Pritish Jetley, Lukasz
378 Wesolowski, Esteban Meneses, Chao Mei, David Kunzman, Osman Sarood,
379 Abhishek Gupta, Yanhua Sun, Ehsan Totoni, Akhil Langer, Cyril Bordage,
380 Harshit Dokania, Prateek Jindal, Jonathan Lifflander, Xiang Ni,
381 Harshitha Menon, and Nikhil Jain. Current developers include Eric
382 Bohm, Phil Miller, Bilge Acun, Ronak Buch, Michael Robson, Eric
383 Mikida, Sam White, Juan Galvez, Seonmyeong Bak, Nitin Bhat, Vipul
384 Harsh, Kavitha Chandrasekar, Jaemin Choi, and Karthik Senthil.