Charm++

       Copyright (C) 1989-2019 Regents of the University of Illinois

INTRODUCTION
============

Charm++ is a message-passing parallel language and runtime system.
It is implemented as a set of libraries for C++, is efficient,
and is portable to a wide variety of parallel machines.
Source code is provided, and non-commercial use is free.


GETTING THE LATEST SOURCE
=========================

You can use anonymous Git access to obtain the latest Charm++ source
code, as follows:

     > git clone http://charm.cs.illinois.edu/gerrit/charm


BUILD CONFIGURATION
===================

QUICK START:
------------
First-time users are encouraged to run the top-level "build" script and follow its lead:
     > ./build


ADVANCED BUILD OPTIONS:
-----------------------

First, you need to decide which version of charm++ to use. The "build"
script takes several command line options to compile Charm++. The command line syntax is:

     > build <target> <version> [options ...]
                                [--basedir=dir] [--libdir=dir] [--incdir=dir]
                                [charmc-options ...]

For detailed help messages, pass -h or --help to the build script, i.e.
     > ./build --help

REQUIRED:
---------
<target> specifies the parts of Charm++ to compile.  The most often used
  <target> is "charm++", which will compile the key Charm++ executables and
  runtime libraries.  Other common targets are "AMPI" and "LIBS".
<versions> defines the CPU, OS and Communication layer of the machines.  See
  "How to choose a <version>" below for details.

OPTIONAL:
---------
<options> defines more detailed information of the compilations, including
  compilers, features to support, etc.  See "How to choose <option>"
  below.
[--libdir=dir] specify additional lib paths for building Charm++.
[--incdir=dir] specify additional include paths for building Charm++.
[--basedir=dir] a shortcut to specify additional include and lib paths for
               building Charm++, the include path is dir/include and lib path
               is dir/lib.


Running build script, a directory of the name of combination of version and
options like "<version>-<option1>-<option2>-..." will be created and
the build script will compile Charm++ under this directory.

For example, on an ordinary Linux PC:

     > ./build charm++ netlrts-linux-x86_64

will build charm++ in the directory: netlrts-linux-x86_64/. The communication
defaults to UDP packets and the compiler to gcc.

For a more complex example, consider a shared-memory build with the
Intel C++ compiler, where you want the communication to happen over TCP sockets:

     > ./build charm++ netlrts-linux-x86_64 smp icc tcp

will build charm++ in the directory: netlrts-linux-x86_64-smp-tcp-icc/.

You can specify multiple options, however you can use at most one compiler
option. The sequencing of options given to the build script should follow the rules:
1. compiler option will be at the end;
2. other options are sorted alphabetically.

**** How to choose a <version> ****

Here is the table for choosing a correct build version. The default setting of compiler
in Charm++ is gcc/g++ on Linux and Clang on MacOS. However, one can use <options> to specify other
compilers. See the detailed explanation of the <options> below.
(Note: this isn't a complete list.  Run ./build for a complete listing)

Charm version                OS         Communication    Default Compiler
-------------             ---------     --------------   --------------------
netlrts-linux-x86_64       Linux            UDP            GNU compiler
netlrts-darwin-x86_64      MacOS X          UDP            Clang C++ compiler
netlrts-win-x86_64         Windows          UDP            MS Visual C++
mpi-linux-x86_64           Linux            MPI            GNU compiler
multicore-darwin-x86_64    MacOS X          Shared memory  Clang C++ compiler
pamilrts-bluegeneq         CNK              PAMI           BGClang C++ compiler
gni-crayxc                 Linux            GNI            CC (whatever PrgEnv module is loaded)
gni-crayxe                 Linux            GNI            CC (whatever PrgEnv module is loaded)
verbs-linux-x86_64         Linux            IB Verbs       GNU compiler
ofi-linux-x86_64           Linux            OFI            GNU compiler


To choose <version>, your choice is determined by two options:

1.)  The way a parallel program written in Charm++ will communicate:

        "netlrts-" Charm++ communicates using the regular TCP/IP stack
(UDP packets), which works everywhere but is fairly slow.  Use this
option for networks of workstations, clusters, or single-machine
development and testing.

        "gni-", "pamilrts-", "verbs-", "ofi-" Charm++
communicates using direct calls to the machine's communication primitives.
Use these versions on machines that support them for best performance.

        "mpi-" Charm++ communicates using MPI calls.

        "multicore-" Charm++ communicates using shared memory within a single node.


2.)  Your operating system:

        "linux-x86_64"       Linux with AMD64 64-bit x86 instructions
        "win-x86_64"         MS Windows with MS Visual C++ compiler
        "darwin-x86_64"      Apple Mac OS X
        "bluegeneq"          IBM's Blue Gene/Q
        "cray{xe/xc}"        Cray's X-Series Supercomputer
        "linux-ppc64le"      POWER/PowerPC


Your Charm++ version is made by concatenating the options, e.g.:

"netlrts-linux-x86_64"   Charm++ for a network of 64-bit Linux workstations,
                         compiled using g++.

"gni-crayxc"             Charm++ for Cray XC systems using the system's compiler.


**** How to choose <options> ****

<version> above defines the most important OS, CPU and Communication of
your machine, and most of time it uses gcc as the default compiler.
To use a different compiler or demand additional special feature support, you
need to choose <options> from the following list:

* icc - Intel C/C++ compiler.
* ifort - Intel Fortran compiler
* xlc - IBM XLC compiler.
* clang - Clang compiler.
* mpicxx - Use MPI-wrappers for MPI builds.
* pgcc - Portland Group's C++ compiler.

* smp - Enable direct SMP support.  An "smp" version communicates using
        shared memory within a machine; but normal message passing across
        machines. Because of locking, "smp" may slightly impact non-SMP
        performance. Try your application to decide if enabling smp mode
        improves performance.

* tcp - for netlrts- version, default communication is via UDP. Using option
        tcp will switch to TCP. TCP version of Charm++ is usually slower
        than UDP, but it is more reliable.
* async - For Blue Gene/Q, this option enables use of hardware communication 
          threads. For applications with significant communication on large
          scale, this option typically improves performance.
* regularpages - on Cray systems, Charm++'s default is to use hugepages. This
                 option disables hugepages, and uses regular malloc for messages.
* persistent - on Cray systems, this option enables use of persitent mode for
               communication.
* pxshm - use Posix Shared Memory for communication between Charm++ processes
          within a shared-memory host.
* syncft - enable in-memory fault tolerance support in Charm++.
* mlogft - enable message logging fault tolerance support in Charm++.
* tsan - compile Charm++ with support for Thread Sanitizer.
* papi - enable PAPI performance counters.
* ooc - build Charm++ with out-of-core execution features.
* bigsim - compile Charm++ as running on the BigSim emulator.

* help - show supported options for a version. For example, for netlrts-linux,
         running:
         > ./build charm++ netlrts-linux help
         will give:
Supported compilers: clang craycc gcc icc iccstatic pgcc xlc xlc64
Supported options: bigemulator bigsim causalft clustermatic flang gfortran ifort local mlogft omp ooc papi persistent pgf90 pxshm scyld smp syncft sysvshm tcp tsan turing


BUILDING THE SOURCE
===================

If you have downloaded a binary version of Charm++, you can skip
this step-- Charm++ should already be compiled.

Once you have decided on a version, unpack Charm++, cd into charm,
and run

     > ./build <target> <version> <opts>

<target> is one of
        "charm++"  The basic Charm++ language
        "AMPI"     An implementation of MPI on top of Charm++
        "LIBS"     Charm++, AMPI, and other libraries built on top of them
        "Tau"      TAU's performance profiling/tracing

<version> is described above

<opts> are build-time options (such as the compiler or "smp"),
or command line options passed to the charmc compiler script.
Common compile time options such as -g, -O, -Ipath, -Lpath, -llib are
accepted.

For example, on a Linux machine, you would run
     > ./build charm++ netlrts-linux-x86_64 -O

This will construct a netlrts-linux-x86_64 directory, link over all
the Charm++ source code into netlrts-linux-x86_64/tmp, build the entire
Charm++ runtime system in netlrts-linux-x86_64/tmp, and link example
programs into netlrts-linux-x86_64/examples.

Charm++ can be compiled with several optional features enabled or
disabled. These include runtime error checking, tracing, interactive
debugging, deterministic record-replay, and more. They can be
controlled by passing flags of the form --enable-featurename or
--disable-featurename to the build command:
     > ./build charm++ netlrts-linux-x86_64 --disable-tracing

The list of optional features available is shown in the output of
     > ./build --help

Production optimizations: Pass the configure option --with-production
to ./build to turn on optimizations in Charm++/Converse. This disables
most of the run-time checking performed by Converse and Charm++
runtime. This option should be used only after the program has been
debugged. Also, this option disables Converse/Charm++ tracing
mechanisms such as projections and summary.

Performance analysis: Pass the configuration option "--with-production 
--enable-tracing" to enable tracing and generation of logs for analysis with 
Projections. This is the recommended way to analyze performance of applications.

When Charm++ is built successfully, the directory structure under the
target directory will look like:

netlrts-linux-x86_64/
   |
   ---  benchmarks/		# benchmark programs
   |
   ---  bin/			# all executables
   |
   ---  doc/			# documentations
   |
   ---  include/		# header files
   |
   ---  lib/			# libraries
   |
   ---  lib_so/			# dynamic libraries
   |
   ---  examples/		# example programs
   |
   ---  tests/			# test programs
   |
   ---  tmp/			# Charm++ build directory


BUILDING A PROGRAM
==================

To make a sample program, cd into examples/charm++/queens/.
This program solves the N-queens problem-- find how many ways there
are to arrange N queens on an NxN chess board such that none may
attack another.

To build the program, type make.  You should get an
executable named "pgm".


RUNNING A PROGRAM
==================

Following the previous example, to run the program on two processors, type

     > ./charmrun ./pgm 12 6 +p2

This should run for a few seconds, and print out:
There are 14200 Solutions to 12 queens. Finish time=4.030000

Charmrun is used to provide a uniform interface to run charm programs.
On some platforms, charmrun is just a shell script which calls the
platform-specific start program, such as mpirun on mpi versions.

For netlrts- version, charmrun is an executable which invokes ssh to start
node programs on remote machines. You should set up a ~/.nodelist that
enumerates all the machines you want to run jobs on, otherwise it will
create a default ~/.nodelist for you that contains only localhost. Here is a
typical .nodelist file:

group main ++shell /bin/ssh
host <machinename>

The default remote shell program is ssh, but you can define a different remote
shell to start remote processes using the ++shell option. You should
also make sure that ssh or your alternative can connect to these machines without
password authentication. Just type following command to verify:
     > ssh <machinename> date
If this gives you current date immediately, your running environment with this
node has been setup correctly.

For development purposes, the netlrts- version of charmrun comes with an easy-to-use
"++local" option. No remote shell invocation is needed in this case. It starts
node programs right on your local machine. This could be useful if you just
want to run program on only one machine, for example, your laptop. This
can save you all the hassle of setting up ssh or charmd daemons.
To use this option, just type:

     > ./charmrun ++local ./pgm 12 100 +p2

However, for best performance, you should launch one node program per processor.


BUILDING DYNAMIC LIBRARIES
==========================

In order to compile Charm++ into dynamic libraries, one needs to specify the
"-build-shared" option to the Charm compiler script "charmc"
at link time. For example, to compile Charm++ under netlrts-linux/tmp, run

     > make charm++ OPTS='-O -build-shared'

Charm++'s dynamic libraries are compiled into the lib_so/ directory.
Typically, they are generated with a ".so" suffix.

Note, "-build-shared" option is automatically turned on when building
Charm++ using the "build" script. So you don't need to pass "-build-shared"
option to "build".

One can compile a Charm++ application linking against Charm++ dynamic
libraries by linking with charmc's "-charm-shared" option.
For example:

     > charmc -o pgm pgm.o -charm-shared

You can then run the program as usual.
Note, linking against Charm++ dynamic libraries produces much smaller size
binaries and takes much less linking time.


FOR MORE INFORMATION
====================

The Charm++ web page, with documentation, more programs,
and the latest version of Charm++, is at
	http://charm.cs.illinois.edu/

The Charm++ mailing list, for questions, comments, suggestions,
improvements, or bug reports is
	charm@cs.illinois.edu


AUTHORS
=======

Charm++ was created and is maintained by the Parallel Programming Lab,
in the Computer Science department at the University of Illinois at
Urbana-Champaign.  Our managing professor is Dr. L.V. Kale; students
and staff have included (in rough time order) Wennie Shu, Kevin Nomura, Wayne
Fenton, Balkrishna Ramkumar, Vikram Saletore, Amitabh B. Sinha, Manish
Gupta, Attila Gursoy, Nimish Shah, Sanjeev Krishnan, Jayant DeSouza,
Parthasarathy Ramachandran, Jeff Wright, Michael Lang, Jackie Wang,
Fang Hu, Michael Denardo, Joshua Yelon, Narain Jagathesan, Zehra Sura,
Krishnan Varadarajan, Sameer Paranjpye, Milind Bhandarkar, Robert Brunner,
Terry Wilmarth, Gengbin Zheng, Orion Lawlor, Celso Mendes, Karthik Mahesh,
Neelam Saboo, Greg Koenig, Sameer Kumar, Sayantan Chakravorty, Chao Huang,
Chee Lee, Fillipo Gioachin, Isaac Dooley, Abhinav Bhatele, Aaron Becker,
Ryan Mokos, Ramprasad Venkataraman, Gagan Gupta, Pritish Jetley, Lukasz
Wesolowski, Esteban Meneses, Chao Mei, David Kunzman, Osman Sarood,
Abhishek Gupta, Yanhua Sun, Ehsan Totoni, Akhil Langer, Cyril Bordage,
Harshit Dokania, Prateek Jindal, Jonathan Lifflander, Xiang Ni,
Harshitha Menon, Nikhil Jain, Vipul Harsh, Bilge Acun, Phil Miller,
Seonmyeong Bak, and Karthik Senthil. Current developers include Eric Bohm,
Ronak Buch, Michael Robson, Eric Mikida, Sam White, Juan Galvez, Nitin Bhat,
Kavitha Chandrasekar, Jaemin Choi, Matthias Diener, Evan Ramos,
Raghavendra Kanakagiri, and Venkatasubrahmanian Narayanan.