Doc #1605: Update FAQ text about C and C++ language usage

[charm.git] / doc / faq / ports.tex

\section{Versions and Ports}

\subsection{Has Charm++ been ported to use MPI underneath? What about OpenMP?}

Charm++ supports MPI and can use it as the underlying communication
library. We have tested on MPICH, OpenMPI, and also most vendor MPI
variants.  Charm++ also has explicit support for SMP nodes in MPI
version. Charm++ hasn't been ported to use OpenMP, but OpenMP can be
used from Charm++.

\subsection{How complicated is porting Charm++/Converse?}

Depends. Hopefully, the porting only involves fixing compiler compatibility
issues.  The LRTS abstraction layer was designed to simplify this process and has been used for the
MPI, Verbs, uGNI, and PAMI layers.  User level threads and Isomalloc support may require special
platform specific support.  Otherwise Charm++ is generally platform independent.

\subsection{If the source is available how feasible would it be for us to do ports
ourselves?}

The source is always available, and you're welcome to make it run anywhere.
Any kind of UNIX, Windows, and MacOS machine should be straightforward: just a
few modifications to {\tt charm/src/arch/.../conv-mach.h} (for compiler
issues) and possibly
a new {\em machine.c} (if there's a new communication system involved).
However, porting to embedded hardware with a proprietary OS may be fairly difficult.

\subsection{To what platform has Charm++/Converse been ported to?}

Charm++/Converse has been ported to most UNIX and Linux OS, Windows, and MacOS.

\subsection{Is it hard to port Charm++ programs to different machines?}

\label{porting}
Charm++ itself it fully portable, and should provide exactly 
the same interfaces everywhere (even if the implementations are 
sometimes different).  Still, it's often harder than we'd like
to port user code to new machines.

Many parallel machines have old or weird compilers, and 
sometimes a strange operating system or unique set of libraries.  
Hence porting code to a parallel machine can be suprisingly difficult.

Unless you're absolutely sure you will only run your code on a
single, known machine, we recommend you be very conservative in 
your use of the language and libraries.  ``But it works with my gcc!''
is often true, but not very useful.

Things that seem to work well everywhere include:
\begin{itemize}
\item Small, straightforward Makefiles.  gmake-specific (e.g.,
``ifeq'', filter variables) or convoluted makefiles can lead 
to porting problems and confusion.  Calling charmc instead
of the platform-specific compiler will save you many headaches,
as charmc abstracts away the platform specific flags.
\item Basically all of ANSI C and fortran 77 work everywhere.  These seem 
to be old enough to now have the bugs largely worked out.
%Thankfully, K\&R (no-prototype) C compilers have now died out.
\item C++ classes, inheritance, virtual methods, and namespaces
work without problems everywhere.  Not so uniformly supported 
are C++ templates, the STL, new-style C++ system headers, 
and the other features listed in the C++ question below.
\end{itemize}

\subsection{How should I approach portability of C language code?}

Our suggestions for Charm++ developers are:

\begin{itemize}
\item Avoid the nonstandard type ``long long'', even though many compilers
happen to support it.  Use CMK\_INT8 or CMK\_UINT8,
from conv-config.h, which are macros for the right thing.
``long long'' is not supported on many 64-bit machines (where ``long''
is 64 bits) or on Windows machines (where it's ``\_\_int64'').
\item The ``long double'' type isn't present on all compilers.  You can protect
long double code with {\em \#ifdef CMK\_LONG\_DOUBLE\_DEFINED} if it's really needed.
\item Never use C++ ``//'' comments in C code, or headers included by C.
This will not compile under many compilers. %including the IBM SP C compiler.
\item ``bzero'' and ``bcopy'' are BSD-specific calls.  
Use memset and memcpy for portable programs.
\end{itemize}

If you're writing code that is expected to compile and run on
Microsoft Windows using the Visual C++ compiler (e.g. modification to
NAMD that you intend to submit for integration), that compiler has
limited support for the C99 standard, and Microsoft recommends using
C++ instead.

Many widely-used C compilers on HPC systems have limited support for
the C11 standard. If you want to use features of C11 in your code,
particularly \verb|_Atomic|, we recommend writing the code in C++
instead, since C++11 standard support is much more ubiquitous.

\subsection{How should I approach portability and performance of C++ language code?}

The Charm++ system developers are conservative about which C++
standard version is relied upon in runtime system code and what
features get used to ensure maximum portability across the broad range
of HPC systems and the compilers used on them. Through version 6.8.x,
the system code requires only limited support for C++11 features,
specifically variadic templates and R-value references. From version
6.9 onwards, the system will require a compiler and standard library
with at least full C++11 support.

A good reference for which compiler versions
provide what level of standard support can be found at
\url{http://en.cppreference.com/w/cpp/compiler_support}

Developers of several Charm++ applications have reported good results
using features in more recent C++ standards, with the caveat of
requiring that those applications be built with a sufficiently
up-to-date C++ compiler.

The containers specified in the C++ standard library are generally
designed to provide a very broad API that can be used correctly over
highly-varied use cases. This often entails tradeoffs against the
performance attainable for narrower use cases that some applications
may have. The most visible of these concerns are the tension between
strict iterator invalidation semantics and cache-friendly memory
layout. We recommend that developers whose code includes container
access in performance-critical elements explore alternative
implementations, such as those published by EA, Google, and Facebook,
or potentially write custom implementations tailored to their
application's needs.

In benchmarks across a range of compilers, we have found that avoiding
use of exceptions (i.e. \verb+throw/catch+) and disabling support for
them with compiler flags can produce higher-performance code,
especially with aggressive optimization settings enabled. The runtime
system does not use exceptions internally. If your goal as an
application developer is to most efficiently use large-scale
computational resources, we recommend alternative error-handling
strategies.

\subsection{Why do I get a link error when mixing Fortran and C/C++?}

\label{f2c}

Fortran compilers ``mangle'' their routine names in a variety
of ways.  g77 and most compilers make names all lowercase, and 
append an underscore, like ``foo\_''.  The IBM xlf compiler makes 
names all lowercase without an underscore, like ``foo''. Absoft f90 
makes names all uppercase, like ``FOO''. 

If the Fortran compiler expects a routine to be named ``foo\_'',
but you only define a C routine named ``foo'', you'll get a link 
error (``undefined symbol foo\_'').  Sometimes the UNIX command-line
tool {\em nm} (list symbols in a .o or .a file) can help you see exactly what the 
Fortran compiler is asking for, compared to what you're providing.

Charm++ automatically detects the fortran name mangling scheme
at configure time, and provides a C/C++ macro ``FTN\_NAME'', in ``charm-api.h'',
that expands to a properly mangled fortran routine name.
You pass the FTN\_NAME macro
two copies of the routine name: once in all uppercase, and again 
in all lowercase.
The FTN\_NAME macro then picks the appropriate name and applies any
needed underscores.  ``charm-api.h'' also includes a macro ``FDECL''
that makes the symbol linkable from fortran (in C++, this expands
to extern ``C''), so a complete Fortran subroutine looks like in C or C++:
\begin{alltt}
FDECL void FTN\_NAME(FOO,foo)(void);
\end{alltt}

This same syntax can be used for C/C++ routines called from
fortran, or for calling fortran routines from C/C++.
We strongly recommend using FTN\_NAME instead of hardcoding your
favorite compiler's name mangling into the C routines.

If designing an API with the same routine names in C and 
Fortran, be sure to include both upper and lowercase letters
in your routine names.  This way, the C name (with mixed case)
will be different from all possible Fortran manglings (which
all have uniform case).  For example, a routine named ``foo''
will have the same name in C and Fortran when using the IBM
xlf compilers, which is bad because the C and Fortran versions
should take different parameters.  A routine named ``Foo'' does
not suffer from this problem, because the C version is ``Foo,
while the Fortran version is ``foo\_'', ``foo'', or ``FOO''.

\subsection{How does parameter passing work between Fortran and C?}

Fortran and C have rather different parameter-passing 
conventions, but it is possible to pass simple objects 
back and forth between Fortran and C:

\begin{itemize}

\item Fortran and C/C++ data types are generally completely
interchangeable:

\begin{tabular}{|l|l|}
\hline
\textbf{C/C++ Type} & \textbf{Fortran Type} \\
\hline
int & INTEGER, LOGICAL \\
double & DOUBLE PRECISION, REAL*8 \\
float & REAL, REAL*4 \\
char & CHARACTER \\
\hline
\end{tabular}

\item Fortran internally passes everything, including 
constants, integers, and doubles, by passing a pointer
to the object.  Hence a fortran ``INTEGER'' argument becomes 
an ``int *'' in C/C++:
\begin{alltt}
/* Fortran */
SUBROUTINE BAR(i) 
    INTEGER :: i
    x=i
END SUBROUTINE

/* C/C++ */
FDECL void FTN\_NAME(BAR,bar)(int *i) \{
    x=*i;
\}
\end{alltt}

\item 1D arrays are passed exactly the same in Fortran and C/C++:
both languages pass the array by passing the address of the 
first element of the array.
Hence a fortran ``INTEGER, DIMENSION(:)'' array is an ``int *''
in C or C++.  However, Fortran programmers normally think of 
their array indices as starting from index 1, while in C/C++ 
arrays always start from index 0.  This does NOT change how 
arrays are passed in, so x is actually the same in both 
these subroutines:
\begin{alltt}
/* Fortran */
SUBROUTINE BAR(arr) 
    INTEGER :: arr(3)
    x=arr(1)
END SUBROUTINE

/* C/C++ */
FDECL void FTN\_NAME(BAR,bar)(int *arr) \{
    x=arr[0];
\}
\end{alltt}

\item There is a subtle but important difference between the way
f77 and f90 pass array arguments.  f90 will pass an array object
(which is not intelligible from C/C++) instead of a simple pointer 
if all of the following are true:
\begin{itemize}
\item A f90 ``INTERFACE'' statement is available on the call side.
\item The subroutine is declared as taking an unspecified-length
       array (e.g., ``myArr(:)'') or POINTER variable.
\end{itemize}
Because these f90 array objects can't be used from C/C++, we recommend 
C/C++ routines either provide no f90 INTERFACE or else all the arrays 
in the INTERFACE are given explicit lengths.

\item Multidimensional allocatable arrays are stored with
the smallest index first in Fortran.  C/C++ do not support
allocatable multidimensional arrays, so they must fake them
using arrays of pointers or index arithmetic.

\begin{alltt}
/* Fortran */
SUBROUTINE BAR2(arr,len1,len2) 
    INTEGER :: arr(len1,len2)
    INTEGER :: i,j
    DO j=1,len2
      DO i=1,len1
        arr(i,j)=i;
      END DO
    END DO
END SUBROUTINE

/* C/C++ */
FDECL void FTN\_NAME(BAR2,bar2)(int *arr,int *len1p,int *len2p) \{
    int i,j; int len1=*len1p, len2=*len2p;
    for (j=0;j<len2;j++)
    for (i=0;i<len1;i++)
        arr[i+j*len1]=i;
\}
\end{alltt}

\item Fortran strings are passed in a very strange fashion.
A string argument is passed as a character pointer and a 
length, but the length field, unlike all other Fortran arguments,
is passed by value, and goes after all other arguments.
Hence 

\begin{alltt}
/* Fortran */
SUBROUTINE CALL\_BARS(arg) 
    INTEGER :: arg
    CALL BARS('some string',arg);
END SUBROUTINE

/* C/C++ */
FDECL void FTN\_NAME(BARS,bars)(char *str,int *arg,int strlen) \{
    char *s=(char *)malloc(strlen+1);
    memcpy(s,str,strlen);
    s[strlen]=0; /* nul-terminate string */
    printf("Received Fortran string '\%s' (\%d characters){\textbackslash}n",s,strlen);
    free(s);
\}
\end{alltt}


\item A f90 named TYPE can sometimes successfully be passed into a 
C/C++ struct, but this can fail if the compilers insert different
amounts of padding.  There does not seem to be a portable way to 
pass f90 POINTER variables into C/C++, since different compilers
represent POINTER variables differently.  

\end{itemize}

\subsection{How do I use Charm++ on Xeon Phi?}

In general, no changes are required to use Charm++ on Xeon Phis. To
compile code for Knights Landing, no special flags are required. To
compile code for Knights Corner, one should build Charm++ with the
{\tt mic} option. In terms of network layers, we currently recommend
building the MPI layer ({\tt mpi-linux-x86\_64}) except for machines with
custom network layers, such as Cray systems, on which we recommend
building for the custom layer ({\tt gni-crayxc} for Cray XC machines,
for example). To enable AVX-512 vector instructions, Charm++ can be
built with {\tt -xMIC-AVX512} on Intel compilers or {\tt -mavx512f
  -mavx512er -mavx512cd -mavx512pf} for GNU compilers.

\subsection{How do I use \charm{} on GPUs?}
\charm{} users have two options when utilizing GPUs in \charm.

The first is to write CUDA (or OpenCL, etc) code directly in their \charm{}
applications. This does not take advantage of any of the special GPU-friendly
features the \charm{} runtime provides and is similar to how programmers utilize
GPUs in other parallel environments, e.g. MPI.

The second option is to leverage \charm's GPU library, GPU Manager. This library
provides several useful features including:
\begin{itemize}
\item Automated data movement
\item Ability to invoke callbacks at various points
\item Host side pinned memory pooling
\item Asynchronous kernel invocation
\item Integrated tracing in Projections
\end{itemize}

To do this, \charm{} must be built with the \texttt{cuda} option. Users must
describe their kernels using a work request struct, which includes the buffers
to be copied, callbacks to be invoked, and kernel to be executed. Additionally,
users can take advantage of a pre-allocated host side pinned memory pool
allocated by the runtime via invoking \texttt{hapi\_poolMalloc}. Finally, the
user must compile this code using the appropriate \texttt{nvcc} compiler as per
usual.

More details on using GPUs in \charm{} can be found in the
\htmladdnormallink{GPU Manager Library}{http://charm.cs.illinois.edu/manuals/html/libraries/6.html}
entry in the larger Libraries Manual.