Documentation: Correct the stated execution context of initproc methods

[charm.git] / doc / faq / charm.tex

\section{Basic \charmpp{} Programming}

\subsection{What's the basic programming model for Charm++?}

Parallel objects using "Asynchronous Remote Method Invocation":

\begin{description}
\item[Asynchronous] in that you {\em do not block} until the method returns--the
caller continues immediately.

\item[Remote] in that the two objects may be separated by a network.

\item[Method Invocation] in that it's just C++ classes calling each other's
methods.
\end{description}

\subsection{What is an "entry method"?}

Entry methods are all the methods of a chare where messages can be sent by other chares.
They are declared in the .ci files, and they must be defined as public methods
of the C++ object representing the chare.

\subsection{When I invoke a remote method, do I block until that method returns?}

No! This is one of the biggest differences between Charm++ and most
other "remote procedure call" systems like, Java RMI, or RPC.
"Invoke an asynchronous method" and "send a message" have exactly the same
semantics and implementation.
Since the invoking method does not wait for the remote method to terminate, it
normally cannot receive any return value. (see later for a way to return values)

\subsection{Why does Charm++ use asynchronous methods?}

Asynchronous method invocation is more efficient because it can be
implemented as a single message send. Unlike with synchronous methods,
thread blocking and unblocking and a return message are not needed.

Another big advantage of asynchronous methods is that it's easy to make
things run in parallel. If I execute:
\begin{alltt}
a->foo();
b->bar();
\end{alltt}
Now foo and bar can run at the same time; there's no reason bar has
to wait for foo.

\subsection{Can I make a method synchronous? Can I then return a value?}

Yes. If you want synchronous methods, so the caller will block, use the {\tt [sync]}
keyword before the method in the .ci file. This requires the sender to be a threaded
entry method, as it will be suspended until the callee finishes.
Sync entry methods are allowed to return values to the caller.

\subsection{What is a threaded entry method? How does one make an entry method threaded?}

A threaded entry method is an entry method for a chare that executes
in a separate user-level thread. It is useful when the entry method wants
to suspend itself (for example, to wait for more data). Note that
threaded entry methods have nothing to do with kernel-level threads or
pthreads; they run in user-level threads that are scheduled by Charm++
itself.

In order to make an entry method threaded, one should add the keyword
{\em threaded} withing square brackets after the {\em entry} keyword in the
interface file:
\begin{alltt}
module M \{
  chare X \{
    entry [threaded] E1(void);
  \};
\};
\end{alltt}

\subsection{If I don't want to use threads, how can an asynchronous method return a value?}

The usual way to get data back to
your caller is via another invocation in the opposite direction:
\begin{alltt}
void A::start(void) \{
  b->giveMeSomeData();
\}
void B::giveMeSomeData(void) \{
  a->hereIsTheData(data);
\}
void A::hereIsTheData(myclass_t data) \{
  ...use data somehow...
\}
\end{alltt}
This is contorted, but it exactly matches what the machine has to do.
The difficulty of accessing remote data encourages programmers to use local
data, bundle outgoing requests, and develop higher-level abstractions,
which leads to good performance and good code.

\subsection{Isn't there a better way to send data back to whoever called me?}

The above example is very non-modular, because {\em b} has to know
that {\em a} called it, and what method to call a back on. For
this kind of request/response code, you can abstract away the ``where to
return the data'' with a {\em CkCallback} object:
\begin{alltt}
void A::start(void) \{
  b->giveMeSomeData(CkCallback(CkIndex_A::hereIsTheData,thisProxy));
\}
void B::giveMeSomeData(CkCallback returnDataHere) \{
  returnDataHere.send(data);
\}
void A::hereIsTheData(myclass_t data) \{
  ...use data somehow...
\}
\end{alltt}
Now {\em b} can be called from several different places in {\em a},
or from several different modules.

\subsection{Why should I prefer the callback way to return data rather than using {\tt [sync]} entry methods?}

There are a few reasons for that:

\begin{itemize}

\item
The caller needs to be threaded, which implies some overhead in creating the
thread. Moreover the threaded entry method will suspend waiting for the data,
preventing any code after the remote method invocation to proceed in parallel.

\item
Threaded entry methods are still methods of an object. While they are suspended
other entry methods for the same object (or even the same threaded entry method)
can be called. This allows for potential problems if the suspending method does
leave some objects in an inconsistent state.

\item
Finally, and probably most important, {\tt [sync]} entry methods can only be
used to return a value that can be computed by a single chare. When more
flexibility is needed, such in cases where the resulting value needs to the
contribution of multiple objects, the callback methodology is the only one
available. The caller could for example send a broadcast to a chare array, which
will use a reduction to collect back the results after they have been computed.

\end{itemize}

\subsection{How does the initialization in Charm work?}

Each processor executes the following operations strictly in order:
\begin{enumerate}
\item All methods registered as {\em initnode};
\item All methods registered as {\em initproc};
\item On processor zero, all {\em mainchares} constructor method is invoked (the ones taking a {\tt CkArgMsg*});
\item The read-onlies are propagated from processor zero to all other processors;
\item The nodegroups are created;
\item The groups are created. During this phase, for all the chare arrays have been created with a block allocation, the corresponding array elements are instantiated;
\item Initialization terminated and all messages are available for processing, including the messages responsible for the instantiation of array elements manually inserted.
\end{enumerate}

This implies that you can assume that the previous steps has completely finished
before the next one starts, and any side effect from all the previous steps are
committed (and can therefore be used).

Inside a single step there is no order guarantee. This implies that, for example,
two groups allocated from mainchare can be instantiated in any order. The only
exception to this is processor zero, where chare objects are instantiated
immediately when allocated in the mainchare, i.e if two groups are allocated,
their order is fixed by the allocation order in the mainchare constructing them. 
Again, this is only valid for processor zero, and in no other processor this
assumption should be made.

To notice that if array elements are allocated in block (by specifying the
number of elements at the end of the {\tt ckNew} function), they are all
instantiated before normal execution is resumed; if manual insertion is used,
each element can be constructed at any time on its home processor, and not
necessarily before other regular communication messages have been delivered to
other chares (including other array elements part of the same array).

\subsection{Does Charm++ support C and Fortran?}

C and Fortran routines can be called from Charm++ using the usual API conventions for accessing them from C++.  AMPI supports Fortran directly, but direct use 
of Charm++ semantics from Fortran is at an immature stage, contact us \htmladdnormallink{charm AT cs.illinois.edu}{mailto:charm AT cs.illinois.edu} if you are interested in pursuing this further.


\subsection{What is a proxy?}

A proxy is a local C++ class that represents a remote C++ class. When
you invoke a method on a proxy, it sends the request across the network
to the real object it represents. In Charm++, all communication is
done using proxies.

A proxy class for each of your classes is generated based on the methods
you list in the .ci file.

\subsection{What are the different ways one can create proxies?}

Proxies can be:
\begin{itemize}
\item
Created using ckNew. This is the only method that actually creates a new
parallel object. "CProxy\_A::ckNew(...)" returns a proxy, as described in
the \htmladdnormallink{manual}{http://charm.cs.uiuc.edu/manuals/html/charm++/}.

\item
Copied from an existing proxy. This happens when you assign two proxies
or send a proxy in a message.

\item
Created from a "handle". This happens when you say "CProxy\_A p=thishandle;"

\item
Created uninitialized. This is the default when you say "CProxy\_A p;".
You'll get a runtime error "proxy has not been initialized" if you try
to use an uninitialized proxy.
\end{itemize}

\subsection{What is wrong if I do {\tt A *ap = new CProxy\_A(handle)}?}

This will not compile, because a {\em CProxy\_A} is not an {\em A}.
What you want is {\em CProxy\_A *ap = new CProxy\_A(handle)}.



%<br>&nbsp;
%<li>
%<b>When sending messages by invoking a method, can we be just in the middle
%of executing another method? I tried to invoke one entry method in one
%object while that target object was in the middle of execution of another
%method, and could not finish until he'd receive the message. Is there something
%wrong with this kind of thinking and can we execute only one method at
%a time? How can I then make two-way communication between methods of two
%objects?</b></li>

%<br>Only one method can execute on a processor at any time. Message sends
%do not interrupt an ongoing execution. Note the lack of <b>blocking receives</b>
%in Charm++.
%<p>The way you implement two-way communication in Charm++ between two objects
%is as follows:
%<p>Object A calls method M on object B. The argument to the method M is
%a message Msg, which contains a field that contains object A's handle (or
%ChareID). Object B's method gets invoked. It constructs a proxy to A using
%A's handle from the message, and invokes a method on A using that proxy.
%<br>&nbsp;

\subsection{Why is the {\em def.h} usually included at the end? Is it
necessary or can I just include it at the beginning?}

You can include the {\em def.h} file once you've actually declared
everything it will reference-- all your chares and readonly variables.
If your chares and readonlies are in your own header files, it is legal
to include the {\em def.h} right away.

However, if the class declaration for a chare isn't visible when you
include the {\em def.h} file, you'll get a confusing compiler error.
This is why we recommend including the {\em def.h} file at the end.

\subsection{How can I use a global variable across different processors?}

Make the global variable "readonly" by declaring it in the .ci file.
Remember also that read-onlies can be safely set only in the mainchare
constructor. Any change after the mainchare constructor has finished will be
local to the processor that made the change. To change a global variable later
in the program, every processor must modify it accordingly (e.g by using a chare
group. Note that chare arrays are not guaranteed to cover all processors)

\subsection{Can I have a class static read-only variable?}

One can have class-static variables as read-onlies. Inside a chare,
group or array declaration in the {\em .ci} file, one can have a readonly
variable declaration. Thus:
\begin{alltt}
chare someChare \{
  ...
  readonly CkGroupID someGroup;
  ...
\};
\end{alltt}
is fine. In the {\em .h} declaration for {\em class someChare},
you will have to put {\em someGroup} as a public static variable,
and you are done.

You then refer to the variable in your program as {\em someChare::someGroup}.

\subsection{How do I measure the time taken by a program or operation?}

You can use {\tt CkWallTimer()} to determine the time on some particular
processor. To time some parallel computation, you need to call CkWallTimer
on some processor, do the parallel computation, then call CkWallTimer again
on the same processor and subtract.

\subsection{What do {\tt CmiAssert} and
{\tt CkAssert} do?}

These are just like the standard C++ {\em assert} calls in {\em \textless assert.h\textgreater}--
they call abort if the condition passed to them is false.

We use our own version rather than the standard version because we have
to call {\em CkAbort}, and because we can turn our asserts off
when {\em --with-production} is used on the build line.  These
assertions are specifically controlled by {\em --enable-error-checking
} or {\em --disable-error-checking}. The {\em --with-production} flag
implies {\em --disable-error-checking}, but it can still be explicitly
enabled with {\em --enable-error-checking}.

\subsection{Can I know how many messages are being sent to a chare?}

No.

There is no nice library to solve this problem, as some messages might be queued
on the receiving processor, some on the sender, and some on the network. You can
still:
\begin{itemize}
\item Send a return receipt message to the sender, and wait until all the
receipts for the messages sent have arrived, then go to a barrier;
\item Do all the sends, then wait for quiescence.
\end{itemize}

\subsection{What is "quiescence"? How does it work?}

Quiescence is When nothing is happening anywhere on the parallel machine.

A low-level background task counts sent and received messages.
When, across the machine, all the messages that have been sent have been
received, and nothing is being processed, quiescence is triggered.

\subsection{Should I use quiescence detection?}

Probably not.

See the \htmladdnormallink{Completion Detection}{http://charm.cs.illinois.edu/manuals/html/charm++/12.html\#SECTION02340000000000000000} section of the manual for instructions on a more local inactivity detection scheme.

In some ways, quiescence is a very strong property (it guarantees {\em nothing}
is happening {\em anywhere}) so if some other library is doing something,
you won't reach quiescence. In other ways, quiescence is a very weak property,
since it doesn't guarantee anything about the state of your application
like a reduction does, only that nothing is happening. Because quiescence
detection is on the one hand so strong it breaks modularity, and on the
other hand is too weak to guarantee anything useful, it's often better
to use something else.

Often global properties can be replaced by much easier-to-compute local
properties. For example, my object could wait until all {\em its} neighbors
have sent it messages (a local property my object can easily detect by
counting message arrivals), rather than waiting until {\em all} neighbor
messages across the whole machine have been sent (a global property that's
difficult to determine). Sometimes a simple reduction is needed instead
of quiescence, which has the benefits of being activated explicitly (each
element of a chare array or chare group has to call contribute) and allows
some data to be collected
at the same time. A reduction is also a few times faster than quiescence
detection. Finally, there are a few situations, such as some tree-search
problems, where quiescence detection is actually the most sensible, efficient
solution.



%<li>
%<b>Can a chare be deleted by using </b><tt>delete this</tt><b>?</b></li>

%<br>You can delete a chare using <tt>delete this;</tt> as long as you do
%not refer to any of its instance variables, or don't send it a message
%after that. <tt>delete this</tt>, by now, is a valid programming construct
%after much debate. The ANSI C++ specification specifically mentions it.
%To delete array elements, use <tt>ckDestroy()</tt> instead of <tt>delete
%this;</tt>.
%<br>&nbsp;
%<li>
%<b>Is there any way to put inheritance in a
%</b><tt>.ci</tt><b> file?</b></li>

%<br>Yes!
%<p>The syntax is exactly like C++, but there's no "public" keyword:
%<pre>array [1D] subArray : parentArray {
%&nbsp; ...the usual...
%};</pre>
%Virtual methods work right away, and entry methods which are declared virtual
%in the .h file are still virtual, even across processors. Multiple inheritance
%works, too. See
%<tt>charm/pgms/charm++/ megatest/inherit.[ihC]</tt> for
%an exhaustive example.
%<br>&nbsp;
%<li>
%<b>Are accumulators supported in Charm++?</b></li>

%<br>No, they are no longer supported. You can get almost exactly the same
%behavior by using a reduction or defining your own group.
%<br>&nbsp;
%<li>
%<b>Can I find out if there are any pending messages for a chare?</b></li>

%<br>No. On a parallel machine, messages destined for a particular chare
%might be queued on the sender, on the network, or queued on the local machine.&nbsp;
%Since the first two are never going to be accessible to you, we didn't
%make the last accessible either.
%<br>&nbsp;</ol>