Document that zero copy api mostly offers only memory footprint benefit

[charm.git] / doc / charm++ / callbacks.tex

\label{callbacks}

Callbacks provide a generic way to store the information required to
invoke a communication target, such as a chare's entry method, at a
future time. Callbacks are often encountered when writing library
code, where they provide a simple way to transfer control back to a
client after the library has finished. For example, after finishing a
reduction, you may want the results passed to some chare's entry
method.  To do this, you would create an object of type
\kw{CkCallback} with the chare's \kw{CkChareID} and entry method
index, and pass this callback object to the reduction library.

\section{Creating a CkCallback Object}
\label{sec:callbacks/creating}

\index{CkCallback}

There are several different types of \kw{CkCallback} objects; the type
of the callback specifies the intended behavior upon invocation of the
callback. Callbacks must be invoked with the \charmpp{} message of the
type specified when creating the callback. If the callback is being
passed into a library which will return its result through the
callback, it is the user's responsibility to ensure that the type of
the message delivered by the library is the same as that specified in
the callback. Messages delivered through a callback are not
automatically freed by the Charm RTS. They should be freed or stored
for future use by the user.

Callbacks that target chares require an ``entry method index'', an
integer that identifies which entry method will be called.  An entry
method index is the \charmpp{} version of a function pointer.  The
entry method index can be obtained using the syntax:

\begin{alltt}
\uw{int myIdx} = CkIndex_\uw{ChareName}::\uw{EntryMethod}(\uw{parameters});
\end{alltt}

Here, \uw{ChareName} is the name of the chare (group, or array)
containing the desired entry method, \uw{EntryMethod} is the name of
that entry method, and \uw{parameters} are the parameters taken by the
method.  These parameters are only used to resolve the proper
\uw{EntryMethod}; they are otherwise ignored. 

Under most circumstances, entry methods to be invoked through a
CkCallback must take a single message pointer as argument. As such, if
the entry method specified in the callback is not overloaded, using
NULL in place of parameters will suffice in fully specifying the
intended target. If the entry method is overloaded, a message pointer
of the appropriate type should be defined and passed in as a parameter
when specifying the entry method. The pointer does not need to be
initialized as the argument is only used to resolve the target entry
method.

The intended behavior upon a callback's invocation is specified
through the choice of callback constructor used when creating the callback. 
Possible constructors are: 

\begin{enumerate}
\item \kw{CkCallback(void (*CallbackFn)(void *, void *), void *param)} - 
When invoked, the callback will pass \uw{param} and the result message
to the given C function, which should have a prototype
like:

\begin{alltt}
void \uw{myCallbackFn}(void *param, void *message)
\end{alltt}

This function will be called on the processor where the callback was created,
so \uw{param} is allowed to point to heap-allocated data.  Hence, this 
constructor should be used only when it is known that the callback target (which by definition here
is just a C-like function) will be on the same processor as from where the constructor was called. 
Of course, you
are required to free any storage referenced by \uw{param}.

\item \kw{CkCallback(CkCallback::ignore)} - When invoked, the callback
will do nothing.  This can be useful if a \charmpp{} library requires
a callback, but you don't care when it finishes, or will find out some
other way.

\item \kw{CkCallback(CkCallback::ckExit)} - When invoked, the callback
will call CkExit(), ending the Charm++ program.

\item \kw{CkCallback(int ep, const CkChareID \&id)} - When invoked, the 
callback will send its message to the given entry method (specified by the 
entry point index - \kw{ep}) of the given
Chare (specified by the chare \kw{id}).  Note that a chare proxy will also work in place of a chare id:

\begin{alltt}
        CkCallback myCB(CkIndex_myChare::myEntry(NULL), myChareProxy);
\end{alltt}

\item \kw{CkCallback(int ep, const CkArrayID \&id)} - 
When invoked,
the callback will broadcast its message to the given entry method
of the given array.  An array proxy will work in the place of an array id.

\item \kw{CkCallback(int ep, const CkArrayIndex \&idx, const CkArrayID \&id)} - 
When invoked,
the callback will send its message to the given entry method
of the given array element. 

\item \kw{CkCallback(int ep, const CkGroupID \&id)} - 
When invoked,
the callback will broadcast its message to the given entry method
of the given group.

\item \kw{CkCallback(int ep, int onPE, const CkGroupID \&id)} - 
When invoked, the callback will send its message to the given entry
method of the given group member.

\end{enumerate}

One final type of callback, \kw{CkCallbackResumeThread()}, can only be
used from within threaded entry methods.  This callback type is
discussed in section \ref{sec:ckcallbackresumethread}.

\section{CkCallback Invocation}

\label{libraryInterface}

A properly initialized \kw{CkCallback} object stores a global
destination identifier, and as such can be freely copied, marshalled,
and sent in messages. Invocation of a CkCallback is done by calling
the function \kw{send} on the callback with the result message as an
argument. As an example, a library which accepts a CkCallback object
from the user and then invokes it to return a result may have the
following interface:

\begin{alltt}
//Main library entry point, called by asynchronous users:
void myLibrary(...library parameters...,const CkCallback \&cb) 
\{
  ..start some parallel computation, store cb to be passed to myLibraryDone later...
\}

//Internal library routine, called when computation is done
void myLibraryDone(...parameters...,const CkCallback \&cb)
\{
  ...prepare a return message...
  cb.send(msg);
\}
\end{alltt}

A \kw{CkCallback} will accept any message type, or even NULL.  The
message is immediately sent to the user's client function or entry
point.  A library which returns its result through a callback should
have a clearly documented return message type. The type of the message
returned by the library must be the same as the type accepted by the
entry method specified in the callback. 

As an alternative to ``send'', the callback can be used in a {\em
  contribute} collective operation. This will internally invoke the
``send'' method on the callback when the contribute operation has
finished.

For examples of how to use the various callback types, please
see \testreffile{megatest/callback.C}

\section{Synchronous Execution with CkCallbackResumeThread}

\label{sec:ckcallbackresumethread}

Threaded entry methods can be suspended and resumed through the {\em
  CkCallbackResumeThread} class. {\em CkCallbackResumeThread} is
derived from {\em CkCallback} and has specific functionality for
threads.  This class automatically suspends the thread when the
destructor of the callback is called.  A suspended threaded client
will resume when the ``send'' method is invoked on the associated
callback.  It can be used in situations when the return value is not
needed, and only the synchronization is important. For example:

\begin{alltt}
// Call the "doWork" method and wait until it has completed
void mainControlFlow() \{
  ...perform some work...
  // call a library
  doWork(...,CkCallbackResumeThread());
  // or send a broadcast to a chare collection
  myProxy.doWork(...,CkCallbackResumeThread());
  // callback goes out of scope; the thread is suspended until doWork calls 'send' on the callback
  
  ...some more work...
\}
\end{alltt}

Alternatively, if doWork returns a value of interest, this can be retrieved by
passing a pointer to {\em CkCallbackResumeThread}. This pointer will be modified
by {\em CkCallbackResumeThread} to point to the incoming message. Notice that
the input pointer has to be cast to {\em (void*\&)}:

\begin{alltt}
// Call the "doWork" method and wait until it has completed
void mainControlFlow() \{
  ...perform some work...
  MyMessage *mymsg;
  myProxy.doWork(...,CkCallbackResumeThread((void*\&)mymsg));
  // The thread is suspended until doWork calls send on the callback

  ...some more work using "mymsg"...
\}
\end{alltt}

Notice that the instance of {\em CkCallbackResumeThread} is constructed
as an anonymous parameter to the ``doWork'' call. This insures that the callback
is destroyed as soon as the function returns, thereby suspending the thread.

It is also possible to allocate a {\em CkCallbackResumeThread} on the heap or on
the stack. We suggest that programmers avoid such usage, and favor the anonymous instance construction
shown above. For completeness, we still present the code for heap and stack
allocation of CkCallbackResumeThread callbacks below.

For heap allocation, the user must explicitly ``delete'' the callback in order to
suspend the thread.

\begin{alltt}
// Call the "doWork" method and wait until it has completed
void mainControlFlow() \{
  ...perform some work...
  CkCallbackResumeThread cb = new CkCallbackResumeThread();
  myProxy.doWork(...,cb);
  ...do not suspend yet, continue some more work...
  delete cb;
  // The thread suspends now

  ...some more work after the thread resumes...
\}
\end{alltt}

For a callback that is allocated on the stack, its destructor will be called only
when the callback variable goes out of scope. In this
situation, the function ``thread\_delay'' can be invoked on the callback to
force the thread to suspend. This also works for heap allocated callbacks.

\begin{alltt}
// Call the "doWork" method and wait until it has completed
void mainControlFlow() \{
  ...perform some work...
  CkCallbackResumeThread cb;
  myProxy.doWork(...,cb);
  ...do not suspend yet, continue some more work...
  cb.thread\_delay();
  // The thread suspends now

  ...some more work after the thread is resumed...
\}
\end{alltt}

In all cases a {\em CkCallbackResumeThread} can be used to suspend a thread
only once.\\
(See \examplerefdir{barnes-charm} for a complete example).\\
{\em Deprecated usage}: in the past, ``thread\_delay'' was used to retrieve the
incoming message from the callback. While that is still allowed for backward
compatibility, its usage is deprecated. The old usage is subject to memory
leaks and dangling pointers.