Bug #757: disable unnecassary network polling in multicore build

[charm.git] / doc / faq / debugging.tex

\section{Debugging}

\subsubsection{How can I debug Charm++ programs?}

There are many ways to debug programs written in Charm++:

\begin{description}

\item[print] By using {\tt CkPrintf}, values from critical point in the program can be
printed.

\item[gdb] This can be used both on a single processor, and in parallel
simulations. In the latter, each processor has a terminal window with a gdb
connected.

\item[charmdebug] This is the most sophisticated method to debug parallel
programs in Charm++. It is tailored to Charm++ and it can display and inspect
chare objects as well as messages in the system. Single {\em gdb}s can be
attached to specific processors on demand.

\end{description}

\subsubsection{How do I use charmdebug?}

Currently charmdebug is tested to work only under net- versions. With other versions,
testing is pending. To get the Charm Debug tool, check out the source code from the repository.
This will create a directory named ccs\_tools. Move to this directory and
build Charm Debug.

\begin{verbatim}
 git clone git://charm.cs.uiuc.edu/ccs_tools.git 
 cd ccs_tools
 ant
\end{verbatim}

This will create the executable {\tt bin/charmdebug}. To start, simply substitute ``charmdebug'' to
``charmrun'':

\begin{alltt}shell> <path>/charmdebug ./myprogram\end{alltt}

You can find more detailed information in the debugger manual in 
\href{http://charm.cs.illinois.edu/manuals/html/debugger/manual-1p.html}{here}.

\subsubsection{Can I use TotalView?}

Yes, on mpi- versions of Charm++. In this case, the program is a regular MPI
application, and as such any tool available for MPI programs can be used. Notice
that some of the internal data structures (like messages in queue) might be
difficult to find.

\subsubsection{How do I use {\em gdb} with Charm++ programs?}

It depends on the machine. On the net- versions of Charm++, like net-linux,
you can just run the serial debugger:
\begin{alltt}shell> gdb myprogram\end{alltt}

If the problem only shows up in parallel, and you're running on an X
terminal, you can use the {\em ++debug} or {\em ++debug-no-pause} options of charmrun
to get a separate window for each process:
\begin{alltt}
shell> export DISPLAY="myterminal:0"
shell> ./charmrun ./myprogram +p2 ++debug
\end{alltt}

%On the SGI Origin2000, you can again run with ++debug, but this only
%prints out the process ID of each processor and waits 10 seconds. In another
%window, you have to manually attach a debugger to the running process,
%like this:
%<br><tt>&nbsp; > ./charmrun ./myprogram +p2 ++debug</tt>
%<br><tt>Running on 2 processors:&nbsp; ./myprogram ++debug</tt>
%<br><tt>CHARMDEBUG> Processor 0 has PID 34554234</tt>
%<br><tt>CHARMDEBUG> Processor 1 has PID 35086430</tt>
%<br><tt>...</tt>
%<br><tt>&nbsp; > dbx -p 34554234</tt>

\subsubsection{When I try to use the {\em ++debug} option I get: {\tt remote
host not responding... connection closed}}

First, make sure the program at least starts to run properly without {\em ++debug}
(i.e. charmrun is working and there are no problems with the program startup
phase). You need to make sure that gdb or dbx, and xterm are installed
on all the machines you are using (not the one that is running {\tt charmrun}).
If you are working on remote machines from Linux, you may need to run ``xhost +''
locally to give the remote machines permission to display an xterm on
your desktop. If you are working from a Windows machine, you need an X-win
application such as exceed. You need to set this up to give the right permissions
for X windows. You need to make sure the DISPLAY environment variable on
the remote machine is set correctly to your local machine. I recommend
ssh and putty, because it will take care of the DISPLAY environment automatically,
and you can set up ssh to use tunnels so that it even works from a private
subnet(e.g. 192.168.0.8). Since the xterm is displayed from the node machines,
you have to make sure they have the correct DISPLAY set. Again, setting
up ssh in the nodelist file to spawn node programs should take care of
that. If you are using rsh, you need to set DISPLAY in {\em ~/.charmrunrc}
which will be read at start up time by each node program.

%<li>
%<b>I've been having some trouble using </b><tt>charmrun</tt><b> with the
%</b><tt>++debug</tt><b>
%option. I have XWinPro running and use ttssh to do X forwarding. I can
%get xemacs to pop up, but when I try to
%</b><tt>charmrun pgm ++debug</tt><b>,
%I receive the following error messages from ttssh:
%</b><tt>Remote X application
%sent incorrect authentication data. Its X session is being cancelled.</tt></li>

%<br>X forwarding generally doesn't work with charmrun, because X forwarding
%assumes all the programs you want to see are running on the machine you're
%logged in to.&nbsp; Charmrun starts your program on the nodes of the parallel
%machine, which generally confuses X forwarding.&nbsp; So forget about forwarding
%and just set the DISPLAY directly.
%<br>&nbsp;
%<li>
%<b>How else can I debug my Charm++ programs?</b></li>

%<br>The usual methods still work in parallel--diagnostic printouts, selective
%code removal, working up from tiny input sizes, etc.


\subsubsection{My debugging printouts seem to be out of order. How can I prevent this?}

Printouts from different processors do not normally stay ordered. Consider
the code:
\begin{alltt}
...somewhere... \{
  CkPrintf("cause\textbackslash{}n");
  proxy.effect();
\}
void effect(void) \{
  CkPrintf("effect\textbackslash{}n");
\}
\end{alltt}

Though you might expect this code to always print ``cause, effect'', you
may get ``effect, cause''. This can only happen when the cause and
effect execute on different processors, so cause's output is delayed.

If you pass the extra command-line parameter {\em +syncprint}, then CkPrintf
actually blocks until the output is queued, so your printouts should at
least happen in causal order. Note that this does dramatically slow down
output.

\subsubsection{Is there a way to flush the print buffers in Charm++ (like
{\tt fflush()})?}

Charm++ automatically flushes the print buffers every newline and at
program exit. There is no way to manually flush the buffers at another
point.

\subsubsection{My Charm++ program is causing a seg fault, and the debugger shows that
it's crashing inside {\em malloc} or {\em printf} or {\em fopen}!}

This isn't a bug in the C library, it's a bug in your program -- you're
corrupting the heap. Link your program again with {\em -memory paranoid} and
run it again in the debugger. {\em -memory paranoid} will check the heap and
detect buffer over- and under-run errors, double-deletes, delete-garbage,
and other common mistakes that trash the heap.

\subsubsection{Everything works fine on one processor, but when I run on
multiple processors it crashes!}

It's very convenient to do your testing on one processor (i.e., with
{\em +p1}); but there are several things that only happen on multiple processors.

A single processor has just one set of global variables, but multiple
processors have different global variables. This means on one processor,
you can set a global variable and it stays set ``everywhere'' (i.e., right
here!), while on two processors the global variable never gets initialized
on the other processor. If you must use globals, either set them on every
processor or make them into {\em readonly} globals.

A single processor has just one address space, so you actually {\em can}
pass pointers around between chares. When running on multiple processors,
the pointers dangle. This can cause incredibly weird behavior -- reading
from uninitialized data, corrupting the heap, etc. The solution is to never,
ever send pointers in messages -- you need to send the data the pointer points
to, not the pointer.

\subsubsection{I get the error: ``{\tt Group ID is zero-{}- invalid!}''. What does
this mean?}

The {\em group} it is refering to is the chare group. This
error is often due to using an uninitialized proxy or handle; but it's
possible this indicates severe corruption. Run with {\em ++debug} and check
it you just sent a message via an uninitialized proxy.

\subsubsection{I get the error: {\tt Null-Method Called. Program may have Unregistered
Module!!} What does this mean?}

You are trying to use code from a module that has not been properly
initialized.

So, in the {\em .ci} file for your {\em mainmodule}, you should
add an ``extern module'' declaration:
\begin{alltt}
mainmodule whatever \{
  extern module someModule;
  ...
\}
\end{alltt}

\subsubsection{When I run my program, it gives this error:}

\begin{alltt}
Charmrun: error on request socket-{}-
Socket closed before recv.
\end{alltt}

This means that the node program died without informing {\tt charmrun}
about it, which typically means a segmentation fault while in the interrupt
handler or other critical communications code. This indicates severe
corruption in Charm++'s data structures, which is likely the result of
a heap corruption bug in your program. Re-linking with {\em -memory paranoid}
may clarify the true problem.

\subsubsection{When I run my program, sometimes I get a {\tt Hangup}, and
sometimes {\tt Bus Error}. What do these messages indicate?}

{\tt Bus Error} and {\tt Hangup} both are indications that your
program is terminating abnormally, i.e. with an uncaught signal (SEGV or
SIGBUS). You should definitely run the program with gdb, or use {\em ++debug}.  Bus Errors often mean there is an alignment problem, check if your compiler or environment offers support for detection of these.