Documentation #1541: fix explanations of Converse Ctv, Cpv, Csv variables

[charm.git] / doc / projections / tracing.tex

\projections{} is a performance analysis/visualization framework that
helps you understand and investigate performance-related problems in
the (\charmpp{}) applications. It is a framework with an
event tracing component which allows to control the
amount of information generated. The tracing has low perturbation 
on the application. It
also has a Java-based visualization and analysis component with
various views that help present the performance information in a
visually useful manner.

Performance analysis with \projections{} typically involves two simple
steps:

\begin{enumerate}
\item 
Prepare your application by linking with the appropriate trace
generation modules and execute it to generate trace data.
\item
Using the Java-based tool to visually study various aspects of the
performance and locate the performance issues for that application execution.
\end{enumerate}

The \charmpp{} runtime automatically records pertinent performance
data for performance-related events during execution. These
events include the start and end of entry method execution, message
send from entry methods and scheduler idle time. This means {\em
most} users do not need to manually insert code into their
applications in order to generate trace data. In scenarios where
special performance information not captured by the runtime is
required, an API (see section \ref{sec::api}) is available for
user-specific events with some support for visualization by the
Java-based tool. If greater control over tracing activities
(e.g. dynamically turning instrumentation on and off) is desired, the
API also allows users to insert code into their applications for such
purposes.

The automatic recording of events by the \projections{} framework
introduces the overhead of an if-statement for each runtime event,
even if no performance analysis traces are desired. Developers of
\charmpp{} applications who consider such an overhead to be
unacceptable (e.g. for a production application which requires the
absolute best performance) may recompile the \charmpp{} runtime with
the \verb|--with-production| flag, which removes the instrumentation
stubs.  To enable the instrumentation stubs while retaining the other
optimizations enabled by \verb|--with-production|, one may
compile \charmpp{} with both \verb|--with-production|
and \verb|--enable-tracing|, which explicitly enables Projections tracing.

To enable performance tracing of your application, users simply need
to link the appropriate trace data generation module(s) (also referred
to as {\em tracemode(s)}). (see section \ref{sec::trace modules})

\section{Enabling Performance Tracing at Link/Run Time}
\label{sec::trace modules}

\projections{} tracing modules dictate the type of performance data,
data detail and data format each processor will record. They are also
referred to as ``tracemodes''. There are currently 2 tracemodes
available. Zero or more tracemodes may be specified at link-time. When
no tracemodes are specified, no trace data is generated.

\subsection{Tracemode {\tt projections}}

Link time option: {\tt -tracemode projections}

This tracemode generates files that contain
information about all \charmpp{} events like entry method calls and
message packing during the execution of the program.  The data will be
used by \projections{} in visualization and analysis.

This tracemode creates a single symbol table file and $p$ ASCII
log files for $p$ processors. The names of the log files will be
NAME.\#.log where NAME is the name of your executable and \# is the
processor \#. The name of the symbol table file is NAME.sts where NAME
is the name of your executable.

This is the main source of data needed by the performance
visualizer. Certain tools like timeline will not work without the
detail data from this tracemode.

The following is a list of runtime options available under this tracemode:

\begin{itemize}
\item
{\tt +logsize NUM}: keep only NUM log entries in the memory of each
processor. The logs are emptied and flushed to disk when filled.
\item
{\tt +binary-trace}:  generate projections log in binary form.
\item
{\tt +gz-trace}:      generate gzip (if available) compressed log files.
\item
{\tt +gz-no-trace}:      generate regular (not compressed) log files.
\item
{\tt +checknested}: a debug option. Checks if events are improperly nested
while recorded and issue a warning immediately.

\item {\tt +trace-subdirs NUM}: divide the generated log files among
  {\tt NUM} subdirectories of the trace root, each named {\tt
    PROGNAME.projdir.K}
\end{itemize}

\subsection{Tracemode {\tt summary}}

Compile option: {\tt -tracemode summary}

In this tracemode, execution time across all entry points for each
processor is partitioned into a fixed number of equally sized
time-interval bins. These bins are globally resized whenever they are
all filled in order to accommodate longer execution times while keeping
the amount of space used constant.

Additional data like the total number of calls made to each entry
point is summarized within each processor.

This tracemode will generate a single symbol table file and $p$ ASCII
summary files for $p$ processors. The names of the summary files will
be NAME.\#.sum where NAME is the name of your executable and \# is the
processor \#. The name of the symbol table file is NAME.sum.sts where NAME
is the name of your executable.

This tracemode can be used to control the amount of output generated
in a run. It is typically used in scenarios where a quick look at the
overall utilization graph of the application is desired to identify
smaller regions of time for more detailed study. Attempting to
generate the same graph using the detailed logs of the prior tracemode
may be unnecessarily time consuming or impossible.

The following is a list of runtime options available under this tracemode:

\begin{itemize}
\item
{\tt +bincount NUM}:   use NUM time-interval bins. The bins are resized and compacted when filled.
\item
{\tt +binsize TIME}:   sets the initial time quantum each bin represents.
\item
{\tt +version}:        set summary version to generate.
%\item
%{\tt +epThreshold}: DOESNT DO ANYTHING YET. LEFT COMMENTED FOR DOC PURPOSES
%\item
%{\tt +epInterval}: DOESNT DO ANYTHING YET. LEFT COMMENTED FOR DOC PURPOSES
\item
{\tt +sumDetail}: Generates a additional set of files, one per processor,
that stores the time spent by each entry method associated with each 
time-bin. The names of ``summary detail'' files will be NAME.\#.sumd where
NAME is the name of your executable and \# is the processor \#.
\item
{\tt +sumonly}: Generates a single file that stores a single
utilization value per time-bin, averaged across all processors. This
file bears the name NAME.sum where NAME is the name of your
executable. This runtime option currently overrides the {\tt
+sumDetail} option.
\end{itemize}

\subsection{General Runtime Options}
\label{sec::general options}

The following is a list of runtime options available with the same
semantics for all tracemodes:

\begin{itemize}
\item
{\tt +traceroot DIR}: place all generated files in DIR.
\item
{\tt +traceoff}: trace generation is turned off when the application
is started. The user is expected to insert code to turn tracing on at
some point in the run.
\item
{\tt +traceWarn}: By default, warning messages from the framework are
not displayed. This option enables warning messages to be printed to
screen. However, on large numbers of processors, they can overwhelm
the terminal I/O system of the machine and result in unacceptable
perturbation of the application.
\item
{\tt +traceprocessors RANGE}: Only output logfiles for PEs present in the range (i.e. {\tt 0-31,32-999966:1000,999967-999999} to record every PE on the first 32, only every thousanth for the middle range, and the last 32 for a million processor run).

\end{itemize}

\subsection{End-of-run Analysis for Data Reduction}
\label{sec::data reduction}

As applications are scaled to thousands or hundreds of thousands of
processors, the amount of data generated becomes extremely large and
potentially unmanageable by the visualization tool. At the time of this{\tt +traceWarn}
documentation, \projections{} is capable of handling data from 8000+
processors but with somewhat severe tool responsiveness issues. We
have developed an approach to mitigate this data size problem with
options to trim-off ``uninteresting'' processors' data by not writing
such data at the end of an application's execution.

This is currently done through heuristics to pick out interesting
extremal (i.e. poorly behaved) processors and at the same time using a
k-means clustering to pick out exemplar processors from equivalence
classes to form a representative subset of processor data. The analyst
is advised to also link in the summary module via {\tt +tracemode
summary} and enable the {\tt +sumDetail} option in order to retain
some profile data for processors whose data were dropped.

\begin{itemize}
\item
{\tt +extrema}: enables extremal processor identification analysis at
the end of the application's execution.
\item
{\tt +numClusters}: determines the number of clusters (equivalence
classes) to be used by the k-means clustering algorithm for
determining exemplar processors. Analysts should take advantage of
their knowledge of natural application decomposition to guess at a
good value for this.
\end{itemize}

This feature is still being developed and refined as part of our
research. It would be appreciated if users of this feature could
contact the developers if you have input or suggestions.


\section{Controlling Tracing from Within the Program}
\label{sec::api}

\subsection{Selective Tracing}
\label{sec::selective tracing}

\charmpp{} allows user to start/stop tracing the execution at certain
points in time on the local processor. Users are advised to make these
calls on all processors and at well-defined points in the application.

Users may choose to have instrumentation turned off at first (by
command line option {\tt +traceoff} - see section \ref{sec::general options}) if some period of time in middle of the
application\'s execution is of interest to the user.

Alternatively, users may start the application with instrumentation
turned on (default) and turn off tracing for specific sections of the
application.

Again, users are advised to be consistent as the {\tt +traceoff}
runtime option applies to all processors in the application.

\begin{itemize}
\item
{\tt void traceBegin()}

Enables the runtime to trace events (including all user events) on the local processor where {\tt traceBegin} is called.

\item
{\tt void traceEnd()}

Prevents the runtime from tracing events (including all user events) on the local processor where {\tt traceEnd} is called.

\end{itemize}

\subsection{User Events}
\label{sec::user events}

\projections{} has the ability to visualize traceable user
specified events. User events are usually displayed in the Timeline view as vertical bars above the entry methods. Alternatively the user event can be displayed as a vertical bar that vertically spans the timelines for all processors. Follow these following basic steps for creating user events in a charm++ program:

\begin{enumerate}
\item
Register an event with an identifying string and either specify or acquire
a globally unique event identifier. All user events that are not registered will be displayed in white.

\item
Use the event identifier to specify trace points in your code of interest to you.
\end{enumerate}

The functions available are as follows:

\begin{itemize}
\item
{\tt int traceRegisterUserEvent(char* EventDesc, int EventNum=-1) }

This function registers a user event by associating {\tt EventNum} to
{\tt EventDesc}. If {\tt EventNum} is not specified, a globally unique
event identifier is obtained from the runtime and returned. The string {\tt EventDesc} must either be a constant string, or it can be a dynamically allocated string that is {\bf NOT} freed by the program. If the {\tt EventDesc} contains a substring ``***'' then the Projections Timeline tool will draw the event vertically spanning all PE timelines.

{\tt EventNum} has to be the same on all processors. Therefore use one of the following methods to ensure the same value for any PEs generating the user events:

\begin{enumerate}
\item
Call {\tt traceRegisterUserEvent} on PE 0 in main::main without specifying
an event number, and store returned event number into a readonly variable.
\item
Call {\tt traceRegisterUserEvent} and specify the event number on
processor 0. Doing this on other processors would have no
effect. Afterwards, the event number can be used in the following user
event calls.
\end{enumerate}

Eg. {\tt traceRegisterUserEvent("Time Step Begin", 10);}

Eg. {\tt eventID = traceRegisterUserEvent(``Time Step Begin'');}

\end{itemize}



There are two main types of user events, bracketed and non bracketed. Non-bracketed user events mark a specific point in time. Bracketed user events span an arbitrary contiguous time range. Additionally, the user can supply a short user supplied text string that is recorded with the event in the log file. These strings should not contain newline characters, but they may contain simple html formatting tags such as \texttt{<br>}, \texttt{<b>}, \texttt{<i>}, \texttt{<font color=\#ff00ff>}, etc.

The calls for recording user events are the following:

\begin{itemize}


\item
{\tt void traceUserEvent(int EventNum) }

This function creates a user event that marks a specific point in time.

Eg. {\tt traceUserEvent(10);}

\item
{\tt void traceUserBracketEvent(int EventNum, double StartTime, double EndTime) }

This function records a user event spanning a time interval from {\tt StartTime} to {\tt EndTime}. Both {\tt StartTime} and {\tt EndTime} should be obtained from a call to {\tt CmiWallTimer()} at the appropriate point in the program.

Eg.
\begin{verbatim}
   traceRegisterUserEvent("Critical Code", 20); // on PE 0
   double critStart = CmiWallTimer();;  // start time
   // do the critical code
   traceUserBracketEvent(20, critStart,CmiWallTimer());
\end{verbatim}

\item
{\tt void traceUserSuppliedData(int data) }

This function records a user specified data value at the current
time. This data value can be used to color entry method invocations in
Timeline, see \ref{sec::timeline view}.

\item
{\tt void traceUserSuppliedNote(char * note) }

This function records a user specified text string at the current time.

\item
{\tt void traceUserSuppliedBracketedNote(char *note, int EventNum, double StartTime, double EndTime)}

This function records a user event spanning a time interval from {\tt StartTime} to {\tt EndTime}. Both {\tt StartTime} and {\tt EndTime} should be obtained from a call to {\tt CmiWallTimer()} at the appropriate point in the program.

Additionally, a user supplied text string is recorded, and the  {\tt EventNum} is recorded. These events are therefore displayed with colors determined by the {\tt EventNum}, just as those generated with {\tt traceUserBracketEvent} are.

\end{itemize}

\subsection{User Stats}
\label{sec::user stats}

\charmpp{} allows the user to track the progression of any variable or value throughout the program execution.
These user specified stats can then be plotted in \projections{}, either over time or by processor.
To enable this feature for \charmpp{}, build \charmpp{} with the {\tt --enable-tracing} flag.

Follow these steps to track user stats in a \charmpp{} program:

\begin{enumerate}
\item
Register a stat with an identifying string and a globally unique integer identifier.

\item
Update the value of the stat at points of interest in the code by calling the update stat functions.

\item
Compile program with -tracemode projections flag.
\end{enumerate}

The functions available are as follows:

\begin{itemize}
\item
{\tt int traceRegisterUserStat(const char * StatDesc, int StatNum) }

This function is called once near the beginning the of the \charmpp{} program. {\tt StatDesc} is the identifying
string and {\tt StatNum} is the unique integer identifier.

\item
{\tt void updateStat(int StatNum, double StatValue)}

This function updates the value of a user stat and can be called many times throughout program execution.
{\tt StatNum} is the integer identifier corresponding to the desired stat. {\tt StatValue} is the updated value of the user stat.

\item
{\tt void updateStatPair(int StatNum, double StatValue, double Time)}

This function works similar to {\tt updateStat()}, but also allows the user to store a user specified time for the update. In 
\projections{}, the user can then choose which time scale to use: real time, user specified time, or ordered.

\end{itemize}