% $Id$
%
% Earth System Modeling Framework
% Copyright (c) 2002-2023, University Corporation for Atmospheric Research, 
% Massachusetts Institute of Technology, Geophysical Fluid Dynamics 
% Laboratory, University of Michigan, National Centers for Environmental 
% Prediction, Los Alamos National Laboratory, Argonne National Laboratory, 
% NASA Goddard Space Flight Center.
% Licensed under the University of Illinois-NCSA License.

%\subsection{Description}
\subsubsection{Profiling}
\label{sec:Profiling}

ESMF's built in {\em profiling} capability collects runtime statistics
of an executing ESMF application through both automatic and manual code
instrumentation. Timing information for all phases of all ESMF components
executing in an application can be automatically collected using the
{\tt ESMF\_RUNTIME\_PROFILE} environment variable (see below for settings).
Additionally, arbitrary user-defined code regions can be timed by
manually instrumenting code with special API calls.  Timing profiles
of component phases and user-defined regions can be output in several
different formats:
\begin{itemize}
\item in text at the end of ESMF Log files
\item in separate text file, one per PET (if the ESMF Logs are turned off)
\item in a single summary text file that aggregates timings over multiple PETs
\item in a binary format for import into the \htmladdnormallink{esmf-profiler}{https://github.com/esmf-org/esmf-profiler} for profile visualization
\end{itemize}

The following table lists important environment variables that control
aspects of ESMF profiling.

\begin{tabular} {|p{6cm}|p{8cm}|p{6cm}|p{6cm}|}
     \hline\hline
     {\bf Environment Variable} & {\bf Description} & {\bf Example Values} & {\bf Default}\\
     \hline\hline
     {\tt ESMF\_RUNTIME\_PROFILE} & Enable/disables all profiling functions & {\tt ON} or {\tt OFF} & {\tt OFF} \\
     \hline\hline
     {\tt ESMF\_RUNTIME\_PROFILE\_PETLIST} & Limits profiling to an explicit list of PETs & ``{\tt 0-9 50 99}'' & {\em profile all PETs}\\
     \hline\hline
     {\tt ESMF\_RUNTIME\_PROFILE\_OUTPUT} & Controls output format of profiles;  multiple can be specified in a space separated list & {\tt TEXT}, {\tt SUMMARY}, {\tt BINARY} & {\tt TEXT} \\
     \hline\hline
\end{tabular}


\subsubsection{Tracing}
\label{sec:Tracing}

Whereas profiling collects summary information from an application,
{\em tracing} records a more detailed set of events for later analysis. Trace
analysis can be used to understand what happened during a program's
execution and is often used for diagnosing problems, debugging, and
performance analysis.

ESMF has a built-in tracing capability that records events into special
binary log files.  Unlike log files written by the {\tt ESMF\_Log} class,
which are primarily for human consumption (see Section \ref{sec:Log}),
the trace output files are
recorded in a compact binary representation and are processed by tools
to produce various analyses. ESMF event streams are recorded in the
\htmladdnormallink{Common Trace Format}{http://diamon.org/ctf/} (CTF).
CTF traces include one or more event streams,
as well as a metadata file describing the events in the streams.

Several tools are available for reading in the CTF traces output by ESMF.
Of the tools listed below, the first one is designed specifically for
analyzing ESMF applications and the second two are general purpose tools
for working with all CTF traces.
\begin{itemize}
\item \htmladdnormallink{esmf-profiler}{https://github.com/esmf-org/esmf-profiler}
  is a tool that ingests traces from an ESMF application and generates
  performance profile plots.
\item \htmladdnormallink{TraceCompass}{http://tracecompass.org/}
  is a general purpose tool for reading, analyzing, and visualizing traces.
\item \htmladdnormallink{Babeltrace}{http://www.efficios.com/babeltrace}
  is a command-line tool and library for trace conversion
  that can read and write CTF traces. Python bindings are available
  to open CTF traces are iterate through events.  
\end{itemize}

Events that can be captured by the ESMF tracer include the following. Events
are recorded with a high-precision timestamp to allow timing analyses.
\begin{description}
\item [phase\_enter] indicates entry into an initialize, run, or finalize ESMF component routine
\item [phase\_exit] indicates exit from an initialize, run, or finalize ESMF component routine
\item [region\_enter] indicates entry into a user-defined code region
\item [region\_exit] indicates exit from a user-defined code region
\end{description}

The following table lists important environment variables that control
aspects of ESMF tracing.

\begin{tabular} {|p{6cm}|p{8cm}|p{6cm}|p{6cm}|}
     \hline\hline
     {\bf Environment Variable} & {\bf Description} & {\bf Example Values} & {\bf Default}\\
     \hline\hline
     {\tt ESMF\_RUNTIME\_TRACE} & Enable/disables all tracing functions & {\tt ON} or {\tt OFF} & {\tt OFF} \\
     \hline\hline                    
     {\tt ESMF\_RUNTIME\_TRACE\_CLOCK} & Sets the type of clock for timestamping events (see Section \ref{sec:TracingClocks}). & {\tt REALTIME} or {\tt MONOTONIC} or {\tt MONOTONIC\_SYNC} & {\tt REALTIME}\\
     \hline\hline
     {\tt ESMF\_RUNTIME\_TRACE\_PETLIST} & Limits tracing to an explicit list of PETs & ``{\tt 0-9 50 99}'' & {\em trace all PETs}\\
     \hline\hline
     {\tt ESMF\_RUNTIME\_TRACE\_COMPONENT} & Enables/disable tracing of Component phase\_enter and phase\_exit events & {\tt ON} or {\tt OFF} & {\tt ON} \\
     \hline\hline
     {\tt ESMF\_RUNTIME\_TRACE\_FLUSH} & Controls frequency of event stream flushing to file & {\tt DEFAULT} or {\tt EAGER} & {\tt DEFAULT} \\
     \hline\hline
\end{tabular}



