% $Id$
%
% Earth System Modeling Framework
% Copyright (c) 2002-2023, University Corporation for Atmospheric Research, 
% Massachusetts Institute of Technology, Geophysical Fluid Dynamics 
% Laboratory, University of Michigan, National Centers for Environmental 
% Prediction, Los Alamos National Laboratory, Argonne National Laboratory, 
% NASA Goddard Space Flight Center.
% Licensed under the University of Illinois-NCSA License.

\subsubsection{User-code {\tt SetServices} method}

Many programs call some library routines.  The library
documentation must explain what the routine name is, what arguments 
are required and what are optional, and what the code does.  

In contrast, all ESMF components must be written to {\it be called}
by another part of the program; in effect, an ESMF component takes the 
place of a library.  The interface is prescribed by the framework,
and the component writer must provide specific subroutines which 
have standard argument lists and perform specific operations.
For technical reasons {\em none} of the arguments in user-provided subroutines
must be declared as {\em optional}.

The only {\em required} public interface of a Component is its
SetServices method.  This subroutine must have an
externally accessible name (be a public symbol), take a component
as the first argument, and an integer return code as the second. 
Both arguments are required and must {\em not} be declared as 
{\tt optional}. If an intent is specified in the interface it must be 
{\tt intent(inout)} for the first and {\tt intent(out)} for the 
second argument. The subroutine name is not predefined, it is set by the
component writer, but must be provided as part of the component 
documentation.

The required function that the SetServices subroutine must provide is to
specify the user-code entry points for the standard ESMF Component methods. To
this end the user-written SetServices routine calls the 

{\tt ESMF\_<Grid/Cpl>CompSetEntryPoint()} method to set each 
Component entry point.

See sections \ref{sec:GridSetServ} and \ref{sec:CplSetServ} for examples of
how to write a user-code SetServices routine.

Note that a component does not call its own SetServices routine;
the AppDriver or parent component code, which is creating a component, 
will first call {\tt ESMF\_<Grid/Cpl>CompCreate()} to create a component object, and then must call into {\tt ESMF\_<Grid/Cpl>CompSetServices()}, supplying the user-code SetServices routine as an argument. The framework then calls into the user-code SetServices, after the Component's VM has been started up.

It is good practice to package the user-code implementing a component into a Fortran module, with the user-code SetService routine being the only public module method. ESMF supports three mechanisms for accessing the user-code SetServices routine from the calling AppDriver or parent component.

\begin{itemize}
\item {\bf Fortran USE association}: The AppDriver or parent component utilizes the standard Fortran USE statement on the component module to make all public entities available. The user-code SetServices routine can then be passed directly into the {\tt ESMF\_<Grid/Cpl>CompSetServices()} interface documented in \ref{GridComp:SetServices} and \ref{CplComp:SetServices}, respectively.

{\em Pros}: Standard Fortran module use: name mangling and interface checking is handled by the Fortran compiler.

{\em Cons}: Fortran 90/95 has no mechanism to implement a "smart" dependency scheme through USE association. Any change in a lower level component module (even just adding or changing a comment!) will trigger a complete recompilation of all of the higher level components throughout the component hierarchy. This situation is particularly annoying for ESMF componentized code, where the prescribed ESMF component interfaces, in principle, remove all interdependencies between components that would require recompilation.

Fortran {\em submodules}, introduced as an extension to Fortran 2003, and now part for the Fortran 2008 standard, are designed to avoid this "false" dependency issue. A code change to an ESMF component that keeps the actual implementation within a submodule, will not trigger a recompilation of the components further up in the component hierarchy. Unfortunately, as of mid-2015, only two compiler vendors support submodules.

\item {\bf External routine}: The AppDriver or parent component provides an explicit interface block for an external routine that implements (or calls) the user-code SetServices routine. This routine can then be passed directly into the {\tt ESMF\_<Grid/Cpl>CompSetServices()} interface documented in \ref{GridComp:SetServices} and \ref{CplComp:SetServices}, respectively. (In practice this can be implemented by the component as an external subroutine that simply calls into the user-code SetServices module routine.)

{\em Pros}: Avoids Fortran USE dependencies: a change to lower level component code will not trigger a complete recompilation of all of the higher level components throughout the component hierarchy. Name mangling is handled by the Fortran compiler.

{\em Cons}: The user-code SetServices interface is not checked by the compiler. The user must ensure uniqueness of the external routine name across the entire application.

\begin{sloppypar}
\item {\bf Name lookup}: The AppDriver or parent component specifies the user-code SetServices routine by name. The actual lookup and code association does not occur until runtime. The name string is passed into the {\tt ESMF\_<Grid/Cpl>CompSetServices()} interface documented in \ref{GridComp:SetServicesShObj} and \ref{CplComp:SetServicesShObj}, respectively.

{\em Pros}: Avoids Fortran USE dependencies: a change to lower level component code will not trigger a complete recompilation of all of the higher level components throughout the component hierarchy. The component code does not have to be accessible until runtime and may be located in a shared object, thus avoiding relinking of the application.

{\em Cons}: The user-code SetServices interface is not checked by the compiler. The user must explicitly deal with all of the Fortran name mangling issues: 1) Accessing a module routine requires precise knowledge of the name mangling rules of the specific compiler. Alternatively, the user-code SetServices routine may be implemented as an external routine, avoiding the module name mangling. 2) Even then, Fortran compilers typically append one or two underscores on a symbol name. This must be considered when passing the name into the {\tt ESMF\_<Grid/Cpl>CompSetServices()} method.

\end{sloppypar}
\end{itemize}

\subsubsection{User-code {\tt Initialize}, {\tt Run}, and {\tt Finalize} methods}

The required standard ESMF Component methods, for which user-code entry
points must be set, are Initialize, Run, and Finalize. Currently optional,
a Component may also set entry points for the WriteRestart and
ReadRestart methods.

Sections \ref{sec:GridSetServ} and \ref{sec:CplSetServ} provide examples
of how the entry points for Initialize, Run, and Finalize are set during
the user-code SetServices routine, using the 
{\tt ESMF\_<Grid/Cpl>CompSetEntryPoint()} library call.

All standard user-code methods must abide {\em exactly} to the prescribed
interfaces. {\em None} of the arguments must be declared as {\em optional}.

The names of the Initialize, Run, and Finalize user-code subroutines do
not need to be public; in fact it is far better for them to be private to
lower the chances of public symbol clashes between different components.

See sections \ref{sec:GridInitialize}, \ref{sec:GridRun},
\ref{sec:GridFinalize}, and \ref{sec:CplInitialize}, \ref{sec:CplRun},
\ref{sec:CplFinalize} for examples of how to write entry points for the 
standard ESMF Component methods.


\subsubsection{User-code {\tt SetVM} method}
\label{sec:AppDriverSetVM}

When the AppDriver or parent component code calls
{\tt ESMF\_<Grid/Cpl>CompCreate()} it has the option to specify a 
{\tt petList} argument. All of the parent PETs contained in this list become
resources of the child component. By default, without the {\tt petList} argument, all of the parent PETs are provided to the child component.

Typically each component has its own virtual machine (VM) object. However, using the optional {\tt contextflag} argument during {\tt ESMF\_<Grid/Cpl>CompCreate()} a child component can inherit its parent component's VM. Unless a child component inherits the parent VM, it has the option to set certain aspects of how its VM utilizes the provided resources. The resources provided via the parent PETs are the associated processing elements (PEs) and virtual address spaces (VASs).

The optional user-written SetVM routine is called from the parent for the child through the {\tt ESMF\_<Grid/Cpl>CompSetVM()} method. This is the only place where the child component can set aspects of its own VM before it is started up. The child component's VM must be running before the SetServices routine can be called, and thus the parent must call the optional {\tt ESMF\_<Grid/Cpl>CompSetVM()} method {\em before} {\tt ESMF\_<Grid/Cpl>CompSetServices()}.

Inside the user-code called by the SetVM routine, the component has the option to specify how the PETs share the provided parent PEs. Further, PETs on the same single system image (SSI) can be set to run multi-threaded within a reduced number of virtual address spaces (VAS), allowing a component to leverage shared memory concepts.

Sections \ref{sec:GridSetVM} and \ref{sec:CplSetVM} provide examples for
simple user-written SetVM routines.

One common use of the SetVM approach is to implement hybrid parallelism based on MPI+OpenMP. Under ESMF, each component can use its own hybrid parallelism implementation. Different components, even if running on the same PE resources, do not have to agree on the number of MPI processes (i.e. PETs), or the number of OpenMP threads launched under each PET. Hybrid and non-hybrid components can be mixed within the same application. Coupling between components of any flavor is supported under ESMF.

In order to obtain best performance when using SetVM based resource control for hybrid parallelism, it is {\em strongly recommended} to set {\tt OMP\_WAIT\_POLICY=PASSIVE} in the environment. This is one of the standard OpenMP environment variables. The {\tt PASSIVE} setting ensures that OpenMP threads relinquish the PEs as soon as they have completed their work. Without that setting ESMF resource control threads can be delayed, and context switching between components becomes more expensive.

\subsubsection{Use of {\tt internal procedures} as user-provided procedures}
\label{sec:AppDriverIntProc}

Internal procedures are nested within a surrounding procedure, and only local to the surrounding procedure.
They are specified by using the CONTAINS statement.

Prior to Fortran-2008 an internal procedure could not be used as a user-provided callback procedure.
In Fortran-2008 this restriction was lifted.  It is important to note that if ESMF is passed an internal
procedure, that the surrounding procedure be active whenever ESMF calls it.  This helps ensure that
local variables at the surrounding procedures scope are properly initialized.

When internal procedures contained within a main program unit are used for callbacks, there is no problem.
This is because the main program unit is always active.  However when internal procedures are used within
other program units, initialization could become a problem.  The following outlines the issue:

\begin{verbatim}
  module my_procs_mod
    use ESMF
    implicit none

  contains

    subroutine my_procs (...)
      integer :: my_setting
      :
      call ESMF_GridCompSetEntryPoint(gridcomp, methodflag=ESMF_METHOD_INITIALIZE, &
          userRoutine=my_grid_proc_init, rc=localrc)
      :
      my_setting = 42

    contains

      subroutine my_grid_proc_init (gridcomp, importState, exportState, clock, rc)
        :
      ! my_setting is possibly uninitialized when my_grid_proc_init is used as a call-back
        something = my_setting
        :
      end subroutine my_grid_proc_init
    end subroutine my_procs
  end module my_procs_mod
\end{verbatim}

The Fortran standard does not specify whether variable {\it my\_setting} is statically or
automatically allocated, unless it is explicitly given the SAVE attribute.  Thus there is no
guarantee that its value will persist after {\it my\_procs} has finished.  The SAVE attribute
is usually given to a variable via specifying a SAVE attribute in its delaration.  However it can
also be inferred by initializing the variable in its declaration:

\begin{verbatim}
      :
      integer, save : my_setting
      :
\end{verbatim}

or,

\begin{verbatim}
      :
      integer :: my_setting = 42
      :
\end{verbatim}

Because of the potential initialization issues, it is recommended that internal procedures
only be used as ESMF callbacks when the surrounding procedure is also active.

