#===============================================================================
# Copyright 2021-2022 Intel Corporation.
#
# This software and the related documents are Intel copyrighted  materials,  and
# your use of  them is  governed by the  express license  under which  they were
# provided to you (License).  Unless the License provides otherwise, you may not
# use, modify, copy, publish, distribute,  disclose or transmit this software or
# the related documents without Intel's prior written permission.
#
# This software and the related documents  are provided as  is,  with no express
# or implied  warranties,  other  than those  that are  expressly stated  in the
# License.
#===============================================================================

#===============================================================================
#
# Intel(R) oneMKL Examples
#===============================================================================

---------------------
 CMake-related Notes
---------------------
* The minimum supported CMake version is 3.13.
* CMAKE_BUILD_TYPE option is locked in "Release" mode for CMake versions below 3.25.2 due to lack of compiler-cmake integration.
* Use CMake version 3.25.2 or newer for full native support of Intel(R) oneAPI DPC++/C++ Compiler.

-------------------
 Examples Contents
-------------------

* c            - examples for the oneMKL C API
* c_mpi        - examples for the oneMKL C Cluster API
* c_offload    - examples for the oneMKL C API with OpenMP Offload
* f            - examples for the oneMKL Fortran API
* f95          - examples for the oneMKL Fortran 95 API
* f_mpi        - examples for the oneMKL Fortran Cluster API
* f_offload    - examples for the oneMKL Fortran API with OpenMP Offload
* sycl         - examples for the oneMKL SYCL API
* sycl_device  - examples for the oneMKL SYCL Device API
* cmake        - CMake scripts
* <MKLROOT>/lib/cmake/mkl/MKLConfig.cmake
               - CMake Config-Mode file for oneMKL
* <MKLROOT>/lib/cmake/mkl/MKLConfigVersion.cmake
               - CMake Config-Mode Version file for oneMKL(helps find the right package version)

NOTE: this list could be limited to what is supported on current OS

-------------------------------
 How to build and run examples
-------------------------------

NOTE: CMake Config-Mode file for oneMKL by default defines MKLROOT as '../../../' 
    and assumes the package structure is '<MKLROOT>/lib/cmake/mkl/MKLConfig.cmake', 
    in case of different package structure $MKLROOT environment variable should be 
    defined by user manually or via oneMKL environment script.

1. Choose the oneMKL API type and change directory to the selected oneMKL API type.

    Example: $> cd $MKLROOT/share/doc/mkl/examples/c

2. Create the build folder and change directory to it.

    Example: $> mkdir build && cd build

3. Call CMake to configure the build and customize build options if needed (see
"Examples build options" below).

    Example: $> cmake .. -G "Ninja" -DCMAKE_Fortran_COMPILER=ifx -DTARGET_DOMAINS=blas

4. Call CMake to build the configured project.
Optional: use the CMake `-j` option to specify the number of parallel jobs.
Optional: use the CMake `--verbose` option to see compilation/link lines for each example.

    Example: $> cmake --build . -j 24 --verbose

5. Run examples using the CTest tool.
Optional: use the CTest `--verbose` option to see examples output.

    Example: $> ctest --verbose

------------------------
 Examples build options
------------------------
CMake Generator option `-G "<value>"`
Supported CMake Generators on Linux   : Ninja, Unix Makefiles
Supported CMake Generators on Windows : Ninja, NMake Makefiles
NOTE: The default Windows generator is NOT supported. The Windows CMake Generator option must be provided.

All build options should be defined as `-D<option>=<value>` at the CMake configuration
step (see #3 in "How to build and run examples" above).

+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| Option                   | Description             | Supported                       | Default      | MKLConfig    |
|                          |                         | values                          | value        | option       |
+==========================+=========================+=================================+==============+==============+
| CMAKE_C_COMPILER         | Define C compiler       | lnx: icc gcc clang icx          | icx          | No           |
|                          |                         | win: icl cl icx                 |              |              |
|                          |                         | C MPI wrappers: mpicc or mpi*   |              |              |
|                          +-------------------------+---------------------------------+--------------+--------------+
|                          | OpenMP Offload API      | icx                             | icx          | No           |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| CMAKE_Fortran_COMPILER   | Define Fortran compiler | lnx: ifort ifx gfortran         | ifx          | No           |
|                          |                         | win: ifort ifx                  |              |              |
|                          |                         | Fortran MPI wrapper mpiifort    |              |              |
|                          +-------------------------+---------------------------------+--------------+              |
|                          | OpenMP Offload API      | ifx                             | ifx          |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| CMAKE_CXX_COMPILER       | Define C++ compiler     | lnx: icpx                       | lnx: icpx    | No           |
|                          | (DPC++ examples only)   | win: icx                        | win: icx     |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| MKL_DIR                  | Specify path to         | Valid full path to              | Search       | No           |
|                          | MKLConfig.cmake         | MKLConfig.cmake                 | Paths**      |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| MKL_LINK                 | Define oneMKL link type | static dynamic sdl              | dynamic      | Yes          |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| MKL_INTERFACE            | Define Integer size for | ilp64 lp64                      | ilp64        | Yes          |
|                          | C/Fortran API           |                                 |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| MKL_THREADING            | Define oneMKL threading | sequential tbb_thread           | intel_thread | Yes          |
|                          | for C/Fortran API       | intel_thread gnu_thread         |              |              |
|                          +-------------------------+---------------------------------+--------------+              |
|                          | OpenMP Offload API      | sequential tbb_thread           | intel_thread |              |
|                          |                         | intel_thread                    |              |              |
|                          +-------------------------+---------------------------------+--------------+              |
|                          | DPC++ API               | sequential tbb_thread           | tbb_thread   |              |
|                          |                         | intel_thread                    |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| MKL_MPI                  | Define MPI vendor       | lnx: intelmpi openmpi mpich     | intelmpi     | Yes          |
|                          | (Cluster examples only) | win: intelmpi mshpc msmpi       |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| TARGET_DEVICES           | Define list of devices  | cpu gpu                         | cpu;gpu      | No           |
|                          | (DPC++ examples only)   |                                 |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| TARGET_DOMAINS           | Define list of domains  | Any domains listed in           | All          | No           |
|                          | to build and run        | <oneMKL API type> folder        |              |              |
|                          |                         | E: TARGET_DOMAINS="blas vml"    |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| TARGET_FUNCTIONS         | Define list of functions| Any function listed in          | All          | No           |
|                          | to build and run as     | <domain>/<domain>.lst           |              |              |
|                          | <domain>/<function>     | E: TARGET_FUNCTIONS="vml/vdsin" |              |              |
|                          | TARGET_DOMAINS  is      |                                 |              |              |
|                          | ignored.                | The regex used is the CMake     |              |              |
|                          | Regex expressions* can  | standard.                       |              |              |
|                          | be used to identify     | E: TARGET_FUNCTIONS="vml/vd.*"  |              |              |
|                          | specific sets of        |   will run all vml examples     |              |              |
|                          | functions to run.       |    starting with "vd"           |              |              |
|                          |                         |    using wildcard ".*"          |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| TARGET_OFFLOAD_PRECISION | Define list of examples | sp (for single precision)       | All          | No           |
|                          | to build and run based  | dp (for double precision)       |              |              |
|                          | on precision. Applies   |                                 |              |              |
|                          | only to c/f_offload     |                                 |              |              |
|                          | examples and to domains |                                 |              |              |
|                          | that have separate lists|                                 |              |              |
|                          | <domain>_[sp|dp].lst    |                                 |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| FAIL_ON_MISSING_DEVICES  | Enable failing of test  | yes no                          | No           | No           |
|                          | if defined device in    |                                 |              |              |
|                          | TARGET_DEVICES variable |                                 |              |              |
|                          | cannot be found while   |                                 |              |              |
|                          | examples are running.   |                                 |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+

* See https://cmake.org/cmake/help/latest/command/string.html#regex-specification for the regex 
specification
** Note: https://cmake.org/cmake/help/latest/command/find_package.html#id5
