Issues with LAPACK detection building on Cray XC40

I am trying to compile Chameleon on our Cray XC40. Unfortunately, I am running into issues with the LAPACK detection. From the logs:

$ module swap PrgEnv-cray PrgEnv-intel
$ module load numlib/intel/mkl/2018.1
$ cmake .. -DBLAS_DIR=$MKLROOT
-- The Fortran compiler identification is Intel 18.0.1.20171018
-- The C compiler identification is Intel 18.0.1.20171018
-- The CXX compiler identification is Intel 18.0.1.20171018
-- Cray Programming Environment 2.5.15 Fortran
-- Check for working Fortran compiler: /opt/cray/pe/craype/2.5.15/bin/ftn
-- Check for working Fortran compiler: /opt/cray/pe/craype/2.5.15/bin/ftn  -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /opt/cray/pe/craype/2.5.15/bin/ftn supports Fortran 90
-- Checking whether /opt/cray/pe/craype/2.5.15/bin/ftn supports Fortran 90 -- yes
-- Cray Programming Environment 2.5.15 C
-- Check for working C compiler: /opt/cray/pe/craype/2.5.15/bin/cc
-- Check for working C compiler: /opt/cray/pe/craype/2.5.15/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Cray Programming Environment 2.5.15 CXX
-- Check for working CXX compiler: /opt/cray/pe/craype/2.5.15/bin/CC
-- Check for working CXX compiler: /opt/cray/pe/craype/2.5.15/bin/CC -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Verifying Fortran/C Compiler Compatibility
-- Verifying Fortran/C Compiler Compatibility - Success
-- Found PythonInterp: /usr/bin/python2 (found version "2.7.13") 
-- CHAMELEON_SCHED_STARPU is set to ON: CHAMELEON uses StarPU runtime
   To use CHAMELEON with Quark  runtime: set CHAMELEON_SCHED_QUARK  to ON
   To use CHAMELEON with PaRSEC runtime: set CHAMELEON_SCHED_PARSEC to ON
   (CHAMELEON_SCHED_STARPU will be disabled)
-- CHAMELEON_USE_MPI is set to OFF, turn it ON to use MPI (unsupported by Quark)
-- CHAMELEON_USE_CUDA is set to OFF, turn it ON to use CUDA (unsupported by Quark)
-- CHAMELEON_ENABLE_TRACING is set to OFF, turn it ON to use FxT (with StarPU)
-- CHAMELEON_ENABLE_EXAMPLE is set to ON, turn it OFF to avoid building examples
-- CHAMELEON_ENABLE_TESTING is set to ON, turn it OFF to avoid building testing
-- CHAMELEON_ENABLE_TIMING is set to ON, turn it OFF to avoid building timing
-- CHAMELEON_SIMULATION is set to OFF, turn it ON to use SIMULATION mode (only with StarPU compiled with SimGrid)
-- CHAMELEON_ENABLE_PRUNING_STATS is set to OFF, turn it ON to build pruning statistics
-- Performing Test HAVE_WALL
-- Performing Test HAVE_WALL - Success
-- Performing Test HAVE_WEXTRA
-- Performing Test HAVE_WEXTRA - Success
-- Performing Test HAVE_FALLTHROUGH
-- Performing Test HAVE_FALLTHROUGH - Failed
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE  
-- A cache variable, namely CBLAS_DIR, has been set to specify the install directory of CBLAS
-- A cache variable, namely BLAS_DIR, has been set to specify the install directory of BLAS
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.28") 
-- Try OpenMP C flag = [-qopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-qopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP Fortran flag = [-qopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -qopenmp  
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - not found
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Looking for MKL BLAS: found
-- A library with BLAS API found.
-- BLAS_LIBRARIES /sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_intel_lp64.a;/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_sequential.a;/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_core.a;/usr/lib64/libm.a
-- Looking for cblas_dscal
-- Looking for cblas_dscal - found
-- Looking for cblas_zgemm3m
-- Looking for cblas_zgemm3m - not found
-- Looking for cblas: test with blas succeeds
-- Looking for cblas_dscal
-- Looking for cblas_dscal - found
-- Found CBLAS: /sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_intel_lp64.a;/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_sequential.a;/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_core.a;/usr/lib64/libm.a  
-- A cache variable, namely LAPACKE_DIR, has been set to specify the install directory of LAPACKE
-- A cache variable, namely TMG_DIR, has been set to specify the install directory of TMG
-- A cache variable, namely LAPACK_DIR, has been set to specify the install directory of LAPACK
-- Looking for MKL BLAS: found
-- A library with BLAS API found.
-- BLAS_LIBRARIES /sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_intel_lp64.a;/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_sequential.a;/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_core.a;/usr/lib64/libm.a
-- Looking for Fortran CHEEV
-- Looking for Fortran CHEEV - not found
-- Looking for Fortran cheev
-- Looking for Fortran cheev - not found
-- Looking for Fortran cheev
-- Looking for Fortran cheev - not found
-- Looking for MKL LAPACK: not found
-- Looking for Goto LAPACK: not found
-- Looking for Open LAPACK: not found
-- Looking for IBM ESSL LAPACK: not found
-- Looking for IBM ESSL MT LAPACK: not found
-- Looking for Apple Accelerate LAPACK: not found
-- Looking for NAS LAPACK: not found
-- Looking for Generic LAPACK: not found
CMake Warning at cmake_modules/morse_cmake/modules/find/FindLAPACK.cmake:782 (message):
  BLA_VENDOR has been set to All but LAPACK libraries could not be found or
  check of symbols failed.

  Please indicate where to find LAPACK libraries.  You have three options:

  - Option 1: Provide the installation directory of LAPACK library with cmake
  option: -DLAPACK_DIR=your/path/to/lapack

  - Option 2: Provide the directory where to find BLAS libraries with cmake
  option: -DBLAS_LIBDIR=your/path/to/blas/libs

  - Option 3: Update your environment variable (Linux: LD_LIBRARY_PATH,
  Windows: LIB, Mac: DYLD_LIBRARY_PATH)

  

  To follow libraries detection more precisely you can activate a verbose
  mode with -DLAPACK_VERBOSE=ON at cmake configure.

  You could also specify a BLAS vendor to look for by setting
  -DBLA_VENDOR=blas_vendor_name.

  List of possible BLAS vendor: Goto, ATLAS PhiPACK, CXML, DXML, SunPerf,
  SCSL, SGIMATH, IBMESSL, Intel10_32 (intel mkl v10 32 bit),Intel10_64lp
  (intel mkl v10 64 bit, lp thread model, lp64 model), Intel10_64lp_seq
  (intel mkl v10 64 bit, sequential code, lp64 model),Intel( older versions
  of mkl 32 and 64 bit), ACML, ACML_MP, ACML_GPU, Apple, NAS, Generic
Call Stack (most recent call first):
  cmake_modules/morse_cmake/modules/find/FindTMG.cmake:77 (find_package)
  cmake_modules/morse_cmake/modules/find/FindLAPACKE.cmake:109 (find_package)
  CMakeLists.txt:345 (find_package)


CMake Error at cmake_modules/morse_cmake/modules/find/FindLAPACK.cmake:793 (message):
  A required library with LAPACK API not found.  Please specify library
  location.
Call Stack (most recent call first):
  cmake_modules/morse_cmake/modules/find/FindTMG.cmake:77 (find_package)
  cmake_modules/morse_cmake/modules/find/FindLAPACKE.cmake:109 (find_package)
  CMakeLists.txt:345 (find_package)


-- Configuring incomplete, errors occurred![CMakeError.log](/uploads/f1c4c87d42a8f940df6820261b88efe3/CMakeError.log)

I am attaching the CMake error log. I assume that there is an issue with the link order of the static libraries.

If I try building with support for dynamic libraries (export CRAYPE_LINK_TYPE=dynamic), LAPACK detection succeeds but linking of executables fails as follows:

[ 87%] Linking Fortran executable time_dgetrs_nopiv_tile
CMakeFiles/time_sgeqrf_hqr.dir/time_sgeqrf_hqr.c.o: In function `main':
time_sgeqrf_hqr.c:(.text+0x0): multiple definition of `main'
/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64_lin/for_main.o:for_main.c:(.text+0x0): first defined here
/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64_lin/for_main.o: In function `main':
for_main.c:(.text+0x2a): undefined reference to `MAIN__'
timing/CMakeFiles/time_sgeqrf_hqr.dir/build.make:114: recipe for target 'timing/time_sgeqrf_hqr' failed
make[2]: *** [timing/time_sgeqrf_hqr] Error 1

If I manually replace ftn with cc the linking succeeds. I tried running CMake with -DCMAKE_LINKER=/opt/cray/pe/craype/2.5.15/bin/cc but that seems to be ignored.

Tested on commit 55938b987201c89f007eae3cc321597536096c06.

Please let me know if I can provide any additional information on this.

Hi, no experience either with Chameleon on that machine.

PRUVOST Florent gitlab@gitlab.inria.fr writes:

It seems that the issue is coming from the linker used to test if the BLAS are available or not. I thought the problem had already been seen and fixed. @abuttari was it you who got the problem ? I don't remember. Or maybe @XL?

And I forgot, @schuchart you are using the last revision of the repository with the last revisions of the submodules, right ?

I am not sure what's happening here. must take a look at the log but I cannot find it. If I understand correctly ftn is nothing else than the Intel ifort compiler. In this case you should add the -nofor_main compiler flag. This will fix the second problem (but the first may be linked to it).

I agree with you that should be the fix but I thought it was a problem you already got :). And remains the pb of how to correctly test the link if we do it with the C linker, or the fortran linker.

In our case, since the testing is in C, The fortran linker should not be used. But maybe we need it for one of the library.

Ahh sorry, I seem to have messed up the attachment: CMakeError.log CMakeOutput.log

The ftn compiler is the Cray fortran compiler wrapper that eventually invokes the ifort Intel Fortran compiler. I hope the logs are helpful.

I checked that I am using commit 55938b987201c89f007eae3cc321597536096c06, the submodules are pulled using --recursive passed to git clone. Morse is at f99bada1af8f986d91f651eb9748d0ac6367ef5e, hqr is at 7ad2afcef2901aa04c772369b3aa0deb4da9158b.

Quick update: passing -DCMAKE_EXE_LINKER_FLAGS=-nofor_main to cmake solves the linker issue when building with shared libraries.

@fpruvost Do you think there is something we should do automatically ? Right now I don't really see what, so I would close the issue. Unless there is really something we can fix.

No ftn is an alias, and it might not be intel behind it. So I don't think it is a good idea.

what about the CMAKE_Fortran_COMPILER_ID variable?

if(${CMAKE_Fortran_COMPILER_ID} STREQUAL "Intel") ...

I forgot to answer but yes, we could use that to add the nofor_main option but you still need to be sure that the linker used is the fortran one, and not the C one.

changed milestone to %Chameleon 0.9.2

mentioned in merge request !144 (merged)

Hello @schuchart Sorry for the long delay. Could you try to patch you chameleon version as I done in !144 (merged) to check it detects correctly that ftn is in fact the Intel compiler please. If yes, I'll merge it.Thanks Mathieu

assigned to @faverge

mentioned in commit 96819268

closed via merge request !144 (merged)

mentioned in commit 4ea65e38