Skip to content
GitLab
Closed
Created by Ghost User@ghost

Issues with LAPACK detection building on Cray XC40

I am trying to compile Chameleon on our Cray XC40. Unfortunately, I am running into issues with the LAPACK detection. From the logs:

$ module swap PrgEnv-cray PrgEnv-intel
$ module load numlib/intel/mkl/2018.1
$ cmake .. -DBLAS_DIR=$MKLROOT
-- The Fortran compiler identification is Intel 18.0.1.20171018
-- The C compiler identification is Intel 18.0.1.20171018
-- The CXX compiler identification is Intel 18.0.1.20171018
-- Cray Programming Environment 2.5.15 Fortran
-- Check for working Fortran compiler: /opt/cray/pe/craype/2.5.15/bin/ftn
-- Check for working Fortran compiler: /opt/cray/pe/craype/2.5.15/bin/ftn  -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /opt/cray/pe/craype/2.5.15/bin/ftn supports Fortran 90
-- Checking whether /opt/cray/pe/craype/2.5.15/bin/ftn supports Fortran 90 -- yes
-- Cray Programming Environment 2.5.15 C
-- Check for working C compiler: /opt/cray/pe/craype/2.5.15/bin/cc
-- Check for working C compiler: /opt/cray/pe/craype/2.5.15/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Cray Programming Environment 2.5.15 CXX
-- Check for working CXX compiler: /opt/cray/pe/craype/2.5.15/bin/CC
-- Check for working CXX compiler: /opt/cray/pe/craype/2.5.15/bin/CC -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Verifying Fortran/C Compiler Compatibility
-- Verifying Fortran/C Compiler Compatibility - Success
-- Found PythonInterp: /usr/bin/python2 (found version "2.7.13") 
-- CHAMELEON_SCHED_STARPU is set to ON: CHAMELEON uses StarPU runtime
   To use CHAMELEON with Quark  runtime: set CHAMELEON_SCHED_QUARK  to ON
   To use CHAMELEON with PaRSEC runtime: set CHAMELEON_SCHED_PARSEC to ON
   (CHAMELEON_SCHED_STARPU will be disabled)
-- CHAMELEON_USE_MPI is set to OFF, turn it ON to use MPI (unsupported by Quark)
-- CHAMELEON_USE_CUDA is set to OFF, turn it ON to use CUDA (unsupported by Quark)
-- CHAMELEON_ENABLE_TRACING is set to OFF, turn it ON to use FxT (with StarPU)
-- CHAMELEON_ENABLE_EXAMPLE is set to ON, turn it OFF to avoid building examples
-- CHAMELEON_ENABLE_TESTING is set to ON, turn it OFF to avoid building testing
-- CHAMELEON_ENABLE_TIMING is set to ON, turn it OFF to avoid building timing
-- CHAMELEON_SIMULATION is set to OFF, turn it ON to use SIMULATION mode (only with StarPU compiled with SimGrid)
-- CHAMELEON_ENABLE_PRUNING_STATS is set to OFF, turn it ON to build pruning statistics
-- Performing Test HAVE_WALL
-- Performing Test HAVE_WALL - Success
-- Performing Test HAVE_WEXTRA
-- Performing Test HAVE_WEXTRA - Success
-- Performing Test HAVE_FALLTHROUGH
-- Performing Test HAVE_FALLTHROUGH - Failed
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE  
-- A cache variable, namely CBLAS_DIR, has been set to specify the install directory of CBLAS
-- A cache variable, namely BLAS_DIR, has been set to specify the install directory of BLAS
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.28") 
-- Try OpenMP C flag = [-qopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-qopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP Fortran flag = [-qopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -qopenmp  
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - not found
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Looking for MKL BLAS: found
-- A library with BLAS API found.
-- BLAS_LIBRARIES /sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_intel_lp64.a;/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_sequential.a;/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_core.a;/usr/lib64/libm.a
-- Looking for cblas_dscal
-- Looking for cblas_dscal - found
-- Looking for cblas_zgemm3m
-- Looking for cblas_zgemm3m - not found
-- Looking for cblas: test with blas succeeds
-- Looking for cblas_dscal
-- Looking for cblas_dscal - found
-- Found CBLAS: /sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_intel_lp64.a;/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_sequential.a;/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_core.a;/usr/lib64/libm.a  
-- A cache variable, namely LAPACKE_DIR, has been set to specify the install directory of LAPACKE
-- A cache variable, namely TMG_DIR, has been set to specify the install directory of TMG
-- A cache variable, namely LAPACK_DIR, has been set to specify the install directory of LAPACK
-- Looking for MKL BLAS: found
-- A library with BLAS API found.
-- BLAS_LIBRARIES /sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_intel_lp64.a;/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_sequential.a;/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64/libmkl_core.a;/usr/lib64/libm.a
-- Looking for Fortran CHEEV
-- Looking for Fortran CHEEV - not found
-- Looking for Fortran cheev
-- Looking for Fortran cheev - not found
-- Looking for Fortran cheev
-- Looking for Fortran cheev - not found
-- Looking for MKL LAPACK: not found
-- Looking for Goto LAPACK: not found
-- Looking for Open LAPACK: not found
-- Looking for IBM ESSL LAPACK: not found
-- Looking for IBM ESSL MT LAPACK: not found
-- Looking for Apple Accelerate LAPACK: not found
-- Looking for NAS LAPACK: not found
-- Looking for Generic LAPACK: not found
CMake Warning at cmake_modules/morse_cmake/modules/find/FindLAPACK.cmake:782 (message):
  BLA_VENDOR has been set to All but LAPACK libraries could not be found or
  check of symbols failed.

  Please indicate where to find LAPACK libraries.  You have three options:

  - Option 1: Provide the installation directory of LAPACK library with cmake
  option: -DLAPACK_DIR=your/path/to/lapack

  - Option 2: Provide the directory where to find BLAS libraries with cmake
  option: -DBLAS_LIBDIR=your/path/to/blas/libs

  - Option 3: Update your environment variable (Linux: LD_LIBRARY_PATH,
  Windows: LIB, Mac: DYLD_LIBRARY_PATH)

  

  To follow libraries detection more precisely you can activate a verbose
  mode with -DLAPACK_VERBOSE=ON at cmake configure.

  You could also specify a BLAS vendor to look for by setting
  -DBLA_VENDOR=blas_vendor_name.

  List of possible BLAS vendor: Goto, ATLAS PhiPACK, CXML, DXML, SunPerf,
  SCSL, SGIMATH, IBMESSL, Intel10_32 (intel mkl v10 32 bit),Intel10_64lp
  (intel mkl v10 64 bit, lp thread model, lp64 model), Intel10_64lp_seq
  (intel mkl v10 64 bit, sequential code, lp64 model),Intel( older versions
  of mkl 32 and 64 bit), ACML, ACML_MP, ACML_GPU, Apple, NAS, Generic
Call Stack (most recent call first):
  cmake_modules/morse_cmake/modules/find/FindTMG.cmake:77 (find_package)
  cmake_modules/morse_cmake/modules/find/FindLAPACKE.cmake:109 (find_package)
  CMakeLists.txt:345 (find_package)


CMake Error at cmake_modules/morse_cmake/modules/find/FindLAPACK.cmake:793 (message):
  A required library with LAPACK API not found.  Please specify library
  location.
Call Stack (most recent call first):
  cmake_modules/morse_cmake/modules/find/FindTMG.cmake:77 (find_package)
  cmake_modules/morse_cmake/modules/find/FindLAPACKE.cmake:109 (find_package)
  CMakeLists.txt:345 (find_package)


-- Configuring incomplete, errors occurred![CMakeError.log](/uploads/f1c4c87d42a8f940df6820261b88efe3/CMakeError.log)

I am attaching the CMake error log. I assume that there is an issue with the link order of the static libraries.

If I try building with support for dynamic libraries (export CRAYPE_LINK_TYPE=dynamic), LAPACK detection succeeds but linking of executables fails as follows:

[ 87%] Linking Fortran executable time_dgetrs_nopiv_tile
CMakeFiles/time_sgeqrf_hqr.dir/time_sgeqrf_hqr.c.o: In function `main':
time_sgeqrf_hqr.c:(.text+0x0): multiple definition of `main'
/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64_lin/for_main.o:for_main.c:(.text+0x0): first defined here
/sw/hazelhen-cle6/hlrs/compiler/intel/Compiler/18.0.1.163/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64_lin/for_main.o: In function `main':
for_main.c:(.text+0x2a): undefined reference to `MAIN__'
timing/CMakeFiles/time_sgeqrf_hqr.dir/build.make:114: recipe for target 'timing/time_sgeqrf_hqr' failed
make[2]: *** [timing/time_sgeqrf_hqr] Error 1

If I manually replace ftn with cc the linking succeeds. I tried running CMake with -DCMAKE_LINKER=/opt/cray/pe/craype/2.5.15/bin/cc but that seems to be ignored.

Tested on commit 55938b987201c89f007eae3cc321597536096c06.

Please let me know if I can provide any additional information on this.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information