Assertion failed in cti_handle_get() in cham_tile_interface.h

Hello, How did you get this ? What is the command line ? Because I don't understand how you can get this issue and why @ahourcau would be involved, since this is not the mixed precision code.

Hi Mathieu,

I used an example program I wrote, see in attachment nosv_starpu_dgemm.c.

I am running it with command line args: 1000 1000 1000 N N. (The program was to test BSC's nOS-V hypervisor support in StarPU, but you can also execute it without nOS-V).

A git blame on the cham_tile_interface.h lists Ana as the last contributor on the cti_handle_get() routine:

84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 57) static inline CHAM_tile_t *
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 58) cti_handle_get( starpu_data_handle_t handle )
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 59) {
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 60)     starpu_cham_tile_interface_t *cham_tile_interface = (starpu_cham_tile_interface_t *)
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 61)         starpu_data_get_interface_on_node( handle, STARPU_MAIN_RAM );
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 62) 
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 63) #ifdef STARPU_DEBUG
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 64)     STARPU_ASSERT_MSG( cham_tile_interface->id == STARPU_CHAM_TILE_INTERFACE_ID,
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 65)                        "Error. The given data is not a cham_tile." );
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 66) #endif
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 67) 
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 68)     return &(cham_tile_interface->tile);
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 69) }
84ba24fbf (Ana Hourcau     2024-06-03 16:51:45 +0200 70)

Best regards,

Olivier

changed the description

Hello,

That weird, starpu register the interface before calling starpu_data_register_new_data :

void starpu_data_register(starpu_data_handle_t *handleptr, int home_node, void *data_interface, struct starpu_data_interface_ops *ops)
{
	STARPU_ASSERT_MSG(home_node >= -1 && home_node < (int)starpu_memory_nodes_get_count(), "Invalid memory node number");
	starpu_data_handle_t handle = _starpu_data_handle_allocate(ops, home_node);

	STARPU_ASSERT(handleptr);
	*handleptr = handle;

	if (ops->interfaceid == STARPU_UNKNOWN_INTERFACE_ID)
	{
		ops->interfaceid = starpu_data_interface_get_next_id();
	}

	/* fill the interface fields with the appropriate method */
	STARPU_ASSERT(ops->register_data_handle);
	ops->register_data_handle(handle, home_node, data_interface);

	_starpu_data_register_ops(ops);

	_starpu_register_new_data(handle, home_node, 0);
	_STARPU_TRACE_HANDLE_DATA_REGISTER(handle);
}

And furthermore the interface_id is initialized in the call to init through a call to starpu_cham_tile_interface_fini() so I really don't get why you would get this issue.

I just ran a command that should be equivalent to what you have in debug mode and I have no assert. So it seems to be difficult to reproduce outside your code:

% ./chameleon_dtesting -o gemm --api 1 --check -n 1000
Id;Function;threads;gpus;P;Q;mtxfmt;nb;transA;transB;m;n;k;lda;ldb;ldc;alpha;beta;seedA;seedB;seedC;tsub;time;gflops;||A||;||B||;||C||;||R||;RETURN
0;dgemm;3;0;1;1;0;384;111;111;1000;1000;1000;1000;1000;1000;4.892778e-01;-1.846424e-01;1649760492;596516649;1189641421;0.000000e+00;1.528008e-02;1.308894e+02;2.636825e+02;2.655066e+02;5.000000e-01;2.664535e-15;SUCCESS

I will prepare a minimal example for you to try.

In the mean time, here are the configure flags that I used for StarPU, as this may have an impact:

--disable-blocking-drivers --enable-static --disable-fast --enable-debug --disable-perf-debug --disable-verbose --with-fxt --enable-shared --disable-cuda --disable-opencl --disable-socl --enable-mpi --enable-mpi-check --disable-starpufft --disable-build-doc --enable-openmp --enable-fortran --enable-starpurm --disable-starpurm-examples --enable-build-tests --enable-build-examples --disable-mlr --enable-nosv --disable-hdf5 --disable-valgrind --enable-blas-lib=mkl --enable-maxcpus=128

Here is a minimal example that still trigger the error. cham_dgemm.tar

Here are the modules loaded on the machine:

module list
Currently Loaded Modulefiles:
  1) tools/git/2.36.0
  2) build/ac269-am114-lt246-m41417/ac269-am114-lt246-m41417
  3) compiler/gcc/11.2.0
  4) hardware/hwloc/2.11.0
  5) linalg/mkl/2020_update4
  6) build/cmake/3.27.0

I have tweaked the StarPU configure flags to shorten the list: --disable-fast --enable-debug

Here are Chameleon's CMake flags: -DCHAMELEON_USE_CUDA=OFF -DCHAMELEON_USE_MPI=OFF -DCHAMELEON_SIMULATION=OFF -DCHAMELEON_SCHED_STARPU=ON -DCHAMELEON_ENABLE_EXAMPLE=OFF -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_Fortran_COMPILER=gfortran -DCMAKE_BUILD_TYPE=Debug

Using GDB, observe the following unexpected sequence:

(gdb) break cti_init
Breakpoint 1 at 0x40c59a: file /home/aumage/Git/Chameleon/chameleon.gitlab/runtime/starpu/interface/cham_tile_interface.c, line 115.
(gdb) break starpu_cham_tile_interface_init
Breakpoint 2 at 0x40d82b: file /home/aumage/Git/Chameleon/chameleon.gitlab/runtime/starpu/interface/cham_tile_interface.c, line 768.
(gdb) r
Starting program: /home/aumage/Git/StarPU/Example_Cham_dgemm/starpu_dgemm 

[...]

Thread 1 "starpu_dgemm" hit Breakpoint 1, cti_init (data_interface=0x1e8ee70) at /home/aumage/Git/Chameleon/chameleon.gitlab/runtime/starpu/interface/cham_tile_interface.c:115
115	    starpu_cham_tile_interface_t *cham_tile_interface = data_interface;
(gdb) n
116	    cham_tile_interface->id = STARPU_CHAM_TILE_INTERFACE_ID;
(gdb) n
117	    cham_tile_interface->allocsize = -1;
(gdb) p cham_tile_interface->id
$1 = STARPU_UNKNOWN_INTERFACE_ID

So, the id assigned to the tile interface is incorrect, and also, breakpoint 2 on starpu_cham_tile_interface_init() is not triggered.

Assertion failed in cti_handle_get() in cham_tile_interface.h

Designs

Child items ...

Activity