Check return value of mallocs in check_zmatrices() and prevent overflows

In the same vein of !330 (merged), an overflow in the mallocs in the function to check the result of computations was leading to a negative value and malloc returning NULL.

This MR adds a check of returned values by mallocs (not sure if returning CHAMELEON_ERR_OUT_OF_RESOURCES is the best solution in case of error) and use longs instead of ints for variables used to compute the memory malloc has to allocate.

I found this bug with the following command:

./testing/chameleon_stesting -o lacpy --n 3200:60000:6400 -H -c --nowarmup

The execution is very long for the matrix size 48000 and actually ends with a segfault (in the BLAS library!) when performing the check.