CUDA tests failing due to high CPU usage on build system
Right now the CUDA tests are failing with OOM not due to any flaw in the test, but because someone else (yang, whoever that is) is heavily utilizing the GPUs on TitanV.
One of my other tasks actually is to look into configuring job scheduling on this server as well as Kepler so that we can use them fairly. Looks like that's more urgent for me now.
It might also be good to put some checks for this in the pipeline, somehow. Like skip GPU tests if the GPU is already too heavily utilized, or add additional GPU builders that can be switched to when another isn't available (I'm not sure how to test for availability though, other than maybe to implement a separate watch script that enables/disables builders depending on their GPU availability--yes, I think that might work...)