[spec] Add checks for removal/addition of GPU

5 jobs for buggpu in 4 minutes and 35 seconds