So I'm teaching crash-course in CUDA that teaches students how to write good GPU code (CUDA 7.5 in this case). They kernels they will be running will do matrix multiply on 2048x2048 floating point matrices, some kernels involving multiple blocks and shared memory.
The students will be ssh-ing into servers that have the GPUs where they will compile and run their programmes.
I haven't a clue on how feasible this is and the only step I can imagine taking to prevent slowdown is to assign groups of students to different GPUs by telling them what value to pass cudaSetDevice().
Edit for posterities' sake: I ended up not taking any precautions and the lab ran smoothly. Wasn't much of test though, since only half (15 or so) students did the lab simultaneously, and we only got to cover really simple hello-world kernels. Second edit: There were two servers, with two GeForce GTX 650 Tis per server