Re: NVRM Xid 26 error
Might be either a driver bug or a hardware fault. If you want nvidia devteam to be able top help you please be kind enough to provide complete details. Most HW/SW configuration details are available in nvidia-bug-report log, but you hadn't shed any light on the exact GPU usage pattern. From your answer I assume that you're doing some kind of parallel compiting using both multicore CPU and GPU. This leads us to question about what low level lib do you use to compute things on GPU? Is it Cuda or OpenCL? Next, AFAIRC, MPI is a high-level standard that had been implemented in a lot of different libs from different vendors. Which one do you use? How does the MPI lib you use gain access to GPU hardware? Is it using Cuda and/or OpenCL directly or passing through some middleware?
Your best bet would be to try to reproduce the GPU hang you experience and write specialized small and separate testcase for it. It would help nVIDIA devteam greatly with reproducing and debugging the problem.