I have the same issue on my system and am currently working on a solution. I have been posting results of my tests to this thread:
I am curious. What nvidia GPU do you have? What variety of Linux are you running?
I am not able to play audio when X is stopped. aplay -l returns "aplay: device_list:215: no soundcards found...". I wonder how you are able to play audio with X stopped?
For me restarting alsa does not restore the HDMI audio. I have to reboot the system.