Adobe Flash Player 126.96.36.199 (a.k.a. 11.0 RC1 from 06/09/2011) 32bit, FireFox 6.0.2 32bit (official linux binary from Mozilla.com), GTS-250 GPU with 1024MB VRAM, nVIDIA driver v.280.13.
Having the following in the mms.cfg (using number and not verbose boolean values is recommended in the official Adobe's Flash Player administrators guide):
EnableLinuxHWVideoDecode = 1
OverrideGPUValidation = 1
With such setup and having "Enable GPU acceleration" checkbox checked in the "Settings..." menu (one may access it by right-clicking on flash applet and selecting "Settings..." menu entry - it might be named slightly different as I have to translate it to English from the system's default locale) there's a number of issues I list below:
* Looks like Adobe had decided to disable OpenGL and/or presentation part of VDPAU with 11.x flash player series. As a result the "best" I can get when playing videos on youtube is "Software video rendering, Accelerated video decoding" and jerky performance when playing fullscreen 1080p. Performance was way better with version 10.3 of flash player and I would call how it was "almost ideal" unless that annoying "overlay" problems that are all well-know to the thread posters.
* "Molehil" acceleration doesn't work as expected. Using testcase at http://labs.adobe.com/technologies/f.../molehill.html
I get about 150% CPU usage while it should be around 3-5% in case "Molehil" were working correctly using OpenGL to accelearate animation. "Zombies" techdemo is also extremely slow and CPU hungry. Unfortunately I had missed times when flash player incubation 11.0 d0 version had been offered for downloads from Adobe Labs so I can't test "Molehil" techdemos with it. There were reports on forums that acceleration had been working as expected with that version of flash player.
* Enabling VDPAU for flash player results in periodical "freezes" of the entire X server followed by quick black "flash" across the entire screen and lines like following written to the kernel log (dmesg):
[765362.445808] NVRM: Xid (0000:02:00): 13, 0001 00000000 00005097 000015e0 00000000 00000080
[767129.336102] NVRM: Xid (0000:02:00): 8, Channel 00000003
[767131.336076] NVRM: os_schedule: Attempted to yield the CPU while in atomic or interrupt context
[767141.345079] NVRM: Xid (0000:02:00): 13, 0001 00000000 00005097 000015e0 00000000 00000080
[767275.622086] NVRM: Xid (0000:02:00): 8, Channel 00000003
[767277.622055] NVRM: os_schedule: Attempted to yield the CPU while in atomic or interrupt context
[767287.632145] NVRM: Xid (0000:02:00): 13, 0001 00000000 00005097 000015e0 00000000 00000080
It seems to me that flash player does something strange with how it interacts with libvdpau leading to HW hang followed by automatic HW recovery after the expiry of GPU hang watchdog timer. Disabling usage of vdpau by flash player either by editing mms.cfg or by unchecking "Enable GPU acceleration" checkbox workarounds this problem.
To summarize the above: with 11.0 RC1 using flash player with VDPAU had become more stable than it was in prior versions but this stability was achieved not by really fixing the bugs but rather by disabling output acceleration through GLX thus skipping renderpaths that triggered the bugs. While I beleive that most of the crappy code is inside flash player there are obviously things to fix on nVIDIA side either. The fact that userland app is able to bring the system - X server part of it - into totally unusable state (and - under some circumstances - it even may lead to a hard hang at kernel level and/or kernel panic) by messing with HW through VDPAU interface isn't acceptable. Userland interfaces providing access to the HW features must be foolproof, it would be a huge security risk otherwise.