VDPAU initialization

[crisalide]

Hi,

actually (with 180.22) if i try to start a second vdpau session (e.g. running a second player), vdpau hangs somewhere in init process (don't remember where but it doesn't matter).
It would be usefull to have vdpau returning an error asap if it has reach its sessions limit, so a player could fallback to e.g. xv.

[crisalide]

Nobody?

[Stephen Warren]

Sorry, didn't see this thread. It is expected that an error will be returned, not a hang.

Can you get a backtrace using gdb, and we'll see if we can work out what's up. Also, an nvidia bug report for the system at the time of problem.

[crisalide]

Starting mplayer while xine-vdpau is running (same with second xine-vdpau player):

VDPAU capture: Enabled                                                   
vdp_imp_device_create_x11(0x91ff478, 0, -, -)                            
    -> 0, 1, 0xb4f1a0d0                                                  
vdp_get_proc_address(1, 5, -)                                            
    -> 0, 0xb4f09220                                                     
vdp_get_proc_address(1, 9, -)                                            
    -> 0, 0xb4f0bc60                                                     
vdp_get_proc_address(1, 10, -)                                           
    -> 0, 0xb4f09150                                                     
vdp_get_proc_address(1, 13, -)                                           
    -> 0, 0xb4f10bb0                                                     
vdp_get_proc_address(1, 12, -)                                           
    -> 0, 0xb4f105d0                                                     
vdp_get_proc_address(1, 24, -)                                           
    -> 0, 0xb4f111c0                                                     
vdp_get_proc_address(1, 22, -)                                           
    -> 0, 0xb4f0f570                                                     
vdp_get_proc_address(1, 18, -)                                           
    -> 0, 0xb4f0b3c0                                                     
vdp_get_proc_address(1, 19, -)                                           
    -> 0, 0xb4f09080                                                     
vdp_get_proc_address(1, 46, -)                                           
    -> 0, 0xb4f0df90                                                     
vdp_get_proc_address(1, 47, -)                                           
    -> 0, 0xb4f0f390                                                     
vdp_get_proc_address(1, 53, -)                                           
    -> 0, 0xb4f08e10                                                     
vdp_get_proc_address(1, 54, -)                                           
    -> 0, 0xb4f0f8d0                                                     
vdp_get_proc_address(1, 55, -)                                           
    -> 0, 0xb4f08d40                                                     
vdp_get_proc_address(1, 56, -)                                           
    -> 0, 0xb4f0a2b0                                                     
vdp_get_proc_address(1, 57, -)                                           
    -> 0, 0xb4f08c70                                                     
vdp_get_proc_address(1, 63, -)                                           
    -> 0, 0xb4f08b60                                                     
vdp_get_proc_address(1, 64, -)                                           
    -> 0, 0xb4f0a020                                                     
vdp_get_proc_address(1, 4096, -)                                         
    -> 0, 0xb4f09d60                                                     
vdp_get_proc_address(1, 37, -)                                           
    -> 0, 0xb4f0a950                                                     
vdp_get_proc_address(1, 38, -)                                           
    -> 0, 0xb4f08ee0                                                     
vdp_get_proc_address(1, 40, -)                                           
    -> 0, 0xb4f0c760                                                     
vdp_get_proc_address(1, 65, -)                                           
    -> 0, 0xb4f09ea0                                                     
vdp_get_proc_address(1, 33, -)                                           
    -> 0, 0xb4f10240
vdp_get_proc_address(1, 23, -)
    -> 0, 0xb4f11620
vdp_presentation_queue_target_create_x11(1, 69206017, -)
    -> 0, 2
vdp_presentation_queue_create(1, 2, -)
VDPAU nvidia: Error detected 1004 824  5
VDPAU nvidia: Backtrace:
01: /usr/lib/libvdpau_nvidia.so [0xb4e58d0e]
02: /usr/lib/libvdpau_nvidia.so [0xb4e59bf9]
03: /usr/lib/libvdpau_nvidia.so [0xb4e7515c]
04: /usr/lib/libvdpau_nvidia.so [0xb4e9f24d]
05: /usr/lib/libvdpau_nvidia.so [0xb4e6ef64]
06: /usr/lib/libvdpau_nvidia.so [0xb4e67189]
07: /usr/lib/libvdpau_nvidia.so [0xb4e6691b]
08: /usr/lib/libvdpau_nvidia.so [0xb4e5eb10]
09: /usr/lib/libvdpau_trace.so [0xb4e4b371]
10: /home/cris/mplayer-vdpau-3263604/mplayer-vdpau/mplayer [0x80bde70]

http://hftom.free.fr/nvidia-bug-report.log

[Stephen Warren]

Did you edit that post? I got an email notification where the VDPAU trace stopped at the presentation_queue_create, and also contained a separate gdb backtrace. Now, the post contains a VDPAU-generated backtrace from where it detected an error. I'm confused...

[crisalide]

yes, i edited the post, but the result is the same

[Stephen Warren]

Oh, but the original gave a lot more information!

For posterity, here is the gdb-generated backtrace, which shows exactly where the hang is happening:

#0  0xb8093430 in __kernel_vsyscall ()
#1  0xb7420fec in sched_yield () from /lib/tls/i686/cmov/libc.so.6
#2  0xb4f2d8e7 in ?? () from /usr/lib/libvdpau_nvidia.so
#3  0xb4f32df5 in ?? () from /usr/lib/libvdpau_nvidia.so
#4  0xb4f34167 in ?? () from /usr/lib/libvdpau_nvidia.so
#5  0xb4f5e24d in ?? () from /usr/lib/libvdpau_nvidia.so
#6  0xb4f2df64 in ?? () from /usr/lib/libvdpau_nvidia.so
#7  0xb4f26189 in ?? () from /usr/lib/libvdpau_nvidia.so
#8  0xb4f2591b in ?? () from /usr/lib/libvdpau_nvidia.so
#9  0xb4f1db10 in ?? () from /usr/lib/libvdpau_nvidia.so
#10 0xb4f0a371 in ?? () from /usr/lib/libvdpau_trace.so
#11 0x080bde70 in config ()

To confirm I understand the issue: Are these true:

a) The VDPAU-generated backtrace you included, and the gdb-generated backtrace were from different runs of the 2nd application, not the same one (I assume this because the address of the first entry point into VDPAU is different).

b) In the 2nd app, when you try to create the presentation queue, you immediately get the VDPAU-generated backtrace, *then* you get a hang for a long time (during which you generated the gdb backtrace in the original post.

If so, then what's happening is that the 2nd app attempts to use overlay mode, finds it's already allocated (hence the VDPAU error backtrace), then attempts to clean up the partially allocated overlay presentation queue, and the cleanup code causes some nasty issue and the hang.

I'll attempt to repro this and fix it.

[crisalide]

ok, i should have added the nvidia backtrace instead of replacing the gdb one
Anyway, yes to a) and b).

[Stephen Warren]

Repro'd and fixed. Looks like a regression introduced between 180.06 and 180.08:-(

[crisalide]

Ok, so the hang is fixed in 180.25, thanx.
But still some problem remains.

The situation:
- A player is running, and has created a vdpau decoder to decode the stream (mpeg12 or h264).
- A second player is started, that tries to use vdpau. vo_vdpau initialization (now) works.
- The second player then tries to create a new vdpau decoder:
- for h264, decoder_create fails. Good, so the player can fallback to software decode.
- for mpeg12, decoder_create returns OK, then subsequent decoder_render returns OK, but nothing is decoded (random garbage in surfaces).

[crisalide]

Oops, my bad, decoder_create fails also for mpeg12.
Sorry