mplayer vdpau

[Stephen Warren]

Originally Posted by uau

This is the basic problem. What's the rationale for this design?

If an application doesn't want a frame displayed, it should not be placed into the presentation queue. The presentation queue's job is to display what the application tells it to, not to perform frame-rate conversion.

Originally Posted by uau

For the application to make the dropping decisions needed because of VSYNC rate requires it to know about VSYNC timing, information which is not readily available and would not be needed otherwise.

That isn't true. The frame timing information MUST be used in *all* cases to ensure correct A/V sync. In a PC environment, the video scanout clock and audio clock are not necessarily locked to each-other in any way at all. As such, correct A/V sync can only be achieved by correlating audio clock (or delay) vs. video presentation timestamps and perhaps vs. PC system time. If that is correctly implemented, VSYNC timing should mostly just work. In cases of low scanout rate, I admit a better experience might require using other information.

Originally Posted by uau

Why would you ever need to explicitly repeat frames?

You don't. Frames that are displayed remain displayed until replaced.

Originally Posted by uau

Anyway skipping is the main issue here, and detecting afterwards that frames already were late and then dropping some (without explicit awareness of VSYNC rate) is not a good solution at least if the display device is low FPS. You don't want to constantly have the video some multiple of 1/24 second late.

Even without extra information, the presentation queue should never be "some multiple of 1/24 seconds late", when used correctly. N frames should be queued into the presentation queue at all times. In order to achieve that, when the very first frame is queued, its presentation time should be set to some time in the future, to allow the buffer to build up (decoding takes a non-zero amount of time). The exact time in the future must be chosen carefully, and matched to when audio is first sent to the sound card, so that the first audio and video align (or are offset by the amount the timestamps in the content are set apart).

Originally Posted by uau

I think it would be possible to implement queuing frames ahead, and no FFmpeg changes would be needed. Video surfaces are allocated through the get_buffer() function in MPlayer's vd_ffmpeg.c, and MPlayer can control their reuse through that.

That would be good news.

Originally Posted by uau

If NVIDIA is unable or unwilling to change the queue behavior to automatically drop frames which should be dropped because of timing then I hope at least a way to directly get the current VSYNC rate of the synced-to device is added to the VDPAU API.

I don't think VDPAU needs to expose this information; unless I'm wrong, detailed mode timing information is already available via other X APIs, which MPlayer already uses.

Arguments re: the frame rate changing apply equally to querying the frame rate through VDPAU or any other mechanism; even if VDPAU was extended with an API to query frame rate, VDPAU would not sent an event/callback/... to the application if the frame rate changed. Also note that NVIDIA's VDPAU implementation does not survive mode-switches anyway.

[uau]

Originally Posted by Stephen Warren

If an application doesn't want a frame displayed, it should not be placed into the presentation queue. The presentation queue's job is to display what the application tells it to, not to perform frame-rate conversion.

So you basically mean it would be harder to implement? ("Not its job" shouldn't otherwise prevent having more useful functionality.)

That isn't true. The frame timing information MUST be used in *all* cases to ensure correct A/V sync. In a PC environment, the video scanout clock and audio clock are not necessarily locked to each-other in any way at all. As such, correct A/V sync can only be achieved by correlating audio clock (or delay) vs. video presentation timestamps and perhaps vs. PC system time. If that is correctly implemented, VSYNC timing should mostly just work. In cases of low scanout rate, I admit a better experience might require using other information.

I think you misinterpreted what I meant by "VSYNC timing"; basically I meant display frame rate information here. You seem to be talking about syncing with VDPAU time. Yes that would be required if you use the queuing-ahead functionality; but frame rate information is needed to make good frame frame dropping decisions (arguably even more if you have a queue, as queuing multiple frames ahead could make the timing fall behind even more before you see any feedback) and would not be necessarily required otherwise.

Even without extra information, the presentation queue should never be "some multiple of 1/24 seconds late", when used correctly.

It would be. Consider what would happen if you queue say 8 frames of 50 FPS content, and the display happens to be a 24 FPS video projector.

I don't think VDPAU needs to expose this information; unless I'm wrong, detailed mode timing information is already available via other X APIs, which MPlayer already uses.

Would this still work if for example VDPAU_NVIDIA_SYNC_DISPLAY_DEVICE was set?

Arguments re: the frame rate changing apply equally to querying the frame rate through VDPAU or any other mechanism; even if VDPAU was extended with an API to query frame rate, VDPAU would not sent an event/callback/... to the application if the frame rate changed.

You could at least query it again occasionally or when frames are falling behind.

Also note that NVIDIA's VDPAU implementation does not survive mode-switches anyway.

So that means that timing maximum-speed frame updates at startup of preemption recovery should give reliable values. Still somewhat tedious to do though, and I doubt you want to guarantee that for future implementations (I mean you probably don't want to make it a requirement on every server-side implementation of the VDPAU API that every mode change will _have_ to trigger a preemption even if it could otherwise be avoided).

[danoscar]

Originally Posted by Stephen Warren

I don't think VDPAU needs to expose this information; unless I'm wrong, detailed mode timing information is already available via other X APIs, which MPlayer already uses.

I have not found any for this. You can get some through X APIs but what I get back is 24hz on a 23.976hz rate. Not good enough. To be able to do really good A/V sync handling you need a resolution of at least micro seconds. nano seconds would be better.

My patches get the time between vsyncs by queuing surfaces to the presentation queue and read when they are displayed. Timing over 3 vsyncs I can get micro second precision when overlay is used, and millisecond precision when blitting is used. It would be much easier to have just an api to fetch the time between two vsyncs. With my method there is a delay before vsync timing can be calculated delaying showing of movie.

My patches also uses the time of last vsync (gotten from query time last surface was displayed by the presentation queue) to adjust speed of movie to match speed of display.
Here it also be simpler if a direct call to get time of last vsync existed.

My next planned step is to use my knowledge of when last vsync occurred and time between vsyncs to make better decisions when to queue a frame for display and when to drop a frame. Like uau talked about. To decode more then one frame in advance should also be possible, without changing that much in mplayer. I planned to look at that next.

[Stephen Warren]

Originally Posted by uau

So you basically mean it would be harder to implement? ("Not its job" shouldn't otherwise prevent having more useful functionality.)

Yes, in some cases, it would be harder to implement. Even if that weren't the case, I still believe that philosophically, the presentation queue internals is the wrong place to make this decision. The application has much better knowledge (e.g. perhaps 3:2 pulldown information) and is the only place that correct frame-dropping decisions can be made.

Originally Posted by uau

It would be. Consider what would happen if you queue say 8 frames of 50 FPS content, and the display happens to be a 24 FPS video projector.

That would be a case of the application using the presentation queue API incorrectly.

Originally Posted by uau

Would this still work if for example VDPAU_NVIDIA_SYNC_DISPLAY_DEVICE was set?

I don't see why not. Obviously the application needs to be aware which display device is being used so as to operate correctly - i.e. use timing information from the correct display.

[Stephen Warren]

Originally Posted by danoscar

I have not found any for this. You can get some through X APIs but what I get back is 24hz on a 23.976hz rate. Not good enough. To be able to do really good A/V sync handling you need a resolution of at least micro seconds. nano seconds would be better.

It looks like XF86VidModeGetModeLine should work; it can give you, amongst other things, the pixel clock rate and the total pixels (including blanking) for both H and V, so the VSYNC rate can be calculated almost trivially.

I hope our driver fully implements that API now that I've mentioned it:-)

[danoscar]

Originally Posted by piotro

Thx for trying help !
Unfortunatelly patch works only for analog audio.
For SPDIF based systems I' receiving error like below.

IMHO Your patch isn't fully resolving issue - it is rather workaround, as wrong synchronization direction is still present (TV->mplayer playback).

Honestly speaking, I'm little surprised. Design where we are not able to sync display to source is IMHO issue - as long as we want to use VDPAU as support for serious player with no judder, no tearing playback.

As I do not have SPDIF I cannot debug what goes wrong for you.
Sorry.

By sync display to source you mean that if I have a 24.123hz movie the display should be changed to 24.123hz? As most display only allow a few refresh rate settings, this will not work for most displays. The only thing you can do is to adjust movie frame rate to match display refresh rate. That is what my patch tries to do.

[uau]

Originally Posted by Stephen Warren

Yes, in some cases, it would be harder to implement. Even if that weren't the case, I still believe that philosophically, the presentation queue internals is the wrong place to make this decision. The application has much better knowledge (e.g. perhaps 3:2 pulldown information) and is the only place that correct frame-dropping decisions can be made.

I don't see any philosophical problem. The presentation queue already has to decide during which and how many display refreshes a frame is shown. The obvious answer is to show it for a refresh if the time of that refresh is later than the earliest presentation timestamp of the frame, and there is no later frame which would also fulfill this condition. This can result in showing the frame for 0, 1, 2 or more refreshes. What reason is there to special-case "0" and disallow that in particular?

That would be a case of the application using the presentation queue API incorrectly.

"Incorrectly"? How? The example only required queuing 8 frames of 50 FPS content. So what's incorrect about that? Placing 8 frames in the queue? Or using the queue to play 50 FPS content? What assumptions exactly are you making if you don't consider that to be "correct" usage? How could the application handle this correctly without having extra information about the display refresh rate?

I don't see why not. Obviously the application needs to be aware which display device is being used so as to operate correctly - i.e. use timing information from the correct display.

Is there an obvious way for the application to be aware of that? In the simplest case the application opens a window, lets the window manager place in wherever (the user could move it later) and then uses the VDPAU API. How can I be sure I get the right values? If VDPAU_NVIDIA_SYNC_DISPLAY_DEVICE actually changes the behavior of the VDPAU implementation how can the application be aware of the changes without being explicitly aware of that particular environment variable and parsing it?

[piotro]

[quote=danoscar;2094755]As I do not have SPDIF I cannot debug what goes wrong for you.
Sorry.[/QUOTE

I really appreciate Your w willingness to help !

Originally Posted by danoscar

By sync display to source you mean that if I have a 24.123hz movie the display should be changed to 24.123hz?

Yes. Indeed, looks horrible, but if we want to have:
-perfect A/V sync (constant sync over hours of playback)
-no judder
This is only IMHO proper solution.
Fortunately most of content has fps from limited pool of possible values: 23.967, 23.98, 24, 25, 30. 50 & 60 are integer multiplies of 25 & 30 so no problem with judder.

Originally Posted by danoscar

As most display only allow a few refresh rate settings, this will not work for most displays. The only thing you can do is to adjust movie frame rate to match display refresh rate. That is what my patch tries to do.

I believe in stand-alone player syncing display to content is standard. Simply, player code is stearing deviation of VCO in clocking sybsystem driving display H/V sequencers.
In PC enviroment it is indeed difficult, so workarround might be based i.e. on free runing of decode and displaying and making sure about speed accuracy.
I.e. 0.1% inaccuracy in speeds causes judder for every 1.76sec.
Assuming that judder with period of tenths of sec is quite unnoticeable - we have to have 0.01% absolute accuracy for player AND display clocks.
So now questions about PLL absolute accuracy arises...

Unfortunately any -even small error - in speed is additive, so it is possible that after hours of playback A/V desync will be noticeable.
Solutions i.e. with frame dropping for keeping A/V sync by monitoring buffer occupancy are IMHO good compromise: acceptable to implement and maybe acceptable judder.
Look: my 23.976 fps move displayed on 24Hz TV has easy noticeable judder for every 1.76 sec. It is visible so easy, that many watches asked me: what is wrong ?

Also Your solution with playback speed adjustments is really nice. It's pity that for SPDIF it has issues.
BTW: I'm surprised by that - taking into account that for analog it works -> mans that D/A converters are able to be clocked faster by few percents. Maybe SPDIF issue is in fact pointing to ALSA spdif non-optimal implementation...

[Stephen Warren]

Originally Posted by uau

I don't see any philosophical problem. The presentation queue already has to decide during which and how many display refreshes a frame is shown. The obvious answer is to show it for a refresh if the time of that refresh is later than the earliest presentation timestamp of the frame, and there is no later frame which would also fulfill this condition. This can result in showing the frame for 0, 1, 2 or more refreshes. What reason is there to special-case "0" and disallow that in particular?

Skipping display of a frame requires skipping sync-to-vblank for that queue entry. There are implementation difficulties doing this; it's probably not possible.

And again I re-iterate: The application should be aware of what it's doing and what the presentation queue's capabilities are. The application is the only software in possession of complete knowledge required to choose the correct frames to skip.

Originally Posted by uau

"Incorrectly"? How? The example only required queuing 8 frames of 50 FPS content. So what's incorrect about that? Placing 8 frames in the queue? Or using the queue to play 50 FPS content?

Attempting to display 50fps content on a 24Hz display doesn't make sense without frame-dropping. The presentation queue does not implement frame dropping. Hence, the application must.

Originally Posted by uau

Is there an obvious way for the application to be aware of that? In the simplest case the application opens a window, lets the window manager place in wherever (the user could move it later) and then uses the VDPAU API. How can I be sure I get the right values? If VDPAU_NVIDIA_SYNC_DISPLAY_DEVICE actually changes the behavior of the VDPAU implementation how can the application be aware of the changes without being explicitly aware of that particular environment variable and parsing it?

I suppose that in a TwinView configuration, it's not currently possible for an application to know which display device VDPAU is syncing to for the blit-based presentation queue (the overlay-based presentation queue syncs to both display devices independently). We'll think about the best solution for this.

[uau]

Originally Posted by Stephen Warren

Skipping display of a frame requires skipping sync-to-vblank for that queue entry. There are implementation difficulties doing this; it's probably not possible.

So no philosophical reasons, just implementation difficulties doing otherwise.

And again I re-iterate: The application should be aware of what it's doing and what the presentation queue's capabilities are. The application is the only software in possession of complete knowledge required to choose the correct frames to skip.

At least the algorithm I currently intend to use in MPlayer will only use the same timing information as is available to VDPAU - MPlayer will drop a frame if the next frame can be shown during the same VSYNC.

Attempting to display 50fps content on a 24Hz display doesn't make sense without frame-dropping. The presentation queue does not implement frame dropping. Hence, the application must.

Well IMO your original claim about how the presentation queue would work without extra information if used correctly becomes quite meaningless if you define "correct" use to require getting extra information about the display frame rate and using that to drop frames before placing them in the queue...

I suppose that in a TwinView configuration, it's not currently possible for an application to know which display device VDPAU is syncing to for the blit-based presentation queue (the overlay-based presentation queue syncs to both display devices independently). We'll think about the best solution for this.

So I suppose the overall result is that for implementation reasons VDPAU won't support cases where multiple queue timestamps fall within the same VSYNC, and applications need to get display device refresh rate information and handle framedropping if they want to avoid the frames being delayed in that case. The best way to get the refresh rate information will be defined later.

[geoffleach]

Originally Posted by danoscar

The problem is how mplayer handles frame display in VDPAU (and some other video outputs). It results in mplayer getting out of A-V sync when refresh rate is nearly the same as fps of movie. To fix this I have patched mplayer to drop frames in VDPAU layer when they cannot be displayed in time to avoid A-V gets out of sync (if frames are dropped it will be displayed on status line)
[snip]
.

I record ATSC (US HD) OTA using an HD HomeRun tuner. I've had serious A-V sync prolbems (dropped frames, large values of AV_delay). I appied danoscar's patche to mplayer (source as of 29 Sept). Since then, I have has absolutely no problems. Great thanks to danoscar. (FWIW, this has nothing to do with Blu-ray)

MB: ASUS P5N7A-VM
CPU: Intel Core 2 Duo E8400 Wolfdale 3.0GHz 6MB L2 Cache LGA 775
GPU: nVidia MCP79 Host Bridge
Video: nVidia GeForce 9300/nForce 730i
Audio: Realtek ALC1200

Sony KDL 40V3000 LCD
1920x1080 @60HZ
Connected to PVR via HDMI

Mplayer: -ao alsa::device=hw=0.3 -vo vdpau -vc ffmpeg12vdpau

[uau]

I implemented smart frame dropping/timing functionality for MPlayer's VDPAU driver to work around the frame rate limitation and avoid jitter due to unstable timing. The code is currently in a temporary branch but can be tested as follows:

First do 'git clone git://repo.or.cz/mplayer-build.git' and build as normal (following the README). Then to switch to the new timing code go to the 'mplayer' subdirectory and run 'git checkout origin/tmp_vdpau' and run make again (you can run it in the subdirectory if you did a top-level compile before; running it at the top level again works too, just recompiles more things than needed).

I'll change the build repo to use this code by default after some more testing and other changes.

Below is the log message of the main commit with more detail:


Implement vblank-aware frame timing for VDPAU

Main things added are custom frame dropping for VDPAU to work around
the display FPS limit, frame timing adjustment to avoid jitter when
video frame times keep falling near vblanks, and use of VDPAU's timing
feature to keep one future frame queued in advance.

NVIDIA's VDPAU implementation refuses to change the displayed frame
more than once per vblank. This set a limit on how much video could be
sped up, and caused problems for nearly all videos on low-FPS video
projectors (playing 24 FPS video on a 24 FPS projector would not work
reliably as MPlayer may need to slightly speed up the video for AV
sync). This commit adds a framedrop mechanism that drops some frames
so that no more than one is sent for display per vblank. The code
tries to select the dropped frames smartly, selecting the best one to
show for each vblank. Because of the timing features needed the drop
functionality currently does not work if the correct-pts option is
disabled.

The code also adjusts frame timing slightly to avoid jitter. If you
for example play 24 FPS video content on a 72 FPS display then
normally a frame would be shown for 3 vblanks, but if the frame times
happen to fall near vblanks and change between just before and just
after then there could be frames alternating between 2 and 4
vblanks. The code changes frame timing by up to one quarter vblank
interval to avoid this.

The above functionality depends on having reliable vblank timing
information available. This is not directly provided by the VDPAU API,
but can be deduced from the frameflip timing information provided by
VDPAU. The code calculating this is somewhat tricky but should work
quite robustly as long as the per-frame timing information is precise.
It works without needing any separate measurement phase and can handle
changes in display FPS in the middle of playback.

After the changes in this commit MPlayer now always tries to keep one
frame queued for future display using VDPAU's internal timing
mechanism (however no more than 50 ms to the future). This should make
video playback somewhat more robust against timing inaccuracies caused
by system load.