Go Back   nV News Forums > Linux Support Forums > NVIDIA Linux

Newegg Daily Deals

Reply
 
Thread Tools
Old 04-10-06, 01:28 PM   #61
chunkey
#!/?*
 
Join Date: Oct 2004
Posts: 662
Default Re: [PATCH, REALTIME] nvidia-1.0-8178 and Linux-2.6.16-rt11

Quote:
Originally Posted by JaXXoN
Hi!

It looks like that the CPU is stopped for several hundred microseconds
while accesing the DMA status byte of the nforce4 sata controller.

True, True... I think, we've all the same problem here...

libata/sata_nv latency on NVIDIA CK804
chunkey is offline   Reply With Quote
Old 04-10-06, 05:02 PM   #62
JaXXoN
Registered User
 
Join Date: Jul 2005
Location: Munich
Posts: 910
Default Re: [PATCH, REALTIME] nvidia-1.0-8178 and Linux-2.6.16-rt11

Quote:
Originally Posted by chunkey
[url=http://www.uwsg.iu.edu/hypermail/linux/kernel/0603.1/2539.html
libata/sata_nv latency on NVIDIA CK804[/url]
Thanks very much for that hint! I was running through this interesting
thread and also came to the conclusion that we are talking about the
same issue. Some ppl have up to 16.6 milliseconds even without
using nvidia 3D graphics (at least nobody mentioned that). The theory
that there are this "SMM BIOS calls" sounds plausible, especially because
it's unlikely that a PCI DMA could hold up the CPU for 16 ms - unless the
PCI arbitration settings are realy realy bad, done by some ignorant
apprentise :-)

However, the dis-assembled code in ata_bmdma_status() only shows
an "ordinary" inb instruction. So if the inb causes an exception
which in turn does an SMM call, then i guess the exception handler should
be somewhere in the linux kernel?! Or at least the kernel needs to setup
an exception gate for some exception handler residing in the BIOS?
In this case, we could redirect that exception gate to our own exception
handler, i.e. dumping the stack trace and in turn invoking the BIOS
exception handler - just to see if this is the problem ...

It's pretty clear that such an SMM called could neither be captured
by /proc/latency (unless the exception handler has been instrumented),
nor with the apic timer demo application (because the CPU stumbles
from the interrupt protected exception straight to the apic timer interrupt
exception).

Do you know where i could learn more about SMM BIOS calls
and the underlaying mechanism? I tried google, but couldn't find
any usefull links in short term.

regards

Bernhard
JaXXoN is offline   Reply With Quote
Old 04-10-06, 07:32 PM   #63
JaXXoN
Registered User
 
Join Date: Jul 2005
Location: Munich
Posts: 910
Default Re: [PATCH, REALTIME] nvidia-1.0-8178 and Linux-2.6.16-rt11

Hi!

The high latencies caused by ata_bmdma_status() have disapeared
after removing the SATA hard drive from the nforce4 SATA controller and
connecting it to the on-board SIL3114 SATA controller (that comes with
my ASUS A8N-SLI Premium), instead. I would say this backs the theory that
the inb instruction causes an SMM call - if there would be any PCI arbitration
issues, then the system should still have suffered from that problem, IMHO.

After half an hour of playing ut2004, worst case interrupt latency measured
was 38.93 microseconds and worst case high priority user context switch time
was 87.34 microseconds. The worst case interrupt latency was caused by
__schedule(), so i guess there is little that can be done to further reduce
latencies. Anyway, i have reached my sub 100 microseconds goal - at least
for now, hopefully there are no more further high latency paths :-)

Thanks again to everybody providing feedback helping me finding a solution.

regards

Bernhard
JaXXoN is offline   Reply With Quote
Old 04-12-06, 09:46 PM   #64
JaXXoN
Registered User
 
Join Date: Jul 2005
Location: Munich
Posts: 910
Default Re: [PATCH, REALTIME] nvidia-1.0-8178 and Linux-2.6.16-rt11

Hi!

I made some small progresses on the "3-seconds glxgears sticky in single
CPU mode" problem:

A. When running glxgears with "strace glxgears 2>/dev/null", then the
X-server doesn't freeze when dragging the glxgears window arround.

B. When adding a "usleep(10);" in the event_loop, after both, draw()
and glXSwapBuffers(), then the sticky problem also disabears.

Can somebody please confirm or deny A and/or B?

regards

Bernhard
JaXXoN is offline   Reply With Quote
Old 04-13-06, 06:17 AM   #65
dmetz99
Registered User
 
Join Date: Mar 2005
Posts: 84
Default Re: [PATCH, REALTIME] nvidia-1.0-8178 and Linux-2.6.16-rt11

I can't confirm A.

If I add "ulseep(5)" after the call to "gettimeofday" in glxgears.c, the 3-second freezes go away. I guess this indirectly confirms your B. observation. But this also slows the glxgears framerate significantly (~200 fps). This behavior only manifests itself if the system is using IO-APIC for interrupt routing. On this box, where IO-APIC is inoperative, and the system uses the PIC for routing, glxgears never freezes.
dmetz99 is offline   Reply With Quote
Old 05-25-06, 05:46 AM   #66
jubiliant_ankit
Registered User
 
Join Date: May 2006
Posts: 8
Default Re: [PATCH, REALTIME] nvidia-1.0-8178 and Linux-2.6.16-rt11

Hello ,
rttest. c gives the latency .. But can somebody tell me what latency r v exactly talkin bout here ..
as in the interrupt latency , scheduling latency or anything else

Thanks in anticipation
Regards
ankit
jubiliant_ankit is offline   Reply With Quote
Old 05-25-06, 12:11 PM   #67
JaXXoN
Registered User
 
Join Date: Jul 2005
Location: Munich
Posts: 910
Default Re: [PATCH, REALTIME] nvidia-1.0-8178 and Linux-2.6.16-rt11

Quote:
Originally Posted by jubiliant_ankit
can somebody tell me what latency r v exactly talkin bout here
The rttest application puts itself into high priority (real time) scheduling
policy and periodically sets an alarm clock to wake up itself. The latency
measured is the delay between the preset alarm time and the time the
process actually proceeds. So it's basically the worst case scheduling latency
(plus timer event handling to be more accurate). For details, please
check the commented source code.

regards

Bernhard
JaXXoN is offline   Reply With Quote
Old 05-26-06, 12:13 AM   #68
jubiliant_ankit
Registered User
 
Join Date: May 2006
Posts: 8
Default Re: [PATCH, REALTIME] nvidia-1.0-8178 and Linux-2.6.16-rt11

hello bernard ,
thanks for ur prompt reply..
I have one more problem ... in ur some earlier post u posted the interrupt latency and high priority context switch time .. can u please tell me how to exactly obtain those figures ..

see basically I am trying to characterise real time linux kernel ..but I don't know bout what tests are to be used for wat purspose..

Hope you can help me in this respect .


Thanks in anticipation ,
Regards,
ankit
jubiliant_ankit is offline   Reply With Quote

Old 05-26-06, 06:56 AM   #69
JaXXoN
Registered User
 
Join Date: Jul 2005
Location: Munich
Posts: 910
Default Re: [PATCH, REALTIME] nvidia-1.0-8178 and Linux-2.6.16-rt11

Quote:
Originally Posted by jubiliant_ankit
in ur some earlier post u posted the interrupt latency and high priority context switch time .. can u please tell me how to exactly obtain those figures ..
There have been two issue:

a "wbinvd" instruction has been invoked when a 3D application has been
started or exited. This causes the caches to be flushed, in turn causing
latencies in the range of several 100 microseconds. The solution was
to disable PAT support which in turn avoids the "wbinvd" instruction
(however, there is a little bit mor to it, see patch above).

The other problem was that the nforce4 SATA controller causes
high latencies up to several 100 microseconds (ppl. reported latencies
up to 16.6 ms) for yet not 100% confirmed reason (the latencies
are probably caused by "SMM BIOS call"). The solution was to
use a SIL3114 SATA Controller instead.

Please note that the above two issues very likely are also affecting
any other Real-Time Linux solutions such as RT-Linux and RTAI, etc.
I didn't tested it, but i technically can't see why theses high latencies
shouldn't show up in those invironments, too. Anyway, the same
solutions as described above could be applied.

regards

Bernhard
JaXXoN is offline   Reply With Quote
Old 05-27-06, 06:53 PM   #70
JaXXoN
Registered User
 
Join Date: Jul 2005
Location: Munich
Posts: 910
Default Re: [PATCH, REALTIME] nvidia-1.0-8178 and Linux-2.6.16-rt11

Quote:
Originally Posted by jubiliant_ankit
in ur some earlier post u posted the interrupt latency and high priority context switch time .. can u please tell me how to exactly obtain those figures ..
Oops, i guess i missunderstood something when doing my former
replay: i understand you want you know how i meassured
interrupt latencies and scheduler latencies?

For the interrupt latencies, i have used an interrupt service routine
that is kicked by the APIC timer. The APIC timer re-starts counting
when the interrupt has been issued and the first thing the ISR
is doing is reading in the current APIC timer value, so you get
the delay between the interrupt was issued and the time the ISR
is doing something. For details, please check
http://www.nvnews.net/vbulletin/show...4&postcount=59

For scheduler latencies, i used rttest and cyclictimer which
work similar to the ISR above: an alarm is set for a certain
time and the first thing the process does after the alarm
has taken place is to do a gettimeofday() and calculates
the delay.

regards

Bernhard
JaXXoN is offline   Reply With Quote
Old 06-08-06, 08:22 AM   #71
jubiliant_ankit
Registered User
 
Join Date: May 2006
Posts: 8
Default Re: [PATCH, REALTIME] nvidia-1.0-8178 and Linux-2.6.16-rt11

Hello jaxxon
As I too premature to assert on the real time characteristics of a kernel I need one more help from you.
here are some of my results for two linux kernels . I have attached the files with this mail..

first kernel is 2.6.16-rt22 and the other one is vanilla 2.6.16...

I have obtained real time characteristics of both of them based on bench test results .
Please see to them and tell me the validity of these results .

please find the files attached with this mail

Thanks in anticipation
ankit
Attached Files
File Type: txt hackbench_results.txt (1.0 KB, 165 views)
File Type: txt lmbench_results.txt (5.1 KB, 159 views)
File Type: txt results.txt (677 Bytes, 154 views)
jubiliant_ankit is offline   Reply With Quote
Old 06-08-06, 08:50 AM   #72
JaXXoN
Registered User
 
Join Date: Jul 2005
Location: Munich
Posts: 910
Default Re: [PATCH, REALTIME] nvidia-1.0-8178 and Linux-2.6.16-rt11

Hi ankit!

Benchmarks such as lmbench and hackbench are IMHO not realy
good real time benchmarks, because these mostly measure the
system throughput and especially because they do not run
at real time priority (AFAIK). Please note that enhanced real time
capabilities typically come with a small throughput penality (as you
can see in some of your results). This discussion is not Linux specific:
determinism comes with overhead :-)

56 microseconds for the scheduler latency seems to be ok,
but 214 microseconds for the apictimer interrupt latencies
is much higher then i would expect. For me, the worst case
has been 11 microseconds (with PAT support disabled, tested
with 2.6.16-rt28).

Could you please post the disassembled portion of code in question?
(especially to know which function causes this high latency).

regards

Bernhard
JaXXoN is offline   Reply With Quote
Reply


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 12:24 AM.


Powered by vBulletin® Version 3.7.1
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright 1998 - 2014, nV News.