Go Back   nV News Forums > Linux Support Forums > NVIDIA Linux

Newegg Daily Deals

Reply
 
Thread Tools
Old 02-16-12, 05:23 PM   #1
interzoneuk
Registered User
 
Join Date: Mar 2005
Posts: 81
Question 295.20 segfaults a game *UNLESS* i launch with gdb ?? - previous drivers fine...

HI.

Here is an odd one.

I can confirm it has to be the driver - this issue is occuring on 2 OS's - Arch Linux and openSuse both using latest driver - 295.20

I am trying to play a game - digital paint2 - www.dplogin.com

- Its a game based on a modified Quake2 engine.

If I try to play using the latest driver - x86_64-295.20 the game segfaults when I run it (always)

- I get this is dmesg

Quote:
[ 1557.418209] paintball2[2607]: segfault at c ip 00000000f7018ea3 sp 00000000f5217180 error 6 in libnvidia-tls.so.295.20[f7018000+3000]
However - (and this I find very odd..) if I run the game with gdb (I tried using it to see what was going on) it works 100% fine (although slower as debugging is occurring) .

How is this possible ?

It 100% must be the driver - if I go back to previous one - x86_64-290.10 it's fine.

Its happening also on 2 separate systems - Arch Linux and OpenSuse 12.1 - on both systems if I revert back it works fine.

Any ideas ?



Specs:

AMD X2 4600
Nvidia 450 GTS
Arch Linux + openSuse (64bit)
Nvidia driver: 295.20
interzoneuk is offline   Reply With Quote
Old 02-17-12, 07:16 AM   #2
Dragoran
Registered User
 
Join Date: May 2004
Posts: 711
Default Re: 295.20 segfaults a game *UNLESS* i launch with gdb ?? - previous drivers fine...

Quote:
Originally Posted by interzoneuk View Post
HI.

Here is an odd one.

I can confirm it has to be the driver - this issue is occuring on 2 OS's - Arch Linux and openSuse both using latest driver - 295.20

I am trying to play a game - digital paint2 - www.dplogin.com

- Its a game based on a modified Quake2 engine.

If I try to play using the latest driver - x86_64-295.20 the game segfaults when I run it (always)

- I get this is dmesg



However - (and this I find very odd..) if I run the game with gdb (I tried using it to see what was going on) it works 100% fine (although slower as debugging is occurring) .

How is this possible ?

It 100% must be the driver - if I go back to previous one - x86_64-290.10 it's fine.

Its happening also on 2 separate systems - Arch Linux and OpenSuse 12.1 - on both systems if I revert back it works fine.

Any ideas ?



Specs:

AMD X2 4600
Nvidia 450 GTS
Arch Linux + openSuse (64bit)
Nvidia driver: 295.20
Sounds similar to http://www.nvnews.net/vbulletin/show...83&postcount=1 i.e I tried to get a backtrace just to find out that it works fine in gdb.
Dragoran is offline   Reply With Quote
Old 02-20-12, 04:48 AM   #3
marisn
Registered User
 
Join Date: Jun 2009
Posts: 9
Default Re: 295.20 segfaults a game *UNLESS* i launch with gdb ?? - previous drivers fine...

Is this game using a Qt based GUI for it's startup and configuration screens? If Yes, this could be related to http://www.nvnews.net/vbulletin/showthread.php?t=174244
marisn is offline   Reply With Quote
Old 02-20-12, 09:45 AM   #4
interzoneuk
Registered User
 
Join Date: Mar 2005
Posts: 81
Default Re: 295.20 segfaults a game *UNLESS* i launch with gdb ?? - previous drivers fine...

I'm pretty sure it doesn't use QT.

http://digitalpaint.org/files/ lists the deps you need for the 64bit version and QT isn't mentioned.

p.s the same thing happens with strace...

i.e if I use strace or gdb I can run the game on the new driver - without running a debugger it segfaults.

I have reverted back to the previous version for now. (290.10)
interzoneuk is offline   Reply With Quote
Old 02-25-12, 05:41 AM   #5
interzoneuk
Registered User
 
Join Date: Mar 2005
Posts: 81
Default Re: 295.20 segfaults a game *UNLESS* i launch with gdb ?? - previous drivers fine...

I managed to get an strace

(/usr/local/bin/check3d-kde4 is just a script that disables 3d in KDE before launching the game - it crashes regardless (i.e if you run normally) -Previous driver no issues at all..)


----------------------------------------------------------------------------------------------

[morgan@arch64 paintball2]$ strace pb2.sh
execve("/home/morgan/scripts/pb2.sh", ["pb2.sh"], [/* 55 vars */]) = 0
brk(0) = 0x97f000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb44fd31000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=207789, ...}) = 0
mmap(NULL, 207789, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb44fcfe000
close(3) = 0
open("/lib/libreadline.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\32 0T\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0555, st_size=326762, ...}) = 0
mmap(NULL, 2380744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb44f8cc000
mprotect(0x7fb44f908000, 2097152, PROT_NONE) = 0
mmap(0x7fb44fb08000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3c000) = 0x7fb44fb08000
mmap(0x7fb44fb10000, 5064, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb44fb10000
close(3) = 0
open("/usr/lib/libncursesw.so.5", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\26 0M\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=378800, ...}) = 0
mmap(NULL, 2475912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb44f66f000
mprotect(0x7fb44f6c7000, 2093056, PROT_NONE) = 0
mmap(0x7fb44f8c6000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x57000) = 0x7fb44f8c6000
mmap(0x7fb44f8cb000, 1928, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb44f8cb000
close(3) = 0
open("/lib/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\34 0\r\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=14784, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb44fcfd000
mmap(NULL, 2109688, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb44f46b000
mprotect(0x7fb44f46d000, 2097152, PROT_NONE) = 0
mmap(0x7fb44f66d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fb44f66d000
close(3) = 0
open("/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\24 0\24\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1974648, ...}) = 0
mmap(NULL, 3795888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb44f0cc000
mprotect(0x7fb44f261000, 2097152, PROT_NONE) = 0
mmap(0x7fb44f461000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x195000) = 0x7fb44f461000
mmap(0x7fb44f467000, 15280, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb44f467000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb44fcfc000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb44fcfb000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb44fcfa000
arch_prctl(ARCH_SET_FS, 0x7fb44fcfb700) = 0
mprotect(0x7fb44f461000, 16384, PROT_READ) = 0
mprotect(0x7fb44f66d000, 4096, PROT_READ) = 0
mprotect(0x7fb44f8c6000, 16384, PROT_READ) = 0
mprotect(0x7fb44fb08000, 8192, PROT_READ) = 0
mprotect(0x6ae000, 4096, PROT_READ) = 0
mprotect(0x7fb44fd32000, 4096, PROT_READ) = 0
munmap(0x7fb44fcfe000, 207789) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
open("/dev/tty", O_RDWR|O_NONBLOCK) = 3
close(3) = 0
brk(0) = 0x97f000
brk(0x9a0000) = 0x9a0000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1863264, ...}) = 0
mmap(NULL, 1863264, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb44fb33000
close(3) = 0
getuid() = 1000
getgid() = 100
geteuid() = 1000
getegid() = 100
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
open("/proc/meminfo", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb44fd30000
read(3, "MemTotal: 4053996 kB\nMemF"..., 1024) = 1024
close(3) = 0
munmap(0x7fb44fd30000, 4096) = 0
rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7fb44f1009f0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7fb44f1009f0}, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7fb44f1009f0}, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fb44f1009f0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fb44f1009f0}, {SIG_DFL, [], SA_RESTORER, 0x7fb44f1009f0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7fb44f1009f0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7fb44f1009f0}, {SIG_DFL, [], SA_RESTORER, 0x7fb44f1009f0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7fb44f1009f0}, {SIG_DFL, [], SA_RESTORER, 0x7fb44f1009f0}, 8) = 0
uname({sys="Linux", node="arch64", ...}) = 0
stat("/home/morgan/games/paintball2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getpid() = 1816
open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/gconv/gconv-modules", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=56368, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb44fd30000
read(3, "# GNU libc iconv configuration.\n"..., 4096) = 4096
read(3, "B1.002//\nalias\tJS//\t\t\tJUS_I.B1.0"..., 4096) = 4096
read(3, "59-3\t1\nmodule\tINTERNAL\t\tISO-8859"..., 4096) = 4096
read(3, "859-14//\nalias\tISO-IR-199//\t\tISO"..., 4096) = 4096
read(3, "CDIC-DK-NO-A//\tEBCDIC-DK-NO-A\t1\n"..., 4096) = 4096
read(3, "\t\tIBM281//\t\tIBM281\t\t1\n\n#\tfrom\t\t\t"..., 4096) = 4096
read(3, "\tIBM863\t\t1\n\n#\tfrom\t\t\tto\t\t\tmodule" ..., 4096) = 4096
read(3, "\tmodule\t\tcost\nalias\tIBM-939//\t\tI"..., 4096) = 4096
read(3, "\t\t\tmodule\t\tcost\nalias\tEUCCN//\t\t\t"..., 4096) = 4096
read(3, "CN//\t\tISO-2022-CN\t1\n\n#\tfrom\t\t\tto"..., 4096) = 4096
read(3, "INTERNAL\t\tISO_5427-EXT//\t\tISO_54"..., 4096) = 4096
brk(0x9c1000) = 0x9c1000
read(3, "e\t\tcost\nmodule\tARMSCII-8//\t\tINTE"..., 4096) = 4096
read(3, "2//\nmodule\tIBM1112//\t\tINTERNAL\t\t"..., 4096) = 4096
read(3, "\tIBM9448//\nalias\tCP9448//\t\tIBM94"..., 4096) = 3120
read(3, "", 4096) = 0
close(3) = 0
munmap(0x7fb44fd30000, 4096) = 0
getppid() = 1815
getpgrp() = 1815
rt_sigaction(SIGCHLD, {0x43b520, [], SA_RESTORER|SA_RESTART, 0x7fb44f1009f0}, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7fb44f1009f0}, 8) = 0
getrlimit(RLIMIT_NPROC, {rlim_cur=31112, rlim_max=31112}) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
open("/home/morgan/scripts/pb2.sh", O_RDONLY) = 3
ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fffbd1b9558) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR) = 0
read(3, "#!/bin/bash\ncd ~/games/paintball"..., 80) = 80
lseek(3, 0, SEEK_SET) = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0
fcntl(255, F_GETFD) = -1 EBADF (Bad file descriptor)
dup2(3, 255) = 255
close(3) = 0
fcntl(255, F_SETFD, FD_CLOEXEC) = 0
fcntl(255, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(255, {st_mode=S_IFREG|0755, st_size=81, ...}) = 0
lseek(255, 0, SEEK_CUR) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "#!/bin/bash\ncd ~/games/paintball"..., 81) = 81
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
stat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/home/morgan", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
stat("/home/morgan/games", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/home/morgan/games/paintball2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
chdir("/home/morgan/games/paintball2") = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0
lseek(255, -6, SEEK_CUR) = 75
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGC HLD, child_tidptr=0x7fb44fcfb9d0) = 1817
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x438cc0, [], SA_RESTORER, 0x7fb44f1009f0}, {SIG_DFL, [], SA_RESTORER, 0x7fb44f1009f0}, 8) = 0
wait4(-1,
Paintball 2 -- Version 2.0
Execing configs/config.cfg.
Console initialized.

------- Sound initialization -------
LoadLibrary("./snd_oss.so")

Initializing OSS Sound System
Buffer size: 524288
Stereo: 1
Samples: 262144
Samplepos: 0
Samplebits: 16
Submission_chunk: 1
Speed: 48000
Sound sampling rate: 48000
------------------------------------
------- Loading ref_pbgl.so -------
LoadLibrary("./ref_pbgl.so")
ref_gl version: PB2GL 0.35
/usr/local/bin/check3d-kde4: line 5: 1819 Segmentation fault $1

[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 1817
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1817, si_status=0, si_utime=0, si_stime=0} (Child exited) ---
wait4(-1, 0x7fffbd1b8e58, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn(0xffffffffffffffff) = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fb44f1009f0}, {0x438cc0, [], SA_RESTORER, 0x7fb44f1009f0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "cd ~/\n", 81) = 6
stat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/home/morgan", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
chdir("/home/morgan") = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0
read(255, "", 81) = 0
exit_group(0)
----------------------------------------------------------------------------------------------
interzoneuk is offline   Reply With Quote
Old 02-28-12, 05:36 AM   #6
sandipt
NVIDIA Corporation
 
sandipt's Avatar
 
Join Date: Dec 2010
Posts: 260
Default Re: 295.20 segfaults a game *UNLESS* i launch with gdb ?? - previous drivers fine...

NVIDIA internally filed bug 943126 to track this issue.
sandipt is offline   Reply With Quote
Old 02-29-12, 06:01 AM   #7
interzoneuk
Registered User
 
Join Date: Mar 2005
Posts: 81
Default Re: 295.20 segfaults a game *UNLESS* i launch with gdb ?? - previous drivers fine...

Thanks for that

How do I follow the bug report ? Or is it just for internal use (i.e I should wait until the next driver is out?)
interzoneuk is offline   Reply With Quote
Old 03-14-12, 08:59 PM   #8
lexa2
Registered User
 
Join Date: Jul 2011
Location: Moscow, Russian Federation
Posts: 58
Send a message via ICQ to lexa2 Send a message via Skype™ to lexa2
Default Re: 295.20 segfaults a game *UNLESS* i launch with gdb ?? - previous drivers fine...

Quote:
Originally Posted by interzoneuk View Post
How do I follow the bug report ? Or is it just for internal use (i.e I should wait until the next driver is out?)
As far as I can tell from my experience of reporting the bugs on this forums: this bug ID is an internal nVIDIA thing that we haven't got access to and thus can't track progress for. So your best bet would be to ping this thread once per month with messages like "any news on bug ###?" and hope that it would eventually be fixed after a while.

As for your issue: crashes, related to 295.20 drivers, also happen in a bunch of other libGL-using apps, most of them are windows games running under Wine. For example, Rage had been working perfectly with drivers 280.x, and now with 295.20 it segfaults on start. So hopes are nVIDIA would be pretty quick to fix this show-stopper bug.
lexa2 is offline   Reply With Quote

Old 03-19-12, 05:13 AM   #9
interzoneuk
Registered User
 
Join Date: Mar 2005
Posts: 81
Default Re: 295.20 segfaults a game *UNLESS* i launch with gdb ?? - previous drivers fine...

lexa2 thanks for the advice.

I fear that Ubuntu 12.04 is going to have this driver as the default Nvidia (non-free) driver, this is going to cause a huge amount of bug reports for the distro.

If they launch with this driver they are going to get a tonne of bug reports and complaints of Ubuntu 12.04 not being stable.

I have opened a bug report here

https://bugs.launchpad.net/ubuntu/+s...rs/+bug/948895

(completely ignored so far)
interzoneuk is offline   Reply With Quote
Old 03-21-12, 12:48 PM   #10
Plagman
NVIDIA Corporation
 
Plagman's Avatar
 
Join Date: Sep 2007
Posts: 254
Default Re: 295.20 segfaults a game *UNLESS* i launch with gdb ?? - previous drivers fine...

This is fixed; the fix should be included in the next 295 release.
Plagman is offline   Reply With Quote
Old 03-22-12, 02:50 AM   #11
RUSTY
Registered User
 
Join Date: Dec 2002
Posts: 44
Default Re: 295.20 segfaults a game *UNLESS* i launch with gdb ?? - previous drivers fine...

Quote:
Originally Posted by Plagman View Post
This is fixed; the fix should be included in the next 295 release.
when are you guys going to release next driver ? windows already got Forceware 300.99 and all we have is not working 295.20.
RUSTY is offline   Reply With Quote
Old 03-22-12, 04:32 PM   #12
Plagman
NVIDIA Corporation
 
Plagman's Avatar
 
Join Date: Sep 2007
Posts: 254
Default Re: 295.20 segfaults a game *UNLESS* i launch with gdb ?? - previous drivers fine...

http://www.nvnews.net/vbulletin/showthread.php?t=176804
Plagman is offline   Reply With Quote
Reply


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 12:50 PM.


Powered by vBulletin® Version 3.7.1
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright 1998 - 2014, nV News.