Mitsubishi 3000GT & Dodge Stealth Forum banner

1 - 20 of 43 Posts

·
Demoted user
Joined
·
595 Posts
Discussion Starter #1
The status of the website is as follows

There are two servers for 3si. One which handles all http requests as well as email. The other machine handles the mysql server and database.

These are the specs:

amd.techdroid.com
runs 3si website and email
Dual AMD Opteron 240 with 2GB of RAM
SATA drives

penguin.techdroid.com
runs 3si mysql database
Dual Intel Pentium III Coppermine 1.0 GHz with 2 GB of RAM
SCSI drives

Both systems run gentoo linux.

The problem with 3si is that mysql keeps causing problems on penguin.techdroid.com. In one case mysql will temporarily refuse connections and in the other worse case it will block amd.techdroid.com from accessing it until I flush the block list or restart it.

Mysql is the only thing running on penguin so it should be fine. However penguin has been giving problems with other applications like SSH which for some reason just stops running. No error msgs or anything. Even after a complete system wipe and fresh installation of gentoo, SSH still crashes. This is very odd as SSH was running for the longest time on this machine. I have the feeling there may be a hardware issue with this machine but I don’t know.

Here is the mysql config. If you have a suggestion for a better configuration please let me know.
Code:
[mysqld]
skip-locking
max_connect_errors = 5000
max_connections = 100
ft_min_word_len = 2
key_buffer = 16M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 1024
thread_cache_size = 64
wait_timeout = 1800
connect_timeout = 10
max_allowed_packet = 16M
query_cache_limit = 1M
query_cache_size = 32M
query_cache_type = 1
skip-innodb

[mysqldump]
quick
set-variable    = max_allowed_packet=1M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
set-variable    = key_buffer=16M
Increasing max_connections will result in a significant slow down of the server when the load on the forum is high. I also haven’t seen the message “too many users” for a long time (correct me if I’m wrong) so I shouldn’t have to change that value.

If there are any other questions I will answer them here.

-Stefan
 

·
Ninja Performance
Joined
·
42,337 Posts
Looks to me that the hardware is plenty adeqoute.
Thank you for posting this info. We have enough techies here, we should be albe to figure out what keeps crashing it.
 

·
Pro xfercase destroyer
Joined
·
5,423 Posts
Without seeing some utilization reports (or a copy/paste of "top" or something) its hard to say - but generally I prefer to put my database on the faster hardware since it has a lot more heavy lifting to do than apache/PHP does.

I don't have time to look close at the config stuff right now (need to rebuild my tranny today), but I'll come back to this thread later.
 

·
Demoted user
Joined
·
595 Posts
Discussion Starter #4
Here top from penguin

Code:
top - 09:06:08 up 120 days, 15:31,  2 users,  load average: 0.22, 0.18, 0.17
Tasks:  95 total,   1 running,  94 sleeping,   0 stopped,   0 zombie
Cpu(s):   0.4% user,   5.6% system,   0.0% nice,  93.9% idle
Mem:    903912k total,   897012k used,     6900k free,     7324k buffers
Swap:   987956k total,     2480k used,   985476k free,   625184k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                             
 8621 cyzoonic  17   0  1008 1008  788 R 11.6  0.1   0:01.91 top                                                 
    1 root       9   0   112   80   76 S  0.0  0.0   1:50.70 init                                                
    2 root       9   0     0    0    0 S  0.0  0.0   0:00.02 keventd                                             
    3 root      19  19     0    0    0 S  0.0  0.0   0:09.23 ksoftirqd_CPU0                                      
    4 root      19  19     0    0    0 S  0.0  0.0   0:09.35 ksoftirqd_CPU1                                      
    5 root       9   0     0    0    0 S  0.0  0.0 167:31.23 kswapd                                              
    6 root       9   0     0    0    0 S  0.0  0.0   0:03.27 bdflush                                             
    7 root       9   0     0    0    0 S  0.0  0.0  14:17.61 kupdated                                            
   10 root       9   0     0    0    0 S  0.0  0.0   0:00.00 ahc_dv_0                                            
   11 root       9   0     0    0    0 S  0.0  0.0   0:00.00 scsi_eh_0                                           
   12 root       9   0     0    0    0 S  0.0  0.0   0:00.02 khubd                                               
   13 root       9   0     0    0    0 S  0.0  0.0   0:09.05 kreiserfsd                                          
  153 root       9   0   384  304  304 S  0.0  0.0   0:00.04 devfsd                                              
  956 root       9   0   364  328  268 S  0.0  0.0   0:07.67 syslog-ng                                           
 1366 root       8   0   168  116   84 S  0.0  0.0   0:15.89 cron                                                
 1382 root       9   0    92   28   28 S  0.0  0.0   0:00.01 agetty                                              
 1383 root       9   0    92   28   28 S  0.0  0.0   0:00.00 agetty                                              
 1384 root       9   0    92   28   28 S  0.0  0.0   0:00.00 agetty                                              
 1385 root       9   0    92   28   28 S  0.0  0.0   0:00.01 agetty                                              
 1386 root       9   0    64    0    0 S  0.0  0.0   0:00.00 agetty                                              
 1700 root       9   0   372  332  292 S  0.0  0.0   0:00.05 xinetd                                              
 1708 root       9   0   216   28   28 S  0.0  0.0   0:00.01 login                                               
 1709 root       9   0   204    0    0 S  0.0  0.0   0:00.00 bash                                                
32416 root       9   0   536  400  380 S  0.0  0.0   0:00.00 mysqld_safe                                         
32456 mysql      9   0  119m 118m 2472 S  0.0 13.4   0:18.76 mysqld                                              
32457 mysql      9   0  119m 118m 2472 S  0.0 13.4   0:00.65 mysqld                                              
32458 mysql      9   0  119m 118m 2472 S  0.0 13.4   0:23.29 mysqld                                              
32459 mysql      9   0  119m 118m 2472 S  0.0 13.4   0:00.00 mysqld                                              
32461 mysql      9   0  119m 118m 2472 S  0.0 13.4   1:32.22 mysqld                                              
32482 mysql      9   0  119m 118m 2472 S  0.0 13.4   1:34.69 mysqld                                              
32491 mysql      9   0  119m 118m 2472 S  0.0 13.4   1:45.88 mysqld                                              
32501 mysql      9   0  119m 118m 2472 S  0.0 13.4   1:18.64 mysqld                                              
32509 mysql      9   0  119m 118m 2472 S  0.0 13.4   1:39.12 mysqld                                              
32515 mysql      9   0  119m 118m 2472 S  0.0 13.4   1:51.98 mysqld                                              
32516 mysql      9   0  119m 118m 2472 S  0.0 13.4   1:16.20 mysqld                                              
32522 mysql      9   0  119m 118m 2472 S  0.0 13.4   1:05.67 mysqld
I just noticed that penguin only sees 1GB of memory. Very odd.
Code:
        total:    used:    free:  shared: buffers:  cached:
Mem:  925605888 918900736  6705152        0  7774208 640741376
Swap: 1011666944  2539520 1009127424
MemTotal:       903912 kB
MemFree:          6548 kB
MemShared:           0 kB
Buffers:          7592 kB
Cached:         625384 kB
SwapCached:        340 kB
Active:         154480 kB
Inactive:       598424 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       903912 kB
LowFree:          6548 kB
SwapTotal:      987956 kB
SwapFree:       985476 kB
I will need to restart it when I get back and check what the bios reports.

-Stefan
 

·
Registered
Joined
·
1,553 Posts
well... if it all fails, try another distro. I've always had little annoying problems with gentoo. I agree with mjannusch. Most of the heavy lifting is being done by mysql, but both server should handle the load without any issues.
 

·
That Others May Live
Joined
·
3,845 Posts
According to the members list, we have 14,910 members. Now obviously very few of those members are even active anymore. Many of them, came on, looked around a little, maybe posted once or twice, then left, never to return again.
Would it do any good by having someone go through and flush out all of these obviously non-active members to free up memory? I realize that the memory is not the root of the problem, but sometimes freeing up some memory space can make things a lot more efficient.
 

·
Registered
Joined
·
2,012 Posts
im running mysql stabily on freebsd 5.3 (and 5.2.1 and 4.10 and a few others but 5.3 mainly) and never run into issues like this. i have a feeling its gentoo. you may wanna investigate into chaning os's.
 

·
Registered
Joined
·
2,012 Posts
sorry for the second post but i have some more info. just googled the problem for a while and it seems to be a bug in the gentoo compilation of mysql thats causing the problem...so now i really say change os...
 

·
I need more cars
Joined
·
2,360 Posts
Thanks for making all this info public.

Have you reviewed mysql's optimization chapter?

- Optimization - http://dev.mysql.com/doc/mysql/en/mysql-optimization.html
- Benchmark suite - http://dev.mysql.com/doc/mysql/en/mysql-benchmarks.html
- Optimizing the server - http://dev.mysql.com/doc/mysql/en/optimizing-the-server.html
- Tuning server parameters - http://dev.mysql.com/doc/mysql/en/server-parameters.html
- Disk issues - http://dev.mysql.com/doc/mysql/en/disk-issues.html

key_buffer_size and table_cache are the most important, but it appears that you have significantly increased these. Definitely figure out where you 1GB memory went first, then maybe consider trying out their benchmark suite. I personally haven't used it however so I'm not sure how much it would really help.

On the OS side of things, you may want to consider RHEL since that is what big corporations are standardizing on and writing apps for. You can be more sure that it'll be more stable. I personally bought it for my dedicated server even though I'm an avid Debian user for my home machines.
 

·
Only Tonic is Chronic
Joined
·
1,549 Posts
CyZooNiC said:
Both systems run gentoo linux.
I have had similar issues w/ PHP and mysql on gentoo...as soon as I switched it over to FBSD everything worked great.

Just my .02$
 

·
Premium Member
Joined
·
4,353 Posts
Definitley sounds like an issue with the OS not the hardware to me.

EDIT: From talking to a friend of mine whose spent alot of time working with servers, he's said he's ran into a TON of issues while using Gentoo, more-so than when he ran Redhat which he had a good amount of issues with as well. I'd say switch to Debian, or *BSD.
 

·
Welcome To The Machine
Joined
·
951 Posts
I know nothing about forum software. I'm just glad that the problem is out in the open and that there is an acknowledgment of the problem and attempts are being made to find a solution.
 

·
Registered
Joined
·
608 Posts
Zoo,

When you build the kernel, are you using genkernel or doing it by hand?

Either way, in order to get past 1GB, when you build your next kernel you need to enable the following option:

Processor Type and Features -> High Memory Support (4GB)

If you use genkernel, you can enable it via (genkernel all --menuconfig)

As far as the my.cnf file -- that looks great. I have not had any staiblity issues with MySQL 4.X on any of my gentoo boxes (over 7) except when there is a hardware issue. So that is the direction I would look.

The load on the mysql server, to be honest, shouldnt get above .40 very often. On the largest forum I run, the mysql server doesnt break a sweat (I'm using pconnect, BTW) -- It rarely gets above .40 load average. I've got 2 seperate machines using round robin DNS for load balancing Apache however, and the load ranges from .50 to 3 on both boxes depending on system load. However, even during these high usage periods, the mysql server rarely breaks any type of sweat.

So, the apache is definetly the machine that needs more beef. At least, in my experience.

From the errors I've seen on the forums, it usually appears to be mysql dying, or mysql table corruption. The emails sent to the admin address should tell you the error. If mysql is dying, you need to check what syslog/dmesg etc says when it happens. It could be running out of memory (with only 1GB, you definetly need to enable that other gig) -- and how much swap? If its table corruption, I would start looking at hardware. Possibly bad memory, or even a CPU problem. I had a dual P3 system fail on me about a year ago, and it acted very flakey... wasnt a cut and dry failure unfortunately.

BTW, If you talk to Eric, I'm interested in buying the forums from him and would love to speak to him about it. Either way, I'm here to help so with more information I'd be happy to offer more advice.
 

·
Demoted user
Joined
·
595 Posts
Discussion Starter #14
I'm will need to check if I compiled that into the kernel. I though I did. The system is running memory from two different manufacturers. Also I believe I'm running a not so new version of mysql and will update it when I get back.

thanks for the advice
 

·
Registered
Joined
·
608 Posts
CyZooNiC said:
I'm will need to check if I compiled that into the kernel. I though I did. The system is running memory from two different manufacturers. Also I believe I'm running a not so new version of mysql and will update it when I get back.

thanks for the advice
If its enabled there, and 2GB still isnt showing up -- if you happen to run lilo, that can also be the culprit (I use GRUB) -- Here's the LILO portion to change if thats the case:

In the kernel section of lilo.conf

Code:
append mem="2048M"
 

·
Demoted user
Joined
·
595 Posts
Discussion Starter #16
ok, I just checked my kernel config and it looks like I forgot to compile that in. I wont recompile it from Brazil as I'm not crazy but I will when I get back sometime next week.

-Stefan
 

·
Demoted user
Joined
·
595 Posts
Discussion Starter #17
I use grub
 

·
Master of the Weaves
Joined
·
555 Posts
Careful upgrading MySQL. If you're running version 3.xx.xx and want to switch to versions 4 or 5, it'll want you to rehash all the passwords for its accounts. The error message it gives when something tries to connect using an old password is not particularly helpful towards solving this problem.

You can, however, avoid the problem by invoking MySQL with the --old-passwords option and it will continue to use the old method.
 

·
Registered
Joined
·
9,645 Posts
Also, you sure the PIII machine is cutting it?

we replaced ALL of our P3 servers at a clients office (and they were xeon too) because they were slooow as shit.

They deal with the same type of stuff, but windblows and SQL server. 4 gigs of ram RAID, etc etc...the whole thing. Upwards of 300+ requests a second to sql, and those machines just couldnt handle it anymore.

Upgraded to dual p4 2.4GHz xeons with RIMBUS memory (this is the BIG help) and the processors are less than 20% utilized at all times.

The searches on here are way more complex as well....people search for "turbo" and you will get 5,000,000 results.

Not sure if mysql does clustered indexing on it (it should) but there should be one on the thread title, and then the body, then the thread title and body together. This will speed up performance as well, if it hasn't already been done.

And what is our load like on the webserver? One box may not cut it for more than 200 or so people on at one time for how much content it has to serve.
 

·
Registered
Joined
·
229 Posts
:stupid: i showed my geeky friend this thread he said that the p3's could be the issue since running a server. but he was sayin software too. but Zoo u should talk to eric lotter and tel him that ekool is interested in 3si or let ekool help u fix the issues with the server.
 
1 - 20 of 43 Posts
Top