why is the forum always down?

Nasty_Savage said:
Database Management and hardware Jedi are two different things. If I recall, the [H] servers are fairly hefty spec wise. I think the baby bs going on here is retarded...this isn't a home box /hobby forum. Its not like you have to pay for it (and paying for General Mayhem is less then nothing). Now that its back and my "shakes" are going away, I can resume goofing off at work. Thanks Kyle/Cliff and all the rest. :D

I am interested in knowing if this came about from an upgrade to Redhat? I thought I read that on the front page. Hopefully the experience can be relayed in a post for us interested nerds ;)
Click to expand...
Actually we were exposed and did not have backups of the database when it cratered this time. We are in middle of big move to FreeBSD. Shit happens. We are working to get this exposure corrected. The three webservers that handle our forums have been moved to FreeBSD, but the forum is still on Redhat 9. We have been afraid to move it due to unknown complications and quite frankly we have some other priorities that pertain to the HardOCP site.

I think the hardware we have is very robust and when the forum cratered, we were not under load. I was using Vbulletin tools to correct tables and optimize them inside Vbulletin and I had turned the forum OFF.

The database is in a single 4U box that has about 125 CFM of airflow through it. Dual 2800 Athlon XPs, 4GB of Ram, with three RAID arrays. All disks are 15K RPM U380SCSI drives 36GB in size.

3 Drive RAID 5 Array for OS.
3 Drive RAID 0 Array for Tables.
2 Drive RAID 0 Array for Index.

OS is on its own LSI SCSI card and the Tables and Index share a 2 Channel card (all 64-bit).

Watching loads and such, we do not think we have any hardware issues.


IN FACT AND THIS IS THE STRANGE THING, we have not seen a database crash with the servers under load since earlier this year. The MySQL database corrupts tables usually when there is hardly any use going on. (Like it did this time.) If you can help us figure this out we are all ears. The HardForums have been working GREAT under load as we generally now have a three to six hour period a day where over 1000 people are on at once.....
 
Oh also, the three webservers are all dual opterons with 4GB of ram.....They are all new boxes and they do also handle HardOCP page serves as well.

Right now with HardOCP and HardForum we are seeing this load.

Server Load Averages0.39 0.30 0.27 | 531 Users Online (235 members and 296 guests)


It generally stays there all day long. Maybe get up to 1 if we are running a big article.
 
Well I am not a MySQL expert at all and have never played with vBulletin.

However since it only craters during low usuage time, do you have a processes that are set "nice" that possibly do some sort of reindexing?

Just a shot in the dark.
 
m1abram said:
Well I am not a MySQL expert at all and have never played with vBulletin.

However since it only craters during low usuage time, do you have a processes that are set "nice" that possibly do some sort of reindexing?

Just a shot in the dark.
Click to expand...
No, not to my knowledge. We use (my)isamchk every 30 minutes to check and fix tables. Since we implemented this several months ago, the downtime here went way down.
 
vapb400 said:
See if you can get ahold of Tony the administrator of PBNation.com. That site runs VB and has almost 125,000 members, but with a little less as the most online record. Their site works 99% of the time, but they have alot of money generated from ads and supporting members. Tony is a great guy and I'm sure he'd be glad to help you out.
Click to expand...
Interesting. He is still running VB 2.3.5. We could not do that here as the folks trying to hack it would bring it down very quickly.
 
People can't bitch about [H] w/o being an asshole. You feel that because you have X hundred posts here you have some sort of right to complain. You don't. You don't pay for bandwidth, server space, Kyle and Cliffs loss of sleep. That is the truth plane and simple.

Was I annoyed the forum was down? Yes. I am glad they are back up? Yes. I am I glad there are people (kyle, cliff) willing to give up women for my personal forum pleaure :D

Thanks again Kyle and Cliff. Love the place. Finally glad after years I decided to regsiter and start participating. Good forum with a low noise level.

-tReP
 
I am glad that the servers are back up and for those of you who are bitching about it all I have to say is a word of wisdom my mother gave to me....if you don't have anything constructive to say STFU
 
Hey Kyle since you have all the SQL on RAID 0 I assume you backup data. How often and what media do you use to back this all up incase you need to do a full restore?
 
Hehe.. this thread cracks me up. Half the people here think since they can overclock ther PC and install a pirated copy of windows on it that they have all the answers to the issues at hand.

Way to call them on it Kyle. I doubt your going to get many replies to your offer to have them come and actually put their ass where their mouth is. I am not refering to the few posts that have what seemed like legitimate offers to help.
 
SnowPunk98 said:
Hey Kyle since you have all the SQL on RAID 0 I assume you backup data. How often and what media do you use to back this all up incase you need to do a full restore?
Click to expand...
If you read the thread, you will see that I already addressed this. Quite frankly, this has nothing to do with the topic at hand.
 
Astronutty said:
My boss is very disappointed with the forums working...
Click to expand...

Yep, the Department Of Labor reported record increases last week with efficiency in the office workplace. Now it is going back down. :D
 
Yeah! It's up again. Woo-hoo!

I think it's funny of all the people saying they'll donate.

I don't think I've seen a single post from most of them on the General Mayhem forum.
 
Met-AL said:
Hehe.. this thread cracks me up. Half the people here think since they can overclock ther PC and install a pirated copy of windows on it that they have all the answers to the issues at hand.

Way to call them on it Kyle. I doubt your going to get many replies to your offer to have them come and actually put their ass where their mouth is. I am not refering to the few posts that have what seemed like legitimate offers to help.
Click to expand...
Nope, I'm pretty sure that for some of us the statements still hold. The downtimes are a bad precidence and without ever knowing whats going on, the people that like this site are going to complain. Not too many of the forums I'm involved in give so little information about whats going wrong when it goes wrong.

edit: stop kissing kyles ass. maybe he likes it but its just obnoxious
 
I would like to thank Kyle and Cliff and everyone who had the forum down last week. Somehow this coincided perfectly with my finals schedule and went down right when I needed to really start working and came back right after my last final. Myself and my grades thank you guys! Perfect timing!
 
Might I ask in regards to the OS of choice:

- Why FreeBSD? I know of the fact that there are many Linux distros out there, I just want to know why you singled out on FreeBSD.
-Have you considered other distros? (Mandrake, Gentoo, etc.)

I'm not saying DON'T go with FreeBSD, once again, I was just curious as to why you chose it. Also, is there another db program you could consider other than mySQL?
 
JSClark said:
Might I ask in regards to the OS of choice:

- Why FreeBSD? I know of the fact that there are many Linux distros out there, I just want to know why you singled out on FreeBSD.
-Have you considered other distros? (Mandrake, Gentoo, etc.)

I'm not saying DON'T go with FreeBSD, once again, I was just curious as to why you chose it. Also, is there another db program you could consider other than mySQL?
Click to expand...

i cannot answer their choice of freebsd..but MySQL i can answer...vbull only runs on mysql right now. I do not know if any other versions are in the works or not.
 
JSClark said:
Might I ask in regards to the OS of choice:

- Why FreeBSD? I know of the fact that there are many Linux distros out there, I just want to know why you singled out on FreeBSD.
-Have you considered other distros? (Mandrake, Gentoo, etc.)

I'm not saying DON'T go with FreeBSD, once again, I was just curious as to why you chose it. Also, is there another db program you could consider other than mySQL?
Click to expand...


keep in mind that FreeBSD is BSD, not Linux. It works differently in the kernel space than linux.

As to why they chose it is up to Kyle to answer.
 
j4zzee said:
If my failing memory serves me correct? I think I remember reading a blog from one of the Yahoo developers about problems with FreeBSD and MySQL. Specifically threading issues on all 4.x versions of FreeBSD, and that Yahoo did all of of their MySQL development on Linux.. This was sometime ago, so maybe FreeBSD 5.x has resolved the high load threading problems with MySQL? If I find his blog in my bookmarks, I will post the link...
Click to expand...
We are talking to one VB owner who is seeing GREAT results with VB 3.X and FreeBSD 5.X and we are planning to move in that direction ASAP.
 
emorphien said:
Nope, I'm pretty sure that for some of us the statements still hold. The downtimes are a bad precidence and without ever knowing whats going on, the people that like this site are going to complain. Not too many of the forums I'm involved in give so little information about whats going wrong when it goes wrong.

edit: stop kissing kyles ass. maybe he likes it but its just obnoxious
Click to expand...
Hard to tell you what is wrong, when WE DONT KNOW THE PROBLEM.
 
JSClark said:
Might I ask in regards to the OS of choice:

- Why FreeBSD? I know of the fact that there are many Linux distros out there, I just want to know why you singled out on FreeBSD.
-Have you considered other distros? (Mandrake, Gentoo, etc.)

I'm not saying DON'T go with FreeBSD, once again, I was just curious as to why you chose it. Also, is there another db program you could consider other than mySQL?
Click to expand...
FreeBSD has proven to be more secure than Linux and it also is a much leaner install that is giving us much better performance than Linux on our webservers. Research pointed to that, so we went with FreeBSD after our web servers were tested on FreeBSD for the DOOM3 Guide. We are now seeing some performance and security benefits on the database side as well....that is why.


As for other databases, MySQL is the only one that works with VB, and other solutions we have looked at start to cost a LOT OF MONEY. If you find something as robust and stable and affordable as VB, please send me links.
 
Do you have an optimized my.cnf file for all the database settings? Query caching, mmcache for php etc. could help greatly for performance. Since the problem wasn't load related this information may be irrelevant.

Check out the setup that wikipedia uses. They have one of the most optimized server farms I've seen. I'm sure their configuration notes would be of some help to you guys.

See=
http://meta.wikimedia.org/wiki/Wikimedia_servers

Obviously they have more financial resources, but their clustering model seems to work remarkably well. They have handled their increased useage with extreme effectiveness. Alot of good info on that page.


In any light, I wish you the best. This is a great site!
 
fireform90 said:
Do you have an optimized my.cnf file for all the database settings? Query caching, mmcache for php etc. could help greatly for performance. Since the problem wasn't load related this information may be irrelevant.

Check out the setup that wikipedia uses. They have one of the most optimized server farms I've seen. I'm sure their configuration notes would be of some help to you guys.

See=
http://meta.wikimedia.org/wiki/Wikimedia_servers

Obviously they have more financial resources, but their clustering model seems to work remarkably well. They have handled their increased useage with extreme effectiveness. Alot of good info on that page.


In any light, I wish you the best. This is a great site!
Click to expand...
Thanks for the heads up, this is surely something we will look at.
 
I'd be glad to help in any way possible. This site is a great resource, and has been for years :)
 
I run IPB on a 500Mhz P3 with 1.5GB of RAM and two 10K RPM SCSI drives in RAID1 using Win2000 Server as the OS and we don't even put a dent on the system....but we also don't come close to the [H]'s traffic.
 
JSClark said:
The one site thats advertised in my sig (and I am a mod at) runs on Invision PowerBoard. If you haven't looked into that, I can talk to the admin and find out whatever necessary details you might need.
Click to expand...

Invision is good for smaller sites. Yours has 260+ members, this has 32,000+. Vbulletin is definately the way to go. It just may need to be optimized. The features are *much* better in Vbulletin imo anyway.
 
Well, my thing is I don't know the ins and outs of BB software, although if I were to sit down with it, I can familiarize myself quickly and figure out how to work it. Moreso, I don't mind doing research into a possible better solution that can minimize problems with a board this big.

I'll see what I can pull up sometime in the next few days if need be. I'm working until like 10 or 11 every night up until Friday.
 
Maybe talk to other forum admins and see what set up their using??
 
loai said:
Maybe talk to other forum admins and see what set up their using??
Click to expand...
Yes, we have done this since day one and have one developer now working on specialty plugins for the software. We have implemented every tweak we could justify that has been suggested to us through reliable channels, most VBbull's own forums. We are members there and do stay up to date with the changes in the software and hacks and tweaks that others do. The last list of "musthave" tweaks that were given to us by members that used to call this board home were nothing more than tweaks that are already known on the VB forums and documented there. In fact, we had all of them implemented.

I am very confident that if we disabled searching, as others have done, we could sail along pretty easily. Remember that most people do come to our forums LOOKING FOR HARDWARE ANSWERS and use our search feature. Most do not come here to post conversations like "What sounds good for lunch today?" and "Only "X" shoplifting days 'till Kwaanzaaa," that lead to hundreds of posts.

Just remember that when making comparisons to other forums stats do not tell the entire picture. Anyway, I thought that was worth pointing out...
 
M|22 said:
wow, that's so much more than we currently have :eek:
i'd kill to have a box like that :p

anyway, i'm sending you a PM with all of my contact information. feel free to forward to cill or whomever :)
Click to expand...
Cool deal, I will get with Cliff and get a time and date set up for you to come in and diagnose our problems and get them fixed.
 
I have seen most forums that have some sort of Google banner, that they use in turn, search for whatever content that the user needs on that specific forum / site. Perhaps, if not already implemented or talked about, that might be a viable solution if it comes down to disabling serach?
 
JSClark said:
I have seen most forums that have some sort of Google banner, that they use in turn, search for whatever content that the user needs on that specific forum / site. Perhaps, if not already implemented or talked about, that might be a viable solution if it comes down to disabling serach?
Click to expand...
Thanks for the suggestion, but we will not be disabling our search function on this forum.
 
M|22 said:
one additonal question... how large is the current database?
Click to expand...
Off the top of my head I think it is right under 1.4GB.
 
so why wont [H] start a fund raiser or something to gain some extra cash for servers and what not. THis could be a good way for the members of [H] to show their appreciation to Kyle and the rest of this wonderful team. Maybe you can make the SNs of the ppl. who donate in color, signifying that they are special or something. Id deffinatley send 20$ bucks since i use this forum alot for its hardware related issues and Gen. May for just conversing with ppl.
 
I really think alot of people here need to chill!
YES, the [H] has gone down alot recently.
YES, the [H] gets just as much traffic as some forums that never go down.

But i think your all forgetting that while people put the effort in for tis place, they dont have too!
This forum is a HUGE source of info, its saved mine, and many others asses when we get a problem, and i find it a sad day when people are DEMANDING answers!

Neigther Hardware nor software is perfect, both can have flaws, SQL is a fine system, when its used properly, but overlooking a minor thing can cause you a weeks downtime, these things happen!

All im going to say is all of you chill, stop moaning, give them a break, enjoy the forums, and accept things go wrong!

Go back to your [H]ard ways and work on keeping the forum a huge pool of knowledge! :D
 
vapb400 said:
That is strange that the DB problems happen at low usage and if I was mySQL buff i would love to help you figure out the problem, but unfortunately I'm not :(.

Anyways, good job, and keep up the great work.
Click to expand...
Thanks for the kind words that everyone has shared. It is nice to know that you guys appreciate what we do here.

As for this last big crash, I am fairly sure that it had to do with optimizing/defragging the tables through the VBull Admin Control Panel. I do this all the time, but this time I think it timed out through the web interface and lead to the issues we have had.

As for the table corruptions that we see every week or two, as mentioned they always seem to come at low usage times. We are going back over cron to see if we have something running amock there, but I really do not think so.

We do run "myisamchk -r" every 30 mintues during normal operations. We did some load tests a few months ago and saw no bad come from this. But if you know different, please let us know.
 
chronic9 said:
so why wont [H] start a fund raiser or something to gain some extra cash for servers and what not. THis could be a good way for the members of [H] to show their appreciation to Kyle and the rest of this wonderful team. Maybe you can make the SNs of the ppl. who donate in color, signifying that they are special or something. Id deffinatley send 20$ bucks since i use this forum alot for its hardware related issues and Gen. May for just conversing with ppl.
Click to expand...
I personally do not think more hardware is the solution here.....thanks for the suggestion though. I would rather get the problem fixed.
 
tunaman said:
Is it possible that there is just something wrong somewhere in the software that is not beeing seen? Have these corruptions been seen elsewhere on different sites?
Click to expand...
I am not ruling out any possibilities. We are not aware of any sites with a similar problem. Then again most VBull sites are not near as large as ours or used in the same manner.
 
That is the whole point of this exercise. Kyle has invited anyone with experience in this area to examine the forum software settings and see if something is out of place that may have been overlooked :cool:

tunaman said:
Is it possible that there is just something wrong somewhere in the software that is not beeing seen? Have these corruptions been seen elsewhere on different sites?
Click to expand...
 
You must log in or register to reply here.
Back
Top