Outage Update

Status
Not open for further replies.
All signs point to a configuration made by the VBulletin engineer.....the same guy that fixed it.

The fact is that nobody at VB has tuned for the hardware we are running here so there is basically no one that has experience. Honestly most VB forums are small and run on ragtag hardware. Even the big ones generally do not have 8-way CPU and extensive RAID arrays like we have. We are truly looking to keep this database box for a number of years. So it comes down to one VB forum is NOT like the other. But there are a lot of folks with bigger boards that share information, just is that many times solutions for one are not solutions for others. :( I wish there were some silver bullet for making this shit work, but even VB cannot point you to a source to hire to FIX or CONFIG the damn things. Everyone is still on their own basically. That said, VB has been a great source of help and has done what they can to help us and I greatly thank them for that.

This was a massive upgrade for us....new database server, new web servers, new OS, new MySQL, new VB....I wish it could all be simple but it just is not. Hopefully we have found our one weakness after a month solid successful run. Cross my fingers..
 
Thanks for all your hard work.
This forum for me is like a cup of coffee..I really need it or I'm grouchy all day :)
 
Blue Falcon said:
It's a wonder things haven't been worse.
Click to expand...

In all honesty, I 100% agree with that statement. We literally pulled two cabs worth of servers out of the Planet and stacked them over at Dataside one afternoon. It has all gone amazingly well considering the absolutely lack of planned attack.
 
I've noticed since the outages that there aren't as many members online as there once was. I remember seeing 120+ member jsut on the Display category. I'm on a new account so I'm back to N00bie. I first joined early last year. adn things just haven't been the same.

BTW maybe you all ought to do soem thread pruning. The database is probably inthe gigs if not hundreds of megs.
 
iGamer said:
I've noticed since the outages that there aren't as many members online as there once was. I remember seeing 120+ member jsut on the Display category. I'm on a new account so I'm back to N00bie. I first joined early last year. adn things just haven't been the same.

BTW maybe you all ought to do soem thread pruning. The database is probably inthe gigs if not hundreds of megs.
Click to expand...


Thanks for the suggestion.
 
eva2000 said:
Slight correction Kyle, the my.cnf config you had during the crash was no where near like the one I recommended for the prior fix. AFAIK, Cliff said he he hired some other guy to do that my.cnf and it was in my opinion what caused your problems - overallocation of memory to mysql memory buffers causing out of memory (mysql issues). I had to correct that my.cnf and pass on the config to Cliff and that's the my.cnf you're probably using now :)

As to the magic silver bullet, there doesn't exist one because as you said, every forum and site has different forum member usage patterns and activity. Tuning a server is a constant act of monitoring and re-tuning to mean the traffic patterns the server is handling. The more consistent those patterns are day to day, the longer those tuned settings stay relevant and the longer the duration before you need re-tuning :)

vB does have a server tuning forum where vB clients can post optimisation request for BASIC advice only to get them up and running optimally. But yes there is no extensive tuning/tweaking via official vB support. vB staff are allowed to do things on the side though which I have been for the past 6+ yrs :)
Click to expand...

Those changes that were made to your configs were made in an attempt to salvage AFTER the crash.

And please don't take my statements in the wrong way. Many people here have been impacted by the events and want answers. The fact is that I don't have the specific answers and am only trying to give our users some insight into what goes on around here. To give our readers a hint at your talents, I think VBulletin would be pretty much screwed without you, because you seem to be the ONLY go to guy they have when it comes to configuring larger forums.
 
eva2000 said:
Ah i see... thanks for clearing it up. As to the actual crash you had not sure what the problem could be as I haven't actually taken a look into the servers. Did you go back to dual quad core intel db server or did this crash occur on the dual AMD dual core opteron db server ?
Click to expand...



We moved over to the dual quad some time ago, it was stable, then did the latest VB upgrade. That was four weeks or so ago, then the crash. Up until the latest issues we had been running solid on the dual quad for weeks.

It troubles me that you are not already clear on this given your involvment. If you guys would copy me on the emails, which neither cliff or you has been doing, maybe I could help out a bit on specifics. [email protected] :)
 
Well, things are still flaky as hell. At this point in time whenever I click a link for any forums other than Forum Help & Feedback I get a database error.
 
Wow, after hearing everything that's happened, I can't believe the forums are still running.

I mean, after a complete host move, complete server move with new processor architecture, new OS, and new VB software, and a second VB upgrade, all that's happened is we lost two weeks of data and we have to occasionally hit the refersh button.

I'd say that's pretty darn good.
 
HOCP4ME said:
Wow, after hearing everything that's happened, I can't believe the forums are still running.

I mean, after a complete host move, complete server move with new processor architecture, new OS, and new VB software, and a second VB upgrade, all that's happened is we lost two weeks of data and we have to occasionally hit the refersh button.

I'd say that's pretty darn good.
Click to expand...

yea I agree good job guys :) btw kyle how much bandwith*traffic* does [H] gets a month?
 
Marvelous said:
This is terrible. You guys need a new tech guy to fix the forum. :(
Click to expand...
I do hope you are kidding. Have you read the thread? It's not exactly [H]ardOCP's fault. They are utilizing VB at a very large scale and with high end hardware, which is apparently new territory-even to the VB support staff. I suppose you'd have [H] buy VB so the forums could be "uber cool".
 
Database problems are no fun, so you have my sympathy. Hopefully they'll be resolved soon, I was wondering why all the blank pages were coming up at 3am! :rolleyes::D
 
This is all uncharted territory and they are literally writing the book as they go. It will take time and work, but it will get done. :cool:

Marvelous said:
This is terrible. You guys need a new tech guy to fix the forum. :(
Click to expand...

ClariorHincHonos said:
I do hope you are kidding. Have you read the thread? It's not exactly [H]ardOCP's fault. They are utilizing VB at a very large scale and with high end hardware, which is apparently new territory-even to the VB support staff. I suppose you'd have [H] buy VB so the forums could be "uber cool".
Click to expand...
 
madmat said:
Well, things are still flaky as hell. At this point in time whenever I click a link for any forums other than Forum Help & Feedback I get a database error.
Click to expand...

had that problem a few days ago. lately I "open in new tab" and as long as I wait
a few seconds for the topic to appear in the tab heading, vs (database error) multiple
tabs with no "database error"'s are a given. surprised that database errors are still
occuring as of yesterday.
 
Enough of the debating, it works for most of us and if it doesn't speed you up then fine...don't use it. End of discussion.
 
I cant see this post going over well, but what the hell, here it goes...

Have you considered a new database? there could be something seriously wrong with the transition over to this new server. an old forum I was part of (suprnova if you must know :p ) had to upgrade the IPB version and move over to a new server, the DB couldnt be saved at all...

Its a grim idea, but maybe set it up on sphinx like before... that was a great way to load test (500000 posts in what, 3 days? :D )

If thats even a considered option, id be glad to try and round up some important threads to carry over.

im not sure if the user database is different from the post database, but if it is, i cant see a reason why anyone other than those with immediate trouble, or the FS FT crowd having a problem with it for the sake of getting the place presentable again.

The amount of time and money you guys are putting in to this free service for us is amazing, and I have to give you major props for that, but I think enough of your time has been spent on it that this should be a serious consideration.
 
beowulf7 said:
Sorry to hear about all the tech difficulties HF has had to endure the past couple months. :( Could the problem be with MySQL; if so, perhaps using Microsoft SQL would alleviate those issues. I'm a fan of open source products and I know MySQL is used by many organizations, so I don't think the problem lies with that. But it is a possibility. Good luck!
Click to expand...

If the current forum's query's are all optimized for mySql's quirks and limitations and the devellopers haven't already already ported the queries over switching to MsSql is probably not going to work smoothly even if the forum was starting to hit mysql's limits. The total amount of data shouldn't be a problem, even access can handle a few gigs before having problems *rolls eyes*, but I don't know how hard the number of querries are hitting it.
 
DanNeely said:
If the current forum's query's are all optimized for mySql's quirks and limitations and the devellopers haven't already already ported the queries over switching to MsSql is probably not going to work smoothly even if the forum was starting to hit mysql's limits. The total amount of data shouldn't be a problem, even access can handle a few gigs before having problems *rolls eyes*, but I don't know how hard the number of querries are hitting it.
Click to expand...

I was so glad when I reached that first period. *Phew*

Anyway, at the very least, the forum seems to be alright now... Even if it is missing 10^n posts, at least people can still come here and use it. ^.^
 
So, what's the latest news? Just checking in, since Hardfolding is still down. Are you still working on the Hardforum problem?
 
One of the things people tend for forget and certainly it is easy to forget sometimes is that this whole thing is Kyle’s source of income. His day and night job so to speak.

Obviously the [H]ardOCP front page gets immediate priority since it has the most hits and produces the most income. The [H]ardforum is a secondary source and no doubt a pretty good one, so it would be next on the list of priorities.

To date Kyle has chosen to not advertise on the [H]ardfolding stats site so that will be the last project on the list unless I’m greatly mistaken. I personally would have no objection what so ever supporting advertising on the stats site. All the other sites like ours do and no one seems to object.

Frankly, the ad’s are a necessity and I always make my Newegg purchases through the [H] links because I like what we have and that is the best way I know to continue supporting things.

I doubt anyone here wants things fixed more then Kyle.:)


 
Is there anyway people can help by donating $$$ for the forum/site's existence?
 
LarsonLake said:
So, what's the latest news? Just checking in, since Hardfolding is still down. Are you still working on the Hardforum problem?
Click to expand...


I have no idea what is up with HardFolding.com, I have not been made aware of any issues.
 
Red Falcon said:
Is there anyway people can help by donating $$$ for the forum/site's existence?
Click to expand...

Nope, not a money problem, an experience and intelligence problem couple with lacking diagnostics for our software.
 
Hmmmmm.....maybe I phrased that poorly. I didn't mean to imply that you weren't working. I just meant that hardfolding being down was kinda a flag that something was wrong, despite the fact that hardforum seems to be cruising along despite whatever forum server problems you've been having.
I was just hoping that you could loop us in on how things were going, since it sounded like you found the problem.
 
LarsonLake said:
Hmmmmm.....maybe I phrased that poorly. I didn't mean to imply that you weren't working. I just meant that hardfolding being down was kinda a flag that something was wrong, despite the fact that hardforum seems to be cruising along despite whatever forum server problems you've been having.
I was just hoping that you could loop us in on how things were going, since it sounded like you found the problem.
Click to expand...


We do not maintain HardFolding.com. If there is something wrong on our hosting end, King would need to drop me a mail.
 
Status
Not open for further replies.
Back
Top