Forum Upgrade Progress Update

Status
Not open for further replies.
FrgMstr

FrgMstr

Just Plain Mean
Staff member
Joined
May 18, 1997
Messages
55,602
I has been a long 6 months or so...... :(

We think we have finally found the problem with MySQL that is causing us issues. We hope to have a couple of options to fix the problem tomorrow and we hope the solution in place by end of next week. This means we should be up and running uber-freaking fast again and back to being stable like were were a year ago.

The problem seems to be a "compatibility" issue between 64-bit Linux, MySQL, and 8-way processors. And the fact of the matter is that when we configured this, there were very few people running dual 4-core processors to rely on for feedback experience and we were told by more than one expert that our configuration was good to go. I will tell you that we did a solid month's worth of research into this and at that time there did not seem to be any problems. Also, we used the configuration for a full 30 days in testing and then were in production a full 30 days before we ever saw an issue. This massive upgrade was not something we rushed into (for a change).

It has taken us about 4 months to figure out what is wrong. Now that we are 95% sure of the issue (which is random MySQL crashes) we should be able to put a plan of action in place to fix everything. At that point we will be back on track and we think the $15,000 worth of hardware (2 new mult-processor database servers as well as a new multi-cpu webserver addition (total of 3 webservers when we are done)) that has been purchased through this upgrade will really shine.

Please be a bit more patient with us, we should have the search server turned back on next week and we will be running at full capacity. At that point, if you have slow page loads, you can most likely blame it on your ISP instead of use for a change! ;) I do appreciate you guys being patient and hanging in there with us. Thank you.
 
Just don't bugger the whole thing up, Kyle ;)


edit: I know people give you guys a hard time sometimes when the forums go down, but you guys do a great job. Sometimes people don't realize the expense of the hardware and amount of time put into something allowing us to chit-chat with each other. :cool:
 
Thanks for all the [H]ard work you and your crew have put into this place. Your efforts are greatly appreciated. :)
 
Our own personal exterminator.... Thanks Kyle, for keeping it [H]ard. :cool:

(I don't think that came out right.... :rolleyes: )
 
Glad to hear it's working!

On a side note, didn't this project start with 64-bit BSD? What caused the change?
 
InorganicMatter said:
Glad to hear it's working!

On a side note, didn't this project start with 64-bit BSD? What caused the change?
Click to expand...

Our webservers for HardOCP(3) and HardForum(3 soon) are 64-bit BSD. HardOCP database servers(3) are are 64-bit BSD. HardForum database servers(2) are 64-bit Linux. We had testing issues with 64-bit BSD/MySQL/VBulletin issues if I remember off hand.
 
About time :D

Seriously, I've never had an issue with HF. It's faster than a lot of forums out there and they are apparently not having problems.

It must have been very frustrating for you guys. Glad to see you were able to solve that issue. Must be a sigh of relief.
 
What seems to be the fix? that way if any of us ever get systems that require all that we can do a quick search and find out how to fix it.
 
It's great to hear how hard you guys are working, it is appreciated.

The update is also another thing that is really appreciated, thanks for letting us know what's going on!
 
Archer1212 said:
What seems to be the fix? that way if any of us ever get systems that require all that we can do a quick search and find out how to fix it.
Click to expand...


We are still waiting for a couple of options for MySQL now, but the simple options are to move to 32-bit OS, or disable one of the processors currently. I would rather move to the 32bit OS obviously, but we need to look at that the memory footprint the search server is using, so right now I dunno, we have yet to see all the options. Hope we know more tomorrow.
 
I was just thinking about how the forums have been working pretty well for me lately as I read through this thread, but as soon as I clicked away the forums disappeared. The irony...

I'm amazed at how hard you guys work to keep this place going, it's definitely worth the effort.
 
Let's blame Sun Microsystems for buying MySQL and making it quail in fear, causing instability in the [H] server farm. :)
 
im glad you guys finally figured out what was wrong. when things do get all the way iron out is there a potability of turning the quick reply back on?
 
Congrats on the diagnosis!

202276
 
Gratz on figuring it out man. That had to not be fun to trouble shoot.

Thanks for doing the [H]ard work you do to keep this place a haven for geeks. You rock.
 
Thanks man for the dedication to resolving the issue.
I know how much of a headache things like this can be.

No wonder you seemed to be in a good mood lately, you figured out what was wrong with the forums. :cool:
 
Since you're going to be messing with the servers can we get pics of the server room and dare I ask inside the servers? :eek:
 
munkle said:
Since you're going to be messing with the servers can we get pics of the server room and dare I ask inside the servers? :eek:
Click to expand...

I second the request for server porn. It never gets old even when you work on them.
 
MrMike said:
So you either lose memory (assuming the servers have 4GB+, I don't remember if you said in another post) and have to rebuild, or lose four cores. Hopefully you have an easy way of displaying performance data so you can see which way to go.

If this is just the search server I'd also be pushing for the 32-bit OS as I imagine it tends to get pummeled with requests at times. However isn't disabling one of the chips simple in this case, so you could bring the search server back up immediately in this sense? It's not exactly a mission critical application, so some downtime isn't the end of the world.

Good luck.
Click to expand...


Yeah, the server was bought to handle searches that are CPU intensive. And yes, it has 8GB of ram in it.....

We checked and did not see a disable processor ability in the bios, and I did not want to start yanking a perfectly working box apart when we found out it is a sofware issue.
 
uzor said:
Indeed. However, their Co-location facility does not allow cameras on-premesis, IIRC, so no pics of the data center. :(

Click to expand...

When there's danger involved pics must be taken, think of it as mission impossible! :p
 
HOCP4ME said:
Hmm. A "compatibility" issue between OS, software, and hardware...

That certainly explains a lot of computer problems.:):p:D
Click to expand...

At least it's not a compatibility issue between user, os, software and hardware :p
 
rgratto2 said:
$15k? Holy shit! Maybe its time I subscribed...
Click to expand...

Hehe, yeah, once you to where there are 2000 people on your forums all the time, it stops getting to be a simple solution to speeding things up without cutting large hunks of functionality out.

I think the hardware we bought this year will last us a good 3 years and give awesome performance. Depending on what features VB puts in over the next years will determine that as well.
 
Status
Not open for further replies.
Back
Top