Simo
21st October 2008, 21:41
All,
For a number of weeks now myself and Mark have been trying to troubleshoot the semi-regular, momentary interuptions to server connectivity, which have been happening. We've had a few theories about what might be happening, but so far have been unable to prove anything.
The current server hardware is (on paper) much, much higher spec than should be required to handle the (rapidly) increasing demands of Saxperience, however we are NOT ruling out the fact that server hardware could be the issue!?
So, a systematic approach is going to be used to finally try to get to the bottom of the issues we are experiencing for a couple of 5-30 minute periods of the day. Between the stages, the situation will be monitored and non-problematic changes will be reverted, where appropriate.
Stage 1 - Adjust some additional server settings to reduce DB load. A peak load re-directing service has been initiated to refuse connections when load is VERY high... Hopefully this won't become a nuisance, if it does changes can be made, but this will also confirm a problem - COMPLETED this evening.
Stage 2 - Disable the Pager system.
I am fully aware this will NOT be a popular move, but the facts are that we have had to butcher the pager system to make it run anything remotely like efficiently AND the pager system has known issues - it should have actually been deleted approx. 12 months ago, but I didn't want to do this as I know how useful it is. Hopefully the pager system will prove NOT to be a problem, meaning it can be re-enabled. Be aware - the number of sites that this pager service has killed is large... its a useful, but horrible (from server side) add-on. COMPLETED this evening.
Stage 3 - De-bug VBExperience system and target slow DB queries.
Optimise VBExperience settings to reduce server load at peak times of day. COMPLE - this appears to have resolved the issue.
Stage 4 - Disable iTrader and Arcade systems, forum upgrade and DOS search.
I do not think these mod's will add significant load to the server, but this needs to be tested out. After stage 4 is complete, the community will be a pretty fresh VB install, with a few customisations thrown in. Further investigations into DOS attacks to be made. Forum upgrade to latest patch version to be performed.
Stage 5 - If we are still having connectivity issues after Stage 3 has been completed then an even more advanced server will be purchased with a DIFFERENT provider. The spec of the current server should be more than adequate... but time will tell. Switching to a new server will be a frustration and will be fairly involved, but we've been there before and we aren't afraid to go there again ;)
We should be at a Stage 4 decision point by this coming weekend.
Stage 6 - If after switching to the highest spec server the community can afford, there are still issues, then this will show the issue is NOT hardware related and is a complex coding/config issue, made more of an issue with increased user-load. At this point, its likely the main website will be disabled awaiting v4, and the forum and associated tools will live on... hopefully trouble free.
Stage 7 - Cry, stamp feet, sulk.
I don't think we'll get to stage 7 ;)
What Im effectively saying is that we WILL resolve this issue. We've always invested in top quality hardware to make sure we are over-spec'd and that has shown in the site performance we've enjoyed for many years.
Whilst we dont fully understand the issues YET... we will and things will return to normal, if not be even better than before. :A:
So, please bare with us (I know Ive said this for the last few weeks). You may have to go through some changes with features and functionality that you wont like... but we're on the same journey, together.
Lets see where we end up...
Simo
For a number of weeks now myself and Mark have been trying to troubleshoot the semi-regular, momentary interuptions to server connectivity, which have been happening. We've had a few theories about what might be happening, but so far have been unable to prove anything.
The current server hardware is (on paper) much, much higher spec than should be required to handle the (rapidly) increasing demands of Saxperience, however we are NOT ruling out the fact that server hardware could be the issue!?
So, a systematic approach is going to be used to finally try to get to the bottom of the issues we are experiencing for a couple of 5-30 minute periods of the day. Between the stages, the situation will be monitored and non-problematic changes will be reverted, where appropriate.
Stage 1 - Adjust some additional server settings to reduce DB load. A peak load re-directing service has been initiated to refuse connections when load is VERY high... Hopefully this won't become a nuisance, if it does changes can be made, but this will also confirm a problem - COMPLETED this evening.
Stage 2 - Disable the Pager system.
I am fully aware this will NOT be a popular move, but the facts are that we have had to butcher the pager system to make it run anything remotely like efficiently AND the pager system has known issues - it should have actually been deleted approx. 12 months ago, but I didn't want to do this as I know how useful it is. Hopefully the pager system will prove NOT to be a problem, meaning it can be re-enabled. Be aware - the number of sites that this pager service has killed is large... its a useful, but horrible (from server side) add-on. COMPLETED this evening.
Stage 3 - De-bug VBExperience system and target slow DB queries.
Optimise VBExperience settings to reduce server load at peak times of day. COMPLE - this appears to have resolved the issue.
Stage 4 - Disable iTrader and Arcade systems, forum upgrade and DOS search.
I do not think these mod's will add significant load to the server, but this needs to be tested out. After stage 4 is complete, the community will be a pretty fresh VB install, with a few customisations thrown in. Further investigations into DOS attacks to be made. Forum upgrade to latest patch version to be performed.
Stage 5 - If we are still having connectivity issues after Stage 3 has been completed then an even more advanced server will be purchased with a DIFFERENT provider. The spec of the current server should be more than adequate... but time will tell. Switching to a new server will be a frustration and will be fairly involved, but we've been there before and we aren't afraid to go there again ;)
We should be at a Stage 4 decision point by this coming weekend.
Stage 6 - If after switching to the highest spec server the community can afford, there are still issues, then this will show the issue is NOT hardware related and is a complex coding/config issue, made more of an issue with increased user-load. At this point, its likely the main website will be disabled awaiting v4, and the forum and associated tools will live on... hopefully trouble free.
Stage 7 - Cry, stamp feet, sulk.
I don't think we'll get to stage 7 ;)
What Im effectively saying is that we WILL resolve this issue. We've always invested in top quality hardware to make sure we are over-spec'd and that has shown in the site performance we've enjoyed for many years.
Whilst we dont fully understand the issues YET... we will and things will return to normal, if not be even better than before. :A:
So, please bare with us (I know Ive said this for the last few weeks). You may have to go through some changes with features and functionality that you wont like... but we're on the same journey, together.
Lets see where we end up...
Simo