Server Performance

[DZComposer]

The server has been performing very crappy lately. I have not really been 100% happy with the current hosting situation, and I do not feel I am getting my money's worth.

I am considering looking into switching server hosts. I can't give a timeframe on this, but I am going to start looking.

It has been suggested to look into VBulletin as opposed to IPB. VBulletin has recently changed their pricing structure, and I find it to be more attractive than the prior one. VB's latest suite looks damn-nice, but there is one stickler in the pricing: Support costs almost as much as a license. While a move to VB is possible, it would be a ton of work and any support issues that come up would cost me money to get resolved. At this point, I think I will stick with IPB. IPB 3.4 is on the horizon and is looking like it will be a nice upgrade, including a new IPB CDN service which I may sign up for.

So, anyway, I apologies for the degraded site performance, and I am looking to do something about it.

[Hidi]

Site hasn't been doing too badly, it functions. I guess we can just wait for the update and hope for the best! Have you considered anything for if this site has more traffic in the near future?

[sroberson]

if this site has more traffic in the near future?

IF?!?!?

I know there is a way to donate to the TF2 server to keep that running, but is there any chance we can donate towards this possible upgrade?

[Hidi]

I'm going to guess that would be part of donating to the site? I like how you highlight my "If" statement. Much of it depends on when a new Star Fox game comes out as we can expect a great amount of traffic then. If the game is moot, non-existent, or terrible, we'll get some for a short while, but it won't be anything too lasting for multiple reasons. A good game, and we could probably see a large spike in activity, and one that lasts.

[sroberson]

Hey I gotta be optimistic about something right? I'm bottling it all up for Star Fox :)

[Hidi]

Hey I gotta be optimistic about something right? I'm bottling it all up for Star Fox

Yes, I suppose it's a good thing to do. I guess we can agree then that a storm of site traffic here is good thing if a Star Fox game comes out. I wouldn't mind another 3DS game though. The 3D effect was beautiful in the remake, and the circle pad was tuned up very nicely. Enough about that though.

Hey, I did notice the site was a little slower today than usual. Thought I'd put that out even though a lot can't be done about it right now.

[Sideways]

I have used the VB software before in the past. It's really nice and from an admin's POV the control panel was not hard to use (as far as I can remember.) I find it stupid however, that you have to pay for support. It should be included free for a certain part since you are paying so damn much for their software..

[Hidi]

Augh, the site's been slow; functional, but not at optimal speed. I was also having "Server is busy" errors earlier this evening too.

[Sideways]

I notice that it's kinda slow too. I don't get any server is busy errors, though.

[Hidi]

I notice that it's kinda slow too. I don't get any server is busy errors, though.

I only had one bout of it, and haven't had any others errors of the type. Funny though, I was just making a post, and the server was "derping" to the point of failing to load icons on the post editor.

[Hidi]

Ok, so today (And apparently much earlier than this according to Orbiter) there have been some issues with the ability to load the site. We've mainly been getting loading errors stating "Oops, something went wrong!" Around 5 EST onwards, had no troubles with it and was able to continue using the site. Sometimes it just feels like small bouts the site has, but they can happen at the most annoying times even with minimal site traffic.

[DZComposer]

That "Oops, something went wrong!" message is the "Server is Busy" message (read further on the page to see that part). I don't like that it uses that heading, but it's part of IPB's error handling code for some reason rather than being its own thing.

[Hidi]

Ah yes, different heading same sub-text is that what you're getting at? Yeah seems a little weird, but either way, it is still something to put in the books.

[DZComposer]

This very very small snippet of the server error log is why Yandex needs to DIAF:

[Thu Dec 13 18:14:16 2012] [error] [client 180.76.6.20] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:14:16 2012] [error] [client 180.76.6.20] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:15:56 2012] [error] [client 180.76.5.162] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:15:56 2012] [error] [client 180.76.5.162] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:17:37 2012] [error] [client 180.76.5.55] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:17:37 2012] [error] [client 180.76.5.55] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:19:18 2012] [error] [client 180.76.5.49] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:19:18 2012] [error] [client 180.76.5.49] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:20:59 2012] [error] [client 180.76.6.28] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:20:59 2012] [error] [client 180.76.6.28] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:22:51 2012] [error] [client 180.76.5.182] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:22:51 2012] [error] [client 180.76.5.182] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:24:35 2012] [error] [client 180.76.5.149] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:24:35 2012] [error] [client 180.76.5.149] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:26:13 2012] [error] [client 180.76.5.162] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:26:13 2012] [error] [client 180.76.5.162] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:27:54 2012] [error] [client 180.76.5.179] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:27:54 2012] [error] [client 180.76.5.179] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:29:35 2012] [error] [client 180.76.5.51] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:29:35 2012] [error] [client 180.76.5.51] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:31:16 2012] [error] [client 180.76.5.64] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:31:16 2012] [error] [client 180.76.5.64] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:32:57 2012] [error] [client 180.76.5.52] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:32:57 2012] [error] [client 180.76.5.52] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:34:40 2012] [error] [client 180.76.5.53] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:34:40 2012] [error] [client 180.76.5.53] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:36:19 2012] [error] [client 180.76.5.61] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:36:19 2012] [error] [client 180.76.5.61] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:37:59 2012] [error] [client 180.76.5.91] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:37:59 2012] [error] [client 180.76.5.91] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:39:40 2012] [error] [client 180.76.5.170] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:39:40 2012] [error] [client 180.76.5.170] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:40:14 2012] [error] [client 178.154.206.251] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:40:14 2012] [error] [client 178.154.206.251] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html
[Thu Dec 13 18:40:50 2012] [error] [client 178.154.211.252] client denied by server configuration: /home/sfoadmin/starfox-online.net/index.php
[Thu Dec 13 18:40:50 2012] [error] [client 178.154.211.252] client denied by server configuration: /home/sfoadmin/starfox-online.net/forbidden.html

And these are just the ones I've blocked.

Biadu is bad too, but not this bad. It will take me a while to find all the Yandex bots and block them, though it should help reduce the server load.

[DZComposer]

I've increased the server load limit, hopefully that will reduce the "server is busy" errors, though it may come at the cost of slowness, but slow is better than no.

I think we've hit a ceiling with DreamHost here. We need more CPU power, but that is not something I can upgrade on a VPS, I'd have to get a dedicated server. That's $120/mo.

[Fana McCloud]

*Awaits the inevitable post about new "premium features" for donations to keep a dedicated server running* XD

[DZComposer]

Ok I've looked at things, and I can afford to do it, so I went ahead and ordered a dedicated server. There may be some downtime, but I don't know for sure. Some time tomorrow we should be running on a dual-core server-grade CPU with 2GB of RAM.

[Sideways]

Nice! Hope that solves some problems.

[DZComposer]

DreamHost has a partnership with CloudFlare, so I have also signed up for that. That should further help reduce loading times.

[DZComposer]

Troubleshooting issues.

 

DNS is not propegating smoothly, so that is why there was some downtime.

 

Also, CloudFlare is not working quite right. Investigating.

[DZComposer]

Things seem to have stabilized. I may try to get CloudFlare to work tomorrow, so some downtime may be possible.

[DZComposer]

Watching the performance on the new server: CPU is not even breaking a sweat ( < 20% usage), still plenty of RAM (~50% usage system-wide), Disk I/Os seem normal. But, strangely, server load is higher than it was on the VPS, and the site is performing poorly. This makes no god-damned sense.

 

Only thing left I can think of is some kind of bottleneck between the web server and the DB server. Working to move the database to the new server.

[DZComposer]

I've put in a ticket with DreamHost. Hopefully they can help speed things up.

 

My apologies for the terrible perfomance of the site over the past few days.

[Steve]

It's been getting better lately, though. Early december it took me almost a whole minute to surf around the site, surprisingly enough, today I haven't had any issues.

[Drasiana]

You're a lucky one then, Steve. The site's unbearably slow on my end and every time I make a post the page just hangs there and never reloads (even though the post is, usually, processed).