Sorry about the downtime!

  • 57 Replies
  • 19461 Views
*

Daniel

  • Administrator
  • 1146
Re: Sorry about the downtime!
« Reply #30 on: May 01, 2011, 11:10:06 AM »
Hi.  Sorry (again) about the downtime!  I'll give everyone a quick explanation of what happened.

Earlier this month, we went down for a day or so because someone downloaded the same PDF file over and over again until my bandwidth allowance was completely used up.  My hosting company was very helpful and reset the quota for me.  Things were running fine until a few days ago when someone (probably the same person) did exactly the same thing again.  The hosting company was, understandably, not willing to reset me quota a second time, so over the limit we stayed.  Everything reset itself today so we're back to normal now.

I'm working on ways to avoid having this happen again in the future.  I have put a couple changes in place now to make it more difficult to take the site down like that and I'm hoping that will be enough.  I'm not going to detail exactly what I've done in case there are bad people reading this.  In any event, my fixes aren't exactly perfect.  So if there are any sysadmin types here with ideas, I'm happy to hear them.  Ideally, I'd like an automated way of preventing the same IP address from downloading excessive amounts of data in a short period of time.  I'm not sure how to go about doing that via cpanel, though, so I'm open to suggestions.

In tangentially-related news, I now own earthnotaglobe.com as well, so if anyone has ideas about what to do with that domain, I'm all ears.

*

theonlydann

  • Official Member
  • 24186
Re: Sorry about the downtime!
« Reply #31 on: May 01, 2011, 11:13:23 AM »
I have an idea of what to do with that website.

Put a picture of the not so globe shaped earth there making a smiley face and sticking its tongue out.


*

PizzaPlanet

  • 12260
  • Now available in stereo
Re: Sorry about the downtime!
« Reply #32 on: May 01, 2011, 11:23:48 AM »
I'm not sure how to go about doing that via cpanel, though, so I'm open to suggestions.
While I'm far from being an expert on the matter, I can say this with confidence: cpanel won't do the trick. What you want to do is play around with Apache's configuration. I don't have a solution at the ready, but if there is one, it's definitely to be found there.
hacking your precious forum as we speak 8) 8) 8)

*

EnglshGentleman

  • Flat Earth Editor
  • 9548
Re: Sorry about the downtime!
« Reply #33 on: May 01, 2011, 11:35:16 AM »
We should probably also consult Parsifal and John on this matter.

*

PizzaPlanet

  • 12260
  • Now available in stereo
Re: Sorry about the downtime!
« Reply #34 on: May 01, 2011, 12:32:18 PM »
hacking your precious forum as we speak 8) 8) 8)

?

Mrs. Peach

  • Official Member
  • 6258
Re: Sorry about the downtime!
« Reply #35 on: May 01, 2011, 12:40:43 PM »
Do away with unlogged-on downloading and 'hidden status' and we can all check on just what the hell people are up to.  Transparency is all the rage now.

Someone will now reply that I am a techno-know-nothing which is a given anyway.  :)

?

Crustinator

  • 7813
  • Bamhammer horror!
Re: Sorry about the downtime!
« Reply #36 on: May 01, 2011, 01:16:29 PM »
PDF file

consult Parsifal

The PDF file format is proprietary. I suggest we publish all materials in postscript format.

*

Trekky0623

  • Official Member
  • 10061
Re: Sorry about the downtime!
« Reply #37 on: May 01, 2011, 03:38:22 PM »
Hi.  Sorry (again) about the downtime!  I'll give everyone a quick explanation of what happened.

Earlier this month, we went down for a day or so because someone downloaded the same PDF file over and over again until my bandwidth allowance was completely used up.  My hosting company was very helpful and reset the quota for me.  Things were running fine until a few days ago when someone (probably the same person) did exactly the same thing again.  The hosting company was, understandably, not willing to reset me quota a second time, so over the limit we stayed.  Everything reset itself today so we're back to normal now.

I'm working on ways to avoid having this happen again in the future.  I have put a couple changes in place now to make it more difficult to take the site down like that and I'm hoping that will be enough.  I'm not going to detail exactly what I've done in case there are bad people reading this.  In any event, my fixes aren't exactly perfect.  So if there are any sysadmin types here with ideas, I'm happy to hear them.  Ideally, I'd like an automated way of preventing the same IP address from downloading excessive amounts of data in a short period of time.  I'm not sure how to go about doing that via cpanel, though, so I'm open to suggestions.

In tangentially-related news, I now own earthnotaglobe.com as well, so if anyone has ideas about what to do with that domain, I'm all ears.

Why not host PDFs on Google docs?

*

markjo

  • Content Nazi
  • The Elder Ones
  • 42529
Re: Sorry about the downtime!
« Reply #38 on: May 01, 2011, 04:17:29 PM »
The PDF file format is proprietary.

Nope.
Quote from: http://en.wikipedia.org/wiki/Portable_Document_Format
Originally a proprietary format, PDF was officially released as an open standard on July 1, 2008, and published by the International Organization for Standardization as ISO 32000-1:2008.
Science is what happens when preconception meets verification.
Quote from: Robosteve
Besides, perhaps FET is a conspiracy too.
Quote from: bullhorn
It is just the way it is, you understanding it doesn't concern me.

*

Parsifal

  • Official Member
  • 36118
  • Bendy Light specialist
Re: Sorry about the downtime!
« Reply #39 on: May 02, 2011, 12:49:01 AM »
I feel that a word needs to be said about the alternative FES that I set up recently. I won't link it here, as I don't want it to be misconstrued as attempting to steal people away from this site; however, I would like to summarise what is the result of a couple of years of (mostly ignored) suggestions and complaints regarding the forum from myself and others.

In general, from a technical (as opposed to community) perspective, this forum accumulates problems without fixing them. The slow searching and the recent bandwidth troubles are just the tip of the iceberg, and I don't have much confidence that any action will be taken to prevent future outages of this sort. That said, I see the future of the FES as taking one of three directions:

  • The forum that I have set up becomes official. I have tried to keep it fairly close in design to this one, and would be very open to discussing a possible merger whereby I host the forum, with Daniel continuing to make administrative decisions regarding the rules, and the CMS part of the site remains here.
  • The upgrade talked about in the past, which would hopefully remedy most or all of the technical issues this forum has been experiencing, is realised directly by Daniel. This would have more or less the same outcome as the above, and if sufficient steps are taken in this direction, I will gladly close down my own forum and set up a redirect back to this one.
  • This forum continues to be full of empty promises, aging software and badly managed hosting. I do hope this will not be the case, but if it is, I will continue to operate my own board as an alternative place for those who want a properly managed discussion forum.

That said, I am glad to see the site back up, and I hope we can see some degree of progress towards improving its uptime and reliability.
I'm going to side with the white supremacists.

*

Daniel

  • Administrator
  • 1146
Re: Sorry about the downtime!
« Reply #40 on: May 02, 2011, 04:33:46 AM »
That said, I see the future of the FES as taking one of three directions:

  • The forum that I have set up becomes official. I have tried to keep it fairly close in design to this one, and would be very open to discussing a possible merger whereby I host the forum, with Daniel continuing to make administrative decisions regarding the rules, and the CMS part of the site remains here.
  • The upgrade talked about in the past, which would hopefully remedy most or all of the technical issues this forum has been experiencing, is realised directly by Daniel. This would have more or less the same outcome as the above, and if sufficient steps are taken in this direction, I will gladly close down my own forum and set up a redirect back to this one.
  • This forum continues to be full of empty promises, aging software and badly managed hosting. I do hope this will not be the case, but if it is, I will continue to operate my own board as an alternative place for those who want a properly managed discussion forum.

I'm leaning toward a combination of the second and third options.  The delay in upgrading forum software isn't my doing.  We've been waiting on a new version of SMF for ages now.  The 2.0RC versions have been out for literally years but there hasn't been a final release yet.  As soon as that comes out, we'll upgrade and hopefully a lot of the current issues will go away.  We actually tried to upgrade to phpBB3 a year or so ago but that didn't work out.  I'm not 100% sure what caused the conversion to break every time, but I think it was something to do with the fact that we were originally phpBB when the site first started in 2004 and then converted to SMF, possibly muddying up the mySQL database in the process.  It doesn't help, either, that the forums database is pretty huge now.  So moving to phpBB3 doesn't look like an option at the moment.  That leaves us with SMF2.0 which is apparently held up for legal reasons but should be available soon.  I'm hoping that I mean "soon" in the normal sense of the word rather than the geological sense.

As for the downtime this month.. well, that had nothing to do with faulty software or inept admins.  That was a malicious attack which didn't take advantage of software flaws or anything as clever as that.  We're working to prevent that from happening in the future.  I've passed on relevant logging data to the correct people and we're put measures in place to (hopefully) stop it from happening again.  We'll see how it goes.

*

frostee

  • Official Member
  • 3555
  • Posts: 1337
Re: Sorry about the downtime!
« Reply #41 on: May 02, 2011, 05:38:05 AM »
But SMF isn't free software BAWWW BAWWW BAWWW
Recently religious due to the impending rapture.

*

markjo

  • Content Nazi
  • The Elder Ones
  • 42529
Re: Sorry about the downtime!
« Reply #42 on: May 02, 2011, 06:09:54 AM »
But SMF isn't free software BAWWW BAWWW BAWWW

Take your bawwing to RM.
Science is what happens when preconception meets verification.
Quote from: Robosteve
Besides, perhaps FET is a conspiracy too.
Quote from: bullhorn
It is just the way it is, you understanding it doesn't concern me.

*

frostee

  • Official Member
  • 3555
  • Posts: 1337
Re: Sorry about the downtime!
« Reply #43 on: May 02, 2011, 07:14:13 AM »
But SMF isn't free software BAWWW BAWWW BAWWW

Take your bawwing to RM.
Are you going to tell Parsifal that too?
Recently religious due to the impending rapture.

*

Parsifal

  • Official Member
  • 36118
  • Bendy Light specialist
Re: Sorry about the downtime!
« Reply #44 on: May 02, 2011, 07:24:05 AM »
I'm leaning toward a combination of the second and third options.  The delay in upgrading forum software isn't my doing.  We've been waiting on a new version of SMF for ages now.  The 2.0RC versions have been out for literally years but there hasn't been a final release yet.  As soon as that comes out, we'll upgrade and hopefully a lot of the current issues will go away.  We actually tried to upgrade to phpBB3 a year or so ago but that didn't work out.  I'm not 100% sure what caused the conversion to break every time, but I think it was something to do with the fact that we were originally phpBB when the site first started in 2004 and then converted to SMF, possibly muddying up the mySQL database in the process.  It doesn't help, either, that the forums database is pretty huge now.  So moving to phpBB3 doesn't look like an option at the moment.  That leaves us with SMF2.0 which is apparently held up for legal reasons but should be available soon.  I'm hoping that I mean "soon" in the normal sense of the word rather than the geological sense.

I'm very glad to hear it. Have you considered the possibility of starting with a clean forum and archiving this one, which may help to keep the database tidy (which I suspect to be a contributing factor to the laggy performance created by searching) in any event?

As for the downtime this month.. well, that had nothing to do with faulty software or inept admins.  That was a malicious attack which didn't take advantage of software flaws or anything as clever as that.  We're working to prevent that from happening in the future.  I've passed on relevant logging data to the correct people and we're put measures in place to (hopefully) stop it from happening again.  We'll see how it goes.

It did, however, take advantage of the fact that the hosting provider cuts us off when our bandwidth limit is reached, rather than handling it gracefully -- for example, I would get notified when I reach 80% of my bandwidth allowance, and if I go over I simply get charged for the excess usage. Simply cutting off the website appears to be a "feature" of CPanel which doesn't really help anybody, and I would consider making use of it to be symptomatic of incompetence on the part of the hosting provider.
I'm going to side with the white supremacists.

*

Daniel

  • Administrator
  • 1146
Re: Sorry about the downtime!
« Reply #45 on: May 02, 2011, 08:02:17 AM »
It did, however, take advantage of the fact that the hosting provider cuts us off when our bandwidth limit is reached, rather than handling it gracefully -- for example, I would get notified when I reach 80% of my bandwidth allowance, and if I go over I simply get charged for the excess usage. Simply cutting off the website appears to be a "feature" of CPanel which doesn't really help anybody, and I would consider making use of it to be symptomatic of incompetence on the part of the hosting provider.

I do get notified of bandwidth usage.  That's not very useful, though, when someone is running a script that eats up 40gigs in the span of about 2 hours. 

*

markjo

  • Content Nazi
  • The Elder Ones
  • 42529
Re: Sorry about the downtime!
« Reply #46 on: May 02, 2011, 09:05:35 AM »
But SMF isn't free software BAWWW BAWWW BAWWW

Take your bawwing to RM.
Are you going to tell Parsifal that too?
Parsifal is discussing the issue at hand.  You are not.
Science is what happens when preconception meets verification.
Quote from: Robosteve
Besides, perhaps FET is a conspiracy too.
Quote from: bullhorn
It is just the way it is, you understanding it doesn't concern me.

?

Horatio

  • Official Member
  • 4016
Re: Sorry about the downtime!
« Reply #47 on: May 02, 2011, 09:08:00 AM »
It did, however, take advantage of the fact that the hosting provider cuts us off when our bandwidth limit is reached, rather than handling it gracefully -- for example, I would get notified when I reach 80% of my bandwidth allowance, and if I go over I simply get charged for the excess usage. Simply cutting off the website appears to be a "feature" of CPanel which doesn't really help anybody, and I would consider making use of it to be symptomatic of incompetence on the part of the hosting provider.

I do get notified of bandwidth usage.  That's not very useful, though, when someone is running a script that eats up 40gigs in the span of about 2 hours. 

Shouldn't the hosting provider red flag such a bizarrely large amount of bandwidth usage in such a short period of time?
How dare you have the audacity to demand my deposition. I've never even heard of you.

*

Lord Wilmore

  • Vice President
  • Flat Earth Believer
  • 12107
Re: Sorry about the downtime!
« Reply #48 on: May 02, 2011, 09:12:48 AM »
I think Daniel's point is that such a warning may be informative, but is not especially helpful.
"I want truth for truth's sake, not for the applaud or approval of men. I would not reject truth because it is unpopular, nor accept error because it is popular. I should rather be right and stand alone than run with the multitude and be wrong." - C.S. DeFord

?

Horatio

  • Official Member
  • 4016
Re: Sorry about the downtime!
« Reply #49 on: May 02, 2011, 09:15:22 AM »
I think Daniel's point is that such a warning may be informative, but is not especially helpful.

And my point is that a hosting company with common sense would red flag such nonsense.
How dare you have the audacity to demand my deposition. I've never even heard of you.

*

Lord Wilmore

  • Vice President
  • Flat Earth Believer
  • 12107
Re: Sorry about the downtime!
« Reply #50 on: May 02, 2011, 09:29:53 AM »
And my point is that a hosting company with common sense would red flag such nonsense.


But they did...


I do get notified of bandwidth usage.  That's not very useful, though, when someone is running a script that eats up 40gigs in the span of about 2 hours. 
"I want truth for truth's sake, not for the applaud or approval of men. I would not reject truth because it is unpopular, nor accept error because it is popular. I should rather be right and stand alone than run with the multitude and be wrong." - C.S. DeFord

*

PizzaPlanet

  • 12260
  • Now available in stereo
Re: Sorry about the downtime!
« Reply #51 on: May 02, 2011, 09:36:58 AM »
It did, however, take advantage of the fact that the hosting provider cuts us off when our bandwidth limit is reached, rather than handling it gracefully -- for example, I would get notified when I reach 80% of my bandwidth allowance, and if I go over I simply get charged for the excess usage.
Good idea. Next time someone decides to download a million billion PDFs, Daniel will just go bankrupt. Yep, sounds viable.

Also, a much better idea than moving the relatively low-bandwidth forum to your server (which is a shit idea, by the way) would be moving the PDFs and other potentially large files somewhere (preferably to a place you can't wipe the next time you get drunk/angry).

EDIT: Daniel, what are the bad guys' IP addresses? Just out of curiosity.
« Last Edit: May 02, 2011, 10:09:37 AM by PizzaPlanet »
hacking your precious forum as we speak 8) 8) 8)

*

Parsifal

  • Official Member
  • 36118
  • Bendy Light specialist
Re: Sorry about the downtime!
« Reply #52 on: May 03, 2011, 02:51:25 AM »
I do get notified of bandwidth usage.  That's not very useful, though, when someone is running a script that eats up 40gigs in the span of about 2 hours.

That's fair enough, I suppose. I still think that hard caps on bandwidth is a dated concept.
I'm going to side with the white supremacists.

*

PizzaPlanet

  • 12260
  • Now available in stereo
Re: Sorry about the downtime!
« Reply #53 on: May 03, 2011, 09:21:03 AM »
I still think that hard caps on bandwidth is a dated concept.
And that is why you're the tech support and someone else is the boss. It's okay, you'll learn in due time.
hacking your precious forum as we speak 8) 8) 8)

*

Chris Spaghetti

  • Flat Earth Editor
  • 12744
Re: Sorry about the downtime!
« Reply #54 on: May 03, 2011, 04:09:27 PM »
Damn bandwidth-stealers.

?

Crustinator

  • 7813
  • Bamhammer horror!
Re: Sorry about the downtime!
« Reply #55 on: May 05, 2011, 03:31:33 PM »
I want more powaz

You can't have them.

?

Mrs. Peach

  • Official Member
  • 6258
Re: Sorry about the downtime!
« Reply #56 on: May 31, 2011, 05:41:58 PM »
Has Daniel solved the bandwidth thief mystery or did the moron (the thief, not Daniel) forget the day of the month?

*

Daniel

  • Administrator
  • 1146
Re: Sorry about the downtime!
« Reply #57 on: June 01, 2011, 02:49:55 AM »
I think it's a combination of the two.