FUCKED: Server 1

edited December 2010 in News & Announcements
Cross posted from http://winboards.net/viewtopic.php?p=119597#119597 :

In regards to the latest wave of problems on Server 1, here's the explanation. I'll update this post periodically to provide updates on this.

The cause of all the fail is DRIVE FAILURE!!!!!! - yay, right? no, not yay. I made changes to some of the epoll behavior to fix some of the forum/other PHP based reset errors, and turned off the global cache, which I forgot I turned off for exactly this reason. However, as far as downloads go, after some investigation there's massive amounts of failures in accessing an assload of blocks on that hard disk.

This very well may be an issue easily fixed by a good fsck, however to umount and fsck that drive will require a maintenance window and datacenter cooperation which is really not 100% feasible at this point. With the luck I've had lately this will probably not solve much of anything.

I'll look into this issue more later today.

Comments

  • I'm going to unannounce this because stickies and announcements don't show up on WinWorld.
  • UPDATE:
    SMART Self-test log structure revision number 1
    Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
    # 1  Extended offline    Completed without error       00%     16763         -
    

    I dunno how much I trust that not-very-smart assessment but I'll start taking stuff offline so I can umount that drive later today.
  • Sorry to hear that. hope you get it up and running again soon.
  • Stitch: Is that the server that has roughly the same chipset as my fileserver? I've had random drive issues lately, I ran checkdisk several times, and the drive is fine, but for some reason it just shits itself every so often.
  • Any news?
  • gdea73 wrote:
    Sorry to hear that. hope you get it up and running again soon.

    Thanks. I keep telling myself I'll put that box out of it's misery one day, I should probably eventually do it.
    BOD wrote:
    Stitch: Is that the server that has roughly the same chipset as my fileserver? I've had random drive issues lately, I ran checkdisk several times, and the drive is fine, but for some reason it just shits itself every so often.

    Yeah and I think even with the latest firmware it hates 1TB disks. Cherokee's epoll() handler spit out a 300MB log file of bad file descriptors in winworld's error log alone.
    Rioter wrote:
    Any news?

    I have a KVM hooked up after steadfast fucked up and hooked it up to the wrong box at first, even with this image, so I should probably plan a time for this to go down for fsck. I'll handle this once my insomnia lately clears up.
  • Well, we're back... for now anyway.
  • So does that mean you're not shipping me the contents of the drive anymore?
  • stitch wrote:
    BOD wrote:
    Stitch: Is that the server that has roughly the same chipset as my fileserver? I've had random drive issues lately, I ran checkdisk several times, and the drive is fine, but for some reason it just shits itself every so often.

    Yeah and I think even with the latest firmware it hates 1TB disks. Cherokee's epoll() handler spit out a 300MB log file of bad file descriptors in winworld's error log alone.

    I ended up installing a hacked up firmware since the latest BIOS didn't update the Sata controllers version to the right one.

    Even so I still get random quirks here and there, but a decent PCI controller that has more than 2 ports is stupidly expensive.
  • Ka0s wrote:
    So does that mean you're not shipping me the contents of the drive anymore?

    I'll do it, just not this second. Also, I don't have an IDE disk, it will have to be SATA.
  • stitch wrote:
    Ka0s wrote:
    So does that mean you're not shipping me the contents of the drive anymore?

    I'll do it, just not this second. Also, I don't have an IDE disk, it will have to be SATA.

    I think wdl2 only supported IDE but I can always have someone at work copy it for me... How many gigs are we talking at this point?
  • Ka0s wrote:
    stitch wrote:
    Ka0s wrote:
    So does that mean you're not shipping me the contents of the drive anymore?

    I'll do it, just not this second. Also, I don't have an IDE disk, it will have to be SATA.

    I think wdl2 only supported IDE but I can always have someone at work copy it for me... How many gigs are we talking at this point?

    I have no idea really, the disk itself will be around 250 GB, I think most of it will be used.
Sign In or Register to comment.