Poster: Eyonix at 5/5/2006 5:19:56 PM PDT
Subject: Select Realm Compensation for 5/04/06 |
| |
On Wednesday, May 5 at approximately 11:50 PM PDT, a hard drive in one of our storage arrays failed, causing a controller to corrupt data files on three realms. In order to resolve the matter, it was necessary to bring down the database that these realms were connected to (17 total realms) for urgent maintenance.
While the realms were down, we began comparing the corrupted data with the clean data in order to restore the affected files. The process to restore the data was involving, and it required us to work hand-in-hand with one of our software vendors to achieve a final resolution. Because this resulted in an extensive period of downtime for the realms listed below, players on those realms will receive a one-day time extension.
UPDATE 1:47 AM PDT
The affected realms are now live and we will be continuing to monitor them throughout the night.
UPDATE 12:30 AM PDT
Our lead database administrator has informed us that the issue has been resolved and the affected realms are now being brought up for internal testing. Our current estimate is to have these realms live in approximately 90 minutes. We will provide an update at that time, or if the current estimate changes.
UPDATE 11:30 PM PDT
Although we had hoped to provide a slightly delayed update with more concrete information, we are in the final stages of the diagnostic tests and are still awaiting the results. We will provide another update in one hour.
UPDATE 10:00 PM PDT
We just got off the phone with our database administrator and we appear to still be on track with the corruption diagnostic tests. We will provide another update in one hour.
UPDATE 9:10 PM PDT
Our lead database administrator has just informed us that the diagnostic process is requiring a greater amount of time than we had previously anticipated. We were able to successfully collect data for one of the affected realms, and are currently working to obtain the data for the other two. We are now currently anticipating the below realms to be available at 4:00 AM PDT. We will provide further updates on this situation in one hour.
UPDATE 7:25 PM PDT
After further monitoring of the diagnostic progression we have found that the current test on the corrupted data is taking longer than previously expected. We anticipate the tests to complete and information to be available at 9:00 PM PDT; we will provide you with another update at that time. Due to this extension, the affected realms are now scheduled to go live at approximately 12:00 AM PDT tonight.
UPDATE 6:30 PM PDT
We are currently running a diagnostic test on the corrupt data. Once we have the results of this test, we will be able to determine a course of action and provide you with more information. The test is scheduled to be completed by approximately 7:45 PM PDT; we will provide you with another update at that time.
UPDATE 6:20 PM PDT
Information copied from original thread found here: http://forums.worldofwarcraft.com/thread.aspx?fn=wow-general&t=8156805
UPDATE 5:20 PM PDT
Our network operations team is still working with our vendors to determine a plan of action. We do know that the data corruption only affects Frostmourne, Sen'jin, and Uther. The data was corrupted by a drive failure that caused two storage controllers to send bad data into the archives for these three realms. We will remain in constant contact with our team leads and will provide you with a status update in one hour.
UPDATE 4:30 PM PDT
We've just been informed that a complication occurred during the restoration process and the corrupt files were not fixed. Our network operations team is currently meeting with our vendors to discuss options on how to move forward. We will provide you with an update within one hour. The affected realms are now scheduled to go live at approximately 11:00 PM PDT tonight.
UPDATE 3:30 PM PDT
We just received an update from the network operations team. The process to restore the corrupt files is almost complete. After that, they will begin the backup process to restore the clean files to the database. We will provide you with another update in approximately one hour. The affected realms are still on schedule to go live before 8:00 PM PDT tonight.
UPDATE 2:30 PM PDT
The network operations team just let us know that the process of restoring the three corrupt files is going well. After the restoration process is complete, we must back up the uncorrupted data before bringing the affected realms back online. We will provide you with another update in approximately one hour. The affected realms are still on schedule to go live before 8:00 PM PDT tonight.
UPDATE 1:10 PM PDT
We just met with the network operations team. The comparison process they ran on the corrupt data is complete. They found three corrupt files and are in the process of restoring them now. We expect all realms to be back online before 8:00 PM PDT. We will continue to gather information on the issue and will provide you with another update in approximately one hour.
UPDATE 12:15 PM PDT
We just got off the phone with our network operations team. The comparison process they are running on the data will finish at approximately 1:00 PM PDT. After the report is complete, we will be able to determine how long it will take to restore the corrupt data and bring these realms back online. Our current ETA remains at 8:00 PM PDT tonight, but we will be able to provide you with more information after they go through the results of the data scan (approximately 1:15 PM PDT).
UPDATE 11:30 AM PDT
We just received another call from our network operations team. They've increased the estimated time it will take to resolve this issue -- the new ETA is approximately 8:00 PM PDT. During the database scan, they discovered that more files were corrupted that we originally thought, and they need more time to continue scanning the database to make sure we detect all the corrupted files before these realms are made live. They are trying to find ways to reduce the amount of time this process takes to complete. We will provide another update on this issue in approximately 30 minutes.
UPDATE 10:53 AM PDT
We just received more information from our network operations team. The realms listed below will remain down until approximately 4:30 PM PDT. During this time, the database administrators will be comparing the corrupted data with the clean data in order to restore the affected files.
The entire team is working hard to reduce the downtime as much as possible. We will continue to provide regular updates on this issue every 30 minutes until it is resolved.
UPDATE 10:30 AM PDT
All of the listed realms are being brought down immediately. The ETA for their return to service is 2 hours:
Blackrock
Cenarius
Frostmourne
Frostwolf
Hyjal
Khaz'goroth
Kil'Jaeden
Kilrogg
Lightbringer
Ner'zhul
Proudmoore
Sen'Jin
Silver Hand
Tichondrius
Uther
Vek'nilash
UPDATE 10:20 AM PDT
Continued Maintenance
Due to a hardware failure last night, we had to bring down several realms in order to replace this equipment. Unfortunately, this failure corrupted data files and affected realm stability. This should not affect character data. We brought down three realms to address this issue, but it now may be necessary to bring down more realms to completely resolve this issue. The realms of Frostmourne, Sen'jin, and Uther will be down for approximately the next 6 hours, as we continue to correct these matters. It may also be necessary to bring down the following realms to assist in these matters.
Blackrock
Cenarius
Frostmourne
Frostwolf
Hyjal
Khaz'goroth
Kil'Jaeden
Kilrogg
Lightbringer
Ner'zhul
Proudmoore
Sen'Jin
Silver Hand
Tichondrius
Uther
Vek'nilash
UPDATE 9:30 AM PDT
We are still running processes to clear out database issues. We are also experiencing some issues with the forums which our network team is addressing. Will provide another update on progress in approximately 30 minutes.
UPDATE 8:45 AM PDT
We are continuing to address database concerns with these realms.
Unfortunately, we are still experiencing stability issues with a few of the realms that were previously brought down due to hardware issues. We are continuing to address these and hope to return these realms to service as quickly as possible. We apologize for this interruption of service. We will provide an update at approximately 8:45 am PDT. The affected realms are as follows:
Frostmourne
Sen'jin
Uther
| | | http://forums.worldofwarcraft.com/thread.aspx?fn=wow-general&t=8176471&p=#post8176471 | | | Poster: Eyonix at 5/5/2006 5:19:56 PM PDT
Subject: Select Realm Compensation for 5/04/06 *edited post* |
| |
On Wednesday, May 3 at approximately 11:50 PM PDT, a hard drive in one of our storage arrays failed, causing a controller to corrupt data files on three realms. In order to resolve the matter, it was necessary to bring down the database that these realms were connected to (17 total realms) for urgent maintenance.
While the realms were down, we began comparing the corrupted data with the clean data in order to restore the affected files. The process to restore the data was involving, and it required us to work hand-in-hand with one of our software vendors to achieve a final resolution. Because this resulted in an extensive period of downtime for the realms listed below, players on those realms will receive a one-day time extension.
UPDATE 1:47 AM PDT
The affected realms are now live and we will be continuing to monitor them throughout the night.
UPDATE 12:30 AM PDT
Our lead database administrator has informed us that the issue has been resolved and the affected realms are now being brought up for internal testing. Our current estimate is to have these realms live in approximately 90 minutes. We will provide an update at that time, or if the current estimate changes.
UPDATE 11:30 PM PDT
Although we had hoped to provide a slightly delayed update with more concrete information, we are in the final stages of the diagnostic tests and are still awaiting the results. We will provide another update in one hour.
UPDATE 10:00 PM PDT
We just got off the phone with our database administrator and we appear to still be on track with the corruption diagnostic tests. We will provide another update in one hour.
UPDATE 9:10 PM PDT
Our lead database administrator has just informed us that the diagnostic process is requiring a greater amount of time than we had previously anticipated. We were able to successfully collect data for one of the affected realms, and are currently working to obtain the data for the other two. We are now currently anticipating the below realms to be available at 4:00 AM PDT. We will provide further updates on this situation in one hour.
UPDATE 7:25 PM PDT
After further monitoring of the diagnostic progression we have found that the current test on the corrupted data is taking longer than previously expected. We anticipate the tests to complete and information to be available at 9:00 PM PDT; we will provide you with another update at that time. Due to this extension, the affected realms are now scheduled to go live at approximately 12:00 AM PDT tonight.
UPDATE 6:30 PM PDT
We are currently running a diagnostic test on the corrupt data. Once we have the results of this test, we will be able to determine a course of action and provide you with more information. The test is scheduled to be completed by approximately 7:45 PM PDT; we will provide you with another update at that time.
UPDATE 6:20 PM PDT
Information copied from original thread found here: http://forums.worldofwarcraft.com/thread.aspx?fn=wow-general&t=8156805
UPDATE 5:20 PM PDT
Our network operations team is still working with our vendors to determine a plan of action. We do know that the data corruption only affects Frostmourne, Sen'jin, and Uther. The data was corrupted by a drive failure that caused two storage controllers to send bad data into the archives for these three realms. We will remain in constant contact with our team leads and will provide you with a status update in one hour.
UPDATE 4:30 PM PDT
We've just been informed that a complication occurred during the restoration process and the corrupt files were not fixed. Our network operations team is currently meeting with our vendors to discuss options on how to move forward. We will provide you with an update within one hour. The affected realms are now scheduled to go live at approximately 11:00 PM PDT tonight.
UPDATE 3:30 PM PDT
We just received an update from the network operations team. The process to restore the corrupt files is almost complete. After that, they will begin the backup process to restore the clean files to the database. We will provide you with another update in approximately one hour. The affected realms are still on schedule to go live before 8:00 PM PDT tonight.
UPDATE 2:30 PM PDT
The network operations team just let us know that the process of restoring the three corrupt files is going well. After the restoration process is complete, we must back up the uncorrupted data before bringing the affected realms back online. We will provide you with another update in approximately one hour. The affected realms are still on schedule to go live before 8:00 PM PDT tonight.
UPDATE 1:10 PM PDT
We just met with the network operations team. The comparison process they ran on the corrupt data is complete. They found three corrupt files and are in the process of restoring them now. We expect all realms to be back online before 8:00 PM PDT. We will continue to gather information on the issue and will provide you with another update in approximately one hour.
UPDATE 12:15 PM PDT
We just got off the phone with our network operations team. The comparison process they are running on the data will finish at approximately 1:00 PM PDT. After the report is complete, we will be able to determine how long it will take to restore the corrupt data and bring these realms back online. Our current ETA remains at 8:00 PM PDT tonight, but we will be able to provide you with more information after they go through the results of the data scan (approximately 1:15 PM PDT).
UPDATE 11:30 AM PDT
We just received another call from our network operations team. They've increased the estimated time it will take to resolve this issue -- the new ETA is approximately 8:00 PM PDT. During the database scan, they discovered that more files were corrupted that we originally thought, and they need more time to continue scanning the database to make sure we detect all the corrupted files before these realms are made live. They are trying to find ways to reduce the amount of time this process takes to complete. We will provide another update on this issue in approximately 30 minutes.
UPDATE 10:53 AM PDT
We just received more information from our network operations team. The realms listed below will remain down until approximately 4:30 PM PDT. During this time, the database administrators will be comparing the corrupted data with the clean data in order to restore the affected files.
The entire team is working hard to reduce the downtime as much as possible. We will continue to provide regular updates on this issue every 30 minutes until it is resolved.
UPDATE 10:30 AM PDT
All of the listed realms are being brought down immediately. The ETA for their return to service is 2 hours:
Blackrock
Cenarius
Frostmourne
Frostwolf
Hyjal
Khaz'goroth
Kil'Jaeden
Kilrogg
Lightbringer
Ner'zhul
Proudmoore
Sen'Jin
Silver Hand
Tichondrius
Uther
Vek'nilash
UPDATE 10:20 AM PDT
Continued Maintenance
Due to a hardware failure last night, we had to bring down several realms in order to replace this equipment. Unfortunately, this failure corrupted data files and affected realm stability. This should not affect character data. We brought down three realms to address this issue, but it now may be necessary to bring down more realms to completely resolve this issue. The realms of Frostmourne, Sen'jin, and Uther will be down for approximately the next 6 hours, as we continue to correct these matters. It may also be necessary to bring down the following realms to assist in these matters.
Blackrock
Cenarius
Frostmourne
Frostwolf
Hyjal
Khaz'goroth
Kil'Jaeden
Kilrogg
Lightbringer
Ner'zhul
Proudmoore
Sen'Jin
Silver Hand
Tichondrius
Uther
Vek'nilash
UPDATE 9:30 AM PDT
We are still running processes to clear out database issues. We are also experiencing some issues with the forums which our network team is addressing. Will provide another update on progress in approximately 30 minutes.
UPDATE 8:45 AM PDT
We are continuing to address database concerns with these realms.
Unfortunately, we are still experiencing stability issues with a few of the realms that were previously brought down due to hardware issues. We are continuing to address these and hope to return these realms to service as quickly as possible. We apologize for this interruption of service. We will provide an update at approximately 8:45 am PDT. The affected realms are as follows:
Frostmourne
Sen'jin
Uther[ post edited by Eyonix ]
| | | http://forums.worldofwarcraft.com/thread.aspx?fn=wow-general&t=8176471&p=#post8176471 |
View all recent official Blue Posts
|
