We learned this week that there has been a major incident affecting the files which have been uploaded by our staff and students to their schools eduBuzz WordPress sites.
We now know the extent of the problem, and this post is to share that information, and advise you of current recovery plans. Please share it with other staff as appropriate.
The symptoms, which you may have seen, are missing images and files such as newsletters and PDF resources becoming inaccessible.
Investigation Findings to Date
The site is now managed by a commercial company which specialises in secure hosting of WordPress sites for the public sector. Their specialists have established that the image and document files involved are now missing, but the reason for that is as yet unknown. The uploaded files are stored on Amazon’s secure and highly resilient infrastructure, and should have been recoverable from a built-in Amazon backup system (versioning). Unfortunately that arrangement has failed on this occasion.
Here is a summary of the situation this afternoon. This is a “worst case” scenario.
– Over the years, we have uploaded 415,804 files (images, documents etc) to the various sites.
– 68,115 of these files are now missing.
– 57,695 of these missing files will be able to be restored from the backup used for site migration.
– Of the 10,400 we cannot replace that way, 7,700 are images and 2600 are documents.
Apart from the missing images, the site is healthy and can be used normally.
Recovery Plans – Documents
We are prioritising the recovery of the 2600 document files, as these are most likely to be resources important to learning. The documents will also include Parent Council minutes.
WordPress retains information about the missing files in its database.
Hosting company specialists will use that information to email site contributors and request new copies of the uploaded documents. They will be able to provide the site name, the file name, and probably the upload date.
Where replacement files can be emailed back, hosting company staff will upload them to exactly replace the lost copies.
Recovery Plans – Images
It is clearly not realistic to take the same approach to the recovery of the 7,700 missing image files. Staff are unlikely to have copies, the file names will be meaningless, and the effort would not be worthwhile. These will generally illustrate blog posts and pages, where the text will remain accessible.
Where header images etc are missing, staff and students can replace them in the normal way (Appearance / Header menu).
Please accept my apologies for this disruption. The problem which affected backups has now been identified and resolved, and the hosting company are making certain that this type of failure cannot reoccur.