Restoring a website when EC2 fails

Recently, one of my websites went down. After noticing, I checked my EC2 dashboard and saw the instance stopped. AWS had emailed me to say that due to a physical hardware issue, it was terminated. When an instance is terminated, all of its data is lost. Luckily, all of my data is backed up automatically every night.

Since I don’t use RDS, I have to manually manage data redundancy. After a few disasters, I came up with a solution to handle it. I trigger a nightly cron-job to run a shell script. That script takes a MySQL dump and uploads it to S3.

As long as I have the user generated data, everything else is replaceable.  The website that went down is a fitness tracking app. Every day users record their martial arts progress. Below are the ten steps taken to bring everything back up.

  1. Launch a new EC2 instance
  2. Configure the security group for that instance –  I can just use the existing one
  3. Install Apache and MariaDB
  4. Secure the database
  5. Install PhpMyAdmin – I use this tool to import the .sql file in the next step
  6. Import the database backup – I downloaded the nightly .sql file dump from my S3 repo
  7. Setup automatic backups, again
  8. Install WordPress and restore the site’s blog
  9. Configure Route 53 (domain name) and SSL (https) – make the website live again
  10. Quality Assurance – “smoke test” everything to make sure it all looks correct

Use this as a checklist to make sure you don’t forget any steps. Read through the blog posts that I hyperlinked to get more details.