Restoring a website when EC2 fails

restore a website with these steps

Recently, one of my websites went down. After noticing, I checked my EC2 dashboard and saw the instance stopped. AWS had emailed me to say that due to a physical hardware issue, it was terminated. When an instance is terminated, all of its data is lost. Luckily, all of my data is backed up automatically every night.

Since I don’t use RDS, I have to manually manage data redundancy. After a few disasters, I came up with a solution to handle it. I trigger a nightly cron-job to run a shell script. That script takes a MySQL dump and uploads it to S3.

As long as I have the user generated data, everything else is replaceable.  The website that went down is a fitness tracking app. Every day users record their martial arts progress. Below are the ten steps taken to bring everything back up.

  1. Launch a new EC2 instance
  2. Configure the security group for that instance –  I can just use the existing one
  3. Install Apache and MariaDB
  4. Secure the database
  5. Install PhpMyAdmin – I use this tool to import the .sql file in the next step
  6. Import the database backup – I downloaded the nightly .sql file dump from my S3 repo
  7. Setup automatic backups, again
  8. Install WordPress and restore the site’s blog
  9. Configure Route 53 (domain name) and SSL (https) – make the website live again
  10. Quality Assurance – “smoke test” everything to make sure it all looks correct

Use this as a checklist to make sure you don’t forget any steps. Read through the blog posts that I hyperlinked to get more details.

Build an image upload gallery

media gallery upload with s3

Allowing users to upload images to your app can be a pivotal feature. Many digital products rely on it. This post will show you how to do it using PHP and AWS S3.

image upload gallery

After launching version 1.0 of SplitWit, I decided to enhance the platform by adding features. An important A/B experiment involves swapping images. This is particularly useful on ecommerce stores that sell physical products.

Originally, users could only swap images by entering a URL. To the average website owner, this would seem lame. For SplitWit to be legit, adding images on the fly had to be a feature.

I wrote three scripts – one to upload files, one to fetch them, and one to delete them. Each leverages a standalone PHP class written by Donovan Schönknecht, making it easy to interact with AWS S3. All you’ll need is your S3 bucket name and IAM user credentials. The library provides methods to do everything you need.

AWS S3

Amazon S3 stands for “simple storage service”. It provides data storage that is scalable, secure, highly available, and performant.

A new bucket can be created directly from the management console.

create new s3 bucket

You’ll want to create a new IAM user to programmatically interact with this bucket. Make sure that new user is added to a group that includes the permission policy “AmazonS3FullAccess”. You can find the access key ID and secret in the “Security credentials” tab.

IAM user in AWS with permissions for S3

Uploading image files

When users select an image in the visual editor, they are shown a button to upload a new file. Clicking on it opens the gallery modal.

<div id="image-gallery-modal" class="modal image-gallery-modal" style="display: none;">
  <div class="modal-content">
    <h3>Your image gallery</h3>
    <p><strong>Upload a new file:</strong></p>
    <input class="uploadimage" id="uploadimage" type="file" name="uploadimage" />
    <p class="display-none file-error"></p>
    <div><hr /></div>
    <div class="image-gallery-content"></div>
    <p class="loading-images"><i class="fas fa-spinner fa-spin"></i> Loading images...</p>
  </div>
</div>

The HTML file-type input element presents a browser dialog to select a file. Once selected, the image data is posted to the S3 upload script. The newly uploaded image then replaces the existing image in the visual editor. 

$(".uploadimage").change(function(){
    
    var file = $(this)[0].files[0];   
    var formData = new FormData();
    formData.append("file", file, file.name);
    formData.append("upload_file", true);         

    $.ajax({
      type: "POST",
      url: "/s3-upload.php",
      xhr: function () {
        var myXhr = $.ajaxSettings.xhr();
        if (myXhr.upload) {
            // myXhr.upload.addEventListener('progress', that.progressHandling, false);
        }
        return myXhr;
      },
      success: function (response) {
        console.log(response);
        
        document.getElementById("uploadimage").value = "";

        if(response !== "success"){
          $(".file-error").text(response).show();
          setTimeout(function(){ $(".file-error").fadeOut();}, 3000)
          return;
        }
        
        $("#image-gallery-modal").hide();
        loadS3images();
        var newImageUrl = "https://splitwit-image-upload.s3.amazonaws.com/<?php echo $_SESSION['userid'];?>/" + file.name;
        $("input.img-url").val(newImageUrl);
        $(".image-preview").attr("src", newImageUrl).show();
        $(".image-label .change-indicator").show();

        //update editor (right side)
        var selector = $(".selector-input").val();
        var iFrameDOM = $("iframe#page-iframe").contents()
        if($(".element-change-wrap").is(":visible")){
          iFrameDOM.find(selector).attr("src", newImageUrl).attr("srcset", "");
          $(".element-change-save-btn").removeAttr("disabled");
        }
        if($(".insert-html-wrap").is(":visible")){
          var position = $(".position-select").val();
          var htmlInsertText = "<img style='display: block; margin: 10px auto;' class='htmlInsertText' src='"+newImageUrl+"'>";
          iFrameDOM.find(".htmlInsertText").remove();
          if(position == "before"){
            iFrameDOM.find(selector).before(htmlInsertText);
          }
          if(position == "after"){
            iFrameDOM.find(selector).after(htmlInsertText);
          }
        }
      },
      error: function (error) {
        console.log("error: ");
        console.log(error);
      },
      async: true,
      data: formData,
      cache: false,
      contentType: false,
      processData: false,
      timeout: 60000
  });

});

The upload script puts files in the same S3 bucket, under a separate sub-directory for each user account ID. It checks the MIME type on the file to make sure an image is being uploaded.

<?php
require 's3.php';
 
$s3 = new S3("XXXX", "XXXX"); //access key ID and secret

// echo "S3::listBuckets(): ".print_r($s3->listBuckets(), 1)."\n";

$bucketName = 'image-upload';

if(isset($_FILES['file'])){
	$file_name = $_FILES['file']['name'];   
	$uploadFile = $_FILES['file']['tmp_name']; 

	if ($_FILES['file']['size'] > 5000000) { //5 megabyte
     	   echo 'Exceeded filesize limit.';
     	   die();
    	}
    	$finfo = new finfo(FILEINFO_MIME_TYPE);
	if (false === $ext = array_search(
	        $finfo->file($uploadFile),
	        array(
	            'jpg' => 'image/jpeg',
	            'png' => 'image/png',
	            'gif' => 'image/gif',
	        ),
	        true
	    )) {
	    	if($_FILES['file']['type'] == ""){
	    		echo 'File format not found. Please re-save the file.';
	    	}else{
		    	echo 'Invalid file format.';
		    }
     	    die();
	 }

	//create new directory with account ID, if it doesn't already exist
	session_start();
	$account_id = $_SESSION['userid'];

	if ($s3->putObjectFile($uploadFile, $bucketName, $account_id."/".$file_name, S3::ACL_PUBLIC_READ)) {
		echo "success";
	}

}
?>

After upload, the gallery list is reloaded by the loadS3images() function.

Fetching image files from S3

When the image gallery modal first shows, that same loadS3images() runs to populate any images that have been previously uploaded.

function loadS3images(){

  $.ajax({
      url:"/s3-get-objects.php",
      complete: function(response){
        gotImages = true;
        $(".loading-images").hide();
        var data = JSON.parse(response.responseText);
        var x;
        var html = "<p><strong>Select existing file:</strong></p>";
        var l = 0;
        for (x in data) {
          l++;
          var name = data[x]["name"];
          nameArr = name.split("/");
          name = nameArr[1];
          var imgUrl = "https://splitwit-image-upload.s3.amazonaws.com/<?php echo $_SESSION['userid'];?>/" + name;
          html += "<div class='image-data-wrap'><p class='filename'>"+name+"</p><img style='width:50px;display:block;margin:10px;' src='' class='display-none'><button type='button' class='btn select-image'>Select</button> <button type='button' class='btn preview-image'>Preview</button> <button type='button' class='btn delete-image'>Delete</button><hr /></div>"
        }
        if(l){
          $(".image-gallery-content").html(html);
        }

      }
    });
}

It hits the “get objects” PHP script to pull the files in the account’s directory.

<?php
require 's3.php';
 
$s3 = new S3("XXX", "XXX"); //access key ID and secret

$bucketName = 'image-upload';
session_start();
$account_id = $_SESSION['userid'];
$info = $s3->getBucket($bucketName, $account_id);
echo json_encode($info);

?>

Existing images can be chosen to replace the one currently selected in the editor. There are also options to preview and delete.

Delete an S3 object

When the delete button is pressed for a file in the image gallery, all we need to do is pass the filename along. If the image is currently being used, we also remove it from the editor.

$(".image-gallery-content").on("click", ".delete-image", function() {
    var parent = $(this).parent();
    var filename = parent.find(".filename").text();
    var currentImageUrl = $(".img-url").val();
    if(currentImageUrl =="https://splitwit-image-upload.s3.amazonaws.com/<?php echo $_SESSION['userid'];?>/" + filename){
      $(".img-url").val(testSelectorElImage);
      $(".image-preview").attr("src", testSelectorElImage);
      var selector = $(".selector-input").val();
      var iFrameDOM = $("iframe#page-iframe").contents()
      iFrameDOM.find(selector).attr("src", testSelectorElImage);
    }
    $.ajax({
      method:"POST",
      data: { 
        'filename': filename, 
      },
      url: "/s3-delete.php?filename="+filename,
      complete: function(response){
        parent.remove();
        if(!$(".image-data-wrap").length){
          $(".image-gallery-content").html("");
        }
      }
    })

}); 

 

<?php
require 's3.php';
 
$s3 = new S3("XXX", "XXX"); //access key ID and secret

$bucketName = 'image-upload';
session_start();
$account_id = $_SESSION['userid'];
$filename = $_POST['filename'];
if ($s3->deleteObject($bucketName, $account_id."/".$filename) ){
	echo "S3::deleteObject(): Deleted file\n";
}

?>

 

Automatic MySQL dump to S3

Automatic MySQL dump to S3

I have had some lousy luck with databases. In 2018, I created a fitness app for martial artists, and quickly gained over a hundred users in the first week. Shortly after, the server stopped resolving and I didn’t know why. I tried restarting it, but that didn’t help. Then, I stopped the EC2 instance from my AWS console. Little did I know, that would wipe the all of the data from that box. Ouch.

Recently, a client let me know that their site wasn’t working. A dreaded “error connecting to the database” message was all that resolved. I’d seen this one before – no sweat. Restarting the database usually does the trick: “sudo service mariadb restart”. The command line barked back at me: “Job for mariadb.service failed because the control process exited with error code.”

Uh-oh.

The database was corrupted. It needed to be deleted and reinstalled. Fortunately, I just happen to have a SQL dump for this site saved on my desktop. This was no way to live – in fear of the whims of servers.

Part of the issue is that I’m running MySQL on the same EC2 instance as the web server.  A more sophisticated architecture would move the database to RDS. This would provide automated backups, patches, and maintenance. It also costs more.

To keep cost low, I decided to automate MySQL dumps and upload to an S3 bucket. S3 storage is cheap ($0.20/GB), and data transfer from EC2 is free.

Deleting and Reinstalling the Database

If your existing database did crash and become corrupt, you’ll need to delete and reinstall it. To reset the database, I SSH’d into my EC2 instance. I navigated to `/var/lib/mysql`

cd /var/lib/mysql

Next, I deleted everything in that folder:

sudo rm -r *

Finally, I ran a command to reinitialize the database directory

mysql_install_db --user=mysql --basedir=/usr --datadir=/var/lib/mysql

Reference: https://serverfault.com/questions/812719/mysql-mariadb-not-starting

Afterwards, you’ll be prompted to reset the root password.

A CLI prompt to reset the root password after installing mariadb

You’ll still need to import your sql dump backups. I used phpMyAdmin to do that.

Scheduled backups

AWS Setup

The first step was to get things configured in my Amazon Web Services (AWS) console. I created a new S3 bucket. I also created a new IAM user, and added it to a group that included the permission policy “AmazonS3FullAccess”.

AWS policy to allow full S3 access
This policy provides full access to all buckets.

I went to the security credentials for that user, and copied down the access key ID and secret. I would use that info to access my S3 bucket programatically. All of the remaining steps take place from the command line, via SSH, against my server. From a Mac terminal, you could use a command like this to connect to an EC2 instance:

ssh -i /Users/antpace/Documents/keys/myKey.pem ec2-user@ec2-XX-XX-XX.us-west-2.compute.amazonaws.com

Once connected, I installed the software to allow programatic access to AWS:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Here is the reference for installing the AWS CLI on Linux.

Shell script

Shell scripts are programs that can be run directly by Linux. They’re great for automating tasks. To create the file on my server I ran: “nano backup.sh”. This assumes you already have the nano text editor installed. If not: “sudo yum install nano” (or, “sudo apt install nano”, depending on your Linux flavor).

Below is the full code I used. I’ll explain what each part of it does.

Credit: This code was largely inspired by a post from Marcelo Gornstein.

#!/bin/bash
AWS_ACCESS_KEY_ID=XXX \
AWS_SECRET_ACCESS_KEY=XXX \
S3_BUCKET=myBucketsName \
MYSQL_HOST=localhost \
MYSQL_PORT=3306 \
MYSQL_USER=XXX \
MYSQL_PASS=XXX \
MYSQL_DB=XXX \

cd /tmp
file=${MYSQL_DB}-$(date +%a).sql
mysqldump \
  --host ${MYSQL_HOST} \
  --port ${MYSQL_PORT} \
  -u ${MYSQL_USER} \
  --password="${MYSQL_PASS}" \
  ${MYSQL_DB} > ${file}
if [ "${?}" -eq 0 ]; then
  gzip ${file}
  AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} aws s3 cp ${file}.gz s3://${S3_BUCKET}
  rm ${file}.gz
else
  echo "sql dump error"
  exit 1
fi

The first line tells the system what interpreter  to use: “#!/bin/bash”. Bash is a variation of the shell scripting language. The next eight lines are variables that contain details about my AWS S3 bucket, and the MySQL database connection.

After switching to a temporary directory, the filename is built. The name of the file is set to the database’s name plus the day of the week. If that file already exists (from the week previous), it’ll be overwritten.  Next, the sql file is created using mysqldump and the database connection variables from above. Once that operation is complete, then we zip the file, upload it to S3, and delete the zip from our temp folder.

If the mysqldump operation fails, we spit out an error message and exit the program. (Exit code 1 is a general catchall for errors. Anything other than 0 is considered an error. Valid error codes range between 1 and 255.)

Before this shell script can be used, we need to change its file permissions so that it is executable: “chmod +x backup.sh”

After all of this, I ran the file manually, and made sure it worked: “./backup.sh”

Sure enough, I received a success message. I also checked the S3 bucket and made sure the file was there.

S3 file dump

Scheduled Cronjob

The last part is to schedule this script to run every night. To do this, we’ll edit the Linux crontab file: “sudo crontab -e”. This file controls cronjobs – which are scheduled tasks that the system will run at set times.

The file opened in my terminal window using the vim text editor – which is notoriously harder to use than the nano editor we used before.

I had to hit ‘i’ to enter insertion mode. Then I right clicked, and pasted in my cronjob code. Then I pressed the escape key to exit insertion mode. Finally, I typed “wq!” to save my changes and quit.

Remember how crontab works:

minute | hour | day-of-month | month | day-of-week

I set the script to run, every day, at 2:30am:

30 2 * * * sudo /home/ec2-user/backup.sh

And that’s it. I made sure to check the next day to make sure my cronjob worked (it did). Hopefully now, I won’t lose production data ever again!

Updates

Request Time Too Skewed (update)

A while after setting this up, I randomly checked my S3 buckets to make sure everything was still working. Although it had been for most of my sites, one had not been backed up in almost 2 months! I shelled into that machine, and tried running the script manually. Sure enough, I received an error: “An error occurred (RequestTimeTooSkewed) when calling the PutObject operation: The difference between the request time and the current time is too large.

I checked the operating system’s current date and time, and it was off by 5 days. I’m not sure how that happened. I fixed it by installing and running “Network Time Protocol”:

sudo yum install ntp
sudo ntpdate ntp.ubuntu.com

After that, I was able to run my backup script successfully, without any S3 errors.

 


Nano text-editor tip I learned along the way:

You can delete chunks of text content using Nano. Use CTRL + Shift + 6 to enter selection mode, move the cursor to expand the block, and press CTRL + K to delete it.

Additional References:

https://www.javatpoint.com/steps-to-write-and-execute-a-shell-script

https://www.sitepoint.com/cron-jobs/