Automatic MySQL dump to S3

Automatic MySQL dump to S3

I have had some lousy luck with databases. In 2018, I created a fitness app for martial artists, and quickly gained over a hundred users in the first week. Shortly after, the server stopped resolving and I didn’t know why. I tried restarting it, but that didn’t help. Then, I stopped the EC2 instance from my AWS console. Little did I know, that would wipe the all of the data from that box. Ouch.

Recently, a client let me know that their site wasn’t working. A dreaded “error connecting to the database” message was all that resolved. I’d seen this one before – no sweat. Restarting the database usually does the trick: “sudo service mariadb restart”. The command line barked back at me: “Job for mariadb.service failed because the control process exited with error code.”

Uh-oh.

The database was corrupted. It needed to be deleted and reinstalled. Fortunately, I just happen to have a SQL dump for this site saved on my desktop. This was no way to live – in fear of the whims of servers.

Part of the issue is that I’m running MySQL on the same EC2 instance as the web server.  A more sophisticated architecture would move the database to RDS. This would provide automated backups, patches, and maintenance. It also costs more.

To keep cost low, I decided to automate MySQL dumps and upload to an S3 bucket. S3 storage is cheap ($0.20/GB), and data transfer from EC2 is free.

AWS Setup

The first step was to get things configured in my Amazon Web Services (AWS) console. I created a new S3 bucket. I also created a new IAM user, and added it to a group that included the permission policy “AmazonS3FullAccess”.

AWS policy to allow full S3 access
This policy provides full access to all buckets.

I went to the security credentials for that user, and copied down the access key ID and secret. I would use that info to access my S3 bucket programatically. All of the remaining steps take place from the command line, via SSH, against my server. From a Mac terminal, you could use a command like this to connect to an EC2 instance:

ssh -i /Users/antpace/Documents/keys/myKey.pem ec2-user@ec2-XX-XX-XX.us-west-2.compute.amazonaws.com

Once connected, I installed the software to allow programatic access to AWS:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Here is the reference for installing the AWS CLI on Linux.

Shell script

Shell scripts are programs that can be run directly by Linux. They’re great for automating tasks. To create the file on my server I ran: “nano backup.sh”. This assumes you already have the nano text editor installed. If not: “sudo yum install nano” (or, “sudo apt install nano”, depending on your Linux flavor).

Below is the full code I used. I’ll explain what each part of it does.

Credit: This code was largely inspired by a post from Marcelo Gornstein.

#!/bin/bash
AWS_ACCESS_KEY_ID=XXX \
AWS_SECRET_ACCESS_KEY=XXX \
S3_BUCKET=myBucketsName \
MYSQL_HOST=localhost \
MYSQL_PORT=3306 \
MYSQL_USER=XXX \
MYSQL_PASS=XXX \
MYSQL_DB=XXX \

cd /tmp
file=${MYSQL_DB}-$(date +%a).sql
mysqldump \
  --host ${MYSQL_HOST} \
  --port ${MYSQL_PORT} \
  -u ${MYSQL_USER} \
  --password="${MYSQL_PASS}" \
  ${MYSQL_DB} > ${file}
if [ "${?}" -eq 0 ]; then
  gzip ${file}
  AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} aws s3 cp ${file}.gz s3://${S3_BUCKET}
  rm ${file}.gz
else
  echo "sql dump error"
  exit 1
fi

The first line tells the system what interpreter  to use: “#!/bin/bash”. Bash is a variation of the shell scripting language. The next eight lines are variables that contain details about my AWS S3 bucket, and the MySQL database connection.

After switching to a temporary directory, the filename is built. The name of the file is set to the database’s name plus the day of the week. If that file already exists (from the week previous), it’ll be overwritten.  Next, the sql file is created using mysqldump and the database connection variables from above. Once that operation is complete, then we zip the file, upload it to S3, and delete the zip from our temp folder.

If the mysqldump operation fails, we spit out an error message and exit the program. (Exit code 1 is a general catchall for errors. Anything other than 0 is considered an error. Valid error codes range between 1 and 255.)

Before this shell script can be used, we need to change its file permissions so that it is executable: “chmod +x backup.sh”

After all of this, I ran the file manually, and made sure it worked: “./backup.sh”

Sure enough, I received a success message. I also checked the S3 bucket and made sure the file was there.

S3 file dump

Scheduled Cronjob

The last part is to schedule this script to run every night. To do this, we’ll edit the Linux crontab file: “sudo crontab -e”. This file controls cronjobs – which are scheduled tasks that the system will run at set times.

The file opened in my terminal window using the vim text editor – which is notoriously harder to use than the nano editor we used before.

I had to hit ‘i’ to enter insertion mode. Then I right clicked, and pasted in my cronjob code. Then I pressed the escape key to exit insertion mode. Finally, I typed “wq!” to save my changes and quit.

Remember how crontab works:

minute | hour | day-of-month | month | day-of-week

I set the script to run, every day, at 2:30am:

30 2 * * * sudo /home/ec2-user/backup.sh

And that’s it. I made sure to check the next day to make sure my cronjob worked (it did). Hopefully now, I won’t lose production data ever again!

 


Nano text-editor tip I learned along the way:

You can delete chunks of text content using Nano. Use CTRL + Shift + 6 to enter selection mode, move the cursor to expand the block, and press CTRL + K to delete it.

 

Drop down with CSS arrow

blog post about css menu with arrow

Here’s a quick one about how to create a drop-down UI element with an arrow via CSS. Our aim is to create a menu that has sub-menus that drop-down when hovering over an item. Each drop-down sub-menu should have an arrow that points up towards the parent element.

Here’s the HTML to structure the menu:

<div class="menu">
  <div class="menu-item">
    <span>Menu Item</span>
    <div class="drop-down-menu">
      <p>Sub-item</p>
      <p>Sub-item</p>
      <p>Sub-item</p>
    </div>
  </div>
  
  <div class="menu-item">
    <span>Menu Item 2</span>
  </div>
  
  <div class="menu-item">
    <span>Menu Item 3</span>
    <div class="drop-down-menu">
      <p>Sub-item</p>
      <p>Sub-item</p>
      <p>Sub-item</p>
    </div>
  </div>
  
</div>

I nest the sub-menu within the parent item, and use CSS to show the it when a user mouses-over:

.menu-item:hover .drop-down-menu{
  display: block;
}

I line the menu-item in a row by setting display to ‘inline-block’. I use that, instead of just ‘inline’, so that its height property is respected. This is important because I will create space between the parent item and sub-menu. If the two elements don’t actually overlap, then the hover state will be lost, closing the drop-down. See what I mean:

Drop down menu with CSS
The sub-menu element overlaps with the parent item, so that the hover state is not lost.

I also set the parent item position to relative, so that the drop-down will be absolutely positioned respective to it.

.menu-item{
  cursor: pointer;
  height: 50px;
  display: inline-block;
  position: relative;
}

Since the menu items have a height larger than the actual content, I apply borders to a child span within them:

.menu-item:first-child span{
  border:none;
}
.menu-item span{
  border-left: 1px solid black;
  padding: 0 10px;
}

The sub-menu styling is straight-forward. I set it to display: none, set a width, add a border and padding, and position it absolutely. Its top value pushes it off of the parent item a bit. Setting a left value to zero ensures that it will be aligned with its parent. (If you don’t set a left value, multiple sub-menus will all stack under the very first parent item.)

.drop-down-menu{
  display: none;
  width: 100px;
  position: absolute;
  background: white;
  border: 1px solid #301B46;
  padding: 20px;
  top: 40px;
  left: 0px;
}

The next step is building the arrow in the drop-down menu. The challenge is making the arrow’s border blend seamlessly with the container’s border. The illusion is achieved by overlapping the :before and :after pseudo-elements.

The triangle shape that forms the arrow is achieved by giving  a bottom border to an element with no height or width. This code pen animation does a phenomenal job of explaining the idea: https://codepen.io/chriscoyier/pen/lotjh

The drop-down’s :before element creates the white triangle that is the heart of the arrow itself. This also creates the gap in the sub-menu’s actual top border

The :after element creates another triangle, this is behind and slightly above the first one – creating the illusion of a border that connects just right with the menu’s.

The illusion can be better revealed by manipulating the position and border-width of these pseudo-elements in the inspector.

Here is the code I used for those pseudo elements:

.drop-down-menu:before {
  content: "";
  position: absolute;
  border-color: rgba(194, 225, 245, 0);
  border: solid transparent;
  border-bottom-color: white;
  border-width: 11px;
  margin-left: -10px;
  top: -21px;
  right: 65px;
  z-index: 1;
} 

.drop-down-menu:after {
    content: "";
    position: absolute;
    right: 66px;
    top: -21px;
    width: 0;
    height: 0;
    border: solid transparent;
    border-width: 10px;
    border-bottom-color: #2B1A41;
    z-index: 0;
}

You can view the whole thing in action here: https://codepen.io/pacea87/pen/OJywqrj

Code generated from CSS Arrow Please helped me a lot when I was first figuring out how to do this right.

Create a WordPress plugin

Distributing software to app and plugin markets is a great way to gain organic traffic. Last year I submitted BJJ Tracker to the Google Play store as a Progressive Web App. Since then, I get signups every few days – with zero marketing effort.

I created a WordPress plugin for SplitWit, to grow its reach in a similar way. SplitWit helps run A/B experiments on the web. A JavaScript snippet needs to  be added to your code for it to work. This plugin injects the code snippet automatically.

Here is the process I took to develop and submit it to the WordPress plugin directory.

Plugin code

Since this is such a simple plugin, all I needed was one PHP file, and a readme.txt file. “At its simplest, a WordPress plugin is a PHP file with a WordPress plugin header comment.

The header comment defines meta-data:

/*

Plugin Name: SplitWit
Plugin URI: https://www.splitwit.com/
Description: This plugin automatically adds the SplitWit code snippet to your WordPress site. SplitWit lets you create a variation of your web page using our visual editor. It splits traffic between the original version and your variation. You can track key metrics, and let SplitWit determine a winner. No code needed. Go to SplitWit to register for free.
Author: SplitWit
Version: 1.0
License: GPLv2+
License URI: http://www.gnu.org/licenses/gpl-2.0.txt

*/

My PHP code defines six functions, uses four action hooks, and one filter hook.

The first function injects the SplitWit snippet code into the WordPress website’s header:

function splitwit_header_code(){
	//inject SplitWit code snippet
	 
	$splitwit_project_id = get_option('splitwit_project_id');
	 
	if($splitwit_project_id){
		wp_enqueue_script( 'splitwit-snippet', 'https://www.splitwit.com/snippet/'.$splitwit_project_id.'.js' );
	}
}
add_action( 'wp_head', 'splitwit_header_code', 1 );

Another defines the WordPress plugin’s menu page:

function splitwit_plugin_menu_page() { ?>

	<div>
		<h1>SplitWit Experiments</h1>
		<p>This plugin automatically adds the <a href="https://www.splitwit.com" target="_blank">SplitWit</a> code snippet to your WordPress site. <a href="https://www.splitwit.com" target="_blank">SplitWit</a> lets you create a variation of your web page using our visual editor. It splits traffic between the original version and your variation. You can track key metrics, and let SplitWit determine a winner. No code needed.
		</p>
		<p>You'll need to create an account at SplitWit.com - it's free. After signing up, you can create a new SplitWit project for your website. Find that project's ID code, and copy/paste it into this page.</p>
		
		<form method="post" action="options.php">
			<?php settings_fields( 'splitwit_settings_group' ); ?>
			<input style="width: 340px;display: block; margin-bottom: 10px;" type="text" name="splitwit_project_id" value="<?php echo get_option('splitwit_project_id'); ?>" />
			<input type="submit" class="button-primary" value="Save" />
		</form>
	</div>

<?php }

I add that menu page to the dashboard:

function splitwit_plugin_menu() {
	add_options_page('SplitWit Experiments', 'SplitWit Experiments', 'publish_posts', 'splitwit_settings', 'splitwit_plugin_menu_page');
}
add_action( 'admin_menu', 'splitwit_plugin_menu' );

And link to it in the Settings section of the dashboard:

function splitwit_link( $links ) {
     $links[] ='<a href="' . admin_url( 'options-general.php?page=splitwit_settings' ) .'">Settings</a>';
    return $links;
}
add_filter('plugin_action_links_'.plugin_basename(__FILE__), 'splitwit_link');

When the SplitWit code snippet is injected into the website’s header, it needs to reference a project ID. I register that value from the menu page:

function splitwit_settings(){
	register_setting('splitwit_settings_group','splitwit_project_id','string');
}
add_action( 'admin_init', 'splitwit_settings' );

If the project ID value has not been defined, I show a warning message at the top of the dashboard:

function splitwit_warning(){
  if (!is_admin()){
     return;
  }

  $splitwit_project_id = get_option("splitwit_project_id");
  if (!$splitwit_project_id || $splitwit_project_id < 1){
    echo "<div class='notice notice-error'><p><strong>SplitWit is missing a project ID code.</strong> You need to enter <a href='options-general.php?page=splitwit_settings'>a SplitWit project ID code</a> for the plugin to work.</p></div>";
  }
}
add_action( 'admin_notices','splitwit_warning');

The readme.txt defines additional meta-data. Each section corresponds to parts of the WordPress plugin directory page. The header section is required, and includes some basic fields that are parsed to the plugin page UI.

=== SplitWit ===
Contributors: SplitWit
Plugin Name: SplitWit
Plugin URI: https://www.splitwit.com
Tags: split test, split testing, ab testing, conversions
Requires at least: 2.8
Tested up to: 5.3.2
Stable tag: 1.0

Optimize your website for maximum convertibility. This plugin lets you use SplitWit to run experiments on your WordPress website.

I also added sections for a long description and installation instructions. Later, I included a screenshots section (see Subversion repo).

Submit for review

Plugin zip files can be uploaded to WordPress.org.  Plugins can also be distributed to WordPress users without this step – but having it listed in the WordPress directory lends credibility and visibility. After my initial submission, I received an email indicating issues with my code and requesting changes. The changes were simple: “use wp_enqueue commands” and “document use of an external service”.

Originally, my “splitwit_header_code()” function include the SplitWit JS snippet directly as plain text. I changed it to use the built-in function “wp_enqueue_script()”.

//wrong:
echo '<script type="text/javascript" async src="https://www.splitwit.com/snippet/'.$splitwit_project_id.'.js"> </script>';

//correct:
wp_enqueue_script( 'splitwit-snippet', 'https://www.splitwit.com/snippet/'.$splitwit_project_id.'.js' );

Next, they wanted me to disclose the use of SplitWit, the service that powers the plugin. I added this to my readme.txt:

This plugin relies on SplitWit, a third-party service. The SplitWit service adds code to your website to run A/B experiments. It also collects data about how users interact with your site, based on the metrics you configure.

After making these changes, I replied back with an updated .zip. A few days later I received approval. But, that wasn’t the end  – I still needed to upload my code to a WordPress.org hosted SVN repository.

Subversion Repo

I’ve used Git for versioning my entire career. I had heard of SVN, but never used it. What a great opportunity to learn!

The approval email provided me with a SVN URL. On my local machine, I created a new folder, “svn-wp-splitwit”. From a terminal, I navigated to this directory and checked out the pre-built repo:

svn co https://plugins.svn.wordpress.org/splitwit

I added my plugin files (readme.txt and splitwit.php) to the “trunk” folder. This is where the most up-to-date, ready-to-distribute, version of code belongs.

In the “tags” folder, I created a new directory called “1.0” and put a copy of my files there too – for the sake of version control. This step is completely optional and is how SVN handles revisions.

In the assets folder I included my banner, icon, and screenshot files. The filenames follow as prescribed by WordPress.org. I made sure to reference the screenshot files in my readme.txt file, under a new “Screenshots” section.

Finally, I pushed my code back up to the remote:

 svn ci -m "Initial commit of my plugin."

You can now find my plugin in the WordPress.org plugin directory. SplitWit is available for a free trial. Give it a try, and let me know what you think.

 


 

Pro-tip: Some WordPress setups won’t let you to install plugins from the dashboard with out providing FTP credentials, including a password. If you use a key file, instead of a password, this is a roadblock.

Install WordPress plugin roadblock
Not everyone uses a password to connect to their server.

You can remedy this by defining the file system connection method in your functions.php file:

define( 'FS_METHOD', 'direct' );

 

Secure a website with SSL and HTTPS on AWS

SSL, certbot, https

My last post was about launching a website onto AWS. This covered launching a new EC2 instance, configuring a security group, installing LAMP software, and pointing a domain at the new instance. The only thing missing was to configure SSL and HTTPS.

Secure Sockets Layer (SSL) encrypts traffic between a website and its server. HTTPS is the protocol to deliver secured data via SSL to end-users.

In my last post, I already allowed all traffic through port 443 (the port that HTTPS uses) in the security group for my EC2 instance. Now I’ll install software to provision SSL certificates for the server.

Certbot

Certbot is free software that will communicate with Let’s Encrypt, an SSL certificate authority, to automate the management of encryption certificates.

Before downloading and installing Certbot, we’ll need to install some dependencies (Extra Packages for Enterprise Linux). SSH into the EC2 instance that you want to secure, and run this command in your home directory (/home/ec2-user):

sudo wget -r --no-parent -A 'epel-release-*.rpm' http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/

Then install it:

sudo rpm -Uvh dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-*.rpm

And enable it:

sudo yum-config-manager --enable epel*

Now, we’ll need to edit the Apache (our web hosting software) configuration file. Mine is located here: /etc/httpd/conf/httpd.conf

You can use the Nano CLI text editor to make changes to this file by running:

sudo nano /etc/httpd/conf/httpd.conf

Scroll down a bit, and you’ll find a line that says “Listen 80”. Paste these lines below (obviously, changing antpace.com to your own domain name)

<VirtualHost *:80>
    DocumentRoot "/var/www/html"
    ServerName "antpace.com"
    ServerAlias "www.antpace.com"
</VirtualHost>

Make sure you have an A record (via Route 53) for both yourwebsite.com AND www.yourwebsite.com with the value set as your EC2 public IP address.

After saving, you’ll need to restart the server software:

sudo systemctl restart httpd

Now we’re ready for Certbot. Install it:

sudo yum install -y certbot python2-certbot-apache

Run it:

sudo certbot

Follow the prompts as they appear.

Automatic renewal

Finally, schedule an automated task (a cron job) to renew the encryption certificate as needed. If you don’t do this part, HTTPS will fail for your website after a few months. Users will receive an ugly warning, telling them that your website is not secure. Don’t skip this part!

Run this command to open your cron file:

sudo nano /etc/crontab

Schedule Certbot to renew everyday, at 4:05 am:

05 4 * * * root certbot renew --no-self-upgrade

Make sure your cron daemon is running:

sudo systemctl restart crond

That’s it! Now your website, hosted on EC2 will support HTTPS. Next, we’ll force all traffic to use it.

* AWS Documentation Reference

Launching a website on AWS

Using AWS for a website

In 2008 I deployed my first website to production. It used a simple LAMP stack , a GoDaddy domain name, and HostGator hosting.

Since 2016, I’ve used AWS as my primary cloud provider. And this year, I’m finally cancelling my HostGator package. Looking through that old server, I found artifacts of past projects – small businesses and start-ups that I helped develop and grow. A virtual memory lane.

Left on that old box was a site that I needed to move to a fresh EC2 instance. This is an opportunity to document how I launch a site to Amazon Web Services.

Amazon Elastic Compute Cloud

To start, I launch a new EC2 instance from the AWS console. Amazon’s Elastic Compute Cloud provides “secure and resizable compute capacity in the cloud.” When prompted to choose an Amazon Machine Image (AMI), I select “Amazon Linux 2 AMI”. I leave all other settings as default. When I finally click “Launch”, it’ll ask me to either generate a new key file, or use an existing one. I’ll need that file later to SSH or sFTP into this instance. A basic Linux server is spun up, with little else installed.

Linux AMI
Amazon Linux 2 AMI is free tier eligible.

Next, I make sure that instance’s Security Group allows inbound traffic on SSH, HTTP, and HTTPS. We allow all traffic via HTTP and HTTPS. That way end-users can reach the website from a browser. Inbound SSH access should not be left wide open. Only specific IP addresses should be allow to command-line in to the server.

Inbound security rules
Don’t allow all IPs to access SSH in a live production environment.

Configure the server

Now that the hosting server is up-and-running, I can command-line in via SSH from my Mac’s terminal using the key file from before. This is what the command looks like:

 ssh -i /Users/apace/my-keys/keypair-secret.pem ec2-user@ec2-XXX-XXX-XX.us-east-2.compute.amazonaws.com

Make sure everything is up-to-date by running “sudo yum update“.  I begin installing the required software to host a website:

sudo amazon-linux-extras install -y lamp-mariadb10.2-php7.2 php7.2

That command gives me Apache, PHP, and MariaDB – a basic LAMP stack. This next one installs the database server:

sudo yum install -y httpd mariadb-server

MariaDB is a fork of the typical MySQL, but with better performance.

Start Apache: “sudo systemctl start httpd“. And, make sure it always starts when the server boots up “sudo systemctl enable httpd

The server setup is complete. I can access an Apache test page from a web browser by navigating to the EC2 instance’s public IP address.

Apache test page
A test page shows when no website files are present.

I’ll take my website files (that are stored on my local machine and synched to a Git repo) and copy them to the server via sFTP.

Copy website files to server
I use FileZilla to access my EC2 public directory

I need to make sure the Linux user I sFTP with owns the directory “/var/www/html”, or else I’ll get a permission denied error: sudo chown -R ec2-user /var/www/html

* AWS Documentation Reference

Domain name and Route 53

Instead of having to use the EC2 server’s public address to see my website from a browser, I’ll point a domain name at it. AWS Route 53 helps with this. It’s a “DNS web service” that routes users to websites by mapping domain names to IP addresses.

In Route 53 I create a new “hosted zone”, and enter the domain name that I’ll be using for this site.  This will automatically generate two record sets: a Name Server (NS) record and a Start-of-Authority (SOA) record. I’ll create one more, an IPv4 address (A) record. The value of that record should be the public IP address that I want my domain to point at. You’ll probably also want to add another, identical to the last one, but specifying “www” in the record name.

Finally, I’ll head over to my domain name registrar, and find my domain name’s settings. I update the nameserver values there to match those in my Route 53 NS record set. It’ll likely take some time for this change to be reflected in the domain’s settings. Once that is complete, the domain name will be pointing at my new EC2 instance.

And that’s it – I’ve deployed my website to AWS. The next step is to secure my site by configuring SSL and HTTPS.

 

React JS & Yup: only require an input, if another is not empty

React JS and Yup

Typically, I avoid using JS app frameworks, and default to plain vanilla JavaScript. But, in keeping up with what is current – and working on projects as part of a team – React is inevitable: “A JavaScript library for building user interfaces” . Yup is the go-to form validation library in this context. Its GitHub page calls it “Dead simple Object schema validation”.

Yup creates validation schemas for inputs. Create a Yup validation object, and wire it up to a form – easy.

The ask

Setting: An existing React project, with a signup form. The form includes address inputs. The “country” input was not a required field – it could be left blank. My assignment was to make that field be required, only if the “state/territory” input was not empty. Sounds straight forward.

Here is a sample of the original code:

export const apValidateMyAddress = () => {
  name: yup.string().required("Don't leave this blank!!"),
  email: yup.string().email(),
  address: yup:string(),
  city: yup.string(),
  state: yup.string(),
  country: yup.string()
}

At first, I wasn’t sure if I should update this schema code directly. I thought about checking if the state field was blank, or not, and applying a different schema object instead. That would have been the wrong approach.

Doing some research, I discovered that the Yup’s when() method was the solution. It would let me “adjust the schema based on a sibling or sibling children fields”.

My first attempt was wrong, and didn’t work::

export const apValidateMyAddress = () => {
  name: yup.string().required("Don't leave this blank!!"),
  email: yup.string().email(),
  address: yup:string(),
  city: yup.string(),
  state: yup.string(),
  country: yup.string().when('state',{
    is: true,
    then: yup.string().required('This is a required field.')
  })
}

Errors were thrown. Documentation examples were hard to come by, and I was new at this. I wanted the condition to be true if “state” was not blank. Setting the “is” clause as “true” would only work if state was validated as a boolean – state: yup.boolean() . Ultimately, I was able to check that the “state” value existed using the value property:

export const apValidateMyAddress = () => {
  name: yup.string().required("Don't leave this blank!!"),
  email: yup.string().email(),
  address: yup:string(),
  city: yup.string(),
  state: yup.string(),
  country: yup.string().when('state',{
    is: (value: any) => !!value,
    then: yup.string().required('This is a required field.')
  })
}

 

This was a great learning experience. I’m excited to learn and use more React JS.

Sending email from your app using AWS SES

amazon ses

Simple Email Service (SES) from AWS

Email is the best way that we can communicate with our users; still better than SMS or app notifications. An effective messaging strategy can enhance the journey our products offer.

This post is about sending email from the website or app you’re developing. We will use SES to send transactional emails. AWS documentation describes Simple Email Service (SES) as “an email sending and receiving service that provides an easy, cost-effective way for you to send email.” It abstracts away managing a mail server.

Configuring your domain name

The first step to sending email through SES is to verify the domain name we’ll want messages coming from. We can do this from the “Domains” dashboard.

Verify a new domain name
Verify a new domain name

This will generate a list of record sets that will need to be added to our domain as DNS records. I use Route 53, another Amazon service, to manage my domains – so that’s where I’ll need to enter this info.

AWS Route 53

Understand deliverability

We want to be confident that intended recipients are actually getting the messages that are sent.  Email service providers, and ISPs, want to prevent being abused by spammers. Following best practices, and understanding deliverability, can ensure that emails won’t be blocked.

Verify any email addresses that you are sending messages from: “To maintain trust between email providers and Amazon SES, Amazon SES needs to ensure that its senders are who they say they are.”

Make sure DKIM has been verified for your domain:  “DomainKeys Identified Mail (DKIM) provides proof that the email you send originates from your domain and is authentic”. If you’re already using Route 53 to manage your DNS records, SES will present an option to automatically create the necessary records.

Route 53 DKIM records

Be reputable. Send high quality emails and make opt-out easy. You don’t want to be marked as spam. Respect sending quotas. If you’re plan on sending bulk email to a list-serve, I suggest using an Email Service Provider such as MailChimp (SES could be used for that too, but is outside the scope of this writing).

 

Sending email

SES can be used three ways: either by API, the SMTP interface, or the console. Each method lists different ways to authenticate. “To interact with [Amazon SES], you use security credentials to verify who you are and whether you have permission to interact with Amazon SES.” We will use the API credentials – an access key ID and secret access key.

Create an access key pair

An access key can be created using Identity and Access Management (IAM). You use access keys to sign programmatic requests that you make to AWS.” This requires creating a user, and setting its permissions policies to include “AmazonSESSendingAccess”. We can create an access key in the “security credentials” for this user.

Permission policy for IAM user
Permission policy for IAM user

Integrating with WordPress

Sending email from WordPress is made easy with plugins. They can be used to easily create forms. Those forms can be wired to use the outbound mail server of our choice using WP Mail SMTP Pro. All we’ll need to do is enter the access key details. If we try to send email without specifying a mail server, forms will default to sending messages directly from the LAMP box hosting the website. That would result in low-to-no deliverability.

Screenshot of WP Mail SMTP Pro
Screenshot of WP Mail SMTP Pro

Integrating with custom code

Although the WordPress option is simple, the necessary plugin has an annual cost. Alternatively, SES can integrate with custom code we’ve written. We can use PHPMailer to abstract away the details of sending email programmatically. Just include the necessary files, configure some variables, and call a send() method.

Contact form powered by SES
Contact form powered by SES

The contact forms on my résumé  and portfolio webpages use this technique. I submit the form data to a PHP file that uses PHPMailer to interact with SES. The front-end uses a UI notification widget to give the user alerts. It’s available on my GitHub, so check it out.

Front-end, client-side:

<form id="contactForm">
    <div class="outer-box">
      
        <input type="text" placeholder="Name" name="name" value="" class="input-block-level bordered-input">
        <input type="email" placeholder="Email" value="" name="email" class="input-block-level bordered-input">
        <input type="text" placeholder="Phone" value="" name="phone" class="input-block-level bordered-input">
       
        <textarea placeholder="Message" rows="3" name="message" id="contactMessage" class="input-block-level bordered-input"></textarea>
        <button type="button" id="contactSubmit" class="btn transparent btn-large pull-right">Contact Me</button>
    </div>
</form>
<link rel="stylesheet" type="text/css" href="/ui-messages/css/ui-notifications.css"> 
<script src="/ui-messages/js/ui-notifications.js"></script>
<script type="text/javascript">
$(function(){
	var notifications = new UINotifications();
	$("#contactSubmit").click(function(){
		var contactMessage = $("#contactMessage").val();
		if(contactMessage < 1){
			notifications.showStatusMessage("Don't leave the message area empty.");
			return;
		}
		var data = $("#contactForm").serialize();
		$.ajax({
			type:"POST",
			data:data,
			url:"assets/contact.php",
			success:function(response){
				console.log(response);
				notifications.showStatusMessage("Thanks for your message. I'll get back to you soon.");
				$("form input, form textarea").val("");					
			}
			
		});
	});
});
</script>

In the PHP file,  we set the username and password as the access key ID and access key secret. Make sure the region variable matches what you’re using in AWS. #TODO: It would be best practice to record the message to a database. (The WordPress plugin from earlier handles that out-of-the-box). We might also send an additional email to the user, letting them know their note was received.

Back-end, server-side:

<?php
//send email via amazon ses
use PHPMailer\PHPMailer\PHPMailer;
use PHPMailer\PHPMailer\Exception;

$name = "";
$email = "";
$phone = "";
$message = "";

if(isset($_POST["name"])){
	$name = $_POST["name"];
}
if(isset($_POST["email"])){
	$email = $_POST["email"];
}
if(isset($_POST["phone"])){
	$phone = $_POST["phone"];
}
if(isset($_POST["message"])){
	$message = $_POST["message"];
}

$region = "us-east-1"
$aws_key_id = "xxx"
$aws_key_secret = "xxx"

require '/var/www/html/PHPMailer/src/Exception.php';
require '/var/www/html/PHPMailer/src/PHPMailer.php';
require '/var/www/html/PHPMailer/src/SMTP.php';
// // Instantiation and passing `true` enables exceptions
$mail = new PHPMailer(true);
try {
	if(strlen($message) > 1){
    //Server settings
	    $mail->SMTPDebug = 2;                                       // Enable verbose debug output
	    $mail->isSMTP();                                            // Set mailer to use SMTP
	    $mail->Host       = 'email-smtp.' . $region . '.amazonaws.com';  // Specify main and backup SMTP servers
	    $mail->SMTPAuth   = true;                                   // Enable SMTP authentication
	    $mail->Username   = $aws_key_id;                     // access key ID
	    $mail->Password   = $aws_key_secret;                               // AWS Key Secret
	    $mail->SMTPSecure = 'tls';                                  // Enable TLS encryption, `ssl` also accepted
	    $mail->Port       = 587;                                    // TCP port to connect to

	    //Recipients
	    $mail->setFrom('XXX@antpace.com', 'Portfolio');
	    $mail->addAddress("XXX@antpace.com");     // Add a recipient
	    $mail->addReplyTo('XXX@antpace.com', 'Portfolio');

	    // Content
	    $mail->isHTML(true);                                  // Set email format to HTML
	  
	    $mail->Subject = 'New message from your portfolio page.';
	    $mail->Body    = "This message was sent from: $name - $email - $phone \n Message: $message";
	    $mail->AltBody = "This message was sent from: $name - $email - $phone \n Message: $message";
	    
	    $mail->send();
	    echo 'Message has been sent';
	}
    
} catch (Exception $e) {
    echo "Message could not be sent. Mailer Error: {$mail->ErrorInfo}";
}

?>

The technical side of sending email from software is straight-forward. The strategy can be fuzzy and requires planning. Transactional emails have an advantage over marketing emails. Since they are triggered by a user’s action, they have more meaning. They have higher open rates, and in that way afford an opportunity.

How can we optimize the usefulness of these emails? Be sure to create a recognizable voice in your communication that resonates your brand. Provide additional useful information, resources, or offers. These kind of emails are an essential part of the user experience and your product’s development.

 

Innovation in 2020

Innovation

Innovation moves the world forward. An original idea can make a measurable difference in the results of a plan or project. Creativity is a prerequisite for innovation. It can take existing concepts and rearrange them to fit a unique pattern.

Creativity

In another post, I highlighted solutions I used to manage a large amount of content for a client project. I leveraged existing features and technology to achieve results in a unique way. Another way to put it: I used technical knowledge to overcome a problem, creatively. Some other examples of creative solutions include:

1) Using an existing CSS feature to deliver a better or unique user experience. When applying animations and effects, the possibilities seem limitless.

2) A doctor prescribing a drug, off label, to treat a problem for which it wasn’t originally intended.

3) A trained athlete combining existing techniques into a unique style.

Although creative, these examples aren’t innovative. Innovation creates something new. It creates new products, new industries, and new markets.

 

Innovation

Innovation changes the space in which you’re working. In the next decade I will focus on that kind of growth, and expand past client-specific work. As a broad stroke, this means building digital products. Specifically, I’ll be taking the opportunity to solve problems in certain areas. Luckily, there are many exciting problems that need solving.

Innovative solutions require energy – and here are some places I’d like to spend mine.

1) Digital accessibility, and solving technology problems for people with disabilities. This will include software, as well as physical products. I’d like to explore how IOT, wearables, and augmented/virtual reality can be leveraged.

2) Privacy. This issue also seems to include homelessness, the justice system, and personal identity.

3) Business and marketing. These solutions are important, because I can re-use them as tools in other ventures. They can be leveraged to solve other important problems.

For example, SplitWit is a product I’ve built that can help companies experiment with user experiences. Failing fast, and measuring results, is imperative to growth and innovation.

SplitWit marketing
www.SplitWit.com

My reason for writing this post is to flesh out, and crystalize, how innovation should direct my work. I’ve discovered that writing helps me to think better. Writing makes me more innovative. These blog entries will continue to be a way that I can solidify my nebula of creative energy into something solid and digestible.

 

Managing content for an art website

Any product, or experience, or artwork – anything you will build – is made up of pieces. And content always sits at the center. Content is the fleshy part of media.

The other pieces include structure, style, and functionality.  These parts layout a skeleton, decorates the aesthetic, and adds usefulness. This  model translates well to modern web development. HTML defines the structure. CSS describes the style. JavaScript adds interactivity. But always, content is King.

That’s why a robust content management system (CMS) is critical. Most clients prefer to have one. It makes content updates easy. WordPress is the modern choice. It’s what this blog is built on.

A website I built featured the work of a visual artist – paintings, etchings, photos. It had a lot of content. A lot of content that needed massaging. As you may have guessed, I chose WordPress to manage it.

This was a situation where I had to be a project manager, and deliver results. Although the content itself was impressive, it was delivered as image files in various formats and different sizes. Filenames were not consistent. And the meta-data – descriptions, titles, notes – was listed is excel files that didn’t always match-up to the image’s filename. This required a lot of spot checking, and manual work. I did my best to automate as much as I could, and make things uniform.

Resizing multiple images

Resizing a batch of images can be done directly in Mac OS by selecting the files, and opening them in Preview. From the ‘Edit’ menu, I clicked ‘Select All’. Then, in the ‘Tool’ menu I found ‘Adjust Size’. Windows has a similar feature, as does other image manipulation apps.

Renaming multiple files

I had to make the filenames match what was listed in the meta-data spreadsheet. Here’s the command I used, in Mac OS, to truncate filenames to the first eight characters:

rename -n 's/(.{8}).*(\.jpg)$/$1$2/' *.jpg

Batch uploading WordPress posts

Each piece of art was a WordPress post, with a different title, meta-values, and image. Once all of the files were sized and named properly, I uploaded them to the server via sFTP. Each category of art (paintings, photos, etc.) was a folder. I created a temporary database table that matched the columns from the meta-data spreadsheet I was given.

CREATE TABLE `content` (
  `content_id` int,
  `title` varchar(250) NOT NULL,
  `medium` varchar(250) NOT NULL,
  `category_id` varchar(250) NOT NULL,
  `size` varchar(250) NOT NULL,
  `date` varchar(250) NOT NULL,
  `filename` varchar(100) NOT NULL,
  `processed` int
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
COMMIT;

I wrote a PHP script that would loop through all records, and create a new post for each. I had to make sure to include core WordPress functionality,  so that I would be able to use the wp_insert_post() method.

require_once('/var/www/html/wp-load.php');

Once I connected to the database, I queried my temporary table, excluding any records that have been marked as already uploaded:

$query = "SELECT * FROM `content` where `processed` != 1"; 
$result = mysqli_query($mysql_link, $query);

While looping through each record, I would look up the WordPress category ID and slug based on the provided category name. This would allow my code to assign the post to the correct category, and to know which folder the image file was in. Once the post is inserted, I take that post ID and assign meta-values. At the end of the loop, I mark this record as processed.

while ($row = mysqli_fetch_assoc($result)) {

    $category = $row['category'];
    $content_id = $row['content_id'];
    $term_id = "";
    $slug = "";
    $category_query = $mysqli->prepare("SELECT * FROM `wp_terms` where name = :name");
    $category_query->bind_param(array(':name' => $category));
    $category_result = $category_query->execute();
    if (mysqli_num_rows($category_result) > 0) {
        while($cat_row = mysqli_fetch_assoc($category_result)) {
           $term_id = $cat_row['term_id'];
           $slug = $cat_row['slug'];
        }
    }
    $post_id = wp_insert_post(array(
        'post_status' => 'publish',
        'post_title' => $row['title'],
        'post_content' => " ",
        'post_category' => $term_id
        
    ));
    
    if ($post_id) { 
        //meta-values
        add_post_meta($post_id, 'medium', $row['medium']);
        add_post_meta($post_id, 'size', $row['size']);
        add_post_meta($post_id, 'date', $row['date']);
        $img = $slug . $row['image'];
        add_post_meta($post_id, 'image_file', $img);
    }

    $update = $mysqli->prepare("UPDATE `content` SET processed = 1 where content_id = :content_id");
    $update->bind_param(array(':content_id' => $content_id));
    $update = $category_query->execute(); 
}

Managing clients, and their content, can be the most challenging  part of web development. Using the right software for the job makes it easier. So does having a toolbox of techniques, and being clever.

Image carousel – update

Image carousel update

In a previous post, I wrote about creating an image carousel using basic web tech: HTML, CSS, and vanilla JavaScript. No frameworks, no jQuery. This is an update to that. The major difference is that it supports multiple carousels on the same page. I also added a try/catch, in case no carousel data is found in the database. I recently used this implementation on a WordPress site. Each carousel was a post (of a custom carousel post-type), that had each image attached. On that post-type archive page, I looped through the posts, and created a separate carousel for each.

Here is the updated JavaScript.

try{
    var galleries = document.getElementsByClassName("carousel-class");
    for(var i = 0; i < galleries.length; i++){
        showGalleries(galleries.item(i), 0);    
    }
}catch(e){
    console.log(e);
}

function showGalleries(gallery, galleryIndex){
    var galleryDots = gallery.getElementsByClassName("dot-button");
    var gallerySlides = gallery.getElementsByClassName("my-slide");
    if (galleryIndex < 0){galleryIndex = gallerySlides.length-1}
    galleryIndex++;
    for(var ii = 0; ii < gallerySlides.length; ii++){ gallerySlides[ii].style.display = "none"; galleryDots[ii].classList.remove('active-dot'); } if (galleryIndex > gallerySlides.length){galleryIndex = 1}
    gallerySlides[galleryIndex-1].style.display = "block";
    var resizeEvent = new Event('resize');
    window.dispatchEvent(resizeEvent);
    galleryDots[galleryIndex-1].classList.add('active-dot');
    //hide gallery navigation, if there is only 1
    if(gallerySlides.length < 2){
        var dotContainer = gallery.getElementsByClassName("dots");
        var arrowContainer = gallery.getElementsByClassName("gallery-arrows");
        dotContainer[0].style.display = "none";
        arrowContainer[0].style.display = "none";
    }
    gallery.setAttribute("data", galleryIndex);
}

//gallery dots
document.addEventListener('click', function (event) {
    if (!event.target.matches('.carousel-class .dot-button')){ return; }
    var index = event.target.getAttribute("data"); 
    var parentGallery = event.target.closest(".carousel-class")
    showGalleries(parentGallery, index);

}, false);

//gallery arrows

//left arrow
document.addEventListener('click', function (event) {
    if (!event.target.matches('.fa-arrow-left')){ return; }
    var parentGallery = event.target.closest(".carousel-class")
    var galleryIndex = parentGallery.getAttribute("data");
    galleryIndex = galleryIndex - 2;
    
    showGalleries(parentGallery, galleryIndex);
}, false);

//right arrow
document.addEventListener('click', function (event) {
    if (!event.target.matches('.fa-arrow-right')){ return; }
    var parentGallery = event.target.closest(".carousel-class")
    var galleryIndex = parentGallery.getAttribute("data");

    showGalleries(parentGallery, galleryIndex);
}, false);

You’ll notice that each carousel section has a data attribute assigned, so our JS knows which one to affect. This version also includes left and right navigation arrows, in addition to the navigation dots we already had.

HTML:

<div class="ap-carousel" data="0">

<?php $num_slides = 0; foreach($posts as $post){ $num_slides++; ?>

	<div class="ap-slide">
		<a href="<?php the_permalink($post->ID); ?>" title="<?php the_title(); ?>">
			<img src="<?php echo esc_url(get_the_post_thumbnail_url($post->ID)); ?>" class="zoom">
		</a>
	</div>

<?php } ?>
<div class="nav-dots">
	<?php $active = "active-dot"; for($x = 0; $x < $num_slides; $x++){ ?>
		<div class="dot"><button data="<?php echo $x; ?>" type="button" class="dot-button <?php echo $active; $active = ''; ?>">b</button></div>
	<?php } ?>
</div>
<div class="gallery-arrows">
    <i class="fas fa-arrow-left"></i>
    <i class="fas fa-arrow-right"></i>
</div>


</div>

I emphasize simplicity when building solutions. I avoid including superfluous code libraries when a vanilla technique works. It’s helpful to keep track of solutions I engineer, and try to reuse them where they fit. And when they need to be adjusted to work with a new problem, I enhance them while still trying to avoid complexity.