I have an S3 bucket that I want to attach to an application’s upload area, but I want to move them out of the bucket accessible to the application after they’ve been uploaded. Eventually, I want to have this be after a small delay, but initially I wanted to test out the concept itself.
Step 1: Have source and destination buckets in S3
Create buckets for source and destination. The ACLs on both of the buckets are the same (non-public) in my case.
Step 2: Create a Lambda Execution Role
Go to IAM > Roles > Create Role
Choose Lambda as a Use Case
Search for S3 and check AmazonS3FullAccess
Search for “lambdabasic” and check AWSLambdaBasicExecutionRole (for CloudWatch logs)
Click [Next: Tags] > [Next: Review] and give your role a name and verify that the S3 and Lambda policies are added:
Be sure to have the correct ruby version (2.7.0 at the time of writing) installed
Change into move_file_on_upload folder
bundle install locally
bundle config set --local deployment 'true' to vendor the gems for the AWS Lambda
zip ../move_file_on_upload.zip ** to package the zip
Step 4: Create the Lambda
Go to AWS Lambda in the AWS Console and click [Create Function]
Name the function, set Ruby 2.7 as the runtime, and use the role you created
Step 5: Add S3 Trigger
Click [+ Add Trigger]
Search and select S3
Fill in your source bucket and select all object create events
If you get this error (“configurations overlap”), select your bucket in S3, click the Properties tab, and you’ll see an Event Notifications that’s been orphaned by a previous config (be sure to delete the dependent Lambda as well if it exists):
Step 6: Upload your code
Go back to the [Code] tab for your lambda and select the [Upload from] dropdown and select zip file.
Go to the [Configuration] section and add a FROM_BUCKET and TO_BUCKET environment variable for your Lambda to know what to process
Step 7: Monitor and test
You can test the Lambda execution via the Test dropdown
S3 put is one of the available test templates
Click “Create” after selecting S3 Put and you’ll be able to watch the event get executed.
Go to CloudWatch -> Logs -> Log Groups and you should see a log group for /aws/lambda/your_function_goes_here
If all else is successful, you should see “the specified key does not exist”
Step 8: Test it live
Create a folder in your source bucket.
Go into that folder
Upload a file.
The file should pretty quickly be copies to the destination bucket in the same folder structure and removed from its original location.
The top level folder under the bucket should remain.
If you’re serving up a static website from S3, especially if you have larger assets stored there, you may want to put monitoring on the requests or bytes downloaded from S3, just to make sure someone’s not running up terabytes of transfers or millions of requests.
Enabling Metrics on S3
You will have to enable metrics on S3 in order to get CloudWatch alarms on them.
Go to Services -> S3 -> Buckets and select the bucket for your static site.
Select Management tab and [Metrics] and then click on the pencil icon next to the bucket icon.
Once enabled, the metrics will take a bit to populate.
First, you’re going to want to be notified. Go to Services -> Simple Notification Service to set up a pathway for that to happen.
Next, click “Topics” and then [Create topic]
Name your topic something that adequately describes the purpose (I just used domainname-com)
Scroll down to [Create Topic]
Set up a subcription
Under “Amazon SNS” left sidebar, click “Subscriptions” and [Create Subscription]
Click on the Topic ARN field and you should be able to see an ARN with your topic name as the last part of the ARN. Click that ARN
Under Protocol, select your preferred method of notification (I’m going with SMS.
Under Endpoint, enter your cell number, including country code (+18005551212 for (800) 555-1212 in the US)
Setting up a CloudWatch Alarm
Go to Services -> CloudWatch -> Alarms and [Create alarm]
[Select metric] and select S3
If you don’t see “Request Metrics per Filter” then the metrics haven’t started populating yet.
Check “GetRequests” or “BytesDownloaded” and [Select Metric]
Set conditions as you would like to have flag any anomalies and click [Next]
Choose “In Alarm” and “Select an existing SNS topic” and click in the box below “Send Notification To…” to get suggestions and select the SNS topic corresponding to the notification method you set up. Click [Next]
In the window that opens from “Request or Import a Certificate with ACM”, enter your domain name (domainname.com) and click [Next]
Select DNS validation and click [Next]
Click [Confirm and Request] if the details look correct.
Expand the domain in validation:
Click [Create record in Route 53] and confirm by clicking [Create] again.
You’ll be waiting from several minutes to half an hour for the validation to happen, during which time status will display as “Pending validation”
Click [Continue] to finish the request process and go back to the Certificate Manager main screen.
Click the (refresh icon) button to update status, and when status turns to “Issued” you are ready to use it in CloudFront.
Setting up a CloudFront distribution
In the AWS Console, go to Services → Cloudfront
Click [Create Distribution]
Click [Get Started] under Web
Under “Origin Domain Name” select the selection under “Amazon S3 buckets” that corresponds to your static web site bucket. (e.g., domainname.com.s3.amazonaws.com)
Optional: Restrict Bucket Access [Yes] so that you can control access through the CloudFront distribution alone.
Set “Origin Access Identity” to “Create a new identity”
Set “Grant Read Permissions on Bucket” to “Yes, Update Bucket Policy”
Under Viewer Protocol Policy I select “Redirect HTTP to HTTPS” just to keep things uniform.
Set up SSL
Under Alternate Domain Names, enter your domain name (e.g., domainname.com)
Select “Custom SSL Certificate”
Click “Request or Import a Certificate with ACM”
If you go back to CloudFront you should be able to select “Custom SSL Certificate” now and the certificate corresponding to your domain name should show up in suggestions:
Scroll down and leave defaults until you get to “Default Root Object”. You’ll want to set this to the name of the document to bring up (e.g., index.html) if the user browses to / on the domain.
Optional: I set Logging to On and selected my logging bucket that I used for the static site as the bucket, adding a log prefix for it.
To finish, click [Create Distribution]
You may be waiting quite a while for changes to propagate to the edge locations, but at some point before the “In Progress” changes to “Deployed” you will be able to pull up via the domain listed under the “Domain Name” column in your list of CloudFront Distributions.
Pointing the domain name at your distribution
Go back to Route 53 and go into the hosted zone for your domain name
Check the checkbox next to your A record and then go up to Actions -> Edit
Change “Value/Route traffic to” from “Alias to S3 endpoint” to “Alias to CloudFront distribution” in the “Choose Distribution” input box.
Enter the domain name (“asdfkjdfasoiadsf9u.cloudfront.net.”) as your domain name. (The new interface wasn’t suggesting distributions like the last version of the interface did… it may change next week, of course.)
Locking down S3
If you selected “Restrict bucket access” and had CloudFront update your S3 policy, your public access setting on the bucket is still unaffected. You’ll want to remove that:
Go back to Services -> Amazon S3
Go to your domainname.com bucket
Click Block public access
Check “Block all public access” and click [Save]
Some other details
Type in a bucket name for static site logging (i.e., domainname.com-logs)
Accept next all the way to Create Bucket
Click [ + Create Bucket ]
Type in your domain name (for example, “domainname.com”)
If you created a logs bucket:
Check “Log requests for access to your bucket” under “Server access logging”
Enter the logging bucket name under the “Target bucket” field.
Hit [Next] for Configure Options
Under Set Permissions, uncheck “Block all public access” and check the box that says “I acknowledge that the current settings may result in this bucket and the objects within becoming public”
Click Create Bucket
Creating a static site
Drag and drop your static HTML files and assets within the site root in your project and drag to S3. Before sure you have the html file you want to use as an index and as an error page.
Click [Next] on the Upload modal.
Under “Manage public permissions” change “Do not grant public read access to this object(s)” to “Grant public read access to this object(s)” and click [Next]
Click [Upload] from the “Set Properties” step (skip Storage Class configuration, etc. screen).
Under the “Properties” tab for your bucket, click on the “Static website hosting” tile
Select “Use this bucket to host a website”, enter the name of your index and error documents, and click [Save].
You should be able to click the link under endpoint and see your index page.
Create a hosted zone for your domain in Route 53
In Route 53, select Hosted zone link on the left of the console
Click [Create hosted zone]
Enter your Domain name
Select “Public hosted zone”
Click “Create hosted zone”
If you have a domain purchased elsewhere than AWS, copy the name servers under the “Hosted zone details” and set on your domain. (e.g., on Namecheap, it’s under Domain->Nameservers->Custom DNS… BE SURE TO HIT THE GREEN CHECKMARK AFTER EDITS!!)
Create a Record in the Hosted Zone
Select “Create record”
Select “Simple record”
Under “Define simple record”
leave the record name blank
Value/Route traffic to
Alias to S3 website endpoints
Select your region
In the field that says “Choose S3 Bucket”, you should see your bucket as an option:
To finish, click [Define simple record] and then [Create records]