DKIM for Cloudflare and Fastmail

After creating the CNAME fm#._domainkey records in Cloudflare DNS Management, I was still getting “DKIM is not configured”. Initially I thought it was a TTL issue on the DNS records…

After playing around with dig fm1._domainkey.DOMAINNAME TYPE CNAME I went back to check the settings on the CNAME records:

Proxied CNAME records!

I went in to edit those to be DNS Only

Now my Fastmail DKIM is correctly configured

Ruby on Jets Webpacker errors and Invalid Configuration Object

The problem

tl;dr to the solution that worked for me

I was trying to get Ruby on Jets up and running and ran into webpacker errors including “CLI for webpack must be installed” (and others) all the way to “Invalid configuration object”.

On creating a new app with jets (3.0.8) with npm 7.11.2, node 16.1.0, and yarn 1.22.5 and creating a basic blog CRUD app like follows:

 jets new blog_app --database=postgresql
 cd blog_app
 jets generate scaffold post title:string post:text
 jets db:create
 jets db:migrate
 jets server

I get an error with browsing to localhost:8888/posts:

ActionView::Template::Error at /posts
Webpacker can't find application.js in /mnt/c/Users/twill/projects/jets/blog_app/public/packs/manifest.json. Possible causes: 1. You want to set webpacker.yml value of compile to true for your environment unless you are using the `webpack -w` or the webpack-dev-server. 2. webpack has not yet re-run to reflect updates. 3. You have misconfigured Webpacker's config/webpacker.yml file. 4. Your webpack configuration is not creating a manifest. Your manifest contains: { }

On the back end, I could see:

[Webpacker] Compilation failed:
warning package.json: No license field
CLI for webpack must be installed.
webpack-cli (
We will use "npm" to install the CLI via "npm install -D webpack-cli".
Do you want to install 'webpack-cli' (yes/no):

Adding dependencies one by one

I added webpack-cli with yarn add webpack-cli and:

[webpack-cli] Failed to load '/mnt/c/Users/twill/projects/jets/blog_app/config/webpack/development.js' config
[webpack-cli] Error: Cannot find module '@rails/webpacker'

Ok, add yarn add @rails/webpacker and run jets server and reload the page:

[Webpacker] Compilation failed:
 warning package.json: No license field
 [webpack-cli] Invalid configuration object. Webpack has been initialized using a configuration object that does not match the API schema.
 configuration.node should be one of these: false | object { __dirname?, __filename?, global? } -> Include polyfills or mocks for various node stuff. Details: configuration.node has an unknown property 'dgram'. These properties are valid:
 object { __dirname?, __filename?, global? }
 -> Options object for node compatibility features.
 configuration.node has an unknown property 'fs'. These properties are valid:
 object { __dirname?, __filename?, global? }
 -> Options object for node compatibility features.
 configuration.node has an unknown property 'net'. These properties are valid:
 object { __dirname?, __filename?, global? }
 -> Options object for node compatibility features.
 configuration.node has an unknown property 'tls'. These properties are valid:
 object { __dirname?, __filename?, global? }
 -> Options object for node compatibility features.
 configuration.node has an unknown property 'child_process'. These properties are valid:
 object { __dirname?, __filename?, global? }
 -> Options object for node compatibility features.  

That led me eventually to Webpack 5: configuration.node has an unknown property ‘dram’. There properties are valid: issue comment on GitHub, with the mention of rails webpacker and that “Rails’ webpacker 5.x.x is only compatible with webpack 4.x.x”

The Solution

After lots of teardowns and tweaks to the project to troubleshoot, I finally ended up with a sequence that produces a scaffold for posts that renders with webpack:

jets new blog_app --database=postgresql
cd blog_app
jets generate scaffold post title:string post:text

# jets db:drop # used while experimenting with yard packages below
jets db:create
jets db:migrate

# this is the key line that makes the currently installed @rails/webpacker work
yarn add webpack@4 
yarn add webpack-cli
yarn add @rails/webpacker
jets server

Automatically Moving Files Between S3 Buckets with Lambda (part 1)

I have an S3 bucket that I want to attach to an application’s upload area, but I want to move them out of the bucket accessible to the application after they’ve been uploaded. Eventually, I want to have this be after a small delay, but initially I wanted to test out the concept itself.

Step 1: Have source and destination buckets in S3

Create buckets for source and destination. The ACLs on both of the buckets are the same (non-public) in my case.

Step 2: Create a Lambda Execution Role

  • Go to IAM > Roles > Create Role
  • Choose Lambda as a Use Case
  • Next: Permissions
  • Search for S3 and check AmazonS3FullAccess
AmazonS3FullAccess selection
  • Search for “lambdabasic” and check AWSLambdaBasicExecutionRole (for CloudWatch logs)
AWSLambdaBasicExecutionRole selection
  • Click [Next: Tags] > [Next: Review] and give your role a name and verify that the S3 and Lambda policies are added:
Verify policies and name role
  • Click [Create Role]

Step 3: Prep the Lambda Code

  • Clone
  • Be sure to have the correct ruby version (2.7.0 at the time of writing) installed
  • Change into move_file_on_upload folder
  • bundle install locally
  • bundle config set --local deployment 'true' to vendor the gems for the AWS Lambda
  • zip ../ ** to package the zip

Step 4: Create the Lambda

  • Go to AWS Lambda in the AWS Console and click [Create Function]
  • Name the function, set Ruby 2.7 as the runtime, and use the role you created
Function name, Runtime, and Permissions
  • [Create function]

Step 5: Add S3 Trigger

  • Click [+ Add Trigger]
  • Search and select S3
  • Fill in your source bucket and select all object create events
  • If you get this error (“configurations overlap”), select your bucket in S3, click the Properties tab, and you’ll see an Event Notifications that’s been orphaned by a previous config (be sure to delete the dependent Lambda as well if it exists):
Configurations overlap error

Step 6: Upload your code

  • Go back to the [Code] tab for your lambda and select the [Upload from] dropdown and select zip file.
Upload .zip file
  • Go to the [Configuration] section and add a FROM_BUCKET and TO_BUCKET environment variable for your Lambda to know what to process

Step 7: Monitor and test

  • You can test the Lambda execution via the Test dropdown
  • S3 put is one of the available test templates
  • Click “Create” after selecting S3 Put and you’ll be able to watch the event get executed.
  • Go to CloudWatch -> Logs -> Log Groups and you should see a log group for /aws/lambda/your_function_goes_here
  • If all else is successful, you should see “the specified key does not exist”

Step 8: Test it live

  • Create a folder in your source bucket.
  • Go into that folder
  • Upload a file.
  • The file should pretty quickly be copies to the destination bucket in the same folder structure and removed from its original location.
  • The top level folder under the bucket should remain.

Inauspicious Start for Oracle Cloud Sign up

I heard via word of mouth and Twitter that Oracle Cloud was offering a Free Tier (with permanently free services [for now]). The always free services looked enticing enough:

AMD and ARM and object storage!

The challenge was, “Who can afford ‘free’ services?” Time is worth something. But I can always make use of another cloud server to run experiments on.

Problem #1: Email confirmation didn’t go through

Self-explanatory, but, yes… I checked my spam and all the auto-sorting tabs. The email confirmation link that’s only good for 30 minutes didn’t deliver in a timely manner. Second attempt, the email showed up immediately.

Problem #2: Password Too Strong

My first 30 character randomly generated password didn’t pass the test:

I think I met the requirements??

Problem #3: Wouldn’t validate my debit card

Maybe there’s a payment glitch right now? Maybe I don’t have enough in the account for a “free” account? Worse… the “try again” link makes you start over from the very first step of creating your account.

Problem #4: Declined my credit card

After going through 2-3 times with a debit card, I tried with a credit card. Maybe I needed five figures of available credit for a free account? This is Oracle, after all.

Upon resubmitting, I’m back to “Error processing transaction”

Aha! Moment

(b) in the above error message was the clue that eventually led me to the right answer… VPN (still US-based) was active, which possibly set off alarm bells with the payment processor. I’m in now and ready to try some VMs!

MySQL Ignores Right Padded Spaces?

When working on an established Rails project, we noticed that trailing spaces weren’t getting trimmed off. Discussion ensued about whether to do data cleanup and then someone noticed that the where method in Rails didn’t care about the spaces. Not having seen this in my PostgreSQL days, I was assuming that there was some magic happening in either the Rails codebase or the MySQL adapter. So I went to the underlying DB: No, searching for column='hi' found values where the column value was actually 'hi '.

In searching on StackOverflow I found this answer states that MySQL ignores trailing spaces in string comparison, echoed in the MySQL 5.7 documentation:

All MySQL collations are of type PAD SPACE. This means that all CHARVARCHAR, and TEXT values are compared without regard to any trailing spaces. “Comparison” in this context does not include the LIKE pattern-matching operator, for which trailing spaces are significant. 

Of course, I’m using MySQL 8.0 and so is the app. So I dug into the collation, setting up the following table:

mysql> select concat('"', thing, '"'), unthing from surelynot;
| concat('"', thing, '"') | unthing |
| "hi "                   | what    |
| " hi "                  | what2   |
| " hi"                   | what3   |
| "hi"                    | what4   |
4 rows in set (0.02 sec)

Setting the collation to MySQL 8.x’s default, I get one row for 'hi':

mysql> alter table surelynot convert to character set utf8mb4 collate utf8mb4_0900_ai_ci;
Query OK, 4 rows affected (0.16 sec)
Records: 4  Duplicates: 0  Warnings: 0

mysql> select concat('"', thing, '"'), unthing from surelynot where thing = 'hi';
| concat('"', thing, '"') | unthing |
| "hi"                    | what4   |
1 row in set (0.05 sec)

However, using the latin1 or latin2 collation that was on the other database, I get two results for 'hi', one for the exact match, and one for the right-padded. Note that left-padding has no impact on the results:

mysql> select concat('"', thing, '"'), unthing from surelynot where thing = 'hi';
| concat('"', thing, '"') | unthing |
| "hi "                   | what    |
| "hi"                    | what4   |
2 rows in set (0.00 sec)

The ucs2_general_ci also has this property as does the utf8_general_ci collation. Ultimately, only the default utf8mb4_0900_ai_ci paid attention to right padded spaces when looking for string equality in the where clause.

XML Handler Bypassing ParseError Handling Rack Middleware on Rails 6

In a project using actionpack-xml_parser, and implementing a modified version of Catching Invalid JSON Parse Errors with Rack Middleware to also handle XML errors, an upgrade to Rails 6 broke the handling of the ParseError. Interestingly enough, JSON was being handled as expected.

TL;DR: The short answer to this is that config/initializers/wrap_parameters.rb only had json mentioned as a format. This can be remedied in the initializer or on a per controller basis. The entire app fix was just adding :xml to the initializer:

ActiveSupport.on_load(:action_controller) do
  wrap_parameters format: [:json, :xml]

In looking through the stack traces/call stack for where the parse error occurs (ActionDispatch::Http::Parameters#parse_formatted_parameters) using the caller array in the gem source and backtrace in the rescue, I was able to catch that ActionController::Metal::ParamsWrapper#_wrapper_enabled? was called for JSON, but not for XML. Ultimately, I was able to track down the difference to ActionController::Metal::ParamsWrapper#_wrapper_formats?, which was only returning [:json], which led to the wrap_parameters functionality in Rails.

Pandemic Brisket

Sliced BrisketAt the beginning of the pandemic last year, I wasn’t dealing well with it. The scare TP and *everything else* phase hit a little hard on top of all of the other panic. So I smoked a brisket. Here’s the general procedure:

Equipment / supplies


  • 1/4 c packed brown sugar
  • 2 tbsp smoked paprika
  • 2 tbsp sea salt
  • 1 tbsp garlic powder
  • 1 tsp onion powder
  • 1 tbsp chili powder
  • 1 tsp black pepper


I had better luck with brisket from Sam’s Club than Costco, but this was at the early phases of the pandemic, so that might have been part of the problem. 10 lb brisket, good marbling and a fairly trim layer of fat on one side.


40% oak wood and 60% charcoal aiming for 250-275℉ on the hood thermometer. I let the oak wood burn fuel for the fire with whatever smoke it produced. I laid the brisket fat side up at the other end of the charcoal grill chamber from the smoker box and maintained charcoal/wood fire for 5 hours.

Low grill indirect heat

After about 5 hours, I’d gotten as much smoke as I wanted, so I put the brisket on an old cookie sheet and wrapped fairly tightly with heavy duty aluminum foil and put in the propane grill chamber opposite the burner that was lit as low as it would burn. At this point I stuck in the digital thermometer and monitored for 195℉. Once it hit target temperature, I brought it inside and let it cool a bit (mostly to make sure everything else is ready).


I know it will be blasphemy to most, but you can carve up extra thick slices (~1 inch thick) and freeze, for later slow warming in the toaster oven.

Vortex Race 3 TKL LED Programming for Mac

I decided to switch to my Vortex Race 3 TKL for a more programming-centric job from the Vortex Tab 90M that felt a little better for email and spreadsheets. The Race 3 has Cherry MX Silent Reds vs. the Vortex Tab 90M’s Cherry MX Silent Blacks.

A couple of things that were a little bit difficult to follow were:

  1. How to set up the Race 3 for Mac (it had been used on a Windows PC most recently and never really used on a Mac). This Reddit thread gives some suggestions on how to optimize that.
  2. How to program the LEDs (for just basic effects):

The LED programming was fairly simple once I understood it (see here for a copy of the manual). The cool LED effects when you type are accessible by pressing the Pn key + one of the ‘1’ through ‘5’ keys. ‘4’ and ‘5’ have multiple effects:

Pn + ‘4’ cycles different “display single color LED mode”s which means that the interactive LED effects appear in context with your

  • [no effect?]
  • Interactive mode – keys light up as you type.
  • Flash vortex mode – keys pulse around the keyboard originating from your keystrokes
  • Aurora mode – keys pulse in color around the keyboard originating from your keystrokes

Pn + ‘5’ cycles different “Display full color LED mode”s which means “display on all the keys regardless of context”:

  • [no effect]
  • Full key light mode – just 100% on
  • Breath mode – keyboard pulses
  • Vortex mode – RGBs cycle in different phases with each other.
  • Rain drop mode – random drops of “color” appear on the keys

Pn + X – brightness down for the above effects

Pn + V – brightness up for the above effects

Pn + [< key] and Pn + [> key] are LED speed up and down.

AWS Certified Developer – Associate (DVA-C01) Experience

It’s been a little over 1 month since my AWS Certified Solutions Architect – Associate (SAA-C02) exam. After finishing the Solutions Architect Associate exam, I immediately registered for the Developer Associate exam, which I took yesterday and passed.

Wasting Time

I took a practice exam (through Linux Academy) shortly after passing the SAA exam, having actually gone through most of a Developer Associate training course. DO NOT DO THIS UNLESS YOU ARE READY TO DIVE DEEP INTO ANYTHING YOU MISSED. I managed to get about 72% on the practice exam, which is essentially a passing score so I let myself lose focus over the holidays.

Next mistake was going through a couple of versions of the course (30 hours each) instead of just using the practice test results to read the white papers, FAQs, and AWS service pages I needed to review. (Linux Academy/aCloudGuru courses provide links to associated AWS reference for this.)

Foundations in SAA

I highly recommend getting the Solutions Architect – Associate first. It covers a good breadth of topics that will give you a solid foundation. I recommend the Linux Academy courses that have labs that you can walk through via a transcript just to traverse all the areas you need to know.

Differences from SAA

All of the AWS Code* services factor into a significant chunk of the DVA exam. If you have solid experience with docker, k8s, and git, you’ll probably have a better than 50% chance on questions about services that you didn’t look into, but there are also things that are about the “AWS way” or the “way that AWS services push you to do things” that I shook my head at. (Physical team organization? Wat?)


  • Learn how to set up X-Ray on the various compute environments.
  • Create and deploy a Serverless Application Model project and dig into the configuration files for it.
  • 0 bytes is the minimum object size, 100 MB is the recommended threshold to use multipart upload to S3, 5GB it is required, 5TB is the object size limit.
  • cfn-init
  • View Protocol Policy in CloudFront (https/http)
  • Bucket policies for enforcing server side encryption
  • Learn supported platforms for CodeDeploy
  • Learn lifecycle hooks for CodeDeploy (just look at the diagrams a few times)
  • Learn which services trigger Lambda asynchronously and synchronously
  • Lambda requires dependency packaging, and CPU is controlled by RAM allocation.
  • Dead letter queues
  • Lambda, by definition, does not do in-place deploy (because in-place requires a tangible server)

Reinstalling a Clean Copy of Mac OS X Yosemite

Use How to get old versions of macOS to download an installer for OS X Yosemite while running on Yosemite. Run the installer and it will install the full installer to /Applications/Install OS X

Use the command line from How to create a bootable installer for macOS substitution El\ Capitan for Yosemite in the El Capitan instructions.

Hold down option while powering on the Mac to get the option to boot from USB Recovery.