Raspberry Pi Zero W to monitor Enphase Envoy Solar Array

I decided to set up some form of monitoring for my solar installation after a fuse and the breaker panel broke down leaving me without solar generation for a couple stretches during near-peak, up to about 1,400 kWh, or about $140-210 worth of solar generation.

Missing output

Components (physical and software)

  • A RaspberryPi Zero W on the same wireless network as the Envoy controller was set up on (initially used PiBakery to configure hostname/wifi/username/password, but the project is a little bit stale at this point).
  • A Nexmo account (part of Vonage APIs now) to allow for SMS alerts on zero output when the sun is up.
  • RubySunrise for only emailing alerts from dusk until dawn.
  • Ruby Gmail and a Gmail account for email informational “down” alerts just to be aware that the cron job is running.
  • cron and gmail

Connections

These are described in the source code repo as well

  • ENVOY_HOST for me was envoy.local, but depending on your DNS situation, your mileage may vary. I got my local DNS in a weird enough state that I just looked up the envoy.local IP on my wireless router’s status page and used that.
  • USERNAME and PASSWORD are the Gmail username and app-specific password credentials I generated for the gmail account I used.
  • INVERTER_COUNT is compared to the number of inverters you should have so that even if the array is producing, you can still generate an error if one of them isn’t reporting (only valid when producing)
  • LATITUDE and LONGITUDE plucked from a site that displays your geolocation… this, along with your TZ represented in a form within the TZInfo::Timezone list, and RubySunrise allow you to figure out if the sun’s up.
  • NEXMO* are api keys and config from the Nexmo site (NEXMO_SMS_TO is your personal mobile to alert to)
  • TO_EMAIL is the email to actually mail to

Code

.config must be of the form that follows but the rest of the code can be cloned from envoy-rpi-zero-monitor

    USERNAME='some.burner.gmail.account'
    PASSWORD='gmai1@cc0untp@$$w0rd'
    TO_EMAIL='an.email.you.read@example.com'
    NEXMO_API_KEY="3ab3789123"
    NEXMO_API_SECRET="123456sSD8dh"
    NEXMO_SMS_FROM="19281123581"
    NEXMO_SMS_TO="15551112222"
    LATITUDE=20.1237899
    LONGITUDE=-57.3364631
    TZ='America/Chicago'
    ENVOY_HOST='192.168.1.222'
    INVERTER_COUNT=100
# crontab runs every hours and inits rbenv to use the right ruby version because
# I didn't really care about "production readiness"... it's a Raspberry Pi Zero W
0 * * * * cd /home/twill/envoy-rpi-zero-monitor && eval "$(rbenv init -)" && ruby read-envoy.rb

YMMV

This all depends on having an Enphase Enlighten Envoy (and a bunch of other random “E” names) as your solar monitor, but if you have a relatively recent solar install and your technician needed to configure the monitor for your wifi, then you probably have a similar device with a pollable endpoint. Look at your wireless router’s web console and you’ll see that monitor:

If you browse to that name or the IP address associated, you’ll probably get a web page with status. If you reload with the network tab up, you’ll probably see it retrieve the data via a .json endpoint:

From there, you can build your own monitor around it.


Referencing one trait from another trait in factory_bot

Sometimes you want to DRY up traits by referencing one trait from another trait in factory_bot. I tried searching on “inheriting traits” (that’s just for one factory inheriting traits from another and was in a factory_bot issue in GitHub). I accidentally stumbled upon the answer in a slightly unrelated StackOverflow question about calling a trait from another trait with params in factory_girl.

Ultimately, you use the trait name from the first trait as a method invocation in the referencing trait:

FactoryBot.define do
  factory :user do
    role
    trait :with_supervisor do
      # complex set up might go
      # here
      after(:create) do |user|
        supervisor { create(:user) }
      end
    end
    trait :with_organization do
      with_supervisor # invoke the other trait first
      organization
    end
  end
end

RAW_POST_DATA in rspec rails for Rails 5.2 and beyond

The last time I was trying to specify RAW_POST_DATA in rspec was probably Rails 3 or 4, but I ran into a situation trying to test an edge case for error handling where I wanted that same functionality. I quickly found this issue [Unable to POST raw request body], but didn’t immediately figure out what wasn’t being set correctly.

In this case the test setup I was using was setting multipart/form-data instead of application/xml on the content types:

{:HTTP_ACCEPT=>"application/xml", :HTTP_CONTENT_TYPE=>"multipart/form-data", :CONTENT_TYPE=>"multipart/form-data"}

Because of this, the Rails controller tests that rspec hooks into was trying to break following malformed xml down to parameters:

            <test>
              <data&nbsp;
              <![CDATA[THIS|IS|SENSITIVE|BUT|MALFORMED]]>
              </data>
            </test>
 Minitest::Assertion:
   Expected response to be a <400: bad_request>, but was a <422: Unprocessable Entity>
   Response body: <errors>
       <error>["<test>\n  <data", "nbsp;\n  <!"] are not permitted parameters</error>
   </errors>

I finally noticed that the mime-type might be involved. In this code, Content-Type was also an issue, so:

  • Removed HTTP_CONTENT_TYPE from the headers
  • Set CONTENT_TYPE header to 'application/xml' instead of 'multipart/form-data' to prevent automatic params parsing in this case.
  • Passed as: :xml into the test to get the 'mime-type' correct.

Ultimately, if your code hasn’t boxed you in, then the as: :xml and passing raw data as a parameter should work:

post things_path, params: raw_xml_data, headers: non_form_data_headers, as: :xml

## replacement for the following:
# @request.env['RAW_POST_DATA'] = raw_xml_data
# post things_path

String#tr in ruby (like tr in Linux) complete with figuring out slashes.

It seems like I’ve seen quite a few programming puzzles in the last few weeks that involved translating mistyped input in which the hands were shifted (right) on the keyboard. My first thought was the tr utility in *nix operating systems, but didn’t immediately go looking for or notice that ruby has a tr method on string. However, after doing a trivial implementation involving keyboard rows like the following, I stumbled on the tr method.

  # initial array of characters/strings to shift back to the left with [index-1]
  KEYBOARD_ROWS= [
    '`1234567890-=',
    'qwertyuiop[]\\', # need to escape the backslash or else debugging pain
    "asdfghjkl;'", # double-quotes here because single quote embedded
    'zxcvbnm,./',
    '~!@#\$%^&*()_+',
    'QWERTYUIOP{}|',
    'ASDFGHJKL:"',
    'ZXCVBNM<>?'
  ].join

Attempting to rewrite this for .tr presented a few challenges, however. If you are substituting for \, -, or ~, you have to escape the characters. You also have to escape them from their string representation, which makes for some head-spinning levels of escaping (zsh users who run shell commands through kubectl might be familiar with this pain as well):

# puts '\\~-'.tr('\\', 'a') # doesn't match because \ is passed to tr and not escaped
a~-
# puts '\\~-'.tr('\\\\', 'a') # now \\ is passed to tr, which is
a~-
# puts '\\~-'.tr("\\\\\\", 'a') # with double quotes, you need an extra pair, for 6 total.
a~-
# puts '\\~-'.tr('\\~', 'b') # the escaping backslash needs to be doubled
\b-
# puts '\\~-'.tr("\\\~", 'b') # the escaping backslash needs to be tripled
\b-
# puts '\\~-'.tr('\\-', 'c') # the escaping backslash needs to be doubled
\~c
# puts '\\~-'.tr("\\\-", 'c') # the escaping backslash needs to be tripled
\~c

So if you’re going to use translate to “shift” hands back to the left, the two arguments to tr, SHIFTED_KEYBOARD_ROWS and UNSHIFTED_KEYBOARD_ROWS would have to be defined with the following escaping:

  SHIFTED_KEYBOARD_ROWS =
    [
      '1234567890\\-=',
      'wertyuiop[]\\\\', # 4x backslash = backslash
      "sdfghjkl;'",
      'xcvbnm,./',
      '!@#\$%\^&*()_+',
      'WERTYUIOP{}|',
      'SDFGHJKL:"',
      'XCVBNM<>?'
  ].join

  UNSHIFTED_KEYBOARD_ROWS= [
    '`1234567890\-',
    'qwertyuiop[]', # need to escape the backslash or else debugging pain
    'asdfghjkl;',
    'zxcvbnm,.',
    '~!@#\$%\^&*()_',
    'QWERTYUIOP{}',
    'ASDFGHJKL:',
    'ZXCVBNM<>?'
  ].join

  def self.translate(string)
    string.tr(SHIFTED_KEYBOARD_ROWS, UNSHIFTED_KEYBOARD_ROWS)
  end

Tracing / Debugging ruby output like set -x in bash

In writing some shell scripts in ruby, I decided that I needed to be able to debug (trace) the lines that were being executed. I even ran across a closed StackOverflow question looking for the same thing.

code=ARGF.readlines.grep_v(/^$/)
eval code.map { |c| %Q|puts "+ #{c.gsub("\"", "\\"").strip}"| }.zip(code).join($/)

After playing around with one of the other answers (see above), I ended up taking a different tactic to try and figure out how to debug the scripts. (By the way, the above code breaks if you have line breaks in a single statement, like the following contrived example):

y = 2
      + 4

The important search term here is “trace”, or Tracer to be exact.

Take the following example:

bind = binding
p bind

bind.local_variable_set(:bind, 2)

p bind

p binding.local_variables

bind = binding

p eval("bind", bind)

if 2 > 3
  puts 2
else
  puts 3
end


if true
  puts "looks like the if true is compiled out"
end

if you run the above (contained in a filename binding.rb) using ruby -r tracer binding.rb then you get the following:

#0:/home/tpowell/.rbenv/versions/2.5.5/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:Kernel:<:       return gem_original_require(path)
#0:binding.rb:1::-: bind = binding
#0:binding.rb:2::-: p bind
#<Binding:0x000055fe6879cd38>
#0:binding.rb:4::-: bind.local_variable_set(:bind, 2)
#0:binding.rb:6::-: p bind
2
#0:binding.rb:8::-: p binding.local_variables
[:bind]
#0:binding.rb:10::-: bind = binding
#0:binding.rb:12::-: p eval("bind", bind)
#0:binding.rb:10::-: bind = binding
#<Binding:0x000055fe68835718>
#0:binding.rb:14::-: if 2 > 3
#0:binding.rb:17::-:   puts 3
3
#0:binding.rb:22::-:   puts "looks like the if true is compiled out"
looks like the if true is compiled out

One interesting difference between how ruby -r tracer works from set -x is that the ruby tracer appears skips evaluating the if true at all. The above runs were against ruby 2.5.5. Looking at 3.0.0 (and as far back as 2.6.x), I only get the output of the script:

#<Binding:0x000055af54e0c730>
2
[:bind]
#<Binding:0x000055af54e0c050>
3
looks like the if true is compiled out

Looking at Tracer, it’s using set_trace_func under the hood:

set_trace_func proc { |event, file, line, id, binding, classname|
  printf "%8s %s:%-2d %10s %8s\n", event, file, line, id, classname
}

Adding that in the 2.6.x+ world returns:

c-return binding.rb:1  set_trace_func   Kernel
    line binding.rb:5
  c-call binding.rb:5     binding   Kernel
c-return binding.rb:5     binding   Kernel
    line binding.rb:6
  c-call binding.rb:6           p   Kernel
  c-call binding.rb:6     inspect   Kernel
c-return binding.rb:6     inspect   Kernel
.
.
.

That output can be filtered by the event type , but the lines of code themselves aren’t output and apparently set_trace_func was apparently obsoleted as of 2.1.10. TracePoint is the updated way to accomplish this:

trace = TracePoint.new(:line) do |tp|
  p tp
end
trace.enable
.
.
.

But we still have the same problem:

#<TracePoint:line@binding_trace_point.rb:8>
#<TracePoint:line@binding_trace_point.rb:9>
#<Binding:0x00005652f2a125e8>
#<TracePoint:line@binding_trace_point.rb:11>
#<TracePoint:line@binding_trace_point.rb:13>
2
#<TracePoint:line@binding_trace_point.rb:15>
[:trace, :bind]

A crude solution I’ve found around this is to read the line from the file mentioned in the TracePoint from within the block (and this apparently doesn’t end up with a stack overflow).

trace = TracePoint.new(:line) do |tp|
  puts "+ #{File.open(tp.path) { |f| f.each_line.to_a[tp.lineno-1] }}"
end

trace.enable
.
.
.

Which produces a somewhat set -x output:

+ bind = binding
+ p bind
#<Binding:0x000055ef6c9d9f70>
+ bind.local_variable_set(:bind, 2)
+ p bind
2
+ p binding.local_variables
[:trace, :bind]
+ bind = binding
+ p eval("bind", bind)
+ bind = binding
#<Binding:0x000055ef6c8321b8>
+ if 2 > 3
+   if 3 > 2
+     puts 3
3
+   puts "looks like the if true is compiled out"
looks like the if true is compiled out


Rails’ and Ruby’s Hash transform_values

Rails: 4.2.1-5.2.3 and Ruby >= 2.5.5 have a transform_values method on Hash that allows you to pass a block to the method and transform the values of the key-value map in the hash based on the block contents. Essentially, it’s map but for the Hash values only, and with no weird Hash/array-element syntax. From the linked API documents above:

h = { a: 1, b: 2, c: 3 }
h.transform_values {|v| v * v + 1 }  #=> { a: 2, b: 5, c: 10 }
h.transform_values(&:to_s)           #=> { a: "1", b: "2", c: "3" }
h.transform_values.with_index {|v, i| "#{v}.#{i}" }
                                     #=> { a: "1.0", b: "2.1", c: "3.2" }


NoMethodError undefined method `shared_examples_for’ for main:Object for bundle gem rspec

If you create a gem stub using bundle gem thing and select rspec as your test suite, you may get an error similar to the following:

❯ bundle exec rspec

An error occurred while loading ./spec/thing_spec.rb.
Failure/Error:
  shared_examples_for 'saying hello' do
    puts "hi"
  end

NoMethodError:
  undefined method `shared_examples_for' for main:Object
# ./spec/thing_spec.rb:1:in `<top (required)>'
No examples found.

whenever using describe, shared_examples, and shared_examples_for, etc…

After a lot of diving into rspec source code to verify where shared_examples_for was defined (rspec-core so…) I noticed the following code in the stubbed spec_helper.rb:

  # Disable RSpec exposing methods globally on `Module` and `main`
  config.disable_monkey_patching!

If you comment out config.disable_monkey_patching!, then those methods will be included at a top level.


Ruby: Enumerable grep, grep_v

UPDATE: After I wrote this, I started finding myself doing a lot of caller.grep(/(project_directory|suspected_gem)/) to aid in debugging obscure interactions with internal gems and projects.

In looking at a pull request and noticing some awkward “first” and “last” iteration detection which also required each_with_index, I started looking into what would be a cleaner way, and my first step was trying to figure out if there was an enumerable context.

Ultimately I landed on Enumerable#grep and Enumerable#grep_v, which somewhat perplexed me. It’s not really a “grep” unless your collection’s values respond to that:

irb(main):056:0> 1000.times.map(&:to_s).grep(/00/)
=> ["100", "200", "300", "400", "500", "600", "700", "800", "900"]

Maybe you want to look for Classes in the ObjectSpace… The argument to #grep is compared against an implicit element for each iteration with the === operator. So you could list everything in the ObjectSpace that’s a Class:

ObjectSpace.each_object.grep(Class) # too long to include here

One situation that I thought of that might be especially useful is Dir globbing:

irb(main):064:0> Dir['*'].grep(/(yarn|json)/)
=> ["package-lock.json", "package.json", "yarn.lock"]

If you were trying to the Ruby REPL as a shell, you could even:

irb(main):005:0> Dir['*/'].grep_v(%r{/packs/}).grep(/(.js$|.erb$|.rb$|.json$|.lock$)/)
=> ["app/controllers/application_controller.rb", "app/controllers/posts_controller.rb", "app/helpers/application_helper.rb", "app/helpers/posts_helper.rb", "app/javascript/src/jets/crud.js", "app/jobs/application_job.rb", "app/models/application_item.rb", "app/models/application_record.rb", "app/models/post.rb", "app/views/layouts/application.html.erb", "app/views/posts/edit.html.erb", "app/views/posts/index.html.erb", "app/views/posts/new.html.erb", "app/views/posts/show.html.erb", "app/views/posts/_form.html.erb", "babel.config.js", "config/application.rb", "config/environments/development.rb", "config/environments/production.rb", "config/environments/test.rb", "config/routes.rb", "config/webpack/development.js", "config/webpack/environment.js", "config/webpack/production.js", "config/webpack/test.js", "db/migrate/20210610023540_create_posts.rb", "db/schema.rb", "Gemfile.lock", "postcss.config.js", "spec/controllers/posts_controller_spec.rb", "spec/fixtures/payloads/posts-index.json", "spec/fixtures/payloads/posts-show.json", "spec/spec_helper.rb"]

And, as with .each, .map, etc… you can pass a block interact with each element. Your return from each iteration will map back to the output.

# array containing the contents of all the files matching:
Dir['*/'].grep_v(%r{/packs/}).grep(/(.js$|.erb$|.rb$|.json$|.lock$)/) { |f| File.open(f).read }

I’m not sure if I’ve seen any code that would be made cleaner by grep (unless for utility scripting), and there’s always the risk of lowering maintainability of the code by using features no one else uses, but it’s nice to know that Ruby always has something more even after using it for many years.


Ruby on Jets Webpacker errors and Invalid Configuration Object

The problem

tl;dr to the solution that worked for me

I was trying to get Ruby on Jets up and running and ran into webpacker errors including “CLI for webpack must be installed” (and others) all the way to “Invalid configuration object”.

On creating a new app with jets (3.0.8) with npm 7.11.2, node 16.1.0, and yarn 1.22.5 and creating a basic blog CRUD app like follows:

 jets new blog_app --database=postgresql
 cd blog_app
 jets generate scaffold post title:string post:text
 jets db:create
 jets db:migrate
 jets server

I get an error with browsing to localhost:8888/posts:

ActionView::Template::Error at /posts
Webpacker can't find application.js in /mnt/c/Users/twill/projects/jets/blog_app/public/packs/manifest.json. Possible causes: 1. You want to set webpacker.yml value of compile to true for your environment unless you are using the `webpack -w` or the webpack-dev-server. 2. webpack has not yet re-run to reflect updates. 3. You have misconfigured Webpacker's config/webpacker.yml file. 4. Your webpack configuration is not creating a manifest. Your manifest contains: { }

On the back end, I could see:

[Webpacker] Compilation failed:
warning package.json: No license field
CLI for webpack must be installed.
webpack-cli (https://github.com/webpack/webpack-cli)
We will use "npm" to install the CLI via "npm install -D webpack-cli".
Do you want to install 'webpack-cli' (yes/no):

Adding dependencies one by one

I added webpack-cli with yarn add webpack-cli and:

[webpack-cli] Failed to load '/mnt/c/Users/twill/projects/jets/blog_app/config/webpack/development.js' config
[webpack-cli] Error: Cannot find module '@rails/webpacker'

Ok, add yarn add @rails/webpacker and run jets server and reload the page:

[Webpacker] Compilation failed:
 warning package.json: No license field
 [webpack-cli] Invalid configuration object. Webpack has been initialized using a configuration object that does not match the API schema.
 configuration.node should be one of these: false | object { __dirname?, __filename?, global? } -> Include polyfills or mocks for various node stuff. Details: configuration.node has an unknown property 'dgram'. These properties are valid:
 object { __dirname?, __filename?, global? }
 -> Options object for node compatibility features.
 configuration.node has an unknown property 'fs'. These properties are valid:
 object { __dirname?, __filename?, global? }
 -> Options object for node compatibility features.
 configuration.node has an unknown property 'net'. These properties are valid:
 object { __dirname?, __filename?, global? }
 -> Options object for node compatibility features.
 configuration.node has an unknown property 'tls'. These properties are valid:
 object { __dirname?, __filename?, global? }
 -> Options object for node compatibility features.
 configuration.node has an unknown property 'child_process'. These properties are valid:
 object { __dirname?, __filename?, global? }
 -> Options object for node compatibility features.  

That led me eventually to Webpack 5: configuration.node has an unknown property ‘dram’. There properties are valid: issue comment on GitHub, with the mention of rails webpacker and that “Rails’ webpacker 5.x.x is only compatible with webpack 4.x.x”

The Solution

After lots of teardowns and tweaks to the project to troubleshoot, I finally ended up with a sequence that produces a scaffold for posts that renders with webpack:

jets new blog_app --database=postgresql
cd blog_app
jets generate scaffold post title:string post:text

# jets db:drop # used while experimenting with yard packages below
jets db:create
jets db:migrate

# this is the key line that makes the currently installed @rails/webpacker work
yarn add webpack@4 
yarn add webpack-cli
yarn add @rails/webpacker
jets server

Automatically Moving Files Between S3 Buckets with Lambda (part 1)

I have an S3 bucket that I want to attach to an application’s upload area, but I want to move them out of the bucket accessible to the application after they’ve been uploaded. Eventually, I want to have this be after a small delay, but initially I wanted to test out the concept itself.

Step 1: Have source and destination buckets in S3

Create buckets for source and destination. The ACLs on both of the buckets are the same (non-public) in my case.

Step 2: Create a Lambda Execution Role

  • Go to IAM > Roles > Create Role
  • Choose Lambda as a Use Case
  • Next: Permissions
  • Search for S3 and check AmazonS3FullAccess
AmazonS3FullAccess selection
  • Search for “lambdabasic” and check AWSLambdaBasicExecutionRole (for CloudWatch logs)
AWSLambdaBasicExecutionRole selection
  • Click [Next: Tags] > [Next: Review] and give your role a name and verify that the S3 and Lambda policies are added:
Verify policies and name role
  • Click [Create Role]

Step 3: Prep the Lambda Code

  • Clone https://github.com/stringsn88keys/move_file_on_upload
  • Be sure to have the correct ruby version (2.7.0 at the time of writing) installed
  • Change into move_file_on_upload folder
  • bundle install locally
  • bundle config set --local deployment 'true' to vendor the gems for the AWS Lambda
  • zip ../move_file_on_upload.zip ** to package the zip

Step 4: Create the Lambda

  • Go to AWS Lambda in the AWS Console and click [Create Function]
  • Name the function, set Ruby 2.7 as the runtime, and use the role you created
Function name, Runtime, and Permissions
  • [Create function]

Step 5: Add S3 Trigger

  • Click [+ Add Trigger]
  • Search and select S3
  • Fill in your source bucket and select all object create events
  • If you get this error (“configurations overlap”), select your bucket in S3, click the Properties tab, and you’ll see an Event Notifications that’s been orphaned by a previous config (be sure to delete the dependent Lambda as well if it exists):
Configurations overlap error

Step 6: Upload your code

  • Go back to the [Code] tab for your lambda and select the [Upload from] dropdown and select zip file.
Upload .zip file
  • Go to the [Configuration] section and add a FROM_BUCKET and TO_BUCKET environment variable for your Lambda to know what to process

Step 7: Monitor and test

  • You can test the Lambda execution via the Test dropdown
  • S3 put is one of the available test templates
  • Click “Create” after selecting S3 Put and you’ll be able to watch the event get executed.
  • Go to CloudWatch -> Logs -> Log Groups and you should see a log group for /aws/lambda/your_function_goes_here
  • If all else is successful, you should see “the specified key does not exist”

Step 8: Test it live

  • Create a folder in your source bucket.
  • Go into that folder
  • Upload a file.
  • The file should pretty quickly be copies to the destination bucket in the same folder structure and removed from its original location.
  • The top level folder under the bucket should remain.