Luis Garcia

Developer

All posts in one long list

Create an Echo Show Skill with VS Code and the Serverless Framework

Jan 3, 2022 on Alexa Echo Show Skill AWS

Lets Make an Echo Show Skill with the Visual Studio Code and the Serverless Framework!

Checkout the completed project on my github!

Introduction

If you’ve been following my posts lately, you’ll probably notice a pattern. I’ve been working a lot with Alexa Skills and Serverless lately. I’m doing a lot of learning and test projects to prepare myself for a bigger project that will combine my efforts. As I learn things, I want to be able to remember what I did and how its done. Thats a big reason why I write these posts; so I can go back later and review things I’ve done. It also has the benefit of possibly helping someone that may have the same problem or questions that I have. So with that in mind, here’s how to make an Echo Show skill with the Serverless Framework.

For more information, here are some other helpful links

https://developer.amazon.com/en-US/docs/alexa/alexa-presentation-language/use-apl-with-ask-sdk.html
https://developer.amazon.com/en-US/docs/alexa/alexa-presentation-language/tutorial-add-first-visual-response-custom-skill.html

Project Setup Review

I’ll review it pretty quickly here, but for more details on how to set up the project checkout my post where I go into this in detail. Additionally, I’ll assume everything’s installed and configured so I’m just going to start creating a new project.

Create The New Skill

First, I’ll launch Visual Studio Code, click the Alexa Skills Toolkit button and click “Create a Skill” PNG

Skill Name - “Show Demo” The name of the skill. I used “Show Demo” so I could invoke the skill by saying something like: “Alexa, open Show Demo.”
Default Language - English
Skill Model - Custom
Choose a method to host your skill’s backend resources - “Provision your own” - A lot of good information on this can be found here. I chose the “Provision your own” to host it in my AWS account. I can deploy it and manage it there without any restriction from the Alexa free-tier. I’ll still have to work with the developer console, but I’ll spend much less time there. Besides, deploying this to AWS with Serverless is one of the points of the article!
Programing Language - Python
Local Directory - The folder/repository created earlier.

After waiting a bit, the code will generate and the skill will be available for you in the Alexa Skill developer console.

My ‘pre-commit’ script

I want to avoid committing my skill-id to these public repositories (although I’ve been deleting these ‘demo’ skills anyway). To do that I’ve created a very rudimentary script to check all the files in the repo for a skill-id type string and prevent committing if one is found. I used the tool pre-commit to help me do that. If you take a look at that script (alexa_skill_id_check_git_hook.py) again, you’ll see its rudimentary, has some TODO’s, etc… but it gets the job done for now. Someday I’ll work on improving it, so look forward to that in another post!

Prepare The Skill For Visuals

In the past I’ve created ‘audio only’ types of skills. Theres a few differences and configuration details that need to change in order for us to add visuals to a skill. This AWS tutorial goes into it thorougly so check it out too.

Enable the APL Interface

For visual skills we need to enable the APL interface. APL stands for “Alexa Presentation Language” its the feature that we’ll use to help us present visuals to users. You can find it in the developer console in the Build tab in the Interfaces section under Alexa Presentation Language Developer Console > Build > Interfaces > Alexa Presentation Language PNG

Create a New APL Document

We’re now going to create an APL (“Alexa Presentation Language”) document to declare how we’ll display information. For this tutorial, we’ll do this in the Alexa Developer Console, but it can be done right in code as well.

In the console, go to the Build tab, then under the “Assets” menu, click “Multimodal Responses”. Once that page opens, there are two types of responses “Audio” and “Visual”. We want “Visual”. (Build > Assets > Multimodal Responses > Visual)

Click “Create Visual Response” PNG

This will take you to a page with templates and fancy ways to display and organize your visual response. These are great and are worth taking a look at, but for our purposes we’re just going to create a blank response for now.

PNG

So click “Blank Document” (in the upper right) which will create a blank response for us. After that we’ll go ahead and save the document and name it “HelloWorldDocument”. So click the save icon in the upper right and type “HelloWorldDocument” as the template name

This will save it as a document, and really, its just JSON. You can check it out by clicking “Code View” next to “GUI View” in the editor in the upper left.

Add an APL Package

In the designer, go to the “Code View”.

PNG

Go ahead and add this object to the “import” array.

{
   "name": "alexa-layouts",
   "version": "1.5.0"
}

The final code should look like this.

{
    "type": "APL",
    "version": "1.8",
    "license": "Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.\nSPDX-License-Identifier: LicenseRef-.amazon.com.-AmznSL-1.0\nLicensed under the Amazon Software License  http://aws.amazon.com/asl/",
    "settings": {},
    "theme": "dark",
    "import": [
        {
           "name": "alexa-layouts",
           "version": "1.5.0"
        }
    ],
    "resources": [],
    "styles": {},
    "onMount": [],
    "graphics": {},
    "commands": {},
    "layouts": {},
    "mainTemplate": {
        "parameters": [
            "payload"
        ],
        "items": []
    }
}

We just imported an APL package into our document. This package contains pre-defined resources that you can use to help display your information. In this case, we’re importing the alexa-layouts package.

Now lets go ahead and use them.

Add a Headline

Lets now add this object to our JSON, this time add it to the mainTemplate.items array.

{
    "type": "AlexaHeadline",
    "primaryText": "Display your text here."
}

Your code should look like this now.

{
    "type": "APL",
    "version": "1.8",
    "license": "Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.\nSPDX-License-Identifier: LicenseRef-.amazon.com.-AmznSL-1.0\nLicensed under the Amazon Software License  http://aws.amazon.com/asl/",
    "settings": {},
    "theme": "dark",
    "import": [
        {
           "name": "alexa-layouts",
           "version": "1.5.0"
        }
    ],
    "resources": [],
    "styles": {},
    "onMount": [],
    "graphics": {},
    "commands": {},
    "layouts": {},
    "mainTemplate": {
        "parameters": [
            "payload"
        ],
        "items": [
            {
                "type": "AlexaHeadline",
                "primaryText": "Display your text here."
            }
        ]
    }
}

And you screen should update to look like this image PNG

Don’t forget to save by clicking the save icon in the top right!

Hook the APL to a data source

Now that we’ve got something to show, lets hook it to a data source so we can display different things to the user other than “Display your text here.”

Lets start by clicking the “DATA” icon in the leftmost column of controls. PNG

You should see the empty code block {} here. Lets go ahead and replace that with this populated code block. Note you won’t see anything change in this step

{
    "helloWorldDataSource": {
        "title": "Hello World!",
        "subtitle": "I hope you're having fun!",
        "color": "@colorTeal800"
    }
}

This adds the data source for our design tool to use. Now we need to reference this and bind it to our APL document we created.

Since we’re done adding the data source and we’re modifying our APL document again, click the APL button in the leftmost column. This takes us back to the design space we were working with earlier. PNG

We’ll update our APL code again here. This time we’ll add to the parameters array. Add the string “helloWorldDataSource” to the parameters list in the mainTemplate object. It should look like this.

"parameters": [
    "helloWorldDataSource"
],

We’ll also bind the data in our “helloWorldDataSource” to the displayed text. This is done by updating the primaryText property to the following value ${helloWorldDataSource.primaryText}

It should look like this

"primaryText": "${helloWorldDataSource.primaryText}"

When this is changed, you should see the text change to “Hello World!”. This is exactly what we want. Now we have a data source that controls the text on screen!

PNG

While we’re here, lets add some of our other properties too. This is the final code I have for my APL document.

{
    "type": "APL",
    "version": "1.8",
    "license": "Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.\nSPDX-License-Identifier: LicenseRef-.amazon.com.-AmznSL-1.0\nLicensed under the Amazon Software License  http://aws.amazon.com/asl/",
    "settings": {},
    "theme": "dark",
    "import": [
        {
           "name": "alexa-layouts",
           "version": "1.5.0"
        }
    ],
    "resources": [],
    "styles": {},
    "onMount": [],
    "graphics": {},
    "commands": {},
    "layouts": {},
    "mainTemplate": {
        "parameters": [
            "helloWorldDataSource"
        ],
        "items": [
            {
                "type": "AlexaHeadline",
                "primaryText": "${helloWorldDataSource.title}",
                "secondaryText": "${helloWorldDataSource.subtitle}",
                "backgroundColor": "${helloWorldDataSource.color}"
            }
        ]
    }
}

Save and Rebuild!

Looking good! Lets make sure to save our work, click the save icon in the upper right. We also have to rebuild our model before we start making any code changes. Click the back button in the upper left. That should take you back to the developer console. You can click the build tab, then the build entry on the left. Alternatively, you can click “Interfaces” then “Build Model” at the top of the page.

Write the Code to Interact with the Visuals

Now we need to add this APL document to our lambda function in code. We need to respond with the APL document when skill is invoked. If you remember that should be in the skill’s ‘lambda’ folder. There should be a ‘hello_world.py’ for us to open up.

In the file, I’m just going to handle this new APL in the launch of the skill, for more complex interactions, checkout some of the other methods.

I’m going to modify the LaunchRequestHandler class’s method “handle”. I’m going to add this code right at the beginning of the function, before any responses are returned. When you’re writing other functions, be sure to think carefully about where you may want to handle any APL interactions.

Here’s my code for adding the APL to the response

supported_interfaces = ask_utils.request_util.get_supported_interfaces(handler_input)
if supported_interfaces.alexa_presentation_apl is not None:
    logging.info('This device supports APL')

    document_name = "HelloWorldDocument" # The name of the APL we saved
    token = document_name + "Token"
    handler_input.response_builder.add_directive(
        RenderDocumentDirective(
            token=token,
            document={
                "src":'doc://alexa/apl/documents/' + document_name,
                "type": "Link"
            },
            datasources={
                "helloWorldDataSource":{
                    "title": "We did it!",
                    "subtitle": "Hello World is coming from code!",
                    "color": "@colorTeal800"
                }
            }
        )
    )

Quick Explaination: I’m using the sdk to figure out what kind of device the user’s device is. If its a device that supports a Presentation APL then we can proceed. I’ll add the “RenderDocumentDirective” to render the APL document that we created. Right now the APL document is stored in the developer console. If you remember we stored it named “HelloWorldDocument”, so we need the name. The developer console will store this saved document at that reference doc://alexa/apl/documents/. A little later, I’ll show you what it looks like if you were to use an APL committed with your code. This Directive is added to the response and is returned to the user. If we’ve done everything right we can save, build, and deploy our skill!

NOTE: Each time you update your APL documents in the developer console, you have to rebuild your model to make sure your changes are available.

Serverless Comes In

So far, this has just been a tutorial on working with APL’s. I want to make sure I’m not forgetting Serverless since its our deployment mechanism. Let’s start working with it. See my first Serverless Alexa Skill post for more details.

First lets initialize serverless by running the command serverless create --template aws-python in the lambda directory.
Rework the serverless.yml as follows

service: DemoEchoShowSkill
frameworkVersion: '2'

plugins:
  - serverless-python-requirements
provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: 20201221
custom:
  pythonRequirements:
    dockerizePip: non-linux

functions:
  hello:
    handler: hello_world.handler
    events: 
      - alexaSkill: amzn1.ask.skill.xx-xx-xx-xx

(Make sure you change your alexaSkill parameter so it matches whatever your skills skill Id is)

Install the plugin serverless-python-requirements the ASK SDK is not in the standard libraries so you have to add the lib in your requirements.txt file and package it up when you deploy
Deploy the skill with the command serverless deploy (Make sure you’ve configured your aws cli and profile)
Once deployed, in the AWS developer console, go to the Build tab, then in the Endpoint section select “Aws Lambda ARN” and copy/paste the ARN of your deployed lambda function into the Default Region box and clid Save Endpoints, then re build the skill.

Test It Out!

Now you can go to the Alexa Developer Console, go to the Test tab, make sure “Device Display” is checked and invoke your skill. I did this by typing ‘hello world’ in the Alexa Simulator textbox. (If you dont know your invocation name, you can go to Build -> Invocations -> Skill Invocation Name). Once your skill has been invoked you should be able to scroll down in the simulator and see the visuals! PNG

Create an Alexa Skill with a Dynamo Database

Dec 9, 2021 on Alexa Echo Skill AWS Amazon

Lets Make an Alexa Skill with a Dynamo Database

Checkout the completed project on my github!

Introduction

In my post about Creating an Alexa Skill with VSCode and the Serverless Framwork I made a simple hello world Alexa Skill so that anyone can get up and running creating and deploying Alexa Skills easily. Now that we’ve seen how easy it can be to develop and deploy, lets start to introduce some complexity!

A large part of many applications is a data layer to store information. I want to be able to have a skill that can store and retrieve custom user data. With that in mind I’ll need a database to store the data and I’ll need to implement code to interact with it.

This project builds upon my previous work. If you haven’t checked it out, now’s a good time to look. With that in mind, I’ll continue by adding a DynamoDB table to my serverless.yml file. This will allow a new DynamoDB table to be created whenever I run the serverless deploy command. Infrastructure as code makes deploying things and identifying and maintaining aws resources a lot easier.

Problem

In expanding our Hello World application, I’ll add a piece to track how many times the application has said hello to the Alexa user.

Designing the Dynamo DB Table

I’m using AWS’s Dynamo DB for a few reasons. Its AWS native resource thats easy to work with. The boto3 python library makes it easy to work with within python and AWS Lambda functions. Its easy to add to a Cloud Formation template or my serverless.yml file. Lastly, I can always use more practice with it.

I mention that I could use more practice with it, because I’m more accustomed to relational databases (think SQL) where data is usually stored across multiple tables in seperate columns per table. Relational databases often use primary keys and foreign keys to join data together across tables to organize, manage, and construct objects to use in code.

DynamoDB on the other hand, is non-relational, sometimes called NoSQL. It doesn’t user joins or other concepts from relational databases. With that in mind, our table design will be very different and so will our access patterns. I’ll do my best to explain my design decisions here, but to learn more about NoSQL database design with Dynamo, you can checkout this AWS post or this post

Columns

Partition Key As I mentioned earlier, there are many differences between relational and non-relational databases. Relational databases have a Primary key to query and help organize the data among other things. For our non-relational Dynamo database though, there is no primary key. We’ll instead use a Partition Key. If you’re familiar with hash tables, sometimes called hash maps or dictionaries, think of this as the ‘key’, they’re very familiar concepts. A partition key is one of the most characteristic parts of a NoSQL database. Our usage patterns will, in a lot of ways, mimic the patterns of a hash map or dictionary. So with that in mind, a partition key will be our first column in our database.

Another thing the differs between relational and non-relational databases is the number of tables. For relational databases, the data is stored across many tables and joined together. In non-relational, its much more appropriate to store and organize all the data in the same table. You can query different objects stored with different styles of keys using different sorting options. That is, we’ll store our data as a JSON string. That JSON string can be a different object across rows of data. Its important to know what that object is and how to deserialize it, so establishing the keys and key data is important.

Let’s consider our access pattern now. Our problem states that we want to know how many times our application has said “hello” to our user. Lets make the partition key relate to the user so that we can query the number of times we’ve said hello by a user’s id. In other words, we’ll ask our database this question: “For user id {x}, what is the number of times we’ve said hello?”. Another way to represent this is with some psuedo-code (Note this code wont be used in our application, but it demonstrates our access pattern)

userId = "x"
numberOfHellos = myDatabase[userId]

For our Alexa ppplication, we’ll use our user’s userId as part of the partition key. According to the Alexa documentation userId is a string so we’ll make the datatype of our Partition Key a string as well. We also want other data in our partition key, so a string is a good way to satisfy both those requirements.

So lets make our first column in our table look like this:

pk
user#abc-123-456
user#def-567-890
user#ghi-456-567

Sort Key

If you expected more than one row to be returned when you queried an individual pk (in our case the userId) then it can be helpful to sort that data so that you can get the exact row you’re after. For the purposes of our problem, this isn’t necessary since we’ll only have one row per user. However, for more complex applications, one could use a Sort Key to enhance and optimize the queries you run to get your data. Checkout those posts I linked earlier for more information.

Data Column

For our data column, I’m just going to store our object as serialized json. Something like this:

{
    "numberOfHellos": 0
}

I’ll just name the column data sticking with the idea that any object can be stored in a non-relational database. That being said, its important to consider how you will know what data is stored in this column. That may be a reason to add an additional column like a sort key, a global index, or just another column.

Our final table will end up looking something like this: | pk | data | | — | — | | user#abc-123-456 | {“numberOfHellos”:0} | | user#def-567-890 | {“numberOfHellos”:5} | | user#ghi-456-567 | {“numberOfHellos”:2} |

Adding to Serverless

Now that our table is designed, lets add it to our serverless.yml file so it can be deployed

resources:
  Resources:
    helloLambdaIamRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: ${opt:stage, self:provider.stage}-helloLambdaRole
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - lambda.amazonaws.com
              Action: sts:AssumeRole
        Policies:
          - PolicyName: ${opt:stage, self:provider.stage}-LambdaDynamoAndLogPolicy
            PolicyDocument:
              Version: '2012-10-17'
              Statement:
                - Effect: Allow
                  Action:
                    - logs:*
                  Resource: "*"
                - Effect: Allow
                  Action:
                    - dynamodb:PutItem
                    - dynamodb:Query
                  Resource: 
                    Fn::GetAtt: [helloWorldDataTable, Arn]
    helloWorldDataTable:
      Type: AWS::DynamoDB::Table
      Properties:
        TableName: ${self:custom.tableName}
        BillingMode: PAY_PER_REQUEST
        AttributeDefinitions:
          -
            AttributeName: "pk"
            AttributeType: "S"
        KeySchema:
          - 
            AttributeName: "pk"
            KeyType: "HASH"

We add the table in a Resources block in the resources section. Yes its a little redundant and silly but you can take that up with Serverless

Our resource is named helloWorldDataTable and if I need to reference this resource anywhere else in my serverless.yml file then I’ll use this as the reference name

From here you can use the AWS Cloud Formation templeate reference for details on the other parameters, but I’ll go over them briefly too.

Type - for DynamoDB tables this is always AWS::DynamoDB::Table
Properties - The specific DynamoDB properties for the resource
- TableName - The name of the table. You can hardcode this, but I named mine elsewhere in the serverless.yml. You can checkout the whole source on github (NOTE this has different meaning than the reference name of the resource we specified earlier)
- BillingMode - This is how AWS bills you for interacting with the resource. I chose PAY_PER_REQUEST in case my skill’s usage pattern varies wildly. Checkout the earlier reference for more information
- AttributeDefinitions - This (in combination with KeySchema) is where our key colums are defined. Note, you don’t have to define all columns here, just the ones that will be used as keys in the KeySchema definitions later
  - AttributeName - the column name
  - AttributeType - the datatype of the column “S” for string, “N” for number, “B” for binary
- KeySchema - This defines the keys for the table
  - AttributeName - the name of the column, this must be equal to one of the AttributeNames defined in the AttributeDefinitions
  - KeyType - the role of the key. “HASH” for partition keys “RANGE” for sort keys

You can also see the IAM role that I defined too. In order to interact with the database, we need to give the role our lambda uses permissions to read and write to the database. So in the resources section I defined a role that is able to read/write to the database, write logs, and interact with lambda. I also made sure the lambda used the role by adding this line to the function definition role: helloLambdaIamRole

Now that our table and security is defined, we can move on to writing the code to use it!

Interacting with our Database

Lets reflect on our problem before we write any code.

Add a piece to track how many times the application has said hello to the Alexa user.

So lets have our application do the following

A user invokes the application
From the database, we fetch the number of times this user has invoked the application
- If the user row doesnt exist in the database for this user, create a row and set the number of times this application has been invoked value to 1
- If the user row does exist, fetch the data, add 1 to it and persist the data
Reply with a message that says “Hello World, I have said hello {x} times” to the user where {x} is the number fetched from the database + 1 (or the number persisted earlier)

Example 1:

Our user invokes this application for the first time.
We attempt to lookup user data in the database but find zero rows
A row is persisted in the database with the userid and the number of times invoked set to 1
Alexa replies saying “Hello World, I have said hello one time”

Example 2:

Our user invokes this application for the third time.
We attempt to lookup user data in the database and find a row with the number of times invoked = 2
the row is updated so that the number of times invoked = 3
Alexa replies saying “Hello World, I have said hello three times”

Now lets write the code!

I’m going to write three methods:

get_number_of_hellos(userId: str) -> int
set_number_of_hellos(userId: str, number_of_hellos: int) -> None
create_hellos_message(number_of_hellos: int) -> str

get_number_of_hellos() will query the database and return the number of hellos found for the user set_number_of_hellos() will add or update the database with a number of hellos create_hellos_message() will create the message to return back to the user

create_hellos_message() is an easy one lets write it below in python

def create_hellos_message(number_of_hellos: int) -> str:
    # NOTE: english is funny, we have to say 'time' or 'times' so it sounds correct depending on whether there is one or more number of hellos
    if number_of_hellos == 1:
        return f"Hello World, I have said hello {number_of_hellos} time"
    else:
        return f"Hello World, I have said hello {number_of_hellos} times"

Querying the Database

Now that we’ve got our easiest method out of the way, lets interact with our database in code for the first time. For this lets import boto3 and make sure boto3 is in our requirements.txt file.

While we’re up in the top of our python file lets also import os and from boto3.dynamodb.conditions import Key, Attr and add some global variables

import boto3
from boto3.dynamodb.conditions import Key, Attr
import os
import json

# Globals that should stay 'warm' from lambda to lambda
TABLE_NAME = os.environ.get('HELLO_WORLD_DATA_TABLE')
REGION = os.environ.get('REGION')
DYNAMODB_RESOURCE = boto3.resource('dynamodb') 
DYNAMODB_TABLE = DYNAMODB_RESOURCE.Table(TABLE_NAME)

Normally, global variables should be something to frown on. However in this case, I want to keep these resource references ‘warm’. That is, if the lambda is invoked in rapid succession, less time will be spent initializing these resources since they’ve been previously initalized, or so I’ve been told. A cursory google neither proves or denies this though.

Of these variables, we’ll just use DYNAMO_TABLE to query our table for data. Let’s start on our get_number_of_hellos method

def get_number_of_hellos(self, userId: str) -> int:
    user_value = f'user#{userId}'
    query_response = DYNAMODB_TABLE.query(
        KeyConditionExpression=Key('pk').eq(user_value)
    )

    query_response_items = query_response['Items']
    if len(query_response_items) < 1:
        return 0
    else:
        # there 'should' only be one item returned, if there is > 1 then we'll just pick the first one
        data_str = query_response_items[0]['data']
        data_obj = json.loads(data_str)
        return data_obj['numberOfHellos']

One item that tripped me up was the difference between the Dynamo resource and the Dynamo client. The query syntax is different between the two.

For more information on querying the database see this AWS article.

Adding to the database

Alright, last method. It should be pretty easy to copy and past our old code with a few changes.

def set_number_of_hellos(self, userId: str, number_of_hellos: int) -> None:
    user_value = f'user#{userId}'
    data_obj = {}
    data_obj['numberOfHellos'] = number_of_hellos
    db_item = {
        'pk': user_value,
        'data': json.dumps(data_obj)
    }

    DYNAMODB_TABLE.put_item(
        Item = db_item
    )

The important part here is to get the item-to-persist correct. Make sure to name and add the columns correctly. Other than that its the same thing!

Putting it all together

Lastly, reflecting on the problem statement from earlier, I reworked the ‘handle’ method in the ‘HelloWorldIntentHandler’ so that we could read, write, and get a good message from Alexa. Since we wrote good methods, the code is very easy.

def handle(self, handler_input):
    # type: (HandlerInput) -> Response
    speak_output = "Hello World!"

    userId = handler_input.request_envelope.session.user.user_id
    number_of_hellos = self.get_number_of_hellos(userId)
    number_of_hellos = number_of_hellos + 1
    self.set_number_of_hellos(userId, number_of_hellos)
    speak_output = self.create_hellos_message(number_of_hellos)

    return (
        handler_input.response_builder
            .speak(speak_output)
            # .ask("add a reprompt if you want to keep the session open for the user to respond")
            .response
    )

Deploying and Testing

Finally, we can deploy and test. I used this line in my powershell (or terminal if you’re using linux) serverless deploy --region us-east-1 --stage dev

Make sure you have your aws profiles configured and it should deploy nicely for you in AWS. If you have problems, go back to my previous post. It should help out.

Like last time you can go into the Alexa Developer Console and test your skill. With any luck you’ll be able to verify the examples from earlier

Checkout the code

You can check-out all the code on my github page

Create an Alexa Skill with VS Code and the Serverless Framework

Apr 19, 2021 on Alexa Echo Skill AWS Amazon

Lets Make an Alexa Skill with the Visual Studio Code and the Serverless Framework!

Checkout the completed project on my github!

I’ve been toying around with more tools, frameworks, and other things I find interesting these days. I’m interested in creating another Alexa Skill, but I’ve never had a good time managing the infrastructure and deployment. I’ve only used the free developer console and its tooling. Its been kind of a pain to provision resources, develop, and prototype rapidly. I’m fully willing to accept that I may be doing something wrong, but I haven’t yet found what.

I’ve been playing with the Serverless Framework lately and have found it pleasant to work with for describing AWS Infrastructure as Code. Serverless will create and deploy an AWS Cloud Formation stack for you. This tool seemed to me like a good way to solve my past problems when developing Alexa Skills. Hopefully it will help solve some of your problems too or at least provide a good guild to this specific case.

Before You Start

Before you continue to read, I wrote this article after I developed a couple Alexa skills. I will skip over a lot of the details of creating your first skill. Instead I’m focusing on the specific case of creating a new skill with Visual Studio Code and deploying the AWS infrastructure with the Serverless Framework. If you’re developing your first ever Alexa skill, I encourage you to keep reading, but a better guide might be the one from Amazon themselves

Visual Studio Code

I’ve worked with the Microsoft Stack pretty extensively and I’m pretty accustomed to Visual Studio. Lately, a lot of my work has been directing me more towards Visual Studio Code though. I enjoy the tool for its relative familiarity, but also how it can integrate and work with a more diverse set of programming languages. It also just so happens supported by Amazon for developing skills via the Alexa Skills Toolkit extension for Visual Studio

There are some pretty good guides out there for working with the toolkit that I used when writing and developing this. Feel free to check them out while you’re here.

Set Up the Environment

I started by getting everything set up.

First, I downloaded and installed Visual Studio Code
I chose to use Python 3.8 as my language of choice for developing this skill, so I made sure to add that extension. Python isn’t required for this though. Alexa skills are supported for a number of programming languages.
I installed the Alexa Skills Toolkit extension.
You’ll need NPM to install Serverless. Once that’s installed, follow the Serverless Install Instructions to get serverless installed. The AWS CLI tool is required too in case you didn’t already have it.
I already had both an AWS account and an Amazon Developer account, so I didn’t need to create those, but if you don’t have them you’ll have to create those accounts. Make sure you’re able to use the AWS cli tool too!
After getting everything installed, I logged in and followed the Getting Started guide. If you’ve never developed a skill before, or have never used the extension, its an important guide to follow.
I created a directory and got it setup with git, check it out here!

Creating the Skill

After launching Visual Studio Code and opening the Alexa Skills Toolkit extension, I clicked the ‘Create new skill’ button. That button opens a new tab for creating a new skill. This tab contains all the parameters that must be filled out to create a new skill. Most of the them are fairly obvious, but I’ll describe what I put in the parameters and the decisions that drove those inputs. PNG

Skill Name - This is the name of your skill. I used ‘Demo’ so I could invoke the skill by saying something like: “Alexa, open Demo.” I can always change this later before publishing if I wanted to publish this or expand the project further.
Default Language - I chose ‘English’ because thats what I speak!
Skill Model - Custom, its the only option right now.
Choose a method to host your skill’s backend resources - A lot of good information on this can be found here. For my purposes, I chose the “Provision your own” option so that I could host it in my AWS account. I can deploy it and manage it there without any restriction from the Alexa free-tier. As much as I want to just have the Alexa developer space host it, I found it a bit restricting. That being said, I’ll still have to work with the developer console, but I’ll spend much less time there. Besides, I’m no stranger to AWS. and deploying this to AWS with Serverless is the point of the article!
Programing Language - Python, as I mentioned above.
Local Directory - I chose the folder/repository I created earlier.

After I filled out all the parameters, I clicked the final ‘Create’ button and waited for the extension to build out some skeleton code for me.

Building and Deploying The Skill

Since I chose to host my AWS resources outside of the Alexa developer space, and since I’m using serverless, I will generally have two deployment steps.

Deploy Skill configuration to Alexa developer account
Deploy AWS resources to AWS account

The steps can technically be done in any order, but its important to make sure they are coordinated well so that any change made to one will affect the other appropriately. That is, if I add some new interaction in the skill configuration, there should be corresponding code to back it up. With that in mind, its probably best to deploy the skill configuration first then the AWS resources. All that being said, I may change my mind on this as time progresses. Any developer in any project should be aware of their deployment order (if they have one) and dependencies.

First Skill Deployment

When I first clicked the ‘Create’ button from the Visual Studio Extension, it automatically created and deployed the default ‘hello-world’ configuration to the developer space. Since this is just a demo, I’m not going to change the default hello-world skill too much. So for now, I’m just going to log-in to the skill console, open up the skill and note the Skill Id. The skill you created is found in the main page of the Alexa developer console. You can click the name of your skill to get into the details of it. Once the skill is open you can go to the ‘Build’ tab, then click ‘Endpoint’. We’re going to use a Lambda for our skill endpoint, so I’ll click the AWS Lambda ARN radio button and then note the Skill ID that appears to the right. Be sure to note the whole string there, it should be unique to your skill. PNG

First AWS Resource Deployment

The next thing on my list was to deploy the AWS resources to make the skill work. Right now, all I’ve got is the hello_world.py file that came by in the lambda directory by default. It should be easy enough to stand it up in AWS and make sure everything works as intended.

If I were doing this manually, I’d zip up the file and its dependencies and then upload it to the lambda section of the AWS console. As I’ve mentioned, I found that to be rather tedious, so I’m choosing to use the Serverless Framework to define, package, and deploy all my AWS resources. This should make things easy to rapidly prototype, test, and deploy.

First I’ll initialize my serverless project by cd‘ing into the lambda directory that was created and I’ll run serverless create --template aws-python. This will create a nice little serverless template file (serverless.yml) for me to use. You can always create this file manually if you know what you’re doing. The command will also create a default function handler.py. I’ll go ahead and remove that file since I don’t need it, but I will tweak my serverless.yml file accordingly.

The serverless.yml file we created has a lot of information in it that is useful if you’re starting out with serverless. Check it all out if you have time, but I only care about a few parameters. First, I’ll change the value in the service: to DemoAlexaSkill since thats a good memorable name for the Cloud Formation Stack that will be created. Under provider: and under runtime: I made sure I specified python3.8. I also like to define the default stage and region too, so I’ll uncomment those lines.

The top of my file looks like this:

service: DemoAlexaSkill
frameworkVersion: '2'

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: 20201221
  stage: dev
  region: us-east-1

Next, we can skip ahead to defining our lambda function.

Lambda functions are defined in the functions: block. By default, Serverless created an example ‘hello’ function with the handler.py file we deleted earlier. We can see how Serverless defined the function in this block of code in the serverless.yml file:

functions:
  hello:
    handler: handler.hello

We only need to change a little bit for now to get our skill working, but I strongly encourage you to read more about Serverless so that you can get a better idea of what it can do.

First, we want to define the lambda’s entry point. A default function was defined in the handler.py file when we created the serverless.yml file. We’re using different lambda code I’ll replace it with the hello_world.py that was created with our skill. That file contains lambda function code so we can just swap some parameters to that the snippet so that our file now looks like this:

functions:
  hello:
    handler: hello_world.handler

hello_world is the name of the file containing the lambda code to execute, as in hello_world.py. The .handler refers to the entry point within that file. In the old, default handler.py that serverless created, the method’s name was hello. In our code, its handler.

Now that the code is ‘done’, we’ll enable it to be used by our skill created in the developer console. With serverless we can just reference it in the serverless file. Its pretty easy as you can see with the updated block.

functions:
  hello:
    handler: hello_world.handler
    events: 
      - alexaSkill: amzn1.ask.skill.xx-xx-xx-xx

This is a good quick reference if you need more info..

The new parameter alexaSkill: should be populated with your Skill Id that I mentioned to note earlier. Do not use amzn1.ask.skill.xx-xx-xx-xx. Use your own skill’s id from the developer console.

Once that’s done we’re ready to deploy! Make sure you have the AWS CLI configured correctly (you need an IAM user that can deploy resources to your AWS account). Also make sure that serverless has the proper AWS configuration as well. If all is configured correctly, you can run this command to see your lambda deployed:

serverless deploy

Look for the Lambda in AWS

We defined the serverless stage as ‘dev’ and we know the service was named ‘DemoAlexaSkill’ in the serverless.yml we created earlier. After running the deploy command, our code was deployed in our region as a Cloud Formation Stack with a name like DemoAlexaSkill-dev. ‘DemoAlexaSkill’ comes from what was defined as the service name and ‘dev’ comes from the serverless stage. To find the lambda function you can click into and explore the ‘Resources’ tab in the Cloud Formation Stack. Alternatively, since we know the name in our serverless.yml file, we can look for a lambda created with the name ‘DemoAlexaSkill-dev-hello’. ‘DemoAlexaSkill’ comes from what was defined as the service name. ‘dev’ comes from the stage and ‘hello’ is from the name we gave to the lambda. PNG

After getting into the lambda function in the console, you can see that it is configured to be invoked by Alexa Skills Kit and it should have the permissions automatically created for it too; the wonders of serverless! Make sure to copy the new lambda function’s ARN. We’ll need it for the next step.

Define Skill Configuration and Test!

To finish things up, we can go back to the Alexa Developer console. Just like before, we need to find the skill that we created before (mine’s named ‘Demo’). After getting into the skill, we need to configure its build. So click the ‘Build’ tab at the top and then the ‘Endpoint’ item from the menu on the left. From there, just like when we copied the Skill Id before, click the AWS Lambda ARN radio button. This time we’ll paste in the ARN of the lambda function we created earlier into the ‘Default Region’ box. This will define our lambda as the endpoint to call when the Alexa skill is invoked. After the value is in the box, you can click ‘Save Endpoints’ at the top of the page and we’re ready to test. PNG

Testing from the Developer console is pretty easy, you can just go to the ‘Test’ tab at the top of the Skill page. Once you’ve gotten to the page, most things will be greyed out. In order to start testing, the tests must be enabled for development by selecting ‘Development’ from the drop down at the top of the page. From there you can toy around with enabling your mic, and treating the test page just like an Alexa device. You can test the skill we created by clicking the mic box and saying ‘hello world’. From an efficiency point of view, I prefer to type ‘hello world’ in the box. Regardless of your choice, Alexa should echo back to you the message “Welcome, you can say Hello or Help. Which would you like to try?”. This is the behavior that was implemented in the hello_world.py file by default. Now you’re all set up to start playing with Alexa!

Cleanup

In order to cleanup the resources we deployed to the AWS account we can remove them with serverless. Its as easy as running the following command in the directory where the serverless.yml exists.

serverless remove

That’ll cleanup all the things we deployed to AWS. If you want to remove the skill from the Alexa Developer console, that can be done too. You can go to the main page in the Alexa Developer Console where all your skills are listed. There should be an ‘Actions’ column with a dropdown menu on the skill that we created. If you choose the ‘Delete’ action, the skill will be deleted.

Final Thoughts

I wanted to have an easier time creating and deploying skills. Serverless, seems to me, to solve this problem pretty well. Its straight forward and I can keep track of everything that I’ve deployed. Its easy to define all the infrastructure as code in the serverless.yml file. Linking the AWS and Alexa Skill is straightforward too. I had a great time learning this and I hope you did too!

If you want to reference the code, check it out on my github page

Wyoming Coronavirus Data by County

Aug 26, 2020 on Coronavirus COVID AWS matplotlib

Updated daily at 12:00 PM Eastern

Check out the details about how this was made

About This Project

What’s shown here?

These graphs show the number of COVID-19 (Coronavirus) cases that have been reported in each of Wyoming’s counties. The number of cases is reported with respect to the reporting date. With those two factors, one can see how the number of COVID-19 cases is changing over time. Here are some examples and explanations of what someone could see in these graphs.

Horizontal Line

A horizontal line would suggest that the number of cases has not changed at all over time. This would mean that COVID-19 has been eliminated or has not been reported from the community.

Straight Diagonal Line

A straight diagonal line from the lower left hand corner to the upper right hand corner means that the number of COVID-19 cases is increasing linearly. That is, the rate of new cases is not increasing, but the rate of new cases is not decreasing. COVID-19 is still spreading, but its rate of spread is not changing. A line like this is not good news or bad news, more people are getting sick which is bad, but the rate of spread is not increasing which is good. Note: You may notice that all the graphs have a period with a VERY straight diagonal line. See the February Update section for why this occurred.

Exponential Curve

A curved line starting small from the left corner and increasing more and more is a bad sign. This is an exponential curve that means that the rate of new cases is increasing over time. This means that COVID-19 is spreading more and more from day to day. This is the worst sign since it indicates that the disease is continuing to spread at a more rapid pace.

Flattening Curve

A curved line that increases quickly but levels out is a good sign. This means that a lot of cases were reported initially but the number of new cases has slowed. This means that COVID-19 is not spreading as much. This is what we want. If it levels out completely, like the horizontal line it means that COVID-19 has been eliminated or is not being reported in the community.

Why Did I Make This?

I made this because it wasn’t shown on the Wyoming Department of Health website, at least not at the time of writing this. The data was there, I saw it change county to county every day. You could also see these statistics from the whole state. However, I wanted to see the graphs on a per-county basis. Since this wasn’t on the website I had to make it myself. Additionally, I wanted to learn more about AWS functionality and offerings. This would be a perfect project to dip my toes in with some other pieces of technology.

How This Was Made

AWS

Like I mentioned, I wanted to play around with more AWS pieces. I found a nice tutorial that gave me a point to start with. You can find that tutorial here. It seemed to mimic some of my needs well. I didn’t really need a front end for this project but I did need some of the other pieces from the tutorial.

The Plan

I knew where to source my data from: the Wyoming Department of Health Website. The problem is, they didn’t provide all the data at once. If I wanted a historical perspective of the county data, I would have to store it my self. Storing the data means that I’ll need a database and to retrieve the data I’ll have to query their website. I can generate my graphs myself with the Python library ‘Matplotlib’. To deliver the graphs I decided initially it would be easy to just shoot out an email.

Starting with the Lambda

I’m still relatively new at implementing AWS tools, so I wanted to make sure that I could get as much working as I could locally before deploying and integrating with other cloud services. So I implemented my first features on my personal computer. I ‘queried’ the Wyoming Department of Health website for data and parsed out the relevant information. I skipped the persistence of that data, since I was developing locally. However, I did use ‘Matplotlib’ to generate some graphs with the single point of data.

Deploying it into the cloud was a little more troublesome. AWS didn’t natively support all of the imported libraries I was using, so I had to create a .zip deployment package to upload to AWS. For this I highly suggest installing Docker. I tried to create my own deployment package, but generating the package from the libraries on Windows was not compatible with the AWS linux environment that the lambda’s run on. Docker simplifies the whole process by spinning up the exact image AWS uses and generating the package there. A quick google will show that this is the command to generate the package with Docker.

docker run -v path_to_project:/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"

Here’s a quick rundown. ‘docker run” Run a command in a new docker container. “-v path_to_project:/var/task” Mount a volume to the container where path_to_project is the path to your project on your local machine and /var/task is the path mapped in the container. ‘/bin/sh -c “pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit”’ Run the shell command pip install and put the output in python/lib/python3.8/site-packages. This also requires that your libraries are listed in a requirements.txt file. For more information on that see documentation here. Running this whole command will dump your package in path_to_project/python/lib/python3.8/site-packages/. From there, add your python file, zip it up, and upload to AWS lambda.

Working with DynamoDB

Since I’d never worked with DynamoDB I wanted to adhere to the tutorial apply its wisdom and deviate only where necessary. With that in mind, I stuck too it. Creating the table was easy. The tutorial showed exactly how to set permissions properly and inserting the data was straightforward. I modified my lambda to insert the data from the Wyoming Department of Health website into the Dynamo database and everything seemed fine. So I let the project sit for a while and collect data while I took a break. If you need a break too, check out what I was doing in the mean time.

Lessons Learned

I came back to the project after about a month to finish it off. There were a couple things still on my list like querying the database for the historical data and sending out the graphs. I started at the top, querying the data. Thats where I ran into my problem. Most of my knowledge and training has come from using relational databases and I didn’t see anything in the tutorial that contradicted that viewpoint. When I created the database I followed my previous knowledge and made the primary key a unique uuid. This is where I encountered my problem. When querying for my data using dynamo db, in order to use the boto3 query command you must know the partition key (a part of the primary key). The way I designed my table, that was the least important piece of my data. Since I didn’t read any documentation beyond the tutorial what I failed to understand was that the database is split and sorted by a combination of the partition key and sort key. Those two keys combined create the primary key. Using those two keys you can index and sort your rows so that lookups are quicker. The primary key still has to be unique per row, but if you carefully choose what data you put in those keys, you can optimize your querying. Since I didn’t choose my keys carefully I could not optimize my data querying. Instead of using the efficient boto3 ‘query’ command, I had to use the ‘scan’ command, much less efficient. The good news is that this was a good learning experience. The bad news is that this project will have some inefficiencies in it until I pick a new database schema and migrate the data. For more information on better DynamoDB patterns see some of these pages: Ten Examples of Getting Data from Dynamo DB… Choosing the Right DynamoDB Partition Key

Going Forward

I reckoned with the fact that my data would need to be migrated in the future. Since this was a test project and the dataset will be small (I’m hoping) it shouldn’t be too bad. The longer the COVID-19 pandemic continues the more expensive my queries will be, so for my database’s and everyone’s sake, lets hope we can bring it to a halt.

Data Delivery

E-mail

So how will someone view this data? In other words, how will I deliver the graphs to a place where they can be viewed? My initial reaction was to send an email. Its quick, easy, and I can attach the graphs without issue. So I started down that path. I implemented the code to my lambda function and started wiring up the infrastructure. I wanted to use one of gmail’s features to use as my email server. I quickly ran into some drawbacks though. I had to provide my email and a plaintext password in the script in order to use gmail. Second, it wouldn’t even work with lambda. I don’t know the specific technical reason it wouldn’t work but its related to signing in to your account. It looked like that I had to sign in via a browser, get a cookie, and then when my script would email, google would look for that cookie or sign in token. When I deployed to AWS there was no way to ‘scriptify’ this login procedure. So I had to abandon that for another idea. The next idea was to utilize AWS’s email utility Simple Email Service (SES). It was simple enough to setup and implement but it didn’t scale for my purposes. That is, I wanted to send out this email to family members in Wyoming. With SES, I had to apply to be able to send an email to any address. Alternatively, I could send out a subscribe email first, and if the recipient accepts, then I could send the email. This is probably good in the scheme of things since it prevents a lot of spam, but it prevented me from easily delivering the data to relevant recipients.

S3

My spouse was the one that suggested sticking it in S3. My first reaction was, “okay I could do that and it would be complex and then I’d have to figure out how to show the file from S3”. However, the more I thought about it, the easier and more correct it seemed. I didn’t have to implement any large S3 uploader code, it was built into the boto3 library. Displaying the data was easy too, I wrote a post on it! I shouldn’t dismiss ideas so quickly. I went forward with this idea, implemented an IAM role for S3 security, added the S3 upload, and changed the filenames of the graphs. I wanted to be sure to overwrite the files every time new data came in. This would keep my S3 bucket from getting too big and allow me to hard-link the S3 files while the data changed on the backend. What you see at the beginning of this page is the result. A series of graphs detailing the number of total cases recorded per day on a county by county basis in Wyoming.

Final Thoughts

This was a fun and practical project for dipping your toes into the AWS landscape. I had to touch a lot of different tools and I learned a bit about each one as I went along. This was is a good project to start with, but the tutorial I followed was useful too if you’re starting. I’m glad I made mistakes along the way because it forced me to learn and I’m happy that this data is now more available to the public! You can see it on my github just note that the code will be a bit messy. Or feel free to take another look at the graphsCheck ‘em out!

Update - February 2021

Wow, looking back on this project, I really did not expect it to run this long. Except for a period between October and December, the code is still running. During that period, the Wyoming Department of Health’s website changed slightly. This caused the function that parsed the data to fail. This meant that the data for that time period was not gathered correctly. I fixed this in December, but since the real data was not there, it had to be extrapolated between the point where it failed and when I fixed it. This is why a very straight diagonal line is present in all the graphs. If this project were to be something more serious, hopefully I would have noticed the errors earlier. Another, better way this could have been handled would be to catch failures in the code execution and have some kind of notification sent out when failures occurred. For a simple project like this though, that much hardening didn’t seem worth it to me. I also didn’t expect to be supporting it for this long, but I am happy that I can still check it periodically to see how Wyoming is doing.

Motorcycle Rebuild

Aug 2, 2020 on 1983 Honda Shadow 500 Motorcycle

1983 Honda Shadow vt500

Checkout the video for the TL;DR

More details below…

COVID-19 and Relocation

In the Spring and Summer of 2020, the COVID-19 pandemic was affecting the world and my spouse and I were no exception. In order to seek some fresh air and space, we ended up spending this time in Wyoming. We were pretty comfortable in our new space but we were still a bit limited in the kinds of activities we could do so I started to look for a new project.

An Old Bike

I started browsing Craigslist and Facebook Marketplace for a project motorcycle and started watching YouTube videos for inspiration. I saw a couple project motorcycles that didn’t work out for various reasons, but I finally found one nearby with a valid title that looked fairly decent. A seller nearby had a 1983 Honda Shadow vt 500. Unfortunately, the description of the motorcycle wasn’t very great. The person selling it received it as a part of a trade. He didn’t know what was wrong with it other than what the previous owner had stated, “A bad motor”. This isn’t usually a good sign, but I had tons of time and the price was right. The only thing I was worried about was a cracked engine block. An issue like that would be pretty difficult and expensive to fix. My spouse and I headed to the seller’s house and inspected the bike. It was in rough shape but we couldn’t see any cracks so we took it home. InTruck

Starting Work on The Bike

Initial Problems

So, what exactly did “rough shape” mean? There were a number of cosmetic issues, missing trim, small rust spots, but that didn’t concern me. What did was a box full of parts that had been taken off the bike. I had to make sense of these parts while trying to address the underlying issue of “A bad motor”. First though, the easiest problem to address was the lack of battery. Thats a pretty easy one to diagnose. First I put the key in the ignition and turned it. Nothing happened. That is pretty symptomatic of a dead battery, but it could be a fuse too. It was easy to see, though, that right where a battery should be, there was none. At this point I started making a list of things I needed at the part store. I couldn’t attempt to start it, but I could check some other simple things before heading out. I checked oil and coolant levels. As far as I saw, there was no coolant and when I checked the oil dipstick it was dry, so oil and coolant went on the list as well. IncompleteWithKiki

Missing Battery

I’ve worked on cars and small engines before so motorcycles didn’t seem that foreign to me. An engine is an engine right? Well, I was about to find out the first difference. After a trip to the store I came back and opened the box with the battery in it. There were two containers in it and some plastic tubing. In replacing the battery I learned my first difference. Car batteries are usually sealed. You buy it and can drop it right in the vehicle. This motorcycle battery came as two parts. There is what looks like the battery with lead in it, and a separate container with acid. You have to put the acid into the battery by yourself. I’d never done it before, but its not a difficult process. The instructions are pretty detailed and the steps are more or less: put the acid into the container. That being said, it is acid so if you are going to do this yourself, be sure to take the proper precautions. After getting the battery prepped, I borrowed a trickle charger from the neighbors and started charging it.

Oil Reveals the Problem

The next problem was the oil, pretty straightforward. Remove the plug and filter and let the oil drain. Then replace the filter and put back the plug. Finally, add oil until the proper level on the dipstick is reached. So I started and by loosening the oil plug and out came a mess. A sluggish pour of what looked like sweetened condensed milk started flowing from the hole. Well, thats a problem. My mind immediately jumped to a crack in the block, what I had feared before. A crack in the engine block can lead to coolant mixing with the oil. When that happens the oil starts to look milky, exactly what my oil looked like. All that being said, these symptoms also apply to other things so it might not be a crack in the block. It could just be a bad head gasket. That being said, both problems required the engine to be mostly disassembled. This is one of the most time consuming repairs one could do to a vehicle. I had high hopes though, maybe the bike had just been sitting for a long time and moisture worked its way in and mixed with the oil. I still wanted to try to start it up to verify that it was indeed a problem and so I could identify any other problems along the way. I replaced the oil, added some coolant and after the battery charger was done, I hooked up the battery and pressed the ignition button.

The engine didn’t start, but after finding the fuel valve and turning it to the “On” position I gave it another try, and lo and behold the bike came to life! Not all was well though, it ran rough and there was smoke pouring out the exhaust. These both seemed like telling signs that I had to take it apart. I turned it off and checked the oil. Sure enough it was getting more milky.

Diving in Deeper

Prep Work

Unfortunately, this was not going to be an easy fix. While I could try to rebuild the engine with just the knowledge in my head, I figured I could probably use some help. Haynes manuals are well-known in the car world for providing all the information one would ever need on a vehicle, so I started looking around for the equivalent in the motorcycle world. I found myself a used Clymer manual with all the information I would need. If you find yourself working on a motorcycle, these manuals are great. They describe how to do all kinds of repairs, they also give all the part specifications right down to the bolt. In addition to the manuals, I found a Honda Shadow forum at hondashadow.net/forums where tons of people with Honda Shadows collaborate and discuss Honda Shadow motorcycles. I posted there with my problem and the great people there confirmed my suspicions and decision to move forward in taking the engine apart.

Taking It Apart.

The Clymer Manual shows exactly how to take the engine out and get it apart. That being said, sometimes it doesn’t need to be followed precisely. The Honda Shadow forums had another manual available that showed the process in a different way. The Clymer manual had extra unnecessary steps so referencing two sources helped avoid unnecessary work. I won’t go into too much detail, the manual can be reference for that, but these are the things I had to remove in order to get the engine out. I started with the seat and fuel tank. These were easy because both were missing bolts that hold them in. There is an air tube under the tank that connects the air filter box to the carburetor that popped off right after. I disconnected the throttle and choke cables that connected to the carburetor and removed it. WithoutCarb Next, I drained and removed the radiator and hoses. I had to remove the rear tire before I could remove the exhaust. Both came off easy. After that were just small trim pieces the shift lever and the rear brake lever. Finally, I placed wood blocks under the engine, took off the sub-frame, and removed the engine mounting bolts from the engine. It took a little wiggling but it came free after a bit. EngineOff

Focusing on the Engine

With the engine free, I could now move on to taking it apart and looking for any coolant leaks into the engine. Most pieces came off easily and as the manual described but I did end up rounding the 12mm socket I was using on some really stuck bolts. There was also another small hidden bolt on each cylinder head that was not described in the manual. If your cylinder heads seem stuck, be sure to check for extra bolts. ValveCoverOff_7 With the heads and cylinders off I immediately saw the problem. The good news is that it wasn’t a crack in the block, or a blown head gasket. The bad news is that it was a hole right in the middle of the rear cylinder. HoleInPiston2 HoleInPiston5

The Real Problem

A hole in the cylinder isn’t a very common problem. Ultimately, it turned out it was gasoline, not coolant, that was turning the oil milky. Gasoline was entering the combustion chamber, it was not being compressed properly, and who knows if it was even igniting. It does explain the engine running rough though. It is a little odd that something like that happened. The only reasons I could think of were if a foreign object entered the combustion chamber or if it maybe got to hot and weakened the metal. These days ethanol is added to some gasoline so maybe the engine got too hot and melted a hole in the piston. This is all just speculation, without being there its hard to determine the actual cause. I could have potentially identified this issue earlier if I had a compression tester, but the outcome would have been the same. In any case, I had to take apart the engine even more to collect the metal parts of the cylinder that had broken off. I cant have metal bits flying around causing more damage. Before doing that though, I ordered a new piston, piston rings, and gaskets from Ebay.

Fixing The Issue

Waiting for Parts

COVID-19 definitely interrupted a lot of the day to day infrastructure we rely on, so shipping was a little slow. In the meantime I could still identify and fix other problems with the bike. First, I removed all the small metal fragments that remained from the hole in the piston. Debris After that I took apart, cleaned and rebuilt the carburetor. It seemed mostly clean on the inside but it was comforting to know that it wouldn’t be a problem. Not only that, but it looked a lot better on the outside after cleaning. The fuel line needed replacing too, it was pretty stiff and cracked. I also started getting into the box of parts. I mostly tried to identify where they belonged, attaching them now would get in the way when I had to put the engine back in. It was a good time to address the front brakes though, they were pretty spongy. The brake fluid was a little low, so I added more and bled the system. I also tried to sort out the electrical system a little bit, there was a mess of wires up front and in the rear. I mounted the front headlight and verified it worked. The last and most time consuming thing I did was cleaning off old gasket material. The material was stuck on pretty well so the task ended up being a bit tedious, but the old material slowly came off.

Replacing the Piston and Engine Reassembly

Piston7 After the packages with the piston and gaskets arrived I got to work. First, I tackled the piston. I made sure to balance the new piston with the old functioning one so that I wouldn’t throw off any engine balance. There are some good YouTube tutorials on this, but its pretty straightforward. Drill a series of small dents until the pistons weigh the same. These dents should definitely be small though, don’t drill deep and don’t drill through anything. Its best make these dents where the material is thick too. The goal is to shave off material not to put holes in things. Once that was done, removing and replacing the broken piston with the new one was easy. I just removed a retaining clip and slid the old piston off. To put on the new one, I oiled up any surfaces that touched and moved, I put on the rings, slid it on the connecting rod, and put back the retaining clip. If it weren’t for the whole engine removal and disassembly, it would have been easy! Piston8

Next came reassembling the engine. The book was a great help here. I replaced the gaskets, assembled the parts, and torqued everything to specification. I did stumble a little bit when it came to the timing chain though. I set the camshaft, timing, and lifters easily on the front cylinder but the when it came to the rear cylinder, things just wouldn’t seat right. It seemed like the chain was too small or under too much tension and I couldn’t fit it onto the camshaft. Setting the timing correctly was out of the question. I had to disassemble almost the whole rear cylinder again to figure out the problem. It turned out to just be a kink in the chain that was a bit stuck so I made sure to lubricate the chain well. Other than that, reassembly went by the book.

Mounting the Engine

Now that the it was back together, I wanted nothing more than to start the engine to see if I had corrected the problem. Unfortunately, I lacked the tools and equipment to test the engine by itself so I had to reassemble the whole motorcycle to test it. With the disassembly process still fresh in my mind, and with the book by my side, this went much faster. First, I had to fit it back in the frame. Mounting and un-mounting the engine from the frame were probably the hardest parts. There wasn’t an easy way for me to lift and adjust the engine. I was working with wood boards and a floor jack so getting it in just the right spot was a little difficult. Getting help from my spouse made the process go easier, but doing this with three people might be even better.

From there it was just disassembly in reverse. I put on the radiator and connected the hoses. The driveshaft (not a chain like most motorcycles) was a little difficult to line up and put on. Putting on the exhaust was simple but took some muscle and it was a dirty job. There were a load of electrical connections that needed to be made but I focused mainly on the ones just related to the engine. I tried not to put everything back, since I just needed to test if the engine ran, so I left a lot of things off for now. Some pieces like the clutch lever and gearshift weren’t essential but would also be good to test worked. I hooked up the fuel system and air-box by the book. I wanted to make sure that if I did run into problems, those two systems wouldn’t be variables. I finally mounted the fuel tank and was ready to start it up.

Ignition! Well, not quite at first. The engine needs to be in neutral to start, so I had to shift it a little. The fuel petcock was also in the off position so no fuel was flowing, I forgot again. Finally, the engine kill switch was in the off position. After that and after a few cranks to get the carburetor primed, it started up and ran!

More Things to fix

With the engine sorted out, I could now pay attention to all of the other problems. Off the top of my head were a couple things. The exhaust was smokey, the engine sounded ‘click-y’, the clutch and shifting felt odd, all of the unmounted parts needed mounting, The junction box looked horrible, there were no turn signals, and a good clean was in order.

I started with the engine and did a lot of reading in the manual. Incorrect valve clearances seemed to explain both of the engine problems. The problem was that I remember setting them correctly when I assembled the engine in the first place. After verifying the clearances again, I noticed my problem. I set the clearances to 1mm when it should have been 0.1mm. Luckily, I didn’t need to take apart the engine for that. Its a bit tight, but there are some convenient valve covers that can be taken off to perform the necessary adjustment. That solved the clicking problem, but the exhaust was still smokey. I’d address the smokey exhaust later the cause and fix seemed easy enough for future me to solve. I’m in a high altitude area so I figured the problem could be related to the air fuel mix. Another thought was that the fuel could be bad. I had only used the fuel that came with the bike and I did not know its history. Whoops!

The clutch problem was next. I started with the simplest idea, adjusting the cable. After fiddling with a couple different adjustments, it didn’t seem to help. I figured something must have gone wrong in installation. More reading I guess. I read through the clutch portion of the manual and prepared myself to tackle the engine again. Luckily, I didn’t have to remove the engine to address the clutch, but I did have to remove a lot of those parts again. The exhaust, radiator, and sub-frame had to come off, but other than that it wasn’t too bad. Maybe I was saying that because I’d already done it once. After the sub-frame came off I drained the oil and removed the clutch cover and saw my problem. The clutch is held in place by a spring plate with four bolts going through it. Although I remember torquing it to spec, it must not have been enough because one of the bolts was more loose than the rest. Clutch2 This was the issue for sure. So I made sure to follow the manual to the letter and I torqued the bolts in a star pattern in three sequences. I also made sure to reference pictures and tutorials online so that I was sure everything went well this time. My guess is that since these bolts are on springs, I tightened one slightly more than another out of sequence and the loose one was more free to move. Regardless, after I reassembled everything, this solved the clutch problem and it felt great after an adjustment. I’m still not great at shifting motorcycles, but at least I know that’s a problem with me not the bike.

The only things left were cosmetic! I put everything on that I needed to ride safely around the block and took it for a spin. It was great to finally enjoy the fruits of my labor, even just for a bit. There was still more to to fix though. Around this time we also relocated from one corner of the state to another. I packed up the bike and its parts and we headed out.

Simpler Problems

A New Problem Appears

We ended up at my parents house for a visit. My parents and I had talked about the bike a lot on the phone and they were curious about it. Once we got it unpacked we started it up and took a few spins around the block. It was a fun time, but it pretty quickly showed another problem, a cooling problem. The engine overheated at the end of our journey and ended up spitting some coolant out the overflow. Well, add that problem to the pile. On a positive note, at my parents house I had a lot more tools and equipment to work with. My Dad’s worked on cars before too so bouncing ideas off him was a great help. After some diagnosis, we thought it was the coolant temperature sensor not sending the ‘turn on’ signal to the cooling fan. So I ordered that sensor in addition to some new turn signals, a fender, a horn, and a new junction box. The old junction box worked fine, but it was missing a cover and was a bit sketchy looking. It might have even been a safety hazard to touch.

The Easy Fixes

The fender was easy, just putting it where it was supposed to go and finding some new bolts. The horn was also easy. I had to fabricate a bracket since it wasn’t the stock horn, but it fit fine and wired up easily. The junction box was simple but a bit time consuming. I had to take off all the electric plugs and plug them back in to the correct spot on the new box. The plugs were stiff so I had to get a little creative but the process was not complicated. JunctionBox1 JunctionBox2

I also wanted to fix the smokey exhaust. I figured it would be best to start with the carburetor. I referenced the manual more, watched some YouTube videos, and finally decided to take another stab at it. At the end of the day, I used what the manual suggested for high altitude as a baseline, but fiddled a little more with the carburetor settings in order to get it right. I still might play with it again someday because the air fuel mixture still seems a little lean. However, problem solved! The smokey exhaust is gone.

Electrical Woes

The turn signals arrived, and I knew there was a problem when I opened them. They had two wires to control the light, where the single signal in my possession had three wires. I should have checked before I bought them, but I thought all turn signals were more or less the same. I read a bit online about turn signals though and these seemed like they would be fine. There wouldn’t be a constant on setting for the signals, but that seemed okay to me. Tackling the electrical was a bit difficult though. There were wires everywhere, none with connectors and none labeled. I started going through them one by one referencing the wiring diagram in the manual. There were a couple issues to fix with the brake light. The switch for the front brakes had a broken contact. Getting a new switch was going to be expensive so instead I modified and added a new contact to the existing one over a couple hours. Getting all the tiny switch pieces to line up took a lot of patience. The rear brake light switch was triggered by a spring so I had to find a suitable spring for it to function. Once I resolved both these problems it was pretty satisfying to watch that brake light turn on and off. Sometimes, its the little things that keep you going. Now, with everything labeled, the turn signals seemed easy to wire up, I just left off the wires for the constant on setting. The only unfortunate part was that the signals did not flash. They were either on or off. The relay was broken so it was off to the parts store to find another. The broken relay was a three contact relay so I got a similar three contact relay. I put it in place, hooked up the wires and nothing. TurnSignalRelay Maybe the contacts were different on this relay. I tried all the combinations, but nothing worked right. I tried rewiring the turn signals and that didn’t work either. It was by happenstance that I didn’t hook up one of the wires onto the relay and that was the trick. I’m still not sure why it was wired that way. It was frustrating to have had to go through all that but the reward was working turn signals. I made sure to label and write down everything I found. I didn’t want to go through that wiring again.

Cooling Mistakes and Fixes

The final problem was cooling. The temperature sensor came in, but getting to it was a problem. It was nestled in a place that was difficult to reach. It was attached to the thermostat housing and I could could not get a wrench in the tight space. I could get a socket in there but the sensor was too long for the shallow socket I had. Ultimately, it had to come apart enough so that a wrench could fit. CoolantCap What seemed like a simple replacement took a couple hours unfortunately. To make things worse, that turned out not to be the problem. The sensor sends a signal to the fan switch which turns on the fan when it gets to a certain temperature. It turned out the fan switch was bad. FanSwitch The good news, is that we identified the problem and knew the fix. The bad news is, its another part that needs replacing. I chose another fix though. The switch should probably be replaced at some point, but instead of replacing it, I chose to short the switch. That is, I chose to bridge the two switch contacts so that the switch was always in the ‘on’ position. This means that the fan will remain on all the time, even when the engine is cool. It might need some time to warm up on chilly days, but it prevents the engine from overheating. I fashioned a nice little wire that could be removed when the switch is eventually replaced. Finally, the bike was finished!

Final Thoughts

I’ve taken the bike on a lot more spins around the block. I still haven’t got my Motorcycle licence endorsement so I can’t go on longer journeys, but its been a great time! This project taught me about almost every part on a motorcycle. The only thing that didn’t need fixing were the front forks and the dials. There’s a couple things I’d still want to do at some point. I’d like to fix that cooling fan switch, probably give it a paint job, and find more trim pieces. However, I think I’d feel pretty confident looking at most other motorcycles out there now, especially with a manual in hand. I would highly recommend anyone fixing a motorcycle to get a shop manual. It helped me with everything that had to be done. It was a fun project to do while I had the time and riding it after all the hard work is one of the best feelings.

Complete1 Complete2 Complete3 Complete4 Complete5 Complete6

Image Hosting with AWS

Mar 8, 2020 on AWS Image Hosting Github Pages

New Website, New Problems

I’m hosting this latest version of my website using Jekyll in combination with Github Pages. If you want to know more about that process you can read my post on that here. Its been great! I’ve still not been contributing as much as I’d like, but I am contributing more! Along with new content, I’m also able to create and deploy site design changes with just as much speed. One of those changes is how I’m hosting the images in the background at the top of the page.

Originally, I only used one background image for my website, the one with four couches and a girl. I wanted was to be able to show off more of my photos, so I added the images and a little javascript to randomize when they appeared. Job well done right? Well, there are a couple problems. First, I use a reasonably decent (read: nice for 2015) camera and its images can be quite large. For that matter, media files in general, depending on quality and compression, can be quite large. Since I’m using Github to host my website, I may run into their size restrictions if I store too many photos. Second, Github is generally used to store code repositories. Using it as a website hosting platform is a nice feature but those two pieces of functionality don’t alway agree on how to do things. Github and other repositories are not generally the ideal place to host images. Third, every deployment with an image change can take a long time to upload to the Github server.

For my initial implementation, I was lucky enough to have the option to host my images in Github, but that needed to change.

My First Idea

AWS costs money, and I didn’t want to pay. I wanted my cake and to eat it to, that is, I wanted my images to be hosted somewhere but I wanted to have strong control over those images and my IP (intellectual property). I probably could have used Imgur, Flickr, or maybe even Instagram as free public solutions to host my images. I didn’t like the inherent public nature of image hosting on the platform. For my pictures with my friends, sure they’re great. Vacation pictures, awesome! For storing my amateur photography? No way. Protected GIF

I tried using a private or even public album with Google Photos, but it seems Google has gotten wise and discourages direct links to photos. It seems like some companies may have come up with ways to get around this, but most of those seemed like hack-y solutions.

In the end, my desire to control my images was more powerful than my desire for free solutions, but AWS S3 bucket hosting, in this case, was cheap enough for me.

Enter AWS

I’m sure there are a ton of ways to configure AWS resources, store your images, and grant access to them. However, I think this method is probably the simplest and most straightforward. I chose to upload my photos to a certain directory in an S3 bucket and grant public read access to that bucket.

Breaking this down a bit, these are the steps that I needed to host images from AWS and use them on this site. 1. Login and Create an S3 Bucket 2. Configure the S3 Bucket 3. Upload Photos 4. Modify Website to Link to Photos

If you don’t have an AWS account, its pretty easy to create one. Just go to https://aws.amazon.com/, click “Create an AWS Account” and follow the instructions. You will have to enter credit card information to cover any costs that you incur with any of the resources you use. If you’re concerned you can look at AWS’s pricing information for more on this. I havent been charged yet and its been a couple months but I’m only using one S3 Bucket with a total of less than 40 MB. After you’ve created your account, or if you already have one, log in and get to the AWS Console.

From there, under AWS services, you can search in the box for “S3” and click the link. Alternatively you can find and click the link under > All Services > Storage > S3. From there it should be pretty easy to find the button with the label “Create Bucket”. Once clicked the “Create bucket” dialogue will appear. Choose a good name for the bucket, AWS has some good information on this if you click the little (i) icon next to “Bucket name” in this dialogue. I named mine luisgarcia.me-background-images for reference. You’ll also need to pick a region for it, this is worth looking into if your new to AWS but for now just pick the region closest to you. Once those two things are filled out, we’ll just accept the default settings on all the other pages, so keep clicking next until you’ve created your bucket. Now we can start configuring it!

Step 2 - Configure The S3 Bucket

Once you have the bucket go ahead and click it! This will take you to the bucket overview where you have a lot of control over your bucket. Right now, lets skip the “Overview” tab, it deals a lot with whats inside the bucket. While thats very important, I usually like to save it until the bucket is configured that way I don’t expose any sensitive data accidentally and I can remove the bucket easily if things change. A lot of the settings and confguration values we are about to change could have been chaged from the bucket creation dialog. I think its probably more important to configure a bucket once its already created though. In a bucket’s lifetime it can only be created once but its configuration could be changed any time you want.

We’ll go ahead and click the ‘Properties’ tab next. Once you’re in there, click ‘Static website hosting’ accept the defaults and click save. Similar to our Github pages, we can use buckets for static website hosting. For our purposes though, since we just want image hosting we’ll use this functionality too.

After that click the ‘Permissions’ tab. This is where we’ll allow read access to our bucket. Its pretty important to get this right, so if you’re not comfortable read up a bit more on AWS S3 bucket permissions. In the permissions tab, click the “Bucket Policy” button which will open the Bucket policy editor. This is where you can copy and paste the following JSON. Be sure to change <bucketname> to the name of your bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowPublicRead",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<bucketname>/*"
        }
    ]
}

This JSON is telling AWS to create a policy named “AllowPublicRead” that grants the “GetObject” action on this S3 bucket to all resources. This policy is applied to all objects in the bucket, designated by the resource arn:aws:s3:::<bucketname>/*. If you wanted to just allow public read to a specific directory you could change this to arn:aws:s3:::<bucketname>/specificDirectory/*

Step 3 - Upload Photos

This is an easy one, go back to the “Overview” tab, click upload, and pick the files you want in the bucket. I’m using images, but you could put anything here. Notably, since we configured it to be publicly accessible, anything you put there can be accessed publicly.

Step 4 - Link to photos.

Since we configured our bucket for static website hosting earlier it should be pretty easy to link to our files, or images in my case. AWS has a pretty standard way of creating url’s from buckets. This can be changed if you want, but we’ll use the default since I already have my real website somewhere else. Anyway, our photos should be avaliable at http://<bucketname>.s3-website.us-east-2.amazonaws.com/<directory>/<filename>, note replace <bucketname> <directory> and <filename> with your relevant values. If you’re still having trouble seeing your files, check the properties tab again and click Static website hosting. Your website url should be at the top of the box that pops up.

Once we have these links, you can alter the HTML in your other site to link to these new images and there we have it! Image hosting with AWS!

Final Thoughts

I like this solution for the control I get over the objects, its relatively low cost, and low complexity. If one doesn’t have an AWS account, I’m sure you could do the same on the Google Cloud Platform or with Azure, other ‘cloud’ providers. I don’t like that its not free, but its a cost I can accept. Going forward, one thing I’d like to validate is the performance of loading theses images from a different place. I generally trust in the uptime and availability of the platforms hosting Github Pages and AWS S3 but it is possible that in certain cases the loading of either the site or the images could differ. Regardless I believe in the solution and I’m happy with it. This method is how I’m loading all the images you see at the top of the page so you can evaluate the solution in real time!

How to Watermark All Your Photos

Jan 20, 2020 on Photography FFmpeg

In looking at images across the internet, an reocurring element that appears in many of them is a watermark. If you’re unfamiliar with the concept, a watermark is a small design or text added to an image. The purpose behind a watermark is to visibly mark a the image to provide the creator with proof of ownership. For instance, if I mark my photos with my name and someone else downloads the image and uses it, its clear who the photo belongs to and who the original creator is. A watermark’s effectiveness is debatable. Depending on the watermark, it can be sometimes be easily cropped off. There’s also software out there designed to explicitly remove watermarks. However, leaving a watermark on an image does offer at least some security when exposing your images to the internet. In this post, I’ll describe an easy way to add a watermark to an image, and a way to automate the process so that watermarks can be added en masse to a large number of photos in a short period of time.

FFmpeg

The tool I’ll be using to help us along is FFmpeg. We’ll only be using it to manipulate our images, but its great open source tool if you want to for other things like video and audio processing. Its a very powerful and versitile tool and I highly recommend it for media processing.

Other requirements

In addition to FFmpeg, you’ll need at least two images, an image that you want to apply a watermark to and the watermark image itself. When designing the watermark image some things to consider are size, color, and transparency. For my watermark image, I just chose my name set at an angle. Additionally, I made sure that the background of the watermark image was 100% transparent. The transparency should allow the watermark to be seen, but the original image can still be mostly intact.

Simple Overlay

Running the following command in a shell will use FFmpeg to do a simple overlay, placing the watermark image on top of the original image

$ ffmpeg.exe -i "originalImage.png" -i "watermarkImage.png" -filter_complex "[0:v][1:v] overlay=main_w-overlay_w:main_h-overlay_h" "outputImage.png"

The resulting image is a combination of the original image and the watermark. We’ve accomplished our goal. Now lets go into what the command is actually doing.

ffmpeg.exe

This is an easy one, we’re calling the executable

-i "originalImage.png" -i "watermarkImage.png"

these two arguments assign the paths to the original image and the watermark image to the first and second inputs.

-filter_complex "[0:v][1:v] overlay=main_w-overlay_w:main_h-overlay_h"

As the argument suggests, this is a complex one to explain. “filter_complex” is an argument that describes various filters to use through FFmpeg. You can read more about it here. Lets break this specific filter down though. The first things we must do is declare the inputs to the filter. In our case we’re using our two source images. The way to declare our inputs to the filter is with the square brackets. [0:v] is stating that we should use the “zero’th” or first input, our original image, and we should use the visual portion, or “video stream” from it. [1:v] is similar, only this time we’re using the second input, the watermark image. These two images will be passed to the “overlay” filter. Finally, the last piece after the “=” symbol states that the watermark’s top left corner should be placed at the point determined by two formulas. “main_w - overlay_w” determines the “x” position, where main_w is the width of the first input and overlay_w is the width of the watermark image. The “y” position is determined in a similar way. I chose to place my image in the lower right hand corner, but by manipulating these arguments the watermark can appear anywhere on the original image. For example “overlay=0:0” will place the watermark in the top left of the original image.

"outputImage.png"

Finally, the last argument is the path to the output image. This tells FFmpeg where to write your output file. If a file already exists there, FFmpeg will ask you if you want to overwrite it. If you dont want to be prompted for this question, you can just use the “-y” flag to always overwrite and “-n” flag to never overwrite.

Making a Better Watermark

Transparency

My watermark started as an opaque image, that is, originally the text in my image was not transparent. A nice feature of watermarks is their transparency, it maintains the idea of marking an image as your own while also preserving as much of the image as possible. Fortunately FFmpeg offers a way for us to do that too, we can just add another filter

[1:v]colorchannelmixer=aa=0.5

This filter will take our watermark image and make it 50% transparent. Putting it all together we can use this command to generate our new image

$ ffmpeg.exe -i "originalImage.png" -i "watermarkImage.png" -filter_complex "[1:v]colorchannelmixer=aa=0.5[opacity];[0:v][opacity]overlay=main_w-overlay_w:main_h-overlay_h" "partialTransparentWatermark.png"

This command may look a little different than expected. Using FFmpeg’s filter_complex, we can do multiple filter operations as long as we keep track of our streams. That is, now that we’ve created an video stream that is 50% transparent, we have to pass that transparent stream onto our overlay filter. I choose to keep things straight by naming the outputs of each filter result, in this case I named the output of my transparency “[opacity]”. There are other ways to pass around streams, for more information, again you can look at the filter documentation.

Consistency

So far we’ve done a great job watermarking our one image. However, if we continue to use the previous command on our images a problem will emerge. The watermark will appear to grow and shrink relative to the size of the image we’re watermarking. That is, if our watermark is 100x100 and we put it on a 200x200 image, the watermark will take up the bottom right quarter of the original image. If our original image is 400x400 and we use the same 100x100 watermark, then it will take up the bottom right sixteenth of the image. One solution is to create a new watermark for every image size. That does seem to get tedious very quickly. Instead lets use FFmpeg to create a more consistent watermark across all image sizes. Unfortunately, there’s not an easy way to scale an image based on the size of another image in one command. We must detect the height and width of the original image so we can scale the watermark appropriately. Fortunately, included with FFmpeg is their ffprobe program, used to gather information about a piece of media. We’ll use it to get the size of our original image.

$ ffprobe.exe -i "originalImage.png"

This command will result in an output that looks like this:

ffprobe version 4.1.3 Copyright (c) 2007-2019 the FFmpeg developers built with gcc 8.3.1 (GCC) 20190414 configuration: –enable-gpl –enable-version3 –enable-sdl2 –enable-fontconfig –enable-gnutls –enable-iconv –enable-libass –enable-libbluray –enable-libfreetype –enable-libmp3lame –enable-libopencore-amrnb –enable-libopencore-amrwb –enable-libopenjpeg –enable-libopus –enable-libshine –enable-libsnappy –enable-libsoxr –enable-libtheora –enable-libtwolame –enable-libvpx –enable-libwavpack –enable-libwebp –enable-libx264 –enable-libx265 –enable-libxml2 –enable-libzimg –enable-lzma –enable-zlib –enable-gmp –enable-libvidstab –enable-libvorbis –enable-libvo-amrwbenc –enable-libmysofa –enable-libspeex –enable-libxvid –enable-libaom –enable-libmfx –enable-amf –enable-ffnvcodec –enable-cuvid –enable-d3d11va –enable-nvenc –enable-nvdec –enable-dxva2 –enable-avisynth libavutil 56. 22.100 / 56. 22.100 libavcodec 58. 35.100 / 58. 35.100 libavformat 58. 20.100 / 58. 20.100 libavdevice 58. 5.100 / 58. 5.100 libavfilter 7. 40.101 / 7. 40.101 libswscale 5. 3.100 / 5. 3.100 libswresample 3. 3.100 / 3. 3.100 libpostproc 55. 3.100 / 55. 3.100 Input #0, image2, from ‘originalImage.jpg’: Duration: 00:00:00.04, start: 0.000000, bitrate: 103560 kb/s Stream #0:0: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown), 2400x953 [SAR 300:300 DAR 2400:953], 25 tbr, 25 tbn, 25 tbc

Thats a lot to take in! Fortunately, we don’t need all of it. All we need is a tiny bit at the end. The resulting outpus shows Input #0 has a video stream, Stream #0:0, and its resolution is 2400x953. As with the other commands, we can simplify this one too.

$ ffprobe.exe -i "originalImage.png" -v error -hide_banner -select_streams v:0 -show_entries stream=width,height -of default=nw=1

This command will get straight to the point and give us our width and height of this form:

width=2400 height=953

Bringing it all back together, we use the original width and height as a base to ensure our watermark is 25% of the size of the original image. This will result in the watermark taking up 1/16 of the original image. The original image does have a weird size though, so lets try to maintain the aspect ratio of our watermark as its applied, rather than distort the watermark image. A quick note here, there are other strategies that could be used here to provide other definitions of “consistent watermark”. I’m choosing to preserve the aspect ratio of the watermark at the expense of the inconsistencies in the height of the watermark. Another strategy could be to apply a standard scale to the both the watermark’s height and width, that could result in a stretched look of the watermark if the aspect ratio of the two images does not match.

I used ffprobe again to find out my watermark’s size is 640x480

Width Scale = (640/480) * 2400 * .25 = x Height Scale = (640/480) * 953 * 0.25 = y

Width Scale = (2400/953) * 640 * 0.25 = 402 Height Scale = (2400/953) * 480 * 0.25 = 302

We can then use these values in our ffmpeg command

$ ffmpeg.exe -i "originalImage.png" -i "watermarkImage.png" -filter_complex "[1:v]scale=402:302,colorchannelmixer=aa=0.5[opacity];[0:v][opacity]overlay=main_w-overlay_w:main_h-overlay_h" "scaledWatermark.png"

Automation

Watermarking a large number of images could take some time if one were to do it by hand, thats why automating this process is essential. There are many ways to automate this process, and I’ve chosen to write a bash script for speed and ease. However, these methods can easily be transferred to another technology or programming language. The important pieces to keep are the processing steps. Those steps consist of the following. 1. Identify the resolution (width and height) of the image intended to be used as a watermark with FFprobe.exe 2. Identify the resolution (width and height) of the image that will receive the watermark with FFprobe 3. Calculate the scale factor to apply to the watermark 4. Create and execute the command to ffmpeg.exe.

If you’d like to see the bash script I put together to accomplish this for myself, you can check it out on github here

New Blog, Who's This

Apr 26, 2019 on Jekyll Github Hexo Integration Deployment

I’ve made a new blog! Again!

Every blog of mine starts out with a post like this. So why not continue the pattern? Yes, it is very redundant to keep making different iterations of blogs every couple years. However, I’m hoping this time its different (said everyone). All of my previous blog iterations have generally been custom solutions. That is, in order to hone and practice my developer skills, I’ve implemented various web frameworks in order to come up with a blog that works for me. The good news is that I definately gained knowledge and experience from the process. The bad news is that since they were custom solutions, it wasn’t always the easiest to contribute to my own blog. I’m hoping that by avoiding custom work and switching to an established solution, contributing and keeping up with the blog will be easier. Basically, I’m trying to make the barrier to entry easier, hoping that I’ll contribute more.

Introducing Jekyll

I chose Jekyll as my ‘established solution’. Its an application developed with Ruby that takes markdown, along with some style information, to generate a static website. The main idea being that a person could write their blog posts in markdown and use Jekyll to quickly interpret and serve the content on a website. For me, the ‘pros’ are that its easy to work with, easy to contribute to, and has a lot of support. Unfortunately, there are some ‘cons’ as well. If I’m hosting a static site, its hard to embed ‘dynamic’ behavior. A few years ago, again, to practice my developer skills, I made my own version of a url shortner like bit.ly. Unfortunately, since this is a static site I am unable to embed and host some of my projects like that (You can read more in the Github Pages section). With that in mind though, I can always host a project somewhere else and link to it if needed. So in the end, the restrictiveness of a static site wasn’t a very big ‘con’ to me. However, there was another major ‘pro’ in using a static site: hosting.

Github Pages

A major feature of Github thats come out in fairly recent history has been their Github Pages concept. In essence, if you commit static web files to a repository in Github, you can configure the repository in such a way that Github will host the static website for you. Given that Github is free (for personal use) this is a great way to host a website for free. If we go back to my running ‘pros’ and ‘cons’ list, this is a pretty good ‘pro’. It also continues the ease of contribution since all I should need is Jekyll and an internet connection to contribute.

A Note On Integration

One of the big concepts being used in modern development is Continuous Integration/Continuous Deployment (CICD). Unfortunately, one of the things I havent quite worked out yet, is how to better improve my deployment pipeline. For example, in writing this post and pushing it live, I have to have Ruby, Jekyll, and Git installed, I have to run Jekyll against the new markdown file, and finally push the new generated site to Github. I have Git, and will continue to have Git installed on all my machines but I’m not a Ruby developer. That is, Ruby and Jekyll are definately a dependency that I don’t work with on a day to day basis. As such, my desktop is the only machine I currently have running Jekyll. This restricts me to contributing to the blog only when I’m on my desktop or when Ruby and Jekyll are installed on a machine I’m working with. This isn’t very portable so CICD is something that I’d like to improve upon going forward. That is, I’d like to be able to commit changes and have the site automatically generated and re-committed so that I only have to worry about committing to Github.

Thoughts on Jekyll Alternatives

In picking which static site generator I had also tried Hexo. Instead of relying on Ruby, hexo uses a Node package. So if you’re not a fan of Ruby, or have different dependencies to manage, Hexo might be a better option for you. In my experience, it does seem more flexible. However, it also seems more verbose and a tiny bit cumbersome. I’ve spent an equally small amount of time with both Jekyll and Hexo so there are definately things I haven’t come across in both options, but either should work for getting a blog up and running quickly with Github pages. There are more static site generators than these two, but since I haven’t worked with them I won’t comment.

Final Thoughts

I’m sure I’ll have more opinions as I continue writing posts and continue to work with Jekyll, but so far my experience has been great. I’ve been able to customize this site to my hearts content and have not run into anything in my way. I hope that going forward, I achieve my goal of contributing more to this blog and that Jekyll helps in that regard.

Test Technical Blog

Apr 14, 2019 on Jekyll GitHub

Technical Test

This is a test for the Technical Blog

Test Non-Technical Blog

Apr 14, 2019 on Jekyll GitHub

Non-Technical Test

This is a test for the Non-Technical Blog

All posts in one long list

Lets Make an Echo Show Skill with the Visual Studio Code and the Serverless Framework!

Introduction

Project Setup Review

Create The New Skill

My ‘pre-commit’ script

Prepare The Skill For Visuals

Enable the APL Interface

Create a New APL Document

Add an APL Package

Add a Headline

Hook the APL to a data source

Save and Rebuild!

Write the Code to Interact with the Visuals

Serverless Comes In

Test It Out!

Further Reading

Sync Changes Back to the Code

Sync Changes Back Upstream to the Developer Console

Must Have More …

Lets Make an Alexa Skill with a Dynamo Database

Introduction

Problem

Designing the Dynamo DB Table

Columns

Adding to Serverless

Interacting with our Database

Querying the Database

Adding to the database

Putting it all together

Deploying and Testing

Checkout the code

Lets Make an Alexa Skill with the Visual Studio Code and the Serverless Framework!

Before You Start

Visual Studio Code

Set Up the Environment

Creating the Skill

Building and Deploying The Skill

First Skill Deployment

First AWS Resource Deployment

Look for the Lambda in AWS

Define Skill Configuration and Test!

Cleanup

Final Thoughts

Updated daily at 12:00 PM Eastern

About This Project

What’s shown here?

Horizontal Line

Straight Diagonal Line

Exponential Curve

Flattening Curve

Why Did I Make This?

How This Was Made

AWS

The Plan

Starting with the Lambda

Working with DynamoDB

Lessons Learned

Going Forward

Data Delivery

E-mail

S3

Final Thoughts

Update - February 2021

1983 Honda Shadow vt500

COVID-19 and Relocation

An Old Bike

Starting Work on The Bike

Initial Problems

Missing Battery

Oil Reveals the Problem

Diving in Deeper

Prep Work

Taking It Apart.

Focusing on the Engine

The Real Problem

Fixing The Issue

Waiting for Parts

Replacing the Piston and Engine Reassembly

Mounting the Engine