Author: Benjamen Pyle

Benjamen is a genuine and resourceful technology creator with over 20 years of hands-on software development, team building and leadership experience. His passion is enabling technology teams to be their best by bridging modern technical design with outstanding business problem-solving. Recognized as an AWS Community leader in the areas of Event-Driven and Serverless Architecture, he brings multiple years of pragmatic experience designing and operating modern cloud-native and containerized solutions. When Benjamen doesn't have his head in the clouds, he's either playing golf with his wife and 2 boys or they are outside with their 12 paws.

Data Serverless

Event-Driven Serverless Data Architecture

Posted on March 25, 2023by Benjamen Pyle

The Preface

Follow me along on a journey toward data unification. One of the applications that I work on is a modern, distributed, event-driven and serverless-based architecture. What that means is that data is completely isolated from other components and evolves at a different pace from its neighbor. This type of architecture is achievable using Event-Driven Serverless Data Architecture with AWS.

This is great if you are building a transactional system. You’ve got isolation, independent component scaling and feature delivery that goes at the pace of the team working on it. So what could be wrong? What possibly isn’t good about this outside of the fact that modern distributed systems are complex? The big issue is that all of this data is not in the same place.

What’s the point in having everything in the same place you ask? Simple. Source of truth for:

Reporting
Public APIs
Versioning
Audits
Data Sandbox

These are just the tip of the iceberg. When you are working on a big system with lots of data, having a single ingress and egress point is important when you are talking about the above.

Infrastructure Serverless

EventBus Mesh

Posted on March 19, 2023by Benjamen Pyle

I’ve been thinking about this topic a lot lately when bringing EventBridge’s EventBus into some applications. On the current projects I’m working on with existing code, I’ve said 100 times, if EventBridge existed when I started them, I wouldn’t have so much SNS->SQS based code lying around. But such is life when working in evolving tech. Enter the EventBus Mesh

Serverless

Subscribe SNS to EventBridge Pipes

Posted on March 12, 2023by Benjamen Pyle

Legacy Serverless to New Serverless

I’ve been thinking and working hard on how I can start to introduce EventBridge and Pipes into some of my existing applications. Unfortunately, I have SNS in front of a lot of my service code and you can’t natively subscribe SNS to EventBridge Pipes. So I’ve started pondering this idea of how to integrate Legacy Serverless Applications into an ecosystem as new features are developed with more modern Serverless concepts. What I really want is a way to connect SNS to EventBridge Pipes.

Infrastructure

Canary Deployment for AWS Lambda

Posted on March 5, 2023by Benjamen Pyle

In life, when working on anything, small and iterative changes give us the best opportunity for feedback and learning. And it’s through that feedback and failure even that we get better. The same thing can be applied to building software. Small, iterative and independent deploys help us as builders understand if we’ve built the right thing and architected it correctly to handle the conditions asked of it. A technique called Canary Deployment is a popular model and the article below will demonstrate how to perform Canary Deployment for AWS Lambda

However, when deploying more frequently, we also need to do it safely. Shipping unfinished or potentially risky changes can have a big impact on our user base. No one wants to be in the middle of using your software only to be interrupted by a bad change. While we can’t be perfect in our ability to predict the impact or blast radius of a change, we can make it so that if the deploy shows signs of not being good, we can roll that change back without the need for human intervention.

Serverless

BatchGetItem with Golang

Posted on February 25, 2023by Benjamen Pyle

I haven’t had to use the Batch API a great deal over the past few years. When thinking more on it, it’s not that I have anything against the API, it is just that I never had a reason to work with it. However, over the past couple of months I saw that I’d used it twice in a project and with good success. My Golang and DynamoDB content has been doing well so I figured there might be some appetite for this one. And with all that said, I wrote this article highlighting how to use DynamoDB’s BatchGetItem with Golang.

Programming Uncategorized

Common AWS CLI commands and explanations

Posted on February 19, 2023by Benjamen Pyle

I tend to lose track of some of the commands or things I run often and by the time I think to script or alias something, I’ve long sense forgotten it. Then I end up running history | grep -i <some phrase> hoping that it’s in my history. The point of this article is just to document and capture some the common AWS CLI commands that I use pretty often.

Data

Parsing a Parquet file with Golang

Posted on February 19, 2023by Benjamen Pyle

I know it’s 2023, but you can’t get away from processing files. In a world of Events, APIs and Sockets, files still exist as a medium for moving data around. And a very common one at that. In recent years I’ve found myself dealing with Apache Parquet format files. And more specifically I often end up dealing with them coming out of AWS S3. If you are a consumer at all of the AWS DMS product when replicating, you will find out that parquet format is a great way to deal with your data as its designed for efficient storage and retrieval. There aren’t too many options for parsing a parquet file with Golang, but I’ve find a library I really enjoy and the article below will describe how to make the best use of it.

As always, here is the link to the Github Repository if you want to skip ahead

Serverless

Streaming DynamoDB to EventBridge Pipes

Posted on February 12, 2023by Benjamen Pyle

There is a real push and thought process around moving as much of your boilerplate code up into your serverless cloud components as possible. By streaming DynamoDB to EventBridge Pipes, you can move a large chunk of that boilerplate into the cloud. The thinking is that for things that really don’t differentiate your solution, why not let your cloud provider take care of that for you. Their integrations are well tested, highly scalable and highly available and can be more cost effective as you don’t waste CPU cycles on things like

Polling
Error handling
Data transformation
Filtering
Enrichment
Event management

All of those things “could” be done say in a container or in a Lambda but again, why pay the cycles, write all of this code over and over and over when you can push it up as configuration and as a part of your CDK or SAM code that handles the deployments

As usual, if you want to skip straight to a working sample, here’s the Github repository. Feel free to pull it and then run cdk deploy npx ts-node bin/app.ts and off you go.

Serverless

Extending and Customizing the JWT from Cognito via AWS Lambda using Go

Posted on February 4, 2023by Benjamen Pyle

I’ve been working a lot lately with Cognito and User Pools in AWS as I’ve been wanting to migrate and existing app into a serverless Identity and Access provider. The promise of Cognito is this “Implement secure, frictionless customer identity and access management that scales” – AWS

Honestly there are so many identity providers out there. This article won’t go into the alternatives and other options out there but will specifically touch upon something that I know was a big question for me when I started with Cognito which was, “how can I customize the private claims in a token?”. So let’s discuss that a little further

As usual, if you want to skip straight to code, feel free to jump over to the repository here

Data

Unmarshalling a DynamoDB Map into a Go Struct

Posted on February 1, 2023by Benjamen Pyle

Short post on unmarshalling a DynamoDB Map into something strongly typed like a Go struct. If you want to jump straight to some code, here are the Github gists