CategoriesServerless

BatchGetItem with Golang

I haven’t had to use the Batch API a great deal over the past few years. When thinking more on it, it’s not that I have anything against the API, it is just that I never had a reason to work with it. However, over the past couple of months I saw that I’d used it twice in a project and with good success. My Golang and DynamoDB content has been doing well so I figured there might be some appetite for this one. And with all that said, I wrote this article highlighting how to use DynamoDB’s BatchGetItem with Golang.

CategoriesProgrammingUncategorized

Common AWS CLI commands and explanations

I tend to lose track of some of the commands or things I run often and by the time I think to script or alias something, I’ve long sense forgotten it. Then I end up running history | grep -i <some phrase> hoping that it’s in my history. The point of this article is just to document and capture some the common AWS CLI commands that I use pretty often.

CategoriesData

Parsing a Parquet file with Golang

I know it’s 2023, but you can’t get away from processing files. In a world of Events, APIs and Sockets, files still exist as a medium for moving data around. And a very common one at that. In recent years I’ve found myself dealing with Apache Parquet format files. And more specifically I often end up dealing with them coming out of AWS S3. If you are a consumer at all of the AWS DMS product when replicating, you will find out that parquet format is a great way to deal with your data as its designed for efficient storage and retrieval. There aren’t too many options for parsing a parquet file with Golang, but I’ve find a library I really enjoy and the article below will describe how to make the best use of it.

As always, here is the link to the Github Repository if you want to skip ahead

CategoriesServerless

Streaming DynamoDB to EventBridge Pipes

There is a real push and thought process around moving as much of your boilerplate code up into your serverless cloud components as possible. By streaming DynamoDB to EventBridge Pipes, you can move a large chunk of that boilerplate into the cloud. The thinking is that for things that really don’t differentiate your solution, why not let your cloud provider take care of that for you. Their integrations are well tested, highly scalable and highly available and can be more cost effective as you don’t waste CPU cycles on things like

  • Polling
  • Error handling
  • Data transformation
  • Filtering
  • Enrichment
  • Event management

All of those things “could” be done say in a container or in a Lambda but again, why pay the cycles, write all of this code over and over and over when you can push it up as configuration and as a part of your CDK or SAM code that handles the deployments

As usual, if you want to skip straight to a working sample, here’s the Github repository. Feel free to pull it and then run cdk deploy npx ts-node bin/app.ts and off you go.

CategoriesServerless

Extending and Customizing the JWT from Cognito via AWS Lambda using Go

I’ve been working a lot lately with Cognito and User Pools in AWS as I’ve been wanting to migrate and existing app into a serverless Identity and Access provider. The promise of Cognito is this “Implement secure, frictionless customer identity and access management that scales” – AWS

Honestly there are so many identity providers out there. This article won’t go into the alternatives and other options out there but will specifically touch upon something that I know was a big question for me when I started with Cognito which was, “how can I customize the private claims in a token?”. So let’s discuss that a little further

As usual, if you want to skip straight to code, feel free to jump over to the repository here

CategoriesServerless

Mapping AWS State Machine output to API Gateway response with VTL

This is a continuation of a previous article I wrote regarding zero code workflows creating Cognito users with Step Functions. Part of using State Machines with API Gateway is the dealing with the response and potentially VTL mapping

Goals of this article are to document some of the tips and things that I picked up along the way.

CategoriesServerless

Creating an AWS Cognito User with an Auto-Incrementing ID using AWS Step Functions

So there are a couple of interesting topics in here.

  1. I’ve been really leaning into code-less workflows with AWS Step Functions and this State Machine has nothing but native SDK integrations which include
    • DynamoDB (Put, Delete, Get)
    • Cognito/User Pools (AdminCreateUser)
  2. I’ve run into some legacy code that requires a Username to be a bigint and I don’t want to use an RDBMS so I’m using DynamoDB to generate one for me while also being “race condition” proof

As always, if you want to jump straight to the code, here is the Github repository

CategoriesData

Querying AWS Healthlake from Go

When working with Healthcare data when of the things that’s often mentioned or discussed is “Is your data interoperable?” As a developer and an architect, that’s a really loaded word to me because if I can expose my data over files, APIs or some consistent channel like TCP, then by definition my system is interoperable. Per my Mac dictionary “interoperable” is an adjective defined like this :: (of computer systems or software) able to exchange and make use of information ::

However where things get a little more nuanced is when the definition includes some common healthcare specific formats and more specifically HL7’s FHIR. So when you have this problem, there are certain tools that you need to use. There are several opensource solutions that you could select but when you are an AWS’ customer, you start with AWS first. And they just so happy to have a set of capabilities wrapped around a product called Healthlake.

CategoriesInfrastructure

Handling “Poison Pill” Messages with AWS Kinesis and Lambdas

Queues and streams are fundamentally different in how they handle readers consuming their information.

With an SQS Queue you can have many consumers but generally one consumer will win reading the message and in the event of success the message is purged from the queue or upon failure that message is returned back to the queue. It technically doesn’t get deleted, yet the its visibility property is changed. Hence why the VisibilityTimeout on the queue matters. If your code processes messages in more time than that property then you are going to get messages that constantly get put back on the queue for retry.