I haven’t had to use the Batch API a great deal over the past few years. When thinking more on it, it’s not that I have anything against the API, it is just that I never had a reason to work with it. However, over the past couple of months I saw that I’d used it twice in a project and with good success. My Golang and DynamoDB content has been doing well so I figured there might be some appetite for this one. And with all that said, I wrote this article highlighting how to use DynamoDB’s BatchGetItem with Golang.
Tag: programming
Common AWS CLI commands and explanations
I tend to lose track of some of the commands or things I run often and by the time I think to script or alias something, I’ve long sense forgotten it. Then I end up running history | grep -i <some phrase
> hoping that it’s in my history. The point of this article is just to document and capture some the common AWS CLI commands that I use pretty often.
Parsing a Parquet file with Golang
I know it’s 2023, but you can’t get away from processing files. In a world of Events, APIs and Sockets, files still exist as a medium for moving data around. And a very common one at that. In recent years I’ve found myself dealing with Apache Parquet format files. And more specifically I often end up dealing with them coming out of AWS S3. If you are a consumer at all of the AWS DMS product when replicating, you will find out that parquet format is a great way to deal with your data as its designed for efficient storage and retrieval. There aren’t too many options for parsing a parquet file with Golang, but I’ve find a library I really enjoy and the article below will describe how to make the best use of it.
As always, here is the link to the Github Repository if you want to skip ahead
Streaming DynamoDB to EventBridge Pipes
There is a real push and thought process around moving as much of your boilerplate code up into your serverless cloud components as possible. By streaming DynamoDB to EventBridge Pipes, you can move a large chunk of that boilerplate into the cloud. The thinking is that for things that really don’t differentiate your solution, why not let your cloud provider take care of that for you. Their integrations are well tested, highly scalable and highly available and can be more cost effective as you don’t waste CPU cycles on things like
- Polling
- Error handling
- Data transformation
- Filtering
- Enrichment
- Event management
All of those things “could” be done say in a container or in a Lambda but again, why pay the cycles, write all of this code over and over and over when you can push it up as configuration and as a part of your CDK or SAM code that handles the deployments
As usual, if you want to skip straight to a working sample, here’s the Github repository. Feel free to pull it and then run cdk deploy npx ts-node bin/app.ts
and off you go.
Extending and Customizing the JWT from Cognito via AWS Lambda using Go
I’ve been working a lot lately with Cognito and User Pools in AWS as I’ve been wanting to migrate and existing app into a serverless Identity and Access provider. The promise of Cognito is this “Implement secure, frictionless customer identity and access management that scales” – AWS
Honestly there are so many identity providers out there. This article won’t go into the alternatives and other options out there but will specifically touch upon something that I know was a big question for me when I started with Cognito which was, “how can I customize the private claims in a token?”. So let’s discuss that a little further
As usual, if you want to skip straight to code, feel free to jump over to the repository here
Unmarshalling a DynamoDB Map into a Go Struct
Short post on unmarshalling a DynamoDB Map into something strongly typed like a Go struct. If you want to jump straight to some code, here are the Github gists
Mapping AWS State Machine output to API Gateway response with VTL
This is a continuation of a previous article I wrote regarding zero code workflows creating Cognito users with Step Functions. Part of using State Machines with API Gateway is the dealing with the response and potentially VTL mapping
Goals of this article are to document some of the tips and things that I picked up along the way.
Creating an AWS Cognito User with an Auto-Incrementing ID using AWS Step Functions
So there are a couple of interesting topics in here.
- I’ve been really leaning into code-less workflows with AWS Step Functions and this State Machine has nothing but native SDK integrations which include
- DynamoDB (Put, Delete, Get)
- Cognito/User Pools (AdminCreateUser)
- I’ve run into some legacy code that requires a Username to be a bigint and I don’t want to use an RDBMS so I’m using DynamoDB to generate one for me while also being “race condition” proof
As always, if you want to jump straight to the code, here is the Github repository
Querying AWS Healthlake from Go
When working with Healthcare data when of the things that’s often mentioned or discussed is “Is your data interoperable?” As a developer and an architect, that’s a really loaded word to me because if I can expose my data over files, APIs or some consistent channel like TCP, then by definition my system is interoperable. Per my Mac dictionary “interoperable” is an adjective defined like this :: (of computer systems or software) able to exchange and make use of information ::
However where things get a little more nuanced is when the definition includes some common healthcare specific formats and more specifically HL7’s FHIR. So when you have this problem, there are certain tools that you need to use. There are several opensource solutions that you could select but when you are an AWS’ customer, you start with AWS first. And they just so happy to have a set of capabilities wrapped around a product called Healthlake.
Handling “Poison Pill” Messages with AWS Kinesis and Lambdas
Queues and streams are fundamentally different in how they handle readers consuming their information.
With an SQS Queue you can have many consumers but generally one consumer will win reading the message and in the event of success the message is purged from the queue or upon failure that message is returned back to the queue. It technically doesn’t get deleted, yet the its visibility property is changed. Hence why the VisibilityTimeout on the queue matters. If your code processes messages in more time than that property then you are going to get messages that constantly get put back on the queue for retry.