CategoriesData

Parsing a Parquet file with Golang

Posted on February 19, 2023by Benjamen Pyle

I know it’s 2023, but you can’t get away from processing files. In a world of Events, APIs and Sockets, files still exist as a medium for moving data around. And a very common one at that. In recent years I’ve found myself dealing with Apache Parquet format files. And more specifically I often end up dealing with them coming out of AWS S3. If you are a consumer at all of the AWS DMS product when replicating, you will find out that parquet format is a great way to deal with your data as its designed for efficient storage and retrieval. There aren’t too many options for parsing a parquet file with Golang, but I’ve find a library I really enjoy and the article below will describe how to make the best use of it.

As always, here is the link to the Github Repository if you want to skip ahead

Continue reading “Parsing a Parquet file with Golang”
CategoriesServerless

Streaming DynamoDB to EventBridge Pipes

Posted on February 12, 2023by Benjamen Pyle

There is a real push and thought process around moving as much of your boilerplate code up into your serverless cloud components as possible. By streaming DynamoDB to EventBridge Pipes, you can move a large chunk of that boilerplate into the cloud. The thinking is that for things that really don’t differentiate your solution, why not let your cloud provider take care of that for you. Their integrations are well tested, highly scalable and highly available and can be more cost effective as you don’t waste CPU cycles on things like

  • Polling
  • Error handling
  • Data transformation
  • Filtering
  • Enrichment
  • Event management

All of those things “could” be done say in a container or in a Lambda but again, why pay the cycles, write all of this code over and over and over when you can push it up as configuration and as a part of your CDK or SAM code that handles the deployments

As usual, if you want to skip straight to a working sample, here’s the Github repository. Feel free to pull it and then run cdk deploy npx ts-node bin/app.ts and off you go.

Continue reading “Streaming DynamoDB to EventBridge Pipes”
CategoriesServerless

Extending and Customizing the JWT from Cognito via AWS Lambda using Go

Posted on February 4, 2023by Benjamen Pyle

I’ve been working a lot lately with Cognito and User Pools in AWS as I’ve been wanting to migrate and existing app into a serverless Identity and Access provider. The promise of Cognito is this “Implement secure, frictionless customer identity and access management that scales” – AWS

Honestly there are so many identity providers out there. This article won’t go into the alternatives and other options out there but will specifically touch upon something that I know was a big question for me when I started with Cognito which was, “how can I customize the private claims in a token?”. So let’s discuss that a little further

As usual, if you want to skip straight to code, feel free to jump over to the repository here

Continue reading “Extending and Customizing the JWT from Cognito via AWS Lambda using Go”
CategoriesServerless

Handling Change with AWS Healthlake

Posted on January 30, 2023by Benjamen Pyle

One of the features that I am currently missing with AWS Healthlake is a proper “event-ing” framework. With DynamoDB you’ve got streams. With RDS you can use DMS. But with Healthlake there is no native change data capture mechanism.

Being that I’m only working on event driven architectures these days, I needed a way to be able to handle change. What I’m going to show you below is not “sanctioned” but it is 100% AWS native and continues with the Serverless theme. With that said, here’s the Github Repository if you just want to jump ahead. The CDK code will deploy a Healthlake instance @ $.27 / hr so please run cdk destroy npx ts-node bin/app.ts` when you are done

Continue reading “Handling Change with AWS Healthlake”