I was working recently with some backend code and I needed to communicate the success or failure of the result back to my UI. I instantly knew that I needed to put together a WebSocket to handle this interaction between the backend and the front end. With all the Serverless and non-Serverless options out there though, which way do I go? How about plain old WebSockets with AWS API Gateway and Serverless?
Tag: programming
CategoriesObservability
Analyzing and Correcting Errors with Advanced SQS Redrive
A good friend of mine is working on a really neat redrive tool with SQS and wanted to write an article to describe its purpose and use. I’m super honored that he asked me to share his writing on my blog. Please find below Adam Tran’s “Analyzing and Correcting Errors with Advanced SQS Redrive”
Analyzing and Correcting Errors with Advanced SQS Redrive
Analyzing dead-letter queues (DLQs) within the AWS ecosystem can be tricky. Receiving and analyzing messages via the AWS Console is very limited, and does not allow for the manipulation of messages in any sensible manner. Sure, you can redrive an entire DLQ, but what if you need to analyze thousands of messages or make changes?
There are many potential solutions to this problem, but a simple solution that I’ve developed is to download your queues’ messages locally where they can be analyzed with any tool of your choosing. I’ve defined a stateful directory structure to reflect where a message is in its journey of analysis so that you can make changes in whatever manner you deem appropriate.
DynamoDB Streams EventBridge Pipes Multiple Items
I’ve written a few articles lately on EventBridge Pipes and specifically around using them with DynamoDB Streams. I’ve written about Enrichment. And I’ve written about just straight Streaming. I believe that using EventBridge Pipes plays a nice part in a Serverless, Event-Driven approach. So in this article, I want to explore Streaming DynamoDB to EventBridge Pipes with multiple items in one table.
Several of the comments I received about Streaming DynamoDB to EventBridge Pipes were around, “What if I have multiple item collections in the same table?”. I intend to show a pattern for handling that exact problem in this article. At the bottom, you’ll find a working code sample that you can deploy and build on top of. I’ve used this exact setup in production, so rest assured that this is a great base to start from.
DynamoDB Streams EventBridge Pipes Enrichment
I’ve been wanting to spend more time lately talking about AWS HealthLake. And then more specifically, Fast Healthcare Interoperable Resources (FHIR) which is the foundation for interoperability in healthcare information systems. I believe very strongly that Serverless is for more than just client and user-driven workflows. I wrote extensively about it here but I wanted to take a deeper dive into building out streams of dataflows. I’ve been using this pattern for quite some time in production, so let’s have a look at EventBridge Pipes enriching DynamoDB Streams.
AWS Step Functions Callback Pattern
Some operations in a system function asynchronously. Many times, those same operations must also happen to be responsible for coordinating external workflows to provide an overall status on the execution of the main workflow. A natural fit for this problem with AWS is to use Step Functions and make use of the Callback pattern. In this article, I’m going to walk through an example of the Callback pattern while using AWS’ HealthLake and its export capabilities as the backbone for the async job. Welcome to the AWS Step Functions Callback Pattern.
Callback Workflow Solution Architecture
Let’s first start with the overarching architecture diagram. The general premise of the solution is that AWS’ HealthLake allows the export of all resources “since the last time”. By using Step Functions, Lambdas, SQS, DynamoDB, S3, Distributed Maps and EventBridge I’m going to build the ultimate Serverless Callback workflow. I feel like outside of Kinesis and SNS, I’ve touched them all in this one.
There’s quite a bit going on in here so I’m going to break it down into segments which will be:
- Triggering the State Machine
- Record Keeping and Run Status
- Running the Export and initiating the Callback
- Polling the Export and Restarting the State Machine
- Working the results
- Wrapping Up
- Dealing with Failure
Hang tight, there’s going to be a bunch of code and lots of detail. If you want to jump to code, it’s down at the bottom here
CategoriesServerless
Building Serverless Applications with AWS – Observability
Building Serverless applications can feel a bit overwhelming when you are first getting started. Sure, Event-Driven Systems have been around for many years but this notion of using managed services to “assemble” solutions vs a more traditional “plugin” style architecture might throw you for a loop. Continuing in the series of Building Serverless Applications with AWS, let’s have a look at the “Debugging and Troubleshooting” aspect.
Series Topics
- Data Storage Choices
- Building the Application (Fargate/Containers vs Lambda)
- Handling Events
- Exposing the API (if there is one)
- Securing it all, including the API
- Debugging and Troubleshooting in Production
CategoriesServerless
Building Serverless Applications with AWS – Security
Building Serverless applications can feel a bit overwhelming when you are first getting started. Sure, Event-Driven Systems have been around for many years but this notion of using managed services to “assemble” solutions vs a more traditional “plugin” style architecture might throw you for a loop. Continuing in the series of Building Serverless Applications with AWS, let’s have a look at the “Security” aspect.
Series Topics
- Data Storage Choices
- Building the Application (Fargate/Containers vs Lambda)
- Handling Events
- Exposing the API (if there is one)
- Securing it all, including the API
- Debugging and Troubleshooting in Production
CategoriesServerless
Building Serverless Applications with AWS – API
Building Serverless applications can feel a bit overwhelming when you are first getting started. Sure, Event-Driven Systems have been around for many years but this notion of using managed services to “assemble” solutions vs a more traditional “plugin” style architecture might throw you for a loop. Continuing in the series of Building Serverless Applications with AWS, let’s have a look at the “API” aspect.
Series Topics
- Data Storage Choices
- Building the Application (Fargate/Containers vs Lambda)
- Handling Events
- Exposing the API (if there is one)
- Securing it all, including the API
- Debugging and Troubleshooting in Production
AWS Step Function Versions and Aliases
Up until last week, when you deployed a new version of your State Machine in AWS Step Functions, the old version was gone and the ability to test or rollback was limited by your ability to re-push a previous commit. However, AWS has rolled out Step Function Versions and Aliases so that you can accomplish just those tasks. Creating a unique combination of a version and ASL gives you the ability to use things like Deployment Preferences to accomplish Canary or Linear-type deployments. In the below article, I’m going to walk you through Step Function Versions and Aliases.
Disclaimer
First up, I’m going to be using SAM to build the infrastructure. I think this is the first SAM-based deployment article I’ve written. I know this makes Allen Helton super happy. Second, I know the article from AWS says support for SAM and CDK, but they haven’t rolled this in as of the writing of this article. However, I’m using the SAM Nightly Builds and it does include the Transforms to make this happen. I could spend another few articles describing SAM and perhaps I will dig deeper later, but for now, here’s the AWS Docs on transforms.
CategoriesServerless
Lambda Extension with Golang
For full disclosure, I’ve been writing Lambda function code since 2017 and I completely breezed over the release of Lambda Extensions back in 2020. Here’s the release announcement. At the core of extensions, you have internal and external options. For the balance of this article, I’m going to focus on building a Lambda extension with Golang and lean into the external style approach.
Extensions and Why
Taking a quick step back, why extensions? From an architect level of thinking, extensions give me the ability to have cross-team reuse of code without being tied to a particular language or build process. For something like Node or Python, you could use a standard Layer to package your Lambda reuse. But for something like Golang, where your code is packaged at build time and not run-time, then you sort of have to look at the shared library. I wrote about that here. But what if you wanted to create some shared functionality that was usable regardless of which language you built your Lamabda in? That seems to have some serious appeal for my current projects where teams are using different stacks to build their APIs due to need and comfort.