Tag: aws

CategoriesData

AWS HealthLake Exports

Posted on August 26, 2023by Benjamen Pyle

In my previous article I wrote about a Callback Pattern with AWS Step Functions built upon the backbone of HealthLake’s export. As much as I went deep with code on the Callback portion, I felt that I didn’t give the HealthLake side of the equation enough run. So this article is that adjustment. Managing exports with AWS HealthLake.

What is HealthLake

AWS HealthLake is a HIPAA-eligible service that provides FHIR APIs that help healthcare and life sciences companies securely store, transform, transact, and analyze health data in minutes to give a chronological view at the patient and population-level. – AWS

My words on that are that HealthLake is a FHIR-compliant database that gives a developer a robust set of APIs to build patient-centered applications. You can use HealthLake for building transactional applications, analyze large volumes of data, store structured and semi-structured information and build analytics and reports.

When building with HealthLake I find it fits in one of two places.

  1. As the transactional center for your Healthcare application. It is highly patient centered, very scalable and contains APIs for working with each resource. In addition, it provides SMART on FHIR capabilities that make it nice choice for building an application on top of.
  2. As the aggregation point for many external and internal systems in a LakeHouse style architecture for interopability and reporting. When you’ve got a distributed system with various datababases and you need your data reunited in one location. HealthLake does that. Or if you are pulling in data from various external sources, HealthLake can do that too. I wrote about doing this with Serverless a while back.
Continue reading “AWS HealthLake Exports”
CategoriesDataProgrammingServerless

AWS Step Functions Callback Pattern

Posted on August 24, 2023by Benjamen Pyle

Some operations in a system function asynchronously. Many times, those same operations must also happen to be responsible for coordinating external workflows to provide an overall status on the execution of the main workflow. A natural fit for this problem with AWS is to use Step Functions and make use of the Callback pattern. In this article, I’m going to walk through an example of the Callback pattern while using AWS’ HealthLake and its export capabilities as the backbone for the async job. Welcome to the AWS Step Functions Callback Pattern.

Callback Workflow Solution Architecture

Let’s first start with the overarching architecture diagram. The general premise of the solution is that AWS’ HealthLake allows the export of all resources “since the last time”. By using Step Functions, Lambdas, SQS, DynamoDB, S3, Distributed Maps and EventBridge I’m going to build the ultimate Serverless Callback workflow. I feel like outside of Kinesis and SNS, I’ve touched them all in this one.

AWS Step Functions Callback Pattern Architecture

There’s quite a bit going on in here so I’m going to break it down into segments which will be:

  1. Triggering the State Machine
  2. Record Keeping and Run Status
  3. Running the Export and initiating the Callback
  4. Polling the Export and Restarting the State Machine
  5. Working the results
  6. Wrapping Up
  7. Dealing with Failure

Hang tight, there’s going to be a bunch of code and lots of detail. If you want to jump to code, it’s down at the bottom here

Continue reading “AWS Step Functions Callback Pattern”
CategoriesServerless

Building Serverless Applications with AWS – Observability

Posted on August 19, 2023by Benjamen Pyle

Building Serverless applications can feel a bit overwhelming when you are first getting started. Sure, Event-Driven Systems have been around for many years but this notion of using managed services to “assemble” solutions vs a more traditional “plugin” style architecture might throw you for a loop. Continuing in the series of Building Serverless Applications with AWS, let’s have a look at the “Debugging and Troubleshooting” aspect.

Series Topics

  1. Data Storage Choices
  2. Building the Application (Fargate/Containers vs Lambda)
  3. Handling Events
  4. Exposing the API (if there is one)
  5. Securing it all, including the API
  6. Debugging and Troubleshooting in Production
Continue reading “Building Serverless Applications with AWS – Observability”
CategoriesServerless

Building Serverless Applications with AWS – Security

Posted on August 12, 2023by Benjamen Pyle

Building Serverless applications can feel a bit overwhelming when you are first getting started. Sure, Event-Driven Systems have been around for many years but this notion of using managed services to “assemble” solutions vs a more traditional “plugin” style architecture might throw you for a loop. Continuing in the series of Building Serverless Applications with AWS, let’s have a look at the “Security” aspect.

Series Topics

  1. Data Storage Choices
  2. Building the Application (Fargate/Containers vs Lambda)
  3. Handling Events
  4. Exposing the API (if there is one)
  5. Securing it all, including the API
  6. Debugging and Troubleshooting in Production
Continue reading “Building Serverless Applications with AWS – Security”
CategoriesServerless

Building Serverless Applications with AWS – API

Posted on August 5, 2023by Benjamen Pyle

Building Serverless applications can feel a bit overwhelming when you are first getting started. Sure, Event-Driven Systems have been around for many years but this notion of using managed services to “assemble” solutions vs a more traditional “plugin” style architecture might throw you for a loop. Continuing in the series of Building Serverless Applications with AWS, let’s have a look at the “API” aspect.

Series Topics

  1. Data Storage Choices
  2. Building the Application (Fargate/Containers vs Lambda)
  3. Handling Events
  4. Exposing the API (if there is one)
  5. Securing it all, including the API
  6. Debugging and Troubleshooting in Production
Continue reading “Building Serverless Applications with AWS – API”
CategoriesServerless

Building Serverless Applications with AWS – Handling Events

Posted on July 27, 2023by Benjamen Pyle

Building Serverless applications can feel a bit overwhelming when you are first getting started. Sure, Event-Driven Systems have been around for many years but this notion of using managed services to “assemble” solutions vs a more traditional “plugin” style architecture might throw you for a loop. Continuing in the series of Building Serverless Applications with AWS, let’s have a look at the “Handling Events” aspect.

Series Topics

  1. Data Storage Choices
  2. Building the Application (Fargate/Containers vs Lambda)
  3. Handling Events
  4. Exposing the API (if there is one)
  5. Securing it all, including the API
  6. Debugging and Troubleshooting in Production
Continue reading “Building Serverless Applications with AWS – Handling Events”
CategoriesServerless

Building Serverless Applications with AWS – Compute

Posted on July 16, 2023by Benjamen Pyle

Building Serverless applications can feel a bit overwhelming when you are first getting started. Sure, Event-Driven Systems have been around for many years but this notion of using managed services to “assemble” solutions vs a more traditional “plugin” style architecture might throw you for a loop. Continuing in the series of Building Serverless Applications with AWS, let’s have a look at the “Compute” aspect.

Series Topics

  1. Data Storage Choices
  2. Building the Application (Fargate/Containers vs Lambda)
  3. Handling Events
  4. Exposing the API (if there is one)
  5. Securing it all, including the API
  6. Debugging and Troubleshooting in Production
Continue reading “Building Serverless Applications with AWS – Compute”
CategoriesDataServerless

Building Serverless Applications with AWS – Data

Posted on July 10, 2023by Benjamen Pyle

Building Serverless applications can feel a bit overwhelming when you are first getting started. Sure, Event-Driven Systems have been around for many years but this notion of using managed services to “assemble” solutions vs a more traditional “plugin” style architecture might throw you for a loop. I haven’t created a series yet, so this is my first attempt at that. My goal is to walk you through the design considerations when Building Serverless Applications with AWS.

Series Topics

  1. Data Storage Choices
  2. Building the Application (Fargate/Containers vs Lambda)
  3. Handling Events
  4. Exposing the API (if there is one)
  5. Securing it all, including the API
  6. Debugging and Troubleshooting in Production

This is an ambitious list, but when I think about what it takes to put together a Serverless application, these are the concepts and decisions that I often end up counseling or guiding developers new to the paradigm. So let’s dig in.

Continue reading “Building Serverless Applications with AWS – Data”
CategoriesInfrastructureServerless

AWS Step Function Versions and Aliases

Posted on June 27, 2023by Benjamen Pyle

Up until last week, when you deployed a new version of your State Machine in AWS Step Functions, the old version was gone and the ability to test or rollback was limited by your ability to re-push a previous commit. However, AWS has rolled out Step Function Versions and Aliases so that you can accomplish just those tasks. Creating a unique combination of a version and ASL gives you the ability to use things like Deployment Preferences to accomplish Canary or Linear-type deployments. In the below article, I’m going to walk you through Step Function Versions and Aliases.

Disclaimer

First up, I’m going to be using SAM to build the infrastructure. I think this is the first SAM-based deployment article I’ve written. I know this makes Allen Helton super happy. Second, I know the article from AWS says support for SAM and CDK, but they haven’t rolled this in as of the writing of this article. However, I’m using the SAM Nightly Builds and it does include the Transforms to make this happen. I could spend another few articles describing SAM and perhaps I will dig deeper later, but for now, here’s the AWS Docs on transforms.

Continue reading “AWS Step Function Versions and Aliases”
CategoriesServerless

Lambda Extension with Golang

Posted on June 18, 2023by Benjamen Pyle

For full disclosure, I’ve been writing Lambda function code since 2017 and I completely breezed over the release of Lambda Extensions back in 2020. Here’s the release announcement. At the core of extensions, you have internal and external options. For the balance of this article, I’m going to focus on building a Lambda extension with Golang and lean into the external style approach.

Extensions and Why

Taking a quick step back, why extensions? From an architect level of thinking, extensions give me the ability to have cross-team reuse of code without being tied to a particular language or build process. For something like Node or Python, you could use a standard Layer to package your Lambda reuse. But for something like Golang, where your code is packaged at build time and not run-time, then you sort of have to look at the shared library. I wrote about that here. But what if you wanted to create some shared functionality that was usable regardless of which language you built your Lamabda in? That seems to have some serious appeal for my current projects where teams are using different stacks to build their APIs due to need and comfort.

Continue reading “Lambda Extension with Golang”