A company is running a custom-built application that processes records. All the components run on Amazon EC2 instances that run in an Auto Scaling group. Each record's processing is a multistep sequential action that is compute-intensive. Each step is always completed in 5 minutes or less.
A limitation of the current system is that if any steps fail, the application has to reprocess the record from the beginning The company wants to update the architecture so that the application must reprocess only the failed steps.
What is the MOST operationally efficient solution that meets these requirements?
* Use AWS Step Functions to Orchestrate Processing:
AWS Step Functions allow you to build distributed applications by combining AWS Lambda functions or other AWS services into workflows.
Decoupling the processing into Step Functions tasks enables you to retry individual steps without reprocessing the entire record.
* Architectural Steps:
Create a web application to pass records to AWS Step Functions:
The web application can be a simple frontend that receives input and triggers the Step Functions workflow.
Define a Step Functions state machine:
Each step in the state machine represents a processing stage. If a step fails, Step Functions can retry the step based on defined conditions.
Use AWS Lambda functions:
Lambda functions can be used to handle each processing step. These functions can be stateless and handle specific tasks, reducing the complexity of error handling and reprocessing logic.
* Operational Efficiency:
Using Step Functions and Lambda improves operational efficiency by providing built-in error handling, retries, and state management.
This architecture scales automatically and isolates failures to individual steps, ensuring only failed steps are retried.
References:
AWS Step Functions
Building Workflows with Step Functions



