Creating a multi-environment AWS solution using an API Gateway, serverless application layer, and deployment via an AWS Code Pipeline
Challenge
The Chartered Institute of Public Finance Accountants (CIPFA) have a program of work to extract better data analytics via a third-party analytics application. As part of the program they needed to be able to access data stored in their CRM application, the existing method for programmatic data access was unsuitable for public access via web services.
Through investigation and evaluation, CIPFA determined a cloud-based target architecture with a service layer that utilised AWS’s API Gateway and Lambda serverless technology: this would allow the analytics application to access the CRM data in real-time without requiring complex fixed integrations.
This approach exploits the advanced scaling, security, and Infrastructure as Code capabilities of the AWS platform, enabling rapid iterations of new applications via Continuous Integration/Continuous Deployment (CI/CD).
CIPFA determined working with an AWS partner would rapidly fulfil the requirements while meeting program delivery dates. CIPFA engaged with CirrusHQ to build the AWS infrastructure
Solution
The solution delivered by CirrusHQ worked with CIPFA to deliver a SOAP (Simple Object Access Protocol) to REST API proxy on AWS that would provide security and scale when accessing core data.
CirrusHQ implemented a multi-environment AWS account structure that makes use of the API Gateway and connects to Lambda within a Virtual Private Cloud environment. This then connected back to CIPFA’s data centre via multiple resilient VPN. By adopting a best practice ‘as a service’ offering from AWS, CIPFA enjoyed the benefits of superior performance and reliability and low costs across all the environments utilising the workload. From a network perspective the solution is entirely privately isolated to the CIPFA network, the only public facing element in the API gateway, which runs as a full proxy.
The solution was delivered via an agile methodology, enabling rapid delivery of the project that met CIPFA’s timelines. In addition, the workload can be extended and aligned with the other projects needing to use the interfaces.
For deployment, CirrusHQ created a mutable pipeline builder to create a pipeline per environment in order to establish a reusable and repeatable deployment process. The process is triggered from GitHub, this triggers the AWS CodePipeline service that automates the end to end process, building the code and deploying it to the target environment. The CI/CD pipeline enables all necessary changes to be delivered rapidly and greatly improves their ability to respond quickly to any defects whilst also providing in-built disaster recovery capabilities. Steps within the pipeline additionally include schema validation and performance service level compliance testing using the artillery framework, leading to a high level of confidence when moving code into production.
For monitoring and supporting the service, all logs from API gateway and Lambda are collated within CloudWatch logs. Additional telemetry about the application is collected at multiple levels and sent to AWS x-ray (below diagram shows a view of x-ray on the Lambda functions).
Results
CirrusHQ were able through the utilisation of AWS best practice and services deliver a secure, performant, and highly scalable API service. The multi-environment workload benefits from costs as it utilises serverless functions and services managed by AWS.
The workload utilises a secure public API gateway means that the solution is both highly resilient to attacks, and ensures governance and scale for the API calls. In addition the application layer is stateless and complies with data governance requirements. As a hosted AWS solution, CIPFA is provided with detailed SLAs that guarantee availability as well as processes built into the infrastructure to ensure recoverability.
The service layer now has multiple replicable environments to enable various isolated stages of testing structure using serverless and cloud-native tooling which allow for hyper scale to extend the architecture further. As this was CIPFA’s first AWS project, it has also established good principles around management, governance, and deployment that can be applied to future AWS workloads. The methodology using CI/CD pipeline all enable best practice and ensure services can be added quickly and innovated incrementally.
By implementing the solution, CIPFA has decreased complexity and increased security which provides for improved agility and adoption of their dataset by third parties, therefore reducing time-to-market for their customer-facing products.
To find out more about CIPFA please click here
Tooling (Tools and Technologies)
Developer tools: Github, AWS CodeBuild, AWS CodePipeline, AWS X-Ray
Management and Governance: Amazon CloudWatch, AWS CloudFormation, AWS CloudTrail, AWS Config, AWS Systems Manager
Security: AWS Identity & Access Management, AWS Key Management Service, AWS Secrets Manager
Application Integration: Amazon Simple Queue Service, Amazon Simple Notification Service
Compute: AWS Lambda
Networking and Content Delivery: Amazon VPC, Amazon API Gateway
