Oxford University’s Gardens, Libraries and Museums (GLAM) engaged CirrusHQ to create an AWS-hosted solution with a fully automated Continuous Integration / Continuous Delivery (CI/CD) pipeline
The University of Oxford has four museums containing some of the world’s most significant collections. Not only are these important for scholarly research, they also represent the front door to the wealth of knowledge and research curated by the University. The goal of the University’s museums and IT Services team was to make the detailed data it holds for these collections accessible – and searchable – online from a single location.
One of the key technology challenges related to this was the creation of an infrastructure able to respond, perform, and deliver accurate search results from the data sets. In the process of bringing the University’s collections under one digital ‘roof’, the in-house development team would be onboarding new data sets on an ongoing basis and they wanted to build up a set of functions able to interrogate existing data from the same single search tool as they did so.
The solution CirrusHQ provided for them utilises managed services by AWS to maximise performance and reliability and features a fully automated Continuous Integration / Continuous Delivery (CI/CD) pipeline. This enabled the development team to initiate builds across the environments, with suitable governance controls and points for testing these updates.
Business Challenge / Problem Detail
GLAM and IT Services wanted each museum’s collection to be viewed as a discrete entity, but they also planned to create a single mechanism facilitating the online search and discovery of items across the University’s collections. The solution would be a key resource allowing scholars, students and the general public to interrogate these world-famous collections online and it was important to ensure prompt (and appropriate) responses from the searches.
The team at GLAM and IT Services have the in-house skills and resources to build the solution’s core functionality, taking the API calls from the website, gathering the data, and sending the responses and any assets associated with the request. However, they recognised the advantages of deploying this application on AWS’s managed cloud services and taking advantage of its serverless technology.
The team recognised that the solution needed to be secure and robust, with capabilities for backup and restoration. In addition, the team required continuous integration and deployment of the solution that could be managed internally. Finally, they wanted to ensure that the solution provided GLAM and IT Services with a blueprint for best practice and was cost-effective.
The AWS solution that CirrusHQ provided was created using AWS best practices and utilising AWS Well Architected Framework.
CirrusHQ built the AWS infrastructure that is shown above. CirrusHQ ensured that all aspects of the infrastructure could be controlled via code using AWS CloudFormation. All source code for both application and infrastructure is managed within the GLAM and IT Services GitHub account.
The diagram above shows, at a logical level, the services used and all application level calls to and from the solution. The use of managed services provided by AWS for this solution ensures best practice, high availability, scale, resilience, and maintainability of each component.
One of the key deliverables of the project was to ensure that the development team had the ability to control integration and delivery of new features to the targeted environment and accelerate delivering new features and improvements to the system. GLAM and IT Services use GitHub to manage their code; but in addition, they wanted to have the ability to include additional governance when releasing to controlled environments (production). CirrusHQ used AWS CodePipeline as a tool to implement CI/CD. This included creating a pipeline builder for each pipeline for the targeted environment so as to make the CI/CD pipelines themselves a repeatable and immutable part of the solution as well.
To deliver the new workload on AWS, CirrusHQ worked with various members of the University of Oxford team, including developers, infrastructure architects, project management, IT support, business analysts and business representatives. They built out the AWS infrastructure environment in a repeatable format, creating an automated process allowing the team to build and deploy on AWS regularly on a continual integration and delivery basis.
Overall, the solution not only enables high performance and resilience from the adoption of resources that are managed by AWS, but it is also cost-effective. As all the infrastructure is in code, it is repeatable, easy to update, and has a delivery pipeline that is able to be managed by the in-house development team to ensure continued updates are quick, and do not constrain their development or deployment processes.
The solution has enabled the GLAM and IT service team to have an AWS blueprint for best practice, demonstrating use of serverless, code pipelines and services managed by AWS to gain the most from current practices within the public cloud.
Please visit the Oxford University Museum of Natural History website to see the new collections online search functionality in action.
Tooling (Tools and Technologies)
Developer tools: GitHub, AWS CodeBuild, AWS CodePipeline, AWS X-Ray
Management and Governance: Amazon CloudWatch, AWS CloudFormation, AWS CloudTrail, AWS Config, AWS Systems Manager
Security: AWS Identity & Access Management, AWS Key Management Service, AWS WAF
Application Integration: Amazon Simple Queue Service, Amazon Simple Notification Service
Compute: AWS Lambda, Amazon EC2
Database: Amazon ElastiCache
Storage: Amazon Simple Storage Solution, Amazon Elastic Block Store
Networking and Content Delivery: Amazon VPC, Amazon API Gateway, Amazon Route53
Quotes from the University of Oxford
Professor Paul Smith
Director of Oxford University Museum of Natural History
“We have seven million specimens, and we have a fraction of those available digitally. And like many natural science collections, we have about 10% of our material digitised at some level or other on a much smaller percentage available online.
Online collections enables our researchers to assess whether the specimen that they’re looking for is the right one – rather having to travel to Oxford to see these or transport it to the research institution concerned”
Divisional Secretary and Chief Operating Officer
“A key objective of GLAM is to share the University’s collections to the public and researchers all around the world. The diversity of collections within GLAM enables connectivity across these different collections which brings additional benefits and insights.
Digitisation provides us with a great opportunity to open our doors virtually to the world and this is the first step in that journey to making the collections accessible to the world. There is also potential for financial gains through this channel by engaging with donors or developing a picture library.”
Dr. Sarah Joomun
Digital Collections Manager
“Previously; it was almost impossible to find information about our collections and what we actually had in the museum. So this project was the chance for us to actually set up a single search for all our collections and a single place to look for collections data, display this and add additional information the collections.
For the first time, we have got a really good search service with images. You can search across all the collections. Previously, if you want to find out about everything we have from a particular country; you had to search about 20 times now you have just search in one place and all the data comes up at once. Now you can search across our fossils and insects at the same time.
Our visitor have started sending enquiries to museum which are getting more specific which is indicative of the fact that visitors will have looked on the Collections Online, being able to find information online and then asking questions based on that.”
Systems Architect and Network Manager – Gardens and Museum (GLAM) IT.
“The level of expertise in solution design and support provided by Cirrus HQ has been a formative experience with this project for Garden and Museums IT. Having a simple – yet robust support framework means setting up our AWS operational infrastructure has been seamless.”