BB2: New serverless data analytics platform
BB2, a startup financial firm based in London, had built a new digital bank on AWS using cloud-native tech, real-time streaming data feeds and serverless components. Creativity and innovation have always been the cornerstone for BB2's business model as the bank is keen on reinforcing this value through innovative digital and electronic banking services.
Challenge
The Client's customer base is growing swiftly generating enormous data and facing a challenge in managing it. Colibri stepped in to help out the BB2 to craft a Data lake solution and to help with navigating through their cloud journey.
Solution
Aligning with the BB2 team goals Colibri engineers designed and built the end solution based around Lambda, S3, Athena, AWS Glue, Kinesis, DynamoDB, Amazon Cloudwatch and Quicksight.
Using terraform as infrastructure as a code solution along with GitLab for CI/CD, the team started by designing and building a strong foundation to form a serverless Data Lake, creating a safe, secured and collaborative environment. CI/CD pipelines were developed to run in bespoke pre-built Docker images for clean configuration and fast deployments.
Lambda functions were used to blend the data via Athena SQL for efficient processing and writing on partitioned tables in Glue and Athena. Then we enabled DynamoDB streams to capture changes to source tables in real time, leveraging Lambda functions to process and blend data in a cost effective and scalable fashion. With the initial version deployed, we worked to create a data strategy to get data from multiple sources and make it available and visible to leadership teams in one place.
Quicksight dashboards were created for every source, supported with CI/CD built with Quicksight APIs to enable propagation of Quicksight objects between different environments - enabling proper QA. The BI solution is entirely self-service, leveraging user management to grant varying permission levels to different user teams, allowing the business to access pre-prepared clean data as well as design custom reports to be shared with their team
Results
With our well-architected serverless Data Lake solution, the client now can capture all of their telematics and metrics, store them centrally, and structure them in a way that they can be easily analysed. The flexibility in the infrastructure to swap different components at low cost is helping BB2 to plan their scaling operations and make decisions on taking their next step in the data journey.
The self-service, scalable BI tool means that users with no SQL experience could now build dashboards based on data relevant only to their role and persona. This allows leadership to focus on growth and product development for the bank - with at-a-glance views of key metrics.
User access to read-only dashboards with custom permissions is helping teams across the organisation to take quick and accurate decisions.