17 Feb Client Story: Modernising Data Management in Government
Client Story:
Modernising data management in Government using Databricks
THE CHALLENGE
This government department runs a survey annually to collect security assessment data, as part of a compliance driven activity. All Service Providers must complete an assessment; the Department want to understand who has provided an assessment and what the actual maturity level ratings are for each provider. A risk rating is calculated for each provider and used to drive a program of change.
The ultimate objective is therefore to support the tracking of security assessments, maturity levels, and risk ratings.
OUR BREIF
The Department established a unified data warehouse to transform, consolidate and report this information. Our role was to build the transformation framework using Databricks. The data was ingested using LogicApps, orchestrated by ADF, transformed by Databricks and analysed/reported in PowerBI.
OUR APPROACH
Key actions included:
- Ingest from 10 security data sources, transform, and integrate from external service providers into the unified data warehouse.
- Store and manage security source data on SharePoint Teams sites to facilitate easy access and collaboration, allowing end users to upload and add new files directly within SharePoint.
- Utilize Databricks and Azure Data Factory to handle data transformation and loading processes, reducing the load at the Power BI level.
- Used the transformation framework to allow users to compare current-year data with snapshot data from previous years and facilitate the development of powerbi dashboards.
KEY OUTCOMES
There were numerous benefits resulting from the project, including:
- Streamlined and enhanced the collection of security data across the department with near real time ingestion and transformation using modern orchestration techniques
- Delivered a highly interactive and user-friendly dashboard solution, enabling stakeholders to compare current-year performance with historical data snapshots for better trend analysis.
- Ensured the solution was scalable, secure, and aligned with the department’s long-term goals for data management, reporting, and analytics.
- Enhanced the department’s ability to track service security assessment completion, maturity levels, and risk ratings in real-time.
- Improved the department’s operational efficiency and risk management capabilities, ultimately supporting better service delivery across the department’s eco-system.