Overview
Having previously been highly paper based, a UK regulator was looking to both take advantage of new opportunities from Data and AI; help the regulator to govern itself in the new business environment and also provide clear, experience based, Data & AI regulatory guidance for its market. Two recent external reports had identified a low level of data maturity across the organisation which a new data catalogue needed to address.
The Challenge
The organisation:
- Had good, dedicated teams dealing with various aspects of the end-to-end data ‘journey’ but these were somewhat siloed with little understanding of, and access to documentation for, the data pipeline outside their own areas of expertise.
- The data quality was variable hindering the Data Science team from developing new AI models.
- Business end users had little understanding of the data pipeline that drove their business processes so had little knowledge of where data quality initiatives would be most cost-effective.
As ever resources of time, money and expertise were limited.
Approach
It was important to:
- Liaise with business and technical units across the organisation to fully understand their needs and develop a formal requirements catalogue to capture those needs.
- Liaise with the Enterprise Architect to add my expertise to a new Data Driven Operating Model for the organisation; identifying key new roles like Data Stewards and Metadata Engineers whilst ensuring the chosen data catalogue product was compatible with the wider operating model options being considered.
The above was then used as the basis for identifying the most suitable product that incorporated:
- Extensive automation of the full data pipeline, from capturing in a proprietary CRM, processing into a Data Warehouse and usage in BI reporting and M/L models.
- The ability to manually augment automated processing of that pipeline and represent it in easy-to-understand formats (tables, maps and charts) to technical and non-technical users.
- Allowing automated and manual data quality checks to be linked to datasets across the end-to-end data pipeline to enable all parties to focus on the most important data assets for data quality improvement initiatives.
- Linking business processes, process maps and policies to datasets for enhanced data ownership and governance across the organisation.
- Would work with existing technologies and those ‘road-mapped’ for planned introduction in the next 3-5 years.
- Could be procured at an acceptable cost with the ability to introduce a ‘bronze’ version, with a clear path to ‘silver’ and ‘gold’ levels of functionality as funds allowed.
Outcomes
We created a solution which delivered:
- A clear view of the optimum catalogue solution for the Regulator, plus three viable alternatives ready for a formal procurement exercise.
- A realistic budget for the delivery of that solution.
- The incorporation of the catalogue into a wider Data Driven Operating Model, facilitating raising the data maturity across the organisation.