Transforming and Linking Data

Overview — This use case aims to improve how the Scottish Government creates, receives and processes the large amounts of structured, semi-structured and unstructured data and information it holds, by testing new ways of making connections between them.

How it is used — Two pilot areas were identified to test new approaches: improvement activity across the Scottish Government, and Programme for Government templates. The Scottish Government currently relies heavily on people to make connections between structured and unstructured information, using approaches that require significant design and governance and may not keep pace with constant change. Newer technologies, such as AI, semantic data and graph databases, have the potential to let computers do more of this work and to keep the structures current as the environment and language change. Working with Wallscope in a secure, private environment, the project centres on building a knowledge graph from sample data (Improvement Activities and Programme for Government) that contains both structured and free-text references, extracting key concepts to reveal relationships between data, organisations, people, places and other relevant concepts.

Scope and limitations — The project uses sample data within a secure, private environment and currently applies a hybrid approach with manual cleaning of results, given the small data volume. Any future deployment would be expected to use more fully automated processing once the models are trained on sufficient data.