Improving Business Survey Targeting Through AI. Our Role in the UK Business Data Survey.
The UK Business Data Survey 2024, commissioned by the Department for Science, Innovation and Technology (DSIT) and delivered by Ipsos, sought to better understand how UK businesses collect, use, store, and transfer data. A key challenge in this wave of the survey was identifying and reaching businesses involved in international data transfers — a relatively small and hard-to-locate subset of UK firms.
To support this effort, DSIT and Ipsos engaged Glass.AI to provide an additional, targeted business sample. This involved using our automated web analysis to identify relevant firms based on open data signals.
AI-Assisted Sample Construction
At the start, Glass.AI confirmed with Ipsos the definition of ‘international data transfer’. It was suggested that the transfer of data to another country could be both internal and external, including UK businesses with operations abroad or UK subsidiaries of overseas companies. Then the first step in our process was to deep-read the UK web to build an initial dataset of potential candidates. The crawler looked for UK businesses that showed the following signals or indicators:
▪ Does the company have offices abroad?
▪ Does the company have people/employees abroad?
▪ Does the company mention international trade/activities on the website (e.g., press releases, mentions of foreign clients, partners, suppliers, JVs, etc.)?
Glass.AI applied its large-scale language processing and web data analysis to scan over 2.2 million UK company websites, along with other digital content, to identify businesses likely to transfer data internationally. The initial crawl resulted in 20,000 candidate companies with evidence and classified by sector, size, and location. After additional filtering and validation of the signals by our data scientists, the final sample had 13,000 firms, all matched to Companies House.
The approach was designed to complement the traditional sampling method used in the main survey and explore whether AI-driven discovery could improve targeting effectiveness.
Comparison of Results
Compared to the main sample, the Glass.AI sample resulted in fewer unusable leads: 33% in the Glass.AI sample, compared to 61% in the main sample source from company databases — and a significantly higher eligibility rate, with 79% of leads found to be in scope compared to 59% for the main sample. This ultimately produced a higher response rate from contact attempts and increased survey completion. These results suggest that AI-assisted identification led to a more targeted and efficient survey sample, particularly for businesses engaged in complex or less-visible data practices.
Role in the Survey
The AI-generated sample helped improve coverage of an underrepresented group: businesses transferring data internationally. This provided valuable support to the research objectives and contributed to a more robust evidence base for policy development.
While the study remains exploratory in its use of AI for business sampling, the results suggest that such methods can be a useful complement to traditional approaches, particularly when targeting specialised or low-incidence business populations.
Read more about the survey here: https://www.gov.uk/government/statistics/uk-business-data-survey-2024