Using AI to Build More Targeted Sample Frames for Market Research and Consulting Firms.

There are many occasions when governments, market research and consulting firms need to take the pulse of businesses. This might be to assess the impact of the changing economic climate, the effect of regulatory changes, the extent of international trade, or to drive future policy changes. The success of these research projects often depends on the quality of the sample frames that are developed to initiate the primary research. Traditionally, these studies have followed a ‘spray and pray’ approach and relied on official data sources and company databases to build the samples of survey recipients. This can work well when a large general business survey is needed however, when more targeted samples of businesses are required, the traditional approach often leads to poor response rates as most targets are not suitable for inclusion. This is problematic for the market research firms doing the studies on behalf of the governments — some projects need to be cancelled due to poor and generic sample frames.

At glass.ai we are transforming how some business samples are put together. We were first introduced to sample framing as part of some of the sector mapping projects that we have contributed to. Our AI capability for large-scale language understanding of web content was used to build a robust evidence base of businesses relevant to many sector studies. Where these studies included a requirement for primary research, our AI capability also identified deeper data/insights about the companies and a set of contact details. The AI has also deep-read other online content to find key individuals that matched the roles relevant to the study questions. The results were then used to run online surveys or telephone interviews that aimed to answer a wide range of questions. As well as identifying a variety of individual contacts, some of the studies were extended further to include parts of the supply chain, adopters and other parts of the sector value chain, giving an even wider variety of perspectives for primary research to investigate.

Beyond sector mapping, there is a large range of other business activities that governments regularly seek to understand better and therefore, many government organisations commission primary research to consulting and market research firms. Often the businesses associated with certain activities are hard to uncover using traditional sources like company databases. Foreign trade (imports/exports) is a good example. It’s difficult to identify businesses that are trading with other countries, especially for certain services sectors. In this and other cases, our AI has been used to read official sources and find the digital footprints that businesses leave online (e.g. in the case of trade sources such as employee details, office addresses, customer locations, news articles, and press releases). This has helped build high-quality samples of businesses that governments and market research firms have used for their studies. Not only can these hard-to-find businesses be uncovered but the evidence base that has been collected has been used to prioritise the order in which targets are contacted to increase response rates even further. This can even be further optimised with feedback as the research progresses.

There is a wide range of business surveys that are ongoing at any one time, however without the right starting point — an effective targeted sample frame — many cannot run efficiently. We’ve been involved in several primary research projects that would just not have been viable without our unique AI research capability that reads the web to discover and understand businesses. With the digital footprint that most business activities leave online, we can build high-quality samples to enable many market research projects to be undertaken that previously would not have been possible and to improve the efficiency of those that were.