Skip to main content
Blog

100x the Speed: Our Data Analysis Tools Engine Upgrade

By November 22, 2023November 27th, 2023No Comments
100x the Speed: Our Data Analysis Engine Upgrade

We rewrote our analysis engine from the ground up, and the results are amazing. The KnowledgeHound analytics engine is now a best-in-class feature of our platform, delivering a verified 100x speed for survey analysis.

  • Do you have a brand tracker survey that has accumulated hundreds of questions over the years? 
  • Do you struggle to sift through matrix questions with a long list of products or competitive brands?
  • Need to understand the awareness of various brands across age groups in multiple regions? 
  • Have hundreds of thousands of respondents, but you still need that trend analysis report broken down by quarter? 
  • Want to see that custom report all over again, but this time segmented by gender?

KnowledgeHound now makes it super fast to analyze these surveys and questions, segmenting/cutting them by other questions or demographics in seconds instead of minutes.

The new analysis engine also underpins our Rapid Reports capability, which allows you to bulk-generate complex analyses across your largest surveys in just a few clicks and seconds. It’s a powerful tool to help you socialize key findings with complex visualizations that convey meaning to the insights across an entire brand tracker!

Rapid Reports capability

Ready to see how the new analytics engine will open the door to you surfacing insights at a pace never seen before? We’re ready to talk!

Skeptical? 

How we did It

For several years, our analysis engine has stored respondent level data as individual rows in our primary database. It was clear that this data model did not scale efficiently when you have over a million respondents and over 1000 questions in a survey. We made two key moves. First of all, we created a new data model to store respondent level data efficiently, and then we transitioned from Python to Rust.

The Results

This work created a new data model: the Unified Survey Data Model. This data model represents survey data in all its forms (dichotomous, non-dichotomous, single choice, multiple choice, matrices, categorical, numeric, open ends etc) and mapped respondent answers into a very efficient set of bitmaps. I.E., 0’s and 1’s, which are much more efficiently stored and retrieved during analysis. We normalize data coming from all sources (SPSS, integrations like Survey Monkey, Qualtrics and Toluna, and our own data analysis tools, API used by AYTM, Sago, etc) to provide a consistent analysis experience irrespective of the data source.

Combining this new data model with a super fast programming language, Rust, we rebuilt our analysis engine to use the latest technology and deliver a user experience that is unrivaled. Our initial proof of concept showed us that we could achieve an improvement in raw analysis that was over 800%.

Have a big survey? We can handle it!

Save time. Curate insights. Share data quickly.

Reach out to learn more about KnowledgeHound.