How Streamkap Reduced Niche.com's Data Latency from 24 Hours to Near Real-Time

In search of the right school to continue their education or the best neighborhood to lay down roots, people come to Niche.com for its rich data. With thousands of entities, from universities to cities, each characterized by hundreds of unique attributes, Niche.com's database comprises millions of records. This wealth of data powers the platform, providing users with the insights they need to make informed decisions about their future.

However, Niche.com faced ongoing challenges moving their core product data into Snowflake quickly and reliably for business analytics purposes. To meet the growing demand for faster and more accurate business insights, Niche.com turned to Streamkap, a real-time data streaming provider that uses change data capture (CDC) to move data with sub-second latency.

"With Streamkap, we went from waiting 24 hours for data refreshes to having it available in near real-time, which has transformed our ability to make data-driven decisions quickly,” says Vikram Chauhan, Head of Data Engineering at Niche.com. 

Here's how Streamkap helped Niche.com reduce their data latency by 95%, slash their data infrastructure costs, and transition from moving limited data to their entire set in less than a day of implementation.

Challenges with a legacy data pipeline

Niche.com's previous data pipeline had several shortcomings, hindering the company's ability to effectively use its data for analytics, reporting, and decision-making. For one, the legacy pipeline was slow to deliver the data, with a latency of up to 24 hours, and was only moving a small subset of the total data. "Our old data pipeline was not running in near real-time and was very limited in scope, picking only certain databases and tables,” Chauhan shares.

Another major issue was the unsupported platform on which the pipeline ran, making scalability a problem. In addition, Niche.com was hosting the platform internally within AWS, requiring the data team to manage all infrastructure, including servers, storage, and databases. "The old pipeline had a lot of overhead,” Chauhan emphasizes. “Not only were we using an insecure platform, but we also had to maintain it."

These challenges highlighted the need for a more robust, scalable, and real-time data pipeline solution, leading to interest in Streamkap as an alternative.

Less than a day of setup makes for easy adoption

When Streamkap caught the attention of one of Niche.com's data engineers, Chauhan was immediately drawn to the platform's affordability and ease of implementation, noting, "I've worked with CDC solutions before. But given that Streamkap seemed like such a low lift and it's super affordable, we weren't going to go with any of those CDC solutions that I have prior experience with."

Streamkap’s user-friendly interface and responsive support team also made the setup process a breeze. “As I got to know the Streamkap platform, I decided — we should implement it, full steam ahead,” Chauhan says. "From that set-up and configuration perspective, I don't even think we spent even a day. It's pretty quick.”

Achieving a 95% decrease in latency and 100% data access

With Streamkap, Niche.com was able to:

• Reduce data refresh latency from 24 hours to near real-time
• Expand data coverage, bringing in data from all relevant databases and tables
• Reduce infrastructure management, saving time and resources
• Retire their previous unsecure and unsupported data pipeline platform
  
  

These improvements have allowed Niche.com to access up-to-date data and make data-driven decisions more quickly and efficiently.

Enabling faster insights and data-driven decisions

The benefits of Streamkap were felt across the organization, particularly in Niche.com's new direct admissions offering, which provides an expedited avenue for a student to gain acceptance into a college. "We had been flying blind in terms of how the direct admissions offering is actually performing," Chauhan says. 

With the data provided by Streamkap, Niche.com's analyst team built reports and dashboards in Tableau to track the performance of the direct admissions product. This visibility has been crucial for making data-driven decisions and has received positive feedback from stakeholders across the company.

"None of that reporting would have been feasible had we not implemented Streamkap because the previous pipeline was lacking the data that we needed to show within those dashboards and reports,” Chauhan shares.

Delivering value at an affordable cost

Streamkap has enabled Niche.com to deliver real-time insights, reduce costs, and focus on growing their business. "We did a cost-benefit analysis, and cost-wise, it was just a no-brainer to move to Streamkap," Chauhan stated. "I think it's probably one of the cheapest tools I've ever used, given all the bells and whistles."

‘Everything just works’

By providing a reliable, affordable, and easy-to-implement solution, Streamkap has empowered Niche.com to access real-time insights and make data-driven decisions that guide the business forward. As Chauhan reflects on the partnership, he notes, "The plug-and-play nature of the tool is what impressed me the most. With Streamkap, everything just works."

‍