Streamkap Enables Fleetio to Ditch Weekend Data Maintenance for Reliable, Real-Time Streaming

When a delivery truck breaks down on the side of the highway, it's more than just an inconvenience—it's a costly setback that can ripple through an entire operation. For companies juggling hundreds, or even thousands of vehicles, each with its own maintenance schedule, fuel consumption, and performance metrics, effectively managing their fleets is a constant challenge.

Enter Fleetio, a leading fleet maintenance management software company that has been empowering businesses of all sizes to take control of their vehicle fleets since its inception in 2012. Serving nearly 6,000 customers across 100 countries, Fleetio has become the go-to solution for a diverse range of organizations, including household names like AAA, Goodyear, and Roto-Rooter. 

At the heart of Fleetio's operations lies a robust data infrastructure that powers its fleet management software. However, as Fleetio's customer base grew, so did the challenges faced by their data team. The company found itself grappling with an outdated batch processing system that couldn't keep up with their expanding data needs.

To address these issues, Fleetio recently replaced its batch data processing solution with Streamkap, a provider of real-time streaming data built on top of Apache Kafka and Flink. Easy to use and zero maintenance, Streamkap offered Fleetio a solution that not only addressed their immediate data replication issues but also paved the way for future innovations. Here’s how this collaboration dramatically improved Fleetio's data infrastructure, leading to improved efficiency, cost savings, and enhanced real-time analytics capabilities.

The taxing burden of recurring data replication failures

With millions of records being updated or created hourly, Fleetio’s data team is responsible for ensuring that this vast amount of information is accurately processed, stored, and made available for analysis in real time. This data encompasses the vital signs of countless vehicles and assets—from delivery trucks crossing the country to equipment on construction sites—as well as analytics ingested from telematics providers.

However, Fleetio's data team faced significant challenges with their batch data processing solution, which had difficulties replicating data from the Fleetio platform into Snowflake for analytics purposes.

"I remember one week when I had to completely replicate our entire database seven times because it just was not working at all," recalls John Michael Mizerany, Senior Data Engineer at Fleetio. "I was working most weekends just to keep it going.

As the company expanded and data volumes increased, the limitations of this approach became glaringly apparent:

• Performance Issues: The batch extraction process caused replication logs in the database to grow over time, leading to downstream problems.
• Frequent failures: The system would often fail, requiring a complete restart of the replication process.
• Time-consuming maintenance: Data engineers spent countless hours, including weekends, managing and troubleshooting the system.
• Delayed insights: The batch process meant that data sometimes wasn't available in real time, hindering the company's ability to provide up-to-date analytics.

Delivering replication at much lower costs

Fleetio's journey to finding a better solution led them to Streamkap. After hearing about Streamkap at a conference and adding it to their evaluation list, the team decided to give it a try. The onboarding process was surprisingly smooth and fast.

"I signed up for a free trial on the site and then requested a meeting," Mizerany explains. "Paul Dudley, who's Streamkap’s CEO, had a meeting on the calendar that same day. We got replicating within the next day. It was pretty straightforward."

Streamkap offered Fleetio a real-time streaming solution that addressed the pain points of their previous batch processing system:

• Real-time data replication: Instead of waiting hours for batch updates, data now flows continuously into their Snowflake data warehouse.
• Improved performance: Streamkap's ability to recover on its own meant fewer interruptions and less manual intervention.
• Cost-efficiency: The solution proved to be more cost-effective, both in terms of direct costs and reduced Snowflake usage.
• Scalability: Streamkap's architecture was better suited to handle Fleetio's growing data volumes.
• Flexible transformation: The ability to perform certain transformations during the replication process added extra value.

"We integrate with some large telematics providers,” Mizerany explains. “So we have a lot of fast-moving data, and we definitely need that data to be real-time on how we're getting from one place to another. Being able to have this real-time data from when an event happens in a fleet to having it in our data warehouse, and even potentially being able to open that to other destinations, is a real value proposition of ours."

Fast, up-to-date data streaming opens up new world of possibilities

The switch to Streamkap resulted in significant benefits for Fleetio:

• Reduced operational overhead: Data engineers were freed from constant troubleshooting, allowing them to focus on more strategic initiatives. “Issues would arise with the batch tool that we were using; it would just fail and we would have to completely replicate our entire database just to get it caught back up,” Mizerany recalls. “Almost every other day I was having to do that.”
• Improved data accuracy: Mizerany notes, "We were finding that data was more accurate with Streamkap than with our other tool. At first we were like 'wait, this is wrong' and then we realized 'wait, no, the other tool was wrong. Streamkap is actually correct.'"
• Cost Savings: Fleetio saw a dramatic reduction in their Snowflake costs related to data loading. "The cost that we have in Snowflake now for loading data on Streamkap is like next to nothing," Mizerany shares. 
• Enhanced analytics capabilities: Real-time data availability opened up new possibilities for timely insights and anomaly detection. "It's a big part of our stack now because these data products that we're going to be releasing heavily rely on the data that Streamkap is producing to Snowflake," Mizerany shares.
• Increased Confidence in data infrastructure: With a more reliable system in place, Fleetio could move forward with developing new data products for their customers. Mizerany notes, "If there's ever any issues, Streamkap can recover itself, which is a nice thing. Compared to our old tool, if one small thing happened, it would just completely break."

Unlocking analytics with the freshest data

Data management remains critical to Fleetio's ability to deliver outstanding fleet management services. Whether it's a local government agency managing snowplows, a nationwide logistics company coordinating cross-country deliveries, or a global corporation maintaining a vast network of service vehicles, Fleetio's cloud-based platform serves as the central nervous system for fleet operations. 

The transition to Streamkap's real-time data streaming solution has not only resolved Fleetio's internal data challenges but has also opened up new possibilities for enhancing their product offerings and customer value. By bringing together vehicle inspections, work orders, preventive maintenance schedules, parts inventory, and fuel management into a single, user-friendly dashboard, Fleetio helps fleet managers make informed decisions that keep their vehicles on the road and their businesses moving forward.  

"Everyone loves the freshest data, and that's a key appealing feature of Streamkap,” says Mizerany. “Instead of waiting hours for our old tool to extract and load data, we now get real-time results. This enables more effective real-time analytical strategies, which is crucial because transactional databases aren't ideal for analytics. By extracting data in real-time from our database to Snowflake, we're able to unlock a lot of our analytical goals.”

‍