DynamoDB ETL and Rockset migration with Streamkap

September 12, 2024

You’ve already heard the news: OpenAI has acquired Rockset, a company known for its innovative real-time analytics and document database support. This acquisition is part of OpenAI's broad strategy to enhance its capabilities in handling complex AI workloads – and Rockset’s technology aligns perfectly with OpenAI's needs, making it a valuable acquisition.

Unfortunately, it will also leave users behind.

Rockset's product will sunset for existing customers by September 30th. If you’re among those seeking a new solution, we can present an efficient, cost-effective, and well-supported path forward.

Migrating from Rockset doesn’t need to be to your detriment. It could ultimately save you a lot of money. Let’s look at the advantages of DynamoDB ETL and Rockset migration with Streamkap.

The impact of OpenAI’s Rockset acquisition

In 2012, engineers at Facebook developed RocksDB — which would eventually become Rockset.

Rockset has some unique advantages, including its document database structure and native ETL for MongoDB and DynamoDB for real-time analytics. Rockset has become a powerful tool for those requiring seamless data integration and high-performance analytics, and it also offers columnar, row, and search-based indexes for a wide variety of use cases.

However, with OpenAI's acquisition, Rockset’s consumer-facing product is being phased out, leaving its customers at a crossroads. With Rockset's sunset, businesses must find an alternative that not only matches but exceeds the performance they’ve come to expect — and hopefully without a significant increase in cost.

Migrating from Rockset

Document databases like MongoDB and DynamoDB have flexible data structures that Rockset handles natively and as such they were common pairings with Rockset. Replacements for Rockset, databases such as ClickHouse and SingleStore, are more SQL-oriented, requiring a structured data format. This means that there is often quite a lot more ETL work to be done to successfully replace Rockset.

For DynamoDB, in particular, there are challenges in finding complete solutions for CDC.

While streaming changes are available via DynamoDB Streams API, only 24 hours of change history is stored.
Backfill options include the Scans API (one-time scanning of the database – usable for smaller databases but problematic when combined with 24-hour retention) and Export to S3 (an elegant solution but not commercially available outside of Rockset).
Migrating semi-structured data from DynamoDB into SQL-oriented OLAP databases can cause problems becuase data must be unnested and variations and type must be handled

Moving from MongoDB or DynamoDB to ClickHouse, PostgreSQL, or SingleStore

We suggest moving from MongoDB or DynamoDB to ClickHouse, PostgreSQL, or SingleStore. Over the past few weeks, we’ve supported our clients during this transition, creating solutions for filtering, ingestion, and transformation.

Connectors and pipelines

Our connectors and pipelines are designed to address any data-related challenges: manage data types, unnest nested data, and ensure data consistency. With a combination of our built-in solutions and our customized solutions, we can support any transition.

Streaming data transformations

Many Rockset customers have SQL and JavaScript transformations directly in Rockset, and these SQL and JavaScript transformations can be a headache to transfer. We support these transformations in the stream for lower cost and lower latency than completing transformations within the destination database.

Migration and engineering

At Streamkap, we provide complete transformation migration. You can offload some of the engineering work from your organization to our team – making tight deadlines, even with complex data challenges.

24/7 support and 99.99% guaranteed uptime

We offer BYOC and SaaS deployment options with 24/7 support and a 99.99% uptime guarantee. You can transition your real-time ETL workloads with Streamkap with assurances that your data will be successfully migrated and remain accessible.

Streamkap: your partner in seamless migration

Migrating from Rockset to a new platform can be daunting, but with Streamkap, the process is simplified. Our platform is designed to handle the complexities of migration, from data transformation to managing data types and unnesting nested data.

We address key challenges in data migration:

Data Structure Differences: Transitioning from document-oriented databases to SQL-based systems.
Data Transformation: Managing complex data types and unnesting nested data during migration.
Consistency and Performance: Ensuring that the migration doesn’t disrupt real-time analytics.

And when you choose Streamkap, you get access to:

Stream-Based Transformations: Offload complex SQL and JavaScript transformations to Streamkap.
Comprehensive Connector Suite: Seamlessly manage data types and nested data.
Tailored Support: 24/7 customer service with a 99.99% uptime guarantee.

Let’s get started

Don’t treat your migration from Rockset as a setback; treat it as a competitive advantage. Our comprehensive migration support, combined with our connectors and pipelines, will ensure that your transition is smooth, efficient, and customized to your needs.

‍