Topic
5 articles found
Google created MapReduce more than 20 years ago to solve the scaling problems in data processing that the then young company was running into. The AI era that we are in now demands efficient, large-scale data processing for everything from training frontier models like Gemini by Google DeepMind to p
Introduction Data engineering is the practice of designing and building systems for collecting, storing, transforming, and managing data so it can be safely used for reporting, analytics, machine learning, and making business decisions. Think of it as the behind-the-scenes work that makes apps, webs
The Problem We Were Actually Solving Our treasure hunt engine at Veltrix was a real-time geospatial matching service that processed 50 million location events daily. By month six it handled bursts of 2M concurrent users during events like Black Friday flash sales. The heap profile from YourKit showe
The week after a major release tends to look quiet on a project's dev list. This one did not. With Iceberg 1.11.0 and 1.10.2 both out the door the week before, and Polaris 1.5.0 shipping right at the start of this window, you might have expected the lakehouse projects to take a breath. Instead the c
Modern data pipelines handle massive volumes of structured and unstructured data every day. As datasets grow, poorly optimized Spark jobs become slower, more expensive, and harder to scale. Common issues include long execution times, excessive shuffling, memory bottlenecks, and inefficient joins. Ef