ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. ETL is a product of the relational database era and it has not evolved much in last decade. With the arrival of new cloud-native tools and platform, ETL...
Kubernetes has emerged as go to container orchestration platform for data engineering teams. Kubernetes has a massive community support and momentum behind it. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of...
On this blog from very early on, we have advocated the concept of service mesh. In fact, our post a sidecar for your service mesh is one of the most viewed posts this year. When Buoyant announced the Conduit - their next-generation lightweight service mesh...
Data integration generally requires in-depth domain knowledge, a strong understanding of data schemas and underlying relationships. This can be time-consuming and bit challenging if you are dealing with hundreds of data sources and thousands of event types (see my recent article on ELT architecture). Various...
A case for ELT (i.e. extract, load, and transform) and difference between ETL and ELT