Cirrus Minor - The nexus of technology, business & people

Skip to content

Discovery of Byte-to-Byte Query Algorithms to Accelerate Dataframe

submitted by /u/100GB-CSV [link] [comments]
“Businesses don’t know how to translate DORA metrics to anything they understand or care about.” Solid blog on what dev metrics actually help and impress the revenue side of companies.

submitted by /u/TriangularHardship [link] [comments]
Rate Limiter For The Real World

In the previous article, we discussed the fundamentals of rate limiting and explored several popular rate limiting algorithms. We’ll continue by exploring how to build some real-world rate limiters.There are many factors to consider when developing a rate limiter for the real world. For example, in many use cases, a rate limiter is in the most critical path of an...
Intro to the Iceberg Kafka Connect sink • Tabular

Welcome to this series of posts that will provide an in-depth look at the new Iceberg Kafka Connect sink, developed by Bryan Keller, a Senior Software Engineer at Tabular. This blog delves into the world of Kafka and Kafka Connect, highlighting their potential applications and mainly focusing on sink connectors. We’ll discuss the merits of employing Apache Kafka Connect for...
Gotchas of Streaming Pipelines: Profiling & Performance Improvements

Discover how Lyft identified and fixed performance issues in our streaming pipelines.BackgroundEvery streaming pipeline is unique. When reviewing a pipeline’s performance, we ask the following questions: “Is there a bottleneck?”, “Is the pipeline performing optimally?”, “Will it continue to scale with increased load?”Regularly asking these questions are vital to avoid scrambling to fix performance issues at the last minute. By...
Temporal Cohesion — Types Of Cohesion

Temporal Cohesion — Types Of Cohesion📚
Temporal Cohesion Definition:Temporal cohesion refers to the characteristic of a software module where its components are grouped together based on the time frame in which they are processed within the program. In simpler words, if a component within the module carries out multiple tasks that are interconnected by their timing, it exhibits temporal cohesion.📚Overview of
Temporal Cohesion:Temporal cohesion...

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Continue reading

Replacing Docker Desktop with hyperkit + minikube

Replacing Docker Desktop with hyperkit + minikube

Published by Arnon Rotem-Gal-Oz on September 2, 2021

Edit June 2023: Added a section on Colima MacOS is a Unix but it isn’t a Linux so, unfortunately, if/when we need to use linux-y…

Continue reading

Intro to Apache Spark (slides)

Intro to Apache Spark (slides)

Published by Arnon Rotem-Gal-Oz on December 16, 2020

I gave a general overview of Apache Spark to our R&D teams. You can find the slides below

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Continue reading

Intro to Docker & Kubernetes (slides)

Intro to Docker & Kubernetes (slides)

Published by Arnon Rotem-Gal-Oz on May 25, 2020

Just had a 2 hours session with my team introducing Docker and Kubernetes (presentation and hands-on). I thought the slides might be useful to others: