Cirrus Minor - The nexus of technology, business & people

Skip to content

Optimizing Apache Spark Performance: Conquering Data Shuffle for Efficient Data Processing

Apache Spark has revolutionized big data processing with its distributed computing capabilities. However, Spark’s performance can be…Continue reading on Medium »
“First” and “Last” Are Not Your Friends in Apache Spark: Here’s Why

Examining Apache Spark’s ‘First’ & ‘Last’ Methods: Understanding Their Pitfalls and Proposing a More Reliable ApproachContinue reading on Medium »
Topiary: A code formatting engine leveraging Tree-sitter

Topiary aims to be a universal formatter engine within the Tree-sitter ecosystem. Named after the art of clipping or trimming trees into fantastic shapes, it is designed for formatter authors and formatter users: Authors can create a formatter for a language without having to write their own formatting engine, or even their own parser. Users benefit from uniform, comparable code style, across multiple languages, with the convenience...
Unlocking Data Recency: Automating Timely Data Delivery and Preventing Downstream Failures

In the world of Data Engineering, the timely delivery of data is crucial for ensuring smooth data processing and preventing downstream…Continue reading on Medium »
The DevOps Paradox: a Shift Away from Ops

A short story about DevOps evolution — DevOps meant Dev and Ops collaboration but removed Ops from the equation insteadContinue reading on Better Programming »
Optimizing Apache Spark Performance: Overcoming Spill for Efficient Data Processing

Apache Spark, renowned for its ability to process vast amounts of data, is a popular choice for big data analytics. However, like any…Continue reading on Medium »

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Continue reading

Replacing Docker Desktop with hyperkit + minikube

Replacing Docker Desktop with hyperkit + minikube

Published by Arnon Rotem-Gal-Oz on September 2, 2021

Edit June 2023: Added a section on Colima MacOS is a Unix but it isn’t a Linux so, unfortunately, if/when we need to use linux-y…

Continue reading

Intro to Apache Spark (slides)

Intro to Apache Spark (slides)

Published by Arnon Rotem-Gal-Oz on December 16, 2020

I gave a general overview of Apache Spark to our R&D teams. You can find the slides below

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Continue reading

Intro to Docker & Kubernetes (slides)

Intro to Docker & Kubernetes (slides)

Published by Arnon Rotem-Gal-Oz on May 25, 2020

Just had a 2 hours session with my team introducing Docker and Kubernetes (presentation and hands-on). I thought the slides might be useful to others: