Cirrus Minor - The nexus of technology, business & people

Skip to content

Building Real-Time Data Systems the Hard Way

What Is Real-Time Data? Real-time data proves to be beneficial for businesses. They use real-time data throughout the organization to derive useful information from it and extract better insights. Real-time data is helpful for monitoring and maintaining IT infrastructure. Real-time data enables businesses and organizations to get better visibility and understanding of the operation of their intricate networks. It is crucial to...
What time is it? A simple question with a complex answer

Ever wondered how your computer or your phone displays the current date and time accurately? What keeps all the devices in the world (and in space) in agreement on what time it is? What makes applications that require precise timing possible? In this article, I will explain some of the challenges with time synchronization and explore two of the most popular protocols that devices use to...
The Magic of Sampling, and Its Limitations

The Magic of Sampling, and its Limitations Posted on Saturday, February 4, 2023. ...
Implementation of Survival Analysis in Customer Analytics

In general, survival analysis deals with topics such as death, churn, and recovery. In this article, we will look at survival analysis from the perspective of customer analytics in the telecom industry.Survival analysis is a statistical tool for analyzing time until some event occurs. This tool answers “How long would it be, before the specified event happens?” question. Therefore, we...
How to boost the efficiency of Kafka message driven services

How to boost the efficiency of Kafka message driven servicesAVRO — The secret recipe of data serializationPhoto by Jeff Sheldon on UnsplashThe transmission of computer records can be done in various formats such as CSV, text and even byte binary format. JSON, without a doubt, is one of the prevalent data formats nowadays. It is extensively adopted in REST APIs....
Reliable event dispatching using a transactional outbox

Using events in a system is great, but how do you know for sure if you've reliably dispatched your events? The transportation of events needs to be done reliably while maintaining overall system consistency, be it eventual or immediately. In a typical setup, a database is used to store information and queues are used to send messages between processes and...

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Continue reading

Replacing Docker Desktop with hyperkit + minikube

Replacing Docker Desktop with hyperkit + minikube

Published by Arnon Rotem-Gal-Oz on September 2, 2021

Edit June 2023: Added a section on Colima MacOS is a Unix but it isn’t a Linux so, unfortunately, if/when we need to use linux-y…

Continue reading

Intro to Apache Spark (slides)

Intro to Apache Spark (slides)

Published by Arnon Rotem-Gal-Oz on December 16, 2020

I gave a general overview of Apache Spark to our R&D teams. You can find the slides below

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Continue reading

Intro to Docker & Kubernetes (slides)

Intro to Docker & Kubernetes (slides)

Published by Arnon Rotem-Gal-Oz on May 25, 2020

Just had a 2 hours session with my team introducing Docker and Kubernetes (presentation and hands-on). I thought the slides might be useful to others: