Cirrus Minor - The nexus of technology, business & people

Skip to content

Fugue: A unified interface for distributed computing

Tutorials API Documentation Chat with us on slack! Fugue is a unified interface for distributed computing that lets users execute Python, Pandas, and SQL code on Spark, Dask, and Ray with minimal rewrites. Fugue is most commonly used for: Parallelizing or scaling existing Python and Pandas code by bringing it to Spark, Dask, or Ray with minimal rewrites. Using FugueSQL to define end-to-end workflows on top...
Logging in Kubernetes: A Guide to Staying Sane in a Sea of Data

Logging is one of the three pillars of observability in distributed systems. As such, we have seen an explosion of popular open-source (e.g. ELK stack) and mature commercial products (e.g. Splunk) to deal with logging at scale. However, in a complex system like Kubernetes, logging remains a hard problem, compounded by the continued growth of data driven by increased adoption...
RT @DefenseUnicorns: Meet Lula, our OSCOWL 🦉. Unicorns have been troubleshooting control validation issues & it's been a hoot😉! We're t…

Arnon Rotem-Gal-Oz (@arnonrgo) retweeted:Meet Lula, our OSCOWL 🦉.Unicorns have been troubleshooting control validation issues & it's been a hoot😉! We're thrilled to see how this will enhance the #compliance ecosystem and pave the way for a valuable #opensource tool. Check it out! github.com/defenseunicorn… pic.twitter.com/u6RX7er4Us- DefenseUnicorns (@DefenseUnicorns)07:31 - Mar 21, 2023
Introducing native support for Apache Hudi, Delta Lake, and Apache Iceberg on AWS Glue for Apache Spark, Part 2: AWS Glue Studio Visual Editor

In the first post of this series, we described how AWS Glue for Apache Spark works with Apache Hudi, Linux Foundation Delta Lake, and Apache Iceberg datasets tables using the native support of those data lake formats. This native support simplifies reading and writing your data for these data lake frameworks so you can more easily build and maintain your...
Multiple Vendors Make Data and Analytics Ubiquitous - The New Stack

In the last few weeks, there has been a glut of news from many of the major players, new and old, in the analytics space. Together, they point to some important trends in the industry as it wraps up its first quarter of the 2023 calendar year. As complicated as the analytics landscape can be, it’s starting to feel like...
Taking Another Stab At Massively Parallel Data Analytics - The Next Platform

For many years, and quite a long time ago in computer time, Hadoop was seen as the best way to store and analyze mountains of unstructured data. But then workloads and data began shifting to the biggest cloud builders, where Amazon Web Services, Microsoft Azure, and Google Cloud were waiting with their own technologies for doing what Hadoop could do...

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Continue reading

Replacing Docker Desktop with hyperkit + minikube

Replacing Docker Desktop with hyperkit + minikube

Published by Arnon Rotem-Gal-Oz on September 2, 2021

Edit June 2023: Added a section on Colima MacOS is a Unix but it isn’t a Linux so, unfortunately, if/when we need to use linux-y…

Continue reading

Intro to Apache Spark (slides)

Intro to Apache Spark (slides)

Published by Arnon Rotem-Gal-Oz on December 16, 2020

I gave a general overview of Apache Spark to our R&D teams. You can find the slides below

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Continue reading

Intro to Docker & Kubernetes (slides)

Intro to Docker & Kubernetes (slides)

Published by Arnon Rotem-Gal-Oz on May 25, 2020

Just had a 2 hours session with my team introducing Docker and Kubernetes (presentation and hands-on). I thought the slides might be useful to others: