Cirrus Minor - The nexus of technology, business & people

Skip to content

Druid Deprecation and ClickHouse Adoption at Lyft

Written by Ritesh Varyani and Jeana Choi at Lyft.IntroductionAt Lyft, we have used systems like ClickHouse and Apache Druid for near real-time and sub-second analytics. Sub-second query systems allow for near real-time data explorations and low latency, high throughput queries, which are particularly well-suited for handling time-series data. For our customers, this means faster analytics on near real-time data and decision...
Learn how to assess the risk of AI systems

Artificial intelligence (AI) is a rapidly evolving field with the potential to improve and transform many aspects of society. In 2023, the pace of adoption of AI technologies has accelerated further with the development of powerful foundation models (FMs) and a resulting advancement in generative AI capabilities. At Amazon, we have launched multiple generative AI services, such as Amazon Bedrock...
OpenTelemetry in a script tag

When I make a website, even a toy one, I want to get telemetry. Has anyone visited it? Did my JavaScript crash? Little things. I want to send this with OpenTelemetry, because it’s the industry standard and all. Currently (November 2023), that requires a whole JavaScript build chain to package up the libraries etc etc. For a toy site, I want...
What about Apache Hudi, Apache Iceberg, and Delta Lake?

In the continuous evolution of Big Data architectures, Delta Lake has worked strongly with the open-source community to make our lives…Continue reading on Medium »
Pandas 2.0: The End of Traditional Data Analysis as We Know It?

Pandas 2.0: The End of Traditional Data Analysis as We Know It?Photo by Jay Wennington on UnsplashIntroductionAs a data scientist, I have spent countless hours working with data using various tools and libraries. One library that has been a staple in my toolkit for years is Pandas. Pandas has been the go-to library for data manipulation and analysis in Python,...
Data Warehouse and Dimensional Data Modeling

Yes, you are imagining it right, it takes inspiration from physical warehouses that you may have seen where companies keep all their…Continue reading on Medium »

Software architecture workshop (slides)

Vasa Ship

Software architecture workshop (slides)

Published by Arnon Rotem-Gal-Oz on November 29, 2023

The title says it all – These are slides from a session I was working on to explain the basics of software architecture based on…

Continue reading

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Continue reading

Replacing Docker Desktop with hyperkit + minikube

Replacing Docker Desktop with hyperkit + minikube

Published by Arnon Rotem-Gal-Oz on September 2, 2021

Edit June 2023: Added a section on Colima MacOS is a Unix but it isn’t a Linux so, unfortunately, if/when we need to use linux-y…

Continue reading

Intro to Apache Spark (slides)

Intro to Apache Spark (slides)

Published by Arnon Rotem-Gal-Oz on December 16, 2020

I gave a general overview of Apache Spark to our R&D teams. You can find the slides below

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Continue reading