Cirrus Minor - The nexus of technology, business & people

Skip to content

Announcing the General Availability of Materialized Views and Streaming Tables for Databricks SQL

We’re excited to announce that materialized views (MVs) and streaming tables (STs) are now Generally Available in Databricks SQL on AWS and Azure...
How I Queried Millions of Rows in Under a Second with AWS Lambda and DuckDB

Many data engineers face the challenge of processing massive datasets quickly. In my recent project, I used AWS Lambda and DuckDB to query…Continue reading on Medium »
Modularity and blast radius in Infrastructure as Code

Photo by deep Bhullar: https://www.pexels.com/photo/vibrant-explosion-of-fire-against-black-background-29052108/As with my recent post on state in Infrastructure as Code (IaC), modularity is a topic that comes up frequently. Code/template reuse is one driver of modularity, but another is blast radius, the scope of resources potentially impacted by a single action, such as terraform apply.Shared fateApply operations are normally done at the root module or...
We're Leaving Kubernetes

Kubernetes seems like the obvious choice for building out remote, standardized and automated development environments. We thought so too and have spent six years invested in making the most popular cloud development environment platform at internet scale. That’s 1.5 million users, where we regularly see thousands of development environments per day. In that time we’ve found that Kubernetes is not...
Large Language Models Are Changing Collective Intelligence Forever

September 20, 2024 New Paper Co-authored by Tepper School Researchers Articulates How Large Language Models Are Changing Collective Intelligence Forever Within teams, organizations, markets, and online communities, ideas from a larger group can help to solve complex problems. Large language models (LLMs) are emerging as powerful tools to unlock even greater potential. Picture an online forum where thousands of voices contribute...
DuckDB Is Not a Data Warehouse

Before I get to DuckDB, I’ve got three house-cleaning items this week: Bluesky, Materialized View’s one year anniversary, and P99 CONF.Let’s begin with social media. I’ve moved to Bluesky 🦋. Follow me @chris.blue if you’ve enjoyed my Twitter posts over the past 15 years. You can crosspost with Fedica or Buffer if you like. There are some great starter packs...

Software architecture workshop (slides)

Vasa Ship

Software architecture workshop (slides)

Published by Arnon Rotem-Gal-Oz on November 29, 2023

The title says it all – These are slides from a session I was working on to explain the basics of software architecture based on…

Continue reading

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Continue reading

Replacing Docker Desktop with hyperkit + minikube

Replacing Docker Desktop with hyperkit + minikube

Published by Arnon Rotem-Gal-Oz on September 2, 2021

Edit June 2023: Added a section on Colima MacOS is a Unix but it isn’t a Linux so, unfortunately, if/when we need to use linux-y…

Continue reading

Intro to Apache Spark (slides)

Intro to Apache Spark (slides)

Published by Arnon Rotem-Gal-Oz on December 16, 2020

I gave a general overview of Apache Spark to our R&D teams. You can find the slides below

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Continue reading