Cirrus Minor - The nexus of technology, business & people

Skip to content

Compaction in Apache Iceberg: Fine-Tuning Your Iceberg Table’s Data Files

The Apache Iceberg format has taken the data lakehouse world by storm, becoming the keystone pillar of many firms’ data infrastructure. There are some maintenance best practices to help you get the best performance from your Iceberg tables. This article takes a deep look at compaction and the rewriteDataFiles procedure. If you are new to Apache Iceberg make sure to check...
Apache Iceberg and the Right to Be Forgotten

Time travel is a fantastic feature that’s available when you adopt a data lakehouse table format like Apache Iceberg, but sometimes this useful feature runs contrary to regulatory requirements such as GDPR (General Data Protection Regulation) and CCPA (California Consumer Privacy Act). This is true of other systems that offer time travel as well, such as Snowflake. This is an issue...
Partitioning for Correctness (and Performance)

Partition design is a critical part of data modeling. Unfortunately, given the constraints of most Hive-based tables, data engineers (myself included) are often forced into choices that have unintentional and potentially dangerous implications. In this post I’m going to illustrate some of the potential pitfalls around partition design and techniques for avoiding them using the popular NYC taxi data set.The...
Ensuring High Performance at Any Scale with Apache Iceberg's Object Store File Layout

Note: You can find this and many other great Apache Iceberg instructional videos and articles at our Apache Iceberg 101 article. Cloud object storage services like AWS S3, Azure Blob Storage and Google Cloud Storage have been transformational in the world of data storage. By drastically lowering the cost and complexity of storing large amounts of data many have migrated away...
Near Real-Time Ingestion For Trino

It is quite popular in today’s data climate for modern data architectures to have some sort of batch processing system to move data into a data lake at an hourly or daily cadence. This is considered an adequate process for lots of analytics needs that do not require immediate updates, like drilling into revenue or user trends. However, the ability...
New Capabilities from Dremio Mark a Turning Point in Data Lakehouse Evolution - Business Wire

SANTA CLARA, Calif.--(BUSINESS WIRE)--Dremio, the easy and open data lakehouse, today announced key features for writing and updating data, enhanced support for semi-structured data, and expanded BI and data ecosystem integrations, that mark a turning point in data lakehouse evolution. On the heels of its inclusion among CNBC’s 25 unranked Top Startups for the Enterprise and its GA support of...

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Continue reading

Replacing Docker Desktop with hyperkit + minikube

Replacing Docker Desktop with hyperkit + minikube

Published by Arnon Rotem-Gal-Oz on September 2, 2021

Edit June 2023: Added a section on Colima MacOS is a Unix but it isn’t a Linux so, unfortunately, if/when we need to use linux-y…

Continue reading

Intro to Apache Spark (slides)

Intro to Apache Spark (slides)

Published by Arnon Rotem-Gal-Oz on December 16, 2020

I gave a general overview of Apache Spark to our R&D teams. You can find the slides below

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Continue reading

Intro to Docker & Kubernetes (slides)

Intro to Docker & Kubernetes (slides)

Published by Arnon Rotem-Gal-Oz on May 25, 2020

Just had a 2 hours session with my team introducing Docker and Kubernetes (presentation and hands-on). I thought the slides might be useful to others: