Cirrus Minor - The nexus of technology, business & people

Skip to content

Diff Models – A New Way to Edit Code

CarperAI is releasing a series of diff models—models trained to predict a code diff, trained on millions of commits scraped from GitHub. We are releasing 3 models of different sizes, all fine-tuned from Salesforce’s CodeGen code synthesis models: The dataset of diffs we scraped to train these models will be released separately in the near future. We hope these models will...
Rust Books Grouped by Level

Awesome Rust Books Rust Books Books Starter Books Welcome! This book will teach you about the Rust programming language. Rust is a systems programming language focused on three goals: safety, speed, and concurrency. It maintains these goals without having a garbage collector, making it a useful language for several use cases other languages aren’t good at: embedding in other languages, programs with...
RT @ClickHouseDB: Postgres is our favorite application database, but did you know you can also use it for analytics (without having to setu…

Arnon Rotem-Gal-Oz (@arnonrgo) retweeted:Postgres is our favorite application database, but did you know you can also use it for analytics (without having to setup a complicated ETL workflow)?We wrote a blog post on how you can use @supabase + @ClickHouseDB for extremely fast analytics queries:clickhouse.com/blog/migrating…- ClickHouse (@ClickHouseDB)11:36 - Jan 27, 2023
PyIceberg 0.2.1: PyArrow and DuckDB

This blog shows how you can load data from an Iceberg table into PyArrow or DuckDB using PyIceberg. The code is publicly available on the docker-spark-iceberg Github repository that contains a full docker-compose stack of Apache Spark with Iceberg support, MinIO as a storage backend, and a REST catalog as a catalog. This is a companion piece to our PyIceberg...
The New Buzzword in Data Engineering: Zero ETL

What is Zero ETL — Definition, Benefits & ChallengesContinue reading on CodeX »
The 31 Flavors of Data Lineage and Why Vanilla Doesn’t Cut It

Data lineage, an automated visualization of the relationships for how data flows across tables and other data assets, is a must-have in the data engineering toolbox. Not only is it helpful for data governance and compliance use cases, but it also plays a starring role as one of the five pillars of data observability.
Data lineage accelerates a data engineer’s ability...

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Continue reading

Replacing Docker Desktop with hyperkit + minikube

Replacing Docker Desktop with hyperkit + minikube

Published by Arnon Rotem-Gal-Oz on September 2, 2021

Edit June 2023: Added a section on Colima MacOS is a Unix but it isn’t a Linux so, unfortunately, if/when we need to use linux-y…

Continue reading

Intro to Apache Spark (slides)

Intro to Apache Spark (slides)

Published by Arnon Rotem-Gal-Oz on December 16, 2020

I gave a general overview of Apache Spark to our R&D teams. You can find the slides below

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Continue reading

Intro to Docker & Kubernetes (slides)

Intro to Docker & Kubernetes (slides)

Published by Arnon Rotem-Gal-Oz on May 25, 2020

Just had a 2 hours session with my team introducing Docker and Kubernetes (presentation and hands-on). I thought the slides might be useful to others: