Cirrus Minor - The nexus of technology, business & people

Skip to content

The Magic of Sampling, and Its Limitations

The Magic of Sampling, and its Limitations Posted on Saturday, February 4, 2023. ...
Fluxsort: A stable quicksort, now faster than Timsort for both random and ordered data

submitted by /u/MrDum [link] [comments]
Article: Designing for Concurrency: the Hilbert’s Hotel Problem in Go

In this article, we want to show how achieving concurrency is the result of an appropriate design. A concurrent solution may turn out to be more elegant and easier to reason about than an equivalent sequential algorithm. To illustrate these concepts we use, as an example, the Hilbert’s Hotel mathematical problem. By Enrico Piccinin
Apache Spark as you don't know it

It's difficult to see all the use cases of a framework. Back in time, when I was a backend engineer, I never succeeded to see all applications of Spring framework. Now, when I'm a data engineer, I feel the same for Apache Spark. Fortunately, the community is there to show me some outstanding features!
Amazon EMR launches support for Amazon EC2 C7g (Graviton3) instances to improve cost performance for Spark workloads by 7–13%

Amazon EMR provides a managed service to easily run analytics applications using open-source frameworks such as Apache Spark, Hive, Presto, Trino, HBase, and Flink. The
Amazon EMR runtime for Spark and Presto includes optimizations that provide over twice the performance improvements compared to open-source Apache Spark and Presto. With
Amazon EMR release 6.7, you can now use Amazon Elastic Compute...
Why Learn Contextual Abstractions in Scala 3 First

TL;DR: Scala 2 is still the most used version of Scala, according to the Scala Developer survey. This means that context abstractions are still giving headaches to Scala developers, especially newcomers. This blog introduces contextual abstractions, first from a Scala 3 point of view, profiting from the API overhaul. Then we will drive home the intuition of contextual abstractions with some code examples,...

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Continue reading

Replacing Docker Desktop with hyperkit + minikube

Replacing Docker Desktop with hyperkit + minikube

Published by Arnon Rotem-Gal-Oz on September 2, 2021

Edit June 2023: Added a section on Colima MacOS is a Unix but it isn’t a Linux so, unfortunately, if/when we need to use linux-y…

Continue reading

Intro to Apache Spark (slides)

Intro to Apache Spark (slides)

Published by Arnon Rotem-Gal-Oz on December 16, 2020

I gave a general overview of Apache Spark to our R&D teams. You can find the slides below

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Continue reading

Intro to Docker & Kubernetes (slides)

Intro to Docker & Kubernetes (slides)

Published by Arnon Rotem-Gal-Oz on May 25, 2020

Just had a 2 hours session with my team introducing Docker and Kubernetes (presentation and hands-on). I thought the slides might be useful to others: