Skip to content

ChibiHash: Small, Fast 64 bit hash function

ChibiHash: Small, Fast 64 bit hash function 16 Nov 2024 If you need a small and fast 64 bit hash function that can be copy-pasted easily, then here's one that I cooked up in an hour or so: chibihash64.h. Some key features: Small: ~60 loc in C Fast: See benchmark table below Portable: Doesn't use hardware specific instructions (e.g SSE) Good Quality: Passes smhasher, so should be good...
The myth that you can’t build interactive web apps except as single page app

You Can't Build Interactive Web Apps Except as Single Page Applications... And Other Myths Tony Alaribe September 20, 2024#An Ode to Browser Advancements. I often encounter discussions on Reddit and YCombinator where newer developers seek tech stack advice. Inevitably, someone claims it’s impossible to build a high-quality application without using a single-page application (SPA) framework like React or AngularJS. This strikes me as odd because,...
Spoiler Alert: The Magic of RAG Does Not Come from AI

Why retrieval, not generation, makes RAG systems magicalQuick POCsMost quick proof of concepts (POCs) which allow a user to explore data with the help of conversational AI simply blow you away. It feels like pure magic when you can all of a sudden talk to your documents, or data, or code base.These POCs work wonders on small datasets with a limited count...
The Mouse Roars

Today in 1970, Douglas Engelbart received a patent for his invention of the first computer mouse.In a 2012 interview, Engelbart saidThe actual invention of the mouse was the result of analyzing the various characteristics of other pointing/input devices. Much as the Periodic Table of the Elements has characteristics which define groups along rows and columns, we laid out a grid...
DuckDB Optimizers: The Low-Key MVP

Optimizers: The Low-Key MVP TL;DR: The query optimizer is an important part of any analytical database system as it provides considerable performance improvements compared to hand-optimized queries, even as the state of your data changes.
Optimizers don't often give "main character" energy in the database community. Databases are usually popular because of their performance, ease of integration, or reliability. As someone who...
Replicate changes from databases to Apache Iceberg tables using Amazon Data Firehose (in preview)

Replicate changes from databases to Apache Iceberg tables using Amazon Data Firehose (in preview)

Tag: Spark

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Continue reading

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Continue reading

Spark, Parquet and S3 – It’s complicated.

Spark, Parquet and S3 – It’s complicated.

Published by Arnon Rotem-Gal-Oz on August 10, 2015

(A version of this post was originally posted in AppsFlyer’s blog. Also special thanks to Morri Feldman and Michael Spector from AppsFlyer data team that…

Continue reading

Apache Spark, ETL and Parquet

Apache Spark, ETL and Parquet

Published by Arnon Rotem-Gal-Oz on September 14, 2014

(Edit 10/8/2015 : A lot has changed in the last few months – you may want to check out my new post on Spark, Parquet…

Continue reading

Is there a future for Map/Reduce?

Is there a future for Map/Reduce?

Published by Arnon Rotem-Gal-Oz on June 3, 2014

Google’s Jeffrey Dean and Sanjay Ghemawat filed the patent request and published the map/reduce paper 10 year ago (2004). According to WikiPedia Doug Cutting and Mike Cafarella created Hadoop,…

Continue reading