Codon: a high-performance Python compiler using LLVM

submitted by /u/arshajii [link] [comments]
Data Mesh vs Data Fabric: a tale of two new data paradigms

Posted on December 8, 2022 Guest post originally published on Mia-Platform’s blog Data is one of the most critical components of any business, as it allows us to personalize and customize our products for potential consumers. As important as data is, studies have shown that about 50‑70% of data collected by organizations goes unused and becomes what Gartner calls Dark Data. We can attribute this...
Streamed Hashing Added to Komihash 4.6. Very Fast Hash Function (in C), competitor to xxhash, wyhash. Java port available.

submitted by /u/avaneev [link] [comments]
Radar Trends to Watch: December 2022

This month’s news has been overshadowed by the implosion of SBF’s TFX and the possible implosion of Elon Musk’s Twitter. All the noise doesn’t mean that important things aren’t happening. Many companies, organizations, and individuals are wrestling with the copyright implications of generative AI. Google is playing a long game: they believe that the goal isn’t to imitate art works,...
Why and How eBay Pivoted to OpenTelemetry

eBay makes a crucial pivot to OpenTelemetry to better align with industry standards for Observability.
How Apache Arrow Is Changing the Big Data Ecosystem

One of the biggest challenges of working with big data is the performance overhead involved with moving data between different tools and systems as part of your data processing pipeline. Different programming languages, file formats and network protocols all have different ways of representing the same data in memory. The process of serializing and deserializing the same data into a different...

Category: Big Data

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Intro to Apache Spark (slides)

Published by Arnon Rotem-Gal-Oz on December 16, 2020

I gave a general overview of Apache Spark to our R&D teams. You can find the slides below

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Big data isn’t – well, almost

Big data isn’t – well, almost

Published by Arnon Rotem-Gal-Oz on March 23, 2019

Back in ancient history (2004) Google’s Jeff Dean & Sanjay Ghemawat presented their innovative idea for dealing with huge data sets – a novel idea…

Big data in the cloud – welcome to cost oriented design

Big data in the cloud – welcome to cost oriented design

Published by Arnon Rotem-Gal-Oz on April 1, 2016

A couple of weeks ago I presented @ BDX2016 The slides are available on Slideshare Big data in the cloud – welcome to cost oriented…