Are You a Stable or Volatile?

I've written before on this blog about the different personality traits of software engineers, and how different traits balance each other on a team.Some of the spectra I came up with were:Dreamers ↔ PragmatistsBig Picture ↔ Detail-OrientedMove Fast & Break Things ↔ Slow & MethodicalOptimists ↔ PessimistsAnswerers ↔ QuestionersRecently I read a comment on Hacker News (which I can't find...
Meta Uses LLMs to Improve Incident Response

In June, Meta released an article titled Leveraging AI for efficient incident response on their engineering blog. In this article, engineers outline how they leveraged large language models to improve Meta's incident response capabilities. The headline metric from this report: Meta was able to use LLMs to successfully root cause incidents with 42% accuracy in their web monorepo. This means...
Understanding the BM25 full text search algorithm

submitted by /u/emschwartz [link] [comments]
Offset Considered Harmful or: The Surprising Complexity of Pagination in SQL

submitted by /u/TheCrush0r [link] [comments]
Building SQL-based Observability With ClickHouse and Grafana

submitted by /u/mmaksimovic [link] [comments]
The Root Cause of Why Organizations Fail With Data & AI

Wave by CLEANPNGDEMYSTIFY DATA STRATEGYA guide to be successful with the strategic groundwork requiredBusiness value through Data and Artificial Intelligence. Everybody talks about it, yet most companies are struggling to monetize their data. I claim that in most cases, this is due to a lack of effective business strategy. This article shows what strategic groundwork is necessary to activate your company’s data assets.This...

Tag: PySpark

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…