Show HN: Trigger.dev V2 – a Temporal alternative for TypeScript devs

Back in February, we posted a Show HN about building a “developer-first open source Zapier alternative” (https://news.ycombinator.com/item?id=34610686). This was v1 of Trigger.dev.During the months since, we’ve gathered a lot of feedback from early users and realized that what developers actually wanted was more like an easier-to-use Temporal with integrations.Here’s what we’ve learned so far:- Serverless timeouts make it hard for...
Paper Notes: Spanner – Google’s Globally-Distributed Database

submitted by /u/varunu28 [link] [comments]
Scribe: Improving Reliability One Nine at a Time

Why does Scribe need more nines? Scribe is a highly scalable, distributed, durable queue that allows writing and reading large-volume streams of data. As of today it serves more than nine terabytes (TB) per second of writes and more than 30 TB per second of reads, providing less than five seconds of latency—that’s in the 90th percentile. Scribe was initially...
Training Foundation Improvements for Closeup Recommendation Ranker

Fan Jiang | Software Engineer, Closeup Candidate Retrieval; Liyao Lu | Software Engineer, Closeup Ranking & Blending; Laksh Bhasin | Software Engineer, Core ML Foundations; Chen Yang | Software Engineer, Core ML Foundations; Shivin Thukral | Software Engineer, Closeup Ranking & Blending; Travis Ebesu | Software Engineer, Closeup Ranking & Blending; Kent Jiang | Software Engineer, Core Serving Infra; Yan...
Cloudy Forecast: How Predictable is Communication Latency in the Cloud?

Many, if not all, practical distributed systems rely on partial synchrony in one way or another, be it a failure detection, a lease mechanism, or some optimization that takes advantage of synchrony to avoid doing a bunch of extra work. These partial synchrony approaches need to know some crucial parameters about their world to estimate how much synchrony to expect....
Yandex open sourced it's BI tool DataLens

Pinned A modern, scalable analytics system ...

Category: Big Data

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)

Published by Arnon Rotem-Gal-Oz on October 16, 2022

Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…

Intro to Apache Spark (slides)

Published by Arnon Rotem-Gal-Oz on December 16, 2020

I gave a general overview of Apache Spark to our R&D teams. You can find the slides below

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Big data isn’t – well, almost

Big data isn’t – well, almost

Published by Arnon Rotem-Gal-Oz on March 23, 2019

Back in ancient history (2004) Google’s Jeff Dean & Sanjay Ghemawat presented their innovative idea for dealing with huge data sets – a novel idea…

Big data in the cloud – welcome to cost oriented design

Big data in the cloud – welcome to cost oriented design

Published by Arnon Rotem-Gal-Oz on April 1, 2016

A couple of weeks ago I presented @ BDX2016 The slides are available on Slideshare Big data in the cloud – welcome to cost oriented…