Data Modernization and Reasoning Step 2: Big Data — distributed data design basics and further…

Data Modernization and Reasoning Step 2: Big Data — distributed data design basics and further refinements from STEP 1You use SQL, partition etc in Spark but do not miss out key requirements in parallel computingSTEP 1 reviews existing data system and reveals issues that need to be fixed. Now we go on real data modernization stage and initialize design phase. Cloud based big...
Amazon EMR 7.5 runtime for Apache Spark and Iceberg can run Spark workloads 3.6 times faster than Spark 3.5.3 and Iceberg 1.6.1

The Amazon EMR runtime for Apache Spark offers a high-performance runtime environment while maintaining 100% API compatibility with open source Apache Spark and Apache Iceberg table format. Amazon EMR on EC2, Amazon EMR Serverless, Amazon EMR on Amazon EKS, Amazon EMR on AWS Outposts and AWS Glue all use the optimized runtimes. In this post, we demonstrate the performance benefits...
Fail Fast, Build Better: How software landscape is changing and embracing failures

submitted by /u/codekidX [link] [comments]
Complain & Propose

Republished from July 2014.͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ...
Use of Time in Distributed Databases (part 2): Use of logical clocks in databases

This is part 2 of our "Use of Time in Distributed Databases" series. We will talk about the use of logical clocks in databases in this post. We will consider three different approaches:vector clocksdependency graph maintenanceepoch service In the upcoming posts we will allow in physical clocks for timestamping, so there is no (almost no) physical clocks involved in the systems...
My DOs and DON’Ts of Software Architecture

Collated from my personal experience as a SW ArchitectPhoto by Pixabay on PexelsThis advice is based on part of a Meetup talk I gave on “Building the Product Right”.As the new year approaches, it’s a time of reflection, and this extends into the workplace. So I thought I’d share some of my own personal “do’s” and “don’ts” at work, which I’ve formulated...

Category: Blog

Is there a future for Map/Reduce?

Published by Arnon Rotem-Gal-Oz on June 3, 2014

Google’s Jeffrey Dean and Sanjay Ghemawat filed the patent request and published the map/reduce paper 10 year ago (2004). According to WikiPedia Doug Cutting and Mike Cafarella created Hadoop,…

Hadoop YARN overview

Published by Arnon Rotem-Gal-Oz on May 27, 2014

I did a short overview of Hadoop YARN to our big data development team. The presentation covers the motivation for YARN, how it works and…

Services, Microservices, Nanoservices – oh my!

Services, Microservices, Nanoservices – oh my!

Published by Arnon Rotem-Gal-Oz on March 25, 2014

Apparently, there’s this new distributed architecture thing called microservices out and about – so last week, I went ahead and read Martin Fowler’s & James…

and with YARN the game changes

and with YARN the game changes

Published by Arnon Rotem-Gal-Oz on October 17, 2013

I’ve been working with Hadoop for a few years now and the platform and ecosystems has been advancing at an amazing pace with new features…

A couple of SOA Q&As

Published by Arnon Rotem-Gal-Oz on September 6, 2013

Every now and then I get some question by email, I usually just answer them directly but considering I got 2 such questions this week…