Creating an IP Lookup Table of Activities in a SIEM Architecture

When working with cyber security data, one thing is for sure: there is no shortage of available data sources. If anything, there are too many data sources with overlapping data. Your traditional SIEM (security information and event management) tool is not really fit to handle the complex transformation and stitching of multiple data sources in an efficient manner. Besides, it...
The release of pyspark package 3.2.0 is now available. 🥳 Apache Spark Python API #sql #java #jdbc #scala #spark #python #pyspark #bigdata #programming

The release of pyspark package 3.2.0 is now available. 🥳Apache Spark Python APIgithub.com/apache/spark #r #sql #java #jdbc #scala #spark #python #pyspark #bigdata #programming- PyPackages (@pypackages)01:53 - Oct 18, 2021
How Time Series Databases Work, and Where They Don’t

Comments
I've written a DSL (kotlin) to transform JSON objects from one format to another

submitted by /u/nomemory [link] [comments]
Presentation: Panel: Event Driven Architectures of Scale

Wes Reisz, Matthew Clark, Gwen Shapira, and Ian Thomas discuss the evolution of event-driven architectures over the decades, the advantages that EDA offers, and thoughts for the future. By
Wes Reisz, Matthew Clark, Gwen Shapira, Ian Thomas
Python 3.10 — Five New Features And Considerations | by Christopher Tao | Oct, 2021 | Towards Data Science

Image by Joshua Woroniecki from PixabayPython 3.10 — Five New Features And ConsiderationsNot only listing but also the examples and considerations.A few days ago, Python 3.10 has released eventually. There are already many articles online that were even published before it was released. However, I found most of those are just listing the new features without too much discussion. Therefore, in my article,...

Category: Big Data

Intro to Apache Spark (slides)

Published by Arnon Rotem-Gal-Oz on December 16, 2020

I gave a general overview of Apache Spark to our R&D teams. You can find the slides below

Where is Apache Spark heading?

Where is Apache Spark heading?

Published by Arnon Rotem-Gal-Oz on December 4, 2020

I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…

Big data isn’t – well, almost

Big data isn’t – well, almost

Published by Arnon Rotem-Gal-Oz on March 23, 2019

Back in ancient history (2004) Google’s Jeff Dean & Sanjay Ghemawat presented their innovative idea for dealing with huge data sets – a novel idea…

Big data in the cloud – welcome to cost oriented design

Big data in the cloud – welcome to cost oriented design

Published by Arnon Rotem-Gal-Oz on April 1, 2016

A couple of weeks ago I presented @ BDX2016 The slides are available on Slideshare Big data in the cloud – welcome to cost oriented…

Spark, Parquet and S3 – It’s complicated.

Spark, Parquet and S3 – It’s complicated.

Published by Arnon Rotem-Gal-Oz on August 10, 2015

(A version of this post was originally posted in AppsFlyer’s blog. Also special thanks to Morri Feldman and Michael Spector from AppsFlyer data team that…