Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below)…
The nexus of technology, business & people
Cirrus Minor Posts
Edit June 2023: Added a section on Colima MacOS is a Unix but it isn’t a Linux so, unfortunately, if/when we need to use linux-y…
I gave a general overview of Apache Spark to our R&D teams. You can find the slides below
I watched (COVID19-era version of “attended”) the latest spark Summit and in one of the keynotes Reynold Xin from Databricks, presented the following two images…
Just had a 2 hours session with my team introducing Docker and Kubernetes (presentation and hands-on). I thought the slides might be useful to others: