Spark to Google Cloud Storage

This is the last post in my series of how to connect Spark to various data sources. Here is how to connect to Google Cloud Storage using Spark 3.x. First, create a service account in GCP, then download the JSON key file. Save it somewhere secure (e.g. import it into Vault or Secrets Manager). Grant …

Spark to Azure Data Lake Storage Gen1

This is another quick post for how to connect Spark to various platforms. I used Azure Data Lake Storage on a project in the past and had a tough time figuring out what to do (there are huge differences between Azure Blob Storage, Azure Data Lake Gen1, and Azure Data Lake Gen2). This guide assumes …

The Manager's Path

Currently reading: The Manager’s Path: A Guide for Tech Leaders Navigating Growth and Change by Camille Fournier 📚