spark - Mike Staszel

S3A on Spark

Quick post mostly for my own reference since I always need to re-learn how to do this. This used to be more difficult in older versions of Spark, but on 2.4/3.0, all you have to do is: wget https://repo1.maven.

pyspark

Installing Spark on Ubuntu in 3 Minutes

One thing I hear often from people starting out with Spark is that it’s too difficult to install. Some guides are for Spark 1.x and others are for 2.x. Some guides get really detailed with Hadoop versions, JAR files, and

github

Spark + Scala Boilerplate Project

After setting up a few Spark + Scala projects I decided to open-source a boilerplate sample project that you can import right into IntelliJ and build with one command. Usually I write Apache Spark code in Python, but there are a few times I prefer

apache

Apache Spark on Google Colaboratory

Google recently launched a preview of Colaboratory, a new service that lets you edit and run IPython notebooks right from Google Drive – free! It’s similar to Databricks – give that a try if you’re looking for a better-supported way

data

Writing Huge CSVs Easily and Efficiently with PySpark

I recently ran into a use case that the usual Spark CSV writer didn’t handle very well – the data I was writing had an unusual encoding, odd characters, and was really large. I needed a way to use the Python unicodecsv

aws

Jupyter Notebooks with PySpark on AWS EMR

One of the biggest, most time-consuming parts of data science is analysis and experimentation. One of the most popular tools to do so in a graphical, interactive environment is Jupyter. Combining Jupyter with Apache Spark (through PySpark) merges two extremely powerful tools. AWS EMR

python

Jupyter Spark Integration

https://github.com/mozilla/jupyter-spark Jupyter Notebook extension for Apache Spark integration. Includes a progress indicator for the current Notebook cell if it invokes a Spark job. Queries the Spark UI service on the backend to get the required Spark job information. This is

aws

Spark and Kinesis Optimization

You’ll definitely want to read this if you’re using AWS Kinesis with Apache Spark to stream data, it’s been extremely valuable: https://aws.amazon.com/blogs/big-data/optimize-spark-streaming-to-efficiently-process-amazon-kinesis-streams/

apache

Apache Spark Cluster with Vagrant

New tiny GitHub project: https://github.com/mikestaszel/spark_cluster_vagrant Over the past few weeks I’ve been working on benchmarking Spark as well as learning more about setting up clusters of Spark machines both locally and on cloud providers. I decided