pyspark Installing Spark on Ubuntu in 3 Minutes One thing I hear often from people starting out with Spark is that it’s too difficult to install. Some guides are for Spark 1.x and others are for 2.x. Some guides get really detailed with Hadoop versions, JAR files, and
apache Apache Spark on Google Colaboratory Google recently launched a preview of Colaboratory, a new service that lets you edit and run IPython notebooks right from Google Drive – free! It’s similar to Databricks – give that a try if you’re looking for a better-supported way
data Writing Huge CSVs Easily and Efficiently with PySpark I recently ran into a use case that the usual Spark CSV writer didn’t handle very well – the data I was writing had an unusual encoding, odd characters, and was really large. I needed a way to use the Python unicodecsv
aws Jupyter Notebooks with PySpark on AWS EMR One of the biggest, most time-consuming parts of data science is analysis and experimentation. One of the most popular tools to do so in a graphical, interactive environment is Jupyter. Combining Jupyter with Apache Spark (through PySpark) merges two extremely powerful tools. AWS EMR
iot Amazon Dash Button Hackery with Python I bought a few Amazon Dash Buttons as part of Prime Day. These are the cheaper $4.99 buttons, not the more expensive $19.99 AWS IoT Buttons. In this blog post I’ll walk through how to make these cheaper buttons do
python Jupyter Spark Integration https://github.com/mozilla/jupyter-spark Jupyter Notebook extension for Apache Spark integration. Includes a progress indicator for the current Notebook cell if it invokes a Spark job. Queries the Spark UI service on the backend to get the required Spark job information. This is
books Fluent Python - Microreview If you really want to get into the details of Python and learn about how the language was built and how some of its internals are implemented, Fluent Python is the book for you. It’s a great book to refresh
aws Spark and Kinesis Optimization You’ll definitely want to read this if you’re using AWS Kinesis with Apache Spark to stream data, it’s been extremely valuable: https://aws.amazon.com/blogs/big-data/optimize-spark-streaming-to-efficiently-process-amazon-kinesis-streams/
books Flask Web Development - Miguel Grinberg Microreview If you’re just getting started with Flask or you want to learn about the innards of Django (yep, that’s right), “Flask Web Development” is the perfect place to start. This book dives right in with creating a full
flask Flask Quick Startup Project I like to start my projects using Flask and Python because it’s fast and quick for most things, yet lightweight. By default, Flask doesn’t give you much in terms of test frameworks, application settings, deployment, or running the application in
python DiskDict - Python dictionaries stored on disk This weekend while running a rather large Python job, I ran into a memory error. It turned out that a dictionary I was populating could potentially become too big to fit into RAM. This is where DiskDict saved me some time. https://github.com/
django Really Simple OAuth v1 with Django Time and time again, developers stumble upon APIs using OAuth. I’ve recently added Fitbit integration to an application I’m working on. FitBit’s API uses OAuth v1 for authentication, and using OAuth with Django was really straightforward. Here’