Hadoop: Accessing Google Cloud Storage

First, go here to choose the hadoop google cloud storage connector for your version of hadoop, likely hadoop 2.

Copy that file to $HADOOP_HOME/share/hadoop/tools/lib/. If you followed the instruction in the prior post, that directory is already in your class path. If not, add the following to your hadoop-env.sh file (found in $HADOOP_CONF directory):

#GS / AWS S3 Support
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/*

Create a service account in google cloud that has the necessary Storage permissions. Download the credentials and save somewhere, in my case I renamed the file and saved it in .config/gcloud/hadoop.json.

Add the following properties in your core-site.xml:
<configuration>
<property>
<name>fs.gs.project.id</name>
<value>someproject-123</value>
<description>
Required. Google Cloud Project ID with access to configured GCS buckets.
</description>
</property>

<property>
<name>google.cloud.auth.service.account.enable</name>
<value>true</value>
<description>
Whether to use a service account for GCS authorizaiton.
</description>
</property>

<property>
<name>google.cloud.auth.service.account.json.keyfile</name>
<value>/Users/tim/.config/gcloud/hadoop.json</value>
</property>

<property>
<name>fs.gs.impl</name>
<value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem</value>
<description>The implementation class of the GS Filesystem</description>
</property>

<property>
<name>fs.AbstractFileSystem.gs.impl</name>
<value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS</value>
<description>The implementation class of the GS AbstractFileSystem.</description>
</property>

</configuration>

Note to change someproject-123 to your actual project-id, which can be found in the google cloud dashboard.

Now test this setup with:

hdfs dfs -ls gs://somebucket

Of course you’ll need to replace somebucket with an actual bucket/directory in your google storage account.

Now you should be setup to use S3 and Google storage with your local hadoop setup.

Hadoop: Installing on macOS

Hadoop is traditionally run on a linux-based system. For learning and development purposes, you may want to install hadoop on macOS.

This is the first in a series of posts that will walkthrough working with Hadoop and cloud-based storage.

First, you’ll want to use homebrew to install hadoop and any related tools you would like.
brew install hadoop apache-spark pig hbase

Next, you’ll want to setup some environment variables. This can be in your shell rc file (.bashrc, .zshrc), or other places if you use a shell config tool like oh-my-zsh.

Make sure you have set JAVA_HOME, which may differ from my setup below.

export HADOOP_INSTALL=/usr/local/opt
export HADOOP_HOME=$HADOOP_INSTALL/hadoop/libexec
export HADOOP_CONF=$HADOOP_HOME/etc/hadoop
PATH="$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$PATH"

Then test your install with the following:

hdfs dfs -ls ~

You should see the contents of your home directory.

You can also run a hadoop example with:
yarn jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi 10 100

You should see a (poor) estimate of pi.

You should now be set to use hadoop. In future posts we will look at using the S3 filesystem from AWS and the Google Cloud Storage as well.