{"id":4098,"date":"2017-12-23T11:33:05","date_gmt":"2017-12-23T16:33:05","guid":{"rendered":"http:\/\/hoolihan.net\/blog-tim\/?p=4098"},"modified":"2017-12-23T11:33:05","modified_gmt":"2017-12-23T16:33:05","slug":"hadoop-accessing-google-cloud-storage","status":"publish","type":"post","link":"http:\/\/hoolihan.net\/blog-tim\/2017\/12\/23\/hadoop-accessing-google-cloud-storage\/","title":{"rendered":"Hadoop: Accessing Google Cloud Storage"},"content":{"rendered":"

First, go here<\/a> to choose the hadoop google cloud storage connector for your version of hadoop, likely hadoop 2.<\/p>\n

Copy that file to $HADOOP_HOME\/share\/hadoop\/tools\/lib\/. If you followed the instruction in the prior post<\/a>, that directory is already in your class path. If not, add the following to your hadoop-env.sh file (found in $HADOOP_CONF directory):<\/p>\n

#GS \/ AWS S3 Support
\nexport HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_HOME\/share\/hadoop\/tools\/lib\/*<\/code><\/p>\n

Create a service account in google cloud that has the necessary Storage permissions. Download the credentials and save somewhere, in my case I renamed the file and saved it in .config\/gcloud\/hadoop.json.<\/p>\n

Add the following properties in your core-site.xml:
\n<configuration>
\n <property>
\n <name>fs.gs.project.id<\/name>
\n <value>someproject-123<\/value>
\n <description>
\n Required. Google Cloud Project ID with access to configured GCS buckets.
\n <\/description>
\n <\/property><\/p>\n

<property>
\n <name>google.cloud.auth.service.account.enable<\/name>
\n <value>true<\/value>
\n <description>
\n Whether to use a service account for GCS authorizaiton.
\n <\/description>
\n <\/property><\/p>\n

<property>
\n <name>google.cloud.auth.service.account.json.keyfile<\/name>
\n <value>\/Users\/tim\/.config\/gcloud\/hadoop.json<\/value>
\n <\/property><\/p>\n

<property>
\n <name>fs.gs.impl<\/name>
\n <value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem<\/value>
\n <description>The implementation class of the GS Filesystem<\/description>
\n <\/property><\/p>\n

<property>
\n <name>fs.AbstractFileSystem.gs.impl<\/name>
\n <value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS<\/value>
\n <description>The implementation class of the GS AbstractFileSystem.<\/description>
\n <\/property><\/p>\n

<\/configuration><\/code><\/p>\n

Note to change someproject-123 to your actual project-id, which can be found in the google cloud dashboard. <\/p>\n

Now test this setup with:<\/p>\n

hdfs dfs -ls gs:\/\/somebucket<\/code><\/p>\n

Of course you’ll need to replace somebucket with an actual bucket\/directory in your google storage account. <\/p>\n

Now you should be setup to use S3 and Google storage with your local hadoop setup.<\/p>\n","protected":false},"excerpt":{"rendered":"

First, go here to choose the hadoop google cloud storage connector for your version of hadoop, likely hadoop 2. Copy that file to $HADOOP_HOME\/share\/hadoop\/tools\/lib\/. If you followed the instruction in the prior post, that directory is already in your class path. If not, add the following to your hadoop-env.sh file (found in $HADOOP_CONF directory): #GS […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55,306,332,19],"tags":[340,337,335],"class_list":["post-4098","post","type-post","status-publish","format-standard","hentry","category-data","category-data-science","category-hadoop","category-os-x","tag-google-cloud","tag-hadoop","tag-macos"],"_links":{"self":[{"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/posts\/4098","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/comments?post=4098"}],"version-history":[{"count":0,"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/posts\/4098\/revisions"}],"wp:attachment":[{"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/media?parent=4098"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/categories?post=4098"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/tags?post=4098"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}