{"id":4091,"date":"2017-12-23T10:25:37","date_gmt":"2017-12-23T15:25:37","guid":{"rendered":"http:\/\/hoolihan.net\/blog-tim\/?p=4091"},"modified":"2017-12-28T11:59:17","modified_gmt":"2017-12-28T16:59:17","slug":"hadoop-installing-macos","status":"publish","type":"post","link":"http:\/\/hoolihan.net\/blog-tim\/2017\/12\/23\/hadoop-installing-macos\/","title":{"rendered":"Hadoop: Installing on macOS"},"content":{"rendered":"<p>Hadoop is traditionally run on a linux-based system. For learning and development purposes, you may want to install hadoop on macOS. <\/p>\n<p>This is the first in a <a href=\"\/\/hoolihan.net\/blog-tim\/?s=hadoop\">series<\/a> of posts that will walkthrough working with Hadoop and cloud-based storage. <\/p>\n<p>First, you&#8217;ll want to use homebrew to install hadoop and any related tools you would like.<br \/>\n<code>brew install hadoop apache-spark pig hbase<\/code><\/p>\n<p>Next, you&#8217;ll want to setup some environment variables. This can be in your shell rc file (.bashrc, .zshrc), or other places if you use a shell config tool like oh-my-zsh. <\/p>\n<p>Make sure you have set JAVA_HOME, which may differ from my setup below.<\/p>\n<p><code>export HADOOP_INSTALL=\/usr\/local\/opt<br \/>\nexport HADOOP_HOME=$HADOOP_INSTALL\/hadoop\/libexec<br \/>\nexport HADOOP_CONF=$HADOOP_HOME\/etc\/hadoop<br \/>\nPATH=\"$HADOOP_HOME\/sbin:$HADOOP_HOME\/bin:$PATH\"<\/code><\/p>\n<p>Then test your install with the following:<\/p>\n<p><code>hdfs dfs -ls ~<\/code><\/p>\n<p>You should see the contents of your home directory.<\/p>\n<p>You can also run a hadoop example with:<br \/>\n<code>yarn jar $HADOOP_HOME\/share\/hadoop\/mapreduce\/hadoop-mapreduce-examples-*.jar pi 10 100<\/code><\/p>\n<p>You should see a (poor) estimate of pi.<\/p>\n<p>You should now be set to use hadoop. In future posts we will look at using the S3 filesystem from AWS and the Google Cloud Storage as well.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hadoop is traditionally run on a linux-based system. For learning and development purposes, you may want to install hadoop on macOS. This is the first in a series of posts that will walkthrough working with Hadoop and cloud-based storage. First, you&#8217;ll want to use homebrew to install hadoop and any related tools you would like. [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55,306,332],"tags":[333,336,335,334],"class_list":["post-4091","post","type-post","status-publish","format-standard","hentry","category-data","category-data-science","category-hadoop","tag-hadop","tag-homebrew","tag-macos","tag-mapreduce"],"_links":{"self":[{"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/posts\/4091","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/comments?post=4091"}],"version-history":[{"count":0,"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/posts\/4091\/revisions"}],"wp:attachment":[{"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/media?parent=4091"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/categories?post=4091"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/hoolihan.net\/blog-tim\/wp-json\/wp\/v2\/tags?post=4091"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}