Posted by: Wildan Maulana | October 15, 2009

Project ideas for Hadoop

Projects ideas for Hadoop. I get this interesting post on hadoop mailing list.

If this is for an undergraduate class, I would suggest something that
allows you to get some work in with basic data structures such as
building an inverted index over a few million documents (maybe Wikipedia
pages?). You will also need to get a general feel for Hadoop.

The University of Washington has some really nice project ideas for
their distributed systems class:

http://www.cs.washington.edu/education/courses/cse490h/09wi/projects/490H.project.ideas.pdf

If you wanted to tackle something a little more advanced, then you could
take a look at Pete Skomoroch’s article on finding trends with Hadoop
and Hive:

http://www.cloudera.com/blog/2009/07/31/tracking-trends-with-hadoop-and-
hive-on-ec2/

http://www.cloudera.com/blog/2009/09/28/grouping-related-trends-with-had
oop-and-hive/

Things to keep in mind:

1.) Hadoop wont be as simple as writing a single Java app
2.) There will be some overhead involved in re-writing algorithms in Map
Reduce
3.) There will also be some overhead involved in setup and maintenance
of the Hadoop Cluster

Take these three things into account when planning how to manage your
time for the project during the semester, semesters can seem a lot
shorter when you spend too much time on things not related to just
implementing and testing your algorithm.

Good luck!

Josh Patterson
TVA


Responses

  1. Thanks🙂

  2. I was just looking for some projects ideas for undergraduate course!
    And your first idea seems interesting ! Could you share some more thought about it?
    Also similar projects suitable for UG course!

    Thanks


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: