VisualStudioFeeds

All Visual Studio blogs in one place

AddThis Social Bookmark Button
More F# Hadoop goodness from Carl Nolan. This time a compositional framework for submitting F# and C# Azure/Hadoop map-reduce jobs.    Framework for Composing and Submitting .Net Hadoop MapReduce Jobs If you have been following my blog you will see that I have been putting together samples for writing .Net Hadoop MapReduce jobs; using Hadoop Streaming. However one thing that became apparent is that the samples could be reconstructed in a composable  framework to enable one to submit .Net based MapReduce jobs whilst only writing Mappers and Reducers types. To this end I have put together a framework that allows one to
submit MapReduce jobs using the following command line syntax: MSDN.Hadoop.Submission.Console.exe -input "mobile/data/debug/sampledata.txt" -output "mobile/querytimes/debug" -mapper "MSDN.Hadoop.MapReduceFSharp.MobilePhoneQueryMapper,MSDN.Hadoop.MapReduceFSharp"  -reducer "MSDN.Hadoop.MapReduceFSharp.MobilePhoneQueryReducer,MSDN.Hadoop.MapReduceFSharp"  -file "%HOMEPATH%MSDN.Hadoop.MapReducebinReleaseMSDN.Hadoop.MapReduceFSharp.dll" Where the mapper and reducer parameters are .Net types that derive from a base Map and Reduce abstract classes. The input, output, and files options are analogous to the standard Hadoop streaming submissions. The mapper and reducer options (more on a combiner option later) allow one to define a .Net type derived from the appropriate abstract base classes. Under the covers standard Hadoop Streaming is being used, where controlling executables are used to handle the StdIn and StdOut operations and activating the required .Net...(Read whole news on source site)

Search

Subscribe

Enter your email address:

Delivered by FeedBurner

Increase your website traffic with Attracta.com

Contact

Email:
Subject:
Message:
Anti-spam: How many eyes has a typical person?

Follow us on FB