Mongodb Map-Reduce

Mongodb Map-Reduce

Map-reduce is a data processing paradigm for condensing large volumes of data into useful aggregated results.

In this map-reduce operation, MongoDB applies the map phase to each input document (i.e. the documents in the collection that match the query condition)

The map function emits key-value pairs. For those keys that have multiple values, MongoDB applies the reduce phase, which collects and condenses the aggregated data. MongoDB then stores the results in a collection.

The output of the reduce function may pass through a finalize function to further condense or process the results of the aggregation.

Why we need Map-Reduce

Map-reduce operations provide some flexibility that is not presently available in the aggregation pipeline.

  • Keeping the results of M/R in a separate collection and updating it from time to time. more
  • Complex queries on large sharded data sets
  • Queries that are so complex that we can’t use the aggregation framework.
  • Map-Reduce runs javascript methods , we have the full power of the language.

Systax

db.collection.mapReduce(
  function(){emit(key, value);}, // map function
  function(key, values) {return reductionFunction},
  {
    out : {[action]: collection},
    query: document.
      sort: document,
  limit: number,
  finalize: finalizeFunction
}
)

In the above syntax

map is a javascript function that maps a value with a key and emits a key-value pair

reduce is a javascript function that reduces or groups all the documents having the same key

out specifies the location of the map-reduce query result

action merge, replace, reduce

query specifies the optional selection criteria for selecting documents

sort specifies the optional sort criteria

limit specifies the optional maximum number of documents to be returned

finalize is a javascript function that modifies reduced result.

Example

In this example, we will try to find out the average sales per customer

[{
 cust_name: "Mr. John",
 ord_date: ISODate("2017-09-03T14:17:00.000Z"),
 status: 'A',
 price: 35
},
{
 cust_name: "Mr. Andrew",
 ord_date: ISODate("2017-09-10T14:17:00.000Z"),
 status: 'A',
 price: 45
},
{
 cust_name: "Mr. John",
 ord_date: ISODate("2017-09-12T14:17:00.000Z"),
 status: 'A',
 price: 27
}]

Firstly, Define the map function to process each input document:

var mapFunction = function() {
  emit(this.cust_name, {count: 1, price: this.price})
}

In the function, this refers to the document that the map-reduce operation is processing

Secondly, Define the corresponding reduce function with two arguments keyCustName and valuesPrices:

var reduceFunction = function(keyCustName, valuesPrices) {
  reducedVal = {count: 0, price: 0}
  for (var idx = 0; idx <valuesPrices.length; idx++) {
    reducedVal.count +=valuesPrices[idx].count;
    reducedVal.price +=valuesPrices[idx].price;
  }
  return reducedVal;
}

Thirdly, Define a finalize function with two arguments key and reducedVal.

var finalizeFunction = function(key, reducedVal) {
  reducedVal.avg = reducedVal.price / reducedVal.count;
  return reducedVal;
}

Finally, Perform the map-reduce operation on the customer_orders collection using the mapFunctionreduceFunction, and finalizeFunction functions.

db.customer_orders.mapReduce(
  mapFunction, 
  reduceFunction, 
  { 
    out: "customer_sales", 
    finalize: finalizeFunction 
  });

 

Now, we can get our final output collection as follows:

/* 1 */
{
  "_id" : "Mr. Andrew",
  "value" : {
  "count" : 1.0000000000000000,
    "price" : 45.0000000000000000,
    "avg" : 45.0000000000000000
}
}

/* 2 */
{
  "_id" : "Mr. John",
  "value" : {
  "count" : 2.0000000000000000,
    "price" : 62.0000000000000000,
    "avg" : 31.0000000000000000
}
}

At the moment , we will how Mongodb perform incremental Map-Reduce. For that we will insert following document in customer_orders collection.

{
  "_id" : ObjectId("59b8a94324194e5c63fae587"),
  "cust_name" : "Mr. Andrew",
  "ord_date" : ISODate("2017-09-14T14:17:00.000Z"),
  "status" : "A",
  "price" : 40
}

And we will change our Map-reduce command as follows:

db.customer_orders.mapReduce(
  mapFunction,
  reduceFunction,
  {
    out: {reduce: "customer_sales"},
    query: { ord_date: { $gt: ISODate('2017-09-13 00:00:00') } },
    finalize: finalizeFunction
  });

 

here Map-reduce only emit recent document based on query we have applied on it. Output the result to the collection customer_sales but reduce the contents with the results of the incremental map-reduce.

Out put collection as follows:

/* 1 */
{
  "_id" : "Mr. Andrew",
  "value" : {
  "count" : 2.0000000000000000,
    "price" : 85.0000000000000000,
    "avg" : 42.5000000000000000
}
}

/* 2 */
{
  "_id" : "Mr. John",
  "value" : {
  "count" : 2.0000000000000000,
    "price" : 62.0000000000000000,
    "avg" : 31.0000000000000000
}
}

 

If the data is constantly growing , the we can perform an incremental map-reduce rather than performing the map-reduce operation over the entire data set each time.

You May Also Like

About the Author: Md. Delwar Hossain

He has 11 years of experience in developing standalone software and web applications for multiple database platforms. He has been passionate about new tools and technologies. He is positive and trustworthy. He is capable to learn and adapt quickly to different situations. He is a great team player and enjoys leading and mentoring. He is specialized in architecting and building complex web and mobile application. He has strong skills to automate POS, inventory, supply chain, trading export/ import, human resource management, manufacturing and production, distribution management system and hospital management system.

20 Comments

  1. Antoine Griezmann may like to sip yerba mate and hang out grilling meat with his Uruguayan chums but his loyalties were not in question as a goal and an assist fired France into the last four of the World Cup. World Cup 2018 quarter-final – Uruguay 0-2 France, RESULT: Antoine Griezmann and Raphael Varane goals help ease Les Bleus through

  2. Hello there,

    My name is Aly and I would like to know if you would have any interest to have your website here at banglatechnologies.com promoted as a resource on our blog alychidesign.com ?

    We are in the midst of updating our broken link resources to include current and up to date resources for our readers. Our resource links are manually approved allowing us to mark a link as a do-follow link as well
    .
    If you may be interested please in being included as a resource on our blog, please let me know.

    Thanks,
    Aly

  3. Thank you so much for providing individuals with such a superb possiblity to read from this website. It’s always very ideal and also packed with amusement for me personally and my office co-workers to search the blog at a minimum three times every week to study the newest tips you have. And lastly, I’m just usually fulfilled with your exceptional suggestions you serve. Certain 4 facts in this post are surely the most efficient we have ever had.

  4. Thank you for all of your efforts on this web site. My daughter take interest in working on research and it’s obvious why. My spouse and i hear all regarding the powerful method you render valuable solutions through the web blog and as well as foster contribution from some other people on that point plus our favorite girl is undoubtedly being taught a lot of things. Take pleasure in the rest of the new year. Your carrying out a terrific job.

  5. I would like to show appreciation to the writer just for bailing me out of this particular issue. Because of exploring through the the net and meeting notions which were not beneficial, I was thinking my entire life was gone. Existing without the approaches to the difficulties you’ve sorted out by way of the report is a crucial case, and the ones that might have badly affected my career if I had not come across your web page. Your skills and kindness in touching all the pieces was very useful. I am not sure what I would have done if I hadn’t come upon such a point like this. I can now look forward to my future. Thanks for your time very much for this expert and sensible help. I won’t think twice to refer your web sites to anybody who should have guidelines about this issue.

  6. Thanks so much for providing individuals with an extremely splendid possiblity to read from this website. It’s usually so terrific plus full of a great time for me personally and my office colleagues to visit the blog minimum three times per week to find out the fresh guidance you have got. And lastly, I’m usually fascinated concerning the unbelievable techniques served by you. Certain 4 facts on this page are truly the very best we’ve ever had.

  7. I intended to send you the little bit of note to help give many thanks again on the pleasant ideas you’ve shown above. It was certainly remarkably generous with you to present openly just what most of us could possibly have supplied for an ebook to help make some dough for themselves, especially now that you might well have done it if you decided. The guidelines in addition acted like a good way to comprehend most people have similar dreams much like mine to understand a great deal more with respect to this condition. I am certain there are thousands of more enjoyable situations ahead for folks who read carefully your website.

  8. I simply had to thank you very much once more. I’m not certain what I would’ve tried without those opinions discussed by you regarding that industry. It truly was an absolute intimidating issue in my circumstances, nevertheless noticing the specialised tactic you solved the issue took me to cry over contentment. I am grateful for the guidance as well as have high hopes you really know what a powerful job you happen to be getting into training most people by way of your web page. Most probably you’ve never encountered any of us.

  9. I intended to post you one little bit of note to finally say thanks a lot the moment again for those incredible knowledge you have featured above. It has been unbelievably open-handed with you to allow publicly precisely what most people would have advertised for an e book to generate some profit for their own end, particularly seeing that you might well have done it if you wanted. Those smart ideas likewise served as a great way to comprehend other individuals have the same zeal really like my very own to see a whole lot more with regards to this matter. I’m sure there are millions of more pleasant situations in the future for those who go through your website.

  10. I must show some thanks to this writer just for rescuing me from such a instance. As a result of researching throughout the search engines and finding opinions which were not productive, I assumed my entire life was well over. Living devoid of the approaches to the difficulties you’ve sorted out by way of your short article is a critical case, as well as the ones which might have badly affected my career if I hadn’t discovered your blog post. Your primary competence and kindness in taking care of all the pieces was important. I’m not sure what I would’ve done if I had not come upon such a subject like this. I can also now look forward to my future. Thank you so much for this specialized and amazing help. I will not be reluctant to refer your web page to anyone who should receive care about this subject.

  11. I’m just commenting to let you understand of the magnificent discovery my friend’s child experienced reading your web site. She came to find a good number of issues, which included what it is like to possess a very effective giving style to have others completely know just exactly certain tortuous things. You really did more than our desires. Thanks for coming up with such invaluable, safe, informative and cool guidance on that topic to Lizeth.

  12. I precisely wanted to appreciate you all over again. I’m not certain the things I might have implemented in the absence of those information discussed by you over my situation. It was before the distressing setting in my position, but encountering this professional way you handled it forced me to cry with fulfillment. Now i am thankful for this support as well as hope that you know what an amazing job your are doing teaching people today all through your web site. I’m certain you have never come across all of us.

  13. I enjoy you because of your own hard work on this web site. Betty loves engaging in investigations and it is obvious why. Almost all hear all of the powerful form you present advantageous strategies on this web blog and foster participation from others about this subject plus our favorite girl is without question learning a great deal. Take advantage of the rest of the new year. You are performing a superb job.

  14. I not to mention my guys ended up studying the great helpful hints from your web blog and then all of a sudden I had a horrible feeling I never thanked the site owner for those strategies. These men became for that reason stimulated to study all of them and now have absolutely been tapping into them. Appreciate your really being very accommodating as well as for deciding on certain decent things most people are really desirous to know about. My very own sincere regret for not expressing appreciation to you sooner.

  15. Thank you so much for providing individuals with remarkably marvellous possiblity to discover important secrets from here. It is often so nice and full of a great time for me personally and my office fellow workers to search your web site at a minimum three times in 7 days to read through the new items you have got. And of course, I’m usually astounded with all the gorgeous guidelines you give. Selected 3 areas in this article are essentially the finest we have ever had.

  16. Thank you for all of your efforts on this site. My mother delights in doing investigation and it’s obvious why. My spouse and i learn all relating to the dynamic form you produce very important tips and tricks on the web blog and boost response from visitors on this theme then our favorite princess has always been starting to learn a great deal. Have fun with the rest of the year. You have been carrying out a first class job.

  17. My husband and i were absolutely lucky that Emmanuel could conclude his web research by way of the ideas he obtained out of your weblog. It is now and again perplexing just to choose to be releasing thoughts which usually some others may have been trying to sell. And now we recognize we’ve got the blog owner to appreciate for that. The type of explanations you’ve made, the easy web site navigation, the friendships your site give support to create – it is everything incredible, and it’s leading our son in addition to us recognize that the matter is exciting, and that’s extremely important. Thank you for the whole lot!

  18. I have to express appreciation to you for bailing me out of this particular predicament. After exploring through the the web and meeting concepts which were not powerful, I was thinking my entire life was done. Being alive without the presence of solutions to the problems you have sorted out all through your entire post is a crucial case, and those that might have adversely affected my career if I hadn’t come across your web site. Your primary expertise and kindness in controlling a lot of things was precious. I don’t know what I would have done if I had not come upon such a stuff like this. I am able to at this point look forward to my future. Thanks so much for your impressive and sensible guide. I will not think twice to refer your site to any individual who wants and needs guide on this area.

  19. I simply had to thank you so much yet again. I’m not certain the things that I could possibly have carried out without the type of suggestions contributed by you regarding such a concern. Certainly was the daunting concern in my circumstances, but looking at a new professional technique you processed it made me to cry with gladness. I’m thankful for the support as well as sincerely hope you find out what a great job you’re putting in teaching other individuals all through a blog. I am certain you have never got to know all of us.

Leave a Reply to hermes belt Cancel reply

Your email address will not be published. Required fields are marked *