“The things we did at Google!” – could now be done at Hadoop, if/when it evolved?


An example of what a Hadoop Cluster infrastructure diagram might look like.

” if you search for “2005 Accord,” Google seems to understand you’re looking for a family sedan, giving you links not only for used Honda Accords but for similar cars with similar prices — a Volkswagen Passat or a Toyota Camry.”

…  But he’s not applauding Google’s famous search algorithms. He’s applauding the infrastructure Google built to support those algorithms — software platforms such as the Google File System (GFS) and Google MapReduce that store and analyze data by spreading it across an army of ordinary servers.

The algorithms are important too, but it’s MapReduce that took all those pages from across the web and put them into a readily searchable index.

“The things we did at Google were incredible,” Srivas says. “I was just blown away by how effectively data was used.”

Hadoop, an open source incarnation of Google’s GFS and MapReduce platforms.

But unlike its competitors, MapR [Sirvas and hios crew] is offering something that’s very different from the open source Hadoop project. The company spent two years rewriting Hadoop behind closed doors,

Hadoop is a prime example of how technologies developed by the giants of the internet are now reinventing the software — and the hardware — used by everyday businesses. It’s a means of analyzing large amounts of unstructured data using a cluster of dirt-cheap servers.

At Yahoo and Facebook, it [Hadoop] feeds information into live web services, and it helps track how these services are performing.

In the age of the internet, with more and more data flooding into the world’s businesses, this is something that can appeal to almost any large operation.

Big-name tech vendors such as Microsoft, Oracle, and IBM are offering tools based on Hadoop, and Srivas is just one example of an engineer who left a big name web outfit to build a startup around the platform.

Cloudera and Hortonworks are the other notable startups.

Each of these startups takes a slightly different approach to Hadoop, and naturally, they spend a fair amount of time criticizing each other’s efforts. But despite his obvious agenda, the message from Srivas provides a nice counterpoint to all the hype surrounding the platform. Hadoop is a tool that still needs work. Like Google. Sorta Kirk Dunn — the chief operating officer of Hadoop startup Cloudera — points to Google, Facebook, and Yahoo as proof that the platform is ready for prime time. “Google and Facebook and Yahoo have many thousands of nodes that have been running many years,” he tells Wired. “The body of evidence there is overwhelming.”

Google doesn’t actually run Hadoop — Yahoo and Facebook and others built Hadoop using Google research papers that describe its back-end infrastructure — but his point is well taken. Yahoo and Facebook use Hadoop to crunch epic amounts of data using thousands of ordinary servers, and most businesses that adopt the technology will run the platform across much smaller clusters of machines.

But as M.C. Srivas points out, the open source version of Hadoop is still plagued by what are commonly called “single points of failure.” If one particular server goes down, it can bring down the entire platform. This sort of thing is something a Yahoo or a Facebook can deal with, Srivas says, but not necessarily the average business. “The reason that Yahoo and Facebook can run it is that they employ 50, 60, 70 engineers to feed the thing,” he says. “Other companies don’t have that.” Before founding MapR, Srivas says, he met with the founders of Cloudera and considered joining their effort. But they wanted to tackle Hadoop the way Red Hat tackled Linux — i.e., offer support, services, and additional software around the open source platform — and he felt that before doing anything else, you had to fix the holes in the platform. Rather than join what would become Cloudera, he found a kindred spirit in John Schroeder, the former CEO of Calista Systems, a desktop virtualization outfit that was acquired by Microsoft in early 2008. Schroeder had a friend at Google who also worked with MapReduce. Like Srivas, he attributes Google’s success not to its search algorithms but to its infrastructure. “From my acquaintance at Google, I observed — earlier than most — the power of MapReduce,” Scroeder says. “In 1998, they were the 19th search engine to enter the market. Remember doing an Alta Vista search, anyone? Google’s implementation of MapReduce on GFS and [Google’s distributed database] BigTable vaulted them to leadership within two years.” Srivas and Schroeder met through mutual acquaintances in the venture capital world, and they founded MapR in 2009. For two years, their team worked to build a proprietary version of Hadoop that would eliminate certain limitations, including those single points of failure, and in May of 2011, they took the wraps off their proprietary Hadoop distribution. It was already being used by an analytics appliance offered by storage giant EMC.

Hadoop of the Future According to Srivas and Schroeder, their Hadoop distro is several years ahead of the open source distributions offered by the likes of Cloudera. Which is only what you’d expected them to say. But it’s indisputable that the company has fixed major flaws that still plague the open source version.

Hadoop consists of a file system (HDFS) and a number-crunching platform (Hadoop MapReduce). The file system lets you spread data across a cluster of machines, and the MapReduce processes this data by sending little pieces of code to each individual server.

During those two years of development, MapR essentially rewrote the file system. “It could not be saved,” Srivas says. The company also revamped Hadoop’s “job tracker,” which distributes jobs across machines and then manages their execution, and its “name node,” which oversees file names across the system. On the open source platform, both are single points of failure, and the name node limits the number of files the platform can handle. Cloudera’s Kirk Dunn acknowledges these shortcomings, but he says there are other things to consider when evaluating the merit of the open source version of Hadoop.

The open source project will eventually eliminate those flaws as well, he says, and in the end, there are advantages to having all your code out in the open. “The conversation must be raised to a higher level than how many goesintas and goesoutas a distribution has,” he says. “With open source, you get the community effect. Would you rather rely on hundreds of engineers working on a very important problem? Or would you rather rely on one company with a handful of engineers?” Indeed, both Cloudera and Hortwornworks, a Yahoo spin-off, are committed to enhancing the open source project. And though much of its code is proprietary, MapR is also making some contributions back to the project. There are other areas where the platform can improve, and MapR is tackling these as well.

For the most part, Hadoop is a “batch” system. You give it a task. It works for a while. And then it churns out a result. It’s not designed to generate information in “real time.” With its search engine, Google has now abandoned MapReduce, moving to a platform called “Caffeine” that can update its search index on the fly, and John Schroeder hints that MapR is moving in a similar direction — though its solution will likely look very different from Caffeine. M.C. Srivas points out that Hadoop is quite different from what Google runs internally. In addition to GFS and MapReduce, Google runs another software layer called Borg, a means of managing server clusters inside its data centers. Google has yet to publish information about Borg, and like all ex-Google employees, Srivas won’t say any more about it, citing a non-disclosure agreement. But his larger message is that you shouldn’t mistake Hadoop for the Google infrastructure. Nor, he says, should you mistake it for what’s running at Yahoo and Facebook. “I’m sure that, like Google, they’re holding back what they see as their secret sauce,” he says. This may or may not be the case. But the fact remains that Google and Yahoo! and Facebook are not your average business.



Ex-Google Man Sells Search Genius to Rest of World | Wired Enterprise |

رسم بياني يقارن بين الشركات الأمريكية حسب قيمة اسعار الأسهم واللون يبين مدى المخاطرة المحتملة نتيجة تقلّب أسعار الأسهم

رسم بياني لأحجام الشركات  المريكية من حيث قيمة الأسهم، اللون يبين شدّة المخاطر المتعلقة بلإستثمار بأسهم الشركات

اكبر خمسة مراكز معلومات في الولايات المتحدة الأميريكية

اكبر خمسة مراكز معلومات في الولايات المتحدة الامريكية

اكبر خمسة مراكز معلومات في الولايات المتحدة الامريكية

Inside 5 of North America's Largest Datacenters
Via: Wikibon

لماذا لم اشترك Google+

لأن الشبكات الإجتماعية تندرج تحت إطار العولمة غير المرغوب بها ، ولأن الربيع العربي لم يزل في بدايته ولما يات بأكله بعد

تطوّر و إنجازات دولة كوريا الجنوبية في تقنية المعلومات

Korean IT Achievements 

كوريا الجنوبية تعرض منتجاتها التقنية في الولايات المتحدة

وصلت الحكومة الإلكترونية في كوريا الجنوبية إلى المرتبة الأولى في العالم حسب تصنيف الامم المتحدة المتعلق بذلك، فيما يلي عرض مختصر لما يقولة الكوريون المعنيون بذلك حول ما أنجزوه :

Ranked first in UN Global e-Government Survey :Korea has actively pursued e-Government as a crucial means to make the government more competitive, by leveraging the world’s best information technology (IT) including broadband Internet. After laying the groundwork for e-Government, including National Basic Information System (NBIS) in the 1980s and streamlining of applicable laws and institutions in the 1990s, the Korean government implemented e-Government as a major national agenda for the 2000s. It concentrated on 11 major tasks for e-Government (2001 ~ 2002) and 31 major tasks for the e-Government roadmap (2003 ~ 2007). As a result, e-Government has become firmly established in all areas of the Korean government.

المجتمع المعرفي الوطني : مجموعة مخططات يتم تطويرها و مراجعتها بإستمرار، تنفذها الحكومة بجديّة وحكمة ، هدفها نقل المجتمع الكوري إلى حالة مجتمع المعلومات.  من 2009 إلى 2012 هناك 205 مهمات/مشاريع ينبغي على الحكومة  تنفيذها للوصول إلى 5 اهداف، و 20 بنداً على الأجندة الوطنية للدولة.

The Korean government establishes a national informatization master plan every five years for the efficient and systematic promotion of national informatization, based on the “Framework Act on National Informatization (1995).” The 4th National Informatization Master Plan (2008~2012)sets the vision of national informatization for the following five years as “Establishing an Advanced Knowledge Information Society Based on Creativity and Trust,” Five goals and 20 agenda items back up the vision for its early achievement. For the successful execution of the 4th National Informatization Master Plan (2008~2012), the ‘National Informatization Action Plan (2009~2012)’ was established in May 2009. There were 205 tasks provided in order to achieve five goals and 20 agenda items.

من النتائج التي تحققت عبر الحكومة الإلكترونية في كوريا : تطوّر نوعي واضح في فعالية الخدمات وشفافية الإجراءآت الحكومية ، و تبعاً لذلك في إمكانية متابعة المواطنين لمعاملاتهم، وتطوّر العمل بشكل عام.

Korea’s e-Government has produced visible results: the efficiency and transparency of administrative work has significantly improved; administrative civil services have been greatly enhanced; and opportunities for people to participate in the policymaking process have been expanded. Accordingly, the effectiveness of the e-Government of Korea is widely recognized by the international community, and various e-Government systems are being exported to foreign countries. The 2010 UN Global e-Government Survey shows that Korea ranked first among all the member countries, given the highest possible scores in the categories of Online Service Index and the e-Participation Index.

الحكومة الذكيّة : مشروع يحقق للمواطنين في كوريا إمكانيّة الحصول بسهولة، ومجاناّ، على شتى الخدمات الحكومية، مهما كانت كيفية توصيل تلك الخدمة الحكومية للمواطن. بناء على ذلك تطمح الحكومة للتوصل إلى حلول لمشاكل إنخفاض نسبة المواليد، وما يتبع ذلك من المشكلات الناتجة عن شيخوخة المجتمع.، بينما تستجيب في الآن ذاته إلى ما يواجهها من تحديات الأمن والرفاهية الإجتماعية.

With the goal of converging the e-Government services with advanced IT technology until 2015, the Korean government now aims to implement ‘Smart Government,’ through which citizens may enjoy easy and free access to government services regardless of different channels of delivery. Accordingly, Korea will be able to tackle its social problems of low birthrates and the aging society, while actively responding to future issues of social security and public welfare.

مع اطيب الأمنيات للشعب الكوري بالوحدة والتقدم والرقي والإزدهار، ربما تعبّر هذه الصورة الليلية لشبه الجزيرة الكورية بوضوح عن الفرق بين الديمقراطية والإستبداد، التطوّر الذي تسمح به الديمقراطية التي تحققت حيث يسطع النور في سيئول ، مقارنة بالتخلف الذي يقود إليه الإستبداد حيث يعوي الظلام حول بيونغيانغ

أهم الإنجازات

Major Achievements

إقرأ المزيد

إذا الشعب يوماً اراد الحياة … فلا بد أن يستجيب القدر


ارادة الحياة

إرادة الحياة

(الشاعر التونسي ابو القاسم الشابي)

إذا الشّعْبُ  يَوْمَاً  أرَادَ   الْحَيَـاةَ          فَلا  بُدَّ  أنْ  يَسْتَجِيبَ   القَـدَر

وَلا بُـدَّ  لِلَّيـْلِ أنْ  يَنْجَلِــي               وَلا  بُدَّ  للقَيْدِ  أَنْ   يَـنْكَسِـر

وَمَنْ  لَمْ  يُعَانِقْهُ  شَوْقُ  الْحَيَـاةِ          تَبَخَّـرَ  في  جَوِّهَـا   وَانْدَثَـر

فَوَيْلٌ  لِمَنْ  لَمْ   تَشُقْـهُ   الْحَيَاةُ           مِنْ   صَفْعَـةِ  العَـدَم  المُنْتَصِر

كَذلِكَ  قَالَـتْ  لِـيَ  الكَائِنَاتُ              وَحَدّثَنـي  رُوحُـهَا    المُسْتَتِر

وَدَمدَمَتِ   الرِّيحُ   بَيْنَ   الفِجَاجِ         وَفَوْقَ  الجِبَال  وَتَحْتَ   الشَّجَر

إذَا مَا  طَمَحْـتُ  إلِـى  غَـايَةٍ              رَكِبْتُ   الْمُنَى  وَنَسِيتُ   الحَذَر

وَلَمْ  أَتَجَنَّبْ  وُعُـورَ  الشِّعَـابِ            وَلا كُبَّـةَ  اللَّهَـبِ   المُسْتَعِـر

وَمَنْ  لا  يُحِبّ  صُعُودَ  الجِبَـالِ          يَعِشْ  أَبَدَ  الدَّهْرِ  بَيْنَ   الحُفَـر

فَعَجَّتْ  بِقَلْبِي   دِمَاءُ   الشَّبَـابِ           وَضَجَّتْ  بِصَدْرِي  رِيَاحٌ   أُخَر

وَأَطْرَقْتُ ، أُصْغِي لِقَصْفِ  الرُّعُودِ      وَعَزْفِ  الرِّيَاح  وَوَقْعِ  المَطَـر

وَقَالَتْ لِيَ الأَرْضُ – لَمَّا  سَأَلْتُ :        ” أَيَـا أُمُّ  هَلْ تَكْرَهِينَ  البَشَر؟”

“أُبَارِكُ  في  النَّاسِ  أَهْلَ  الطُّمُوحِ       وَمَنْ  يَسْتَلِـذُّ رُكُوبَ  الخَطَـر

وأَلْعَنُ  مَنْ  لا  يُمَاشِي  الزَّمَـانَ          وَيَقْنَعُ  بِالعَيْـشِ  عَيْشِ  الحَجَر

هُوَ الكَوْنُ  حَيٌّ ، يُحِـبُّ  الحَيَاةَ           وَيَحْتَقِرُ  الْمَيْتَ  مَهْمَا  كَـبُر

فَلا  الأُفْقُ  يَحْضُنُ  مَيْتَ  الطُّيُورِ         وَلا النَّحْلُ يَلْثِمُ مَيْتَ الزَّهَــر

وَلَـوْلا   أُمُومَةُ    قَلْبِي   الرَّؤُوم          لَمَا ضَمَّتِ  المَيْتَ تِلْكَ  الحُفَـر

فَوَيْلٌ لِمَنْ  لَمْ  تَشُقْـهُ   الحَيَـاةُ                   مِنْ   لَعْنَةِ   العَـدَمِ   المُنْتَصِـر!”

وفي   لَيْلَةٍ   مِنْ   لَيَالِي  الخَرِيفِ        مُثَقَّلَـةٍ  بِالأََسَـى   وَالضَّجَـر

سَكِرْتُ  بِهَا  مِنْ  ضِياءِ   النُّجُومِ          وَغَنَّيْتُ  لِلْحُزْنِ   حَتَّى  سَكِـر

سَأَلْتُ الدُّجَى: هَلْ  تُعِيدُ   الْحَيَاةُ          لِمَا   أَذْبَلَتْـهُ   رَبِيعَ    العُمُـر؟

فَلَمْ   تَتَكَلَّمْ     شِفَـاهُ    الظَّلامِ             وَلَمْ  تَتَرَنَّـمْ  عَذَارَى   السَّحَر

وَقَالَ  لِيَ  الْغَـابُ   في   رِقَّـةٍ            مُحَبَّبـَةٍ  مِثْلَ  خَفْـقِ  الْوَتَـر

يَجِيءُ  الشِّتَاءُ  ،  شِتَاءُ الضَّبَابِ         شِتَاءُ  الثُّلُوجِ  ، شِتَاءُ  الْمَطَـر

فَيَنْطَفِىء السِّحْرُ ، سِحْرُ الغُصُونِ        وَسِحْرُ  الزُّهُورِ   وَسِحْرُ  الثَّمَر

وَسِحْرُ  الْمَسَاءِ  الشَّجِيِّ   الوَدِيعِ        وَسِحْرُ  الْمُرُوجِ  الشَّهِيّ  العَطِر

وَتَهْوِي    الْغُصُونُ     وَأَوْرَاقُـهَا         وَأَزْهَـارُ  عَهْدٍ  حَبِيبٍ  نَضِـر

وَتَلْهُو  بِهَا  الرِّيحُ  في   كُلِّ   وَادٍ         وَيَدْفنُـهَا  السَّيْـلُ  أنَّى  عَـبَر

وَيَفْنَى   الجَمِيعُ   كَحُلْمٍ   بَدِيـعٍ           تَأَلَّـقَ  في  مُهْجَـةٍ   وَانْدَثَـر

وَتَبْقَى  البُـذُورُ  التي   حُمِّلَـتْ            ذَخِيـرَةَ  عُمْرٍ  جَمِـيلٍ  غَـبَر

وَذِكْرَى  فُصُول ٍ ،  وَرُؤْيَا   حَيَاةٍ        وَأَشْبَاح   دُنْيَا   تَلاشَتْ   زُمَـر

مُعَانِقَـةً  وَهْيَ  تَحْـتَ الضَّبَابِ            وَتَحْتَ الثُّلُوجِ وَتَحْـتَ  الْمَدَر

لَطِيفَ  الحَيَـاةِ الذي  لا  يُمَـلُّ            وَقَلْبَ  الرَّبِيعِ   الشَّذِيِّ   الخَضِر

وَحَالِمَـةً  بِأَغَـانِـي  الطُّيُـورِ                     وَعِطْرِ  الزُّهُورِ  وَطَعْمِ   الثَّمَـر

“ويًَمشيْ الزَّمانُ، فتنموْ صُروفٌ           وتذْوي صُروفٌ، وتحْيا أُخَر

وتُصبح ُ أحلامُها يَقْظةً،                    مُوَشَّحةً بغُموضِ السَّحر

تُسائِلُ:  أينَ ضَبابُ الصَّباحِ،               وَسِحْرُ المساءِ؟ وضوْئُ القَمر؟

وَأسْرابُ ذاكَ الفَراشِ الأنيقِ؟              ونَحْلٌ يُغَنيْ، وغَيمٌ يَمُرّ

وأينَ الأشِعَّةُ والكائِناتُ؟                     وأينَ الحياةُ الَّتي أنْتظِر

ظمِئتُ إلى النُّور، فوقَ الغُصونِ!                   ظمِئتُ إلى الظِلِّ تحْتَ الشَّجار!

ظَمِئتُ إلى النَّبْعِ، بَيْنَ المُروجِ              يُغَنّين ويّرْقُصُ فَوْقَ الزّهَر!

ظَمِئتُ إلى نَغَمَتِ الطُّيورِ،                  وهَمسِ النَّسيم، ولَحْنِ المَطر!

ظَمِئتُ إلى الكونِ! أيْنَ الوُجودُ             وأنَّي أرَى العالَمَ المنتظر

هو الكَوْنُ، خَلْفَ سُباتِ الجُمود             وفي أثفُقِ اليَقَظاتِ الكُبَر”

وَمَا  هُـوَ  إِلاَّ  كَخَفْـقِ  الجَنَاحِ            حَتَّـى  نَمَا شَوْقُـهَا  وَانْتَصَـر

فصدّعت  الأرض  من    فوقـها         وأبصرت الكون  عذب  الصور

وجـاءَ    الربيـعُ      بأنغامـه             وأحلامـهِ  وصِبـاهُ   العطِـر

وقبلّـها   قبـلاً   في   الشفـاه              تعيد  الشباب الذي  قد   غبـر

وقالَ  لَهَا : قد  مُنحـتِ   الحياةَ            وخُلّدتِ  في  نسلكِ  الْمُدّخـر

وباركـكِ  النـورُ   فاستقبـلي              شبابَ  الحياةِ  وخصبَ   العُمر

ومن  تعبـدُ  النـورَ   أحلامـهُ             يباركهُ   النـورُ   أنّـى   ظَهر

إليك  الفضاء  ،  إليك  الضيـاء          إليك  الثرى   الحالِمِ   الْمُزْدَهِر

إليك  الجمال  الذي   لا   يبيـد            إليك  الوجود  الرحيب  النضر

فميدي كما  شئتِ  فوق  الحقول         بِحلو  الثمار  وغـض الزهـر

وناجي  النسيم  وناجي  الغيـوم          وناجي النجوم  وناجي  القمـر

وناجـي   الحيـاة   وأشواقـها             وفتنـة هذا الوجـود  الأغـر

وشف  الدجى  عن  جمال عميقٍ         يشب  الخيـال ويذكي   الفكر

ومُدَّ  عَلَى  الْكَوْنِ  سِحْرٌ  غَرِيبٌ          يُصَـرِّفُهُ  سَـاحِـرٌ  مُقْـتَدِر

وَضَاءَتْ  شُمُوعُ النُّجُومِ  الوِضَاء       وَضَاعَ  البَخُورُ  ، بَخُورُ   الزَّهَر

وَرَفْرَفَ   رُوحٌ   غَرِيبُ   الجَمَالِ       بِأَجْنِحَـةٍ  مِنْ  ضِيَاءِ   الْقَمَـر

وَرَنَّ  نَشِيدُ   الْحَيَاةِ    الْمُقَـدَّسِ           في  هَيْكَـلٍ حَالِمٍ  قَدْ  سُـحِر

وَأَعْلَنَ  في  الْكَوْنِ  أَنَّ   الطُّمُوحَ        لَهِيبُ الْحَيَـاةِ  وَرُوحُ الظَّفَـر

إِذَا   طَمَحَتْ   لِلْحَيَاةِ    النُّفُوسُ           فَلا  بُدَّ  أَنْ  يَسْتَجِيبَ  الْقَـدَرْ