【51CTO獨(dú)家特稿】2006年創(chuàng)建的Twitter已經(jīng)是如今全球最為火熱的SNS服務(wù)之一，更是微博客這一概念的頭號(hào)先行者。Twitter目前的注冊(cè)用戶數(shù)已經(jīng)超過(guò)了一億，而現(xiàn)在整個(gè)Twitter團(tuán)隊(duì)大約只有180名員工，其中技術(shù)工程師占據(jù)70到80個(gè)左右。巨大的信息流量不斷沖擊著Twitter服務(wù)器的上限，而做為一個(gè)SNS服務(wù)，功能的更新也同樣不可忽視。對(duì)于這樣一個(gè)發(fā)展迅速的服務(wù)，我們要如何有效地把握它的發(fā)展脈絡(luò)？在今年的北京QCon大會(huì)上，51CTO開發(fā)頻道有幸邀請(qǐng)到了Twitter的系統(tǒng)工程師Nick Kallen來(lái)和我們分享Twitter技術(shù)團(tuán)隊(duì)的經(jīng)驗(yàn)。

Nick Kallen原本是一位軟件咨詢師，同時(shí)也是Rails3框架的基礎(chǔ)Arel、NamedScope、分布式緩存框架Cache Money和JavaScript行為驅(qū)動(dòng)開發(fā)框架Screw.Unit等多個(gè)開源項(xiàng)目的作者。兩年前他應(yīng)邀解決Twitter的可伸縮性問(wèn)題，并由此加入了Twitter的技術(shù)團(tuán)隊(duì)。對(duì)于現(xiàn)在Twitter的規(guī)模而言，可高效查詢的可伸縮式數(shù)據(jù)模式是最為緊迫的任務(wù)之一，而Nick現(xiàn)在的主要關(guān)注方向之一就是一個(gè)通用的分布式數(shù)據(jù)庫(kù)。

Nick的Twitter是@nk。

Twitter的發(fā)展早期

做為流行度增長(zhǎng)最快的SNS服務(wù)，Twitter常常被拿來(lái)與2004年創(chuàng)建的Facebook作比較。與Facebook日益劇增的新功能與新應(yīng)用形成鮮明對(duì)比的是，Twitter在誕生之后的三年內(nèi)一直鮮有新特性的加入，直到最近這一年才逐漸加入了圖片支持、地理位置顯示等新功能。根據(jù)Nick的介紹，早期的Twitter團(tuán)隊(duì)對(duì)Twitter的定位是以“最小化服務(wù)”為目標(biāo)的，當(dāng)時(shí)團(tuán)隊(duì)認(rèn)為添加新的功能將會(huì)對(duì)Twitter“最小化服務(wù)”的定位產(chǎn)生不好的影響。

不過(guò)對(duì)當(dāng)時(shí)常年被Fail Whale所糾纏的Twitter團(tuán)隊(duì)而言，的確也抽不出足夠的人力和精力放在開發(fā)新功能上面。Twitter誕生后的三年時(shí)間中，用戶數(shù)量一直飛速發(fā)展，數(shù)據(jù)量驟然攀升。Twitter最開始以LAMP架構(gòu)（Linux+Apache+MySQL+PHP）創(chuàng)建，這個(gè)系統(tǒng)很快便不堪重負(fù)。Nick十分坦誠(chéng)的向我們描述了Twitter團(tuán)隊(duì)早些年的狀況：

[[10406]]

“系統(tǒng)的可伸縮性一直是如此緊迫的任務(wù)，導(dǎo)致我們的工程師們幾乎沒(méi)有什么機(jī)會(huì)考慮新特性方面的事情。所有的工作就是維持服務(wù)運(yùn)作，以及將其擴(kuò)展。……在一開始的LAMP架構(gòu)下，通常就是一個(gè)MySQL主數(shù)據(jù)庫(kù)來(lái)做垂直伸縮。這當(dāng)然不是不可伸縮的設(shè)計(jì)，不過(guò)無(wú)法滿足我們的需求。”

Twitter的可伸縮性

在用戶接觸不到的后臺(tái)，Twitter在四年間其實(shí)一直經(jīng)歷著很多改變。比如在應(yīng)用層，服務(wù)器后臺(tái)處理這一塊，Twitter在08到09年間使用Scala語(yǔ)言重寫了后臺(tái)應(yīng)用，大大增強(qiáng)了多進(jìn)程異步處理的能力，提升了性能。

這期間***的變化，應(yīng)該就是數(shù)據(jù)層的變革了。2009年開始掀起的“NoSQL革命”在Web領(lǐng)域造成了極大的反響，Twitter也在今年年初開始對(duì)其Tweets數(shù)據(jù)類型引進(jìn)新生的NoSQL數(shù)據(jù)庫(kù)Cassandra。而根據(jù)Nick的介紹，Twitter早在一年半之前便開始專注于高效的分布式數(shù)據(jù)存儲(chǔ)解決方案，該方案的重點(diǎn)在于Partitioning策略，即數(shù)據(jù)的分割。

[[10407]]

“以前我們將所有的數(shù)據(jù)和服務(wù)存儲(chǔ)在一個(gè)組件上。而數(shù)據(jù)分割的做法，就是將數(shù)據(jù)分割成小塊，然后存儲(chǔ)在多個(gè)組件之上。因?yàn)榇髩K的數(shù)據(jù)被切割成了小塊，我們就可以并行的、以小任務(wù)的方式完成查詢和操作的工作。無(wú)論是我們開始發(fā)一個(gè)推，還是我們開始一個(gè)社交圖（social graph），還是我們開始一個(gè)搜索，每一個(gè)主要組件都在過(guò)去的一年半中通過(guò)不同的策略完成了數(shù)據(jù)分割。這就是現(xiàn)在Twitter可伸縮性的實(shí)現(xiàn)。”

近一年多持續(xù)增長(zhǎng)的時(shí)間內(nèi)，Twitter Fail Whale的出現(xiàn)次數(shù)已經(jīng)降低了很多，應(yīng)用層與數(shù)據(jù)層改進(jìn)可謂是卓有成效。

Twitter API與新功能

相比Twitter的流量和用戶數(shù)，現(xiàn)在的Twitter技術(shù)團(tuán)隊(duì)仍然顯得人手不那么充足。不過(guò)Twitter有一個(gè)很有意思的地方，就是在于其第三方應(yīng)用的流行。2009年7月的統(tǒng)計(jì)顯示Twitter的第三方桌面應(yīng)用、手機(jī)應(yīng)用、Web應(yīng)用和瀏覽器擴(kuò)展總數(shù)剛剛過(guò)萬(wàn)，而截止到現(xiàn)在，這個(gè)數(shù)量已經(jīng)超過(guò)了十萬(wàn)。雖然Twitter的核心功能一直沒(méi)什么重要改變，但在全世界上萬(wàn)開發(fā)者的擺弄之下，這個(gè)社交平臺(tái)顯示了驚人的生命力與創(chuàng)新能力。像是TwitPic圖片存儲(chǔ)服務(wù)和iPhone客戶端這種應(yīng)用都是來(lái)自第三方開發(fā)者，而這與Twitter的開放API是密不可分的。可以說(shuō)，Twitter的大部分活躍和創(chuàng)新都應(yīng)該歸功于它的開放API。

Twitter開發(fā)者大會(huì)

對(duì)于Twitter API的由來(lái)，Nick做了一點(diǎn)簡(jiǎn)單的介紹：

“其實(shí)一開始的API開發(fā)，就是因?yàn)橐粋€(gè)以前在Twitter的工程師離職去了德國(guó)，而他想要把Twitter集成到自己的聊天機(jī)器人里面。最初的API就是為了這位老兄的小玩具而設(shè)計(jì)的，不過(guò)我們很快就看到，將有更多人使用API來(lái)創(chuàng)造更多的東西。因此我們?cè)诤茉绲臅r(shí)候就開始在API上投入精力。”

和Facebook一樣，Twitter對(duì)于開發(fā)者社區(qū)非常關(guān)注。Twitter剛剛在4月14日和15日舉辦了Twitter開發(fā)者大會(huì)Chirp，剛好在Facebook的F8開發(fā)者大會(huì)之前一周。Chirp大會(huì)上剛剛公開了Twitter開發(fā)者網(wǎng)站的上線，相信對(duì)于Twitter開發(fā)者而言是個(gè)令人興奮的消息。

更多的變革

正如同Nick所介紹的那樣，Twitter已經(jīng)發(fā)生了轉(zhuǎn)變。與之前的“最小化”定位不同，Twitter已經(jīng)開始積極的推出新的特性。而對(duì)于開發(fā)者社區(qū)，Nick也表示了自己的期待：

“開發(fā)者社區(qū)的一大挑戰(zhàn)便是，我們構(gòu)建的產(chǎn)品核心功能要如何通過(guò)API來(lái)實(shí)現(xiàn)更多的創(chuàng)新——那些我們將不會(huì)在核心中包括的創(chuàng)新，而不僅僅是Web的一個(gè)替代品。程序使用API的方式和人使用網(wǎng)站的方式是有著極大區(qū)別的。程序總是不停地問(wèn)Twitter：有更多數(shù)據(jù)么？有新的信息么？如此這般。不過(guò)人是不同的，他們?cè)谝惶飚?dāng)中的特定時(shí)間段查看頁(yè)面，比如午飯的時(shí)候。所以很有意思的是，所有的API使用都很同質(zhì)化——功能相似，速度很快，重復(fù)性強(qiáng)。如何在這種方式下進(jìn)行高效的訪問(wèn)，是工程師需要思考的問(wèn)題。而每個(gè)用戶不同的、不規(guī)范的使用習(xí)慣，也是另一個(gè)需要思考的問(wèn)題。”

#p#

附錄：Nick Kallen專訪文字整理

（右為Twitter系統(tǒng)工程師Nick Kallen，左為51CTO開發(fā)頻道編輯楊賽）

51CTO: How did you join twitter, and how many people are there in the current twitter development team?

NK: I joined twitter about 2 years ago. Originally I consulted for them to help with scalability issues, and I really enjoyed working there. They wanted to hire me, they made me an offer and I accepted, that’s how I originally joined. I believe there is 180 employees now, approximately. As for how many engineers are there, I think there is about 40% or 50% of engineers, so about 70 to 80.

51CTO: Twitter has been cautiously adding new features over the past 4 years. How do you decide whether a new feature should be added?

NK: There is a lot of reasons why twitter has in the past been cautious adding new features. For the first couple of years of twitter history, and until recently, scalability has been such an urgent concern that there hasn’t been as much opportunities for the engineers to work on any new features, they’ve been so busy keeping the site up on, making it scale. I think also early on, twitter is sort of a minimal service, I mean, many people contrast it to Facebook. Facebook is a rich set of features like photos, all sorts of things. And traditionally twitter has been very minimal, hasn’t added like extensive conversation functionality abilities like photo features. And so, the culture for a while, we have been reluctant to add features, which distracts the minimal of twitter. I think that is changing now, though, I think we are pretty aggressively adding new features, and riching new feature sets has been rather experimental. So, vary to the minimalism we used to be.

51CTO: Did you consider high scalability from the very beginning? How did Twitter's scalability improve over the past year?

NK: Well it definitely wasn’t designed for scalability from the beginning. It was designed using kind of the traditional LAMP style – Linux, Apache, MySQL, PHP architecture – usually the single MySQL master database, vertically scaled. That is kind of how the original version of twitter was architectured. That is definitely not unscalable design.

Last year and a half, we were focused on basically partitioning strategies for our data storage. That means, instead of storing all the data or service in one component, you take that data and divide it into small pieces, and you store it across multiple components. So you can answer queries and manipulate it in parallel and in smaller jobs, because you’ve take a huge amount of data and divided it into pieces. So every major components, from how we start tweets, to how we start a social graph, to how we start search indices, has basically been partitioned using different strategies over the last year and a half. That’s what makes twitter scale now.

51CTO: Twitter APIs has been a main reason for Twitter's success as a service. How does such an architecture different from a normal Web 2.0 product?

NK: Originally the API was developed because, when the first engineers of twitter left and moved to Germany, and he wanted to integrate twitter with an IRC bot, and the original API was designed to support him doing his little toy, and it quickly became apparent that people could have created things using the API, so we early on invested on the API functionality. That gives twitter a main advantage since we have been a small engineering team for a long time, and by opening the API we allow other people to build core functionality for us, an obvious example would be twitpic, where we didn’t have the resources to build photo storage/services, because there weren’t enough engineers. But by having the API, those core services could be built by other people, including like an iPhone client these days.

The challenge in the community now, though, is as we are able to build central parts of the products for people to have more creative uses of the API, and use it for not just as an alternative to the web, but creative kinds of things that we are not going to build into the core product. For scaling, APIs vs. the web, there is a big difference between the way software queries APIs vs. the way human beings use the websites. Programs would constantly pull – they keep pinning twitter, can I get more data, is there anything more recent, etc. The human beings don’t behave that way, they check it a few times, during lunch or something like that. So the interesting thing about API usage is that it’s very homogeneous, very similar, and is very high velocity and repetitive. And so you need to engineer your system to support that style of access efficiently. And that’s a different problem in supporting the kind of varied and irregular use cases of human beings.

51CTO專訪Nick Kallen視頻請(qǐng)見(jiàn)下一頁(yè)

#p#

視頻采訪實(shí)錄