David Carr at Ziff Davis' Baseline has a very useful article, Inside MySpace.com about how MySpace faced the problems of scaling to their current size of 40 billion page views and 26 million users. Some of the more interesting observations:
- The thorniest problem as they scaled was the database. Once their original, single database couldn't handle the load, they tried various methods of partitioning (in order): separating update and retrieval, having different databases for different parts of the web site and, finally, splitting the users among different databases. There were further refinements to virualize the SAN, add a caching layer and move to 64-bit SQL Server.
- Unlike most large sites which are built on some variety of LAMP, MySpace is built on Microsoft ASP.NET and SQL Server. This may be the largest such site, and it appears that Microsoft is powerfully motivated to make it a success. I will say from our own experience at Convoq that this platform does do a lot for the programmer and scales quite well.
- The reliability problems that MySpace's users experienced as the site went through its growing pains did not seem to hurt the business. Users complain but they don't leave the service and, unlike a bank, no money is misplaced. In some cases MySpace consciously chooses performance over reliability, for example extending the time between database checkpoints.
- The decision to allow users to use HTML tags and CSS was "kind of a mistake" when the developer neglected to strip them out. Although this results in messy looking pages and costlier performance, giving the end user such freedom is now credited with much of the site's success as a place where users go to express themselves.
Larry Dignan commented on the Baseline article on his blog at ZDNet and there is a lively thread on Slashdot.
