Designing Scalable Infrastructure
I think it is time someone spends some time writing a book about designing scalable infrastructure to help those who are currentlty having to go through the learning curve to learn how to design a scalable infrastructure for their application.
Thinking about who could qualify for writing this book I would imagine someone from Live Journal or Omniti could easily write this book. I have from a certain degree not had to scale a perl application in ages, but I tend to have to scale PHP ones which I've designed.
The problem with sites gaining in popularity is that they get increased traffic and they require more CPU time. So throwing in more hardware all the time to make the site scale to many may look like the solution to their problem.
Profiling your code can also help. Then you can optimise your SQL queries, cache content, etc.
Looking back at various projects which I've designed only one of them required having a replicated database setups for doing queries to search the database. All fulltext searches where ofset to the slave database server which happened to be much faster at dealing with fulltext searches.
Then you get opcode caches for PHP such as the Zend Performance Suite which does content acceleration. There are other free content accelerators for PHP which can be used, but most companies would want to use Zend's as it is a commercially supported product.
Then you can tune your hardware to see how much more performance you can get out of it. Add more RAM (we had 128Mb's of RAM in the server and upgraded the box to 512Mb) which also helps.
Tame the Apache HTTP Server. Reverse proxy infront of the servers or use LSNAT so that the boxes behind the firewall are like 10.0.0.2, 10.0.0.3, 10.0.0.4 and randomly gets the requests sent to them.
Rsync is very useful for replicating data across the network. MogileFS looks good if it can work under FreeBSD, but FreeBSD's NFS implementation does work whereas the NFS implementation under Linux does not work well.
I think that it is important for others to learn from our mistakes as Brad has said before rather than they make the same mistakes, which I think could in the long run save them losts of time and money. One thing I would have liked to have known what I know today when I was starting to write an application. I suppose I was lucky to rewrite a similar application and figure out as I was going along, but a book would have helped.
Also one needs to look at various things when coding. A lot of people suggest that one should not being using database abstraction layers or that we need to form our own opinion about using them.
Then we get template engine such as Smarty which enables us to remove HTML code out of the PHP code and have seperate template files, which I like and is one of the reasons why I use Smarty. No longer do web designers break my PHP code when they edit, they just edit the template files, and learn a simple easy to use template language.