On Wednesday night spoke about an 'Introduction to Memcache' at the FB Developers Garage which was held at UCT. Chatted about various things ranging from Memcached, Starling, the Evented version of Starling, background queue processing, etc.
I will be writing a more detailed post during the course of the next few days or so.
Further reading in the interim and the patch for the memcache-client:
Amazon are launching their SimpleDB product to a limited group of people in the next couple of weeks which exposes a webservice for running queries on structured data in real time. Apparently this service works with both their existing S3 and EC2 services.
Interesting idea to offload database queries to a third party, but the question is how reliable is it to offload data processing to a third party when you ideally want your SQL to run in less than 10 ms?
An interesting article linked to from James Cox, which brings back memories while working on php3 code for a local .za e-commerce start-up called Pitlane, where I developed code for a car auction site. But enough strolling down memory lane for one day:
In a past job, I worked for a startup e-commerce firm. The architecture was management not customer driven (it was a battle to even get permission to interact with customers and solicit feedback). The structure came from ideas, concepts but not planning: the end result was meaningless and frustrating.
I remember having a conversation with the not-much-older-than-I Managing Director. I had been researching some potential vendor and raised the name with my boss. Yes, he said - "This company are our biggest competitor". I neglected to mention we hadn't sold our product to a proper external client at that time and this competitor was powering some of the country's biggest blue chip names, and instead focused on the company I believed was the bigger and most dangerous competitor: Amazon.
So I responded - "have you considered Amazon?" he dismissed the idea - "don't be silly, amazon just sells dvds and books - it's a web store not an ecommerce platform!" It was about this time that I realized he was a moron.
The thing is - Amazon is the biggest threat to e-commerce developers. It's also their saving grace - they just don't know it yet.
Amazon happens to have innovated and grown a massive distributed service oriented architecture. After they gave up their monolithic 1.0 version back in 2001, they realized that to scale up they had to scale down: distributed architecture is often synonymous with the emergence of insect colonies. In this case Ants - each with their own role to play but without each the whole fails its joint objective.
By taking out core elements of the business - from backend supply chain to front end ratings, rankings and search - and all the others in between and sub divided, Amazon are able to delegate ownership, management and process of these tasks to small efficient crack teams within their organization. This ratifies the belief that small teams develop and ship, big teams discuss and plan: clearly developing and shipping product outweighs the lost efficiency of wrong paths.
It also escapes the need to mass scale intelligence: to try and architect the entire amazon operation from a whole world view is very hard, where as breaking it down to sub units is easier to manage. It breaks away from the bikeshed problem too: developers don't need to find something small enough to argue over - the parts are bite size and the teams are scaled to cope.
So technically they have it right, even if they are struggling to figure out how to test - a problem I totally understand: trying to figure out how to simulate a customer with a specific problem I hadn't even encountered yet was quite hard - instead I focused on making error reporting more robust and iterate faster.
I see the biggest problems that Amazon face are two fold: it's hard to see the usage requirements of a whole if everything is in pieces - it's important to keep a hold on all of them. Calling 100 applications on the front gateway page may be OK - but what impact does that have on the user? Especially if the user is a mobile device, a heads up display or a refrigerator.
The second problem is in interface. Amazon (in my opinion) need to go through the process of scaling back the architecture and direction. I often look at www.amazon.com and struggle to decide what the purpose is. (hint: making money by customers buying or consuming?)
Amazon are having that teenage crisis about their identity - are they really just a technology company who sells partner product, or are they really just a big shop with an amazing technology department?
It's hard being both, and if not managed the needs of one conflict and fight with the other.
Personally, I'd like to see amazon the retailer distance itself from amazon the technology company. (perhaps A9 will become the technology face of amazon?) I'd like to see that technology company focus on e-commerce, and not get into being a Microsoft competitor: we don't need another social networking site, search engine or similar web2.0 contribution - we do need better, simpler and more encompassing shopping user experience.
To find out more about Werner's work - read his weblog.
