So I just finished a big tour de Southeast Asia. There is really only one story going on over there – Mobile. Check out this research from BCG (BRICI = Brazil, Russia, India, China, Indonesia):
1.2B new internet users coming from BRICI countries by 2015
Mostly coming from mobile, much of that will be advanced feature phones (not smart phones)
Social in Indonesia & Brazil, Email in India, IM in China (QQ FTW!!!)
That will only result in 20% internet penetration in those countries (i.e. lots more to come!)
All of this is happening here and now. There are huge opportunities for any mobile startups to break into these countries. Let me give you a taste of Indonesia.
Two years ago Nokia had 60% of the phone market. A retailer called Nexian had just started up with the idea of importing and reselling Chinese manufactured phones and branding them as Nexian phones.
Fast forward to today. Nokia is down to 30% of the market!!! Nexian is one of the dominant retailers, they just sold to the Spice Mobile family (large conglomerate in India exploding across Asia, Middle East and Africa). You can get a stylish QWERTY phone with a big display with pre-loaded apps including email, Opera Mini, Facebook, Twitter, Mig33, Nimbuzz IM, private messenger, widgets, news reader, etc, all for < $40 (in some cases closer to $20). You can also get a Nexian Android phone for $150-$200.
[caption id="attachment_9327" align="alignleft" width="640" caption="This Bad Boy = $20"][/caption]
Everybody is ripping off Nexian’s go to market strategy. The telecos have launched new brands to sell these phones. The retailers have gone to China, sourced phones and are re-selling them. The Chinese manufacturers have opened up retail in Indonesia (Huaiwei has their own stores!). On top of that I counted at least 20 Nexian clones, small companies with the same strategy as Nexian.
Blueberry Probably Outsells BlackBerry 2:1 In Indonesia
Nexian has countered by offering apps & services on these $20-$40 phones. People don’t have data plans, so Nexian does micro-billing via the carriers. 10c/week for Facebook, 5c/day for IM, $3/month for chat as examples. It works, they make millions a month from it.
There are roughly 170M phone users in Indonesia.
For Canadian startups, this means that there has never been a better opportunity to build mobile apps and “go global”. Check out if your app has a competitor in markets like Indonesia. All these phones in Indonesia (and China and India) are on a single platform with a single OS -> MediaTek. Port your app to MediaTek and go get deals with any of the manufacturers, the Nexians and their clones, the operators, etc. You can either try and get a royalty up front with your app pre-installed on the phone, or you can work out a rev share agreement on the service, or whatever other monetization scheme you have in mind.
There are plenty of folks who can help you with local business development. For instance, if you are a mobile app startup and you think Indonesia looks hot, Andy Zain runs the Indonesian Mobile Monday and Founder’s Institute. He is incredibly well connected in the teleco scene and he has helped startups like mig33 and Payfone (you are also free to reach out to me for contacts and info as well). In every country there are guys or companies like him. A bit of research and networking and you can find them.
Next week I’ll have some more on the start-up scene in Singapore, including some info from the leading startup blog over there (sgentrepreneurs.com).
If you haven’t heard by now, RIM is having a horrible year. Their earnings meeting yesterday was chock full of bad news:
Q1 revenue: $4.9 billion vs. $5.15 billion consensus
Q1 EPS: $1.33 vs. $1.32 consensus
Q1 shipments: 13.2 million vs. 13.5 million units expected
Q2 revenue: $4.2-$4.8 billion vs. $5.46 billion consensus
Q2 EPS: $0.75-$1.05 vs. $1.40 consensus
Q2 shipments: None given vs. 13.5-14 million units expectation
One caption I read put it best – RIIMMMMMBEEEERRRRRRR.
Out of the downfall the Globe and Mail was hypothesizing that the fall of RIM was catastrophic for Canada’s tech eco-system. The article was a bit light on fact as to why it would rip apart the Canadian eco-system, and my initial gut reaction was “RIM has almost no impact on any of the startups I know.” But then I decided to go and look at the facts.
Since roughly 2008 RIM has bought the following Canadian startups:
So they’ve probably flushed about $60mm-$80mm into the Toronto ecosystem over 3 years in exits. On top of that they have the BlackBerry Partners Fund (with about $150mm in cash) which has invested in several Canadian startups. Lets also not forget that the eco-system around their partners. BlackBerry’s platform has created opportunity for mobile dev shop’s like Fivemobile and Xtremelabs to exist. But it feels like those guys do most of their business in iPhone and Android.
So between exits and investment via BB partner funds they have probably kicked in about $100mm to the Canadian startup eco-system over the past 2-3 years. Which is not something to sneeze at. Having said that, Google (not HQ’d in Canada) has kicked in probably close to $100mm in the past 12 months… just in exits. So maybe its also not something to brag about either.
Putting these numbers together, makes me feel more ambivalent about RIM’s impact on the tech eco-system in Canada. Lets be clear, we’re talking about a decline in the short to medium term, not a total shutdown. In that decline I expect RIM to take an even lesser role in the eco-system than before. And I’m not sure it matters.
(Small end note as a UW alumnus. I’m not sure RIM’s downfall will have that big an impact on the school either. Big companies like Microsoft, Google, Facebook are still going to fight over UW’s top talent – their won’t a shortage of jobs for UW’s engineering community anytime soon. Maybe Laurier’s business & marketing grads… oh low blow).
Last week Tim Ferris, of 4 Hour Work Week fame interviewed the guys from Pivotal Labs and learnt the secret of their success – pair programming! It was a very thought provoking piece on how to get stuff done.
Well here in Canada we have our own super hot mobile dev shop, Xtreme Labs, @xtremelabs, who are absolutely killing it, picking up huge customers like GroupOn. The guys at Xtreme Labs are also infamous amongst the tech scene for being pushers of pair programming.
Now at Peek, where I’m CTO, we are all “Lonely Hero Programming” all the time. Our “bus count” is precisely 1. If somebody died tomorrow you know what I’d do? I’d hire somebody new!
The Best Way To Program – By Vinoth Chandar, Some Rights Reserved
So, being from this totally opposite world to pair programming, I wanted to learn a bit more about why a company would choose pair programming, and I thought I’d start up a friendly dialogue with Farhan Thawar of Xtreme Labs to gather this thoughts.
Dan: Hey Farhan, how are things? I’d love to hear more about how you guys program at Xtreme. How are you setup?
Farhan: Our work environment is what I call an Agile Team room. Super low cubes, everyone in one area, with programmers focussed in a particular technology seated close to each other (so iOS devs sit together, Android devs sit together, etc.).
Dan: I’m going to start right off the bat with some basic math. If I have two people solving one problem versus two people solving two problems, doesn’t that automatically make me less productive? You know 1+1=2? How can I as a lean running startup afford to be half productive???? You guys have to pay your 2x workforce in real cash, right?
Farhan: So the simple math is misleading, i.e. that 2 programmers working on the same task is less effective. For example, most agile shops typically get 4-5 hours of coding done per day. The rest is spent in email, meetings, Facebook, Twitter, Youtube, reading blogs, etc. With pairing, we can focus our folks for 8-9 hours together to get a full days work done. Yes, it’s 8-9 hours / 2 = 4-4.5 hours of coding each, so at a minimum it’s the same hours. However, we have instant robustness (bus count), higher quality (i.e. less hacking), instant knowledge transfer (two people touch every line of code).
Dan: I am a programming god. I go on tears for 2-4 weeks where I am an absolute machine. Knuth himself would have a hard time keeping up with me as I weave around in my language du jour (i.e. Haskell, gods only program in Haskell.. the language of the heavens). The last thing I need is some assclown developer tethered to my leg slowing me down. How do you handle god coders like me in pair programming?
Farhan: Lol, badass programmers get better by pairing, just one example: http://cycle-gap.blogspot.com/2007/09/extreme-pair-programming-guy-steele-and.html. The reason is that it’s an intense learning environment, much like apprenticeship in the old days (you apprenticed to learn your trade). There’s no better way to get better and faster at your trade than to work with someone else who is as smart as you are.
Dan: How does this work if you work on a lot of projects? What if you are doing maintenance on a few projects as well as new dev’t on another project? We have lots of guys who probably touch 2-3 projects on any given day? Do they have to find their pairing buddy for each project every time?
Farhan: So the key is to not swap projects often. Almost 100% of our pairs are on a single project for the entire week. So week to week things my change (most time they don’t), but our devs are focused just on one project, on one platform. We do have R&D pairs and a Sustained Engineering pair who focus on multiple updates and fixes, but they aren’t dedicated full-time to any project. Folks also don’t find a pair, they are allocated before the week starts. We take tons of feedback on pairings and it’s very easy to see pairs that aren’t working well (they don’t talk much, or one person is driving the whole time)
Dan: In a debate in my company a great coder I know basically said something like this “don’t most coders do pair whiteboarding/pair designing anyways? By the time it comes to actually coding, isn’t that the trivial part?” Lets pretend this is a law office or consultancy shop, would you pair (i.e. do teamwork) at the strategy phase/problem solving phase or would you pair during the document creation phase?
Farhan: So the coding isn’t the trivial part, as the end product is based on the code. It’s almost like saying the ingredients and the recipe are the hard part but the cooking is trivial. It’s not (and I know, cause I can’t cook for shit). You want to write code that is elegant, understandable, maintainable, etc. and pairing forces that as two people have to understand what is going on at all times.
Dan: Have you guys ever tried doing like a race? Put two coders doing 1+1 vs two coders pairing on the same project? Should we try to sponsor this type of event?
Farhan: Bring it. We’d love this. Don’t forget, ACM programming competitions only use one computer đ
Process Matters
So here’s my take overall. Process matters. Even for dev teams of 1-3 people selecting the right language, technology, tool chain, and software process make a big difference in productivity and quality. Software engineering has improved by leaps and bounds over the past 5-10 years. Here are some of the best changes:
Opensource software and the explosion of re-usable software components
Iterations and demos
Continuous Integration, build servers and automated test suites
Continuous Deployments, i.e. the newer devops movement
Explosion of tools & infrastructure in the build, test, deploy area (think of tools like Heroku, Chef, Capistrano, build tools, etc)
Test driven development, test automation & significantly improved testing frameworks
(Sometimes I write in ANSI C on proprietary embedded plaforms, where the tool chain and quantity of re-usable code is a fraction of what I want. You take for granted how much Java, Ruby on Rails, Python, PHP, etc have been built up.)
These days, I rarely come across startups that haven’t adopted most of the above to a certain degree. And pair programming, truthfully its just not widely adopted compared to the list above. To me pair programming seems to be an overkill solution to solve the real programming productivity problem – communication. Engineers are classically trained to solve problems independently (though this is changing at the university level finally). Dr Amol Sarva, CEO at Peek, always mentored me with the following advice on communication, picked up during formal structured problem solving training at McKinsey.
“Basically there are three stages of any problem where one can communicate and practice team work:
1. Structuring – during framing of the problem. What are the steps I am going to take to solve this problem? Read the internet, talk to an expert, write quick hacks to test it, etc.
2. Solving – during solving the problem. “Hey, I did this and this and here are the raw results, interesting, right?”
3. Synthesis – communication while synthesizing the results of problem solving. “Hey guys, I solved this, here’s how. Code is checked in.”
Engineers classically wait until synthesis to solve the problem. Which is too late. Others, (e.g. your classic ice-breaking consultant) do too much communication during structuring.”
Pair programming forces engineers to communicate much earlier on in the problem solving process. Which is good! It also forces them to communicate across the whole problem solving spectrum from structuring to synthesis. Also good! But god it sure feels like an inefficient oversolve to the problem. Managers can bake in communication to their dev process without forcing pair programming in my opinion.
Having said all that, I am always willing to try something new (good engineering demands experimentation and learning), so we are going to try pairing it up on a project at Peek and see if it works well or not. I’ll report back the results.
I’d love to hear what others think. Is pair programming the real deal, have you found it to be more efficient in getting your code on? Or do you like to pound it out solo with headphones and your favourite hoodie?
The launch of iMessage was an announcement with pretty significant impact on a few big/hot tech companies in Canada. Kik has obviously built a private messenger service around iPhone, RIM has taken an aggressive stance on promoting BBM, and of course lets not forget all the Canadian telecos.
Being an investor in Kik and some others impacted by the new iOS announcements, Fred Wilson wrote a brilliant post analyzing the impact. “Expect platform owners to work against you”.
In fact it was so interesting, that the product manager for RIM’s BBM service re-tweeted it (note that Theban is a good friend).
@dpmorel lol – it’s a balancing act. Platforms need to protect themselves from the ‘tragedy of the commons’
It is a fascinating lens into how RIM thinks of BBM and their own platforms. Despite my “ass” comment, I agree with him. Innovation often comes from the community who build things for the platform. The platform then needs to pull the best ideas back into the mothership – sometimes as an acquisition, sometimes they build it themselves and take on the “incumbent” in the community. This is in theory a good thing. We want the best innovations to be widely adopted. We want private messenger everywhere so we can all stop paying atrocious rates for SMS.
And those atrocious SMS rates charged by carriers, brings me to my next conversation. A good friend of mine runs messaging at a major Canadian carrier. Him and I have had the debate several times over the past few months on whether or not private messenger is a big threat today and now for them.
What i mean is that the argument about building SMS over data to drive costs down will go away in a few years as soon as LTE is deployed so ideally I want to invest in a solution I can leverage for more than a couple of years or so…
Customer experience and scalability are key for me. If I am spending money to improve the customer experience, it has to be open to all subs without having to invite people, therefore is has to scale.
The problem I have with the kik, textplus and the likes is that they are closed communities of 100,000 in 30 different countries or so and only address their own P2P niche market whereas SMS is an open community of a couple billion compatible devices; no need to register, invite friends, etc… While P2P SMS is plateau’ing in Europe, A2P SMS traffic is growing at a 25% rate yoy. In Canada P2P SMS is still growing at a 20% rate yoy and the A2P side of the business is still non existent.
P2P = peer 2 peer (i.e. customer to customer), A2P = application 2 peer (i.e. app to customer).
That list contains some some big, big knocks on ALL private messenger services – iMessage, BBM, Kik, TextPlus, etc. They are closed communities!! Meanwhile, with SMS, I can literally text ANYBODY in the world. Also, private messenger solutions don’t (yet) provide a solution for applications to talk to customers (A2P). And the carriers have a very real solution coming – LTE with better SMS clients on phones. SMS was literally one of the best services ever created – open, standards driven, no registration, etc. Awe-inspiring engineering. And with LTE the carriers will be able to take advantage of the cost savings of routing SMS over data, and they’ll have the biggest community (5b GSM users) and simply the best solution.
Buuuutttt… its not here yet. And thats what this is all about. Messaging is in transition and there’s an opportunity to own “messages over data”. Richer messages, more social messages, better user experiences, more cost effective for customers, etc. Its a fat, fat bazillion dollar market. And thats why this has happened:
Apple launched iMessage, Facetime, etc
Facebook bought Beluga
Google + GoogleVoice + GoogleChat + Android push notifications
RIM doubles down on BBM
Venture industry putting big money into potential disruptors like Kik, Textplus, Color, etc
By the time carriers get LTE out the door with new devices in market, the top end of this market may have already been won to the tune of 200-500 million users.
There is maybe some good news for Kik out of this announcement. For a startup the biggest problem is almost never a competitor. The biggest problem is typically non-adoption. Nobody knows there is something better than texting available. Nobody knows that Kik exists. Well, Apple & iMessage are about to blow up the “private messenger” space. In an ideal world, as the market grows, Kik goes along with it.
It’d be amazing if Kik & RIM (or others from up north) could figure out how to own this market. There are huge opportunities still – the A2P problem, the “interconnect” problem, etc. Lets hope they can figure them out and build ginormous services.
Edit 2 – I have been corrected that the Coradiant acquisition was more likely $100m-$150m. Updated below. Bigger than I thought! Glad to see I was “under-reporting”
Interesting first 6 months to 2011. Check out this list of exits:
Firstly, this is a perfect example of how VC exit math works and the power of a fundmaker like Radian6.
But get this, we are exactly 6 months in and almost half way to…. well… July/August/December all kind of suck for doing deals… buuuutttt… maybe… maybe if I utter the amount… people will dream and we could maybe dreamily hit 1 BILLION DOLLARS in returns this year. Wow, 1 BILLION. Thats nine 0’s.
The Net Value of All Exits for 2011?? (by Adam Crowe, some rights reserved)
We’d need another big mama ala Radian6, and another 4 $20-$30mm exits. I could hazard a guess at a few companies that could exit for $20mm+ today and now, but the big, big question is, where will the big exit come from?
They just raised $50mm. Not sure about the valuation, but we could size it at say $150mm – $300mm (I’d go higher because they are very young and having a $50mm raise means they have a vertigo inducing growth/revenue curve). And lets not forget the mainstream press chatting about an Apple acquisition. This could be a high 9 figure to billion dollar plus exit if something happens.
They state over 2mm users on their home page. Lets say 10% are paying customers and they pay on average $30/month (looking at their pricing plans). 200k * $30/month * 12 months = $72mm in annual revenue. Even at 5% freemium conversion they are at $36mm. Thats a big customer base and a big chunky, sticky subscription revenue base.
Would love to hear from folks. What other startups do we have hanging about that could do a big exit? Am I missing any of the 2011 exits to date?
The cost of an engineer is inflated right now. Informally, I poll salary expectations each semester from Waterloo co-ops. 2-3 years ago it was $65k-$70k. Last year/this year its $95k (the base Google, Facebook new grad offer). On top of that lots of engineers are able to source funding for their own social-mobile-cloud startup, more incentive not to join your own startup. Its not even a North American phenomena, in India I know technology managers making $120k+, in China I know guys turning down $100k offers now. There’s really no such thing as low cost development anymore.
So, whats a startup looking to build product to do?
âIdeally, to protect against inflation, you want a royalty on someone elseâs sales so you donât have to invest any more capitalâyou license it to them and you make money as their volume grows.â Warren Buffett, Berkshire Hathaway AGM, April 30th 2011
Don’t build new stuff and overpay inflated development costs if you don’t have to. Find partnerships so you can sell more product to your customers. Find more channel to resell your product. And so on. Maybe its time for more hustle and less hack??? (I’m a hacker, so writing this hurts a little bit).
Once upon a time in my startup life, we stumbled across the deal of a lifetime. A large company was spinning off a subsidiary, and they were paying someone a few million bucks to take it off their hands.
Most folks looked at the numbers and said “no thanks” – $7mm in revenue and spending about $20mm… yuck. But these guys were running the exact same business as us, similar subscriber counts and all, and we knew they could be run spending $3-$5mm per annum (which is what we were spending). For instance their CEO was getting $4mm/annum and ours was making $1/annum (perfect example of why big companies can be bad at launching new products). So we put in a bid for -$2mm (yes thats a minus in front). I.e. pay us $2mm to take your company, and have it generate $2-$4mm a year in cash for us. Booya. You could imagine how excited we were. We basically re-enacted this Monty Pyton scene every day in the office for 2 weeks (word of caution – this is 10 minute video and there’s a part 2):
Yaaaaar, corporate raiders be we.
Until, sadly, of course, somebody outbid our -$2mm offer. Damn. I suppose -2mm isn’t that hard to outbid.
I’m telling this story to give another “meme” to startupdom. Lean product development, social marketing, customer development, iterations, pivots, etc – these are the more popular memes of today. Well there’s another one thats not mentioned enough – Hustle and Flow – doing deals, business development, partnerships, strategics, m&a, etc. There are many big famous startups who had deals with a big elephant: Google powering Yahoo, Amazon powering Target, the Microsoft/Apple/IBM/Xerox tangle, RIM’s pager deal with Ericsson. Its a crucial part of growing your startup, you gotta be able to do deals.
One of the current killer “deal-oriented” startups in Toronto right now is Kobo Books (@mserbinis). Kobo got frickin’ Li Ka Shing to back them, the guy is a business legend! Why waste time with tiny business punks like Paul Graham and Dave McClure (I joke) when you can have a business God invest in you. Plus Kobo has done huge, killer deals from top to bottom in every category of their business – checkout their partner list in here. That my friends is big pimpin’ Canadian startup style.
Here’s another one, check out the list of deals Fixmo (@ricksegal, @shyamsheth) has done. They acquired a company (Conceivium) as a year and a half old startup! How many of you entrepreneurs in your first year or so wake up and say “lets buy a company”. On top of that, as an unknown one year old startup they walked into the Department of Defense in the US and nailed a massive deal. That is pure brass-balled, biz dev game.
Would love to hear some other great Canadian business hustler success stories from folks (or near misses, or disasters), or give us your favourite links/resources for networking, bd, m&a, hustling, pimping, whatever. We’ll be following up with some resources and tips to help your biz dev game.
I’ve always thought it weird that there is a perception that risk is front-loaded for startups, i.e. the person(s) who start it take all the risk. In some ways, the risk of starting a company (especially a web startup) is lower than ever:
there are great grant/tax credit programs like IRAP and SRED
there are more incubators, angels and providers of small seed funds than ever
its faster than ever to go from concept to commercialization -> you can have paying customers in < 6 months
It just isn’t as hard as it once was to raise $20-$50k and/or have a company generating a few thousand dollars in revenue to cover the early founder(s) costs. The big “risk” is that your life & business align to this cost structure. Your business needs to be able to run with only 1-3 people at first. Your life – no fancy sports cars, no big mortgage, no massive piles of credit card debt, etc. I’d hypothesize, there is a strong parallel between managing personal finances and being able to start a company.
Another big “de-risker”. You have control as the early founder! Only you can lay yourself off. And you control the culture and lifestyle of the company, i.e. you are a lot less likely to hate your job or get fired for hating your job or leave suddenly or have a heart attack or just generally hate life.
some rights reserved Search Engine People Blog
Reward-wise, you have a huge chunk of the reward. Shares, not options. Big founders cut. Maybe no vesting. Dividends. There are a lot of paths for you being well rewarded for the risk you took. On top of that, your experience as a first-time entrepreneur will make the second time around all that much smoother, it’ll be easier to raise money, easier to hire, easier to find business partners, and so on. You can have a career as an entrepreneur.
Now, lets compare that to what is traditionally thought of as the “low-risk” employee, lets say employee #8. Poor employee #8 takes on massive life risk, and often gets very little in reward.
Reward-wise, they get something like .5% of the company, so on a typical $20-$30mm exit they get $100-$150k. Hardly life-changing money. If you are a super-star in the company you may get granted up to 1 or 2% but they’ll be vested over an annoyingly stupid schedule such that you’ll have to be at the company for 6-8 years to “earn” them.
Now compare that to all the risks of being employee #8:
you are far more likely to get laid off than any of the founding team
you are far more likely to get laid off than a non-startup job
you may simply not get paid a few times… missed payroll is no uncommon event
you probably in fact took a pay cut, or at best, you’ll miss out on bonuses when there are a few tight years
you will likely work a lot longer hours
there’s no “fast” trade-off, you’ll need to work there 4-7 years to earn your $100-$150k stock option reward, and you may have given up way more than that in time & salary to get there
So, in summary, its a lot better to start your own company than to be employed by a startup. And in many ways its less-risky and better to start your own company than to “have a job”. And I wish I could make more people take the entrepreneurial leap themselves because its simply not as scary as it seems.
I started my career in teleco, so I learned a very old school style of managing tech ops. 99.999% availability, planned everything, control-room style networks, etc. 10-levels of permissions & sign-offs to change anything. So imagine me, coming from this world, building a new tech ops stack and processes, and I stumbled across the above presentation from John Allspaw at Velocity. The idea of doing 10+ deploys/day, coming from a world of big releases and maintenance windows, completely blew my mind away. This presentation and ideology has morphed into what is now known as “devops”, which I think is really the only way for a startup to operate their service.
Stumbling around the internet back in the founding days, I found some other tech ops resources & people that helped me a lot in changing my view.
@netik – John Adams who heads up ops at Twitter. He presents often and you can find a lot of his materials on how Twitter handles tech ops topics. This is one of his more recent presentations: http://www.ritholtz.com/blog/2011/05/twitter-ops-clouds-scale/. (I’m amazed that Twitter continues to rely on MySQL as its backbone (so do we at Peek, but obviously at a fraction of their scale))
http://codeascraft.etsy.com/ – Etsy’s notes on deploying and engineering, note that John Allspaw is now at Etsy. Their one-click deployinator tool is intriguing.
@vvuksan – Vladimir has a great blog http://vuksan.com/blog/ with dense information, especially around gathering data and stats.
@cgoldberg – Corey Goldberg has generated a ton of tools for performance testing and understanding the capabilities of your software.
On top of a lot of the information provided above, I’d suggest a few other practices for your initial tech ops infrastructure.
Decouple from your hosting provider
You should only ever require an instance (virtual or physical) + a base OS build from your hosting provider. If you depend on them for more, you make it difficult to switch providers. So if they have an embarrassing outage, you won’t be able to do anything about it. To be more specific in Amazon this means you should always use “raw” AMIs, such as the Canonical Ubuntu AMIs, and then add on the software you need via Puppet, Chef, and your favourite config & deploy tools. If you did this, it would have meant launching a new server in RackSpace would have taken about 20 minutes, and you would have avoided the whole Amazon outage.
Favour tools, automation & culture over process & documentation
Personally, I think wikis are bad, nobody reads them (or remembers logins to them). With the time you could write a procedure in a wiki, you could have probably built a ruby or bash script that did whatever you were explaining. So if you install apache or ejabberd or something, don’t take notes on installation, setup a script in Chef or Puppet, so next time it takes 2 minutes to setup. Everything should be one click/command tools or scripts.
Own your own hardware brother
At some point Amazon and the cloud will stop making sense. This post called Petabytes on a Budget does a great job on outlining why that is the case. Eventually with enough scale, the cost of expertise < cost of cloud.
One last note, a pitch for a product I rely on a lot. If you need an app to wake you up late at night in your tech ops world when things go wrong, I'd take a look at Toronto's own PagerDuty.com. We used them to replace a $2500/month service we had with a $50/month service Pagerduty offers. (I have no connection to them other than liking their product)
I’d love to hear some thoughts from others on how they run their infrastructure.
A few years ago the two memes of minimal viable product and lean product development were my favourites. There was this old-world, chronic problem of products getting stuck in product development until they reach “nirvana” – feature-itis, scope creep, etc. There was nothing more frustrating than sitting in a meeting where a product manager (with zero research or understanding) would state something like “we need to support groups, we absolutely can’t launch with out it or else it will be embarrassing”. So 18 months later product dev’t would complete, a few million would have been burnt and then we’d test it on the market and lo and behold nobody cared about that feature. Expensive. We can call this the “Microsoft Excel” product development methodology. Why Lean Product Development Evolved… copyright Phil Gwinn
Now I am seeing the opposite problem. Startups are launching crap under the excuse of “testing the market”. I think as a collective group we need to come together and put down a firm set of launch requirements:
1. Crashing should not be a launch feature. Please stress your stuff.
2. Whatever your apps core purpose is, it should do it fast. i.e. you should do performance testing.
3. You should use real designers – it should be easy to use and be easy on the eye.
You should still aim for simplicity & a bare bone set of features, but your product must work and meet the expectation of a professional product. A great example of a company doing this well has been Kik. Their core messaging product worked, worked fast and looked great – was super simple but also had some well thought out viral hooks. They even survived eXtreme scaling!
An exception to MVP, I don’t think it would make a lot of sense to use MVP if you were attacking an existing industry. Sometimes you know the business model, you don’t need to test it out. And your product has to exceed the current bar of products out there. This, for instance, is why it was ridiculous for the Playbook to be missing major features (like email) at launch.
[/caption]
