By Chow Lin
A number of us, coming from Google and other large companies, have seen various styles of management and decision-making, each with their pros and cons. In the beginning, there was Decision Making 1.0: business people made all the decisions (IBM/GE/Bloomberg) and the engineers did all the work; it was a nice prototype that worked for some, but was clearly rushed to market and needed iteration. Then, an upstart breed of tech companies including Microsoft and Google released Decision Making 2.0: this is where PMs look at the market and make decisions, but engineers can ignore them; engineers had an effective veto, but their level of participation was otherwise unclear.
After leaving Google, a bunch of us asked ourselves, is there a way to further improve on this? We had some ideas, did a lot of research on how others have tackled the problem, but in the end, we had to run some experiments–and if they didn’t work, we’d roll them back and try something else. As an engineering-heavy organization, we began with the premise that engineers are amazing number-crunching decision-making machines (every line of code requires a number of tradeoffs between memory usage, run-time, readability, verbosity and a bunch of other factors). What if we could find a system that would allow all of the engineers and PMs to work together to make decisions in a logical and metrics-based way? Later on, as we grew, the question would encompass not just engineers and PMs, but every role in the company.
Goal
A core part of the culture at WorkSmart Labs is making sure that everyone plays a central role in decision-making. That’s not because we don’t have a strong vision (we do); however, we believe that if a person does not love what they are working on, they’re less likely to be successful. And especially in a startup, everyone has to pitch in on every key project, so every voice needs to be heard: everybody has an equal seat at the table and all votes are equal. But a big question we had to address early on was, how do you avoid falling into the trap of just working on ideas that seem cool but are not at all aligned with the company’s goals or are relevant only to a small portion of our users? And how do you avoid the bureaucratic paralysis that plagues most group decision-making?
First, let’s define exactly what we’re trying to achieve:
- We want to make the best decisions possible; that is, choose the course of action that will result in the largest improvement of the product for our customers and increase in the value of the company in the shortest amount of time. And do this in highly ambiguous market conditions.
- We want to make sure that everyone has a say in how decisions are made and that all voices are respected equally.
- We want to open the floor to any idea, no matter how crazy, and give it a chance to live. The group might kill it later, but we shouldn’t be afraid to think big or think crazy.
- We want to be as quantitative and metrics-driven as possible. Logic and data should trump “gut feeling” whenever possible.
How Do Other People Do It?
With something as important as our basic decision-making process, we figured it was important to figure out how others did it, and, specifically, what were the best companies doing? Then, of course, not to blindly copy them (a lot of processes from big proven companies would kill a startup), but to figure out how we can adapt the parts that make sense to us.
In a lot of modern tech companies, there is a Product Manager whose role is to figure out a product vision that is aligned with the company goals and maximizes value. This person is expected to be the liaison between engineering and business, a person with a strong business and technology background; they are trusted with dictating the direction of the team.
Many companies that we strongly respect use this system; however, there are a number of flaws with the “PM as Hero” approach. The most prominent is that there’s a single point of failure: the PM. If the PM is really good and is a market whisperer, you win; but if they’re not, and they make strategic mistakes, you can waste a lot of valuable time going in the wrong direction. Another consequence of the PM as Hero approach is that it implicitly creates a top-down culture where engineers do not have an equal say in the direction of the project and are discouraged from thinking about whether or not the strategy makes sense. This makes it difficult for engineers to evolve into metrics-driven PMs or to feel like we own what we’re building. It also results in the engineers and PMs speaking different languages, often talking past each other, and not understanding each others’ priorities. At WorkSmart, we’d rather make sure that everyone understands why we’re going in a particular direction and has a say in it instead of creating a system where some people can dictate, “pull rank”, or ignore relevant data.
37 Signals (http://37signals.com/svn/posts/2640-another-round-of-lessons-learned-from-our-new-team-based-way-of-working) has tried to keep things democratic by using a group ideation/voting system: a repository where people can think of ideas, refine them as a group, and vote them up or down. We generally liked this approach: everyone is involved and working on the “best” features. The main drawback with many existing systems is that the only vote you can make is a binary “like/dislike” decision, and the ideas with the most likes wins; oftentimes, people that are making these like/dislike decisions are not thinking about key product metrics. The end result is a tendency to build the ideas that we think are “coolest” or that we would like to have for ourselves, but that might only impact a very small percentage of our users. So this is clearly a step in the right direction, but can we do more to stay focused on maximizing impact for our users? The answer lies in metrics.
Voting on Metrics
As a quick backgrounder into startup metrics, if you haven’t checked out Dave McClure’s awesome presentation, check it out here: http://www.youtube.com/watch?v=irjgfW0BIrw&noredirect=1 The ten-second summary is that you can measure a small set of metrics (acquisition, activation, retention, referral, and revenue), assign different values or weights to each metric, and use that as a tool to evaluate how you’re doing over time (cohort analysis).
You can find lots of great information out there on the importance of metrics; there are others that explain that importance far better than I, so I’ll just present a quick summary. Metrics help a company in a number of ways:
- Measuring metrics allows you to know you’re moving towards the right goals, and allows you to see if you’re increasing impact (however you define it).
- Coming up with metrics forces you to think through what’s most important for the company and how to measure it. They help you stay focused on what’s important and say no to the things that are not as important (you have finite resources, after all).
- By measuring progress across time and experiment cohorts, you can see the impact of various features and experiments on your customers.
The question we tried to answer at WorkSmart is, how do you inject quantitative metrics-based thinking into the early planning phases? After all, if it makes sense to measure your progress, it makes sense to try to figure out which activities will most likely allow you to maximize your progress (for the engineers out there, it’s like the difference between A* Search and an undirected search).
Metrics in the Planning Phase
In principle, it seems to make sense to extend the metrics to the planning phase; if you’re trying to optimize for some result, it’s important to be thinking about those results when you’re choosing what actions to do. In practice, we take our existing AARRR (acquisition, activation, retention, referral, revenue) numbers, and we vote by estimating the impact of each feature on each of the AARRR metrics. For example, how does adding a point-based game mechanic affect our product? I think that, most likely, it’ll have a greater impact on retention than on acquisition, so I would cast a strong vote along the retention metric and a weaker vote along the acquisition metric. How do you then compare it to, say, building a seasonal promotion that will increase acquisition? Easy–you plug the numbers into the AARRR formula for calculating total impact, and voila, you get a stack rank of your ideas based on impact.
This sounds like a lot of work. After all, if we have a hundred big ideas every quarter, and we have to vote along five different dimensions, that’s 500 estimates we each need to make. It does take time, but with something this important, the extra time is worth it: we end up making better decisions and come out of it a stronger and more unified team. Of course, there are things you can do to speed this up: in the beginning, we sat around a table and did the estimates and discussion in person, but this got slow as we grew; then we did it in a spreadsheet, but building it was a pretty complex process; eventually, we built a tool that allows us to enter ideas, comment on ideas, and vote using AARRR metrics.
The tool takes care of the number crunching and stack ranking in real-time to produce a list like the following.
This means that at any point in time, we have a good idea of the best ideas for our company to pursue; it’s a living repository that’s growing and being used.
But, as with all of our hypotheses, we need to ask, does it work? We’ve been doing this for over a year now, and we’ve learned a couple of things. First, surprisingly, an organized system actually allows you to be more open and creative because you trust the system and how decision making is done. Second, you need to throw in a sanity checking stage at the end. This means going over the stack-ranked list together and making sure that the numbers didn’t produce something weird where a great idea got buried or a horrible idea topped the list; this last quarter, we took about an hour and a half to do the sanity check. After sanity-checking the list, everyone decides on what to work on. What we’ve found is that, generally, the top third of the list is packed full of great ideas, the bottom third, mostly weaker ideas, and the ones in the middle are hit or miss. Usually, we have more ideas than we have time and resources, so the ideas in the bottom half don’t even matter so much; by the time we get to them, we will have had another voting session and have even better ideas from new learnings. We’ve found this system hugely helpful in figuring out the most important things to work on to maximize user happiness.
Conclusions
The main thing we got out of this series of experiments was that we now know that it’s possible to make decisions as a group quickly and efficiently. Does this mean it’s the best method in all circumstances? No, of course not. There are definitely cases where it’s really important for a person with a totally unsupported idea to take the lead and flesh it out before other people can intelligently chime in on it (it’s hard to argue when there’s no data). And what works for us might or might not work for you. But what we’ve found is that our culture of constant experimentation and willingness to try different things, even with things like decision-making, has allowed us to find something empowering, efficient, pretty scalable, and works quite well for us.
If you like to see this in action, check out idea-tool.appspot.com. Or if you like the idea of directly impacting the strategic direction of a company, of being able to suggest any idea and have it heard, and want to build amazing wellness technology to improve the lives of millions of people really appeals to you, come check us out (http://www.worksmartlabs.com/jobs.php).
