Analytics X is an ongoing contest to apply analytics, modeling, and statistics to solve the social problems that affect our cities.  It combines the fields of statistics, mathematics, and social science to understand the root causes of dysfunction in our neighborhoods.  Understanding these relationships and discovering the most highly correlated variables allows us to deploy our limited resources more effectively and target the variables that will have the greatest positive impact on improvement.

Current Contest - 2010 - Predicting Homicides in Philadelphia

Philadelphia is a city with 5.8 million people spread out over 47 zip codes and, like any major city, it has its share of crime.  The goal of the Analytics X competition is to use statistical techniques and any data sets you can find to predict where crime, specifically homicides, will occur in the city.  The ability to accurately predict where crime is likely to occur allows us to deploy our limited city resources more effectively.  Full rules can be found on the Rules & FAQ page.

Jakob Nielsen has written about this:

In my two examples, the probability of making the right design decision was vastly improved when given the tiniest amount of empirical data.

If there’s one thing we should all take to heart, it’s that humans are strange: They rarely behave the way we expect (or want) them to. Testing often reveals issues we would never have found out by merely thinking about a design. Conversely, something that looks wrong might actually work perfectly

He was trying to sell them on the idea of a high-powered analysis platform that could scan multiple databases simultaneously— a tool that government officials and corporations could use to tackle complex problems.

“It was very scary since doing enterprise software [from] 2005 to 2009 was a little bit like starting a circus in the middle of Palo Alto with engineers,” Karp says, “Enterprise is a dirty word and that’s the business we’re in, and government is also not very popular in the Valley, [we combined] both.” [See our interview with Karp above]

Bump, which launched in March 2009, has so far has been downloaded 15 million times. The three founders projected a million downloads at best in the first year. Much of that growth has been organic — no plugs from Apple in their commercials — and very viral. When analyzing the usage data of its customers, the company realized that nearly two-thirds of its customers were going from download to sharing within 15 minutes. “To us, it said that people were making their friends download the app and share information with them,” Lieb said.

That little discovery became the driving force for the company; they’re now ruthlessly trying to reduce the time it takes to download and start using the application. It’s driving the company’s entire design process and user experience, and in doing so, the company has seen its usage explode, especially on the weekends. “We see a sharp increase in usage and growth over the weekends,” said Lieb. That trend follows the enhanced social interactions — swapping contacts and exchanging photos —  that take place over the weekend.

Clearly you need to look at the data and see the patterns. Once you recognize the pattern and see its relevance to the business make it a priority to exploit it.

To create value for the customer – which I hope we agree is how we should be earning our living – a series of steps must be conducted properly in the proper sequence.  These steps collectively are what we call the value stream for each product.  As I walk through any value stream – and I walk a lot every year as I visit many companies in many countries -- I ask the following very simple questions about each step:

Is the step valuable?  Or would the customer be equally happy with the product if the step could be left out?  If the latter is the case, the step is at best what Toyota would call “incidental work” and what I often call Type One muda.  Get rid of it as soon as you can!

Is the step capable?  Can it be conducted with the exact same result every time?  This is the starting point, but never the end point, for Six Sigma.

Is the step available?  That is, can it be performed whenever it is needed?  Or is the step subject to breakdowns and varying cycle times so you are never sure what will happen?  This is the starting point, but again not the end point, of Total Productive Maintenance.

Is the step adequate?  That is, is there capacity to perform it exactly when the value stream requires it?  Or is there a bottleneck?  Bottleneck analysis is, of course, the starting point of the Theory of Constraints.  Or, and more likely in the current era, is there too much capacity?  Toyota tries to avoid this by adding production capacity in small increments rather than in big hunks, increments that can be flexed by adding or subtracting employees.

Is the step flexible?  Can it shift over quickly from making green ones to making red ones quickly?  And can it changeover without compromising capability, availability, and adequacy?  Flexibility is the key to rapid response to changing customer desires while avoiding the inefficient production of big batches.

If all the steps in your value streams are valuable, capable, available, adequate, and flexible, you are well on your way.  What remains is to perfect the linkage between the steps.

The Logic of Scientific Discovery

'One of the most important documents of the twentieth century.' Peter Medawar, New Scientist

‘Wonderfully exhilarating’ Naomi Bliven, New Yorker

'One of the most important philosophical works of our century.'
Richard Wolheim, The Observer

First published in English in 1959, The Logic of Scientific Discovery revolutionized contemporary thinking about science and knowledge and is one of the most widely read books about science written in the twentieth century. Described by the late philosopher A.J. Ayer as a work 'of great originality and power', it present succinctly Popper's view of science and his solutions to two fundamental problems of the theory of knowledge: the demarcation of science from non-science, and the role of induction in the growth of scientific knowledge.

Popper recognised that scientific theories are the result of a creative imagination and that the growth of scientific knowledge rests on the doctrine of falsifiability: that only those theories that are testable and falsifiable by observation and experiment are properly open to scientific evaluation. These stirring ideas had a hugely significant influence on the philosophical and scientific communities and are central to the development of the philosophy of science. Translated into many languages, The Logic of Scientific Discovery ranks alongside The Open Society and Its Enemies as Popper's most important book and a major contribution to modern thought.

The Lean Startup movement has increased the interest in lean and agile development.  I have studied and practiced this stuff for years and seen legit ideas and execution, as well hype and mysticism.

Here are some of the key lessons based on my own experience and the experience of other practitioners I have worked with.

A/B Testing

Powerful analytics provide some amazing capabilities for businesses such as ‘A/B Testing’. This is the process where a business may make a minor change and then observe the effects of that modification to decide if the change was fruitful.

A store like Makro may put up discounted confectioneries at check-out counters in one store on any given day and compare performance of the store against an outlet where no changes were introduced. Using analytics software, a retail chain can make hundreds of similar changes in a month and determine the success of products with accuracy. Purely software companies like Google go so far as to running all changes to their homepage through an A/B test and vetting out any change that does not yield positive results.

While this is only the tip of the iceberg, business analytics technology has evolved considerably to incorporate real-time reporting, statistical tools and reporting. There is plenty of room to improve with the right set of tools.