Big data is about more than Hadoop and a bunch of fancy technology: there are some very real organisational barriers too.
It's a bit of a mirage. As soon as you get your head around it, it ceases to exist.
How so? The accepted definition for Big Data talks about exploiting “data sets whose size is beyond the ability of commonly used tools to process it within tolerable time”. By that definition, as soon as you’re comfortably handling the data, it ceases to be big.
Nonetheless, Big Data is clearly trending amongst the tech analysts, and it’s doing so for good reasons. The volume of data we’re handling is growing dramatically, Social media, the internet of things. The mass of data produced by smart electric grids, intelligent traffic systems, etc.
90% of the data ever created has been created in the last two years...
Small, frequent, fine-grained interactions erode organisational boundaries.
How will organisations evolve in the face of mobile interaction patterns?
When a team can't meet customer demand, that demand starts to go underground.
This can have unintended side effects...
Problems rarely kill projects. What kills them is failure to recognise and address problems.
A colleague said recently that online projects fail for three main reasons – poor governance, weak communication or problematic technology, and in roughly equal proportions.
I’m not sure I agree about the proportions, but the categories feel useful.
For a start, each has its own distinctive failure modes.
Service level agreements get broken all the time. That's as true in the physical economy as it is in the digital one.
I learnt a lot about service levels when I worked in a coalmine.
At one level, coal mining is simple. Coal is valuable. The shale that surrounds it isn’t. Mining is about digging out as much coal as possible while leaving the shale behind.
In practice, things are a little more complex It’s often hard to dig out the coal without shifting a lot of shale, and the boundary between the two is often unclear – strata that are 90% coal and 10% shale grade gradually into strata that are 90% shale and 10% coal.
And predicting the exact shape of the coal seam as it meanders through the earth requires a lot of skill. But what makes things really complex are the contracts, the mine’s equivalent of service level agreements.
If the Cloud is just about cost savings, then it's pretty dull. It could be about so much more.
The reality behind the cloud is pretty boring. When you look closely at what most people are doing there, it’s all about cost savings.
Take a commodity application from your on-premise servers and push it out to the cloud and your costs (probably) go down.
Few applications are heavily loaded 24 hours a day, seven days a week, so paying only for the compute capacity you need as you need it makes a lot of sense, but it's hardly exciting.
Projects don't come about through some law of nature. They're a mental construct we've created to help ourselves manage our work.
I spend most of my time on projects.I’ve worked on projects to build e-commerce systems, projects to build and update websites, projects to create new products, even projects to define the way we do projects within an organisation.
The idea of a project as the fundamental unit of work is pretty pervasive within our industry. Many organisations structure themselves almost entirely around the portfolio of projects that they are undertaking.
And a large industry, think of PRINCE2 and the PMI and various other project management bodies, has emerged solely to service the interests of projects and project managers.
But how real are these projects?
IT departments can create a lot of value when they take responsibility for Integration Technology, bringing together the activities of people across the organisation and beyond its increasingly porous boundaries.
But if they sit behind their firewalls chanting verses from the ITIL, then they deserve to die.
Making decisions requires mental energy. When your energy is low, 'decision fatigue' sets in and some fairly predictable biases result.
Too many teams set themselves up to operate in an almost permanently fatigued state. How can you avoid this?
What does a good strategy look like? How can we recognise the right one when we see it? The problem is, even in retrospect it's hard to recognise a good strategy. As often as not, we end up creating a myth.
Successful organisations often point to a clear strategy underlying their growth. They analysed the market, identified leverage points, focused their resources on those points, and reaped the benefits. Their success is a reward for good strategy and hard work.
Or so they say. I don’t think it really happens that way.