Defining Business Analysis for Big Data

In Big Data, Business Analysis by IRM UKLeave a Comment

Print Friendly, PDF & Email

In the beginning of the Information Era, there was business analysis for application development. Waterfall methods, Rapid Application Development, Agile methods,.. all were based on delivering a functioning piece of information technology that supports a well defined business process. There are clear signs of an evolution in the application development area.

Introduction to an enhanced methodology in business analysis.

Bert Brijs,

Speaking at the Business Analysis Conference Europe 2015, 21-23 September, London

Automating the value chain

Core operations like manufacturing and logistics came up with automation of human tasks and the IT department was called the “EDP department”. Some of the readers will need to look up that abbreviation. I can spare them the time: Electronic Data Processing indicated clearly that the main challenge was managing the data from these primary processes.


This schema gives a few hints on the progress made in automation of business processes: the core operations came first: finance, logistics and manufacturing which evolved into Enterprise Resource Planning (ERP). Later sales, marketing and after sales service evolved into customer relationship management which later on extended into Enterprise Relationship Management (ERM) incorporating employee relationship management and partner relationship management. Finally ERP and ERM merged into massive systems claiming to be the source of all data. The increase in productivity and processing power of the infrastructure enabled an information layer that binds all these business processes and interacts with the outside world via standardized protocols (EDI, web services based on SOAP or REST protocols).

The common, denominator of these developments is: crisp business analysis to enable accurate system designs was needed to meet the business needs.

The “Information is the new oil” era

Already in the mid nineties, Mike Saylor, the visionary founder and CEO   from Microstrategy stated that information is the new oil. Twenty years later, Peter Sondergaard from Gartner repeated his dictum and added “and analytics is the combustion engine”. A whole new discipline –already announced since the 1950’s- emerged: Business Intelligence (BI). Connecting all available relevant data sources to come up with meaningful information and insights to improve the corporate performance dramatically.


The metaphor remains powerful in its simplicity: drill for information in the data and fuel your organization’s growth with better decision making.


Luther at the Diet of Worms in 1521

Yet, the consequences of this new discipline on the business analysis practice remained unnoticed by most business analysts, project managers and project sponsors. The majority was still using the methods from the application development era. And I admit in the late nineties I have also used concepts from waterfall in project management and approached the products from a BI development track as an application where requirements gathering would do the trick. But it soon became clear to me that asking for requirements to a person who has an embryonic idea about what he wants is not the optimum way. The client changes requirements in 90 % of the cases after seeing the results from his initial requirements. That’s when I started collecting data and empirical evidence on which approach to a business analysis method leads to success. So when I published my book “Business Analysis for Business Intelligence” in October 2012, I was convinced everybody would agree this new approach is what we need to develop successful BI projects. The International Institute of Business Analysis’s (IIBA) Body Of Knowledge has increased its attention to BI but the mainstream community is still unaware of the consequences on their practice. And now, I want to discuss a new layer of paradigms, methods, tricks and tips on top of this one? Why face the risk of leaving even more readers and customers behind? I guess I need to take Luther’s pose at the Diet of Worms in 1521: “Here I stand, I can do no other.” So call me a heretic, see if I care.

The new, enhanced approach to business analysis for business intelligence in a nutshell deals with the bridging three gaps. The first gap is the one between the strategy process and the information needed to develop, monitor and adjust the intended strategic options.

The second gap is about the mismatch between the needed and the available information and the third gap is about the available information and the way data are registered, stored and maintained in the organization.

Now, with the advent of Big Data, new challenges impose themselves on our business analysis practice.

Business Analysis for Big Data, the new challenges

But before I discuss a few challenges, let’s refer to my definition of Big Data as described in the article “What is really Big About Big Data” In short: volume, variety and velocity are relative to technological developments. In the eighties, 20 Megabytes was Big Data and today 100 terabytes isn’t a shocker. Variety has always been around and velocity is also relative to processing, I/O and storage speeds which have evolved. No, the real discriminating factor is volatility: answering the pressing question what data you need to consider as persistent both on semantic and on a physical storage level. The clue is partly to be found in the practice of data mining itself: a model evolves dynamically over time, due to new data with better added value and / or because of a decay in value of the existing data collection strategy.

Ninety percent of “classic” Business Intelligence is about “What we know we need to know” . With the advent of Big Data the shift towards “What we don’t know we need to know” will increase. I can imagine in the long run the majority of value creation will come from this part.

From “What we know we need to know” to

“What we don’t know we need to know”

is the major challenge in Business Analysis for Big Data

Another challenge is about managing scalability. Your business analysis may come up with a nice case for tapping certain data streams which deliver promising results within a small scope but if the investment can’t be depreciated on a broader base, you are dead in your tracks. That’s why the innovation adage “Fail early and fail cheap” should lead all your analytical endeavors in the Big Data sphere. Some of you may say “If you expect to fail, why invest in this Big Data Thing?”. The simple answer is “Because you can’t afford not to invest and miss out on opportunities.” Like any groundbreaking technology at the beginning of its life cycle, the gambling factor is large but the stakes are also high. As the technology matures, both the winning chances and the prize money diminish. Failing early and cheap is more difficult than it sounds. This is where a good analytical strategy, defined in a business analysis process can mitigate the risks of failing in an expensive way.

Business Analysis for Big Data is about finding scalable analytical solutions, early and cheap.

So make sure you can work in an agile way as I have described in my article on BA4BI and deliver value in two to three weeks of development. Big Data needs small increments.

Data sources pose the next challenge. Since they are mostly delivered via external providers, you don’t control the format, the terms and conditions of use, … In short it is hard if not impossible to come with an SLA between you and the data provider. The next challenge related to the data is: getting your priorities right. Is user generated content like reviews on Yelp or posts in Disqus more relevant than blog posts or tweets? What about the other side of the Big Data coin like Open Data sources, process data or IOT data? And to finish it off: nothing is easier than copying, duplicating or reproducing data which can be a source of bias.

Data generates data and may degenerate the analytics

Some activist groups get an unrealistic level of attention and most social media use algorithms to publish selected posts to their audience. This filtering causes spikes in occurrences and this in turn may compromise the analytics. And of course, the opposite is also true: finding the dark number, i.e. things people talk about without being prominent on the Web may need massive amounts of data and longitudinal studies before you notice. Like a fire brigade, you need to find the peat-moor fire before the firestorm starts.

The architectural challenge is also one to take into account. Because of the massive amount amount of data and their volatility which cannot always be foreseen, the architectural challenges are bigger than in “regular” Business intelligence.

Data volatility drives architectural decisions

There are quite a few processing decisions to make and their architectural consequences impact greatly the budget and the strategic responsiveness of the organization. In a following article I will go into more detail but for now, this picture of a simplified Big Data processing scheme gives you clue.


Enabling Business Analysis for Big Data

We are at the beginning of a new analytical technology cycle and therefore, classical innovation management advice is to be heeded.

You need to have a business sponsor with sufficient clout, supporting the evangelization efforts and experiments with the new technologies and data sources.

Allow for failures but make sure they are not fatal failures: “fail fast and cheap”. Reward the people who stick out their necks and commit themselves to new use cases. Make sure these use cases connect with the business needs, if they don’t, forward them to your local university. They might like to do fundamental research.

If the experiments show some value and can be considered as a proof of concept, your organization can learn and develop further in this direction.

The next phase is about integration:

  • integrate Big Data analytics in the BI portfolio
  • integrate Big Data analytics in the BI architecture
  • integrate Big Data analytical competences in your BI team
  • integrate it with the strategy process
  • integrate it in the organizational culture
  • deal with ethical and privacy issues
  • link the Big Data analytical practice with existing performance management systems

And on a personal note, please, please be aware that the business analysis effort for Big Data analytics is not business as usual.

What is the Added Value of Business Analysis for Big Data?

This is a pertinent question formulated by one of the reviewers of this article. “It depends” is the best possible answer.

The Efficiency Mode

It depends on the basic strategic drive of the organization. If management is in a mode of efficiency drive, they will skip the analysis part and start experimenting as quickly as possible. On the upside: this can save time and deliver spontaneous insights. But the downside of this non directed trial-and-error approach can provoke undesirable side effects. What if the trials aren’t “deep” and “wide” enough and the experiment is killed too early? With “deep” I mean the sample size and the time frame of the captured data and with “wide” the number of attributes and the number of investigated links with corporate performance measures.

The Strategy Management Mode

If management is actively devising new strategies, looking for opportunities and new ways of doing business rather than only looking for cost cutting then Business Analysis for Big Data can deliver true value.

It will help you to detect leading indicators for potential changes in market trends, consumer behavior, production deficiencies, lags and gaps in communication and advertising positioning, fraud and crime prevention etc…

Today, the Big Data era is like 1492 in Sevilla, when Columbus went to look for an alternative route to India. He got far beyond the known borders of the world, didn’t quite reach India but he certainly changed many paradigms and assumptions about the then known world. And isn’t that the essence of what leaders do?

All articles are © 2015 by the authors.

Bert Brijs,

Speaking at the Business Analysis Conference Europe 2015, 21-23 September, London


Leave a Comment