In the face of continuing economic turmoil, business, government and, indeed, all of us face increasing challenges in making even the most basic decisions.
Old certainties can no longer be relied upon; traditional approaches no longer work. In this situation, new technologies and tools in Business Intelligence (BI) seem ever more attractive. But, will that be enough to face down the challenges, to overcome the turmoil? We review some recent technological developments in BI and conclude that together they drive a new way of thinking about business intelligence, data warehousing and information usage in the broadest sense in the enterprise.
For almost two decades, general purpose relational databases have held centre stage in data warehousing and since the mid-90s have become the mainstay of information management. In the past few years, however, hardware developments and software advances have combined to produce a variety of new approaches—from data warehouse appliances to mobile BI, from BI in the cloud to big data—that redefine the very foundations of BI and are even causing some to question the future of relational databases. It would be easy to be overwhelmed by the array of technologies now available. But, if we focus on three key trends, we can see the road ahead relatively clearly.
The first and very obvious trend is big data—the volumes and varieties of information available to support decision making is growing rapidly. According to IDC’s Digital Universe Study, so-called “unstructured” or soft information such as video, image and text, is growing at over 60% compound annual growth rate (CAGR). Even highly structured or hard data within enterprises is growing at over 20% CAGR. Database vendors have responded to the latter growth with parallel processing and solid state disks in hardware and software advances such as columnar storage and compression. This has enabled database technology to achieve an order of magnitude improvement in size and speed. But it is the enormous growth of soft data that has made headlines, along with non-database solutions such as MapReduce and Hadoop. These big data programmatic approaches, mostly Open Source, have reopened the old debate between the procedural and declarative models for accessing data.
While more information cannot guarantee better decisions, it certainly allows decision makers to view the world in a broader context and potentially with more clarity. Big data applications typically focus on analysis of large web usage and scientific/engineering data sets, graph analytics, text analysis and so on. Its programmatic approach makes it ideal for one-off analyses using throw-away data or for urgent one-off reports. Much of the “social analytics” of web usage, Twitter or Google trending, identification of influencers and so on uses these techniques, allowing rapid decisions that mitigate costly PR mistakes or develop products that anticipate public demand.
The second key trend is accountability and, in some ways, is in direct conflict of the big data trend. This growing demand for accountability, auditability and tracking of decision making has emerged as a result of the financial and economic collapse since 2008, where it became clear that much decision making was far from transparent. Anti-crime and -terrorism measures also demand increased information accountability. This trend reemphasises the need for consistent, cleansed and historically accurate information and, of course, harks back to the principles and practices of data warehousing. However, it is clear that the volumes and varieties of big data cannot and should not be forced unthinkingly through the traditional architectural structures of the data warehouse. Soft information is inherently unmodelled and contains diverse and changing content—so how would one prepare it and load it into the data warehouse? Given volumes of the order of terabytes of even petabytes, does it make sense to store even one copy of the original data?
The third trend is empowerment: as BI comes ever closer to the business in self-service approaches, whether on tablet devices or in the cloud, users can freely explore and play with relevant information for sales innovation, cost balancing and more, free from their old dependency on IT. Empowerment is similar to the big data trend in its implications for accountability, tending to drive more copies of more data onto local or cloud storage. However, demographic changes in the population of workers indicate that empowerment and collaboration will be key drivers for innovation in the coming years.
So, we have some very important and contradictory trends, and with the growing wealth of information and power of analysis come the insidious threat of data chaos. How can we know which information is most correct or timely? Which analysis the most trustworthy or relevant? A renewed emphasis on data governance, process quality, management and organisational disciplines and the bigger architectural vision becomes far more important than evaluations of whose tool is biggest or fastest.
Business Integrated Insight (BI2), as described in a series of articles on B-eye-Network (Devlin, 2010) and in the Business Intelligence Journal (Devlin, 2010a), provides a framework to resolve these conflicts. For some, the answer will be challenging, because it requires looking more broadly than traditional BI and considering all the information used by the enterprise, all of the processes that interact with that information and all of the people who operate the processes. In short, it involves breaking down all the barriers between different parts of the business—even between business and IT— and bringing together all of the silos of information in a single virtual store. Virtual is the key word here: information will continue to exist in the most appropriate stores and technologies, depending on its type and use. However, this virtual store requires at its heart a core set of highly managed and dependable information, which has been extracted from many locations, modelled and reconciled. This core business information, unlike the traditional data warehouse, is not seen as the single source of all BI; rather it is the reference point and anchor information that tie together the different valid and valuable views of the business.
The recent major advances in relational database performance, parallel processing (used by both databases and big data applications) and solid state storage, together with the developments in distributed processing and virtual storage seen in cloud computing all enable the architectural vision described by BI2. These same technologies, as well as social networking and collaborative tools, are already driving new decision making applications of information in the business sphere. And technology already on the horizon, such as in-memory databases, soft information modelling and the semantic web (Web 3.0) will provide the further tools needed to make the transition.
However, while such technology advances certainly do drive BI2 architecture and use, and often receive media and analyst attention, major innovations within businesses in the softer areas of organisation and governance will be required to plan, implement and use this new approach effectively. As information comes to be seen as an ever more significant asset of the business, the IT organisation—who has most knowledge of and responsibility for information, whether they accept it or not—will have to take a more business-oriented role. The Chief Information Officer will have similar power and influence as the Chief Financial Officer in organisations that successfully exploit their information to the maximum extent.
It was often claimed that business intelligence played a strong role in successful business decision making—and in some few cases it actually did. However, the successful integration of traditional BI, big data applications, long-established operational processing and novel collaborative working approaches, emerging as Business Integrated Insight, will be the foundation for significant business advantage in the coming years.
Barry Devlin è uno dei più autorevoli esperti mondiali di Business Insight e Data Warehousing. Consulente, speaker e autore del famoso libro “Data Warehouse – from Architecture to Implementation”. L’interesse di Mr. Devlin oggi si estende a un Business completamente integrato, coprendo ambienti informational, operational e collaborative per offrire una esperienza olistica del Business attraverso l’IT. E’ fondatore di 9sight Consulting, una società di consulenza specializzata nel progettare soluzioni ampie di Business Insight lavorando con prestigiosi analisti, vendors di BI e molto di più.
Presenterà a Roma per Technology Transfer il seminario “BI²: dalla Business Intelligence all’Enterprise IT Integration” dal’11-12 aprile 2011, inoltre sarà uno degli speaker dell’ “
International Data Warehousing and Business Intelligence Summit 2011” che si svolgerà a Roma dal 22 al 24 giugno.
