bi trend monitor 2020 cover

Infographic: BI Trend Monitor 2020

BARC’s BI Trend Monitor 2020 study gives BI practitioners a platform to have their say on the trends currently shaping the BI and data management market, supplemented by additional commentary and analysis from BARC analysts.

This infographic summarizes the importance that BI users and consultants attach to a series of 20 trends. It also reveals how these trends have evolved in popularity over the last five years.

Click here to find out more and to download the full study.

 

BI Trend Monitor 2020 infographic

 

Click here to find out more and to download the full study.

modernizing the data warehouse

Infographic: Modernizing the Data Warehouse – Challenges and Benefits

Modernizing the Data Warehouse: Challenges and Benefits, a study based on a worldwide survey examining companies’ approaches to get their data warehouse to the next level. In particular, it provides insights regarding technologies used, benefits achieved and lessons learned by companies that are in the process of modernizing their data warehouse.

Click here to find out more and to download the full study.

 

data warehouse modernization infographic

 

Click here to find out more and to download the full study.

modernizing the data warehouse

The Challenges and Benefits of Data Warehouse Modernization

PRESS RELEASE – WÜRZBURG, November 27th, 2019
The Business Application Research Center (BARC) publishes Modernizing the Data Warehouse: Challenges and Benefits, a study based on a worldwide survey examining companies’ approaches to get their data warehouse to the next level. In particular, it provides insights regarding technologies used, benefits achieved and lessons learned by companies that are in the process of modernizing their data warehouse.

Retrofitting the data warehouse: time-consuming development processes call for action

It is the central approach of IT to make cumbersome processes more efficient through automation. Therefore, it seems obvious that users should seek to streamline their time-consuming development processes for the data warehouse by means of data warehouse/ETL automation. In fact, these approaches provide benefits. Above all, these include broad data warehouse usage and the ability to handle a larger variety of data to support enhanced analytical requirements. Greater efficiency in data management processes, on the other hand, only ranks 8th among the benefits achieved.

Lack of agility and skills as well as data landscape complexity limit the ability to act

Inflexibility in on-premises environments is driving companies to the cloud. Yet on average only 29 percent of our participants are adopting cloud services for their data warehouse. Investments in infrastructure and technology, however, are of limited usefulness. Data landscapes tend to be too complex and not many organizations have established data governance frameworks with both business and IT participation. So not surprisingly, a lack of business and technical skills is seen as a central challenge (38 percent) in data warehouse modernization.

data warehouse modernization challenges

Figure 1: What are the biggest challenges your company faced in modernizing its data warehouse environment? Top five answers (n=380)

 

Leaders profit from their investments in a flexible data and process architecture and agile data services

Companies with greater data landscape maturity are more likely than others to use cloud services. They build leaner data architectures based on central data warehouses backed by analytical databases and implement fewer data marts. Microservices and API management play a greater role in their landscape. Their future prospects are more often focused on real-time analytics and streaming as well as new data sources and open data. They are therefore more likely to better support BI and advanced analytics requirements.

About ‘Modernizing the Data Warehouse: Challenges and Benefits’

‘Modernizing the Data Warehouse: Challenges and Benefits’ by Jacqueline Bloemen, Senior Analyst Data & Analytics at BARC, is a study based on a worldwide survey of 368 people with a professional interest in data management, conducted from January to May 2019. The study covers approaches to modernizing the data warehouse, technology use and current challenges. The full report is available free of charge thanks to sponsorship from Wherescape.

More information about ‘Modernizing the Data Warehouse: Challenges and Benefits’

Download the full study from the sponsor’s website:

About BARC

BARC (Business Application Research Center) is an enterprise software analyst and consulting firm with a focus on Analytics/Business Intelligence (BI), Data Management, Enterprise Content Management (ECM), Customer Relationship Management (CRM) and Enterprise Resource Planning (ERP). With experts in Germany, Austria, Switzerland and the United Kingdom, BARC supports clients in their software selection and strategy through specialist consulting, leading-edge research and a program of conferences and highly-focused seminars. For more information, visit:
www.barc-research.com

Contact

For all enquiries, please contact:

Regina Schmidt
Communications Manager
Tel: +49 (0) 931 880651-0
rschmidt@barc.de

BI trends 2020 ranked

BI Users Name Data Quality the Biggest Success Factor in Their Work

PRESS RELEASE – WÜRZBURG, November 13th, 2019
Which BI trends will shape the coming year? The enterprise software analyst BARC published the BI Trend Monitor 2020 at this year’s Big Data World in Frankfurt. With 2,865 BI users surveyed, the study offers a comprehensive overview of current trend topics in the global business intelligence market. For the third time in a row, BI users name master data and data quality management as the top trend.

Going into a new decade, the two most important trends – master data and data quality management in first place and data discovery in second place – remain unchanged compared to last year.

“BI users see the value of data. This means they are focusing on data quality, data governance and data strategy,” said Dr. Carsten Bange, founder and CEO of enterprise software analyst BARC. “Basically, they have realized that even the best ways to prepare data in dashboards or provide analysis functionality are worthless if they don’t have the right data or if the quality is not up to scratch.”

Third spot is now occupied by establishing a data-driven culture. This trend, which was newly introduced last year and went straight into fifth place in the rankings, is considered even more important this year. Self-service BI, on the other hand, dropped down to fifth place this year while data governance remains in fourth. All in all, these five top trends are the foundation for companies to manage and use their own data.

BI trends 2020

Figure 1: Importance of BI trends from “not important at all“ (0) to “very important“ (10), n=2865


Artificial intelligence not yet on the agenda of BI decision-makers

Some trends have slightly gained in importance since last year – including real-time analytics and integrated platforms for BI and PM.  However, no single trend has made any major gains.

Mobile BI dropped from twelfth to fifteenth place this year, continuing its downward trend. The use of BI functions on mobile devices may no longer be very interesting for BI users because it is either available already or because the requirements of companies are changing.

BI trends 2020 ranked

Figure 2: Development of rankings of BI trends since 2016, n=2794/2772/2770/2679/2865

Advanced Analytics, Machine Learning and AI are on everyone’s lips at the moment. Nevertheless, BI users rate this trend lower than they did last year (from nine to ten). More important than the difference in rank, however, is the reason behind this slight downward trend: In 2018, many hopes were based on new tools with artificial intelligence, so an increase for this topic might have been expected.

“Advanced analytics is probably not so highly valued as BI practitioners currently see other even more important trends in their work. Perhaps users are also skeptical about how the current, somewhat overblown topic of artificial intelligence can actually generate benefits in companies. Decision-makers are in the process of separating the hype from reality and identifying the most useful areas of application,” suggested Dr. Carsten Bange.

About the BI Trend Monitor 2020

With 2,865 participants, the BARC BI Trend Monitor 2020 is the largest worldwide trend survey on business intelligence and data management. It offers a detailed evaluation and weighting of twenty individual trends by region, industry and the self-assessment of companies (“best-in-class” vs. “laggards”). The full report is available free of charge thanks to sponsorship from Cubeware, Information Builders, MicroStrategy, TARGIT and Unit4 Prevero.

For more information on the BARC BI Trend Monitor 2020 and to download the full report, visit:
https://bi-survey.com/top-business-intelligence-trends

About BARC

BARC (Business Application Research Center) is an enterprise software analyst and consulting firm with a focus on Analytics/Business Intelligence (BI), Data Management, Enterprise Content Management (ECM), Customer Relationship Management (CRM) and Enterprise Resource Planning (ERP). With experts in Germany, Austria, Switzerland and the United Kingdom, BARC supports clients in their software selection and strategy through specialist consulting, leading-edge research and a program of conferences and highly-focused seminars. For more information, visit:
www.barc-research.com

Contact

For all enquiries, please contact:

Regina Schmidt
Communications Manager
Tel: +49 (0) 931 880651-0
rschmidt@barc.de

The Future of Reporting

Infographic: The Future of Reporting

Modernizing and optimizing enterprise reporting applications to support shifting and increasing demands is vital to continuously deliver trusted information for decision support in rapidly changing environments.

This study investigates current and future challenges with information delivery and how organizations plan to improve their enterprise reporting to gain benefits from their data treasures.

Click here to find out more and to download the full study.

Future of Reporting Infographic

 

Click here to find out more and to download the full study.

Data Monetization

BARC Survey Finds Data Monetization Is In The Early Stages of Adoption But Is Expanding

PRESS RELEASE – WÜRZBURG, April 23rd, 2019
The Business Application Research Center (BARC) publishes “Data Monetization – Use Cases, Implementation and Added Value”, the results of a survey sponsored by Tableau examining current approaches to data monetization in European companies.

17 percent of companies have established data monetization initiatives, a further 12 percent are currently building prototypes and another 10 percent are still developing concepts, according to new research from BARC. Large companies from retail, services, finance and banking are leading the way and, in general, data monetization is principally being implemented by larger companies. 25 percent of large companies and 23 percent of larger medium-sized companies responding to the survey have already launched data monetization products. In contrast, only 9 percent of small and 13 percent of small to medium-sized companies have already done so.

Data Monetization press release - Chart 1

Figure 1: Is data being monetized in your company? (n=200)

 

“Companies still have the opportunity to secure a competitive edge by engaging in data monetization”, said Sebastian Derwisch, BARC analyst and author of the report. “Our results show there is value in data monetization, but management needs to give priority to these initiatives. Therefore, data monetization projects must be tied to business goals and strategy.”

Providing results for process improvement is the main way to monetize data

Providing analysis results is the most common form of monetizing data. 40 percent of survey participants employ this type of data monetization, whereby data analytics is involved. The provision of data via reporting and benchmarking is almost as important (37 percent). Less common methods include establishing digital platforms (22 percent), extending existing products (17 percent), providing new services (16 percent) and building new business models based on data (6 percent).

Data monetization occurs mainly via the distribution of analysis results and through reporting and benchmarking. Fewer companies establish digital platforms, enhance existing products with new services or create new business models.

“The main focus is on the internal distribution of results as external data monetization is more complex, requiring companies to rethink business models”, commented Derwisch. “Furthermore, the higher standards of data quality and security required for external data monetization may be seen as a barrier.”

Data Monetization press release - Chart 2

Figure 2: How does your company monetize or plan to monetize data? (n=137)

 

Benefits of data monetization are tangible 

Data products can bring a broad range of benefits, from new revenue sources to a better understanding of customers and product improvements. New revenue sources are the most prevalent benefit of data products, reported by 69 percent of respondents. For 66 percent, the provision of new services is a benefit, and improved customer loyalty is cited by 63 percent of participants. Internal provision of the results of data analysis is a motivation for data monetization for more than half of respondents (59 percent), as is the internal provision of data and benchmarks (53 percent). Better insight into customers and improved customer experience – for example by personalization – is achieved by about 50 percent of participating companies. Generating new data is only viewed as a benefit by 38 percent, and binding partners and suppliers by 31 percent.

Data Monetization press release - Chart 3

Figure 3: Which benefits have you achieved by monetizing data? (n=32) 

 

About ‘Data Monetization – Use Cases, Implementation and Added Value’

‘Data Monetization – Use Cases, Implementation and Added Value’ is a survey-based study offering insights into how European companies are approaching data monetization. The online survey was carried out in October and November 2018, attracting 200 participants from a wide range of company sizes and industries. The full study is available free of charge thanks to sponsorship from Tableau. On completion of a short registration form, it can be downloaded from the Tableau website. BARC and Tableau will be presenting the survey results in a series of webinars in English, German and French on June 4th. For further information, visit: www.tableau.com

About BARC – Business Application Research Center – a teknowlogy Group company

The Business Application Research Center (BARC) is an enterprise software analyst and consulting firm with a focus on Analytics/Business Intelligence (BI), Data Management, Enterprise Content Management (ECM), Customer Relationship Management (CRM) and Enterprise Resource Planning (ERP). Along with Ardour, Le CXP and PAC, BARC forms part of teknowlogy Group – the leading independent European research and consulting firm in the fields of digital transformation, software and IT services with a network of 155 experts in eight countries.

For further information or media enquiries, please contact:

Nikolai Janoschek
Head of Market Research
Phone: +49 (0) 931 880651-0
Email: njanoschek@barc.de

BARC Logo

Standard vs Best-of-Breed

Introduction

Choosing the right business intelligence software for an enterprise is a complex job. BI solutions range from data management to BI tools and applications with varying feature sets and user groups. Mixing and matching software packages from several vendors is referred to as the best-of-breed approach. The alternative is to choose a single strategic vendor that provides adequate coverage of all the functional areas required.

Each approach has its advantages and disadvantages. For the best product for each functional area, best-of-breed solutions usually provide richer functionality. Typically, there are disparate cases to be addressed and no single solution will be able to deal with them all. Furthermore, users of existing tools often have a vested interest in keeping them, which makes the introduction of a suite of standard tools for the entire enterprise difficult. This is important because, as The BI Survey shows, satisfying users is a key to successful business intelligence projects.

However there are several arguments in favor of buying a tool suite from a single vendor including price, administration effort and effective data sharing between users. The promise of integrated systems is to provide multiple applications with common infrastructure elements such as user administration, and data management tools as well as a consistent user interface to reduce training costs. The risk is that some pieces of the combined offering may not be very good, which will discourage users from adopting them, increase implementation and maintenance costs, or slow down change management.

There is no practical way to make an exact cost calculation for a business intelligence project. There are simply too many variables. But when attempting to calculate the costs of software it is important to keep issues such as training and implementation costs in mind. Unfortunately, these costs are difficult to know in advance without detailed information about the project. In many cases this information is not available in the earliest phases of the selection process. It is also important to keep in mind that software maintenance can be expensive, so calculations of total cost of ownership have to include some estimation of how long the product will be used and how the number of applications and users might develop.

Best-of-breed systems, which lack the coverage of all the use cases but are designed to excel in one or a few application areas, can also pose challenges. These include increased training and support costs, difficulties integrating with other systems and data management issues caused by redundant data storage.

At the same time, nearly all suites are the result of larger companies acquiring several best-of-breed systems. As a result, it is often the case that customers are sold a set of incompatible products which have been relabeled to appear to be parts of the same product. In this case, the more technical arguments in favor of a suite do not apply, although organizational arguments (see below) may.

There is no simple answer to the question of whether to choose best-of-breed or an integrated system. However, there are some basic criteria to be considered and applied to individual cases.

Purchasing and Licenses

License fees and the subsequent maintenance fees are an important issue in choosing business intelligence software. The BI Survey shows that costs are a common argument against wider deployment.

Systems provided by a single vendor tend to have lower licensing costs. However, they also tend to have more complex licensing schemes, making the true costs harder to analyze. The price advantage of a suite is partly driven by the suite vendors’ common practice of throwing in extras to sweeten the deal, or bundling the product with other offerings such as transactional applications. In most cases, this option is not available to best-of-breed vendors. On the other hand, customers only looking for a subset of all the features a suite offers may find a best-of-breed solution cheaper.

Best-of-breed systems with specific feature sets and more focused product management offer the customer more. However, they tend to be more expensive. A cost justification may be needed to show that the advantages of the system compensate for higher up-front costs and the additional expense of maintaining it. This is often difficult for BI products.

Having fewer products means less effort for the purchasing department. It is easier for the purchasing department to keep track of existing contracts when there are fewer vendors. This argument is probably fairly weak in most cases because the major costs of business intelligence software are not created in the purchase cycle itself. But the purchasing department may have an unduly large influence in the selection process.

Choosing a product to replace several best-of-breed products with a single corporate standard can be problematic. It can be difficult for a company to analyze the requirements of its employees if they are satisfied with the product they are currently using. Business users tend not to have much interest in participating in software projects they do not see as bringing benefits.

Vendor Relationship

Many companies find that they are operating too many different software packages from different vendors and see the need to reduce both the number of products in use and the number of vendors they talk to. There are several advantages in having a small number of vendors.

Suite providers often enjoy a good negotiating position with their established customers. The more different products from the same vendor that are present on a single site, the more tightly the customer is locked into the system. This affects negotiations when the customer needs new modules for an existing system and the customer chooses to replace part or all of the system with a competing tool. However, best-of-breed solutions can sometimes provide an antidote to this problem by allowing the customer to replace a complete system piecemeal.

Large customers are more important to each of the remaining vendors and have an advantage in negotiations as a result. Negotiating with BI vendors on price often leads to significant reductions from the list price, and large customers are usually in a better position to apply pressure to the sales team to cut the price.

Training

In a best-of-breed environment, IT will require additional training to administrate each system. This could also mean supporting separate platforms — operating systems, databases, Web servers and programming languages. The same applies to the end-users, who may find it more convenient to have the same interface for separate functions. This offers another strong argument in favor of an integrated solution.

If there are fewer software packages in use, there is less need for training. This applies to end-users, programmers and administrators. Training costs can be very high, and a lack of proper training can also incur significant costs.

If the same product is reused across the enterprise, existing skills can be redeployed on new projects. This reduces the time required to carry out individual software projects. As The BI Survey 9 clearly shows, short project times lead to better results. Software that is more widespread in the company also reduces dependencies on the skills of individuals. In some cases, departmental business intelligence projects are carried out on a shoestring budget without temporary outside support.

Support

An environment in which multiple vendor offerings are being used side-by-side can be more difficult to maintain if the various vendors do not cooperate or point fingers at each other when problems arise. In practice, however, the potential problems caused by conflicts between vendors have little impact in the BI industry. The BI Survey 9 shows that large vendors with broad suites consistently underperform in the area of support. As a result, the argument that a single product offers better support is not strong.

Shared Data and Metadata

With a truly integrated system developed by one company, it is easier and faster to access shared data. However, it is doubtful that the BI tools themselves are the best place to focus on creating a unified data structure. As a general rule, the best way to provide shared data is at the data warehouse level, not the level of the data mart or reporting system. On the other hand, there are situations in which data warehouses are just too cumbersome to keep up with the needs of a rapidly changing company. In these situations, a semantic layer with direct access to source systems or smaller scale but more flexible departmental solutions might be feasible.

Furthermore, metadata management with an exchange of objects and information between different tools and layers of the architecture can be of benefit to developers and end-users. Since metadata standards are rarely available, or followed by the vendors, it is usually much easier (but not always possible) to exchange metadata between components of one vendor’s suite.

Functionality

The strongest case for the best-of-breed solutions is richer functionality. Some areas of business intelligence are becoming commoditized, which reduces this argument, but it still exists. This is especially true of planning and other content rich CPM applications.

Best-of-breed vendors are more likely to provide timely updates of customer-specific features. Specialized systems may be technologically advanced since it takes longer for companies to re-write a whole suite of applications. But the best Web interfaces come from larger companies, and smaller vendors have difficulty providing highly scalable solutions, either in terms of quantities of data or in terms of the number of users.

It is difficult to find a single vendor that meets all the needs of the enterprise. This reduces the number of choices a company has when it is selecting a strategic partner.

Just because a suite is good in one area, it does not mean it is good in another area. In fact it is quite common for vendors to have functional “sweet spots” with their tools while other BI application areas might be rather cumbersome. Choosing a single vendor to fulfill all company requirements runs the risk of having second rate software in certain areas.

Administration and Maintenance

Suites can reduce the complexity of maintenance, as they usually have centralized administration tools and fewer internal interfaces.

  • IT requirements are reduced when a single vendor is used because there are fewer servers to maintain. Each application has its own specific overhead.
  • Upgrading to new versions of the software is easier if there are fewer products involved. Upgrading is always an expensive and difficult process, and one that can be risky. Many software vendors recommend several upgrades per year. By keeping the number of different products level, companies can reduce the costs incurred by upgrades.
  • Collaboration between separate departments is easier if the same software is being used.
  • If all employees are using the same software, it is easier to define a single point of truth and to avoid multiple definitions and calculations of the key performance indicators.
  • Most vendors who offer wide-ranging products have gotten so big by buying other companies that their software is not very well unified.

However, it is worth noting that all of these arguments apply to situations when more than one application or more than one tool is being used. For smaller scale solutions, best-of-breed products tend to be easier to administer, and in many cases the business users can maintain their own installations. The problems arise when large numbers of such solutions are in use in the same company.

Conclusions

The following table gives an overview of the advantages and disadvantages of each approach.

Best-of-breed

Company standard

Purchasing

More precise feature selection

Low price, but lack of transparency. More leverage for the customer

Training

Tends to be more expensive, and support a smaller community

Knowledge tends to be more widespread, and can be reused in other projects

Administration & maintenance

Simpler, and often more accessible to end-users

Better suited for enterprise-wide applications

Functionality

The best available, but usually missing some application types

Wide-ranging, but often missing key features in particular areas

Figure 1: Comparing standard software with best-of-breed

There is no general recommendation as to whether to use a best-of breed solution. The final decision will depend on the customer’s size, culture and management style. Decentralized organizations or organizations that often change their structures will find it difficult to maintain a single vendor over a longer period. In stable highly centralized organizations, suites are easier to maintain.

The current market situation for larger customers is that many are beginning to adopt an integrated systems strategy and are filling in the gaps with best-of-breed systems. But the systems they are buying are often the cobbled-together suites that were created in the wave of takeovers that peaked around 2007 and has continued to affect the market. In fact, there are very few options on the market today for truly unified suites so, at least to some extent, best-of-breed specialists are to be found in nearly every company.

 

BARC Logo

BIaaS – Business Intelligence as a service

Background information on SaaS

The idea of hosting software off-premises is not new. In fact, software as a service (SaaS) is just a rewording of an older term referring to companies providing the service, application service provider (ASP). ASPs were a very popular investment target during the dotcom era, but not much ever came of the business model. Like so many Internet-based startup ideas in this era, most of these companies disappeared. In fact, the only major exception to this rule was Salesforce.com, which used to refer to itself as an ASP, before the SaaS term became popular. And indeed Salseforce.com is often taken as an example in discussions of the future of Business Intelligence as a service.

But why was the ASP wave a failure and why should SaaS be different? Even the often quoted success of Salesforce.com is far removed from the company’s once loudly trumpeted goal of replacing all the ERP features of SAP and PeopleSoft. Furthermore, do the forces that could make SaaS more successful than ASP also apply to the BI market? After all, Salesforce.com is arguably more valuable to users as a means of automating sales processes than as a means of providing timely analysis for strategic or even tactical decision making. It can be viewed as an ERP company, and the history of ERP companies attempting to conquer the BI market is not particularly encouraging. So even if SaaS spawns more companies like Salesforce.com than ASP did, it does not necessarily mean the SaaS companies will be BI companies.

SaaS is not about the software that is delivered, but how it is delivered. The resources of a cloud environment such as storage, processing, memory, and network bandwidth are generally pooled and delivered in a multi-tenant model. Thus, the discussion must focus on the advantages of the delivery model itself.

One key to understanding SaaS is that there are two different ideas involved that are often mixed freely: the issue of how the software is hosted and how usage is billed. The internet allows companies to deliver software off-site, and this is the essence of cloud computing. Cloud computing environments require broadband networks to provide timely response times especially for users of thin clients. But an SaaS application is provided on demand, meaning that the users only pay for what they use, not a flat fee. This is an additional feature that is not necessarily a given in a cloud environment.

Self Service

Self service is a popular idea in the BI space. In fact one of the key ways of speeding up BI projects and providing more flexibility is removing IT from the loop, thus empowering users to carry out data analysis on their own without being tied to slow bureaucratic IT processes designed to maintain the integrity of enterprise transactional systems.

And there is no question that SaaS provides a kind of flexibility that normal BI tools cannot. In fact one of the key features in cloud computing is that systems can scale up and down seamlessly, and this can be done in the simplest cases without involving complex processes on the customer side. Obviously, more complex processes such as user management are not simplified by changing the platform. This includes role management and security management.

This latter point leads to a key question: does the added flexibility provided by BIaaS actually address the key flexibility issues that BI users face? And the answer is probably no. For it is true that BI users commonly have difficulties with enterprise driven BI initiatives because of the lack of flexibility — but the lack of flexibility relates to data access, data modeling and other site specific issues that BIaaS does not improve at all and in some ways may even make more difficult.

As a result, the issue of BIaaS versus on-site hosting is probably not a key factor in choosing a product. Instead, prospective buyers should weigh issues such as costs, performance and features when choosing a BI tool. A possible exception involves data acquisition for BI tools. It could be useful to combine a BIaaS solution with a data as a service solution. Hosted data management could be interesting, if the data originates from other SaaS solutions.

Project types

CRM, which is often seen as closely related to BI, has seen a good deal of success in the SaaS sector. The most commonly cited example is SalesForce.com, but it has a growing field of competitors. But is the Salesforce model applicable to BI products?

Larger companies have been slow to adopt BIaaS. The exceptions tend to be vendors such as Nielsen and some logistics companies, but these companies supply data as well as a service. One commonly cited reason is the lack of features in BIaaS, but this is probably not an issue with the delivery model itself. As new vendors enter the market, the range of sophisticated feature sets has improved, and BIaaS is is now available for corporate performance management suites and on-demand predictive analytics.

Another major issue is data integration. It is not at all clear how an application that resides in the cloud can efficiently process on-site data. This is particularly problematic for real-time applications, which would need to support a constant stream of data being to be moved off-site and back on. For the time being it is not unreasonable to say that BIaaS is not suitable for real time BI.

There are also very few planning tools that use BIaaS. In fact, dashboards, reporting and simple analysis are by far the most common applications.

BIaaS is often combined with other, usually transactional software as a service solution. The hosted BI solution reports or analysis data that resides in the cloud so neither software nor data is needed to be transferred on premise.

Costs

The main argument offered by vendors in favor of SaaS BI is that it is less expensive than the more typical on-premises software. The upfront capital costs of hardware contribute to these savings. The other major factor is that the company is not required to operate separate servers for the BI application.

However, both of these savings result from the investment required to own and operate the hardware for BI software. Similar savings can also be realized with strictly in-house approaches such as server virtualization. Furthermore, the use of Web technology has already eliminated one of the key costs of BI initiatives, which is the cost of rolling out the front-ends to large numbers of users.

Another key cost advantage that BI SaaS brings is the ability to switch off the service when the licenses are not required. This is a major issue because it is very hard to anticipate BI tool usage in larger organizations, making BI shelfware a widespread problem. But it is also an interesting subject for applications that are only used intermittently, or seasonally. Planning tools are a good example of this kind of software.

Agility

Many, especially larger organizations are burdened by the time and effort needed to keep their software up to date. The installation of updates and upgrades can take several months until they can be used in a productive system. BIaaS solutions often offer frequent updates and new features. They relieve IT organizations from cumbersome test cycles creating additional maintenance work and gives users the opportunity to benefit from new features and functions faster.

Another important issue that sometimes confronts large projects is the sudden or temporary need to significantly increase the data storage for the application. Because cloud solutions used shared users farms with very large capacity, this is much easier to deal with than on dedicated servers.

Security

Security is one of the key reasons why potential customers have shied away from BIaaS. Companies tend to be vigilant about securing their data, and the strategic data often found in BI applications is particularly important. Vendors recognized this challenge and have addressed it with third-party auditing and certification. The two best known certifications are SAS-70 and Systrust. Both certifications provide information to potential customers on how the vendor maintains client data. They also address issues arising from governance, regulatory and compliance (GRC).

At one level, data stored with outside providers may well be even more secure than data stored by companies themselves. This is particularly true of smaller organizations that need to work on a shoestring budget. The large server farms run by cloud operators tend to offer much more security than most companies can.

Nevertheless, there is still of good deal of nervousness in the market about using SaaS to store key data. In particular, companies seem to worry that data stored in the cloud will leak to other users. Perhaps one reason for this is the fact that SaaS providers are perceived as having their origins in the consumer business rather than the secure networking industry. Most potential users may associate BIaaS with products like Facebook, which is not known for its privacy features. Many users have also noticed that Google reads their mail to provide advertising in its email client, which is hardly reassuring. And the recent decision of Amazon — which is certified as a trusted provider — to bow to political pressure and simply cut off Wikileaks.org without notice also sent a shiver through the market. Privacy concerns are further spurred with some legislative demands, such as a law that crucial personal data may not be stored outside the country. This obviously complicates the use of service providers with a general purpose, globally active infrastructure.

Another issue that creates risk is the lack of control over data. This can only be addressed on a vendor-by-vendor basis by careful analysis of the license agreements made in advance, but it undermines the argument that BIaaS companies are more flexible. Exporting data stored off-site may bring unexpected costs. Furthermore, it may be that providers limit the usage of the data they store, and additional licensing for uses not originally planned may be expensive.

BIaaS-specific selection criteria

Companies looking into any BI solution have to analyze a wide variety of issues. These include performance features and usability of the product. For more information on general best practices for purchasing software, see our guide to purchasing BI software.

However, there are certain aspects of SaaS offerings that merit special attention. These include the following:

  • Security: Security is a key consideration because BIaaS involves moving strategic data off-site and storing it at a third party location. Outsourcing IT is nothing new, but in this case the issues of security seem more acute than in most, because the data used in BI applications is specifically modeled to be easy to analyze and reuse.
  • Performance: Performance may well be a major issue when using off-site tools. In particular, this method of product delivery introduces additional performance risks into this sensitive area, and the risks may be more difficult for the customer to control that if the problems were on-site. On the other hand a good service level agreement – if the provider agrees to one – might hand the task over to the service provider to tune the systems according to performance demands.
  • Training: There may be a lack of resources for a BIaaS provider in the customers locale. Part of the price advantage of BIaaS is that the expense of creating a geographically widespread organization is reduced. However, this also means that customers have less access to experienced consultants.
  • Service levels: One of the key long-term costs of BI software. Since BIaaS may not have an organization in your country, this may be a problem.
  • Customization: BIaaS products are often distributed as finished applications with little flexibility. Creating an on-line development system with the power of locally hosted applications represents a significant technical challenge.
BARC Logo

Business Intelligence and Web Front ends

Introduction

The complexity of Web technology has increased greatly since the first simple browsers became popular in the nineties. The original idea was very simple: Web servers provided access to simple formatted texts which could contain hyperlinks to other texts, creating a worldwide web of interlinked documents.

Since then the technology has flourished in many ways, but it is possible to cut through the almost impenetrable thicket of technologies by classifying the developments. Once you do this, things look much simpler.

  • The original hypertext markup language HTML has gotten much more complex, allowing much better formatting and interaction. Most importantly, HTML pages can embed various non-text objects, a capability that was originally intended for displaying images.
  • Originally, HTML files were stored on the Web server and served up on request. This has changed a great deal, and now most Web pages are provided by content management systems which may serve static HTML files or assemble the HTML files based on the requests that come from the browser using data held in various databases. Thus the business intelligence software that creates Web content often comes with a simple BI portal.
  • The HTML pages now embed scripts that run in the browser and allow the documents to be dynamic – that is, to modify themselves on-the-fly while the user interacts with them. The language of these scripts is JavaScript, which was originally invented by Netscape. Netscape created a lot of confusion when it chose this name, because JavaScript has nothing whatsoever to do with Sun’s Java language, except for some superficial similarities in the syntax.
  • The objects embedded in the HTML pages have gotten more and more complex. The original images have now been replaced by various interactive objects. The Java programming language actually embeds a complete virtual machine – a sort of toy computer – into the document. This approach has been copied by several vendors, including Microsoft with its .NET technology, which is nearly a clone of Java, and increasingly by Adobe’s Flash technology, which was originally intended for cute little scripted animations but now has many more capabilities.

The main point of Web technology, from the point of view of BI deployments, is to reduce the effort involved in rolling out an installation of the software to every user. Web front ends also make it possible to use the software outside the company in the extranet. Installation is usually automated in larger organizations, but even then additional applications cost the IT department additional expense and risk, and these costs are often passed on to the other parts of the organization. So called “zero footprint” solutions do not require any installation at all, reducing the total cost of ownership of the tool, assuming all other factors are equal.

That said, there are other costs in addition to the rollout costs. For example, zero footprint deployments require more processing to be done on servers that would not otherwise be required, and more data to be sent across the network than would be the case with a good client/server application, which has local caching, generates charts locally, a local calculation engine and so on. This increases hardware and network costs, while reducing the quality of the user experience.

But on balance, Web-based tools are most popular in enterprise environments, where IT issues dominate the discussion. Note that this discussion of deployment costs contains an implicit criticism of the personal computer with desktop software installations as an enterprise tool, which is why the Web is widely seen as an alternative or even a threat to Windows. By the same token IT departments which are skeptical of Windows technologies tend to strongly favor Web-based applications.

At the same time, IT departments tend to view programmed components embedded in Web pages with a good deal of suspicion because of the potential security issues involved. Every time users load a Web page, they are loading code from a server (potentially one outside the company) into the company’s network – a nightmare for busy administrators. Microsoft’s ActiveX in particular is a target of IT ire. The problem is not that there is anything inherently wrong with ActiveX – in fact it is heavily used in Windows. Furthermore, in an intranet that is the typical environment for business intelligence solutions, all the code being loaded is already on the company’s network. The only difference in an intranet environment is that some of this software runs in the browser, and some on the Web or application server. The problem is that if you allow one ActiveX object to run on a browser there is no satisfactory way to prevent other potentially malign ActiveX objects from entering from outside as well.

For the end user Web-based BI front ends offer little in the way of advantages over Windows. Because Web applications tend to be less interactive than Windows clients, and tend to have restricted access to local data sources for security reasons, there is a strong tendency for business users to prefer Windows client tools. Here’s a brief list of the main issues to be kept in mind when selecting a Web-based business intelligence tool:

  • Web applications allow the user to access the application from any networked PC, not just his own PC which happens to have the software installed.
  • BI applications running in browsers lose usable screen space to the browser, which displays tools bars, tabs, and an additional menu bar which the application does not use. This loss of screen space at the top of the screen is exacerbated by modern wide-screen laptops which have shallow screens that were originally designed for the larger television market but have spilled over into the computer business because they are already mass-produced.
  • Unlike Windows applications, Web applications often fail to take advantage of wide or tall screens, so they end up wasting even more screen space. All this forces users to scroll sideways or up and down much more than in a decent Windows application, thus reducing the usability.
  • The menu of the browser creates more problems than just taking up valuable space on the screen. Users often confuse the browser’s back button with the application’s undo button. If they accidentally hit the back button, they may be taken right out of the BI application, instead of just undoing their most recent action.
  • The same confusion between browser functionality and application functionality makes it difficult for users to control font sizes and other aspects of the user interface.
  • Printing may be an issue and involve server-based PDF generation. In fact, creating a PDF file on the server and presenting it in a new window for the end-user is about the only really viable way to print a document presented in a Web interface. A few applications attempt to print in the browser but this is unlikely to work well since it cannot take advantage of locally-installed fonts, or wide page capabilities. Again, all this can be confusing to end users to whom we will seem logical to use the print function built into the browser.
  • Performance will usually be much worse than with a good client/server or local Windows application. This is a serious issue for BI deployments, as The BI Survey regularly finds, and performance issues are a major cause of user dissatisfaction.
  • Web interfaces often lack strongly interactive GUI features, like double click or providing context menus when the user right clicks an object. Drag and drop is another form of interactivity that is often missing or weakly implemented in web interfaces. The lack of context menus is most noticeable in supposedly highly interactive Flash interfaces. It is only one example of a common problem that is often unclear whether user actions invoke browser functions or application functions.
  • It is hard to drag content from the Web applications to local PC applications, such as Excel. This may require special features in the application to support such links, which would be natively available in Windows applications.

As Web usability continues to improve these GUI arguments are getting weaker, but they still apply to the self-service BI market. For security reasons, Web-based tools usually do not have access to local files, and in self-service BI scenarios, end users often provide their own data management and prefer to remain independent of IT. Furthermore, as convenient as Web user interfaces are to IT departments, they are more complicated to administer in a departmental environment, because they require a Web server.

Common Web technologies

Business users facing a decision on which software to choose are often confronted with a slew of formats to choose from, and little guidance on their differences. The following provides information on the most common Web formats. Note how Microsoft has brought out its own version of each technological idea. This phenomenon is related to the perceived threat of the Web to Windows, mentioned above.

ActiveX

ActiveX elements are reusable native Windows software components based on Microsoft’s (pre .NET) COM technology. Internet Explorer allows them to be embedded in HTML pages.

AJAX

AJAX is a special type of advanced DHTML that allows Web pages to get data from the server incrementally – without refreshing the entire page. AJAX is a major improvement in interactive Web pages, because refreshing the entire page is slow and causes the browser’s screen to go briefly blank, interrupting the user’s interactive experience.

DHTML

DHTML is short for Dynamic HTML, which is HTML enhanced with JavaScript to make it interactive. The scripts actually modify the content of the document while it is being displayed and can react to user actions to provide interactivity. DHTML is the format most commonly referred to as “zero footprint”. The scripts that make these interfaces dynamic and interactive are actually executed in the browser, not in any add-on object. This is where the term zero footprint comes from — as long as the user has a browser that complies to Web standards they won’t need any additional software to make the application work. DHTML interfaces are somewhat clumsy to program however, especially because the JavaScript language was never really intended for creating large applications. They also tend to be plagued by minor bugs caused by slightly incompatible browsers. However, in recent years this technology has gotten surprisingly good. The most spectacular implementations include a small amount of data stored in the HTML file itself, which allows this technology to be used to provide off-line capabilities.

Flash

Originally a format for vector-based animations, Adobe Flash is now a general platform for creating small, highly interactive Web applications and games. It requires an add-in for the browser to execute, but most IT departments do not object to the Flash add-in the way they would object to other add-ins, especially ActiveX add-ins.

Some of the most spectacular new BI interfaces are created using this technology. One feature that is typical of these interfaces is the glazed jellybean look that the objects sometimes have. Flash interfaces also tend to be highly interactive, allowing drag-and-drop and lassoes for grabbing multiple objects. But they do not have the Windows standard context menus available by right clicking the mouse. They also feature animations so changing the selections results in the charts smoothly morphing from one state to the next instead of just switching from view to view.

These objects can also store quite a bit of data. Several BI companies have made use of this ability to create a limited off-line functionality based on Flash.

Flex

Flex is a software development kit for Flash. The term is often used interchangeably with Flash.

HTML

A simple syntax for describing page layouts featuring links to other documents. Browsers convert HTML into formatted text.

Java

Java is Sun Microsystems’ programming language based on a virtual machine. The virtual machine has its own internal language which mimics the machine language of a real computer and provides most of the services a computer with an operating system would provide – it is a closed world. The Java virtual machine supports any programming language that compiles to its internal syntax , but the Java language is almost always used. Sun has now been bought by Oracle, so this key technology now belongs to them. Because the Java program is completely executed inside its virtual machine, Java programs run the same way on any computer that has Java virtual machine software installed. Java applets are complete virtual machines embedded in Web pages, but Java programs also exist as stand-alone tools outside the browser. Java programs also run on mobile devices.

JavaScript

JavaScript is a scripting language that runs in the browser itself. JavaScript is very different from Java although it does have a similar syntax. The name was a marketing ploy by Netscape back in the days when Netscape was battling Microsoft in the browser wars. Microsoft also created a version of JavaScript. The Microsoft version is called Jscript.

.NET

.NET is Microsoft’s Windows-only near clone of the Java virtual machine. .NET comes with compilers for multiple languages including C++, Basic.NET and the Java-like C#, but all these compile to the same internal format. Microsoft created .NET after losing a lawsuit on the future of Java to Sun. .NET built on Microsoft’s excellent development tools and has been quite successful. It is now used for building many Windows applications, but not for building Office or Windows itself. It has not caught on as Web front end.

Silverlight

Silverlight is a Microsoft platform based on .NET that closely imitates Adobe Flash. Silverlight uses Microsoft’s newest .NET visualization, and can also be viewed as a new viable Web front end for .NET.

XHMTL

XHTML is a version of HTML that is cleaned up a bit to conform to the more general XML format. XHTML is not really of any interest from the application user’s point of view.

XML

XML is a markup language for describing objects. Even in the hype-rich Web environment the amount of hype surrounding XML is remarkable. XML is not really a standard, as is often claimed, but simply a way to make non-standard content a little easier to deal with.

BARC Logo

Relational Databases

Relational databases have been used by businesses for decades and they have steadily developed the features required for large-scale implementation, including scalability (in terms of user count and database size). The idea of relational databases was first suggested by Edgar Codd in 1969, and by the nineties, they had largely replaced hierarchical and network databases for business purposes. But the original definition is excruciatingly abstract, and no Codd-standard “true” relational database has ever been commercially released, although there are academic implementations. Textbook discussions of relational databases descend into pure math almost immediately. For practical purposes, relational databases are databases that can be queried using SQL, but purists point out that the current ISO standard definition of SQL does not mention the relational model and ISO SQL deviates from the relational model in several ways.

Roughly speaking, a relational database is a set of data connected by relations. To take a simple example, an invoicing system could have one table with a list of invoices and another with a list of customers. Each table has one record (referred to as a tuple) per line, and defines a relation – for example, the invoice number is related to the date the invoice was issued. In fact, relational tables are often referred to as “base relations”.

But there are other relations in the database. The data in the two tables mentioned above are also related to each other by customer numbers, which appear on each line of both tables. To see how much a given customer bought in sum, filter the invoice table by the customer number and sum the total column of the result. To get the address of the customer for a given invoice, filter the customer table by the customer number in the invoice and look at the address column. These are derived relations, more commonly known as views or queries.

Relational databases are now so widespread that it is hard to imagine a modern company without one. But it is not just the details of the relational model which give them such a firm position. The ubiquity of technical skills required to administer relational databases and a huge number of applications that use them guarantees that relational databases will be used for many years in the future. Another strength of relational databases is the widespread use of SQL, which makes it easy for an application to work with multiple relational databases. This was not true of hierarchical databases. Establishing SQL as a standard probably led to the development of the whole ERP industry and the rest of the OLTP applications market.

One of the great advantages of relational databases is their ability to support transactions. The physical storage of relational data on the disk is usually by row. Row-based databases are better for short rows and situations where all columns are needed in most queries. This applies to transactions, which often write all the columns of a new row at the same time. However, it is worth noting that key parts of transactions are actually add-ons to the basic relational model and are achieved in the following steps:

  • Lock affected areas of the database until the transaction is complete
  • Log all changes to the database in a separate log file
  • Write back the result immediately
  • Roll back if the transaction fails by undoing the changes recorded in the log.

Relational databases used for large scale business environments have highly refined versions of these extra features.

A variety of other features have clustered round the relation concept and allowed the databases to be used in many different contexts and to scale almost unimaginable heights. Strictly speaking, these additions have nothing to do with the relational model and, indeed, non-relational databases also have similar capabilities. So the relational model owes much of its success to the development of external concepts.

  • Highly developed indexing methods: Indexing sounds boring, but it has been a rich field for technical innovation over the past few decades. Most modern relational databases use multiple indexes simultaneously and indices often consume more storage space than the raw data in the database. Part of the database’s internal query optimization is an automatic on-the-fly selection of which index to use. Individual index types tend to work best in specific use cases. For example, bitmap indices are mostly commonly used for analysis on tables with many columns of yes/no data. They are not much use in other applications.
  • Redundancy: Another feature of modern relational databases is the ability to spread data storage across multiple servers, and provide redundancy for safety.
  • Multiple interfaces: Relational databases come with their own proprietary interfaces, but there are also standard interfaces to allow third-party products to access them generically. Providing multiple technical interfaces, all of which support SQL, is another way of opening up the database to as many third-party products as possible. The most common interfaces are the Windows-based ODBC and the Java-based JDBC. As a rule, the native interfaces provide the best performance.
  • Stored procedures: Stored procedures were originally introduced by individual vendors and are now part of the SQL standard as optional features. They are stored in the database and are fragments of procedural code that can be called from the client and executed on the server. They allow highly complex applications to run on the database server itself. Unlike SQL itself, which is reasonably well standardized, the syntax for procedures to manipulate data is completely different in every database.
  • Large-scale databases also take good advantage of multi-threaded servers, so processing loads can be spread across multiple CPUs.

Other types of database also have these features, and relational databases are not the only type around. IBM IMS, a hierarchical database, is probably still the leading OLTP database, and is reputed to be IBM’s biggest billing software product. IDMS is also still around, as well as Adabas, Model 204, and other non-relational databases. Of course, some of these now support SQL queries, even though the underlying storage model is not relational.

One of the weaknesses of relational databases is the way that related data tends to be spread around the database. Even the simplest invoicing system would require four tables to be useful: a customer table, an invoice header table, a product table and an invoice line table. The latter is needed because each invoice can have multiple lines. Printing the invoice means filtering all four tables and combining the results into a single report. And real-world applications are much more complicated than that.

Normalization, which results in splitting the data up into small tables, is the usual mode for storing relational data. Normalized databases are optimized to avoid inconsistencies in transactional systems. But data that is spread around this way is not at all easy to analyze by hand. Most applications based on relational databases provide the user with highly abstract views of the data, which make sense from a business perspective but are very difficult to see in the underlying data. The queries needed to analyze transactional data tend to be extremely complicated because related data is distributed around the database as a result of the normalization. The queries are typically difficult to define, requiring multiple joins and a complete understanding of the complexity of the schema. They are also quite slow, because the database has to join the data from the various tables back together to answer the query.

Another problem with analyzing transactional data directly is that the data model is not designed to return aggregated views such as the sum of all the transactions in a given time period. But this kind of aggregation is typical for analysis. Attempting to carry out analysis directly on data in this format results in poor performance. It should never be attempted on anything but the smallest transactional systems.

The data structures in applications built on relational databases are sometimes very complex. Large ERP systems can contain tens of thousands of tables. Vendors of business intelligence software commonly say they support various relational databases, but they rarely mention the fact that without semantic support the ability to connect to a complex relational database is almost useless. In other words, if the software that is extracting data does not know where to find the data in the right form, then the user will have to know where the data is, and build queries by hand. This issue can lead to significant cost increases in business intelligence projects.

To make analysis of transactional data feasible, many companies create data warehouses. This involves extracting data from its original source, transforming and cleansing it and finally loading it into a new database in a form that is suitable for analysis. Data warehouses are usually stored in relational databases and are often the first place in the enterprise where basic analytic concepts such as a time axis appear. The data management processes involved can be quite complex and often involve exporting the data to large text files and re-importing it into another database. The issue of data quality is particularly vexing. In many cases the data in the warehouse comes from multiple, otherwise unconnected, sources.

However, data warehouses can be very large and unwieldy. To make analysis practical, small subsets of the data are often extracted into so-called data marts for analysis purposes. Data marts are sometimes stored in relational databases, but in many cases other forms of database are used, particularly multidimensional databases. The idea here is to trade off details and complexity for speed and simplicity. Attempting to retain the complexity of the underlying ERP system in the data mart is trying to have your cake and eat it too, and is probably doomed from the start. Well-designed data marts always lack most of the complexity of the transactional world they are based on. They are optimized for accessibility, and usually hold aggregated data only. Because they feature reduced size and complexity, data marts can often deliver query results very quickly, making them popular with business users.

General-purpose relational databases designed for enterprise transactional systems are not always the best choice for storing data warehouses. The major vendors of relational databases have tried to optimize their products for this market, but face stiff competition from a relatively new technology in this area – data warehouse appliances, which offer a combination of hardware and software for high speed performance on very large quantities of data, sometimes even in the petabyte range. Data warehouse appliances use parallel architecture with optimized custom hardware and operating systems to deliver performance. Some of the systems also come with chips built into the disk controller that carry out simple filters on the data as it comes from the disk. In these systems only a fraction of the data needs to be loaded and processed by the server itself, and appliances typically make much less use of complex indices, which means they also store data much more compactly. These innovations allow appliances to process huge quantities of data on relatively cheap custom hardware.

The physical data storage model also varies from database to database. Again, this is an implementation detail and, strictly speaking, it has nothing to do with the relational model itself. The standard way to store data in transactional databases is to store by row – all the data in the first row of the table, followed by all the data in the second row, and so on.

In columnar databases, the data is stored by column, not by row, and this has an effect on how the database is used. Storing by column makes typical transactional operations such as writing or reading every column of a single row from a large database slow, so columnar databases are not well adapted to typical transactional systems. To see why, consider a command to add a new invoice to a system. It adds a new row to the invoice table, so it would require accessing and changing all the columns of the entire table, because the table is stored by column. So typical transactional operations are fairly clumsy on columnar databases. But reading one or two columns from every row in a table is fast using a columnar database. Reading a few columns and ignoring the rest is typical for OLAP operations. Because columnar databases allow you to ignore columns they are better for databases with large numbers of columns and allow separate indexes per column for optimal access speeds. They are more efficient for adding new values of a column for all rows because only one column is touched.

Most columnar databases support SQL. The distinction between row-based and columnar databases is in the storage engine. The queries may use SQL and optimize to the storage engine, for executing the actual write and read actions. In fact one columnar database, InfoBright, is actually delivered as a database engine for MySQL and accessed using SQL queries in the MySQL dialect.