The Role of the Application Managers and Developers

In Part 1 of this series I addressed the role of the Director of Data Management and/or the Director of BI/Data Warehousing and the key information needed to achieve their objects and goals in managing the IT team responsible for the integrated delivery of the data warehouse.

This installment I will address role of the Application Managers and Developers and what is needed by them to efficiently manage the BI users and applications deployed.

The BI Application managers and developers are responsible for the application infrastructure and the management of the business users. The key functions include:

• Responds to the business user requirements and develops applications (reports and analysis) to meet business needs.

• Develop and manage the complete lifecycle of the BI Applications

• Collaborate with the data architects, data warehouse developers and database administrators to Ensure the performance of the applications and effectively address end-user complaints

The user activity in a BI and data warehouse environment is characterized by unpredictability and constant change. As deployments get larger, the user’s typically are allowed to develop and run reports in an ad-hoc manner (as opposed to pre-tested and canned applications).

The key goals for the Application managers and developers in managing the BI and data warehouse deployments typically are:

• Respond faster to end-user complaints

• Improve BI application performance

• Ensure that the applications and access provided meet audit requirements for security or regulatory requirements

To effectively meet these goals, the BI Application Managers need the ability to clearly understand how their end-user activity is impacting the applications that are deployed. Rather than chasing one or two problematic SQL queries – the application managers would be far better served at first analyzing the most frequent user activity that impact performance and then isolate the sources of issues. In addition, Application managers need end-to-end view that correlates the users and application activity with data usage and database query performance metrics in order to diagnose problems and respond faster.

Examples of information needed by the Application managers and developers are:

• Who are the most frequent users running problematic queries - i.e. User = Application User_ID (not generic database ID)

• Who are the Users running Ad-hoc reports – and creating very expensive query activity or poorly written queries? E.g. Unconstrained queries that return 1000’s of rows unnecessarily

• What are the most frequent or repetitive queries that are long running? Who are the Application users? What data is involved? What other operations are occurring on the data that contributes to issues?

• Who is using sensitive data, when and how? E.g. Which user queried the table “Credit_card ID” and returned more than 100,000 rows?

It is clear from the above examples that to manage BI users and applications requires more than just the application server level metrics. Instead it requires visibility to applications’ user activity that is correlated with data usage and database performance metrics.

In my next blog I will discuss the role of the Application Database Administrators and Data Warehouse Architects.

The Role of the Director of Data Management or Director of BI and Data Warehousing.

Any BI and Data Warehouse design and implementation requires the collaboration of several functional groups within IT, after spending several months documenting the requirements of business users. However the real challenges usually begin to occur after the implementation when you begin to deploy in production and start supporting a growing number of users. It is a fallacy and in the long run disastrous to consider that the Warehouse will remain the same once it is deployed. In reality, the way the Business users use the system will continually change and requires every one of the functional groups to be in lock step with the evolution of the BI and Data Warehouse deployment to ensure optimal performance and control support costs.

In this multi-part series I am going to address the role of each functional group and what information would each need to ensure that a BI and Data Warehouse deployment is managed efficiently and cost effectively.

So lets take a look at a typical IT team that is responsible for the integrated delivery of BI applications and Data Warehouse for a department within an enterprise or across the enterprise (as a shared environment).

A business unit or departmental team is led by a Director of Data Management or
Director of BI and/or Data Warehousing who has oversight over the IT team responsible for the integrated delivery the Data Warehouse – including ETL, BI Applications and the Data Warehouse databases.

The Director of Data Management or Director of BI and/or Data Warehousing manages a team responsible for the design, implementation and support of the integrated system. This team typically includes:

• Application Managers/Developers

• Data warehouse Architects/Developers

• Application DBAs (not to be confused with System DBAs)

• ETL Developers and Data Architects

• System DBAs (most likely a shared resource from the database infrastructure IT team)

I will discuss each role in detail in this multi-part series.

Lets start with the Director of Data Management (or BI and Data Warehousing). What are their key objectives and need in managing an enterprise BI and DW deployment? And what kind of information is needed to help them be successful?

Since this individual has oversight over the IT team responsible for the integrated delivery the Data Warehouse, including ETL, BI Applications and the Data Warehouse databases, the major goals for a Director typically are:

• Control operational and support costs while meeting the growing needs of the business

• Communicate effectively with his/her managers to justify current investments and request funding and plan for future growth (people and capacity)

• Ensure that any corporate security and regulatory compliance requirements are met (if required)

More often than not there is not enough information available to assist in meeting these goals effectively. The kind of information that Director level managers need to ensure they have readily available are:

• What are the weekly and monthly trends on user activity and utilization?

• How are we doing with meeting our internal SLAs? What is the trend in activity that is not meeting our internal goals of performance?

• Do we have information to meet our Audit requirements (who is using what data and when)?

• What resources should be plan for based on trend in activity – add capacity, more people etc.?

This type of information is not any different to what business users regularly leverage their BI applications to make informed decisions.

Except that the data is related to IT performance metrics.

In my next blog I will discuss the role of the Application Managers/Developers in managing a BI and Data Warehouse deployment.

It’s actually a simple concept. If you don’t know what is happening in a process you can’t possibly manage that process. A natural corollary is that the better you measure a process the better you can manage it.

Most Data Warehouses have some level of measurements in place. However most measurements are single purpose metrics and therefore are of limited or single value. The System Administrator can measure CPU usage; the DBA can measure tablespace usage; the Network administrator can track dropped packets and finally the BI manager can see which reports are being run.

This is analogous to a sale and marketing operation without an integrated data warehouse (did you ever wonder why everyone works so hard to integrate multiple data sources in a data warehouse, is it possible they know something special?). Consider trying to manage a sales promotion where the sales manager only knows about her sales forces call records; the marketing manager only knows about product movement through the stores; and the warehouse manager only knows his inventory level. So what happens? The marketing manager creates a coupon drop in specified markets; however she has no idea if there is sufficient inventory in the warehouse to fulfill the customer demand. The sales manager has no idea if his sales forces effort as had any affect on the promotion and the warehouse manager has no idea that he is going to need more product. Each individual is doing their best to manage their own area of responsibility but without a view to the other areas the entire marketing campaign is not managed.

Managing a BI infrastructure is very similar. You need to collect measurements of all levels of the BI stack to be able to manage the stack. To simply measure at the DBA level or only at the application server limits how well you can manage the environment. To truly manage the stack you need to apply many of the same disciplines common in building a robust data warehouse. Multiple, integrated data sources matched with an analytic application.

The day was March 1, 2007.  The event - Oracle’s purchase of Hyperion.  For years Gartner foretold the day that BI would be integrated into the business process - it became a self fulfilling prophesy.  It was then that I realized that BI as we knew it would die.  For many years, the rumors of consolidation were rampant, but the innovation and open competition continued.  End-users benefited from the consolidation of functionality - query and reporting, multi-dimensional analysis, analytics converging onto single unified platforms from Business Objects, Cognos and Hyperion.  This innovation was good for enterprises since it led to simplified architectures and infrastructures and lower costs.  BI was out in the open with vendors touting their latest and greatest capabilities and analysts tracking market share, relative positioning etc.  Since that infamous day, Business Objects has been absorbed by SAP and Cognos by IBM.  The lights are now out.  BI has been snuffed out, buried in someones stack.  Is this progress?  Enterprise IT - you wanted fewer vendors to deal with, you’re getting it.  Fewer vendors charging higher prices!