‘WHAT MAY HAVE CHANGED?’ ANALYSIS FOR PREVIOUSLY STABLE BPC-MS SOLUTIONS
Should a BPC-MS solution behave unexpectedly, there are a number of points one can consider, in order to narrow down the possible root causes for the new behavior observed. Proper consideration of these can save time in identifying a root cause.
The primary purpose of this document is to serve as a checklist/guide only. The scenarios referred to in each item are only for examples purposes. Not all items listed apply to any or all situations.
What may have changed in BPC
Often issues relate to a table/cube/dimension/member/file/registry key associated with a particular component is corrupted, inaccessible otherwise improperly configured. It is important to learn about the component-specific aspects of the issues early-on in order to start a structured troubleshooting effort. Knowing these help carry out valuable comparisons between similar components within the same environment as well as between different environments.
Report-specific: Identify if the issue is reproducible with only a certain report or set of reports
Application (=’Model’ in BPC10) Specific: Only reproducible when a particular Application is selected.
Dimension Specific: Only reproducible when a particular Dimension is selected.
Member Specific: Only reproducible when a particular Member is selected.
Hierarchy specific: Only reproducible when a particular Hierarchy level is selected
Appset (=’Environment’ in BPC10) Specific: Issue only reproducible when a particular Appset is used.
Appserver Specific (Load balanced scenario): Where an environment has a multiple Application Servers used with a load balancer and when the load balancer is by-passed, the issue is either no longer reproducible or only reproducible when connecting via a particular (or a particular group of) Appserver.
SQL Server Specific (Cluster Scenario): Where a cluster exists, the issue is only reproducible when connecting via a particular database server.
User-Account or User-Group specific: The issue is only reproducible when a particular user account (or a particular group of accounts) is used. User rights comparison is a major starting point in troubleshooting of this scenario.
*Client machine specific: *The issue is only reproducible with use of certain client machines (regardless of the user account being used to logon). In this scenario, the issue most likely related to BPC client tools installed on the machine, the O/S or other 3rd party software installed on the machine, or the machine’s network connectivity and or the security policy within the network.
Virtualization-Specific (Citrix, VMWare etc.): The issue is only reproducible when a particular virtualized system(s) is being used as a part of the workflow.
Data-package Specific: The issue relates to use of a particular data package (watch for custom logics)
Load-Specific : The issue is only reproducible when a certain amount of data or data from a specific data region is selected.
Time-Specific : The issue is only occurring during certain time intervals. (especially for performance troubleshooting, it is useful to investigate, what else happens in the environment during that period). How frequently the issue occurs may be relevant information.
What may have changed in an Environment
If the new, undesirable behavior in environment is not due to corruption or malfunction of a previously stable software or hardware component, it is usually due to a manual or automatic change. For this reason it is a good troubleshooting step to investigate and identify what, if any, manual and automatic changes occurred shortly before the new behavior observed. Below are some examples of possible changes in a BPC-MS environment.
Changes in Network
Infrastructure/domain re-segmenting networks (Switch-Bridge configs)
Server names
IP addresses
Network cards and their drivers
Name resolution configuration (DNS servers)
Changes in firewall Rules
Changes in client machine
O/S patches updates
WIFI and Wired connections
BPC client version
3rd party apps.
Excel version
Changes in Servers
RAM/AVAILABLE MEMORY
Hard disk capacity/format
Network Card Driver settings
Network bindings
Recent O/S patches (ie Security updates)
3rd party apps.
Changes in Storage device
Configuration (read/write speeds)
Network Settings
Partitioning settings
Compression and indexing settings.
Security Settings
Fault tolerance configuration
Changes in User security (Domain and local users)
Rights to files/folders/registry
Looking For Differences in the logs
While investigating the root cause of a new behavior, comparison of logs for the same workflow (different users, different systems etc) can be helpful, here are few things to keep in mind when working with logs.
Check Timezones : Prior to enabling logs, check the Regional settings of the client/Servers involved and also ensure the time zone differences between the timestamps are put into account.
Limit capture : If possible, enable logs only step-before the issue is reproducible, and disable as soon as the issue is reproduced.
Filter : When examining very long logs, look for keywords ‘fail’, ‘error’, ‘ not ‘ , ‘cannot’, ‘unable’,’ too ‘ for a quick scan of errors and once a relevant error is found, work backwards to find out any logged failures immediately before that occurrence.
Patterns : Identify the long chunks of errors that may be repeating in a long log file.
Associate : Look at all logs captured (for example IIS, DB logs, SQL profiler, client and Server event logs, and any 3rd party monitoring tools like procmon, procexplorer, fiddler,wireshark etc) for the same timestamp.
Consider : If enabling auditing can capture useful information for the given situation. Consider if BPC Management Console can provide helpful information for the given situation.