CRSIC Domain #3 Risk Monitoring

Risk MonitoringWeek #4 – CRISC Exam Prep ~ Domain #3 : Risk MonitoringWeek #4 – CRISC Exam Prep ~ Domain #3 Bill PankeyTunitas Group

Job Practice : Job Practice Collect and validate data that measure key risk indicators (KRIs) to monitor and communicate their status to relevant stakeholders. Monitor and communicate key risk indicators (KRIs) and management activities to assist relevant stakeholders in their decision-making process. Facilitate independent risk assessments and risk management process reviews to ensure they are performed efficiently and effectively. Identify and report on risk, including compliance, to initiate corrective action and meet business and regulatory requirements. CRISCRisk Monitoring Domain

Agenda : Agenda Key Risk Indicators What are they How to construct How are they used How are they improved How are they reported Data Aggregation Benchmarking No distinct RiskIT ‘monitoring’ process

Expansive View of “Risk Monitoring” : Expansive View of “Risk Monitoring” Risk Governance Objective  Monitor the overall performance / effectiveness of the risk management program and recommend improvement (COSO) Risk Management Objective  Ensure the current and emerging levels of risk are within tolerance levels.

2nd Order Uncertainty : 2nd Order Uncertainty Risk is a statement about the potential for [future] loss events Manage by control; avoidance; sharing … Risk monitoring identifies and evaluates changes in risk Whether potential for loss is increasing / decreasing Added opportunity to manage risk Surprisingly, statements about changes is risk are less ambiguous, more objective than the statement of risk

Key Risk Indicator (KRI) : Key Risk Indicator (KRI) Metric / observation used to track risk level at specific time point where likely unacceptable loss or ‘trouble ahead’ Indicator becomes ‘key’ when Tracks an important risk Is reliable, cost effective, … # of unpatched systems is a risk indicator, but may not be ‘key’ What is the risk that is being tracked? How important is that risk?

KRI are not KPI …KPI could be KRI : KRI are not KPI …KPI could be KRI KRI are leading indicators Intended to be predictive of future loss / outcome Key Performance Indicators are lagging indicators Report on accomplishment of activity / process A given KPI could be used as a KRI or component of KRI % of [expected] function points delivered on time Measure of project efficiency / could be used as indicator of project delivery risk

KRI proxies Risk Measures : KRI proxies Risk Measures Source: Risk Management Association 2005 survey of Financial institutions Operational risk management Strategy Normalization Risk Communication Compliance ??? KRI Uses

Why KRI are Important : Why KRI are Important Risk factor ~ condition influencing frequency, magnitude business impact of the loss event / scenario Change in risk factor  [some] risk indicator Logically and [typically] temporally prior to risk event

KRI Selection : KRI Selection Unlimited # of risk indicators in logs, alarms, reports … What to select for regular monitoring as KPI Reflects management priorities Stakeholder concern Strategic and / or operational business impact Management utility / basis of management report Basis for risk communication “What gets measured, gets done”, Drucker, The Practice of Management

KPI Goodness Criteria : KPI Goodness Criteria Associated with one or more specific risks Measureable at specific points in time Objective finding rather than a subjective assessment Track at least one risk factor Actionable Effectiveness

KPI Goodness Criteria : KPI Goodness Criteria Quantified (#, %, ratio, rate) Well defined / reproducible Time independence [Business] Process independence Auditable Comparable across organizations (?) Comparability

KPI Goodness Criteria : KPI Goodness Criteria Timely, readily available in reasonable time frame Cost effective to collect as a production of automated system, by-product of process or service Obvious … easily understood and communicated Efficiency

KPI Process Steps : KPI Process Steps Data access Ensure timely, reliable data delivery Data validation Match definition; complete; within range; missing data(?); duplicates (?); reliability of derived values; ‘referential’ integrity reasonableness checks! Data analysis Statistical computations Conclusions / inference Reporting Right people, right format ISACA Best Practice

KPI Optimization : KPI Optimization Optimization criteria Sensitivity Appropriate level of alert; # of red flags; Critical conditions, etc Timing How much lead time Frequency How rapidly may new risk condition develop; express themselves Corrective action Utility for tracking remediation effort; assigning priority (MBO) KPI autocorrelations, correlations with each other, loss and performance data provide an empirical basis for optimization Regression analysis ISACA Best Practice

KPI Validity: Causal Factors : KPI Validity: Causal Factors Utilize expertise of subject matter experts, process and service owners Identify the risk scenarios of greatest concern Decompose scenario into leading risk factors Identify indicators for those factors Observable measures; critical thresholds or change Develop KPI reporting scheme

KPI Timing : KPI Timing Clinger-Cohen Act of 1996 – new demand for certified IT professional KRI: Industry Salary Index (annual) Loss of key IT personnel KRI: Mid-level Staff retention rate Failed IT project Early indication  greater opportunity for correction

Pro-Active Risk Management : Pro-Active Risk Management

Aggregation : Aggregation Address management concern regarding ‘overall’ risk : Specific business objectives Strategic | Operational Customer Product Regulation Business Unit … arbitrary level of specificity Management interest is greatest when ‘all variety of risk’ is reported

Aggregation : Aggregation Diversity Problem: For different risk domains (say IT, Legal, Finance ..) KRI are specialized KRI have different time periods KRI have different granularity KRI have varying sensitivity KRI have varying relevance, reliability and validity This is a problem that is not solved so much as overcome

Aggregation Heuristic : Aggregation Heuristic Report risk as dimensionless quantity (kounts, %) Report risk as % red, yellow, green Code each KPI relative to threshold (red or green) For each risk, count the number of associated KPI above of below threshold Report risk as increasing of decreasing # KPI indicating lesser risk / # KPI indicating greater risk Loss data Cumulative impact of loss events Possible impact of “managed events”(intervention)

Management Report : Management Report Visual display of risk based on risk indicators With thanks to Excel, easy to produce Dashboards ….Red light / green light … ‘gauges’ Heat Maps Spider Diagrams

Industry Risk Benchmarks : Industry Risk Benchmarks Collection & summarization of risk data Loss data KRI Source for: Validation of enterprise results Trend analysis Risk analysis data Comparative benchmarking (company)

e.g. Standard KPI Specification : e.g. Standard KPI Specification www. KRIeX.org Repository ~ 2500 KPI specified and monitored

e.g., Loss Data : e.g., Loss Data

Bottom Line : Bottom Line Express organizational 'risk appetite’ in terms of a KPI threshold value Alert management to trends that may affect achievement of objectives Use KPI to initiate mitigation activity Provides measurable data conducive to aggregation Assists in demonstrating compliance

Next Week : Next Week CRISC Domain #4 Control Design & Implementation