Testing NERC Compliance Evidence for Quality – Perspective of an Ex-Regulator

One of the most important audit preparation activities to ensure success during an ERO audit engagement is often missing from an entities audit preparation practices. This activity is a systematic process to test and evaluate compliance evidence using criteria set forth in the Generally Accepted Government Auditing Standards (GAGAS) Yellow Book, and used by ERO auditors to help them determine an acceptable level of evidence quality.

During my many years as a NERC regulator, testing and validating the quality of NERC compliance evidence, using a three step baseline criteria set forth in the GAGAS Yellow Book was the keystone to formulating our compliance determinations. I often trained Regional Entity staff on the criteria and how it applied to the data and information provided in RSAWS and during audits. We practiced using professional judgement and exercised professional skepticism if the evidence was of borderline quality.

It is KEY that organizations understand that ERO auditors do not approach the review of compliance evidence without a systematic structure that removes as much subjectivity from their decision making as possible. Auditors are looking for persuasive evidence that is supported by the key criteria.  Why not adapt this model as a best practice to test and evaluate your evidence before it is submitted? A best practice approach to testing NERC compliance evidence should include the following three key steps of evaluation. Steps 1 & 2 are detailed in GAGAS Yellow Book Chapter 6.

Quality evidence represents the following:

• Sufficient: the collective weight of the evidence is enough to lead a prudent person (persuasive support) to the same valid conclusions on which the entities opinion is based.
• Appropriate: Relevant, valid, and reliable in providing support for findings and conclusions. Bears a clear, logical, and repetitive relationship to the Standard Requirements.
• Adequate: Evidence that is of high enough quality to be used for analysis and proof (e.g. version control, approved and signed by appropriate authorities).

When testing, and evaluating compliance evidence, determine if the collective evidence will lead a prudent person to the same valid conclusions that your team reached (Sufficient). Stronger evidence may allow less evidence to be used (Sufficient). In some cases, one quality piece of evidence may be sufficient for the requirement. For a more comprehensive or complex requirement, one document may not be sufficient. This would require complementary evidence to support your case (Adequate). Evidence is often considered more reliable when it is complemented with different sources (Appropriate). However, submitting a large volume of evidence does not compensate for the lack of relevance, validity, or reliability of that evidence (Appropriate).

 It is important to understand that evidence is not sufficient nor appropriate when:

• An unacceptable high risk that It could lead an auditor to an incorrect or improper conclusion.
• The evidence has significant limitations, given the audit objectives and intended use of the evidence.
• The evidence does not provide an adequate basis for addressing the audit objectives or supporting the findings and conclusions

Questions to Ask:

• What methods do you use to ensure the evidence you include in RSAWs and provide to auditors during a compliance engagement will withstand the scrutiny of a highly skilled auditor?
• Are you evaluating your evidence from the same criteria and guidelines as the auditor?
• Is your evidence of borderline quality? Will auditors need to use professional judgment and professional scrutiny to make determinations?

If you have any questions or would like more information on testing NERC compliance evidence for quality, please feel free to give me a call or drop me a line.