One of the most important audit preparation activities to ensure success during an ERO audit engagement is often missing from an entities audit preparation practices. This activity is a systematic process to test and evaluate compliance evidence using criteria set forth in the Generally Accepted Government Auditing Standards (GAGAS) Yellow Book, and used by ERO auditors to help them determine an acceptable level of evidence quality.
During my many years as a NERC regulator, testing and validating the quality of NERC compliance evidence, using a three step baseline criteria set forth in the GAGAS Yellow Book was the keystone to formulating our compliance determinations. I often trained Regional Entity staff on the criteria and how it applied to the data and information provided in RSAWS and during audits. We practiced using professional judgement and exercised professional skepticism if the evidence was of borderline quality.
It is KEY that organizations understand that ERO auditors do not approach the review of compliance evidence without a systematic structure that removes as much subjectivity from their decision making as possible. Auditors are looking for persuasive evidence that is supported by the key criteria. Why not adapt this model as a best practice to test and evaluate your evidence before it is submitted? A best practice approach to testing NERC compliance evidence should include the following three key steps of evaluation. Steps 1 & 2 are detailed in GAGAS Yellow Book Chapter 6.
Quality evidence represents the following:
When testing, and evaluating compliance evidence, determine if the collective evidence will lead a prudent person to the same valid conclusions that your team reached (Sufficient). Stronger evidence may allow less evidence to be used (Sufficient). In some cases, one quality piece of evidence may be sufficient for the requirement. For a more comprehensive or complex requirement, one document may not be sufficient. This would require complementary evidence to support your case (Adequate). Evidence is often considered more reliable when it is complemented with different sources (Appropriate). However, submitting a large volume of evidence does not compensate for the lack of relevance, validity, or reliability of that evidence (Appropriate).
It is important to understand that evidence is not sufficient nor appropriate when:
Questions to Ask:
If you have any questions or would like more information on testing NERC compliance evidence for quality, please feel free to give me a call or drop me a line.
Do you have questions regarding your organization, compliance, risk, strategy or operations? Get your questions answered.Schedule a call
One of the most important audit preparation activities to ensure success during an ERO audit engagement is often missing from an entities audit preparation practices. This activity is a systematic process to test and evaluate compliance evidence...
There has been a lot angst, discussion, and Scuttlebutt (Navy term) since NERC levied the largest and most significant CIP regulatory fine. 10M dollars sure got the attention of the energy industry. Since the announcement, I received calls from...
I received a lot of great feedback on my last publication “Testing NERC Evidence for Quality”. During discussions about this article, several people asked me to write an additional article and share my perspective on best practices and key...
The Western Interconnection Regional Advisory Body (WIRAB) commissioned a report from InPOWERd LLC on the NERC Reliability Issues Steering Committee’s (RISC) November 2016 report on ERO Reliability Risk Priorities.