21st Century Cures Real World Test Plan

Background & Instructions

Under the ONC Health IT Certification Program (Program), Health IT Developers are required to conduct Real World Testing of their Certified Health IT (45 CFR 170.556 and 170.523(i)). The Office of the National Coordinator for Health Information Technology (ONC) issues Real World Testing resources to clarify Health IT Developers’ responsibilities for conducting Real World Testing, to identify topics and specific elements of Real World Testing that ONC considers a priority, and to assist Health IT Developers to develop their Real World Testing plans.

Health IT Developers have maximum flexibility to develop innovative plans and measures for Real World Testing. As developers are planning for how they will execute Real World Testing, they should consider the overall complexity of the workflows and use cases within the care settings in which they market their Certified Health IT to determine which approaches they will take. This Real World Testing plan template was created to assist Health IT Developers in organizing the required information that must be submitted for each element in their Real World Testing plan. Health IT Developers must submit one plan for each year of Real World Testing (see resources listed below for specific timelines and due dates). ONC does not encourage updating plans outside the submission timeline and will not post updates on the Certified Health IT Product List (CHPL). If adjustments to approaches are made throughout Real World Testing, the Health IT Developer should reflect these adjustments in their Real World Testing results report. ONC would expect that the Real World Testing results report will include a description of these types of changes, the reasons for them, and how intended outcomes were more efficiently met as a result. This resource should be read and understood in conjunction with the following companion resources, which describe in detail many of the Program requirements referenced in this resource.

Health IT Developers should also review the following regulatory materials, which establish the core requirements and responsibilities for Real World Testing under the Program.

21st Century Cures Act: Interoperability, Information Blocking, and the ONC Health IT Certification Program final rule, 85 FR 25642 (May 1, 2020) (Century Cures final rule)

Section VII.B.5 — “Real World Testing”

General Information

Plan Report ID Number: [For ONC-Authorized Certification Body use only]

Developer Name: Elation Health, Inc

Product Name(s): Elation EMR

Version Number(s): Version 3

Certified Health IT Product List (CHPL) ID(s): 15.04.04.2717.Elat.03.00.1.181231

Developer Real World Testing Page URL: https://www.elationhealth.com/real-world-test-plan

Justification for Real World Testing Approach

Consistent with the ONC’s recommendation that “Real World Testing verify that deployed Certified Health IT continues to perform as intended by conducting and measuring observations of interoperability and data exchange”, this test plan focuses on capturing and documenting the number of instances that certified capability is successfully utilized in the real world. In instances where no evidence exists due to zero adoption of a certified capability or the inability to capture evidence of successful use for other reasons, we will demonstrate the required certified capability in a semi-controlled setting as close to a “real world” implementation as possible.

It is important to note that Real World Testing is only one component of the Health IT Certification program used to demonstrate compliance with the program requirements. Real World Testing should augment and support testing that was conducted prior to certification being granted. It is not intended to duplicate the methods or results previously demonstrated. Instead, this test plan was developed to demonstrate that the certified capabilities have been successfully deployed for providers to use at their discretion in live settings.

We are using a 2-fold approach to demonstrate successful real-world implementations.

Summative Testing
Interactive Testing

Summative assessments will be used to measure which certified actions were performed at the conclusion of a given time period. These will be conducted by generating reports and examining audit logs from within the certified health IT module to help demonstrate the frequency of actions within the given time frame, and where possible, whether those actions were successful or unsuccessful. High success rates should be an indicator of a successful implementation of a given certified capability in a real-world setting.

Interactive testing will be used to demonstrate conformance to requirements where the adoption rate of a given certified capability is zero and to demonstrate ongoing compliance with updated standards and code sets (SVAP). Interactive tests will require a live test as opposed to examining historical usage statistics. The goal is to allow a user to demonstrate the certified Health IT module being used in a way consistent with their own practice or care setting.

Standards Updates (Including Standards Version Advancement Process-SVAP and USCDI)

Standard (and version): All certified criteria meet the Cures Updates. 170.315 (g)(10) is updated using SVAP HL7 FHIR US Core Implementation Guide Version 4.0.0

Date of ONC ACB notification: 11/22/22

USCDI updated certification criteria: USCDI v 1

Care Settings

Elation EMR is marketed to primary care providers and certified accordingly for the ambulatory care setting.

Measures Used In Overall Approach

Summative Assessment Metrics

The following metrics will be measured by viewing audit logs and reporting systems available to track the behavior of the certified Health IT module during a given time frame. All metrics are designed to reflect the core elements of the criteria, demonstrate interoperability, and demonstrate the success rate of the certified capability being used. In most cases we elected to record these metrics over a 90-day period to reflect the reporting periods typically required for compliance with the federal incentive programs.

The continued measurable use of certified capabilities will provide implicit evidence of successful implementation of the required certified capability. This is especially meaningful in cases where interoperability with outside systems is demonstrated. In cases where it is not possible to determine “success” via an explicit confirmation by a receiving system, success will be defined as a transmission was made where no error was received from the destination system or its intermediaries. Additionally, we will review internal customer and vendor issue tracking systems for reports of failures or unsatisfactory performance in the field.

Criterion	Metric	Care Setting	Justification and Expected Outcome
170.315 (b)(1) Transition of care	Over a 90-day period: 1) Number of CCDAs created 2) Number of CCDAs sent via edge protocols 3) Number of CCDAs received via edge protocols	Primary Care	This criterion requires the ability of a certified Health IT module to create CCDAs according to specified standards and vocabulary code sets, as well as send and receive CCDAs via edge protocols. However, it is not possible to consistently and reliably demonstrate that all required standards and code sets were used because not all CCDAs created in a real-world setting contain all the necessary data elements. Furthermore, it is not feasible to obtain copies of CCDA documents from “outside” developers or providers who have no incentive to participate in this exercise. Therefore, we intend to demonstrate the required certified capabilities by demonstrating how often CCDAs are created and exchanged with other systems to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be high utilization by providers with a high success rate. Surescripts Clinical Direct Messaging will be used as relied upon software to achieve this outcome.
170.315 (b)(2) Clinical information reconciliation and incorporation	Over a 90-day period: 1) Number of times a user reconciled medication list data from a received CCDA 2) Number of times a user reconciled allergies and intolerance list data from a received CCDA 3) Number of times a user reconciled problem list data from a received CCDA	Primary Care	This criterion requires the ability of a certified Health IT module to take a CCDA received via an outside system and match it to the correct patient; reconcile the medication, allergy, and problem lists; and then incorporate the lists into the patient record. The expectation is each of these steps is done electronically within the certified Health IT module. While this certified capability is available to our users, most providers in the real world typically prefer to perform these steps manually and elect to save any outside received CCDAs as attachments to the patient record. Therefore, we intend to record the frequency that providers are electronically reconciling and incorporating CCDAs that were received from outside providers to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be low utilization by providers with a high success rate.
170.315(b)(3) – Electronic prescribing	Over a 90-day period: 1) Number of prescriptions created 2) Number of prescriptions changed 3) Number of prescriptions canceled 4) Number of prescriptions renewed	Primary Care	This criterion requires the ability of a certified Health IT module to perform prescription-related electronic transactions (eRx) using required standards. However, it is not possible to demonstrate the correct standards were used because it is not feasible to obtain copies of eRx documents from “outside” companies or pharmacies who have no incentive to participate. Therefore, we intend to demonstrate the required certified capabilities are effective by demonstrating how often eRx transactions are performed by examining reports from our eRx partner. This will demonstrate that not only are the eRx transactions sent from the certified Health IT module, but that the transactions are successfully received by the eRx clearinghouse. Our expectation is there will be high utilization by providers with a high success rate. Surescripts ePrescribing will be used as relied upon software to achieve this outcome.
170.315(b)(9) Care Plan	Over a 90-day period: 1) Number of care plans recorded 2) Number of care plans created	Primary Care	This criterion requires the ability of a certified Health IT module to record, change, access, create, and receive care plan information according to the specified format. We intend to record the frequency that care plans are recorded, and created to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be moderate adoption of this certified capability by our users.
170.315(b)(10) Electronic Health Information Export	Over a 90-day period: 1) Number of single patient exports 2) Number of patient population exports	Primary Care	This criterion requires the ability of a certified Health IT module to export a single patient export by a designated user, or an entire patient population export of all EHI. We intend to record and report the number of individual patient EHI exports and patient population exports to demonstrate availability and conformance. Our expectation is that there will be moderate utilization by providers with a high success rate.
170.315(c)(1-3) Clinical quality measures (CQMs)	Over a 90-day period: 1) Number of measures recorded during the period 2) Number of QRDA Category 1 files exported 3) Number of QRDA Category 1 files imported (if applicable) 4) Number of QRDA Category 3 aggregate report(s) created over the period	Primary Care	These criteria will be tested together. C1 requires a certified Health IT module to record required data, calculate CQMs from the recorded data, and export the data in QRDA Category 1 format. C2 requires a certified Health IT module must be able to import data from a QRDA Category 1 formatted file and calculate the CQMs based on that data. C3 requires a certified Health IT module must be able to create a QRDA Category 1 formatted file and a QRDA Category 3 aggregate report to be used for transmitting CQM data to CMS. We intend to record the frequency that CQM files are imported and/or exported by providers to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be moderate utilization by providers with a high success rate.
170.315(e)(1) View, download, and transmit to 3rd party	Over a 90-day period: 1) Number of views of health information by a patient or authorized representative 2) Number of downloads of health information by a patient or authorized representative 3) Number of transmissions of health information by a patient or authorized representative	Primary Care	This criterion requires the ability of a certified Health IT module to provide patients access to a patient portal with the ability to view, download, and send their health care records to other providers via encrypted or unencrypted transmission methods in CCDA format. We intend to record the frequency that patients are viewing, downloading, and transmitting their records from the portal using the certified capabilities to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be low utilization by patients for view and download and lower utilization for transmit with a high success rate for all certified capabilities.
170.315(f)(1) Transmission to immunization registries	Over a 90-day window: 1) Number (or percentage) of immunization records submitted to the immunization record	Primary Care	This criterion requires the ability of a certified Health IT module to transmit immunization data to a registry using a specified format. We intend to record the frequency that immunization data is submitted to registries by providers to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be moderate adoption of this certified capability by our users, and high success rate.
170.315(f)(2) Transmission to public health agencies — syndromic surveillance	Over a 90-day window: 1) Total number of syndromic surveillance events created and submitted	Primary Care	This criterion requires the ability of a certified Health IT module to transmit syndrome-based public health surveillance data to a registry using a specified format. We intend to record the frequency that syndromic surveillance data is submitted to registries by providers to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be zero adoption of this certified capability by our users, so we have added interactive testing methodology for these capabilities to the test plan below to demonstrate the feature is available and functions as expected should any users elect to begin using this feature.
170.315(g)(7) Application access — patient selection	1) Number of requests for a patient ID or token 2) Number of requests that provided sufficient information to provide a valid response 3) Number of follow-up requests made using the provided patient ID or token	Primary Care	This criterion requires the certified Health IT module to provide an API and supporting documentation that enable external applications to request a unique patient identifier from the certified Health IT module that can be used to request additional patient data. We intend to record the frequency that patient ID requests are received by providers via API to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be high utilization by providers with a high success rate.
170.315(g)(9) Application access — all data request	1) Number of requests for a patient’s Summary Record made by an application via an all data category request using a valid patient ID or token 2) Number of requests for a patient’s Summary Record made by an application via an all data category request using a valid patient ID or token for a specific date range	Primary Care	This criterion requires the certified Health IT module to provide an API and supporting documentation that enable external applications to request all categories of patient data defined in the CCDS from the certified Health IT module. We intend to record the frequency that patient data requests for all categories are received by providers and fulfilled via API to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be moderate utilization by providers with a high success rate.
170.315(g)(10) Standardized API for patient and population services	1) Number of requests for developer access to the FHIR environment. 2) Number of calls to FHIR resources to access patients USCDI data.	Primary Care	This criterion requires the certified Health IT module to provide an API meeting FHIR requirements and supporting documentation that enable external applications to request patient data by category from the certified Health IT module. We intend to record the frequency that patient data requests for USCDI data are fulfilled via API to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be low utilization with a high success rate.
170.315(h)(1) Direct Project	1) Number of Direct Messages sent 2) Number of Direct Messages received	Primary Care	This criterion requires the ability of a certified Health IT module to record the frequency that direct messages are sent and received by providers, along with how often MDNs are sent and received. Since not all systems respond with MDNs, we cannot reliably use that metric to define success. Furthermore, it is not feasible to obtain copies of Direct Messages from “outside” developers or providers who have no incentive to participate in this exercise. Therefore, we intend to demonstrate the required certified capabilities by demonstrating how often Direct Messages are exchanged with other systems to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be moderate utilization by providers with a moderate success rate. Surescripts Clinical Direct Messaging will be used as relied upon software to achieve this outcome.

Interactive Testing

The following test plans will be executed to demonstrate Real World certified capabilities for criteria where metrics are not available, because there is 0 or very low adoption of the criteria in the real world, either due to unanticipated lack of interest or other factors.

High Level Interactive Test Plan:

Care Settings: All interactive testing will be performed for each of the care settings listed above.
Test Environment: All interactive testing will be performed in a live production environment. Evidence will be collected via screenshots and logs.
Test Data: Interactive testing will be performed using test patient data in the live production environment in order to be as representative as possible of real-world deployments. Test patient data is used to ensure no exposure of PHI occurs.

Criterion	Interactive Test Plan	Care Setting	Justification and Expected Outcome
170.315(f)(2) Transmission to public health agencies — syndromic surveillance	Elation Health will work with a practice to perform a real life test in production with live patient data. The provider will record syndromic surveillance content and generate the HL7 ADT. 1) Provider will document the visit reason for the patient 2) Provider will generate the HL7 ADTs	Primary Care	This criterion requires the ability of a certified Health IT module to transmit syndrome-based public health surveillance data to a registry using a specified format. While Elation EMR provides this capability to our providers, we’ve seen low adoption of the capability and so we will demonstrate real world performance through interactive testing. We will select an Elation EMR provider to run a real life test with real patient data. Reported results will remove all PHI but demonstrate the following expected outcomes: – Record syndromic surveillance content – Generate the HL7 ADTs – ADTs are formatted correctly

Schedule of Key Milestones

Real World test planning will commence in first quarter of 2024. Each phase is expected to take 90-days to complete, with report writing to occur end of 2024/early 2025.

Key Milestones	Care Setting	Date/Time Frame
Data instrumentation	Primary Care	90 days
Data collection	Primary Care	90 days
Review and collate data	Primary Care	90 days
Writing report	Primary Care	90 days

Attestation

This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. All information in this plan is up to date and fully addresses the health IT developer’s Real World Testing requirements.

Authorized Representative Name: Conan Fong

Authorized Representative Email: conan.fong@elationhealth.com

Authorized Representative Phone: 415-231-5164

Authorized Representative Signature:

Authorized Representative Signature: Conan signature

Date: October 21, 2024