Major Award for Process-Aware EMR / EHR BPM Workflow System: Congratulations and Thank You!

Congratulations to the Cardiff School of Computer Science & Informatics and the Velindre NHS Trust Cancer Centre  (check out Welsh language version of their website) for bringing the future of process-aware EMR / EHR BPM workflow systems closer to today. Their prototype workflow management system/business process management system for healthcare won a 2011 Global Awards for Excellence in Adaptive Case Management. I urge you to read the full text of the case study.

cardiff-wordle

Wordle based on sixteen short quotes
from the awarding winning application

I regularly scour the Web looking for evidence of convergence between the worlds of EMR/EHR and workflow management systems/business process management (check out the EHR.BZ REPORT on EMR / EHR Workflow, Usability and Productivity). To quote myself:

“The next step in the evolution of ambulatory EMRs is squarely at the intersection between two great software industries: electronic health record systems and workflow management/business process management systems. The hybrid EMR workflow systems that result will be more usable and more systematically optimizable than traditional EMRs with respect to user satisfaction, clinical performance, patient satisfaction, and practice profitability.” (A White Paper About EMR Workflow, Usability, and Productivity in Pediatric and Primary Care)

Eight years before that I co-authored a paper about the first EMR / EHR workflow management system, about which I wrote:

“Developments that are beginning to affect collective thinking of the HIT industry today:

  • Workflow management systems (business process management today)
  • Computer-supported collaborative work (groupware and workflow systems today)
  • Componentized EMR architecture (software modules/plugins and service-oriented architecture today)”

(Ten Years Ago, Dallas HIMSS: Landmark Presentation on Modular Pediatric EMR Workflow Groupware)

It’s too soon to write “I told you so!” but that delicious moment draws ever closer. Health information technology and business process management industries increasingly overlap and interact, giving rise to much needed hybrid vigor.

Below are quotes from the case study. If you follow healthcare discussions about EMR / EHR usability, interoperability (especially workflow, or pragmatic, interoperability), clinical intelligence, population health management, clinical groupware, collaboration, coordination, continuity of care, they will surely pique your interest. Skim the bold emphases I added and think (”Just think!”) about their relevance to solving many of the problems plaguing today’s generation of EMRs and EHRs.

Mind the Gap! (those strategically placed three-dot thingies)

open-quoteTeamwork, collaboration and coordination are key aspects of the patient-centric approach taken by modern healthcare….The work described includes tracking care teams and individual team members dynamically as the patient progresses along the dynamic care pathway. It proposes the integration of a Business Process Management (BPM) system into the HISs as it will better support both the individual work of health and care practitioners as well as improve support for team communication, and care coordination throughout the patient’s care….The proof of concept prototype showed that the application of Workflow Technology in the healthcare domain is a very promising development. It can be used to evolve the functionalities of existing HIS, so that they can be used to support implementation of ICP services and associated treatment flow for a patient. It is believed that these functionalities are important as they result in safer more effective and efficient care and treatment. Functionalities of the proposed system include: providing a pro-active system, routing and information filtering….The strength in the BPM systems is in its ability to invoke existing systems at any stage during the process flow in a way which overcomes the system heterogeneity challenge….BPMs can be adjusted to enforce a specific sequence and\or enable an extremely flexible order of processes….At a business level, the system will facilitate multi-professional care team communication and therefore care coordination across the multiple healthcare organisations involved in the treatment of patients….The proof of concept prototype showed that the application of Workflow Technology in the healthcare domain is very promising….Functionalities of the proposed system include: providing a pro-active system, routing, task automation and information filtering. These functionalities are: Pro-active System: this is the primary advantage of using Workflow Technology in the healthcare domain. It is the difference between having a reactive or a proactive system. In the case of a reactive system, which most traditional HISs are, reactions are a response to requests made by users while proactive systems are capable of identifying the need to take an action and activities it activates. The workflow engine within the workflow management system can be coded to fetch triggers, understand which of the users or roles are affected by this trigger, how they are affected and finally, take appropriate actions to inform them. These pro-active functionalities can be used to execute many different actions, such as: alert, notify, refer, schedule and set timers….Routing: this helps the system determine the sequence of the processes and the consequences of any decision made. This is either by suggesting the next stage or automating a set of processes. While routing is a tool that WFMSs provide, routing is only made with a user’s approval or suggestion. This is done by providing a message to the user showing the alternative routes according to the ICPs’ logic and the user can approve any or simply skip it….Task automation: is performed when a number of tasks need to be processed as a set. In this context, it includes tasks that do not require user interaction….Information extraction and filtering: to ensure summary of important information is visible to healthcare professionals when viewing a patient’s records. This aims to facilitate tracking a patient and improving the decision-making process by making healthcare professionals aware of the development of the care process and therefore making better use of their time. This includes improved visibility of: treatment history, milestones, order and time, and acting healthcare professionals….At the system level, the HIS will become more pro-active and capable of performing the following actions: alert, notify, refer, schedule, and set timers. It also provides a flexible system that handles dynamic changes happening during a patient’s treatment. This includes routing the flow and performing automatic tasks. Moreover, the workflow system provides a tool to track patients and ensure continuity of the flow by filtering and extracting important information. The treatment information extracted includes: history, milestones, order and time and involved care team professionals….The coordination problem…need for a more proactive system that facilitates care coordination among care team members…Business Workflow: This is required to model the huge number of processes interacting in a healthcare system. Business workflows support human interaction with the system. This is necessary in healthcare systems, where different care team professionals interact with the system and support the decision making process and therefore the routing of the flow….Activities represent all the treatment and diagnosis options a patient can follow. These options should be modelled in the WFMS and form the main block of the system….A Process: this is the actual workflow map of the clinical guidelines…An Activity: the steps of the clinical guidelines….A Process object: this provides control or audit to the case according to its state….A case: this is the scenario in progress. Each case represents a patient’s treatment flow. For each patient, the treatment pathway is unique and is processed by considering a patient’s health condition and the available resources. The case hierarchy at run time usually shows treatment history, the progress, the state in each and the roles or users involved in different stages.

close-quote

 

Again, congratulations…and thank you!

Posted in healthcare-BPM | Tagged , , , , , , , , , , , , , , | Leave a comment

Clinical Intelligence, Complex Event Processing and Process Mining in Process-Aware EMR / EHR BPM Systems

Short Link: http://ehr.bz/8c

Last fall I presented a paper, co-written with Mark Copenhaver, at MedInfo2010 in Cape Town, South Africa.

Webster C. & Copenhaver, M. Process-aware EHR BPM Systems: Two Prototypes and a Conceptual Framework. In: Proceedings of the 13th World Congress on Medical Informatics, Studies in Health Technology and Informatics, Volume 160, 2010, pp 106-110. (indexed in MedLine)

You may have noted my photo travelogue at the time.

Process-aware health information system research and related industry undertakings have evolved since we built clinical intelligence and process mining prototypes in 2009 and thought how to bridge between healthcare IT and business process management. One of the goals of the blog is to draw readers, equally, from the realms of EMR/EHR/Health IT and BPM/Workflow/groupware. If you are from one of these industries but not the other, you’ll know some of the terminology, but not all. Hence a lengthy editorial preface plus a glossary of EMR / EHR workflow terminology as an addendum.

You can keep reading, skip to the abstract/slides/notes, or peruse the following outline and cut the (slide) deck wherever you like.

Short “Editorials” on EMR, EHR, BPM, BI, CI, CEP, Productivity and Usability

  • EMR, Electronic Medical Record
    • Perhaps the simplest definition of EMR is a “computerized ’systematic documentation of a single patient’s long-term individual medical history and care’” where all the words between the single quotes are simply the definition of a “medical record” (wiki)
  • EHR, Electronic Health Record
    • Some use “EMR” and “EHR” synonymously. I often do. Others regard an EMR as being within a healthcare organization, such as medical office or hospital, while an EHR is a sum of capabilities to share and coordinate data and care across organizations. I’m OK with that too, but note use of the EHR acronym did not take off until the Federal government began using it instead of EMR. I sometimes hedge my semantic bets by using the phrase “EMR / EHR”. If a reader thinks they mean the same thing, then I appear to be noting synonymy. If they believe EMR and EHR mean different things, then I appear to refer to the totality of EMR plus EHR. Either works for me.
  • Workflow Management Systems, Business Process Management, BPM
    • As noted in Wil van der Aalst’s 2004 book Workflow Management: Models, Methods, and Systems, by analogy a workflow management system is to a workflow system much as a database management system is to a database system (for more on this distinction). In each case, the former creates and manages the latter. Workflow management systems are narrower in scope than business process management systems, sometimes designed to do little more than flexibly automate collections of tasks. Business process management systems, or suites, add capabilities from business activity monitoring and business intelligence to process mining and simulation to flexible user-customizable user interfaces.
  • Business Intelligence

“There is no clear definition for BI. On one hand it is a very broad term that includes anything that aims at providing actionable information that can be used to support decision making. On the other hand, vendors and consultants tend to conveniently skew the definition towards a particular tool or methodology. Clearly, process mining can be seen as a new collection of BI techniques. However, it is important to note that most BI tools are not really “intelligent” and do not provide any process mining capabilities. The focus is on querying and reporting combining simple visualization techniques showing dashboards and scorecards….Under the BI umbrella many fancy terms have been introduced to refer to rather simple reporting and dashboard tools.” (p. 21)

“Many vendors offer Business Intelligence (BI) software products. Unfortunately, most of these products are data-centric and focus on rather simplistic forms of analysis….process-centric, truly “intelligent” BI is possible due to advances in process mining.” (p. 261)

“BI products do not show the end-to-end process and cannot zoom into selected parts of this process….Another problem of mainstream BI products is that the focus is on fancy-looking dashboards and rather simple report, rather than a deeper analysis of the data collected. This is surprising as the “I” in BI refers to ‘intelligence’.” (p. 263, emphasis in original)

Also see How Process Mining is Related to BI where they write:

“The added value of process mining over traditional BI reporting tools lies in the depth of the analysis.

Traditional BI reporting tools focus on the display of Key Performance Indicators (KPIs) for executives in the organization. For example, the cycle times of a customer-facing process may be key in meeting certain service levels that have been agreed.

If the cycle times are out of the acceptable bounds, dashboards can highlight this problem. However, they cannot do much to uncover the root causes for this problem. Process mining can help to provide much deeper insight into the actual processes by uncovering the process flows and bottlenecks based on existing IT logs in a bottom-up manner.

Essentially, BI assumes that the underlying processes are known. Process mining takes the stand that even well-defined processes usually don’t go as planned and need to be brought into light objectively.” (my emphasis)

Out-of-bound KPI cycle times explained by process mined bottlenecks…this is exactly the capability demonstrated by one of the two EMR / EHR BPM modules presented below.

  • Clinical Intelligence
    • Definitions of clinical intelligence also vary according to vendor and consultant, tool and methodology. At this point, it is perhaps best defined as business intelligence tools and methods applied to patient care and health, and left at that. Later I’ll describe a specific clinical intelligence tool presented at MedInfo2010 (slides and notes also below).
  • Clinical Groupware
    • Clinical groupware is a combination of the “intentional care team processes and procedures pertaining to the observation and treatment of patients plus the tools designed to support and facilitate the care team’s work.” Note the emphasis on “team”. It’s unusual to see that same word, especially a noun, used more than once in a definition, so it must be important! I sometimes refer to clinical groupware as teamware (as opposed to “singleware”). Clinical groupware includes workflow systems, workflow management systems, business process management systems, and adaptive case management systems when applied to clinical coordination and collaboration.
  • Process-Aware Information Systems
    • PAISs, or Process-Aware Information Systems, include business process management systems (which in turn include workflow management systems). While database systems and email programs, for example, may execute steps in a process, they do not contain, consult, or are “aware” of, any explicit process models. Most current traditional EMR / EHR systems are not process aware. While EMR / EHRs are gradually incorporating more-and-more sophisticated task management features, most of these capabilities are relatively frozen, their workflow not amenable to editorial control by EMR / EHR users.
  • Process Mining
    • Since I own Process Mining by, arguably, the world’s expert on the topic, I might as well just quote Wil van der Aalst again:

“The goal of process mining is to use event data to extract process-related information, e.g., to automatically discover a process model by observing events recorded in some enterprise system.” (Process Mining: Discovery, Conformance and Enhancement of Business Processes)

“The healthcare industry includes hospitals and other care organizations. Most events are being recorded (blood tests, MRI scans, appointments, etc.) and correlation is easy because each event refers to a particular patient. The closer processes get to the medical profession the less structured they become. For instance, most diagnosis and treatment processes tend to be rather Spaghetti-like…. Medical guidelines typically have little to do with the actual processes. On the one hand, this suggests these processes can be improved by structuring them. On the other hand, the variability of medical processes is caused by the different characteristics of patients, their problems, and unanticipated complications. Patients are saved by doctors deviating from standard procedures. However, some deviations also cost lives. Clearly hospitals need to get a better understanding of care processes to be able to improve them. Process mining can help as event data is readily available. (emphasis in original)

I’ll illustrate the use of process mining to generate a process model for comparing nine busy pediatric practices in slides and speaker notes below.

If you are interested in process mining applied to healthcare, a good place to start are these three recent introductory posts.

Four Challenges for Process Mining in Healthcare

Process Mining in Healthcare - Case Study No. 1

Process Mining in Healthcare - Case Study No. 2

  • Usability, Human Factors
    • Usability is “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.” EMR / EHR usability, applying human factors principles and methods to EMR / EHRs, is a hot topic because physicians are “resisting” adoption of EMR / EHRs and there must be a good reason. The current reason célèbre is that EMRs are “clunky” and that usability engineers can fix this clunkiness. I’m a great fan of the cognitive science behind usability, but I do have a bone to pick (hmm, more like axe to grind). The relationship between EMR / EHR usability and workflow is both profound and misunderstood. When users complain about EMR usability they are often really complaining about EMR workflow that gets in their way, instead of paves their way. The biggest reason that current traditional-style EMRs are difficult to use is not so much because they weren’t created by an army of usability engineers, but because EMR / EHRs don’t rely on executable process models under editorial control (at both design- and run-time) of the users relying in them. It’s the users who should make their EMRs more usable, not EMR programmers or even usability experts. If non-programmer users could more easily push-pull-poke malleable EMR workflow into more usable workflow (for them, their goals, and their context), more usable EMRs and EHRs would result. I am *not* against usability testing. Usability testing *will* result in more usable EMRs. However, the most important form of EMR / EHR usability is a kind of meta-usability: usability of the tools users need to improve EMR usability, themselves. The most important meta-usability is the ability to improve the usability of EMR workflow. All the usability testing in the world won’t convert current document-based, data-centric, non-process-aware EMRs into workflow-based, process-aware EMRs. It’s too much of a paradigm shift and there is too much investment (and therefore design inertia) built into current legacy EMR / EHR product infrastructure.
  • Productivity
    • There’s also been a lot of press about current EMRs / EHRs reducing physician productivity (this letter to the New York Times is typical). Much of the current impetus to improve EMR usability comes from this press. Yes, usability is part of the problem. However, a traditional approach to EMR usability focuses on solitary users in front of solitary computers accomplishing (relatively) solitary tasks. The alternative is to study teams of users coordinating accomplishment of coordinated tasks “in the wild” (a reference to one of the first books about distributed cognition). The problem with the first approach is that it’s difficult to generalize from simulated laboratory experiments with individual users back to the real world. On the other hand, studying teams using groupware in the real world is fraught with its own problems. However, studying how better workflow leads to better usability and higher productivity will require it. We need hard data (such as, for example, process models generated from individually time-stamped user clicks) from teams of users actually using EMRs in the real world. Process mining can provide this. In fact, process mining is already used to study usability.
  • Complex Event Processing, Event-Driven Architectures
    • An event is a change in state, such as a patient who gains weight and moves from obese to morbidly obese state categories. A complex patient event is a pattern of detected events amidst a patient event stream (such as moving from obese to morbidly obese combined with being diabetic). Complex event processing, implemented in conjunction with a BPM system, provides means to react to events in real-, or almost real-, time. In the case of a process-aware information systems such as EMR / EHR workflow management (or business process management) systems, patient events can drive automated clinical workflows (such pushing action items to worklists) via workflow engines executing process definitions (more below).

Abstract, Slides, and Speaker Notes:

“Process-aware EHR BPM Systems:
Two Prototypes and a Conceptual Framework”

Abstract

Systematic methods to improve the effectiveness and efficiency of electronic health record-mediated processes will be key to EHRs playing an important role in the positive transformation of healthcare. Business process management (BPM) systematically optimizes process effectiveness, efficiency, and flexibility. Therefore BPM offers relevant ideas and technologies. We provide a conceptual model based on EHR productivity and negative feedback control that links EHR and BPM domains, describe two EHR BPM prototype modules, and close with the argument that typical EHRs must become more process-aware if they are to take full advantage of BPM ideas and technology. A prediction: Future extensible clinical groupware will coordinate delivery of EHR functionality to teams of users by combining modular components with executable process models whose usability (effectiveness, efficiency, and user satisfaction) will be systematically improved using business process management techniques.

Keywords: EMR, Electronic Medical Record, EHR, Electronic Health Record, WfMS, Workflow Management Systems, Business Process Management, BPM, Business Intelligence, BI, Clinical Intelligence, Clinical Groupware, PAIS, Process-Aware Information Systems, Process Mining, Usability, Human Factors, Productivity, Complex Event Processing, CEP, Event-Driven, Clinical Quality Measures, Protocols, Guidelines, Compliance, Outcomes, Population Health Management, KPI, Key Performance Indicators, Closed-loop Patient Care

1-title

Thank you for attending this session on Process-Aware EHR Business Process Management Systems: Two Prototypes and a Conceptual Framework.

2-outline

My presentation outline is as follows…

I’ll…

  • introduce EMR / EHR productivity, which can be systematically improved with business process management technology.
  • discuss a prototype clinical intelligence BPM module, called PROCARE, intended to systematically improve the state of health of patients in an EHR database.
  • describe a prototype process mining BPM module, called PROCESS, intended to systematically improve EHR workflow efficiency.
  • Finally I’ll list seven general advantages of process-aware EHR BPMS systems over EHRs that lack workflow engines, process definitions and the functionality these enable.

The red numbered bullet points (2, 2.a-d, 3, 3.a, 3.b) correspond to upcoming numbered slides.

EHR Productivity = Information Value / Information Cost

3-framework

The concept of EMR / EHR productivity can bridge between EHRs and BPM technology realms. EHR productivity is the value of information contained in an EHR divided by the cost of obtaining that information. Information value and information cost can be systematically improved, maximized and minimized respectively, using BPM techniques such as business intelligence, process mining, and complex event processing.

Closed-Loop Optimization can Systematically Improve EHR Productivity

4-closed-loop-optimization

I am sure you are familiar with negative feedback loops such as implemented by thermostats operating to minimize the difference between observed and desired temperature. The difference between observed and desired state/output steers system state/output toward desired state/output. Many complex systems, from missiles to reactors, use sophisticated implementations of negative feedback loops to optimize system behavior.

[See Closed-Loop Strategies for Patient Care Systems for further overview of closed-loop control and its history, use, and future potential in healthcare.]

Healthcare information technology increasingly seeks to implement closed-loop systems, using estimated measures of clinical outcome and resource consumption to improve performance.

In order to systematically improve EHR productivity the information value numerator should be systematically increased while the information cost should be systematically decreased. In our formulation information cost is inversely proportional to efficiency level.

Closed-Loop Population Management and Closed-Loop Process Improvement

5-ehr-productivity

This slide is a graphical representation of an outline to this presentation. The numbered red boxes (2, 2.a-d, 3, 3.a, 3.b) correspond to upcoming slides.

You can think of improving EHR productivity in terms of negative feedback loops within a negative feedback loop. Two inner feedback loops implement systems for systematically increasing EHR information value and systematically decreasing EHR information cost. The outer feedback loop systematically increases the ratio of EHR information value to its cost to create.

Clinical Intelligence Plus CEP Drives Process Definition Execution

6-closed-loop-population-management

PROCARE, PROvision-based Clinically Active Reporting Environment, was a prototype BPM clinical intelligence module created to interact with an EMR / EHR workflow management system that relies on a workflow engine to execute process definitions.

A clinical intelligence reporting system without ability to trigger automated workflow is a passive reporting system (in which reports must be handed to staff for disposition, “Please put a note in each patient’s chart so that the next time they have an appointment…”). An active reporting system feeds directly back to a workflow engine executing clinical process definitions to automatically perform useful tasks–hence the “Active” in PROCARE’s Provision-based Clinically Active Reporting Environment.

“Provision” is borrowed from legal terminology. It means forward-looking restriction or qualification in a contract or agreement. For example, a patient can be in a predefined class of patients provided they meet that class’s predefined criteria (age between 0 and 18, BMI > 30, etc.). Many clinical intelligence reporting systems use predefined, or user defined, criteria to include or exclude patients from numerators and denominators of clinical performance measures.

To implement an EMR / EHR-based population health management you need a measure of health state, or a surrogate such as clinical performance (those clinical quality measures, with their numerators and denominators and exclusion categories and so forth). This direct or indirect measure (or combined measures) is compared to a goal value. The difference, or at least direction, is used configure states, events, policies, and process definitions to use patient events to drive automated workflows improving health state/clinical performance. A human user, reacting to patient state reports and clinical dashboards, provides an important part of this negative feedback loop.

Clinical Dashboard Displays Patient On-Protocol/Compliant, Measured, Controlled Percentages

7-closed-loop-population-management

This slide shows a clinical dashboard relative to its function within this negative feedback formulation (”Health Monitor” in red box).

8-procare-dashboard1

procare-clinical-dashboard

PROCARE’s clinical dashboard displays KPI’s (Key Performance Indicators) for each measure of clinical performance four numbers corresponding to the four levels of a “patient class event hierarchy” (which I’ll display several slides from now):

  • number of patients in the class for which the measure applies,
  • percentage of patients in each class that are compliant with a predefined protocol,
  • percentage of patients for whom appropriate and timely measurements are available, and
  • percentage of patients for whom observed measures are controlled (within target normal limits).

These colorful graphs on the right represent the same information in comparison to goal thresholds:

  • Green means performance measure above threshold
  • Yellow means at or near clinical performance threshold
  • Red means below clinical performance threshold.

Patient List Manager Enables Ad-Hoc and Policy-Based Intervention Planning

9-patient-list-manager

procarepatientlist540wide

Selecting a measure of clinical performance in the summary display brings up a patient list management screen for intervention planning. Creation or refinement of automated workflow policies link patient class events to automated workflows. For example, process definition steps could include sending work items to roles or users, work items that appear when the patient next is physically present, instructions that appear automatically whenever a patient chart is opened, or messages to external systems that trigger email or phone calls.

While interventions can be triggered manually (from the patient list manager for individual patients or groups of patients meeting prior or user specified criteria) or automatically (via clinical CEP policies linking patient events to automated workflows) one optimization goal is to gradually replace manual interventions with automatic policy-based interventions, decreasing resource consumption while increasing predictability. Over time, slow, inconsistent, manual workflow “compiles” into fast, consistent, automated workflow.

Linking a clinical business intelligence system and a workflow engine automatically executing process definitions is what makes PROCARE an example of clinical complex event processing. A non-programmer, a clinical user, can create and edit patient state definitions, policies linking patient events (changes in patient state), and executable process definitions, turning an EMR / EMR, with a workflow engine, into an active clinical assistant, tirelessly working to achieve goals its human users program into it.

Patient Class Event Hierarchy Intermediates Patient Event Stream and Automated Workflow

11-closed-loop-population-management

Now we’ll drill down into the patient class event hierarchy (red box labeled Patient Class Events) used to trigger automated EHR workflows.

This decision tree is the critical intermediate representation mediating between low level patient events (state changes) and higher level concepts clinical concepts such as “on-protocol,” “compliant”, “measured”, and “controlled.”

12-patient-class-event-hierarchy1

Here you can see where the numbers in the PROCARE clinical performance dashboard come from:

  • #P+EHR, number of patients in EHR;
  • #P+MC, number of patients meeting clinical criteria;
  • #P-MC, number of patients not meeting clinical criteria;
  • #P+P/C, number of patients on-protocol/compliant,
  • #P-P/C, number of patients not on-protocol/compliant;
  • #P+MM, number of patients for whom target metrics have been measured within specified time interval;
  • #P-MM, number of patients for whom target metrics have not been measured within specified time interval;
  • #P+C, number of patients for whom target measures have been measured within specified time interval and are under control (within normal limits);
  • #P-C, number of patients for whom target measures have been measured within specified time interval and are not under control (not within normal limits).

Regarding #P+/-P/C (Patients On Protocol/Compliant, or not), if you have direct compliance-relevant data feeds from devices in the home, for example, this patient class event hierarchy likely should separate into On Protocol (#P+/-P) and Compliant (#P+/-C) levels.

To summarize…

Execution of appropriate automatic policy-based workflows (in effect, intervention plans),

  • for patients who aren’t on protocol but should be,
  • aren’t being measured but should be,
  • or whose clinical values are not-controlled,

moves patients from

  • off-protocol to on-protocol,
  • non-compliance to compliance,
  • unmeasured to measured, and from
  • uncontrolled to controlled state categories,

improving individual and collective patient health state, causing a shift from red to yellow to green graphical indicators on the clinical dashboard.

KPIs and Process Mining Flag and Identify EMR / EHR Workflow Bottlenecks

13-closed-loop-process-improvement

Now let’s take a look at the denominator of the EHR productivity formula. This module was called PROCESS, for PROcess Comparison for Efficient System Specification.

PROCESS is an example of process mining. Process mining generates process models from workflow, or event, logs. An event data point can be as little as a number identifying a patient encounter, the name of a task (”Record Allergies”), and a time stamp.

14-closed-loop-process-improvement

We needed a measure of global efficiency to optimize. We chose average throughput time, also called cycle time by industrial engineers. [Note red box, "Efficiency Monitor".]

nine-medical-practices-productivity-statistics2

We process mined the workflow logs of nine pediatric practices to compare productivity measures and workflows, and highlight possible bottlenecks that could be alleviated by changing executed process definitions–hence the “Comparison”, “Efficient”, and “Specification” in PROCESS’s Process Comparison for Efficient System Specification. We benchmarked practice throughput volume and times against each other. Three practices stood out [see circled practices in previous slide]. We noticed that one of the three busiest pediatric practices (Practice 9, in blue) had a dramatically longer throughput time. Practices 5 (red) and 7 (green) took only 23 minutes and 44 minutes, respectively, to open and close a patient chart. In contrast, practice 9 took over eight hours to complete its charts. Obviously the patient was long gone by then.

15-closed-loop-process-improvement

Now we’ll drill down into a process model generated from the combined workflow/event logs of these nine pediatric practices. [Note red box, "Compare Processes".]

Individual EMR / EHR Workflow Steps are Time-stamped and Logged

16-ehr-workflow-steps

This EHR workflow management system has screens devoted to each possible data review and entry and order entry step. Table letters A through Y index the names of EMR screens: Allergies, Anticipatory Guidance, Chart Review and so on. Let me draw your attention to two pairs of task screen steps, Get Patient (H) and then Current Meds (E) versus Examination (F) and then New Note (G). In the former case (H to E) a nurse gets the patient and then asks about current medications. In the latter case (F to G) a patient examination is followed by creating a new note about the patient.

Process Mined Workflow/Event Logs Generate Detailed Process Models

17-compare-improve-processes

This is the transcribed result of process mining the workflow logs for nine pediatric practices for the busy month of October 2008. The letters correspond to the individual screen tasks. Reviewing the process models revealed that practice 9 differed from practices 5 and 7 primarily in that many charts appeared to pile up between the Examination and a New Note steps and then stay there (red arrow from F to J). Practices 5 and 7 also showed some degree of congestion earlier in their workflow (red arrow from F to G), but this apparently did not have a dramatic impact on throughput time. The practice skills instructor took one look at this and said “Who is practice nine? They are doing something wrong and I need to fix their workplans!” (Process definitions are called “workplans” in this EHR workflow management system.)

[Recall that second quote about process mining and business intelligence? PROCESS is an example of what it described, explaining an out-of-bounds cycle time (the KPI, or key performance indicator) via a potential bottleneck in the process model generated by process mining EMR / EHR event data.]

Active Clinical Intelligence and Systematically Improvable Clinical Processes Require Process-Aware Foundations

18-need-process-aware-ehrs

Business process management systems, or suites, rely on workflow engines and process definitions but add additional value, such as user-friendly user interfaces or visual analytics to better understand processes. More recently, in the US, the phrase “clinical groupware” has also become popular. Workflow systems are classical examples of groupware. So, in keeping with recent trends in both the health information technology and business process management industries, I sometimes refer to these systems as “process-aware clinical groupware.” What all these technologies have in common is that they are “process-aware.”

Without a process-aware foundation, that is, without an executable process model, neither PROCARE nor PROCESS (or EMR / EHR BPM modules similar to them in functionality) would be possible and have practical effect. Actionable clinical intelligence, that is, active as opposed to passive clinical reporting requires some means to automatically detect salient patient events and then automatically trigger automated workflows, transparently and usably at physician behest. These automated workflows interleave with other workflows, manual and automated, to generate a deluge of time-stamped data, the basis for generating sophisticated operational clinical process intelligence to explain and improve clinical processes. Process mining is not just about improving efficiency. Any KPI–clinical outcomes; practice productivity and profitability; patient and user satisfaction–can be compared across medical practices and difference in KPI values (good or bad) explained by processes generating or influencing them. These fact- and process-based explanations can direct further investigation and intervention.

Seven Advantages of Process-Aware EMR / EHR BPM Over Process-Unaware Alternatives

19-advantages-process-aware-ehr-wfms-bpm

Process-aware EMR / EHR workflow management systems/business process management systems have numerous advantages over their process-unaware cousins.

Non-process aware EHRs do not distinguish between unitary tasks at the same fine degree of granularity as process-aware EHRs. Traditional EHRs often have high resolution screens with a multitude of simultaneous data review and entry and order entry options. Multiple user events, spanning multiple tasks, are often committed together to the underlying database, conflating together logically separate workflow steps. In contrast, a process definition-driven EHR can present just the data review and entry and order entry options on each screen that are relevant to a single step in a task workflow sequence. For example, a nurse checking allergies and then current medications are two different tasks that at highly granular resolution should be distinct and acquire different time stamps.

Non-process aware EHRs do not capture all the potential meaningful timestamps for those events that they do log. They may log when data and orders are committed to a database but they do not typically log when tasks are first available to be accomplished, when they begin, when they complete, and other relevant timed-stamped events such as cancellation, postponement, or forwarding. Much of this missing temporal information is invaluable for understanding why bottlenecks occur, why certain tasks are subject to rework, and what slack resources are available elsewhere in the system.

Non-process aware EHRs, even if their event logs result in useful process models and actionable insights, lack means to actively influence changes to workflow. There are no process definitions or workflow engines to execute them; so there are no process definitions to change and thereby influence and improve effectiveness and efficiency. With respect to EHR effectiveness, a clinical intelligence reporting system without ability to trigger automated workflow is a passive reporting system (in which reports must be handed to staff for disposition, “Please put a note in each patient’s chart so that the next time they have an appointment…”). A more active clinical intelligence reporting system feeds directly back to a workflow engine and process definitions to automatically perform useful tasks. With respect to EHR efficiency, even if a process model has an obvious flaw, there is no way to consistently and automatically deflect behavior at critical process junctures in order to improve throughput and throughput time.

In summary, compared to process-aware EHR workflow management, or business process, management systems, traditional EHRs (1) do not track tasks at a sufficiently high degree of resolution, (2) do not distinguish among the large number of possibly useful time-stamped events, and (3) have no means for process model insights to drive improvement at the point-of-care through automated workflow.

The next four advantages of process-aware EHR BPM systems (or process-aware clinical groupware, if you will) are generally acknowledged advantages of BPM systems over non-BPM systems.

  • EHR BPM systems can be used to model and understand workflow,
  • coordinate patient care tasks handoffs,
  • monitor task execution in real time, and
  • systematically improve clinical workflow and outcomes.

The next step in the evolution of ambulatory EMRs is squarely at the intersection between two great software industries: electronic health record systems and workflow management/business process management systems. The hybrid EMR workflow systems that result will be more usable and more systematically optimizable than traditional EMRs with respect to user satisfaction, clinical performance, patient satisfaction, and practice profitability.

Thank you!

Epilogue: EMRs / EHRs Need to Perceive and Respond to Clinical Threats and Opportunities in Real-Time

Referring back to van der Aalst’s quote about business intelligence, he clearly considers process mining to be an example of sophisticated operational business intelligence. Just as clearly, therefore, both the numerator (PROCARE) and the denominator (PROCESS) in the EMR / EHR productivity ratio are examples of clinical business intelligence/clinical intelligence. The difference between this formulation of EMR / EHR-mediated business intelligence and most other formulations is the important role of an executable process model yoked to clinical complex event functionality. Without both capabilities–to both perceive and react to clinical threats and opportunities in real-time–transparently and under flexible human control, EMRs / EHRs will not become capable of automatable closed-loop patient care or its systematic improvement.

Addendum: Glossary of EMR / EHR Workflow Terminology

Phrase Definition Medical Example
Work Item Task to perform Vitals signs awaiting performance during a patient encounter
Workflow/ Process Definition Description of a process detailed enough to drive EMR / EHR behavior. van der Aalst refers to this as a formal process model, that is, one that can executed. Get the Patient, Take Vitals and a Chief Complaint, Review Allergies, Review Medications, Review of Systems, Examination Screen, Evaluation and Management, Billing Approval
Worklist List of tasks to perform A nurse’s To-Do list
Case Particular application of a EMR / EHR workflow management system / business process management suite A particular patient’s encounter managed by EMR / EHR workflow management system /business process management system
Process Order (though not necessarily sequence) of tasks to be performed and resource requirements A Well Child pediatric visit
Resource Something that accomplishes tasks (often a user) A physician, nurse, technician
Role Set of related skills accomplished by a resource The role of nurse or physician
Routing Types of routing include sequential, parallel, conditional, or iterative task execution Routing a recording to a transcriptionist and the report back to the physician
Task Unit of work carried out by a resource Obtain vital signs
Trigger An event that changes a work item into an activity Starting to accomplish the task of responding to a phone message by selecting a To-Do list item
Workflow A process and its cases, resources, and triggers The tasks and people involved in accomplishing a patient encounter
Workflow/ Process Definition Editor User application or interface for creating workflow/ process definitions An ordered picklist or flowchart diagram representing Get the Patient, Take Vitals and a Chief Complaint, Review Allergies, Review Medications, Review of Systems, Examination Screen, Evaluation and Management, Billing Approval
Activity Performance of a task Obtain vital signs within a patient encounter
Posted in clinical-intelligence | Tagged , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , | Leave a comment

NIST EMR / EHR Usability Workshop: A Highly Annotated Tweetstream

Short link: http://ehr.bz/73

A.S. Check out the EHR.BZ REPORT on Workflow, Usability and Productivity.

June 7th I attended the well-run and thought-provoking NIST EHR Usability workshop in Gaithersburg, Maryland (http://ehr.bz/nistux). I tweeted my notes. I’ve tried this before with mixed results (Dr. G’s Workflow Management EMR presentation at HIMSS, Tweeting Live from HIMSS, Tweeting Live from Process.gov).

I decided to try again!

So I…

  • Tweeted my “notes.”
  • Copied the tweets from Twitter.
  • Reversed their order (”tail -r tweets.txt” for you UNIX folks).
  • Pasted them into Wordpress.
  • Edited for readiblity.
  • Added more thoughts and material such as links and specific slides.

This is a long (and sometimes meandering and ruminative) document, so here is a table of contents if you’d rather proceed directly to one or other presentation topic.

I should make a disclaimer. I am biased. Lack of EMR usability has more do with document-based versus process-based approaches to building EMRs than it has with government versus industry approaches to driving EMR innovation. Most current EMRs rely on structured documents and unstructured processes. Until EMR users are given tools to model, execute, monitor and systematically improve the standardizable processes generating the structured documents (and potentially redirect those processes on the fly), lack of usability will continue to slow EMR adoption. (See EHR/EMR Usability: Natural, Consistent, Relevant, Supportive, Flexible Workflow) Whether government or industry or both accomplish this, and how, is an important debate, but more about ideology than what is technically retarding EMR usability.

Let the tweets begin!

c_wb Looking forward to the NIST EHR / EMR Usability workshop tomorrow

nist-ehr-emr-usuability-screen

c_wb I’ve arrived at the NIST EMR / EHR usability workshop, my gadgets are charged, I have my coffee, presentations are about to begin …

By the way, I read NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records to prepare

c_wb Today I’m attending Measuring, Evaluating & Improving Usability of EHRs EMRs Workshop NIST Gaithersburg, MD

A Community-Building Workshop: Measuring, Evaluating and Improving the Usability of Electronic Health Records, June 7th, 2011, Gaithersburg, MD

Original announcement and agenda:

http://ehr.bz/nistux (cached announcement and agenda)

Opening remarks A (cached)

Welcome to: A Community-Building Workshop: Measuring, Evaluating and Improving the Usability of Electronic Health Records

Opening remarks B (cached)

A Community-Building Workshop: Measuring, Evaluating and Improving the Usability of Electronic Health Records

c_wb NIST: innovation/competitiveness by advancing measurement/standards/tech 2 enhance econ security/quality of life

Cramming NIST’s mission statement into 140 tweetable characters resulting in something a bit terse, so…

NIST’s mission:

“To promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life.”

Why is Improved Usability of EHRs Important?

c_wb Starting: Intro 2 workshop

c_wb Usability: strong/direct relationshp w/ clinical productivity, error rate, user fatigue & satisfaction

Also a bit terse, but the cool thing is that these were great search terms to find the original quote

http://www.google.com/search?q=usability+strong+direct+relationship+clinical+productivity+error+rate+user+fatigue+satisfaction

“usability is one of the major factors—possibly the most important factor—hindering widespread adoption of EMRs. Usability has a strong, often direct relationship with clinical productivity, error rate, user fatigue and user satisfaction–critical factors for EMR adoption. Clinicians lose productivity during the training days and for months afterward as they adapt to the new tools and workflow. Some productivity losses are sustained, mostly due to longer time needed for encounter documentation in complex patients” (Defining and Testing EHR Usability)

c_wb ONC goals include EHR usability transparency

There were several other goals regarding usability, but I wasn’t a quick enough so, from presentation slides distributed later…

ONC’s Goals

  • Improve transparency on usability
  • Promote technology that fully supports care
  • Identify and address potential safety issues, but also factors that affect efficiency and effectiveness, user satisfaction, etc.
  • Enable constructive innovation

c_wb #ehrusability is the hashtag for NIST EHR usability workshop

Introduction: The Promise of EMRs

Presentation PDF / Cached

@ Starting: Introduction: Why is Improved Usability of EHRs Important?

c_wb one pediatric growth chart takes 8 clicks to get to open…

Wish I had a nickel for every time “click” was mentioned at this workshop. It also on the minds of folks not attending the workshop (pro and con: here, here, here, here, here and here are representative).

The number of clicks is a fairly superficial measure of (lack of) usability; three easy, fast, automatic clicks is can be more usable than one long tortuous click. However, cognitive effort is harder to measure then physical user events, so it’s a convenient surrogate. This point was made several times by members of the audience. I suspect that most people attending the workshop agree, but that “click” is shorthand for a wide variety of data and order entry costs: time (to target), effort (to find target), and error (if missed).

Perhaps entering data into an EHR/EMR should resemble playing a musical instrument. A pianist can effortlessly click on a lot of keys in a very short period of time.

c_wb poor usability can cause medical errors. We need to figure out how to test, identify and prevent them

The workshop focussed more on error minimization and patient safety than other possible usability goals such as speed, productivity and user satisfaction, though each of these topics were indeed represented in presentations and discussion. The following slide is from a later presentation. Notice that “critical errors that impact patient safety” and “errors and failures” are highlighted in red.

summative-ehr-usability-test-plan

Overview of Current Programs for Improving EHR Usability (at NIST)

c_wb Starting: Overview of Current Programs for Improving EHR Usability

NIST EHR Usability Program

Presentation PDF / Cached

c_wb Starting: NIST EHR Usability Program

c_wb NIST funds EMR usability research, such as Human Factors Guidance to Prevent Healthcare Disparities with EHRs http://1.usa.gov/lXcPYT

Resources for the Regional Extension Centers

Presentation PDF / Cached

c_wb Starting: Resources for the Regional Extension Centers Usability of EMRs

c_wb accessibility and usability of data was a theme of 2010 HITPC Meaningful Use hearings

Themes that Emerged from 2010 HITPC/ Meaningful Use Workgroup Hearings

  • Achieve 4 Es: engage, educate, empower, and enable
  • Meet needs of diverse population
  • Accessibility/usability of data – Need for mobile apps(esp.for vulnerable populations) – Contextualizing information – Multiple languages – Compatible with assistivetechnologies
  • Patient-provider secure messaging
  • Incorporate patient-generated data into EHR
  • Provide ample training on all functionalities

c_wb regional extension center: need training to get to 4 clicks instead of 15

No kidding. I know of two EHRs that both allow a physician users to approve a refill request; one takes four clicks :) and the other takes 25 :( . Yeah, I know, now I seem to be contradicting myself. But in this case it’s four fast automatic effortless clicks versus 25 slow laborious clicks.

c_wb RECs to help train to “vendor specific workflow”

Interesting phrase: “vendor specific workflow”

The slide says “Work directly with providers on a regular basis. Use the HITRC for usability tools and resources. More participation with RECs; more participation with RECs; Webinars, Training, Response; Vendor specific workflow.(my emphasis)

Given that a majority of the total cost of owning an EHR / EMR can associated with workflow customization issues (I do have reference on this, but not immediately at hand), it’ll be interesting to observe how RECs cope with this particularly important and problematic aspect of implementing traditional EHRs.

See my Mirror, Mirror, On the Wall, Which EMR is Least Traditional Of All? for some tongue-in-cheek comments on how darned difficult it is to cope with the relatively uncustomizable workflow of traditional EMRs / EHRs.

Evidence-based Usability Guidelines for Promoting Safety and Efficacy

Presentation PDF / Cached

c_wb Starting: Evidence-based Usability Guidelines 4 Promoting Safety & Efficacy

c_wb took sixty yrs to get to standardized time zones, can we get 2 usability faster?

c_wb lessons other industries: apple microsoft android “usability guidelines are powerful”

Usability Guidelines are Powerful

  • Apple, Microsoft, Android, …
  • NASA, FAA, DoD, …
  • HFES, WWW-Consortium, …
  • RAISE QUALITY
    • Promote Consistency
    • User Performance Speed Development
    • Programmer Productivity Reduce Errors
    • Organization Reputation

Interesting though to contrast the slide above and the slide below (from a later presentation)!

if-the-gov2

This second slide shouldn’t be taken out of context though. It was presented as a typical criticism of government-led EMR usability test that must be acknowledge and addressed.

c_wb for a good model of research-based web design and usability guidelines see http://usability.gov

c_wb HIT usability researchers find it difficult to obtain user docs, screenshots, demos from EMR EHR vendors for research purposes

Yeah, I know what he is talking about. When I was in academia it was very difficult to get cooperation from EHR / EMR vendors to help train students let alone conduct research. When I worked for an EHR / EMR vendor I suddenly had access to a plethora of such materials, butdid not have time (or charge) to train students (as opposed to users) or conduct and present research. I started this blog in an attempt to bridge this gap and resolve (to my satisfaction) some of these inherent contradictions. I wrote about this in Walking the Fine Line between Marketing and Education.

TURF – A Unified Framework for Defining, Evaluating, Measuring, and Designing EHR Usability

Presentation PDF / Cached

c_wb Starting: TURF: Unified Framework 4 Defining, Evaluating, Measuring & Designing EHR usability

TURF = UFuRT? Below is a diagram from 2007 paper…

ufurt-diagram1

UFuRT: A Work-Centered Framework and Process for Design and Evaluation of Information Systems (cached)

c_wb TURF = task + user + function + representation

Note the common components between the previous and subsequent slides: task, users, function, representation, intrinsic difficulty/complexity, extrinsic usability/difficulty

turf-framework1

c_wb “function saturation”

function-saturation1

c_wb “overhead in designer model” less is better

overhead-designer-model

c_wb measuring usability: learnability, efficiency, error prevention and recovery

How to Measure Usableness?

  • Learnability
    • trials to reach a certain performance level
    • items that need to be memorized
    • sequences of steps that need to be memorized
    • Etc.
  • Efficiency
    • Time on task
    • Task steps
    • Task Success
    • Mental effort
  • Error Prevention and Recovery
    • Error occurrence rate
    • Error recovery rate

c_wb efficiency measure: time on task in seconds (5 minutes on CPOE )

time-on-task2

c_wb number of steps, mental effort (thru cognitive modeling)

turf-action2

For an overview of cognitive modeling see Toward Cognitive Modeling for Predicting Usability

c_wb TURF in action: reduced steps from 91 to 14

Note that if an EHR does not have some means for its users to modify its workflow then this means going back to the programmers to “unfreeze”, change, and then “refreeze” its workflow. I addressed this in my post Litmus Test for Detecting Frozen EHR Workflow.

c_wb in summary: usability is definable, measurable, doable

By the way, usability approaches that emphasize an entire team as the user instead of a single user before a single display (groupware instead of “singleware”) are complementary with workflow-oriented approaches such as TURF/UFuRT. This is why, in 2004 I wrote the following about the importance of workflow management systems (today business process management systems and suites) to EHR / EMR usability:

Workflow Management and EHR Usability

EHR workflow management concepts mesh with research initiatives to improve EHR usability. For example, Human-Centered Distributed Information Design [6] (there applied to EHR usability issues) distinguishes four levels of distributed analysis: user, function, task, and representation, which correspond well to workflow management architectural distinctions.

[1] Distributed user analysis can be interpreted to include allocation of tasks, relationship between roles, and task-related messaging, all of which are important workflow management concepts.

[2] Distributed function analysis involves high-level relationships among users and system resources. From a workflow management perspective, this includes who reports to whom and who is allowed to accomplish what.

[3] Distributed task analysis roughly corresponds to the creation of process definitions that in turn drive EHR behavior: What is to be accomplished by whom, in what order, and what needs to happen automatically.

[4] Distributed representational analysis corresponds to something that workflow management systems intentionally do not address. Workflow management system design tends to be agnostic about how information is displayed to, transformed, or collected from the user. The underlying workflow engine is intended to be a general purpose tool that can be used to sequentially launch whatever screen or initiate whatever behind the scenes action that the implementer of the workflow system deems most apt as part of workflow analysis and design. However, by remaining orthogonal to the choice of screen, by not mandating or hard coding, the designer/implementer is free to bring to bear the powers of representational analysis to use whatever screen and attendant representation is most appropriate.

Thus, workflow management concepts are consistent with human-centered distributed information design, an important emerging area of medical informatics research. “Task-specific, context-sensitive, and event-related displays are basic elements for implementing HCC [human-centered computing] systems,” (p. 46 [6]) and they are the basic elements provided by EHR workflow management systems, too.” (my emphases)

Overview of HIMSS Usability Taskforce Initiatives

Presentation PDF / Cached

c_wb Starting: Overview of HIMSS Usability Taskforce Initiatives

c_wb promotes industry education about usability principles & measurement

c_wb usability maturity model white paper

Defining and Testing EMR. Usability: Principles and Proposed Methods of EMR Usability Evaluation

c_wb subgroups include HIMSS Celltop Design Workgroup: smartphones, handheld design tenets, partnership NIH & HIMSS

c_wb next steps: maturity model checklists, attention education, more white papers

Safety, Usability and User Interface Standards in the NHS (Virtual Presentation)

Presentation PDF / Cached

c_wb starting virtual presentation from britain on NHS usability program

c_wb CUI = Common User Interface, speaker has background in aviation human factors (as have I, sort of) which is advanced over healthIT with respect to usability

c_wb common user interface example: patient banner eg “display comma after family name”

c_wb NHS lessons learned: usability + safety strongest message, considerable common user interface docs online at portal located at http://www.cui.nhs.uk

c_wb “there’s some very poor product out there” “woefully clunky”

Amen! Question: what is the best way to make EHRs generally less clunky?

  • Government?
  • Industry?
  • Both?
  • How?

Even bigger question: Is above the right question?

Human Factors Approaches to Improving EHR Usability

Presentation PDF / Cached

c_wb Starting: Human Factors Approaches to Improving EHR Usability

c_wb amusing user videos “we only have 3 screens, we don’t need 500 buttons”

The actual quote was “We only have three screens–preop, intraop, postop–we don’t need five hundred buttons” How do I know? I used the Looxcie wearable camcoder to provide a resource to consult when I want the exact wording of something.

RT amalec NHS reps bemoan lack of visibility into ehrusability; no recourse after you sign the ehr contract

c_wb +1 RT @amalec: Super cool ehrusability lab in Canada with live video recorded simulation

c_wb challenges: generalizability, resource requirements, risk vs usability, comprehensiveness…

c_wb question from audience: from an CMIO, how to tap into your (the folks presenting at NIST EHR usability conference) usability expertise?

Answer: Join this EHR usability community that NIST is in the process of creating.

c_wb question: how to make sure requirements are done right?

c_wb answer to previous question from virtual NHS folks: usability needs to be brought up in contract phase, otherwise it is two late for it to influence requirements

c_wb member of panel: patient is also a user, an example is a patient identifying an error in chart when looking at the EMR screen

As an aside, I once suggested that pediatric EMR EHR user interfaces ought to, and eventually will, evolve to look like illustrated children’s books (as part of a large exercise to make pediatric offices more-and-more child friendly). Why? Because the pediatric patient is an EMR user too. EHR buttons ought to be big enough for not only the pediatrician to hit on the fly (respecting Fitts and Hicks laws of target acquisition) but for the sharp-eyed child to see. An EMR is not just [insert standard EHR definition here], it is a form of persuasive technology. Big buttons displaying soccer balls and report cards carrying the letter “A” ought to be part and parcel of pediatric EMR user interfaces.

soccer-ball-report-card

Cute representations of real and imaginary animals *ought* to scamper about an EMR UI, drawing in the child EHR user, not just making them less fearful, but entrancing and motivating them. Imagine a pediatrician clicking on a big animated button while saying “Come sit here and let’s see what the big blue bear thinks you should do about your cough!” EHRs, really usable EMRs, by Disney or Nickelodeon are in our future. Just not sure when–how soon or how long.

c_wb question: can NLP natural language processing help? docs wont tolerate lots of clicking

I took all the courses necessary for a degree in Computational Linguistics before switching into Intelligent Systems, including phonetics, phonology, morphology, syntax, semantics, pragmatics, and NLP I, II, and III, plus knowledge representation and NLG (natural language generation). The problem I have with replacing properly-managed structured data entry with speech recognition and natural language processing is this: a human, either the original speaker or someone else (perhaps some sort of post-editor) still needs to proof the string of linguistic tokens emitted by a speech recognition-based EMR user interface, or canonical representations of what those tokens mean.

Even if speech recognition is 99.5% correct, as long as non-automated proofing and post-editing is required, the fire-and-forget nature of clicking (or, preferably, touching) a picklist item, which does not require proofing or post-editing, is superior. And as for slowing down the physician user (the so-called “clickorrhea” I’ve occasionally tweeted about, again, if Fitts and Hicks laws are respected, structured data entry can actually be faster and more accurate than other data input modalities. Where speech recognition makes more sense, at this point in the evolution of the degree of intelligence possible by the technologies we can bring to bear on this problem, is in the mobile smartphone interface. However, here the context is severely constrained and the amount of required proofing and post editing is minimal (though still required).

By the way, computational linguistics is relevant to not just processing medical language generated by humans, but communication between EMR systems as well (especially at the higher, currently less considered, levels: pragmatics and discourse).

c_wb Answer: we’ll compare input modalities, including NLP Question: can NLP help?

RT ahier Following tweets from ehrusability workshop http://bit.ly/kexzxu

Hi Brian!

Collaboration and Consensus through Standards – The National Technology Transfer and Advancement Act

No Presentation PDF

c_wb Starting: Collaboration & Consensus thru Standards – National Technology Transfer & Advancement Act

c_wb by its very nature standards are a “collaboration” vehicles and “consensus” processes

c_wb NIST has a standard for creating standards (hmm, is there a standard for creating standards for creating standards?)

A Community Approach to EHR User Experience Measurement

Presentation PDF / Cached

c_wb Starting: Community Approach to EHR User Experience Measurement

c_wb classic tradeoff between focus vs participation: so “focussed collaboration” approach

From slide presentation speaker notes:

“Focused collaboration means that we’re engaging a broad array of stakeholders in the development process, but managing their work property to ensure the most efficient and effective process. What we don’t want to see is a high degree of focus (top down, heavy-handed government-driven process) with little participation from outside stakeholders; nor do we want a highly participatory process with no strong focus that results in a lot of great ideas, but no results. We’re seeking the best of both worlds.”

c_wb some say: if the gov created usability tests we wouldn’t have iphone or android

We saw this slide earlier

c_wb others say usability is indeed a science but that clinical workflow are “nuanced”

I presume (perhaps too much, though) that “nuanced” means not susceptible to formalization, formal analysis, or automatic execution in the service of greater data and order entry usability. I absolutely disagree.

From EMRs, EHRs, and Clinical Groupware Need to Solve “The BPM Problem”: Why Not Use BPM to Help Do So?

While it is true that most current traditional EMRs lack facility to model and execute workflow, future EMRs based on workflow management systems (WfMSs: workflow engines plus process definitions) and business process management technology (WfMS plus business intelligence, business activity monitoring, process mining, complex event processing, process simulation optimization, adaptive case management, etc.) inevitably will.

c_wb vendors don’t want government say what products are good or bad, we [vendors] don’t want the government to incent us into creating technology that stinks

c_wb BTW funny typo on slide, text was “…we want…” instead of “we [don't] want the government to incent us into creating technology that stinks”

Audience reaction to the type was good humored laughter. I certainly appreciated the lighter moment after hours of seriousness.

After the lighter moment, there is this: “So we have this classic tradeoff of the vendor community not wanting, for absolute valid reasons, the government to tell the world what products are good or bad, and the provider community saying, look we want to use electronic health records but we need to improve practice workflow, we want to make sure we have the right signals about the products we purchase”

Again, a central question appears to be: How to make EMRs, on the whole, “less clunky?

c_wb EHR EMR Usability professional: “sometimes i feel like an island in my own organization: the product is done, marketing brings me in, ‘the usability stinks’”

From my Intuitive vs. Intuitable EMRs, EHRs, and Clinical Groupware: Do We Need Smarter Users or Smarter User Interfaces?

“Usability can’t be “added” to EMRs, EHRs, or clinical groupware. It has to inform and influence the very first design decisions. And there are no more fundamental early design decisions than what paradigm to adopt and platform to use.

No matter how “intuitable,” EMRs without executable process models (necessary to perceive, reason, and act, and later systematically improve), cannot become fully active and helpful members of the patient care team. Wrong paradigm. Wrong platform.

Truly “intuitive” process-aware clinical groupware, on the other hand, has a brain, variously called a BPM, workflow, or process engine. This is the necessary platform for delivering context-aware intelligent user interfaces and user experience to the point of care. Right paradigm. Right platform.”

Community of Profession Model

Presentation PDF / Cached

Human Factors / Usability for Medical Devices at FDA: An Historical Perspective

Presentation PTT / Cached

c_wb Starting: Human Factors / Usability for medical devices at FDA: Historical Perspective

c_wb medical device milestones, 1976 bureau of MD, 1984 congress hearings on deaths

c_wb 1999 “To error is human” 98,000 deaths, 5th cause death, cost $29B

c_wb key: FDA review of pre-market submissions, outreach 2 industry

c_wb feedback: sales increase w/ satisfaction of customers for devices w attention to human factors driven by FDA

c_wb FDA concerns about device usability include relying on checklists and rating scales instead of systematic usability reviews

See my own comments about the use of checklists to evaluate EMR EHR usability

Building More Usable EHRs – Supporting the Needs of Developers “Focus on Faster & Usable Clinical Documentation”

Presentation PDF / Cached

c_wb Starting: Building More Usable EHRs: Supporting Developers “Focus on Clinical Documention” usability EMRs NIST

c_wb one study’s conclusion: “current EMRs frustrate physician collection of data”

current-ehrs-frustrate

Another study:

summary-conclusions

c_wb “the ‘we computerized the paper, so we can go paperless” fallacy” displays not as portable, flexible or well designed as paper

c_wb remove tension btwn free text vs structured documentation

This from this presentation’s key slide:

Recommendations From Literature

Remove tension between free text versus structured documentation

Clinical documentation needs to support both seamlessly

  • Usability and semantic interoperability go hand in hand
  • Refuse systems that do not deliver both
  • Remove tension between clinician/physician documentation as a billing vehicle and as a clinical documentation tool
  • Improved data input and richness of documentation can coexist if you design the system properly
  • Usability is perhaps more crucial than interoperability
  • The question of interoperability will be unresolved if clinicians fail toaccurately record the data

c_wb improved data input & richness of clinical documentation can coexist

Also the point of my rejoinder to the many criticisms of structured EMR EHR data entry

Developers: Supporting the Needs of Patients

No Presentation PDF (no slides were used)

c_wb Starting: Developers: Supporting Needs of Patients Usability of EHRs EMR

c_wb among users: physicians have the highest standards but also lowest tolerance (hmm, not bad, just a fact)

In fact my experience has been that part of the difficulty in developing usable EMRs has been creating an EHR that is useful and usable enough to physicians, in spite of its flaws and their high standards, to sustain physician engagement in the necessary process of improving EMR usefulness and usability.

c_wb As soon as you talk about certification your are talking about an idealized model EMR vendor customers are skeptical

c_wb “i don’t believe the price of adding government [to this mix] will be rewarded w/ better outcomes”

Again consulting my Looxcie:

“What you are saying is that your design is meaningful for every purchaser out there and your standards, your metrics, your way of saying yes we’ve have these numbers and tests are going to translate into provider happiness. As I said, I challenge the assertion that the industry has failed but I find it even more surprising the assertion we can’t do better. We think there is a role for usability in healthcare because true usability has to start out with the tasks the users have to provide, or have to perform everyday, it’s their requirements. So let’s start to look at some of those requirements that they have for meaningful use right now and figure out the way we can make those processes more efficient and more usable while still maintaining their usefulness. I think ultimately then what we need to do is to first of all provide support to those organizations like ISMT [???] and give the purchasers the tools that they need. Because we have clinicians who demand quality based on markets that have to bow to those demands in order to survive. I don’t believe the price of adding government into this dynamic will be rewarded with better outcomes. Thank you…”

Educate, Motivate, and Improve: In Favor of Inspecting and Rating UCD/Usability Processes

Presentation PDF / Cached

c_wb Starting: Educate, Motivate & Improve: In Favor of Inspecting/Rating UCD/Usability Processes

c_wb +1 “who is the user? it’s a cooperative *group* of users” [that is the user, not an individual user] #ehrusability #EMR clinical #groupware

I absolutely agree with this point. Focussing on the individual user in front of a single screen will ultimately be counterproductive. I tagged this tweet with the hashtag #groupware because I think that the phrase (and movement) associated with “clinical groupware” has this particularly right. Clinical Groupware, Care Coordination, and EMR Workflow Systems: Key Ideas provides an overview of this history of groupware as it relates to workflow (and getting workflow right is so important to usability, and, in my view, considerably misunderstood)

c_wb vendors are worried they will spend more time getting certified instead of improving usability

EMR certification does have the danger of running afoul of the famous software development “Iron Triangle”. Given the same level of resources a software product can increase one or two of 1) features, 2) quality, 3) time to market only at the expense of one of two of the remaining goals. By diverting resources to meet certification not only can one set of features (those required for certification) crowd out other features, but also constitute an obstacle to quickly getting a stable and usable EHR release to market. Usability certification may, ironically, potentially forestall the very usability innovation it seeks to advance.

c_wb summary: need incremental improvement & competitive dynamic to invent new solutions

c_wb Love “Focus on the ‘Five Big Tasks’” RT @healthfinch “War on EMR usability” Should gov get involved? post.ly/2Aagh

c_wb Agree RT @MetaMetaThinker @c_wb we should continue to automate where it makes sense and let providers focus on practicing medicine

One of the fun bonuses of tweeting notes live from a presentation is that folks (in the audience or not present) will occasionally publicly (or privately, by direct message) chime in. I’ve previously written about this Twitter-mediated conference back channel.

Usability is the Key to Stimulating EHR Innovation and Adoption

Presentation PDF / Cached

c_wb Starting: Usability is the Key 2 Stimulating EHR Innovation & Adoption

c_wb Usability: extent EMR EHR can B used by specified/S users 2 achieve S goals w/effectiveness, efficiency & satisfaction in S context NIST

usability-definition

It’s an interesting exercise to extend this usability definition to a clinical groupware approach to EMRs:

“Great definition…but it just seems so, well, “singleware-ish.” Clinical groupware needs a less abstract definition of usability that is more direct about groupware’s unique usability issues (see the sixth, seventh, eighth, ninth, and tenth quotes from my Clinical Groupware…Key Ideas post). How about:

Clinical groupware usability is…

“The extent to which clinical groupware can be used by specified teams of users to coordinate activity and achieve specified collections of goals with overall effectiveness, efficiency and satisfaction in specified contexts of use.”

[Those links to the sixth through tenth quotes about groupware usability? I pull that material into this post after its conclusion below.]

c_wb meaningful use has emphasized functionality over usability, 100 clicks 2 doc’mt smoking status!

Surely an exaggeration, but point well taken.

c_wb 2 types of usability: individual usability vs workflow usability, how can we reduce physician work/steps/task?

two-types-ehr-usability

Absolutely agree. Also see material on clinical groupware usability in postscript.

c_wb Government should do 3 things: define usability measures, promote open platforms and APIs, national EHR usability database

c_wb compare users to find “positive deviance” find people doing well and find out what they are doing to learn from it ehrusability

Wrote about the potential to do something similar relative to comparing medical practice productivity measures and explaining them in terms of differences in workflow.

c_wb individual vs workflow usability: think more about how usability affects team-based care ehrusability NIST emr clinical groupware

Again, incredibly important insight, this! In my post The Cognitive Science Behind EMR Usability Checklists I wrote:

“[T]here is no guarantee that optimizing single user usability won’t in suboptimize higher level global system goals. So I prefer a definition of usability that emphasizes team, rather than individual, performance.”

Again, see postscript.

Promoting Usability in Healthcare Organizations with a New Usability Maturity Model

Presentation PDF / Cached

c_wb Starting: Promoting Usability in Healthcare Organizations with a New Usability Maturity Model

c_wb Speaker defined usability as “sexier products using a process”

Now that is a sexy definition of usability!

c_wb Usability Maturity Model: unregulated, preliminary, implemented, integrated, strategic

usability-maturity

While we are on the topic of maturity models I’d like to mention two others 1) the Software Capability Maturity Model developed at CMU’s Software Engineering Institute (the original MM that has inspired other MMs, and 2) the Business Process Management BPM Maturity Model.

Relative to the Capability Maturity Model, Wikipedia says more generally:

“A maturity model can be viewed as a set of structured levels that describe how well the behaviors, practices and processes of an organization can reliably and sustainably produce required outcomes. A maturity model may provide, for example :

  • a place to start
  • the benefit of a community’s prior experiences
  • a common language and a shared vision
  • a framework for prioritizing actions.
  • a way to define what improvement means for your organization.

A maturity model can be used as a benchmark for comparison and as an aid to understanding - for example, for comparative assessment of different organizations where there is something in common that can be used as a basis for comparison. In the case of the CMM, for example, the basis for comparison would be the organizations’ software development processes.”

Relative to a BPM Maturity Model here is a similar five level chart from “Towards a Business Process Management Maturity Model”

bpm-mm

Why do I include these maturity models? The Software CMM is the grandaddy of maturity models and therefore important context for considering any EHR usability maturity model. And the healthcare industry has the lowest BPM maturity of any major industry segment. At least some of the (lack of) usability issues afflicting EHRs is that they are not process-aware in the sense as workflow management systems and business process management suites.

c_wb Launching usability: wake-up calls, individual infiltration, internal champion, external experts

Guidelines for Improving Usability: Proposed EHR Usability Evaluation Protocol

Presentation PDF / Cached

Presentation PDF / Cached

c_wb Starting: Guidelines for Improving Usability: Proposed EHR Usability Evaluation Protocol

c_wb EHR Usability Protocol focuses on *most critical* issues first, others later

Presumably preventing errors threatening patient safety

From the slide:

EHR Usability Protocol (EUP)

  • The EUP provides a methodology for identifying and eliminating risks to patients due to poor user interface design.
  • This focus is the foundation of many existing, validated protocols for evaluating the usability of systems where safety is a critical component of user operation.
  • EUP focuses on the most critical issues first.
  • Other dimensions of usability are important.(my emphasis)

My interpretation is that while speed, productivity, profitability, satisfaction, engagement, etc. are important and may eventually be tackled, that patient safety is paramount and therefore will drive creation of the EHR Usability Protocol.

c_wb objectives: eliminate “never events”, ID/prevent critical use errors…

c_wb objectives:… ID areas for improvement and report in Common Industry Format CIF

c_wb EUP does not describe “look & feel” & therefore will not discourage innovation

eup-is-not

Is it possible to improve usability without changing the look and feel of a user interface? From the Wikipedia entry on “look and feel”:

“Look and feel in operating system user interfaces serves two general purposes.

First, it provides branding, helping to identify a set of products from one company.

Second, it increases ease of use, since users will become familiar with how one product functions (looks, reads, etc.) and can translate their experience to other products with the same look and feel.” (my emphasis)

c_wb Errors of commission vs omission (harder to detect)

c_wb Never events: commission, ommission wrong med, [2 more didn't get!]

Here’s the complete list (from the slide):

“Never Events

The proposed categories of never events are:

  • Wrong patient action of commission event: Actions with potentially fatal consequences are performed for one patient that were intended for another patient because two patient identifiers were not displayed in an area of the screen that is visible without scrolling
  • Wrong patient action of omission event: A patient is not informed of the need for treatment because the wrong patient’s name was displayed on clinical data for another patient
  • Wrong medication event: A patient receives the wrong medication, dose, or route because the displayed information was not accurate or required viewing information on hidden screens to be accurate
  • Delay in care event: A patient should not receive a life-threatening delay in the provision of critical care activities due to design decisions made for administrative, billing, or security objectives
  • Unintended care event: A patient should not receive unintended care actions due to actions taken to test software, train users, or demonstrate software to potential customers.”

c_wb More errors: sequence & timing errors (both subclasses of errors of commission)

c_wb Quant vs qual/attitude vs behavior: summative usability testing is quantitative results from behaviors

usability-eval-context1(red circle added)

c_wb “discount usability testing” not repeatable across multiple designs

summative-ehr-usability-test-plan

(red highlights in original)

Had to look that phrase up–Jakob Nielson popularized the phrase and idea of discount usability engineering, about which he wrote:

The Discount Usability Engineering Approach

“Usability specialists will often propose using the best possible methodology. Indeed, this is what they have been trained to do in most universities. Unfortunately, it seems that “le mieux est l’ennemi du bien” (the best is the enemy of the good) [Voltaire 1764] to the extent that insisting on using only the best methods may result in having no methods used at all. Therefore, I will focus on achieving “the good” with respect to having some usability engineering work performed, even though the methods needed to achieve this result are definitely not “the best” method and will not give perfect results.

It will be easy for the knowledgable reader to put down the methods proposed here with various well-known counter-examples showing important usability aspects that will be missed under certain circumstances. Some of these counter-examples are no doubt true and I do agree that better results can be achieved by applying more careful methodologies. But remember that such more careful methods are also more expensive — often in terms of money, and always in terms of required expertise (leading to the intimidation factor discussed above). Therefore, the simpler methods stand a much better chance of actually being used in practical design situations and they should therefore be viewed as a way of serving the user community.”

c_wb usability testing process: kickoff/discovery, preparation, data collection, analysis/reporting

c_wb key differences summative testing for EHRs: requires more moderators with greater expertise & more tasks mandated by meaningful use than non EMR testing

difference-ehr-usability-test-plan

(red highlights in original)

c_wb many are tasks tied to MU meaningful use criteria

c_wb test administrators will need advanced degrees in human factors and minimum 3 years experience

evaluators-ehr-usability

c_wb Next steps: protocol development, test protocol examples, data sheets, develop more specific tasks

Government Best Practices in System Usability: Brief History & Status

Presentation PDF / Cached

c_wb Starting: Government best practices in system usability: Brief history & status

c_wb Human factors in design of safety critical systems: Book: The Chapanis Chronicles: 50 years of HF Research, education & design

chapanis

Great to see the well-reviewed biography of aviation human factors/ergonomics researcher Alphonse Chapanis. I became peripherally aware of Chapanis when I took a course at the University of Illinois Institute of Aviation (home of U of I’s Human Factors and Ergonomics program). I’ve frequently cited his contemporary, Paul Fitts, relative to Fitts Law and the need for big buttons for EHRs. I even wrote a poem about him. :)

“The early educators in the field-Alex Williams, Al Chapanis, Paul Fitts, Ross McFarland, Len Mead, Lick Licklider, Neil Warren, John Lyman, Jack Adams, George Briggs, and Ernest McCormick-had in common a recognition of the importance of a multidisciplinary approach to aviation problems, and their students were so trained.” (The Adolescence of Aviation Psychology)

As we are currently in the adolescence of “EMR Psychology”, I think there is indeed a historical model in aviation psychology to inspire and guide us today.

c_wb Human factors was born 70 years ago in the aviation industry when planes were falling out of the skies

I enjoyed seeing connections drawn between the history of aviation human factors and EMR / EHR usability. For more on the subject see EHR/EMR Workflow System Usability–Roots in Aviation Human Factors.

helmet

c_wb Notable UI Incidences: NORAD false alarms, 3 Mile Island, Flight 965 Cali CA

c_wb NIST involved in refinery user interface: reorg control room operator info, reduced plant incidents to one third of previous

c_wb FAA Order 9550.08*: “human factors shall be systematically integrated…all FAA elements & activities

FAA order 9550.08*

“Human factors shall be systematically integrated into the planning and execution of the functions of all FAA elements and activities associated with system acquisitions and system operations. FAA endeavors shall emphasize human factors considerations to enhance system performance and capitalize upon the relative strengths of people and machines.”

c_wb Human factors @ Dept of Defense: Human System Integration/Manpower Personnel Integration (MANPRINT, I kid you not)

The Relationship between Health IT Usability and Patient Safety: Towards an EHR Usability Safety Framework

Presentation PDF / Cached

Presentation PDF / Cached

c_wb Starting: Relationshp btn HealthIT Usability/Patient Safety: 2wards EHR Usability Safety Framework

c_wb NIST working on Usability-Safety Framework max benefits 2 users & patients & min harm

safety-framework

I like this diagram and look forward to where it leads. What I like most is that it is “closed loop” and appears amenable to a “process-aware” approach (process design, implementation, enactment and diagnosis) to systematic optimization (minimization of error, maximization of safety).

c_wb Use Errors: Patient ID, mode, data accuracy, visibility, consistency, recall, feedback, data integrity

c_wb Evaluation indicators: workarounds, redundancies, burnout

c_wb Patient harm: never events, substandard care, morbidity, mortality

c_wb Risk Factors: severity, frequency, detectability, complexity (1-4 scale)

c_wb handoff & interruptions increase complexity

Better supporting handoffs and managing interruptions is key. See Interruptions, Usability, and Pediatric and Primary Care EMR Workflow on the subject.

Audience Questions/Comments During Technical Feedback on EUP (EHR Usability Protocol)

c_wb Now moving to Breakout sessions: (Red) Tech Feedback on EUP (Green) Building Collaborative Community 4 Improving Usability

I stayed for the Red technical feedback session

c_wb question: will testing occur in typically noisy & distracting environment of real EMR user?

Great question!

“Similar to an under-attack fighter, a busy airport control tower, or a hectic lunchtime restaurant, medical practices can be high cognitive load environments (especially during the flu season in primary care). All four require multitasking and prioritization in the face of interruption and distraction.” (EHR/EMR Workflow System Usability–Roots in Aviation Human Factors)

Going to be very interesting to see how EHR usability, relevant to real-life use of EHRs, will be measured.

c_wb comment: warning that dealing with test system versions, configurations, and seeding realistic patient data will be very difficult

Traditional EMRs are highly customizable (often based on table-driven development). However, they are still not customizable enough, especially when it comes to workflow (which will be necessary to improve EMR workflow from 91 to 14 steps, as mentioned in a presentation). As workflow engines executing user customizable process definitions become more prevalent, EMRs will become even more customizable. So, which set of EMR process definitions will be tested? Since users will likely change these definitions, how do EMR workflows “in vivo” get tested for usability? I suspect we are ultimately looking as more use of participant observation in the wild than summative testing in simulated environments. But then that is more qualitative than quantitative, which makes it difficult to apply to programs for certification of EMR / EHR usability.

c_wb question: are looking where in workflow error occurs? team setting, where in setting?

c_wb Answer: interface is built based on workflow, it is the interface that is tested

What does it mean to say an “interface is built based on workflow”? I think it means that someone analyzes the workflow and then writes software that fits the workflow. The problem is we (developers) aren’t very good at doing this. Is there an alternative? I think there is. Build EMRs with workflow engines executing process definitions so users don’t have to be programmers to change workflow. There are even so-called Design by Doing approaches that allow users to create their own workflows without having to deal with workflow editors. Instead of trying to make sure an EMR fits medical practice workflow before it is installed, give its users the tools to more easily change its workflow. But, then, how do you measure usability? Isn’t usability then really about the tools used to change EMR screens and workflows, not the resulting screens and workflows?

c_wb question: what about variability? comment: reducing usability to number of clicks problematic

Agree: See my comments above…

c_wb audience: need more detailed cognitive model of error besides omission/commission?

Might want to check out the Rouse Human Error Scheme (disclaimer: he was my Industrial Engineering advisor). I’ve not been able to find a relevant paper of his that is not behind a paywall, but a table appears at the end of Improving Human Factors in Marine Maintenance by Clive K. Bright BA, PhD and Simon P. Bell BSc, CEng. (cached)

error-table

c_wb audience: would need 2 test 100 users at least for each release, which are frequent

Summative testing does require a test design that averages over a number of similar users performing similar tasks in similar environments (in contrast with current certification scripts that rely on one user (perhaps pretend) performing each task once over the Internet. So presumably more work than current certifications.

summative-formative-table1

Formative and summative usability testing are compared here.

c_wb response: if user interface doesn’t change, don’t need to test, only need 15 test users per category

c_wb response: can still identify critical use errors with smaller groups even if not statistical significant

c_wb audience: different sites can run very different version of same EMR version, test each?

c_wb audience: most important causes of critical error may B in the variability between sites such as how they handle interruptions ehrusability

c_wb audience: 8 week release cycles, could be a lot of testing…

BTW: all of these questioners were thanked and their comments and questions appreciated, that was the purpose of this portion of the workshop

c_wb audience: can eye-specialty vendor adapt usability to their subspecialty workflow?

I recall this audience member commented that in order to obtain EMR certification that they had to add pediatric growth chart to their EHR even though it was of no possible use to any of their customers.

Response: likely should just concentrate those aspects of workflow that are relevant to them [tempted to go back to my Looxcie video to find the exact wording]

c_wb audience: where can we get credible test data? response: building community to provide this

c_wb audience: radiology oncology errors must be reported to state, 1/10000, pool size statistical too small

c_wb audience: why so difficult for gov & vendors to collaborate? response: Question not for this forum (ask ONC)

Believe this question came from a European representative in the audience…

c_wb @ re medicine where aviation was in 1940 < coincidence! > conf @ NIST this topic https://twitter.com/c_wb/status/78171827809173505

That it! There are tradeoffs when you tweet instead of write your notes. I’m not a fast typist so I miss stuff. On the other hand there is less transcription from difficult to decipher handwriting. Plus the tweets provide an electronic outline that begs for further electronic annotation. Another thing I like about tweeting notes is that I can retweet other folks in the audience (or sometimes not even in the audience but nonetheless following along) and interact with them.

Cheers!

P.S. Here is that material regarding clinical groupware usability that I promised earlier:

“Great definition…but it just seems so, well, “singleware-ish.” Clinical groupware needs a less abstract definition of usability that is more direct about groupware’s unique usability issues (see the sixth, seventh, eighth, ninth, and tenth quotes from my Clinical Groupware…Key Ideas post). How about:

Clinical groupware usability is…

“The extent to which clinical groupware can be used by specified teams of users to coordinate activity and achieve specified collections of goals with overall effectiveness, efficiency and satisfaction in specified contexts of use.”

“Distributed Cognition takes as its unit of analysis a complex cognitive system: collections of individuals and artifacts that participate in the performance of a task. The external structures exchanged by agents of complex cognitive systems comprise its “mental” state and unlike individual cognition, where mental states are inaccessible, these states are observable and available for direct analysis.”

“The Human Factors in Computing community has a…challenge [to] find ways to test and evaluate technological impacts on groups. It’s difficult enough to get meaningful results that take into account differences in experience and individual differences of users to their reactions to user interfaces. But at least it’s possible to get volunteers to sit down with word processing systems and spreadsheet programs for relatively self-contained tasks. It is more difficult to “stage” a realistic group-work setting in a lab and have volunteers use the system in a way that provides meaningful data. Methodologies for testing individual user interfaces don’t apply as well to group support systems. As a result, CSCW [Computer-Supported Cooperative Work] is looking more to anthropology to find methodologies for studying groups at work in their natural settings.”

“Until recently, most user interface research has focused on single-user systems. Groupware challenges researchers to broaden this perspective, to address the issues of human-user interaction with the context of multiuser or *group* interfaces. Since these interfaces are sensitive to such factors as group dynamics and organizational structure—factors not normally considered relevant to user interface design—it is vital that social scientists and end users play a role in the development of group interfaces.”

“Evaluating groupware ‘in the field’ is remarkably complex because of the number of people to observe at each site, the wide variability of group composition, and the range of environmental factors that play roles in determining acceptance”

“Five factors contributing to groupware failure…:

  1. Groupware applications often fail because they require that some people do additional work, and those people are not the ones who perceive a direct benefit from the use of the applications.
  2. Groupware may lead to activity that violates social taboos, threatens existing political structures, or otherwise demotivates users who are crucial to its success.
  3. Groupware may fail if it does not allow for a wide range of exception handling and improvisation that characterizes much group activity.
  4. We fail to learn from experience because these complex applications introduce insurmountable obstacles to meaningful, generalizable analysis and evaluation.
  5. The groupware development process fails because our intuitions are especially poor for multiuser applications.”

Follow me on Twitter at @c_wb.

Posted in usability | Leave a comment

Introducing the EHR.BZ REPORT on EMR & EHR Workflow, Usability, Safety & Productivity

Short link: http://ehr.bz/71

Check out the EHR.BZ REPORT on EMR / EHR Workflow, Usability, Safety and Productivity!

I’ve been using EHR.BZ as an EHR-themed URL shortener for tweeting about EMR / EHR workflow, usability and productivity from my @c_wb Twitter account. I decided to use EHR.BZ’s home page to archive select links about these topics. After looking at the usual free open source content management systems (Wordpress, Joomla! and Drupal) I decided to create my own. Since by some accounts the DRUDGE REPORT is one of the most usable websites around, I borrowed some ideas, wrote some PHP, and the EHR.BZ REPORT on EMR / EHR Workflow, Usability and Productivity resulted.

Got some other ideas about what I might do, so check back! Or forget about the EHR.BZ REPORT on EMR / EHR Workflow, Usability, Safety and Productivity and just follow me on Twitter: @c_wb.

Posted in usability | Leave a comment

Twitter 1 Blog 0: Haven’t Posted For a While, But You Can Find Me on Twitter

Short Link: http://j.mp/icCBOj

I haven’t posted here for a while. I’ve got a lots of ideas for future posts, plus drafts I need to get around to polishing and publishing. In the mean time, Twitter sure is fun and I’ve met some great folks there with similar interests.

Ironic that I started tweeting to get attract traffic to the blog (“A Twitter Holiday: Sun, Sand, Surf, Smartphones, Short URLs, Social MEdia, and EMR Workflow Systems”) but was myself attracted from my blog to Twitter. Perhaps the pendulum will eventually swing back.

In the meantime…

I tweet about basically the same stuff (plus a dumb joke once in a while). I hope you’ll follow me (mention this blog and I’ll be sure to follow you back). Or, if you don’t have an account, head over to Twitter and get one.

Cheers

Chuck

http://twitter.com/chuckwebster

@chuckwebster

Posted in EHR Workflow | Leave a comment