Evaluating your pilot study
Through this step we will consider the importance of demonstrating an improvement as a result of your innovation. If no improvement can be demonstrated, it is unlikely that your innovation will be adopted. We will help you to identify appropriate evaluation techniques for your own innovation pilot study.
Measuring the success of innovation is important to demonstrate the benefits for all stakeholders. This includes patients, hospital clinicians, family doctors, administrators and the wider healthcare community.
Measuring the success of your innovation requires careful planning and you should begin to think about how you measure success early in your project planning. A key aspect of being able to measure success is being able to establish a baseline performance for your service. The exact measures you will use, will depend on your exact aims and objectives but could commonly include, for examples, sample turnaround time in laboratory medicine or waiting list time in patient facing disciplines. In addition to establishing what baseline measures you will use, you should also establish ‘criteria for success’ for example, a 20% increase in sample turnaround time.
You should evaluate the success your innovation, during the pilot testing phase. Measuring success during the pilot phase is important as it also provides an opportunity to refine your project and also to consider the impact of any changes on other parts of your service. Once you have implemented your innovation proposal into routine clinical service, you should continue to measure the success of your innovation. For example, you may wish to evaluate your innovation over a range of time points using the same measures and techniques used to establish the service baseline. This will enable you to objectively quantify the impact of your innovation.
As alluded to in Step 3.9, you may wish to consider using a model for improvement which may briefly be summarised as what are you trying to achieve, how will you know your innovation is an improvement and what needs to change
The Plan, Do, Study, Act or PDSA Cycle is a convenient tool to help you do that. PDSA cycles are a structured approach for making small changes and then planning, implementing, testing and identifying further changes. Further, PDSA Cycles are a tried and tested method and can be used to incrementally to introduce innovative service improvements.
Plan: What exactly are you going to do?
Do: What and how?
Study: What were the results? What worked and what did not? Do you need to think again and think of other ideas?
Act: What changes are you going to make based on your findings?
Repeated PDSA cycles can be used to evaluate and continually re-evaluate the impact of your innovation.
In summary, in order to measure success, you should think early about what you are trying to achieve and what you would accept as your criterion for success. This will inform the measures you will use to establish your baseline and to measure the impact of your innovation. Below are some online resources which you may find helpful.
You may find the references below helpful in designing your evaluation:
Thinking about the discussions so far, and what you now know about evaluating your pilot study, could your pilot require any refining? In Step 3.11 we will consider refining your study/proposal.