Skip to 0 minutes and 11 seconds Grading is the central process in a diabetic retinopathy screening programme. A high quality grading service requires competent grading staff with appropriate training and accreditation, manages staff workloads to ensure timely grading and decision making, maximises grading accuracy through quality assurance and ensures grading takes place in a suitable environment. Who takes on the task of grading retinal images varies across health systems. From ophthalmologists to optometrists or specially trained nonophthalmic graders. Following a grading pathway involves examining the fundus image, grading the pathology along an agreed protocol, deciding to refer for further assessment or return the patient for recall based on an agreed screening interval and undertaking surveillance for ungradeable images and examining the patients appropriately.
Skip to 1 minute and 9 seconds Grader training. All graders must receive specialist training in grading and decision making based on the local screening protocol. Ideally the training should be accredited managed as a recognised qualification. There are international training programmes, but every local programme should put together a training schedule to enable graders to achieve the required standards. [So this is…] Trainee graders should spend time on a one to one basis with a senior grader or ophthalmologist to review images for diabetic retinopathy and differentiate the levels of pathology seen. Performance of trained graders should be monitored regularly for their accuracy.
Skip to 1 minute and 50 seconds For example, in the English diabetic retinopathy screening programme graders are required to take an online monthly test and achieve sensitivity and specificity scores above 80% to retain their accreditation. Managing graders’ workload. Screening generates a large workload. Where possible a team approach should be taken to manage this. At least two levels of graders should be used. The basic grader differentiates cases as having no pathology or pathology. And identifies the level of pathology seen, The senior grader confirms the level of pathology and makes a referral decision.
Skip to 2 minutes and 29 seconds All graders are supervised by a consultant ophthalmologist who provides regular training and feedback. Each grader has a responsibility to ensure that the images are graded in a standardised manner and make identifications and decisions according to the local protocol. Let’s look at an example of graders’ workload to see how it works in practice. A DR screening programme has 10,000 patients in its database. The local protocol requires that each patient is screened annually. This means 200 people will need to be screened each week over 50 weeks of the year. Each patient needs a minimum of four images at each screening. This is called an image set.
Skip to 3 minutes and 9 seconds Over a year the centre ‘s basic grader will see the image sets of all 10,000 patients. Of these, 60% are likely to have no visible retinal changes. That is 6,000 patients’ image sets show no pathology. These patients are reassured and advised to come back the following year. The remaining 4,000 patients have image sets which show some abnormality, either pathology or being ungradeable. These are sent to the senior grader. In addition for quality assurance 10%, or 600 of the image sets, where no abnormality is found are also sent to the senior grader. The senior grader examines these, confirms the pathology, and makes a referral decision.
Skip to 3 minutes and 52 seconds Finally, the programme ‘s clinical lead should also review about 10%, or 400, of the 4,000 image sets which show some pathology. It’s essential to plan and manage this process and workload for both the patients and the team.
Skip to 4 minutes and 9 seconds Charlotte Wallis [Senior grader, Dorset
Skip to 4 minutes and 12 seconds diabetic eye screening programme]: Within my programme I monitor grading cues daily to decide where I need to place my graders. Whether they need to be sat looking at the second full disease grading or they need to be looking at the ROG grading. And also in addition to that when my screeners are out taking the photographs they would triage those images. So if they see anything particularly worrying or urgent its triaged as red. If there’s anything that they feel is soon and against our criteria for soon they will label that as amber. And anything routine that’s not worrying will go as a green.
Skip to 4 minutes and 48 seconds And through monitoring these cues and triaging appropriately we should be able to make sure that all our patients are graded in a timely manner. Because any delay in grading patients with proliferative disease could have an effect on their eyesight. Quality assurance of grading should underpin every DR screening programme. Accurate timely grading and correct patient referral decisions enable the DR screening programme to offer good quality and value for both patient and the programme.
Skip to 5 minutes and 20 seconds Indicators used to monitor graders’ workload include number of patients screened by the programme per month. This tracks the programme output.
Skip to 5 minutes and 31 seconds Number of patient image sets graded per grader per month. This tracks each grader’s output. Number of ungradeable images requiring surveillance per month. This tracks the quality of the screening method being used.
Skip to 5 minutes and 47 seconds Delivering quality assurance in a DR screening service requires adherence to local protocols and ongoing training and supervision of the grading team.
Skip to 5 minutes and 58 seconds Charlotte: So we regularly manage our quality assurance through the online national test and train and the graders have to score above 80%; otherwise, they get put onto a recovery test set. So I will monitor that very carefully every month to make sure that my graders do hit that target or above ideally. I will also monitor their sensitivity and specificity on their live grading also. And they also have 10% of their negatives checked again to make sure they are indeed negative. So we will review that also. And then our screeners have their performance monitored by their unassessable results because having a high level of unassessables can indicate that the screener needs more training. Requirements for good grading environment.
Skip to 6 minutes and 53 seconds Grading is an intensive activity and ideally graders should work in a space that is quiet, well ventilated and sufficiently dark to enable concentration. This is not always possible in optometry practices and should be a consideration when using optometrists to grade All grading should take place on adequately sized computer screens which meet the specifications for grading. Graders should have regular breaks built into their day to reduce tiredness and loss of attention. A short break once an hour away from the screen is a good idea. Protocols and standardisation are key for quality assurance. All graders should work to an agreed local protocol with set standards and this should be regularly reviewed to allow for improvements.
Skip to 7 minutes and 38 seconds In summary, grading is an intensive activity that requires a trained workforce, graders need an appropriate environment to maintain high quality performance, managing grading involves coordinating graders’ workload, monitoring performance and implementing responsive quality assurance.
Grading diabetic retinopathy: Who can do it?
Grading determines the level of disease present in the retina using a nationally agreed classification system and management protocols. Depending on the local setting, grading can be done by studying digital images of the retina or through slit lamp biomicroscopy and each grader follows a specific, and feature based technique
Grading must achieve both a high sensitivity and high specificity for a diabetic retinopathy (DR) programme to successfully prevent vision loss in a population. Trained DR graders use screening software or guidelines to assign the correct grade for the level of DR they see in each eye. There are four possible outcomes for the patient depending on the grade provided for each eye:
- Recall for routine digital screening
- Further slit lamp biomicroscopy by a senior grader or assigned clinician
- More frequent monitoring by the screening unit
- Referral to the hospital eye service
It is straightforward to define thresholds between levels of disease but, in practice, there will be areas of uncertainty within some grading decisions that need further consideration. Therefore, each DR programme needs to ensure there are clear systems and protocols and that these are followed.
The role of the grader is demanding and needs considerable training to maintain a high and consistent sensitivity and specificity. Each person undertaking grading needs a:
- Clear job description
- Support system to ensure they can provide the best service.
Who can be a grader, can vary across different settings, e.g. non clinical specialist technicians, optometrists or clinical nurses assistants. Each grader needs to be linked with and supported by an ophthalmologist, ideally a retinal specialist.
Local programmes need to constantly review their grading staff capacity and facilities to adjust for changes in demand for, and capacity of, the service and to make contingency plans for long-term sickness, maternity leave and annual leave. Ultimately, programmes need to work towards ensuring that enough accredited graders are employed at any given time to meet the service objectives and provide an effective service to patients.
As you watch this video, consider the different grading roles in your own setting. What are the challenges for quality assurance? In the future, do you think grading could be assigned to artificial intelligence systems instead of trained graders?
© London School of Hygiene & Tropical Medicine CC BY-NC-SA 4.0