ICML 2009: Great Hall 2 Session 4

Theme: Performance and Quality Measures

Time: 10.45-12.15
Date: Wednesday 2nd September 2009
Chair: Veronica Delafosse

An evaluation of the TOOL (Toowoomba Online Literature Search) Protocol for improving literature searches in a hospital setting
Mr Daniel McDonald, Queensland Health, Australia
Patrick O'Connor, Queensland Health, Australia

Daniel McDonald has worked in a variety of libraries including establishing consumer health information services for a large public library service. Currently he is developing a clinical library service for the multidisciplinary Pain Clinic in the Toowoomba Health Service. He has never knowingly transmitted MRSA to a patient during ward rounds.

Patrick O’Connor has worked in health libraries for a decade. He has provided specialist clinical library services in cardiac and paediatric units. His interests include outcomes assessment for library services in hospital settings and improving literature searching.

Abstract:
This paper describes the development, application and assessment of the TOOL Protocol. The protocol was developed for use in non university hospitals with a broad clinical base, undergraduate education programs and limited local research capacity. Protocols used in other comparable health services were adapted to meet local needs by creating versions that incorporated an evidence synopsis and local information sources.

The protocol has facilitated the following activities:

• Documenting the planning, management and execution of comprehensive literature searches
• Benchmarking and audit of librarians’ literature searching skills and knowledge
• Demonstrating good literature searching practice to clinicians
• Educating clinicians about information resources, systems and scholarly publishing
• Supporting evidence-based practice and effective knowledge translation
• Marketing library services

Assessment of the protocol’s impact has been conducted via clinician self report and librarian peer review. This has guided further refinements and modifications to the protocol, including the development of abridged, discipline specific and complete versions. Further assessment of the protocol as a tool for teaching literature searching is required before it can be adopted more widely. In the absence of any previous program for locally assessing literature searching practices and techniques the protocol has been valuable for providing baseline performance data. An electronic version is under development and it will provide a map for clinicians seeking information beyond their normal scope of practice. The electronic version of the protocol will articulate with electronic databases to improve efficiency.

What is the evidence in evidence-based practice? Citation analysis of papers referenced in Australian cancer clinical practice guidelines
Dr Berenika M Webster, Thomson Reuters Scientific, Australia
Ms Danielle Penn, School of Population Health, The University of Queensland, Australia

Dr Berenika M Webster is a Strategic Business Managers for the ThomsonReuters Scientific for Asia Pacific markets (in particular in Australia and New Zealand). She supports regional users of ThomsonReuters research evaluative products. She is also responsible for the regional liaison with the evaluative products development team. Berenika holds Masters degree in Library and Information Science from Western Ontario and PhD in bibliometrics from Warsaw University. She has published in the area of scholarly communication and research evaluation. Prior to joining ThomsonReuters in July 2008, Dr Webster held academic posts in UK (Northumbria University in Newcastle upon Tyne and City University in London) and New Zealand (Victoria Univerity in Wellington) and was an Academic Project Manager (Research Quality) at Office of the DVC-R at University of Queensland in Australia.

Danielle Penn commenced as the Research Development Coordinator in the School of Population Health on 14th May 2007. Previous to this appointment, she was employed in research and clinically based roles in the not for profit, university, public hospital and private pathology sectors in Brisbane and Sydney. Danielle has obtained a Bachelor of Biomedical Sciences, Masters of Science, Masters of Public Health and a Justice of the Peace (Qual). Her experience is largely in health informatics, program evaluation and chronic disease management. She is also interested inequity and inequality in health, social justice and the environmental impacts of climate change. Danielle's role at the School of Population Health is to provide analytical and academic support and advice to enhance the School's research performance, in the development of competitive grant applications and publications and bibliometric analyses.

Abstract:
The Excellence in Research in Australia (ERA) initiative aims at assessing the quality of Australian research. It is relying on the use of publication and citation counts in scientific literature. While the bibliometric approaches are established and accepted for evaluation of the impact research has within the scientific community, less is known about assessing impact of research on clinical practice and outcomes. One of the ways for medical research to influence practice is through the inclusion in clinical guidelines. Clinical practice guidelines, as defined by Cancer Council Australia, bring together the best available evidence to underpin scientifically-valid recommendations for the prevention and diagnosis of cancer and treatment of care of patients.

This study provides a bibliometric picture of literature referenced in Australian clinical practice guidelines. In Australia NHRMC develops and/or approves clinical guidelines. Cancer practice guidelines are produced in cooperation with Australian Cancer Network (up to date -- 21 guidelines), National Breast Cancer Centre (5 guidelines) and a small number with Royal Australian Colleges of General Practitioners and of Surgeons.

26 Australian cancer clinical guidelines have been identified for this study. References on these guidelines were analysed by date of publication, field of research, type of research (basic to applied), country of origin(institutional affiliation of authors) and citation impact. Analyses were normalised for field of research, year and type of publication. Finally, the bibliometric profile of the literature in the Australian guidelines was compared to that in UK guidelines (as presented in paper by Lewison and Sullivan (BrJCancer, 2008).

Research performance indicators at Aalborg Hospital – initial analysis
Jens Peter Andersen, Medical Library, Aalborg Hospital, Aarhus University Hospital, Denmark

Abstract:
Objectives: To apply and evaluate a system of research performance indicators at the department level of a university hospital.

Methods: The performance indicators used in the study are differentiated in five main areas of research activities, namely funding, scholarly activities, mediation, merits and networking, innovation and technology transfer. These areas are divided into various indicators, using a scoring system which allows for some types of comparisons. The scholarly activities stand out as being the only activity without a threshold for maximum score. The model used for this score is akin to the Norwegian model for academic performance as well as the Australian research quality framework.

Results: The complete system of performance indicators is applied on a 2007 dataset for the first time, hence results are purely indicators of which departments perform well in given areas, and in which fields certain activities are focused. Scholarly activities in particular are reported from 2002-2007 offering a timeline for these activities.

Conclusions: The 2007 performance analysis mainly serves as a reference point for future evaluations, as it is being used for the first time as a strategic research management tool. However, the results of the initial analysis also serve as actual indicators. These indicators manage to express a general overview of the five main areas of performance as well as pinpoint areas of specialized activity and potential areas of concern.