Tag Archive for: Realistic Evaluation

Transforming care and health through information and technology: Local Investment Programme – Interim Report

By Dr Adam Hoare

Local Government Association – Local Investment Programme

Local Government Association Local Investment Programme Interim Report

Local Investment Programme – Interim Evaluation Report

In 2017 the Local Government Association in collaboration with NHS Digital introduced the Local Investment Programme. The programme recognised that the imperative for local care services to transform has never been greater due to rising needs, citizen demands and expectations alongside sustained pressure on resources. Although technology will never be a replacement for the delivery of care it can, if used appropriately, support people to live at home for longer, enable professionals to work more effectively together and help commissioners target services where they have the greatest impact. The Local Investment Programme (LIP) supported transformation through one-off local investment funding of up to £50,000 for projects that seek to use information and technology to improve adult social care and health outcomes and deliver financial savings as well as supporting more resilient care services and in improving outcomes, including people’s experience of care.

The programme was underpinned by a number of key principles:

  • encouraging open and replicable approaches which enable other LAs to adopt digital ways of working
  • supporting innovation and new thinking within adult social care and health, assisting its digital maturity within a challenging financial environment
  • assisting cross-council or multi-agency use of information and technology
  • developing an evidence-base and tools to support ongoing local digital investment in adult social care and health.

It was expected that the outputs and learning from these projects would be reusable and replicable, enabling them to be shared and implemented widely across the sector.

Councils who were successful in their bids for funding received support from an external partner organisation to help evaluate benefits, capture learning and support dissemination to assist other local areas. Participating local authorities were required to actively support this work, over the course of the programme, including in the development of guidance and tools for the sector.

The Bayswater Institute and OPM Group (now Traverse) were successful in winning the opportunity to evaluate the 19 technology interventions funded by the programme. In April 2018 an interim report was produced capturing the learning on the ground as well as considering the programme itself.

 

Early Evaluation of the Organisational Implications of CSCW Systems

Evaluation of Computer Systems which Support Cooperative Work (CSCW)

Abstract:
Computer systems which support cooperative work will undoubtedly change the way people interact with one another in a working setting. In stimulating these changes, the CSCW system will be a force for organisational change. Over the past twenty years there has been a tradition of impact research, in which investigators have studied the impact computer-based information systems implemented within organisations have had upon those organisations. The results have been many and varied. Some studies show that computer systems lead to the centralisation of power, while others show decentralisation of power. At the level of the individual job there are studies which show the empowerment of the individual and the opposite: the creation of the “white collar assembly line.” A study from 1983 demonstrated both job enrichment and deskilling from two different computer systems in the same organisation. There is widespread agreement that there can be substantial change, but very little agreement on the form that it takes. We have argued elsewhere (Eason 1988) that the reason for these diverse results is that computer systems are not deterministic, and that they can be used to achieve many different organisational effects. There is an opportunity in the design and development process to plan the organisational outcomes and to achieve the impact that is desired by the members of the organisation.

Early Evaluation of the Organisational Implications of CSCW Systems Ken Eason

Springer Link to Article

Eason K.D. and Olphert C.W. (1996) Early evaluation of the organisational implications of CSCW Systems, In Thomas P. (ed) ‘CSCW Requirements and Evaluation, Springer-Verlag, London 75-89

Psychological Processes in the Use of Electronic Journals

Paper presented at the UKSG 23rd Annual Conference, Keele, April 2000

Abstract:
Evidence of user behaviour with electronic journals from the SuperJournal project is reviewed to identify the psychological processes being employed. The article reviews patterns of use, the dominance of browsing as the means of seeking information, the depth, breadth and range of use and the tendency to print. The article concludes that the dominant user approach is a coping strategy that maximises success for minimum psychological effort.

PDF Link to Paper

Eason K.D. and Harker S.D.P. Psychological processes in the use of electronic journals, Serials 13(2) 67-72

The Use and Usefulness of Functions in Electronic Journals: The Experience of the Superjournal Project

Value to users of a range of functions of electronic journals and their usefulness in the specific context of the SuperJournal Project.

Abstract:
For the evaluation of each of the functions three types of data were analysed in relation to each other and in light of other contextual data: logged data of usage, survey data on user satisfaction, and survey data on the perceived importance of the function. The analysis shows that basic browsing, printing and search make up the core functions of electronic journals; other functions, such as saving of bibliographic data, alerting, customising, links with external resources and communication, serve as peripheral functions. The usefulness of both the core functions and the peripheral functions in a specific service is influenced by various implementation factors. However, it is the realised usefulness of the core functions which determines the use of a service.

Emerald Insight Link

Eason K. D. Yu, L. and Harker S.D.P. The use and usefulness of functions in electronic journals: The experience of the Superjournal Project. Program 34(1) 1-28

 

Patterns of Use of Electronic Journals

This paper classifies a spectrum of user behaviour with electronic journals into a typology of eight categories of user/use.

Abstract:
On the basis of a twenty‐two month transaction log of SuperJournal and using K‐Means cluster analysis, this paper classifies a spectrum of user behaviour with electronic journals into a typology of eight categories of user (or eight patterns of use): the searcher, the enthusiastic user, the focused regular user, the specialised occasional user, the restricted user, the lost user, the exploratory user and the tourist. It examines the background and experience with SuperJournal of each type of user to illuminate its formation. The examination shows that the contents (both coverage and relevance) and ease of use of a system as they were perceived by the user were the most significant factors affecting patterns of use. Users’ perceptions of both factors were affected by a range of intervening factors such as discipline, status, habitual approach towards information management, availability of alternative electronic journal services, purpose of use, etc. As any service is likely to attract a great variety of users, so will it lead to differing patterns of use. This paper demonstrates the need for a service to meet the requirements of users with these varied patterns.

Emerald Insight Link to Paper

EASON, K.D., RICHARDSON, S, YU, L., Patterns of use of Electronic Journals, Journal of Documentation, Vol 56(5) pp 477-504. ISSN 00220418.

Productivity? – Don’t Just Fund the Technology Phil! A response to the budget of the 22nd November 2017.

By Dr. Adam Hoare

The Organisation for Economic Co-operation and Development has downgraded its 2017 growth forecast for the UK to 1.5% from a 1.6% estimate made in September, making Britain the weakest economy in the G7. The office for budget responsibility had taken the rosy view that after 2008 UK productivity growth would return to previous levels of around 2% It has now admitted, after years of getting it wrong, that it is likely to sit around 1.3-1.5% until 2020. Last week’s budget reverberated with the recurring issue of low productivity growth. The solution presented was an industrial strategy. Something that had fallen out of favour as Government interference.

The announcements based on borrowing came thick and fast:

  • Digital skills and startup funding to reinvigorate the UK’s waning productivity.
  • £3 billion to cushion the landing of a potential hard Brexit, the chancellor said: “This Budget is about much more than Brexit. For the first time in decades Britain is genuinely at the forefront of this technological revolution. Not just in our universities and research institutes, but this time in the commercial development labs of our great companies, and on factory floors and business parks across this land. But we must invest to secure that bright future for Britain.”
  • Last year’s £23 billion National Productivity Investment Fund was to provide £31 billion in funding over six years, compared to the originally planned five. R&D to receive another £2.3 billion investment, under the government’s Industrial Strategy aim to ramp up R&D spending to 2.4% of GDP.
  • To double the number of tech startups founded in Britain with the goal to see one created every half an hour.
  • A £10 million Regulators’ Pioneers Fund to help regulators find new ways to bring emerging tech – AI and 5G – to market.
  • Tech City UK, to be rebranded as Tech Nation, a body with a remit to spend £21 million on developing the UK’s various startup hubs.
  • In a bid to tackle the UK’s stark digital skills gap, the chancellor also outlined fresh cash to retrain people and provide a greater focus on maths and computing for children and teenagers.

The idea of a strategy and a long list of funding opportunities for new technology seems to overlook some very important evidence. Figures for various IT projects (some of the figures originating from the National Audit Office no less) demonstrate a persistent gap between the projected benefits and the reality:

  • Child Support Agency – £500m estimated loss;
  • DEFRA Rural Repayments Agency – £130m estimated loss;
  • Inland Revenue NIHS – £3-4 billion estimated loss;
  • Magistrates Court LIBRA – £232 million estimated loss;
  • HM Prison Service C-NOMIS – £690 million estimated loss;
  • Fire and Rescue FiReControl – £469 million estimated loss;
  • NHS NPfIT – £20 billion estimated loss.

As we borrow money to fuel a technological “hail Mary pass,” it would seem a good time to think about why we fail to convert so many such passes to a touchdown. The Bayswater Institute has been extensively involved in embedding and evaluating digital technologies in health and social care over several decades. Over the last decade alone there have been hundreds of initiatives to improve productivity in care provision by elevating the use of technology to 21st century standards. Although there has not been an overall assessment of the impact of these initiatives the experience of care provision points to low impact from these initiatives. From seeing these projects from the inside, we have developed a level of understanding of why they struggle – and it is not the technology. Two things work against the use of technology in many of these scenarios:

  1. The technology does not exist in isolation it is part of a system that involves the people using it and the people receiving services. If it does not work for them it is not productive.
  2. Where there is an increase in productivity it usually means a single person can handle more work or the workforce can be reduced. This inevitably generates resistance.

Both of these challenges are rooted in social science and the interface between people and technology. Understanding these sociotechnical systems is essential in successfully capturing the benefits the technology can bring. Looking back over the announcements we cannot see where this is mentioned. Throwing money at the technology and expecting social transformation is an interesting approach but the evidence is – it has been done before and it will fail.

A third issue that recurs in productivity considerations. To know a system is more productive than it was before it must be measured in a meaningful way. This links back to point 1 above. If it works for the professional but not the citizen – it does not work. Hence, the outcomes of productivity must include social value and social impact otherwise public money is spent on making the system happy and the service recipient unhappy.

We spend much of our time providing summative evaluations of where the barriers and challenges are in technology projects that are trying to embed into practice. We have a special interest in formative evaluations of interventions where we can draw on our experience and anticipate some of the problems ahead of the development and have the opportunity to have an impact on the NAO estimated loss. If the focus remains on the technology and not the combined scoiotechnical system the return on investment is likely to be negative. The last thing that Phil wants.