Who shapes AI in care—and who gets to challenge it? Katie Thorn, Director of Innovation at Digital Care Hub, explores why responsible adoption depends on lived experience, frontline expertise and partnerships that keep human judgement at the centre.
An AI-generated care plan can sound convincing while missing what matters most to the person. A scheduling tool can produce an efficient rota while overlooking the relationships that make someone feel safe. Technology may work as designed and still fall short of good care.
That is why decisions about AI cannot sit with a supplier, an IT team or a care provider alone. People drawing on care, families, frontline workers, commissioners and technology specialists all see different parts of the picture. Bringing those perspectives together is essential to understanding whether a tool will help in practice.
The sector-led Digital Excellence in Care report for working-age adults provides wider context for this connected approach. Here, the question is how we apply that thinking to AI: who shapes its use, how decisions are challenged and what safeguards make it worthy of people’s trust.
Frontline staff understand how care is delivered and where systems create extra work. People drawing on care and their families know whether a tool supports independence, communication, dignity and control. Leaders, commissioners and suppliers must provide the resources, governance and connections needed to turn those insights into consistent practice.
People drawing on care and frontline staff should help define the problem, shape safeguards, test tools in real settings and agree how success will be judged. Their involvement needs to begin before a purchasing decision, while there is still room to question assumptions and change direction.
This takes time, trust and honest discussion, particularly when buying decisions emphasise cost or features and evidence is limited. A useful starting question is simple: whose experience might reveal something we have missed?
Without frontline and lived experience, plausible assumptions can cause problems. A scheduling tool may overlook travel, continuity of care or preferences for particular workers. AI-generated care plans may sound polished while missing nuance or weakening the person’s voice. Tools must also work across different accents, speech impairments and communication styles.
Keep human judgement central
Generative AI identifies patterns and predicts likely responses; it does not understand a person or situation as a care worker does. Its answers may be wrong, incomplete or biased even when they sound convincing, so human judgement must remain central.
LDC Care shows how a human-in-the-loop approach can work. Staff review AI-supported suggestions before they enter a support plan, while care, quality, compliance and IT colleagues test tools before introduction. People can add notes securely to their records. Adam, who is supported by LDC Care, explained: “I find it hard to say things – but I can write it down and share it.”
Technology supports the relationship rather than replacing it. The person has another way to express themselves, while staff remain responsible for listening, interpreting and responding.
Responsible AI use raises practical questions. Do people know when it is used and what information it draws on? Can they refuse or challenge an outcome? Is their data safe? Does the tool work fairly for different needs, and who is responsible when something goes wrong?
The AI in Care Alliance offers a useful framework. AI should support human rights, independence, choice and control, dignity, equality and wellbeing—not weaken them. Its principles can be applied through three checks. Is the output accurate and fair? Do people understand the tool, its limits and how to challenge it? Does it strengthen human care, and is its impact reviewed?
These questions belong in everyday decisions about buying, introducing and reviewing technology. They need to be understood by the people using a tool, as well as those approving it.
Suppliers have a part to play too. Their pledge commits signatories to ethical innovation, responsible use of data, openness, accountability and collaboration, with annual updates on progress. It is voluntary and not formally monitored, so providers must still carry out their own checks. Even so, it sets clearer expectations for suppliers working with social care.
Build a connected approach
The AI in Care Alliance, developed from the Oxford Project on responsible generative AI in social care, brings together people drawing on care, carers, workers, providers, suppliers, researchers, commissioners and policymakers. It aims to turn shared principles into practical guidance, learning and action, helping the sector challenge poor practice and learn from experience.
Its success will depend on whose voices shape it. Smaller providers, frontline workers, family carers, people who communicate in different ways and communities at risk of digital exclusion must influence decisions from the start, not comment at the end.
The sector can act now by starting with a care problem, not an AI product. Providers should involve people early, test tools in care settings and review their impact. Commissioners and policymakers can strengthen procurement and evidence, while suppliers should explain limitations as clearly as benefits.
The aim is bigger than better records or faster administration. Good technology should give people more choice and control, help staff make safer decisions and protect the relationships at the heart of care. If it does not help people live the lives they choose, it is not transformation.
