Focus on Specific AI Applications

Ethical AI must start from a clear care problem. When the application is aimed at a concrete need in care, it is more likely to genuinely contribute to the wellbeing of patients and not to become technology without clear added value.

Organisational A clear problem definition is the starting point for a responsible deployment of AI. When the purpose and the scope are clear, it is easier to assess what is technically possible and meaningful. This prevents technology from being introduced without a clear need, and allows AI to contribute in a targeted way to better work processes and to the broader mission and strategy of the organisation.

Legal A patient has a right to purposeful care and to high-quality care. If this includes the use of AI, it must be done in a considered and well-thought-out way. Re-evaluate the purpose and the value after a reasonable period of time. Be alert to situations of overtreatment and loss of efficiency. The implementation of AI, too, can lead to such situations, possibly completely unexpectedly.

Ethical Possible broad ethical benefits: patient wellbeing, wellbeing of care staff, patient self-determination, fair allocation of resources, etc.

Clinical Performance outcomes must be defined prospectively in order to allow objective evaluation.

Organisational AI often influences several work processes at the same time. It is therefore important to consider in advance how processes need to be adapted. If processes do not evolve along with it, automation can even be counterproductive. By examining this impact carefully, AI can contribute better to quality of care, efficiency, workability and financial sustainability.

Ethical Possible broad ethical benefits: better collaboration increases quality of care and therefore patient wellbeing, etc.

Organisational AI influences how different actors work together in the care process. It is therefore important to map work processes and roles clearly. This makes visible where collaboration is needed between care professionals, IT and other stakeholders, and also makes it possible to provide checks to verify the results of AI applications.

Psychosocial AI-driven task shifts influence team dynamics and responsibilities.

Clinical Patient experiences with AI-driven care vary and require proactive communication.

Organisational AI can change the role and work experience of care providers and patients. It is therefore important to monitor this impact actively, for example via clear indicators regarding quality of care, patient satisfaction and the work experience of staff, such as autonomy, learning opportunities and technostress.

Psychosocial Changes in the professional role influence work experience and job satisfaction.

Ethical Transparency about the use of AI is important in order to maintain trust between care provider and patient. Open communication helps patients to make informed choices and supports their right to self-determination.

Legal When specific AI applications are used in care, the patient's right to information does not change. This means that not only the question of whether AI applications are used, but also all information about the name, the role, the impact, etc. of the AI application follows the legal rules we have long known, first and foremost as laid down in the Patient Rights Act and the GDPR. (1) The Patient Rights Act focuses on understanding the state of health, the intervention and the possible alternatives. For each application it must therefore be examined to what extent it is necessary for patients to know and understand it in order to comply with their rights as patients. (2) The GDPR focuses on information about the processing of personal data. Obtaining an answer to the question of whether AI applications are used in a care process does not, in concrete terms, form part of the information to be provided. Nevertheless, in order to be able to comply with all the rights of data subjects, it may become necessary at a given moment to communicate explicitly about the use of AI, whether or not proactively. That will be the case, for example, when the valid legal basis for the processing is the consent of the data subject.

Organisational Corporate communication can play a role here, as can giving guarantees about the human role in the care pathway.

Ethical Patients must be able to ask questions and to receive information about their care process. That supports informed consent, protects their autonomy and helps to prevent misuse or harm, for example through privacy breaches. At the same time, responsibility for the use of AI remains with the care provider and the organisation, so that it is not placed with the patient. This can also contribute to lasting trust in the care system.

Organisational AI is often invisibly embedded in work processes, which means that patients cannot always refuse a specific application. It is therefore important that it remains clear that human specialists retain a central role in care. Clear communication about the use and the benefits of AI for patient care helps to maintain trust.

Ethical The ability to contest decisions stimulates self-determination and the principles of fairness and responsibility.

Legal The care provider remains primarily responsible for the decision taken, also when AI was used as support. Bear in mind, however, that anyone “who has grounds to consider that there has been an infringement of the provisions of [the AI Act] may submit a reasoned complaint to the relevant market surveillance authority” (Article 85 AI Act, read together with recital 170). Patients are therefore also free to contest decisions before the market surveillance authorities on the grounds of an infringement of the AI Act.

Organisational Complaint processes around AI decisions require clear escalation and review procedures.

Legal For a limited number of very specific AI applications, the AI Act makes it mandatory for the institution or care provider qualified as a “deployer” to be able to explain substantively to patients what the role of the AI system and its output was in decisions taken in respect of the patient (see Article 86 AI Act). The right to an explanation arises when the decisions have had adverse consequences for the health, safety or respect for the fundamental rights of the patient, or when the patient has experienced other legal effects (such as higher costs, the loss of a reimbursement or benefit, etc.). Be prepared for this when you introduce AI applications that (a) support the assessment of the eligibility of patients (or other persons) for public benefits and services (including health services); or (b) are intended to evaluate and classify emergency calls by natural persons or to be used for dispatching, or for establishing priority in the dispatching of, emergency services, including police, fire brigade and ambulance, as well as systems for the triage of patients in need of urgent medical care. In addition, for all AI applications that process personal data, the provisions of Article 22 GDPR on automated individual decision-making, including profiling, remain applicable, just as is the case for other types of medical software.

Legal When AI is used in health care, the same rules apply regarding collaboration (including patient consent), data sharing and professional secrecy. The use of AI must not be a licence to bypass these rules.

Legal AI can only be a (medical) aid: human-centred and an instrument for people, with the ultimate aim of increasing human wellbeing. It is important to monitor, for each individual application, whether this legal red line is not crossed.

Ethical Value-driven and purposeful policy is important in terms of integrity and mutual trust between care providers, patients, institutions, policy makers, etc.

Organisational It is important to evaluate the impact of AI systematically. This can be done via clear indicators that align with the strategy, values and objectives of the organisation. By measuring, for example, process time, costs, quality and flexibility before and after the introduction of AI, it is possible to determine whether the application really contributes to better care and to the mission of the institution.

Psychosocial Alignment with the values of care providers is crucial for sustainable adoption of AI on the ground.

Legal Within the diagnostic and therapeutic freedom of the care provider, the preferences of the patient must always be taken into account. This implies that the impact of AI on high-quality care must not be assessed solely on the basis of efficiency gains for the care provider or the institution, but also from the patient's perspective.

Ethical The wellbeing of the patient can decline unnoticed if adequate validation is lacking. It is important to realise that in certain settings non-validated applications could also do more harm than good.

Clinical AI applications must be validated locally, because performance can differ between patient groups and care contexts. External validation alone is not sufficient to guarantee that the system works equally well within one's own institution.

Organisational Before an AI application is introduced widely, it is important to first consider whether local validation is necessary or desirable. By first testing the application on a smaller scale and validating it internally, errors can be detected, confidence among staff grows and the roll-out can later proceed more safely and efficiently.

Legal Have the legally required opinions been obtained? Think first and foremost of the ethics committee of the care institution (for research) and the DPO. Have other opinions also been obtained that may not be legally required, but that are described in internal procedures? Depending on the application, these may differ. Consider, for example, an ethics committee in the event of new ethical questions, the occupational prevention and protection service in the event of AI with an impact on employees, a medical technology committee in the event of questions about medical validation and quality, an IT department in the event of an impact on information security, the legal department because of liability risks, etc.

Economic Without evaluation, an investment in AI remains mainly a matter of belief instead of a well-founded choice.

Ethical Value-driven policy is important in terms of integrity and mutual trust between care providers, patients, institutions, policy makers, etc.

Organisational The added value of an AI application must be evaluated by comparing the performance of the care process before and after its introduction. It is important not to look only at efficiency, but at several dimensions, such as quality of care, the experience of patients and care providers, and the sustainability of the system. It is also useful to consider different AI alternatives before proceeding to roll-out.

Ethical Collecting feedback about the use of AI is important in order to determine whether the application actually contributes to the wellbeing of the patient and to good care.

Clinical Systematic collection of feedback detects problems early that are not visible in data.

Organisational Systematic feedback from users is important in order to keep improving the functioning of AI and the care process. That feedback can differ depending on the phase of the application: in research it also concerns the technology itself, whereas for a proven application it focuses mainly on integration into the work process. By collecting feedback from various stakeholders and following it up within the process team and together with IT, the performance and impact of the application can be better monitored and adjusted.

Legal Only collect feedback in compliance with the GDPR.

Ethical By re-evaluating AI applications regularly after implementation, it is possible to determine whether they continue to contribute to the wellbeing of the patient and to good care.

Clinical AI applications must also be re-evaluated regularly after implementation, because their performance can change in practice. Periodic monitoring is therefore technically necessary in order to keep guaranteeing the quality and reliability of the system.

Organisational Integrate this evaluation into the standard processes with which other tools are also continuously monitored and evaluated. If it turns out that there are process performance problems, improvements must be considered. Or if there are new IT opportunities, one can explore how the process can benefit from them. So a combination of exploitation and exploration, in other words ambidexterity.

Legal In so far as the AI application is considered high risk, it must be borne in mind that virtually all obligations are formulated as having to be complied with “throughout the entire lifetime”. Post-market surveillance and (clinical) evaluation are then necessary. A few examples: (1) As part of your risk management system, the adequacy of the risk control measures will have to be examined and a risk evaluation will have to be carried out (Article 9 AI Act); (2) The technical documentation and the automatic logs require systematic monitoring and evaluation (Articles 10–11 AI Act); (3) Where non-conformity is suspected, information obligations arise for the provider. The provider will also have to take corrective measures, which must be implemented by the care institution (Article 20 AI Act).

Ethical Signals such as problems with patient safety, privacy, self-determination or fairness can give rise to a reassessment or adjustment of an AI application. These ethical values help to determine whether the use of AI is still responsible.

Clinical Performance indicators for AI must include both technical and clinical parameters.

Organisational Clear and predefined KPIs help to monitor objectively whether an AI application contributes to the strategic objectives of the organisation. When these indicators are not met, or when many complaints come in from patients or staff, that can be a signal to adjust or reassess the application.

Legal The legislator emphasises that the management of an AI system must be a continuous and iterative process. Some specific indicators that should trigger particular scrutiny are: (1) Serious incidents (e.g. harm to the health of a patient or member of staff, harm to the rights of the patient, disruption of critical infrastructure, etc.); (2) The levels of accuracy or robustness indicated in the instructions for use are not achieved; (3) Risks that had not been identified during the development phase emerge during use, including problems caused by interaction effects.

Ethical Predefined criteria help to determine when an AI application must be suspended or discontinued. Ethical considerations play an important role here, such as the wellbeing of the patient, privacy, self-determination and fairness.

Legal Explicit exit criteria are essential for patient safety and legal liability. The AI system must be built in such a way that it allows an exit.

Clinical Predefined stop scenarios prevent poor AI systems from remaining in use for too long.

Organisational These predefined criteria or scenarios are needed in order to be able to act quickly.

Ethical Involving care providers and staff in the selection or development of AI helps to safeguard the quality of care and to avoid moral stress among staff. In this way the use of AI remains aligned with the wellbeing of patients and with the professional values of care providers.

Organisational Involving care providers and other stakeholders in the selection or development of AI is important because they know the working context best. Co-creation helps to align the application better with practice, increases acceptance among staff and thus increases the chance of a successful implementation.

Psychosocial Involvement in the selection increases willingness to adopt and reduces resistance.

Legal An institution must provide for the AI literacy of care providers and other staff. Care providers must keep evidence of this; if not, they may not use the AI tool.

Organisational The introduction of AI can gradually cause changes in work processes and in the division of tasks. It is therefore important to monitor these developments actively. When processes change, the staff involved must receive appropriate, job-specific training in order to make the implementation successful.

Psychosocial Deliberate monitoring of changes in working methods protects employees' rights and wellbeing.

Legal For high-risk applications, the obligation to guarantee human oversight must also be taken into account. Care providers, but for example also staff members who exercise this oversight when AI is used in an HR context, must have the necessary competence, training and authority, as well as the required support to be able to monitor the impact on working methods.

Ethical An AI system cannot itself bear responsibility. It must therefore be clearly established which person or organisation is responsible for the settings, the use and the decisions surrounding the application. Ultimate responsibility must always lie with a natural or legal person, not with the technology.

Organisational A clear division of roles is needed in order to steer AI applications properly. A process owner usually monitors the functioning and makes adjustments, possibly together with the IT department. Ultimate responsibility, however, remains with top management, so that the lines of accountability within the organisation remain clear.

Legal The choice between AI as a safety net or as a first adviser has important consequences for patient safety and liability. It must therefore be clear that human responsibility is always retained when AI is used in care. Bear in mind that, although the AI Act does not prohibit a hammock, (a) human oversight must remain effective; and (b) certain care tasks may, because of sector-specific (Belgian) legislation, never be carried out by an autonomously operating AI application.

Clinical Hammock configurations increase automation complacency; a safety net preserves human alertness.

Organisational This depends from process to process. Employees must not, however, get the impression of being monitored, because this can be negative for their job satisfaction and can limit their sense of autonomy. A healthy human-AI collaboration in which human talent is recognised is necessary for acceptance and success.

Clinical Targeted AI training increases correct use and reduces the risk of dangerous errors.

Organisational Care providers need training and support in order to use AI in a safe and considered way. Insight into how the application works increases their resilience and helps to integrate AI correctly into the work processes. Initiatives such as training, coaching, peer learning, IT support and clear communication support adoption and help to share knowledge and good practices.

Legal The management of an institution (the deployer) is responsible for providing ways to create sufficient AI literacy among staff and care providers.

Economic A negative effect on collaboration, culture, etc. can undo the positive effect of the AI application.

Ethical Documenting negative effects is the basis for learning and for future improvement.

Organisational The introduction of AI can have various effects on processes, roles and collaboration. It is therefore important to monitor and document possible negative consequences actively. By following up processes and making timely adjustments when problems or undesirable effects occur, the change can be better guided and the implementation can continue to be supported by staff.

Clinical Explicit reflection on human-machine collaboration prevents the erosion of professional expertise.

Organisational When deploying AI it is important to pay attention to task shifts and to the collaboration between people and technology. The limitations and possibilities of the system must also be clear to care staff. By involving care providers in rethinking work processes and asking for feedback regularly, their expertise continues to be recognised. This also helps to preserve their sense of autonomy, connectedness and professional worth, which is important for motivation and job satisfaction.

Psychosocial A deliberate task-shift policy protects care providers against undesirable role ambiguity.

Legal On the basis of their professional autonomy, care providers may only perform those acts for which they have the necessary demonstrable competence and experience. This also applies to AI.

Clinical In order to keep staff alert when using AI, training is essential. Care providers must understand how the specific application works and where its limitations lie. Exercises, for example with incorrect AI output, can help to strengthen their critical eye and their vigilance.

Organisational Training, communication and HR initiatives help to encourage a critical attitude and to strengthen the values the organisation wants to realise with AI.

Ethical Transparency about the limitations of AI is an ethical obligation towards users.

Clinical Understanding the limitations of AI is essential in order to interpret output correctly and to recognise errors.

Organisational Staff must understand the limitations of AI applications in order to use them correctly and to recognise possible problems. Basic knowledge about AI is therefore important for everyone, supplemented with more targeted, job-specific training and possibly a general training course to increase awareness of AI.

Legal Both when using high-risk AI systems and when using general-purpose AI models, it is mandatory to communicate transparently about the limitations of the AI systems. For a high-risk system, the provider makes this information available via instructions for use. These must be available to the member of staff. For a general-purpose AI model, the provider must make technical documentation available that allows the institution and its staff to assess how the model can or cannot be integrated (e.g. assessment of the unsuitability of the data sources for the intended task).

Ethical Because AI applications can make mistakes, it is important that staff continue to assess AI output critically. By allowing room to contest AI suggestions and to test them against their own expertise or through consultation, the professional autonomy of care providers is preserved and it remains clear who is responsible for the final decision.

Clinical Critical thinking about AI output is a learned skill that must be actively practised.

Organisational It is important that staff do not follow AI output blindly. Through clear communication about the AI policy and targeted training via HR, a critical attitude towards AI can be encouraged and strengthened. In this way care providers keep testing AI results against their own expertise and professional judgement.

Legal When an institution introduces AI systems, it is responsible for ensuring that all persons who use the AI system on its behalf (staff and others) have “a sufficient level of AI literacy”. Where sector-specific or profession-specific obligations regarding lifelong learning already exist, the obligation regarding AI literacy can be read as a new component to be added to these. In other situations, separate training will have to be provided. It should be noted here that this obligation is not limited to high-risk AI systems.

Clinical Even when AI is used, staff must continue to develop their knowledge and skills. Because AI results have to be checked, it is important that expertise is retained. By making room to keep practising actively and to think along with the system, deskilling can be prevented and the clinical competences of care providers preserved.

Organisational Also when deploying AI, it is important that staff continue to learn and can develop further. Room to learn and lifelong learning help to preserve and strengthen expertise. Moreover, when AI creates room for further specialisation, this can contribute to greater job satisfaction, motivation and quality of care.

Ethical Data bias in AI is an important driver of health inequality.

Clinical It is important that the training data of an AI application are representative of the patient population. If that is not the case, the system may work less well and make systematic errors for certain patient groups.

Organisational Every digital innovation must be tested thoroughly before being rolled out on a larger scale.

Legal As part of the data and data governance obligations that apply under the AI Act to high-risk AI systems, this check must always form part of the internal procedures. Training, validation and test datasets must be relevant and sufficiently representative in the light of the intended purpose. Account must be taken of the specific characteristics of the geographical, contextual, functional and behavioural setting.

Legal Emergency procedures are necessary both in order to meet the requirements imposed by the right to high-quality care provision and those imposed by the GDPR. A distinction must be made between situations in which life-threatening impact can arise and other situations.

Clinical Proactive risk identification is required for safe AI implementation in high-risk care contexts.

Organisational Mapping risks is important in order to integrate AI into the care process in a responsible and sustainable way. By systematically analysing possible strengths, weaknesses, opportunities and threats, the organisation can better anticipate problems and take appropriate measures.

Ethical Residual risks after mitigation must be explicitly communicated to users.

Organisational Risks with a significant negative impact must be avoided or limited as far as possible. Effective risk management therefore requires both technical measures and attention to the behaviour and working methods of staff.

Legal Known risks, but also reasonably foreseeable risks that can have an impact on health, safety or respect for fundamental rights, must be accompanied by risk-mitigating measures. These measures can relate both to the design of the system and to the design of the implementation process. They can be of a technical or of an organisational nature.

Legal There must be responsibility on the part of a natural or legal person.

Clinical AI systems can make mistakes or fail. Fallback procedures are therefore needed in order to guarantee the continuity of care. By retaining human control and providing alternative working methods, people and technology can complement each other and errors can be absorbed better.

Organisational Fallback procedures are needed in order to prevent an organisation from becoming too dependent on AI. When the system fails or produces incorrect output, care providers must be able to intervene and to overrule decisions. Ultimate responsibility remains with people, who moreover retain important human skills such as assessing context, showing empathy and making ethical considerations.

Ethical Transparency about the algorithm and the training data is required for trust and auditability.

Legal A broad transparency obligation has been imposed by the AI Act on the providers of AI. Via instructions for use, they must inform the deployer (the institution or self-employed care provider) about the functioning, the provider, the characteristics, etc. of the system. The deployer will in turn have to ensure that this information is examined critically.

Clinical Algorithm characteristics are especially important if you are looking for systems whose decisions are explainable.

Organisational Information about the training data is essential in order to be able to assess performance in one's own population.

Ethical AI often does not have all the relevant information about a patient. Care providers must therefore always have the possibility to set AI recommendations aside and to base their decision on their professional judgement and on the full context.

Legal For high-risk AI systems it is mandatory to provide, for everyone who exercises human oversight, a clear procedure that allows them to intervene in the operation of the AI system by means of either a “stop button” or a comparable safe procedure (Article 14(4)(e) AI Act).

Organisational AI output must not be regarded as absolute truth. There must therefore be clear agreements about when care providers can disregard AI advice and let their own expertise prevail. These agreements can be laid down in a central AI policy and further elaborated in procedures within the work processes.

Clinical Reporting AI incidents is essential for learning and for improving system safety.

Organisational AI-related errors and incidents must be included in existing safety and incident reporting systems. By registering errors systematically and following up processes, underlying problems in the system or in the work process can be detected. Where necessary, this can lead to adjustment of the process or the technology, or to escalation to IT, an AI team or management.

Legal See questions 12 and 13.

Clinical Consistent follow-up of AI incidents is required for system safety and for a learning organisation.

Organisational The follow-up of AI incidents is best integrated into existing quality and incident reporting systems. In this way the use of AI does not stand apart from other care tools and the threshold for reporting and following up problems is lowered. This strengthens overall quality assurance within the care organisation.

Ethical Staff members should not have to fear for their jobs, because they assume they will be replaced one-to-one.

Organisational A member of staff will rarely be replaceable one to one. It will usually concern subtasks, which makes a reorganisation of the work necessary.

Organisational Providing job-specific training on working together with AI, and allowing people to specialise further.

Organisational Research shows that people and AI can strengthen each other. This does, however, require a critical human eye.

Organisational Providing job-specific training that encourages a critical view of AI, and sharing stories about poor AI advice at process level, with advice on how to deal with this.

Legal When high-risk AI is used, the information needed to understand the output of the AI system properly forms part of the minimum information requirements for the instructions for use that must be supplied by the provider to the institution and, via the institution, to the actual user. However, mind that the legal obligation is formulated rather vaguely and is intended more to prevent output from being uninterpretable. These legal measures do not prevent automation complacency from arising.

Organisational In case of doubt, an additional verification of the decision is needed.

Organisational Training that encourages a critical view of AI, and sharing stories about poor AI advice with advice on how to deal with this. Ultimately, tackling these biases is all about proper awareness and training.

Legal It is advisable to draw continuous attention to automation bias through post-implementation procedures and through training, among younger as well as older staff.

Organisational New staff must be given the chance to acquire skills before they use AI applications.

Organisational Allowing people to specialise, learning to deal with human-AI collaboration, and good training.

Legal On the basis of various legal frameworks, different conditions must be met before an institution can work with a provider. Write conditions such as the submission of a Data Protection Impact Assessment (GDPR), a Fundamental Rights Impact Assessment (FRIA) and other minimum requirements that the organisation must meet in the context of the GDPR, the AI Act, cybersecurity, etc. into the documents before issuing a call for tender.

Ethical The use of patient data requires clear agreements and transparency. That increases the trust of patients and helps to handle ethical questions correctly, for example when data are compared or when AI detects possible health problems without the patient having explicitly asked for this. It must therefore be clear how these data are used and within which legal framework.

Legal Make a distinction between the use of patient data for the development of AI, for the training of AI and in the use of AI systems. Different legal bases are relevant for each of these phases, and for each of these phases it must be examined whether the rights of data subjects are respected. Important: the AI Act, as a measure to support innovation, provides a new legal basis for the use of personal data within AI regulatory sandboxes (Article 59) but not for the use of personal data for testing AI in real-world conditions outside the AI regulatory sandbox (Article 60). For the latter, the explicit consent of the data subject is required (Article 61).

Legal Transparency about the use of patient data is essential. Patients must know how their data are used and, where necessary, be able to give their consent, as is also required under the GDPR.

Economic A structured compliance check at the time of purchase avoids costly adjustments afterwards.

Legal Subject the AI application to an integrated legal assessment: provisions of the AI Act must always be read together with existing obligations. Think of obligations arising from sector-specific legislation such as the Patient Rights Act, the Quality Act and the MDR or IVDR, but also of other horizontal legislation such as the GDPR or public procurement legislation.

Legal Depending on the legal position of the care provider under employment law, the care provider will be qualified under the AI Act as a user or as a deployer. Each role has its own responsibilities. Carry out the analysis for the institution and then discuss the support that is needed.

Organisational Compliance with legal obligations should be built into the processes. Individual care providers should need to think about it as little as possible.