Implementations in Machine Ethics: A Survey

Increasingly complex and autonomous systems require machine ethics to maximize the benefits and minimize the risks to society arising from the new technology. It is challenging to decide which type of ethical theory to employ and how to implement it effectively. This survey provides a threefold contribution. Firstly, it introduces a taxonomy to analyze the field of machine ethics from an ethical, implementational, and technical perspective. Secondly, an exhaustive selection and description of relevant works is presented. Thirdly, applying the new taxonomy to the selected works, dominant research patterns and lessons for the field are identified, and future directions for research are suggested.


INTRODUCTION
Autonomous machines are increasingly taking over human tasks.Initially, simple and limited assignments such as assembly line labor were taken over by machines.Nowadays, more complex tasks are transferred to so ware and robots.Even parts of jobs that were previously deemed purely human occupations, such as being a driver, credit line assessor, medical doctor, or soldier are progressively carried out by machines (e.g., [36,42]).As many believe, ceding control over important decisions to machines requires that they act in morally appropriate ways.Or, as Picard puts it, "the greater the freedom of a machine, the more it will need moral standards" [102, p. 134].
For this reason, there has been a growing interest in Machine Ethics, de ned as the discipline "concerned with the consequences of machine behavior towards human users and other machines" [6, p. 1]. 1 Research in this eld is a combination of computer science and moral philosophy.As a result, publications range from theoretical essays on what a machine can or should do (e.g.[46,122]), to prototypes implementing ethics in a system (e.g., [3,142]).In this eld, the emphasis lies on how a machine could act ethically in an autonomous fashion.
ere have been several a empts to classify current approaches of machine ethics.A rst high-level classi cation was proposed by Allen et al. [2] in 2005, distinguishing between top-down theory-driven approaches, bo om-up learning approaches, and hybrids of the two.Subsequent work tried to further determine types of procedures [29,65,147], but these works were either mixing di erent dimensions (e.g., mixing technical approach and ethical theory in one category) [147] or o ering an orthogonal dimension that did not t the existing taxonomy (e.g., whether normative premises can di er between ethical machines) [29].Also, because these works did not provide an extensive and systematic overview of the application of their taxonomy, veri cation of the taxonomy with papers from the eld was missing.A recent survey from Yu et al. [65] on ethics in AI has some overlap with this work, but 1) does not systematically apply the ethical theory classi cation to selected papers and 2) takes a broader perspective to include consequences of and interaction with ethical AI, while this paper focuses speci cally on machine ethics implementations.Hence, compared to previous works, this survey covers more related work, provides a more extensive classi cation, and describes the relationship between di erent ethics approaches and di erent technology solutions in more depth than previous work [2,29,65,147].Furthermore, gaps are identi ed in the implementation of ethics in existing systems.
is paper is created as a collaboration between ethicists and computer scientists.In the context of implementing machine ethics, it can be a pitfall for philosophers to use a purely theoretical approach without consulting computer scientists, as this can result in theories that are too abstract to be implemented.Conversely, computer scientists may implement a faulty interpretation of an ethical theory if they do not consult a philosopher.In such an interdisciplinary eld, it is crucial to have a balanced cooperation between the di erent elds involved.
e contributions of this article are as follows: • Based on previous work [2], a taxonomy is de ned to analyze the eld based on three di erent dimensions: types of ethical theory (Section 4), implementation of ethics (Section 5), and technological details (Section 6).• An exhaustive selection and description of relevant contributions related to machine ethics implementations is presented (Section 8).

•
e reviewed publications are classi ed and research pa erns and challenges are identi ed (Section 7).
• A number of general lessons for the eld are discussed and further important research directions for machine ethics are outlined (Section 9).
As such, this survey aims to provide a guide, not only to researchers but also to those interested in the state of the art in machine ethics, as well as seed a discussion on what is preferred and accepted in society, and how machine ethics should be implemented.e rest of this paper is structured as follows.Section 2 introduces the eld of machine ethics, its importance, and justi cation of used terminology throughout the paper.Section 3 lists the methodology used to create this survey, including the search methodology, the process of creating the classi cation dimensions, and the actual classi cation process.Section 4, 5 and 6 introduce the three classi cation dimensions presented in this survey.Section 7 discusses the results of the classi cation of the selected papers, whereas Section 8 summarizes those papers.Finally, Section 9 outlines which future avenues of research may be interesting to pursue based on the analysis, as well as the limitations of this survey.

INTRODUCTION TO MACHINE ETHICS
Before going into more depth on the implementation of ethics, it is important to establish what is considered machines ethics, why it ma ers, and present the relevant terminology for the eld.

Relevance
So ware and hardware (combined under the term "machine" throughout this survey) are increasingly assisting humans in various domains.ey are also tasked with many types of decisions and activities previously performed by humans.Hence, there will be a tighter interaction between humans and machines, leading to the risk of less meaningful human control and an increased number of decision made by machines.As such, ethics needs to be a factor in decision making to consider fundamental problems such as the a ribution of responsibility (e.g., [122]) or what counts as morally right or wrong in the rst place (e.g., [135]).Additionally, ethics is needed to reduce the chance of negative results for humans and/or to mitigate the negative e ects machines can cause.
Authors in the eld give di erent reasons for studying (implementations in) machine ethics.Fears of the negative consequences of AI motivate the rst category of reasons: creating machines that do not have a negative societal impact [12,86].With further autonomy and complexity, ethics need to be implemented in a more elaborate way [24,38,49,75,95,99].Society needs to be able to rely on machines to act ethically when they gain autonomy [6,48].A second category of reasons for studying machine ethics focuses on the ethics part: by implementing ethics, ethical theory will be be er understood [24,58,95,99].Robots might even outperform humans in terms of ethical behavior at some point [4,8].
Some authors contend that in cases with no consensus on the most ethical way to act, the machine should not be allowed to act autonomously [5,122].However, not acting does not imply the moral conundrum is avoided.In fact, the decision not to act also has a moral dimension [55,78,144] -think, for example, of the di erence between active and passive euthanasia [106].Additionally, by not allowing the machine to act, all the possible advantages of these machines are foregone.Take, for example, autonomous cars: a large number of tra c accidents could be avoided by allowing autonomous cars on the road.Moreover, simply not allowing certain machines would not stimulate the conversation on how to solve the lack of consensus; a conversation that can lead to new, more practical ethical insights and helpful machines.

Terminology
An o en-used term in the eld of machine ethics is "Arti cial Moral Agent" or AMA, to refer to a machine with ethics as part of its programming.However, to see if this term is appropriate to use, it is important to identify what ethical agents mean in the context of machine ethics and how ethical machines should be regarded.In an o en-cited paper, Moor [95] de nes four di erent levels of ethical agents: Ethical-impact agents are types of agents that have an (indirect) ethical impact.An example would be a simple assembly line robot that replaces a human in a task.e robot itself does not do anything (un)ethical by acting.However, by existing and doing its task, it has an ethical impact on its environment; in this case, the human that performed the task is replaced and has to nd another job.Implicit ethical agents do not have any ethics explicitly added in their so ware.
ey are considered implicitly ethical because their design involves safety or critical reliability concerns.For example, autopilots in airplanes should let passengers arrive safely and on time.Explicit ethical agents draw on ethical knowledge or reasoning that they use in their decision process.ey are explicitly ethical, since normative premises can be found directly in their programming or reasoning process.Fully ethical agents can make explicit judgments and are able to justify these judgments.Currently, humans are the only agents considered to be full ethical agents, partially because they have consciousness, free will, and intentionality.
While these de nitions can help with a rst indication of the types of ethical machines, they do not allow for distinctions from a technical perspective and are also unclear from a philosophical perspective: Moor [95] does not actually de ne what an (ethical) agent is.For example, it can be debated whether an autopilot is an agent.erefore, a clearer de nition is needed of what an agent is.Himma [74] investigates the concepts of agency and moral agency, drawing from philosophical sources such as the Stanford Encyclopedia of Philosophy and Routledge Encyclopedia of Philosophy.He proposes the following de nitions: Agent : "X is an agent if and only if X can instantiate intentional mental states capable of performing actions." [74, p. 21] Moral agency : "For all X, X is a moral agent if and only if X is (1) an agent having the capacities for (2) making free choices, (3) deliberating about what one ought to do, and (4) understanding and applying moral rules correctly in the paradigm cases." [74, p. 24] With regards to arti cial agents, Himma postulates that the existence of natural agents can be explained by biological analysis, while arti cial agents are created by "intentional agents out of pre-existing materials" [74, p. 24].He emphasizes that natural and arti cial agents are not mutually exclusive (e.g. a clone of a living being).He further claims that moral agents need to have conscious and intentionality, something that state-of-the-art systems do not seem to instantiate.It is worth noting that Himma a empts to provide a general de nition of moral agency, while for example Floridi and Sanders [54] propose to change the current description of a moral agent.For example, they proposed description includes the separation the technical concepts of moral responsibility and moral accountability, a distinction that was not evident thus far: "An agent is morally accountable for x if the agent is the source of x and x is morally quali able […] To be also morally responsible for x, the agent needs to show the right intentional states".Wallach and Allen [137] rate AMAs along two dimensions: how sensitive systems are to moral considerations and how autonomous they are.Sullins [126] has a partially overlapping concept of requirements for robotic moral agency with Himma's, that intersects with Wallach and Allen's relevant concepts: autonomy (i.e."the capacity for self-government") [27]), intentionality (i.e."the directedness or 'aboutness' of many, if not all, conscious states" [27]) and responsibility (i.e."those things for which people are accountable" [27]).ese are just some notions of how concepts such as agency, autonomy, intentionality, accountability and responsibility are important to the eld of machine ethics.However, it is challenging to summarize and de ne these concepts concisely while doing justice to the work in philosophy, and computer science that has been done so far, including the discussions and controversy around di erent relevant concepts (as the di erent concepts of moral agency display).e goal of this survey is not to give an introduction to moral philosophy, so this section merely gives a glimpse of the depths of the topic.Rather, the goal is to summarize and analyze the current state of the eld of machine ethics.To avoid any assumption on concepts, the popular term Autonomous Moral Agent is not used in this survey: as shown above, the term "agent" can be debated in this context and the term "autonomous" has various meanings in the di erent surveyed systems.Instead, a machine that has some form of ethical theory implemented-implicitly or explicitly-in it is referred to as an "ethical machine" throughout this paper.

SURVEY METHODOLOGY
is section describes the search strategy, paper selection criteria, and review process used for this survey.

Search Strategy
A literature review was conducted to create an overview of the di erent implementations of and approaches to machine ethics.e search of relevant papers was conducted in two phases: automated search and manual search.Automated Search.e rst phase used a search entry that re ected di erent terms related to machine ethics combined with the word 'implementation': implementation AND ( "machine ethics" OR "arti cial morality" OR "machine morality" OR "computational ethics" OR "roboethics" OR "robot ethics" OR "articial moral agents") ese terms were cumulated during the search process (e.g.[139, p. 455]); each added term resulted in a new search until no new terms emerged. 2 No time period of publication was speci ed, to include as many items as possible.

Manual
Search.e second phase included checking the related work and other work by the same rst authors of phase one.Twenty-nine promising results from phase one did not meet all criteria, but were included in the second search phase to see if related publications did meet all criteria.is process was repeated for each newly found paper, until no more papers could be added that t the selection criteria (see 3.2). is resulted in a total of 49 papers, describing 48 ethical machines.

Selection Criteria
A er the selection process, two more coauthors judged which papers should be in-or excluded to verify the selection.Papers were included only if they adhered to all of the following inclusion criteria.e paper • implements a system OR describes a system in su cient (high-level) detail for implementation OR implements/describes a language to implement ethical cases, • describes a system that is explicitly sensitive to ethical variables (as described by [95]), no ma er whether it achieves this sensitivity through top-down rule-based approaches or bo om-up data-driven approaches (as described by [2]), • is published as a conference paper, workshop paper, journal article, book chapter, or technical report, • and has ethical behavior as the main focus.e following exclusion criteria were used.e paper • describes machine ethics in a purely theoretical fashion, • describes a model of (human) moral decision making without an implementable model description, • lists results of human judgment on ethical decisions without using the data in an implementation, • is published as a complete book, presentation slides, editorial, thesis, or has not been published, • describes a particular system in less detail than other available publications, • focuses on unethical behavior to explore ethics (e.g., a lying program), • mentions ethical considerations while implementing a machine, but does not focus on the ethical component and does not explain it in enough detail to be the main focus, • simulates arti cial agents to see how ethics emerge (e.g. by using an evolutionary algorithm without any validation), • and describes a general proposal of an ethical machine without mentioning implementation related details.
Given the focus on algorithms implementing moral decision making and the limitations of space, we will not go into further detail as regards recent interesting work on AI and moral psychology (cf.e.g.[17,28,85,118]).

Taxonomy Creation and Review Process
In order to be able to identify strengths and weaknesses of the state of the art, we created di erent taxonomies and classi ed the selected papers accordingly.It was clear that both an ethical taxonomy (cf.Table 1) and a technological taxonomy (cf.Table 4) were necessary.All authors agreed that there were some aspects of implementations that did not relate exclusively to either the ethical or the technical dimension but were still important for the classi cation.Hence, we de ned and applied a third taxonomy (cf.Table 3) related to implementation choices.e rst version of these three taxonomies was created using knowledge obtained during the paper selection.
Before the classi cation process started, one third of the papers were randomly selected to review the applicability of the taxonomy proposed and adjust the assessment scheme where required.Any parts of the taxonomies that was unclear and led to inconsistent classi cations was adapted and clari ed.
Each selected paper was categorized according to the features of the three di erent taxonomies (discussed in Sections 4-6).Since the ethical classi cation is perhaps the most disputable, it was determined by three distinct assessors: two philosophers and a computer scientist.Two computer scientists evaluated the implementation and technical details of all proposed systems.
To provide a classi cation for all selected papers, multiple classi cation rounds took place for each dimension.In between classi cation rounds, disagreements across reviewers were discussed until a consensus was reached.In the case of the ethical dimensions, four papers could not be agreed upon a er multiple classi cation rounds.As such, these papers were labeled as 'Ambiguous'.

ETHICAL THEORIES
is section introduces the rst of three taxonomies introduced in this paper: a taxonomy of types of ethical theories, which is the basis for the categorization of ethical frameworks used by machines (in Section 7).Note that this section is not a general introduction to (meta-)ethics, which can for example be found in [26,41,94].

Overview of Ethical Theory Types
It is commonplace to di erentiate between three distinct overarching approaches to ethics: consequentialism, deontological ethics, and virtue ethics.Consequentialists de ne an action as morally good if it maximizes well-being or utility.Deontologists de ne an action as morally good if it is in line with certain applicable moral rules or duties.Virtue ethicists de ne an action as morally good if, in acting in a particular way, the agent manifests moral virtues.Consider an example: an elderly gentleman is harassed by a group of cocky teenagers on the subway and a resolute woman comes to his aid.e consequentialist will explain her action as good since the woman maximized the overall well-being of all parties involved-the elderly gentleman is spared pain and humiliation which outweighs the teenagers' amusement.e deontologist will consider her action commendable as it is in accordance with the rule (or duty) to help those in distress.e virtue ethicist, instead, will deem her action morally appropriate since it instantiates the virtues of benevolence and courage.
Consequentialist theories can be divided into two main schools: according to act utilitarianism, the principle of utility (maximize overall well-being) must be applied to each individual act.Rule utilitarians, by contrast, advocate the adoption of those and only those moral rules that will maximize well-being.Cases can thus arise where an individual action does not itself maximize well-being, yet is consistent with an overarching well-being maximizing rule.While act utilitarians would consider this action morally bad, rule utilitarians would consider it good.
Deontological ethics can be divided into agent-centered and patient-centered approaches.Agentcentred theories focus on agent-relative duties, such as, for instance, the kinds of duties someone has towards their parents (rather than parents in general).
eories of this sort contrast with patient-centered theories that focus on the rights of patients (or potential victims), such as the right, postulated by Kant, not to be used as a means to an end by someone else [81].Finally, there are some approaches that question the universal applicability of general ethical principles to all situations, as put forward by deontological ethics, virtue ethics or consequentialism.For such a particularist view, moral rules or maxims are simply vague rules of thumb, which cannot do justice to the complexity of the myriad of real-life situations in which moral agents might nd themselves.Hence, they have to be evaluated on a case-by-case basis.
We would like to highlight that a moral theory is a set of substantial moral principles that determine what, according to the theory, is morally right and wrong.Moral theories can take di erent structures -they might state their concrete demands in terms of hard rules (deontological ethics); virtues that should guide actions, with reference to an overall principle of utility maximization, or else reject the proposal that there is a one-size-ts-all solution (itself a structural trait, this would be particularism).In this work, we are interested in these structures, which we label "ethical theory types".

Categorizing Ethical Machines by Ethical Theory Type
Based on the distinct types of ethical theories introduced above, this sub-section develops a simple taxonomy for categorizing ethical machines, summarized in Table 1.An evaluation of existing approaches to moral decision making in machines can make use of this typology in the following way.Deontological ethics is rule-based.What ma ers is that the agent acts in accordance with established moral rules and/or does not violate the rights of others (whose protection is codi ed by speci ed rules).Accidents occur, and a well-disposed agent might nonetheless bring about a harmful outcome.On o -the-shelf deontological views, bad outcomes (if non-negligently, or at least unintentionally, brought about) play no role in moral evaluation, whereas the agent's mental states (their intentions, , and beliefs) are important.If John, intending to deceive Sally about the shortest way to work, tells the truth (perhaps because he himself is poorly informed), a Kantian will consider his action morally wrong, despite its positive consequence. 3In the context of machine ethics, the focus is solely on agent relative duties.Hence, no distinction is made between agent-centered and patient-centered theories of deontological ethics in the taxonomy summarized in Table 1.
Consequentialists, by contrast, largely disregard the agent's mental states and focus principally on outcomes: what ma ers is the maximization of overall well-being.Note that, procedurally, a rule-utilitarian system can appear very similar to a deontological one.e agent must act in keeping with a set of rules (potentially the very same as in a Kantian system) which, in the long run, maximizes well-being.However, the two types of systems can still be distinguished in terms of the ultimate source of normativity (well-being vs. good will) and will-standardly-di er in terms of the importance accorded to the agent's mental states.us far, nearly all consequentialist machine ethics implementations utilize act utilitarianism.For this reason, the distinction between act and rule utilitarianism is not relevant enough to be included in this survey.
Virtue ethics di ers from the aforementioned systems in so far as it does not focus principally on (the consequences or rule-consistency of) actions but on agents, and more particularly on whether In contrast to the other three major approaches, on the particularist view, there is no unique source of normative value, nor is there a single, universally applicable procedure for moral assessment.Rules or precedents can guide our evaluative practices.However, they are deemed too crude to do justice to many individual situations.us, according to particularism, whether a certain feature is morally relevant or not in a new situation -and if so, what exact role it is playing there-will be sensitive to other features of the situation.
Table 2 gives a schematic overview of key characteristics of the di erent types of ethical systems that might be implemented in an ethical machine.Note that it does not take some of the more ne-grained aspects di erentiating the theories (e.g., the before-mentioned complications regarding act and rule utilitarianism) into account.
As an alternative to implementing a single determinate type of ethics, systems can also combine two or more types, resulting in a hybrid ethical machine. is approach seems enticing when one theory alleviates problems another one might have in certain situations, but it can also generate con icts across types of ethical approaches.Hence, some proposals enforce a speci ed hierarchy, which means that one theory is dominant over the other(s) in the system.For example, a primarily deontological system might use rules, but turn to the utilitarian approach of maximizing utility when the rules are in con ict.In other cases, the hierarchy is non-speci c and di erent theories are present without a speci ed dominant theory.Some authors do not se le on a particular type of ethical theory.Instead, they provide a con gurable technical framework or language and exhibit how di erent types of ethical theories can be implemented.e choice of which theory type should be selected is essentially le to the person implementing the system in an actual use case.
Finally, some contributions were classi ed as ambiguous from a meta-ethical perspective.For these, not enough details were given by the authors to classify a paper, or the theories used to implement were not ethical theories but retrieved from domains other than moral philosophy.
e next section introduces the second dimension of ethical machines: implementing ethics into a machine.

IMPLEMENTING ETHICS
e second taxonomy that was created for this survey considers the implementation aspect of machine ethics.An important part of creating an ethical system is to decide how to implement ethics.at entails de ning whether an implementation can follow di erent approaches, how to evaluate the system, and whether or not domain speci cations need to be taken into account.Important features concerning the implementation dimension are summarized in Table 3.Furthermore, this section highlights the implementation challenges that the various ethical theories entail.

Approaches
Di erent taxonomies have been proposed to determine how ethics types are implemented.e most in uential and widely referenced scheme, also applied in this survey, stems from Allen, Smit and Wallach [2].ey distinguish three types of implementation approaches, namely top-down, bo om-up, and hybrid.
Top-down approaches : Top-down approaches assume that humans have gathered su cient knowledge on a speci c topic; it is a ma er of translating this knowledge into an implementation.e ethical theory types described in Section 4 are examples of normative human knowledge that can be translated into usable mechanisms for machines.
e system acts in line with predetermined guidelines and its behavior is therefore predictable.In AI, strategies using a top-down approach mostly make use of logical or case-based reasoning.Given general domain knowledge, the system can reason about the situation that is given as input.Usually, human knowledge is not speci ed in a very structured or detailed way for concrete cases, so knowledge needs to be interpreted before it can be used.is process presents the risk of losing or misrepresenting information.e positive aspect of this approach is that existing knowledge is applied and no new knowledge needs to be generated.Bottom-up approaches : A di erent method to implementing ethics is to assume the machine can learn how to act if it receives as input enough correctly labeled data to learn from. is approach, not just in machine ethics but in general, has gained popularity a er the surge of machine learning in AI and the recent success of neural networks.Technologies such as arti cial neural networks, reinforcement learning, and evolutionary computing fall under this trend.Increased computing power and amounts of data allow learning systems to become more successful.However, data has to be labeled consistently and the right data properties need to be described in a machine-processable way to obtain an accurate training of machines.ere is a risk that the machine learns the wrong rules or cannot reliably extrapolate to cases that were not re ected in its training data.However, for certain tasks, such as feature selection or classi cation, machine learning can be very successful.Hybrid approaches : As the term suggests, hybrid approaches combine top-down and bo om-up approaches.As Allen et al. phrase it: "Both top-down and bo om-up approaches embody di erent aspects of what we commonly consider a sophisticated moral sensibility."[2, p 153] ey indicate that a hybrid approach is considered necessary, if a single approach does not cover all requirements of machine ethics.e challenge consists in appropriately combining features of top-down and bo om-up approaches.
Bonnemains et al. [29] suggest adding a fourth category, called "Personal values/ethics system".Essentially, it acknowledges that two di erent agents may rely on di erent ethical systems or may rely on di erent precedence in case of con icts in a hybrid system.In this survey, this is regarded as diversity consideration: the authors of a machine ethics paper consider the possibility that not all ethical machines adhere to the same ethical theory type, and their contribution includes the choice of diverse types of ethics to be implemented.As Bonnemains et al. recognize, this category is somewhat orthogonal to the previous three, as all of those can be seen to implement distinct normative principles.For example, a machine ethics implementation with diversity consideration could allow for multiple ethical theory types to be implemented (i.e., a top-down approach) or allow for di erent machines to learn di erent types of ethics (i.e., a bo om-up approach).It is considered part of the implementation dimension rather than the ethics dimension since diversity considerations can also exist within the same ethical theory, for example, by allowing deontological machines to have di erent rules to adhere to while still all being deontological in nature.is survey regards structures of normative frameworks and their implementation rather than substantial normative principles (c.f.Table 3).

Type of Contribution
Ethical systems can be intended to enact di erent aspects of ethical behavior.is section discusses the di erent types of contributions published to implement ethical machines.
Model representation : is contribution type focuses on representing current ethical knowledge.
e goal is to determine how to appropriately represent a theory, dilemma, or expert-generated guidelines whilst staying true to the original theory.Model selection : Given a set of alternative options to implement an ethical machine, some systems limit their action to selecting the most ing elements to be included in the system.Judgment provision : ese contributions focus on judging an action given a scenario and a set of possible actions.Example outputs are binary (acceptable/non-acceptable) or responses on a scale (e.g., very ethical to very unethical).Action selection/execution : Here the proposed system chooses which action is best given multiple possible actions for a scenario.Some systems then assign the action to a human, while others carry out the selected action themselves.Part of the action selection task can also be action restriction, when some possible actions are not morally acceptable (enough).

Evaluation
Most artifacts-simple or complex, concrete or abstract-can be evaluated in virtue of their capacity to ful ll their constitutive function or purpose.A good knife cuts well, a good thermostat reliably activates the heating if the temperature drops below a predetermined threshold, and a good translation system adequately and idiomatically converts grammatical sentences from one language into another.Whereas there are objective and measurable criteria for the evaluation of thermostats, things are more cumbersome when it comes to moral machines.is is not because their purpose does not standardly consist in simply "acting morally", but in executing certain tasks (taking care of the elderly, counselling suicidal people, evaluating risk of recidivism etc.) in a moral fashion.Much rather, the complication arises from the question of what exactly is to count as executing the task at hand in morally appropriate ways, or against what exactly the behavior of the system should be evaluated.ere are objective facts as to whether an image represents a certain type of animal or not.ese facts constrain whether the image is correctly classi ed as representing an animal.e existence of objective, universal moral values, by contrast, is controversial (cf.e.g., [72,88,105,133]).Furthermore, and as objectivists readily acknowledge, delineating what is morally permissible poses an epistemic challenge of a di erent order than identifying, say, a gira e in an image, or determining the weight of an object.e ontological and epistemic complications that arise in the moral domain thus make it di cult to se le on standards against which the performance of a moral machine could be evaluated.More fundamentally, it is not even evident what kinds of considerations should guide the process of choosing such standards.
While complications as to the evaluation of a moral machine are worrying, their practical signi cance should not be exaggerated.Although there is disagreement as regards complex cases, in ordinary life situations in which one is confronted with extremely di cult ethical decisions or run away trolleys are exceedingly rare.In many domains, moral dilemmas are unlikely to arise or be of much import, and there is widespread convergence (not only among the folk, but experts, too) on what constitutes adequate moral behavior.Overall, then, the challenge of evaluation might raise metaphysical and epistemic complications of limited pragmatic importance, at least when care is exercised to limit the decision capacity of moral machines to mundane contexts that steer clear of complex ethical paradoxes.

5.
3.1 Test.When a system is tested, the system outcome needs to be compared against a ground truth.ese may have the following origins: Non-experts : One possibility consists in making folk morality the benchmark.Problematically, there is substantial evidence of moral parochialism across cultures (e.g., [53,87,113]), and it is not di cult to nd topics on which a single nation is roughly divided -just think of abortion, euthanasia, or same-sex relations in the US [112].Furthermore, the existence of widespread convergence in moral opinion does not necessarily make such opinions true or acceptable (consider that until a century and a half ago, there was broad agreement in considerable parts of the world that slavery is morally acceptable).Experts : To escape the tyranny of a potentially mistaken or self-serving majority, one might adopt the standard of experts in normative ethics.Problematically, however, experts themselves are sometimes deeply divided on fundamental issues of moral import as well as meta-ethical intuitions [31] and their very expertise can be called into question [119,120].Laws : One might side-step the complications raised by retreating to a second-best solution: the law. is strategy, however, is not without drawbacks either, as the law is simply silent on most questions of day-to-day morality.It is, for instance, not illegal to lie in most contexts, yet it would be regarded as outrageous to be perpetually deceived by "moral" machines.Still, it might be suitable to draw on the law to provide restrictions where they exist, for example, as concerns the "Laws of War" or "Laws of Armed Con icts" for the lethal weapons domain [8], or speci c domain rules such as the Code of Ethics for Engineers [91].As Arkin [8] suggests, scoping the problem using domain-speci c requirements can make it more easily implementable and testable.

Prove.
Another approach, typically based on some type of logic, consists of proving that the system behaves correctly according to some known speci cations. is approach can be divided into the following types: Model checker : Given an ethical machine, a model checker exhaustively and automatically ascertains that it adheres to a given set of speci cations.Logical proof : is approach provides a logical proof that given certain premises, the system does what it should do.Proofs of this sort can be e ected manually, or by using a theorem prover which employs automated logical and mathematical reasoning.
Note that this approach assumes that a correct speci cation exists a priori and is widely accepted.Within the logic community, model checking and theorem proving are o en considered an implementation issue rather than a type of evaluation (e.g., see [68]).In some cases, authors do not even explicitly mention that they employ a model checker, because it is inherent in their approach to logic programming.However, given the multidisciplinary nature of the eld of machine ethics, it is vital to explicitly state which approach has been used.Furthermore, while logical/internal validity and consistency may be inherent in the system, a form of evaluation is necessary to ensure the system acts as expected in di erent cases and exhibits external validity.

5.3.3
Informal evaluation.Some authors refrain from formally evaluating their implementation.Instead, they only describe their work and, in some cases, show a few example scenarios or exhibit application domains.Whilst these approaches may have limited validity, they may be warranted given the evaluation complications outlined above or when the authors principally engage in theory building [50].
Example scenarios/case studies : To showcase that the system works as intended, one or multiple scenarios are presented to demonstrate the system's performance.is procedure gives a rst indication of the functionalities of the machine or may help in theorizing about certain properties of a system, but it does not cover all possible situations or give a complete performance indication.Face validity : O en described as "reasonable results", authors using this approach state that the results of a few example tasks are as expected.It is o en unclear what this means and to what extent these results are desirable.

None.
When no evaluation could be discerned, papers were categorized as having none of the evaluation types present.

Domain Specificity
What is deemed an appropriate action can depend on the domain in which the ethical agent is operating, such as the principles in the domain of biomedical ethics as proposed by Beauchamps and Childress [21] for the medical domain, or the Rules of Engagement and Laws of Armed Con ict for autonomous weapon systems [8].Hence, some contributions focus on a speci c application domain, which limits the scope of an ethical machine implementation, and thus the endeavor is more manageable [8].

Domain speci c
Yes (domain speci ed) No

Implementation Challenges
ere are certain challenges inherent in the di erent types of ethics when they are being implemented.Since these obstacles need to be taken into account to select an ethical theory type for an ethical machine, this subsection provides a (non-exhaustive) list of complications.
Challenges when implementing deontological ethics: At a rst glance, the rule-based nature of deontological ethics seems to lend itself well for implementation.However, at di erent stages of implementation, challenges arise.e rst issue is which rules should be implemented.Rules are expected to be strictly followed, implying that for every exception, the rule must be amended, resulting in an extremely long rule.Determining the right level of detail is important for the success of an application: when the rules are not practical and at the right level of detail, they will not be interpretable for the machine [8].Second, there might be con icts between rules [29]-in general or in speci c situations.Whilst ordering or weighing the rules might address this issue from an implementational perspective, determining an order of importance can be di cult.Also, this assumes that all relevant rules are determined before they are used.
Challenges when implementing consequentialist ethics: ere are three main categories of di culties for consequentialist ethics.First, it is hard to identify consequences and determine the right level of detail and aggregation in terms of time and size.Some outcomes might have resulted regardless of the action theorized to have caused it.In real-life situations, all possible consequences are not always that clear beforehand given the lack of epistemic transparency and causal interdependence.
A second issue is concerned with quantifying consequences.As consequentialism is about maximizing utility, the problem is how to de ne utility.In simple scenarios like the Trolley problem, utility is o en de ned as how many people survive or die.In the real world, more complex concepts, such as happiness and well-being, are preferred to de ne utility.ere are measures available (e.g., QALY [64]), but using a di erent measure can give a di erent outcome.Even more so, even if each consequence is assigned a utility, it might still be inappropriate to simply aggregate them (e.g., see [83]).
Finally, there might be a signi cant computational cost when computing utility [141] requiring heuristics or approximations to derive a correct answer in time.is, in turn, requires a veri cation of whether these results are still correct.
Challenges when implementing virtue ethics: Virtues are positive character traits, character traits that should be manifested in morally good actions.De ning what "character" a machine has is troubling, if a machine can be claimed to have a character at all.To judge whether a machine-or a human for that ma er-is virtuous is not possible by merely observing one action or a series of actions that seem to imply that virtue; the reasons behind them need to be clear [123].Perhaps the best way to create a virtuous machine is to let a machine mimic the behavior of a virtuous person.But how is a certain virtue measured, and who decides which virtues are more important and how to pick the perfect role model?Coleman [39] even proposes di erent virtues that are more desirable for machines rather than human virtues, implying merely mimicking a virtuous person is not su cient.
To circumvent these challenges, machine ethics researchers have not used virtue ethics o en, as the alternatives might be more appealing.For example, Haber [66] states that virtue ethics and principle-based ethics are complements and that for each trait there will be a principle that expresses that trait and vice versa.While not everyone agrees with Haber, it is easier and more detailed from a computational perspective to implement rules than generic virtues to adhere to.Arkin [8] also concludes that principle-based and act-centric models allow for stricter ethical implementations, which is desirable in machine ethics.
Challenges when implementing particularism: In particularism, the system needs to take the entire context into account.is implies that it needs to either be trained for all possible cases, which is not possible, or be able to extrapolate without using generalizations, which is highly challenging.For each feature of the context, the system would have to recognize whether it is morally relevant in the given case and how it will in uence the result.Case-based methods or instance-based classi cations come closest to allowing an implementation of particularism.More recently, some contributions are trying to approximate particularist ethics using neural networks (e.g., [62,71]).
Challenges when implementing hybrid approaches: Each type of ethical theory raises its own set of complications, but combining them introduces additional issues.First, when di erent types of ethical theories are used in a non-hierarchical way, the interaction between them can be problematic: how should the results from di erent ethical approaches be combined to guarantee morally appropriate outcomes?What happens when the results of di erent implemented ethical theory types stand in con ict, and how should such con icts be resolved?
Second, when a hierarchical approach is employed, it is not evident when the system should employ one theory rather than another.One standard approach resorts to the secondary set of ethical principles when the rst does not deliver a verdict.While this alleviates some of the challenges of hybrid systems, it is still possible that the second ethical theory proposes something that con icts with the rst ethical theory type.

TECHNICAL DIMENSION
e third and nal taxonomy introduced regards the technical dimension of machine ethics.is includes the type of technology chosen for the implementation, the input the system relies on, the ethical machine's availability (i.e., implementation details are published) and some implementation features: whether it relies on speci c hardware or feedback from users, provides explanations for its conclusions, has a user interface (UI), and whether the input for the system needs to be preprocessed.Important features pertaining to the technical dimension are surveyed in Table 4.

Types of Technology
Inspired by Russell and Norvig [110], di erent types of technologies can be distinguished.While these types of technology are not always clearly delimited, this categorization allows comparing implementations.Deductive logic : is is the classical type of logic: knowledge is represented as logical statementspropositions and rules-that allow deriving new propositions.Pure deductive systems typically involve no learning or inference involved but only derive what can be known from their set of statements and inputs.Non-monotonic logic : Non-monotonic logic allows the revision of conclusions when a con ict arises, for example, in light of new information.Abductive logic : In abductive logic, the conclusions drawn are the most likely propositions given the premises.Deontic logic : is type of logic stems from philosophy and is speci cally designed to express normative propositions.Naturally, this type of logic is inherently suited for the representation and deduction of moral propositions.
Rule-based systems : As the name suggests, rule-based systems are systems that function based on a set of rules.ese can be ethical rules the system has to adhere to.Note that many of the di erent types of logic above are typically implemented as some form of rule-based system.Event calculus : Event calculus allows reasoning about events.When a machine needs to act ethically, di erent events can trigger di erent types of behavior.Knowledge representation (KR) and ontologies : A KR approach focuses on representing knowledge in a form that a computer system can utilize.In other words, the emphasis lies on improving the quality of the data rather than (just) improving the algorithm.Inductive logic : When relying on inductive logic, premises are induced or learned from examples, rather than pre-de ned by a human.
6.1.2Probabilistic reasoning.Recently, probabilistic reasoning has gained more a ention.Different types of probabilistic reasoning approaches can be distinguished.
Bayesian approaches Based on Bayes' rule, these approaches rely on prior knowledge to compute the likelihood of an event.In an ethical context, a machine can then act based on this predicted information.Markov models Markov models focus on sequences of randomly changing events, assuming that a future event only depends on the current (and not the previous) event(s).Statistical inference By retrieving probability distributions from available data, the system can try to predict the chances of future events happening.

6.1.3
Learning.e increased computational power, the amounts of data available, and the GPU-driven revival of neural networks have made learning systems more popular.
ere are di erent learning approaches to be characterized.
Inductive logic : In inductive logic, a rule-base for reasoning is learned.As such, it is listed under both the "Logic" and "Learning" categories of this taxonomy.Decision tree : Decision trees are a supervised learning method to solve a classi cation problem by exploring the decision space as a search tree and computing the expected utility.ey are, thus, useful to identify and interpret the features that are most important to classify cases.Reinforcement learning : A system can learn from its actions when they are reinforced with rewards or punishments received from its environment.Neural networks : A neural network can be trained on many cases, to be able to classify new cases based on their relevant features.Evolutionary computing : Evolutionary algorithms are used when, for example, di erent competing models of an ethical machine exist.Models evolve in an iterative fashion, based on actions inspired from the concept of evolution in the eld of biology (e.g., selection, and mutation) [79] 6.1.4Optimization.e most common form of optimization relies on a closed-form formula for which some optimal parameters are sought.Di erent actions get assigned di erent values based on a predetermined formula, and the best value is chosen (e.g., the highest value).
6.1.5Case-based reasoning.In case-based reasoning, a new situation is assessed based on a collection of prior cases.Similar cases are identi ed and their conclusions are transferred to apply to the current situation.

Input
To be able to respond appropriately, ethical machines need to receive information about the environment (or situation at hand).Input is the information that the system receives, not the transformation the system itself performs on the data a erwards.
Sensor data : In the case of (simulated) hardware, the machine perceives the input through its sensors.e sensor data is interpreted and processed to serve as the input.Case: Logical representation : Systems using a form of logic o en need an input case represented using logic.Case: Numerical representation : Other systems, for example ones using neural nets, need their input in a numerical form. is can be a vector representation or a set of numbers.Case: Language representation : Language inputs can be natural language or input translated into structured language.

Implementation availability
As mentioned before, part of the eld of machine ethics tends to be of a theoretical nature.is becomes apparent in the level of detail of implementation proposals.While some authors implement an idea and provide the source code, this is fairly rare in machine ethics.Some authors only give a few implementation details, and others merely specify a high-level description of their idea.Usually, the focus lies on sketching an idea rather than its complete implementation.
Speci cation details : is level has the fewest implementation details: the author speci es the proposed idea (e.g., textually) without any additional detail.Implementation details : is next level provides implementation details illustrating how the speci cation is implemented in the described machine.Code (link) provided : is nal level provides the link to the code of the machine, so the prototype can be used and the experiments can be replicated.

Other Implementation Categories
is section introduces di erent and independent categories that are of interest for the implementation of an ethical machine.
6.4.1 Hardware.Robots can have direct physical results rather than "just" digital or indirect physical consequences.Hardware can change the way people interact with a system and how it should be able to function, making it an interesting and important feature to classify.

Feedback.
No ma er which ethical approach is used, feedback is a valuable component of an ethical system.For example, the user can be asked whether the provided output was the best given the input or whether the system was clear during its decision process.

Explanation.
Transparency is important when it comes to algorithmic decisions, both from a user perspective [149] and, in some cases (such as the General Data Protection Regulation in the European Union [60]), from a legal perspective.To achieve this goal, an understandable explanation should be provided by the system.

User Interface (UI).
Systems should be easy to interact with.
is is important for all machines, including ethical machines.6.4.5 Automated processing.Sometimes, initial prototypes focus on the concept of a system, not the (detailed) implementation, and may require some pre-processing of the input data.Ideally, systems should be able to process input from the environment automatically.

ANALYSIS
e goal of this section is to classify the surveyed moral machines according to the taxonomies introduced in Sections 4-6 and elicit pa erns in the literature based on this classi cation.Specifically, every publication is categorized according to the ethics, implementation, and technology taxonomies (as described in Section 3).

Ethical Classification
e classi cation results for the ethical dimension of machine ethics implementations can be found in Table 5; the ratio of single vs. hybrid theory papers is visualized in Figure 1.Among the papers, several constitute clear-cut cases instantiating one of the four main ethical systems.For example, [5,97,121] are clearly deontological; and [1,37,49,132] constitute uncontroversial examples of consequentialist systems.Furthermore, a considerable number of papers invoke elements from multiple systems.Finally, there are papers in which the hierarchy across theory types remains ambiguous.Examples of ambiguous papers are implementations where authors try to mimic the human brain [34], or focus on implementing constraints such as the Pareto principle [86], which does not strictly speaking constitute a moral theory.Note that categorizing a paper as "ambiguous" does not imply a negative assessment of the implementation.It simply means that the proposal cannot be adequately placed within our classi cation framework.About 50% of the proposals draw on a single type of ethical theory (see Figure 1).As can be seen in Table 5, deontological and consequentialist ethics are used most o en.It stands out that particularism is barely used and pure virtue ethics is not used at all. is may be explained as follows: rst, a generalist approach is much easier to implement than a particularist approach, as it is more straightforward to encode generalist rules than to build systems that may have to handle as of yet unknown, particular cases.Second, virtue ethics can be considered a very high-level theory focusing on characteristics rather than actions or consequences, which is di cult to interpret in an application context.go beyond the two main types of theory.Examples are virtue ethics and particularism [79], and a re ective equilibrium approach that combines consequences, rules, and other in uences [147].
A li le less than 10% of the papers do not have a speci c theory implemented.Instead, they provide various proposals on how to implement di erent ethical theory types without choosing a particular one.is can be considered a computer scientist approach, where the goal is to devise a general framework which the users can adapt to their preferences.
It is surprising that despite previous calls that a single classical theory is not enough to create an ethical machine and hybrid methods are needed (e.g., [2]), there is relatively li le work on hybrid ethical machines.While most hybrid systems have emerged over the last ten to een year, we could not nd evidence for an increase in the creation of such systems.

Implementation Classification
Table 6 provides an overview of the classi cation of the implementation dimension.
Approximately consistent with the number of single theory and hybrid theory approaches identi ed in Section 8.1, most authors choose a top-down approach.Hybrid approaches account for a li le less than 25% of those chosen (see Figure 2a).
Most authors use a general approach to machine ethics: almost three out of four do not use a domain-speci c approach, but focus on a general proposal of implementing machine ethics (see Figure 2b).
In terms of contribution type, there is a relatively balanced division between authors investigating how an ethical machine should be shaped (model selection and model representation) and authors focusing on the output of the ethical machine (action judgment and action selection/execution, see Figure 2c).Most papers address action selection/execution.About 15% of all the papers focus on action judgment: the system judges a situation but leaves it up to the human to actually act on this.From a broader scienti c perspective, it is good that both model shaping and output-oriented contributions are investigated.However, it would be ideal to have both things connected.
A possibility for future improvement regards system evaluation: over half of the authors either provide no or only an informal evaluation of their system.Of the rest, about 50% use a test approach and 50% validate their claims with some form of formal proof (see Figure 2d).
Finally, about half of the selected papers (51%) acknowledge diversity in implementable ethics, while the other half presents work allowing for or assuming only one ethical theory type.7. Of the di erent techniques, logical reasoning is the most frequent.Figure 3a shows the distribution of types of technology used.About a quarter of the papers adopt more than one technology type.Only about 10% of the authors focused on a pure learning approach.Case-based reasoning and probabilistic reasoning are the least popular.Mostly classical AI approaches are used-perhaps due to the direct correspondence of rules with deontological ethics.
e level of implementation detail provided is somewhat limited (see Figure 3b): although most authors include a speci cation of their idea in the paper, implementation details (or even source code) are rarely included.Both from a computer science perspective and a general science perspective, this is quite undesirable, as it hampers the reproducibility and extensibility of systems and empirical studies.
e di erent types of input used are fairly distributed: in about 36% of the ethical machines the input is de ned as logical cases, in 21% the input has a numerical representation, in 30% the input is wri en in (natural or structured) language, and 34% use (simulated) sensor data as input.Of all cases, ve selected papers had more than one type of input for their system.Around 25% of the authors used a (simulated) robot, corresponding with the amount of sensor data used as input.
In terms of user friendliness, the implemented systems score poorly.While it is important to note many of these machines are in their prototype phase and more focused on the ethics than the user, it should be important to keep the user in mind from the start of development.Nearly 35% of the machines provide an explanation of their output.27% process the input automatically, implying that about three out of four implementations require the user to pre-process the input manually in some way -which does not make it easy for the user.Only around one out of ve machines include a user interface and less than 17% o er the option for the user to give feedback.In summary, there is still plenty of room for improvement as regards user friendliness.

Interactions Between Dimensions
Given that machine ethics is an interdisciplinary eld, it is interesting to look at the interaction between the ethical theory types and their implementation, Figure 4 shows the interactions between ethical theory, ethical implementation approach, and technology type used to implement an ethical machine.For researchers, it can be useful to see which combinations have not yet been tried out that might be promising.For example, Figure 4a shows that (to the best of our knowledge) a hybrid approach (including both top-down and bo om-up elements) to implementing pure consequentialism does not yet exist.Similarly, bo om-up approaches to optimization (see Figure 4b) or pure deontological approaches to learning (see Figure 4c) (e.g., seeing which input leads to behavior adherent to a certain set of rules) have not yet been explored.7.5 General Observations ere are some general observations to be made about the eld.Firstly, the focus is on one universal and objective ethical agent.ere are no options for adding cultural in uences or societal preferences in any of the classi ed papers.All systems assume the user cannot in uence the output of the system.A recent publication shows indication of cultural di erences in ethical preferences [17], and the development of societal preferences within an ethical machine would improve the chance of acceptance of ethical machines.However, it is still under debate whether the eld should move towards a "universal moral grammar, " such as that proposed by Mikhail [93].
Secondly, there are some issues inherent to the eld.For instance, there are no benchmarks to verify if a system is working as it should.ere are no speci c tasks to be implemented, no consensus as to what the correct output is, and few data sets to use in an implementation.A helpful tool to recur to in this context is the work by Whitley [140], who provides a four-dimensional schema for analyzing a research eld.Two of the dimensions refer to the uncertainty of the task at hand, and two refer to the mutual dependence between the elds and scientists in them.e eld of machine ethics scores highly on all of these dimensions: High technical task uncertainty: there is unpredictability and variability in which methods are used in the eld and how results are interpreted.In this regard, it is a fragmented eld.High strategic task uncertainty: there are problems present in the eld that are valued di erently (e.g., some authors focus on the theoretical, others on the implementation, and the ethical theories or even ethical theory types they focus on diverge).High strategic dependence: there is much disagreement on the relevant topics, so there is a high reliance on peers for validation and reputation in the eld.Medium functional dependence: in terms of physical dependence of resources, there is none.Anyone with a computer can add to the eld; no expensive equipment is needed.However, there is a high dependence on results of others and acceptance by the eld.Another potentially helpful perspective can be derived from Whitley's theory, where the eld of machine ethics would be a "polycentric oligarchy", implying there are several independent clusters of scholars that con rm each other's assumptions and do not communicate much with other clusters that have very di erent views.At rst glance, such clusters can indeed be detected: the multi-agent norm domain (e.g., [97,128]), the logical translation of ethical theories (e.g., [61,69,99]) or the modern learning approach to machine ethics (e.g., [1,146]).While exploratory research in many directions is valuable, the eld would bene t from more standardization and more communication between clusters to exchange knowledge on ethics and technology.

SELECTED PAPER DESCRIPTIONS
For readers that are interested in a more detailed description of the classi ed papers, the following section provides a short summary of each of the selected papers.To structure their presentation, the papers were categorized across two orthogonal dimensions: (i) implementation (top-down, bo om-up, and hybrid, cf.[138]), and (ii) type of ethical theory (deontological, consequentialist, virtue ethics, particularism).

Top-Down
is section introduces papers that adopt a top-down approach to implementing ethics.
8.1.1Deontological Ethics.Among top-down deontological approaches, di erent kinds can be distinguished: papers that use predetermined given rules for a certain domain, papers focusing on multi-agent systems (MAS), and other papers that do not t either of these two categories.Domain rules.In the medical domain, Anderson and Anderson [3] use an interpretation of the four principles of Beauchamp and Childress [21] from earlier work by Anderson et al. [7] to create an ethical eldercare system.e system, called Ethel, needs to oversee the medication intake of patients.Initial information is given by an overseer, including, for example, at what time medication should be taken, how much harm could be done by not taking the medication, and the number of hours it would take to reach this maximum harm.Shim et al. [121] also explore the medical domain, but focus on mediating between caregivers and patients with Parkinson's disease.Instead of a constraint-based approach from previous work, their paper builds on the work by Arkin [8], who employs a rule-based approach.Based on expert knowledge, a set of rules is created to improve communication quality between patient and caregiver and to ensure that the communication process is safe and not interrupted.Among other things, each rule has a type (obligation or prohibition) and response output when triggered.e rules are prohibition rules, for example about yelling, and obligations rules regarding, for instance, how to keep the patient safe.
ere are verbal and non-verbal cues for each action, retrieved through sensors.For the military domain, Reed et al. [108] use a model that balances the principles of civilian non-male cence, military necessity, proportionality, and prospect of success.e resulting principles are ranked in order of importance.A scenario is used to calibrate the relative ethical violation model by updating the weight for each principle.en, a survey is conducted to collect both expert and non-expert assessment of the situation.Rule-based systems trained on human data perform at the level of human experts.For the air tra c domain, Dennis et al. [48] developed the ETHAN system that deals with situations when civil air navigation regulations are in con ict.e system relates these rules to four hierarchical ordered ethical principles (do not harm people, do not harm animals, do not damage self, and do not damage property) and develops a course of action that generates the smallest violation to those principles in case of con ict.McLaren [91] collaborated with the National Society of Professional Engineers to adopt the principles in their code of ethics for a system called SIROCCO.Its primary goal is to test whether it can apply existing heuristic techniques to identify the principles and previous cases that are most applicable for the analysis of new cases, based on an engineering ethics ontology.SIROCCO accepts a target case in Ethics Transcription Language, searches relevant details in cases in its knowledge base in Extended Ethics Transcription Language and produces advised code provisions and relevant known cases.
Multi-Agent Systems (MAS).Wiegel and van den Berg [141] use a Belief-Desire-Intention (BDI) model to model agents in a MAS se ing.
eir approach is based on deontic epistemic action logic, which includes four steps: modelling moral information, creating a moral knowledge base, connecting moral knowledge to intentions, and including meta-level moral reasoning.Moral knowledge is linked to intentions and if there is no action that can satisfy the constraints, the agent will not act.Neto et al. [97] also implement a BDI approach for a MAS.eir focus is on norm con ict: an agent can adopt and update norms, decide which norms to activate based on the case at hand, its desires, and its intentions.Con ict between norms is solved by selecting the norm that adds most to the achievement of the agent's intentions and desires.Norm-adherence is incorporated in the agent's desires and intentions.Also, Mermet and Simon [92] deal with norm con icts.ey distinguish between moral rules and ethical rules that come into play when moral rules are in con ict.ey perform a veri cation of whether their system called GDT4MAS, is able to choose the correct ethical rule in con ict cases.
Other.Bringsjord and Taylor [32] propose a normative approach using what they call "divinecommand ethics".ey present a divine-command logic intended to be used for lethal autonomous robots in the military domain.is logic is a natural-deduction proof theory, where input from a human can be seen as a divine command for the robot.Turilli [129] introduces the concept of the ethical consistency problem.He is interested in the ethical aspects of information technology in general.He proposes a generic two-step method that rst translates ethical principles into ethical requirements, and then ethical requirements into ethical protocols.
8.1.2Consequentialism.Among papers that use a top-down consequentialist approach, this survey brie y discusses (i) those that focus on the home assistance domain, (ii) those that focus on safety applications, and (iii) a variety of others.
Home domain.Cloos [37] proposes a service robot for the home environment.
e system, called Utilibot, chooses the action with the highest expected utility.Because of the computational complexity of consequentialism, the ethical theory is a decision criterion rather than a decision process.e description of the system seems a realistic thought experiment, mentioning features the system could have, based on previous research.
e system controlling the robot, Wellnet, consists of Bayesian nets and uses a Markov decision process to optimize its behavior for its policies.Van Dang et al. [130] focus a similar use case but opt for a di erent technical approach: they adopt a cognitive agent so ware architecture called Soar. e robot is given information about family members.When it receives a request, each possible action is assigned a utility value for each general law of robotics as proposed by Asimov.e action with the maximum overall utility is selected to be executed, which can be to either obey, disobey, or partially obey (meaning proposing an alternative option for) the human's request.
Falling prevention.ree related papers focus on the use case where a human and robot (both represented by a robot in experiments) are navigating a space that has a hole in the ground.e robot has to decide how to intervene in order to prevent the human from falling into the hole.
Win eld et al. [142] add a "Safety/Ethical Logic" layer that is integrated in a so-called consequence engine.is mechanism for estimating the consequences of actions follows rules very similar to Asimov's laws of robotics.ey address each law in an experiment.Dennis et al. [49] continue the work of Win eld et al. [142], by using and extending their approach, and introduce a declarative language that allows the creation of consequence engines within what they name the "agent infrastructure layer toolkit" (AIL).Systems created with AIL can be formally veri ed using an available model checker.e example system that is implemented sums multiple possible unethical outcomes and minimizes the number of people harmed.Vanderelst and Win eld [132] have a similar approach and implement two robots representing humans and a robot that follows Asimov's laws respectively.In their case study, there are two goal locations, one of which is dangerous, and the Asimov robot has to intervene.
Other.In early work by Anderson et al. [6], a simple utilitarian system is introduced based on the theory of Jeremy Bentham that implements act utilitarianism (i.e., calculates utilities of options and chooses the one with the highest utility).[14] describe a system that "compares cases that contain ethical dilemmas about whether or not to tell the truth." ey use a case-based reasoning approach to compare the di erent cases in its database.e program, called Truth-Teller, compares di erent real-world situations in terms of relevant similarities and distinctions in justi cations for telling the truth or lying.Representations for principles and reasons, truth telling episodes, comparison rules, and important scenarios are presented.

Particularism. Ashley and McLaren
8.1.4Hybrid: Specified Hierarchy.is section contains papers that use a top-down ethical hybrid approach with a speci ed hierarchy.Di erent groups can be distinguished: papers where deontological ethics are dominant over consequentialism, and a paper where consequentialism is dominant over deontological ethics.Deontological dominance.While the following three systems all have the same approach, they are very di erent in their implementation.In the system by Dehghani et al. [47], the ethical theory type is very clear.e system, called MoralMD, has two modes: deontological and utilitarian.A new case is processed into predicate calculus and the presence of principles and contextual features are compared to a determined set of rules in a knowledge base.e order of magnitude reasoning module calculates the relationship between the utility of each choice.If there are no sacred values involved in the case at hand (i.e., the deontological component), the system will choose the proper output based on the highest utility (i.e., the consequentialist component).Govindarajulu and Bringsjord [61] provide a rst-order modal logic to formalize the doctrine of double e ect and even of triple e ect: "the deontic cognitive event calculus."e calculus includes the modal operators for knowledge, beliefs, desires, and intentions.To be able to be useful in non-logic systems, they explain what characteristics a system should have to be able to use the proposed approach.e doctrine of double (and triple) e ect combines deontological and consequentialist ethics, where deontology has a greater emphasis than consequentialism.Pereira and Saptawijaya [99] use prospective logical programming to model various moral dilemmas taken from the classic trolley problem and employ the principle of double e ect as the moral rule.Once an action has been chosen, preferences for situations are judged a posteriori by the user.
e authors show their implementation in a program called ACORDA.
Consequentialist dominance .In earlier work, Pontier and Hoorn [104] introduced a "cognitive model of emotional intelligence and a ective decision making" called Silicon Coppélia to be used in the health domain.An agent has three moral duties (autonomy, bene cence, and non-male cence) with a certain ambition to ful ll each duty (i.e., weights).e system's decisions are based on actionspeci c expected utilities and consistency with the predetermined duties.While most authors make an act utilitarian system, Pontier and Hoorn create a rule utilitarian system by trying to maximize the total amount of utility for everyone.While they use rules (i.e., deontological ethics), they implement them in a consequentialist way, making this the dominant ethical theory type.8.1.5Hybrid: Unspecified Hierarchy.Both systems in this category focus on a modular approach, where di erent ethical theory types can be combined in an ethical machine.e goal of the system by Berreby et al. [25] is to create a modular architecture to represent ethical principles in a consistent and adjustable manner.ey qualify what they call "the Good" and "the Right" as the ethical part of their system (implying both consequentialist and deontological constraints).Besides these system components, the system consists of an action model (i.e., "it enables the agent to represent its environment and the changes that take place in it, taking as input a set of performed actions") and a causal model (i.e., "it tracks the causal powers of actions, enabling reasoning over agent responsibility and accountability, taking as input the event trace given by the action model and a speci cation of events containing a set of events and of dependence relations") [25].e implementation is done in Answer Set Programming using a modi ed version of Event calculus.Using a medical scenario, they provide a proof of concept.Lindner et al. [86] have created a so ware library for modelling "hybrid ethical reasoning agents" called HERA.Based on logic, they create a prototype called IMMANUEL, which is a robotic face and upper body that users can interact with.e system's ethical constraints draw on consequentialist calculations, the Pareto principle from economics, and the principle of double e ect.Uncertainty and belief in permissibly of an action are added as extra variables in the system.8.1.6Configurable ethics.e papers in this subsection have a top-down approach and proposed various ways in which ethics can be implemented.One paper has machine ethics tailored for a speci c domain, while another uses di erent techniques in a more domain-general way.A third focuses on multi-agent systems.Domain-speci c. ornton et al. [127] combine deontology, consequentialism, and virtue ethics to optimize driving goals in automated vehicle control.Constraints and costs on vehicle goals are determined on the basis of both deontological and consequentialist considerations.Virtue ethics generates speci c goals across vehicle types, such that a tra c infraction of an ambulance is assessed as less costly than that of a taxi cab.Domain-general.Ganascia [58] claims to be the rst to a empt to model ethical rules with Answer Set Programming (cf.[19]) to model three types of ethical systems -Aristotelian ethics, Kantian deontology, and Constant's "Principles of Politics" (cf.[40]).Drawing on [103] situation calculus, Bonnemains et al. [29] devise a formalism in which moral dilemmas can be expressed and resolved in line with distinct ethical systems, including consequentialism and deontological ethics.
Multi-agent systems.Cointe et al. [38] extend ethical decision making to multi-agent systems.e judgment function can accommodate a wide variety of inputs and is not restricted to the format of a single type of ethical system.8.1.7Ambiguous.Arkoudas et al. [12] reason that well-behaved robots should be based on "mechanized formal logics of action, obligation and permissibility".A er introducing a domainspeci c deontic logic, they describe a previously published interactive theorem proving system, Athena, that can be utilized to verify ethical systems based on rst-order logic.Murakami [96] presented an axiomatization of Horty's utilitarian formulation of multi-agent deontic logic [77], while Arkoudas et al. [12] present a sequent-based deduction formulation of Murakami's system.While deontic logic is used, each deontic stit frame contains a utility function.e contribution lies in the new approach to Murakami's system, which is implemented and proven in Athena.In a di erent approach, the proposed system by Cervantes et al. [35] devise a computational model for moral decision-making inspired by neuronal mechanisms of the human brain.e model integrates agential preferences, past experience, current emotional states, a set of ethical rules, as well as certain utilitarian and deontological doctrines as desiderata for the impending ethical decision.
With an entirely di erent focus, Atkinson and Bench-Capon [16] depart from Hare's contention [70] that in situations with serious consequences, we engage in complex moral reasoning rather than the simple application of moral rules and norms.Moral norms are thus considered not an input to, but an output of serious moral deliberation.e authors model situated moral reasoning drawing on Action-Based Alternating Transition Systems (cf.[143] as well as [15]).
Verheij [134] draws on Bench-Capon's framework of value-based argumentation ( [22,23]), which is inspired by case law (new cases are decided on past cases where there is no clear legislation, cf.[67]).e paper breaks new ground in so far as the formal model is not restricted to either qualitative or quantitative primitives, but integrates both.

Bo om-up
is section introduces selected papers that use a bo om-up approach to implementing ethics.
8.2.1 Deontological Ethics.Malle et al. [90] argue that robots need to have a norm capacitya capacity to learn and adhere to norms.Drawing on deontic logic, the authors explore two distinct approaches of implementing a norm system in an arti cial cognitive architecture.Noothigattu et al. [98] collect data on human ethical decision making to learn societal preferences.ey then create a system that summarizes and aggregates the results to make ethical decisions.8.2.2 Consequentialism.Armstrong [13] observes that equipping arti cial agents directly with values or preferences can be dangerous (cf.[30]).Representing values as utility functions, the author proposes a value selection mechanism where existing values do not interfere with the adoption of new ones.Abel et al. [1] pursue a related goal.In contrast to Armstrong, the agent does not maximize a changing meta-utility function but instead draws on partially observable Markov decision processes (cf.[82]) familiar from reinforcement learning.e system is tested with respect to two moral dilemmas.

Hybrid: Unspecified Hierarchy. Howard and Muntean [79]
In contrast to the dominant action-based models of autonomous arti cial moral agents, Howard and Muntean [63] advocate an agent-based model, which combines traits of virtue ethics and particularism.e implementation draws on neural networks optimized by evolutionary computation and is given a test run with the NEAT (NeuroEvolution of Augmenting Topologies) package (cf.[52,109,124,125]).8.2.4 Ambiguous.Furbach et al. [57] demonstrate how deontic logic can be transformed into description logic so as to be processed by Hyper-a theorem prover employing hypertableau calculus by aid of which normative systems can be evaluated and checked for consistency.Wu and Lin [146] are interested in "ethics shaping" and propose a reinforcement learning model.e la er is augmented by a system of penalties and rewards which draws on the Kullback-Leibler divergence [84].

Hybrid
is section introduces selected papers that use a hybrid approach to implement ethics by combining top-down and bo om-up elements.

Deontological
Ethics.e following papers, all by the same set of authors, use a hybrid approach to implement deontological ethics.In 2004, Anderson et al. [6] introduced W.D.: a system based on the prima facie duties advocated by W.D. Ross.W.D. leaves the encoding of a situation up to the user, who has to a ribute values to the satisfaction and violation of the duties for each possible action.
e system pursues the action with the highest weighted sum of duty satisfaction.Two years later, Anderson et al. [7] introduced MedEthEx, an advisory system in medical ethics.MedEthEx has three components: a basic module trained by experts, a knowledge-based interface that guides users when inpu ing a new case, and a module that provides advice for the new case at hand.In 2014, Anderson and Anderson [5] created GenEth, a general analyzing system for moral dilemmas.e system is capable of representing a variety of aspects of dilemmas (situational features, duties, actions, cases, and principles) and can generate abstract ethical principles by applying inductive logic to solutions of particular dilemma cases.e principles are evaluated by a self-made Ethical Turing Test: if the system performs as an ethical expert would, it passes the test.8.3.2Particularism.Guarini [62] explores whether neural networks can be employed to implement particularist ethics, as occasionally hinted at by Dancy, one of particularism's most renowned advocates (cf.[43][44][45]).Using the action/omission distinction (cf.[145] for a review) as a test paradigm, neural networks are trained with di erent types of cases in order to investigate whether they can competently judge new ones.8.3.3Hybrid: Specified Hierarchy.Arkin [8] explores constraints on the deployment of lethal autonomous weapons in the ba le eld (it was subsequently published as a series of three articles [9][10][11].
e proposed system is predominantly governed by deontological rules, namely international laws of war and the US Army's rules of engagement.Its architecture relies on four central constituents: an Ethical Governor that suppresses lethal action; an Ethical Behavior Control that constrains behavior in line with the rules; an Ethical Adaptor, which can update the agent's constraint set to a more restrictive one; and a Responsibility Advisor, which is the human-robot interaction part of the system.
Azad-Manjiri [18] develops an architecture for a deontological system constrained by Beauchamp and Childress's biomedical principles.e system determines its actions on the basis of said principles and a decision tree algorithm trained with expert ethicist judgments in a variety of cases from the biomedical domain.Building on early work by Ganascia [59], Tu s ¸and Ganascia [128] augment a belief-desire-intention rational agent model with normative constraints.ey devote particular a ention to the problem arising from the acquisition of new norms, which frequently stand in con ict with existing ones (for an alternative approach building on the belief-desire-intention model, see Honarvar and Ghasem-Aghaee [75,76] discussed below).8.3.4Hybrid: Unspecified Hierarchy.Yilmaz et al. [147] survey the eld of machine ethics and propose a coherence-driven re ective equilibrium model (cf.[107]), by aid of which con icts across heterogenous interests and values can be resolved.Honarvar and Ghasem-Aghaee [75] build a belief-desire-intention agent model whose decisions are based on a number of weighted features drawn from hedonic act utilitarianism (e.g., the amount of pleasure and displeasure for the agent and other parties a ected by the action).8.3.5 Ambiguous.Most of the work of Saptawijaya and Pereira (c.f.[99][100][101][115][116][117]) focuses on logic programming and prospective logic to model ethical machines.In Han et al. [69], they introduce uncertainty as a factor in decision making and draw on abductive logic to accommodate it.Madl and Franklin [89] call for limits on ethical machines for safety reasons.Developing on Franklin et al.'s [56] LIDA architecture-an arti cial general intelligence (AGI) model of human cognition-they suggest that deliberate actions could be constrained top-down during run time, and ethical meta-rules (such as certain Kantian principles) could be implemented on a metacognitive level.Rather than start from a complete set of rules, the la er can gradually expand.e approach is exempli ed by CareBot, an assistive simulated bot for the home care domain.Wallach et al. [139] also discuss the LIDA model.ey demonstrate how emotions can be integrated into a LIDA-based account of the human decision making process and extend the approach to arti cial moral agents.

FUTURE AVENUES AND LIMITATIONS
Based on the results of the analysis and description of the selected papers, some literature gaps are identi ed that can be of interest for future work.Additionally, the limitations of this survey are discussed.
Ethical dimension.In view of earlier calls for hybrid systems when it comes to ethical theory, a surprisingly low percentage of authors consider a multi-theory approach in which machines can interchangeably apply di erent theories depending on the type of situation.In terms of the content (and not the structure) of ethical theories, it is important to acknowledge and harness the nuances of speci c theories, but human morality is complex and cannot be captured by one single classical ethical theory.Even experts can have rational disagreement amongst themselves on an ethical dilemma. is leads to the next important point: an ethical machine will not be of use if it is not accepted by its users, which can be the risk of focusing on one ethical theory and, thus, not covering human morality.Ethical theory needs to be combined with domain-speci c ethics as accepted by domain experts and, as identi ed in the analysis of this paper, this is not the case in the majority of the related work.Moreover, it is necessary to discuss the ethical theory/theories in the system with its possible users.Some examples of using folk morality in machine ethics can be found in Noothiga u et al. [98], as well as in [111].However, it is important to note that just as ethical theories have their challenges, so does folk morality.ree challenges are who to include in the group whose values should be considered (standing), how to obtain their values (measurement), and how to aggregate their values (aggregation) [20].Implementations should start from ethical theories combined with domain-speci c ethical theory, a er which acceptance by the users and deviation from socially accepted norms should be discussed (cf.e.g., [17,28,85,118]).
Implementation dimension.ere is a need for more systematic evaluations when ethical machines are created in order to be able to rate and compare systems.To this end, there is a strong need for domain-speci c benchmarks.Based on input from domain experts, data sets need to be created containing the types of cases prevalent in that domain, with respect to which ethical machines must be assessed.e gathering of typical tasks and respective answers that domain experts agree on is just as important as the actual creation of ethical machines.is implies the need for more collaboration between elds.Computer scientists and philosophers, as well as domain experts and social science experts, have to work together to ensure the interaction with and e ects of the ethical machines are as desired.Even within the eld, collaboration is needed between di erent clusters of topics in the eld of machine ethics, for example between clusters specializing in MAS and machine learning respectively.Finally, in general, implementation requires more a ention.While on a higher level, theoretical discussion remains important in this eld, especially to prepare for possible future scenarios, the testing of theory in practice can enrich the discussion on what is (or is not) possible at that moment and what practical implementations and consequences certain ethical machines can have.
Technical dimension.When a system is implemented, it is imperative to provide exhaustive speci cation detail, including availability of the code, which is predominantly lacking.Another frequent shortcoming regards usability: the system should have a user interface so that the future user can interact with the system without having to know how to code.Furthermore, automatic processing of input cases deserves more a ention, so as to avoid having to encode each variable manually as a vector for a neural network.Considering the increased need for transparency in algorithmic decision making, as well as the fundamental role of reasons in ethics, the system should also provide an explanation of why it took a certain decision.In a next phase, the user should be able to give feedback on the ethical decision the system makes.Finally, the association of a given type of technology with a certain type of ethics requires an adequate technical justi cation, beyond using just the most acquainted technology.
Further Points of Interest.Current technology allows for successful application of narrow AI geared towards speci c tasks.While steps are being taken towards AGI, the technology does not yet exist [80].Hence, domain-speci c applications seem suitable.A domain-speci c non-AGI approach to machine ethics alleviates some of the risks and limitations on machine ethics posed by [33], such as those related to an "insu cient knowledge and/or computational resources for the situation at hand."However, there are still risks and limitations.For instance, in the context of lethal autonomous weapons systems, the loss of "meaningful human control" [114] is a risk, as humans would not have the same control over ethical decisions such as target selection.A limitation of using domain-speci c ethical machines is that the process of one domain may not be transferable to other domains.Furthermore, not everyone is ready to accept a machine taking over the ethical decision making process [73].
A slightly di erent way to address ethics in machines is to de ne (and implement) an ethical decision support, rather than leaving the machine to make an autonomous ethical decision.For an overview of di erent types of moral mediation, see Van de Voort et al. [131].Etzioni agrees that the focus should lie on decision support, stating "there seem to be very strong reasons to treat smart machines as partners, rather than as commanding a mind that allows them to function on their own" [51, p. 412].One of those reasons is that AGI will not exist in the foreseeable future.is approach will also help with acceptance of machines with ethical considerations in society.ere are di erent possible levels of autonomy the system can have, for example only summarizing available data, interpreting available data, summarizing possible actions, or even suggesting/pre-selecting a possible action the system deems best.Di erent types of support and collaboration might be necessary for di erent applications, and according to the literature review done in this paper, further research is needed in this direction.
Limitations. is survey has some limitations that need to be mentioned.First of all, the scope of the paper selection was limited to explicit ethical theories (i.e., theories directly programmed into the machine).While some of the works reviewed can still be of interest and provide inspiration for implementation, papers devoid of implementation details were excluded from this survey.Examples are emerging ethics based on human data to research folk morality (e.g., [146]) or models of human morality to determine relevant features in input cases (e.g., [136]).Furthermore, we limited the survey to one paper per author whenever similar systems were discussed across multiple publications, selecting the most comprehensive one.is does not do full justice to the work of certain authors (e.g., Guarini working on explainability of neural networks making ethical decision [63]).While the paper selection procedure was designed to be as exhaustive as possible, it is still possible that a few important papers were missed.Finally, three authors reviewed the ethical dimension and two reviewed the implementation and technical dimension, but it is still possible there was bias in the classi cation due to the limited number of people involved in the classi cation process and the process of discussion until agreement was reached.

CONCLUSION
e future of the eld of machine ethics will depend on advances in both technology and ethical theory.Until new breakthroughs change the eld, it is important to acknowledge what has been done so far and the avenues of research that make sense to pursue in the near future.To accomplish this, the contribution of this survey is threefold.Firstly, a classi cation taxonomy with three di erent dimension is introduced: the ethical dimension, the dimension considering implementing ethics into a machine, and the technical dimension.Secondly, an exhaustive selection of papers describing machine ethics implementations is presented, summarized, and classi ed according to the introduced taxonomies.Finally, based on the classi cation, a trend analysis is presented that leads to some recommendations on future research foci.It is important to keep in mind how machine ethics can be used in a meaningful way for its users, with increasing agreement on what a system should do, and in what context.

ACKNOWLEDGEMENT
is work is partially funded by the Swiss Center for Drones and Robotics of the Department of Defense, Civil Protection and Sport via armasuisse S+T under project number 050-43.

6. 1 . 1
Logical reasoning.ere are di erent types of logic or logic-based techniques used in machine ethics.

Table 1 .
Ethical theory types taxonomy

Table 2 .
High-level overview to ethics categories in the context of ethical machine implementation

Table 3 .
Ethical implementation taxonomy

Table 4 .
Technical taxonomy.As explained in Section 6, "Inductive logic" is present twice.

Table 5 .
Ethical theory classification.Hybrid dominance D-C implies both D and C are implemented, but D is dominant.The reverse is true for Hybrid dominance C-D.For the Hybrid undefined dominance the theories that are combined are noted in parentheses following the citation.

Table 6 .
Ethical implementation classification.Diversity consideration: implies yes, an empty cell implies no/not present.

Table 7 .
Technical classification.implies yes/fully, • implies partially, an empty cell implies no/not present.