News recommender systems: a programmatic research review

ABSTRACT News recommender systems (NRS) are becoming a ubiquitous part of the digital media landscape. Particularly in the realm of political news, the adoption of NRS can significantly impact journalistic distribution, in turn affecting journalistic work practices and news consumption. Thus, NRS touch both the supply and demand of political news. In recent years, there has been a strong increase in research on NRS. Yet, the field remains dispersed across supply and demand research perspectives. Therefore, the contribution of this programmatic research review is threefold. First, we conduct a scoping study to review scholarly work on the journalistic supply and user demand sides. Second, we identify underexplored areas. Finally, we advance five recommendations for future research from a political communication perspective.

based on inferred as well as explicitly stated user preferences  has fuelled discussions about algorithms' potential to drive selective exposure and, ultimately, the establishment of filter bubbles (Pariser, 2011) and political polarisation (Nelson & Webster, 2017;Slechten et al., 2021). These reflections show that NRS may have important democratic implications on the supply and demand sides of political news, which makes an investigation of these systems through a political communication lens crucial. A political communication perspective on NRS can illuminate the role of processes related to both the journalistic production and audiences' acquisition of political information in shaping democratic society.
The growing relevance of NRS as an object of study is reflected in the increased scholarly interest in NRS on the journalistic supply and the audience demand sides. However, scholarly work on these two sides often follows distinct theoretical and methodological approaches, whose findings have yet to be synthesised. At the same time, similar to research on other technological innovations, there is a risk of overestimating the novelty, significance, and change related to NRS (Steensen & Westlund, 2021, pp. 82-85), making the collection and synthesis of existing research even more important. Therefore, our main contribution is the provision of a programmatic review of the field to date from a political communication perspective. Such a review is necessary to lay the foundation for the integration of theoretical, methodological, and empirical insights from the journalistic supply and user demand sides. A political communication lens allows us to analyse two aspects: a) how NRS affect processes related to the journalistic representation of political reality and its dissemination in news stories, and b) how NRS affect processes involved in the acquisition of political information by the audience (see also Van Aelst et al., 2017). 1 As NRS touch both the supply and demand of political news and thus carry important democratic implications, a political communication perspective is particularly relevant to the research field of NRS.
To assess the breadth and depth of previous research addressing NRS, our review follows the methodology of the so-called 'scoping study' by Arksey and O'Malley (2005) (see also Levac et al., 2010). A scoping study is a specific approach to reviewing scientific evidence in which the main interest is to systematise literature that has developed in a short time around a new and complex phenomenon, and thus map an evolving field of research. We first clarify the concept of NRS and identify the specific research questions that guide our review. Second, we describe the procedure we followed when searching and selecting relevant studies for inclusion in this review and how we classified these studies according to larger research perspectives. Third, we describeby way of a thematic synthesisin more detail the extent, range, and nature of research on NRS. This allows us, in a fourth step, to delineate areas of emphasis in existing scholarship and, more importantly, to identify research gaps and development opportunities. The logical fifth and final step is to make recommendations for future research.
We conclude by suggesting five promising directions for news recommender research: (1) more research integrating the demand and supply sides, (2) more theory-explicated work, (3) more comparative research, (4) more interdisciplinary work, and (5) more industry-academia collaboration.

News recommender systems: an emerging research field
News recommender systems (NRS) rely on algorithms that provide users with personalised recommendations based on past data about interactions with news content, for example, past user or similar users' behaviour, explicitly stated user preferences, overall popularity metrics, or other content-specific features related to articles' actual content such as their topic, length, or presence of multimedia elements . The most common filtering mechanisms include content-based filtering, collaborative filtering, and hybrid approaches which combine content-and collaborative-based criteria. Content-based filtering considers characteristics of the news piece itself and its relevance to one's interest (as determined by, e.g., a user's consumption history), while collaborative filtering also considers what users with similar news preferences consume .
Recommender systems in the specific field of political news have greater social relevance than those in other areas  as NRS touch both sides of the political information environment (Van Aelst et al., 2017): the supply of political news by journalistic media and the demand for political information by audiences (see also Guzman, 2019).
On the supply side, the introduction of algorithmic solutions such as NRS, which rely on automation and machine learning , can challenge traditional journalistic practices (Belair-Gagnon & Holton, 2018). Regarding NRS, such challenges primarily concern the distribution of news, as NRS can supplement or even replace traditional news curation by human editors with algorithmic selection (Diakopoulos, 2019). Yet, the implementation of NRS may ultimately also affect journalistic selection and production practices. Since NRS can give some preference to certain content over other, owing, for instance, to its popularity or similarity to previously consumed stories, rather than based on human judgement, 'the question of "what deserves attention?" at the heart of the professional judgment of journalism shifts to a different, personalized query of "what does this person want?"' (Carlson, 2018(Carlson, , p. 1765. This shift, further accelerated by the availability of metrics-oriented insight into user demand, 'can change newsroom cultures and affect how journalists understand good journalism' (Møller, 2022b, p. 7;Carlson, 2018). As a result, technologies such as NRS may influence the amount, shape, and quality of political news provided by the media (Davis, 2013;Napoli, 2014).
On the demand side, NRS may affect the amount and quality of political news people consume . If NRS rely heavily on preference matching, employing personalisation for political news could, in theory, facilitate exposure to primarily attitude-consistent content and thus restrict citizens' opportunities to encounter counter-attitudinal information . Such insistent preference matching by recommender systems can work against the ideal of a broadly and diversely informed public in which each citizen weighs conflicting alternatives and can undermine the chances for deliberative forms of democracy .
The use of NRS is, therefore, a prototypical example of how supply and demand sides of political communication are intertwined. On the one hand, the supply of algorithmically curated news affects how users interact with and make sense of political content. On the other hand, news organisations 'tailor [NRS] to a particular view of their audience and how these technologies may impact audience members' engagement in society' (Guzman, 2019(Guzman, , p. 1186. This anticipation of audience preferences and its potential impact on the supply of political news are testament to the growing interconnectedness of supply and demand sides in high-choice media environments (see also Van Aelst et al., 2017, p. 6).
Despite increasing scholarly interest in NRS, there is a lack of reflection on the need to bring the two perspectives together and a lack of awareness of the additional potential for insight of such a step. To date, the related research on the supply and demand sides often follows different theoretical and methodological approaches, and the empirical findings await synthesis. Thus, to lay the foundation for a more integrated scholarly approach to NRS, we review and evaluate studies on both the supply and the demand sides. The research questions that guide this scoping study thus read: RQ1) What scientific evidence exists regarding the use and impact of NRS with respect to news organisations (i.e. the supply side)? RQ2) What scientific evidence exists regarding the use and impact of NRS with respect to news audiences (i.e. the demand side)? By mapping the literature around NRS, we identify underexplored areas, thereby creating an assessment that serves us as a starting point for advancing recommendations for future research on NRS which place particular emphasis on a hitherto neglected political communication perspective.

Conducting the scoping study: literature search
To identify relevant peer-reviewed studies for our scoping study, we first conducted manual searches of the following databases: Web of Science and EBSCO Communication & Mass Media Complete. The main keywords guiding the search were 'news recommender systems', 'news recommendation', and 'news personalisation'. 2 We limited our search to peer-reviewed English-language studies published between 2011 and mid-2022. We deemed a 12-year time span appropriate for tracing the development of this nascent field. 3 Another exclusion criterion was the discipline of the studies. On Web of Science, we excluded disciplines such as medicine, engineering, and natural sciences from our search. We then analysed the title and abstract of each study to determine if it relates to political news. Therefore, we included papers which investigated recommender systems on news aggregators like Google News or Apple News and social networking sites like Twitter, YouTube, and Facebook, if their title or abstract made a connection to political news. We excluded articles that were not related to the political news domain (e.g. articles dealing with Spotify or Netflix) and/or primarily dealt with the technological implementation of recommender systems. Following this relevance check, we removed duplicates. These first steps yielded 41 relevant studies, of which 34 investigated NRS on news websites, and 7 examined content recommender systems on social media or news aggregators. In a second step, following a snowball method, we examined the initial set of 41 articles in greater detail to identify additional literature (49), of which 29 explicitly dealt with NRS on news websites and 20 with the recommendation of political content on news aggregators or social media.
The search query method, followed by snowballing, produced a total of 90 English-language studies. Of these 90 studies, 63 explicitly addressed NRS on news websites. The remaining 27 focused on news personalisation on intermediaries such as Google News, Apple News, Twitter, and Facebook. Of the 63 studies on NRS, 21 addressed the supply side, 24 the demand side, and 18 pursued other perspectives which are not discussed in detail in the following because they are not empirical. 4 Attesting to the rapidly increasing interest in the topic of NRS is the fact that more than two thirds (43) of the 63 studies were published between 2019 and mid-2022. A full list of articles considered in this review can be found in the Appendix. Figure 1 displays our data selection procedure in a flowchart.
Part of the second procedural step of our scoping study was developing a data-charting form to classify all studies. We created an Excel matrix using common dimensions to compare relevant study characteristics, such as the methodological and theoretical approaches used, the sample, and main findings. Following Levac et al. (2010)'s guidelines for scoping studies, two members of the research team responsible for the classification of the studies independently extracted relevant data from the sample to agree on the inclusion criteria and parameters of the data-charting form. They met several times to determine whether their approach to data extraction corresponded to the purpose of the review. The research team repeatedly discussed the interpretation of the studies to ensure the validity and reliability of the findings. Following this, the first author adapted and completed the datacharting form. A consolidated data-charting form is an important prerequisite for reporting the outcome of a scoping review, namely collating, and summarising the studies in the form of a thematic synthesis and identifying previous research foci and remaining research gaps. We take these steps first for supply research and then for demand research on NRS.
Research evidence on the use and impact of NRS with respect to news organisations (supply side) In addressing the first research question (RQ1), we find that supply-side research can be categorised into three main areas: 1) current state of NRS implementation, 2) journalistic attitudes towards NRS and their effects on news practice and role perceptions, and 3) organisational dynamics surrounding NRS.

Current state of NRS implementation
Several studies have analysed the extent to which NRS are currently employed (e.g. ; Van den Bulck & Moe, 2018). These studies point to the increasing use of NRS. Overall, however, current research indicates that newspapers' use of personalisation remains limited .
News organisations that have NRS in place employ different approaches that range from more basic solutions based on content popularity to more advanced ones based on user profiles and past behaviour . Current NRS consider implicit feedback, including geolocation and reading history, as well as explicit signals, such as users' stated topic preferences . Several studies document the use of explicit personalisation, wherein users can customise how and which content is displayed to them through e.g. subscribing to specific topics . Recommendations are oftentimes visible either on the front page, below articles, in sidebars, in 'Recommended for you' sections, or grouped together with advertising content . Research has also shown that news organisations do not always develop their NRS solutions in-house, but sometimes rely on third-party software solutions that combine editorial with promotional content based on users' previous consumption . However, to date, studies documenting current NRS usage pertain to a small set of countries such as Germany (e.g. , the UK, (e.g. , and the US (e.g. .
Individual studies indicate that NRS strategies can vary across organisations depending on their respective business models, target audiences, and technological affinities. For example, news outlets interested in fostering loyalty and encouraging subscriptions employ algorithms that aim to recommend less popular and niche content such as local, cultural, or sports news . Organisations that are generally more apprehensive of technology may be more reluctant to use NRS (Van den Bulck & Moe, 2018). Particularly public service media that aim to address a broad audience can be wary of advanced NRS applications . Conversely, the increased usage of targeted and personalised advertising can drive experimentations with NRS Lewis & Westlund, 2015;. News outlets' target audiences can also affect how NRS are conceptualised . At the same time, a recent study by  finds no substantial differences in front-end news personalisation between quality and tabloid media (see also  for similar results). However, as studies detailing the current state of NRS solutions have often focused on individual organisations (but see e.g. , it is difficult to draw conclusions about which factors best explain NRS adoption across different organisation types.
This notwithstanding, current research indicates that organisational characteristics such as the editorial mission and business model may influence whether and how NRS are implemented. Simultaneously, recent findings also imply that there might be a certain homogenisation and suggest that the close observation and imitation of other news organisations' NRS solutions potentially play an important role (e.g. . Underexplored Areas. As a comparative approach has not been at the forefront of studies yet, a set of comparable factors driving NRS usage across different organisational and national contexts has yet to be identified. Future research may, for example, simultaneously examine how macro-level factors related to the political and media system or the level of technological development and meso-level factors like media organisations' financial models affect NRS adoption (see also Recommendation 3 below). As of now, however, a) the application of NRS across different types of news organisations and contexts as well as b) the reasons underlying NRS usage or differences therein remain underexplored.

Journalistic attitudes towards NRS and implications for news work
This area investigates journalists' attitudes towards NRS and how the integration of such solutions relates to prevalent editorial practices, journalistic role perceptions, and ideals.
Overall, despite initial scepticism Linden, 2017), research on AI and data-driven technologies indicates that journalists have become more optimistic about algorithmic technologies (e.g. Schapals & Porlezza, 2020). One reason for this is the belief that such solutions can take over manual tasks and free up resources for more creative, contextual, and investigative work (Schapals & Porlezza, 2020).
Similar positive perceptions can also be observed regarding the adoption of NRS within news organisations (e.g. . A perceived benefit of NRS lies in their potential to increase customer satisfaction and boost subscriptions . Thus, NRS are sometimes seen as a vehicle to alleviate potential budgetary constraints . According to news professionals, NRS can also be used to recommend relevant content that users might have otherwise missed due to the rapidly changing makeup of news websites' homepages and the high volume of new articles published every day . Similarly, some journalists believe that NRS can be employed to promote niche and less popular content such as local, sports, and cultural stories, thereby better addressing the interests of previously underserved audiences .
Research also indicates that news workers are conscious of problems relating to NRS. Such concerns can pertain to technological limitations and the complexity of current NRS solutions . Typical challenges for NRS include, for example, the short lifespan of news stories (Gulla et al., 2022;, a limited content pool of articles , data and privacy protection , the difficulty of measuring the long-term success of such systems beyond user clicks , or explaining NRS applications in an understandable way to users who are not necessarily technologically wellversed . News professionals are also aware that news organisations might lack sufficient user and cookie data as well as metadata to deliver content recommendations that match audiences' preferences (Gulla et al., 2022;. Additionally, studies have shown that news workers express doubts about user demand for farreaching news personalisation and are conscious of concerns their users might have, such as filter bubbles, lack of a user-friendly design, or privacy protection . Similarly,  and Møller (2022a)'s findings reveal that news workers are sceptical of the extent to which core values such as journalistic autonomy, public service, or the provision of information on important issues can be achieved through the usage of NRS. Therefore, they voice concerns over a potential dominance of commercial imperatives over journalism's democratic mission (see also Gulla et al., 2022; for similar findings).
Attitudes towards NRS may also vary depending on the individual characteristics of news workers. Recent research on NRS indicates that professional background and position within the organisation account for differences in expectations as to which goals NRS should pursue and the traditional journalistic values to which they should adhere (e.g. . Moreover,  discovers that perceptions towards how NRS affect a news organisation's reputation vary across news workers with different job profiles, such as editors and data scientists. To date, however, most studies on journalistic attitudes towards NRS have been qualitative and based on case-specific, non-representative samples (e.g. , which does not allow to draw generalised conclusions about what factors explain attitudinal differences at an individual level.
The integration of algorithmic technologies into news work can also affect journalistic role perceptions. For example, relying on algorithmic solutions such as NRS may lead to a re-examination of journalists' role as information gatekeepers, as they must now share their editorial autonomy with new human and non-human actors . Nevertheless, studies have also shown that classical ideals and norms (e.g. objectivity, autonomy, and reporting on what is considered relevant) remain crucial to journalists (e.g. .
Closely connected to changes related to the fulfilment of traditional norms and values are implications of NRS for news work more generally. To date, studies have primarily examined which strategies news organisations employ to counteract potential shortcomings of algorithmic technologies that stem from their non-journalistic origin. These involve stronger human oversight over algorithms , editorial control (Gulla et al., 2022;, the technological implementation of features such as transparency and diversity , or forgoing the use of NRS for news that journalists think everyone should read . However, except for studies on coping strategies, research has yet to examine to what extent NRS can alter work practices related to the everyday journalistic production of political news, such as story selection, framing, newswriting, and content creation. Underexplored Areas. At the time of writing, the following research gaps become apparent: only few studies have identified d) a comparable set of factors that explain journalistic attitudes towards NRS, or studied e) the impact of NRS on work practices beyond coping strategies as well as relatedly f) how the adoption of NRS affects the representation of political reality on the production side.

Organisational dynamics surrounding NRS
Current research on algorithmic and data-driven technologies indicates that the integration of datadriven technologies into news work can alter organisational dynamics and create new sites of struggle e.g. Belair-Gagnon et al., 2020;Hagar & Diakopoulos, 2019). Similarly, studies on NRS find that the introduction of NRS can affect decision-making processes and create tensions between members of different organisational divisions (e.g. .
Recent research suggests that the implementation of NRS can shift power dynamics, as new technological actors, such as data scientists, who are responsible for the maintenance and design of NRS, are integrated into the news organisation . As a result, technological actors become increasingly involved in decision-making and strategy development within news organisations . Consequently, editorial and managerial teams also need to consider 'choices regarding the technical design and parameterization of recommender algorithms' (see also Bodó, , p. 1063.
Findings also indicate that news personalisation strategies can be shaped by marketing departments  and user-experience researchers , as they are traditionally responsible for the collection and analysis of user-related data for commercial purposes.
Svensson (2021)'s recent interview study reveals tensions between editorial staff and tech actors stemming from different goals and perceptions towards the usefulness of algorithms. Similarly,  shows that the successful implementation of NRS necessitates negotiations between different stakeholders such as editors, technologists, and managerial staff, concerning, for example, a shared understanding of what is important to users or what is technically implementable.
Underexplored Areas. More than the other two research areas on the supply side, systematic research has yet to be conducted on how the adoption of NRS affects and structures organisational dynamics. Thus, two additional research gaps become evident: a closer examination of the impact of NRS on g) organisational processes and decision-making, as well as their effect on h) the relationships and power dynamics between members of different organisational divisions involved in the implementation of NRS.

Interim conclusion
Research on the supply side has investigated how and to what extent NRS are implemented in media organisations, and to a lesser extent, how NRS relate to journalistic attitudes and practices as well as the organisational dynamics and decision-making processes surrounding the adoption and maintenance of NRS projects.
Supply-side scholarship has employed different methods such as content analyses (e.g.  and analyses based on browsing data (e.g. . To date, studies investigating journalistic reactions and attitudes toward NRS have mostly relied on qualitative in-depth interviews with news professionals (e.g. . However, few studies have looked at the role of technologists and managers involved in NRS projects (but see e.g. .
Regarding theoretical frameworks, several studies on the supply side have consulted classical journalism theories such as gatekeeping (e.g. . Similarly, authors like  and  have drawn on Deuze's (2005) notion of journalism as an ideology to examine how traditional journalistic values and ideals are challenged by the adoption of NRS. Individual studies have also employed an institutionalist approach (e.g. . In some cases, NRS research builds on previous empirical knowledge, or follows a descriptive or case-study approach  rather than formulating theory-driven assumptions (see also e.g. .
Most research focuses on Western countries such as the US (e.g. , the UK (e.g. Bodó, 2019), Germany (e.g. , the Netherlands (e.g. , and Belgium (e.g. Malcorps, 2019; Møller, 2022a) (but see  for non-Western countries). Only a handful of studies follow a comparative approach across countries (e.g. ; Van den Bulck & Moe, 2018), types of media organisations (e.g. Gulla et al., 2022;, or time (e.g. . More than other algorithmic technologies that primarily affect journalistic production processes, NRS are at the intersection between the supply and demand of news. As Guzman (2019) argued, research on the supply side of NRS 'necessarily involves, or at least evokes, the audience' (p. 1186). Therefore, research also needs to model how NRS both affect and are affected by news audiences (see also Van Aelst et al., 2017). Thus, we turn to the crucial demand side.
Research evidence on the use and impact of NRS with respect to news audiences (demand side) In addressing the second research question (RQ2) of our scoping study, we find that demand-side research can be categorised into three main areas: 1) user attitudes towards algorithmic technologies, 2) effects of NRS usage on political news consumption and information reception, and 3) political implications of NRS usage.

User attitudes towards algorithmic technologies and NRS
This research area frequently draws a connection to the concept of the 'machine heuristic' (Sundar & Kim, 2019). The underlying theoretical model argues that positive perceptions towards algorithmic technologies stem from implicit, albeit possibly faulty, beliefs that some of these technologies' characteristics (e.g. their impartiality) make them fairer and less biased than humans (Sundar & Kim, 2019). Accordingly, the term algorithmic appreciation has been coined to summarise favourable attitudes towards algorithms in decision-making processes (Logg et al., 2019), whereas algorithmic aversion subsumes negative attitudes and general distrust of such technologies (Dietvorst et al., 2015). These two opposing assessments have also been used to describe user attitudes towards algorithmic decision-making in the news sector.
To investigate attitudes towards algorithmic technologies in the realm of news, several studies have juxtaposed human editorial curation with algorithmic selection (e.g. . On the one hand, these studies show that NRS are perceived to be as fair and useful as human editors . On the other hand, survey results also indicate that users think NRS which recommend content based on their previous consumption are a better way to get news than either NRS that consider similar users' behaviour, what is currently popular ( These findings allude to users' preference for recommendations based on content-based filtering rather than based on popularity-driven and collaborative filtering. However, as some findings reveal that users prefer content-based filtering over editorial selection (e.g. , whereas others suggest users perceive them to be equally useful (e.g. , it is, as of now, unclear whether algorithmic selection takes precedence over manual curation. Diverging user attitudes towards NRS have been traced to a multiplicity of individual characteristics such as privacy concerns ; but see Bodó et al., 2019 for nonsignificant effects), level of educational attainment , attitudes towards a shared public sphere , duty to keep informed , overall knowledge about algorithms , confidence in one's ability to judge the relevance of information (van der Velden & Loecherbach, 2021), technology optimism , and, similarly, trust in algorithmic systems (Shin, 2020) and the media . General attitudes towards the news outlet can also translate to perceptions towards the outputs of its recommender system. For instance, users might assess the trustworthiness of personalised newsfeeds based on the degree of overall trust they have in the news brand . However, as existing studies have each focused on a subset of these factors, it is difficult to assess their relative importance and potential interactions.
Research has also analysed the extent to which users are aware of personalisation algorithms on intermediaries such as social media (e.g. Powers, 2017;Proferes, 2017;, and, to a lesser extent, on news websites (e.g. . These studies paint a mixed picture, with some pointing to low awareness of the existence of algorithms and limited understanding of how they filter information Powers, 2017;, while others detect a high awareness thereof (Proferes, 2017;. Additionally, research on social media has found that user knowledge of how filtering algorithms work and affect news feeds is often based on unproven assumptions (so-called 'folk theories') about algorithms' inner workings  as well as misconceptions and erroneous beliefs about algorithms' prowess .
Several studies have further investigated users' perceived benefits (e.g.  and concerns (e.g. , as well as improvement wishes regarding NRS (e.g. . A main benefit of NRS is seen in the perception that they enable users to receive only news that matches their interests, saving them time spent looking for relevant content Shin, 2020). At the same time, users are cognisant of risks such as manipulation or stereotyping , lack of privacy protection , and limited exposure to counter-attitudinal or important information . Trust towards NRS is further influenced by the degree of transparency about the process of personalisation itself and the people in charge of the underlying algorithms Shin, 2020), as well as the level of exercisable user control . However, a desire for more control over algorithmic recommendations does not necessarily mean that users will actively exercise their agency ; see also ter Hoeve et al., 2017 for news aggregators).
As of now, few studies have examined users' perceived benefits and concerns simultaneously, making it difficult to assess whether negative or positive perceptions currently take precedence (but see . Additionally, a clear comparison between studies examining benefits and concerns is hampered by diverging study designs, as some employ a qualitative interview approach (e.g. , whereas the remaining ones make use of quantitative surveys (e.g. . Underexplored Areas. Studies in this research area have analysed various aspects and influencing factors on users' attitudes towards news personalisation, making it difficult to assess these factors' relative importance and potential interactions between them. Moreover, few studies have examined algorithmic awareness, specifically regarding NRS on news websites. In light thereof, the following research gaps materialise at the time of writing: j) a more systematic comparison of individual factors influencing user attitudes, k) the juxtaposition of perceived benefits and concerns related to NRS, l) a closer examination of algorithmic awareness relating to news websites.

Effects of NRS on political news consumption
Studies have analysed how news personalisation relates to news consumption more generally. Anecdotal findings indicate that recommender systems can drive engagement with news content , influence which stories users click on , and increase usage time and user satisfaction .
However, more important from a democratic perspective is the assumption that personalised news exposure can affect political information reception and, more generally, news consumption patterns . Oftentimes, studies in this research area have aimed to investigate whether recommender systems facilitate the establishment of information cocoons (so-called 'filter bubbles') due to a stronger exposure to similar and attitude-consistent content (Pariser, 2011). These endeavours frequently consult the selective exposure paradigm, which has its origin in cognitive dissonance theory (Festinger, 1962). Selective exposure refers to biases in news consumption that stem from citizens' tendency to select political content that is congenial to their point of view (Guess et al., 2018). This ingrained predisposition can further intensify and self-reinforce due to algorithmic filtering, whereby certain information is hidden from users because it does not match their preferences or is deemed unpopular. This, in turn, can propagate the eventual establishment of filter bubbles  and undermine the establishment of a shared common core of political issues, which is vital for democracy Zuiderveen Borgesius et al., 2016).
To examine the empirical validity of this theoretical assumption, studies have often examined the filtering algorithms of news aggregators and social media and their effect on political news output (e.g. . Overall, research indicates that the risk of getting stuck in such a bubble on intermediaries such as Google News (e.g.   Bruns, 2019;Möller, 2021;Zuiderveen Borgesius et al., 2016). Though relatively few studies have examined effects of recommender systems on news websites on the breadth and depth of political news consumption, their findings also largely disprove fears about filter bubbles  for simulated user behaviour see ; but also  for divergent findings). Scholars have, however, highlighted the importance of comparing different platforms and filtering mechanisms (e.g. . One possible explanation for the lack of empirical evidence for filter bubbles is that users themselves do not necessarily strive to avoid attitude-incongruent information. Various studies have shown that individual choice can counteract involuntary selective exposure due to algorithmic filtering (e.g. Bakshy et al., 2015;. Non-experimental surveys indicate that users' news diets in digital settings are relatively diverse, disproving claims about users' ingrained tendency for selective exposure (e.g. Dubois & Blank, 2018;Guess et al., 2018). Effects on news reception can further be moderated by various technological and nontechnological factors, such as recommendations' placement on the webpage , the level of exercisable user control , the ability to customise recommendation preferences , or users' desire to seek out balanced political information due to its higher information utility .
To date, however, only few studies have looked specifically at the impact of recommender systems on news websites on users' political news consumption choices (but see . One recent exception constitutes a longitudinal experimental study by , which concludes that recommender systems on news websites can effectively nudge news consumers to engage with opposing political viewpoints if they are programmed to promote exposure to both counter-and pro-attitudinal information (see also  for similar findings using simulated user behaviour).
Furthermore, NRS scholarship until now has mostly modelled effects on political news reception with the selective exposure paradigm (e.g. . Few studies have thoroughly examined the impact of NRS on the breadth and depth of political information acquisition through the lens of other theories (but see e.g. Beam (2014)'s consideration of the elaboration-likelihood-model).
Underexplored Areas. Even though the implications of recommender systems for audiences' political news consumption have been studied to a moderate degree, they have often been investigated in the context of social media and news aggregators. Therefore, the following research gaps emerge: The specific effects of NRS on m) news consumption, particularly related to news websites, have yet to be thoroughly explored empirically and in view of other theoretical frameworks besides the selective exposure paradigm. Moreover, studies have not fully investigated n) the longer-term impacts of NRS on the breadth and depth, for example the diversity as well as the quality, of users' political information acquisition.

Political implications of NRS usage
Scholars have argued that NRS might impair political knowledge due to stronger exposure to onesided information Bruns, 2019;Eskens et al., 2017). However, empirical findings paint a more nuanced picture. For example,  finds that user-driven news customisation leads to a higher elaboration, i.e. more thoughtful processing of the news message and higher attention to its content, which, positively affects political knowledge. As users are presented with information that matches their interests and is, therefore, more familiar, they are likely to engage with the content more systematically, leading to knowledge gain . However, more recently,  shows that the repeated usage of a news platform that is optimised to promote exposure to both counter-and pro-attitudinal political viewpoints does not enhance the users' perceived political knowledgeability.
Several studies have also analysed the effects of algorithmically recommended content on political opinion polarisation. Dylko and colleagues' (2018) experimental study of a custom-built news page and Cho and colleagues' (2020) laboratory study of YouTube discover that system-driven personalisation, wherein users do not explicitly customise their feeds and are mostly exposed to attitude-consistent information, increases attitudinal and affective polarisation (see also  for similar effects on attitude reinforcement). Conversely, Beam et al.'s panel study (2018) finds no attitude polarisation effects from Facebook use for political news but modest depolarisation effects due to increased exposure to counter-attitudinal news sources. Similarly, Heitz et al.'s experimental study (2022) shows that diversity-optimising recommender systems, which promote exposure to both counter-and pro-attitudinal political information, can enhance tolerance for opposing views.
The effects of the reliance on algorithmically driven news platforms on political participation have rarely been studied. For example,  find that exposure to algorithmically curated news on Google and Facebook is associated with higher online political participation.  panel study uncovers similar effects of exposure to political information through Facebook's algorithmically curated newsfeed on political participation. In contrast, 's findings suggest that the diversity-optimising news recommenders that foster the exposure to opposing political viewpoints do not enhance political participation.
Overall, to date, few studies have specifically examined the effects of recommender systems on news websites on political attitudes and behaviour (but see . Underexplored Areas. As empirical examinations of NRS and their implications for political behaviour specifically are scarce, two additional research gaps on the demand side emerge: To date, there has only been sparse examination of the effects of NRS on o) political knowledge and p) political attitudes and behaviour.

Interim conclusion
Research on the demand side has mostly investigated user attitudes towards and awareness of algorithmic decision-making. Studies have also addressed the extent to which news personalisation affects the establishment of filter bubbles, the diversity of readers' political news consumption, and to a lesser extent political behaviour, knowledge, and opinion polarisation. However, research has often focused on news aggregators such as Google News and social media such as Facebook rather than on recommender systems on news websites. There has been little systematic research on individual-level differences that affect attitudes towards NRS specifically and how the reliance on NRS steers subsequent news consumption, information reception, and political behaviour more generally.
Existing investigations have mostly measured cross-sectional news exposure with the help of quantitative surveys (e.g.  or experiments (e.g. . In contrast to studies on the supply side which are often qualitative in nature, such an approach is seldomly employed in research on the demand side (but see e.g. . Individual studies on NRS have employed longitudinal designs (e.g. ; see also e.g. Stark, 2022 for intermediaries). Recent studies have also made use of simulations and agent-based approaches, particularly in the context of news aggregators or social media (e.g. ; but see Haim, 2020; for NRS on news websites), as well as real browsing and tracking data (e.g. . However, thus far, studies that aim to incorporate communication and computer science perspectives, for example, by programming NRS that are responsive to real users' or computer simulated agents' interaction with them, remain scarce (but see Shin, 2020; for a simulation approach .
Demand-side studies have thus far focused on a handful of Western countries, including the Netherlands (e.g. , Germany (e.g.  and the US (e.g.  (but see e.g.  for non-Western samples). To date, demand-side studies have rarely employed a comparative approach across countries (but see e.g. , time (but see e.g.  for NRS), types of platforms (but see e.g. , or filtering mechanisms (but see e.g. .
Lastly, studies that explore the effects of NRS on news consumption, political polarisation, and knowledge have largely employed the selective exposure paradigm. Several studies have drawn on previously established empirical work rather than formulated theory-explicated assumptions (e.g. . Other theories that could potentially model and explain effects on political news use and behaviour have seldomly been considered (but see e.g. .

Conclusions: recommendations for news recommender research
Having summarised the literature on NRS, we are now able to draw conclusions about (1) the extent and range of scholarly literature on NRS; (2) the issues NRS-related research has focused on so far on the supply and demand sides; and (3) the diversity of NRS research in terms of methodology, theory, interdisciplinarity, and geographic scope.
Our first conclusion is that while a significant and steadily growing amount of scholarly work on NRS has been assembled in the past years, the theoretical, methodological, and empirical insights from supply and demand perspectives have yet to be synthesised. Such an encapsulated view on NRS can contribute to a fragmented research field. Thus, to help researchers see relevant connections in this emerging research field, we pursued an integrative approach. Drawing on political communication as an overarching framework and considering the potential significance of NRS for an informed democratic public, we aimed to lay the groundwork for a more holistic, systematic approach to NRS which considers the implications of NRS for the production, dissemination, and consumption of political information.
Our second conclusion about previously well-studied topics allows us to answer our two research questions. RQ1 asked, 'What scientific evidence exists regarding the use and impact of NRS with respect to news organisations?' Here, our review revealed that research can be categorised into three broader areas: 1) the current state of NRS implementation in news organisations; 2) journalistic attitudes towards NRS and their effects on news practices and role perceptions; and 3) organisational dynamics and hierarchies surrounding NRS. However, there are still significant research gaps in all three areas.
Although research on the actual implementation of NRS has recently increased, we still know little about how NRS adoption varies across different types of news organisations or political and media systems. On a micro level, research, as it is often case-specific and relies on qualitative data, has yet to systematically examine what factors influence journalistic attitudes towards NRS. Additionally, research has yet to thoroughly analyse whether the use of NRS has longer-term impacts on what political stories are selected and curated on news pages or how the adoption of NRS affects the representation of political reality on the production side. On a meso level, still little is known about how the inclusion of technological actors who are responsible for the maintenance of NRS shape organisational dynamics. Even though supply-side studies do not always examine the adoption of NRS in the political news domain in detail, our review indicates that such systems may have important implications for news work, organisational practices, and the realisation of values such as public service and autonomy which, in turn, can carry over to the representation of political reality and its dissemination in news content.
RQ2 asked, 'What scientific evidence exists regarding the use and impact of NRS with respect to news audiences?' Here, our review revealed that research can be categorised into three areas: 1) user attitudes towards NRS; 2) the effects of NRS usage on political news consumption; and 3) the political implications of NRS usage. Again, there are still significant research gaps in all three areas.
Although research has explored user attitudes toward NRS, the examination of a multiplicity of relevant factors using different study designs has hampered the clear identification of individual factors influencing user attitudes toward NRS. Furthermore, the relative importance and juxtaposition of perceived benefits and concerns as well as algorithmic awareness, specifically regarding NRS on news websites, still need to be analysed more systematically. The effects of personalisation on political news consumption and information receptionparticularly regarding the potential emergence of filter bubblesand implications for political behaviour have often been analysed in the context of social media and news aggregators. Thus, there is a lack of research specifically investigating NRS on news websites, their effects on the breadth and depth of political information reception, and their democratic implications for political outcomes, such as knowledge, attitudes, and participation.
Our third conclusion relates to the diversity of NRS research in terms of methodology, theory, interdisciplinarity, and geographic scope. Our review showed that NRS-related research has employed a wide range of quantitative and qualitative methods. Overall, research on the supply side has most often used a qualitative approach (e.g. , whereas studies on the demand side have, for the most part, quantitively examined user attitudes and effects on news consumption (e.g. .
Regarding the extent of theory-explicated work, NRS research on the supply side has often drawn on classical journalism or institutionalist theories. On the demand side, studies have predominantly employed the selective exposure paradigm. However, we also found studies that have drawn on previously established empirical work rather than on theory-explicated assumptions.
In terms of geographic scope, research on both the supply and demand sides is often casespecific and limited to a small number of Western countries, such as the US, Germany, or the Netherlands.
Taken together, our evaluation of the existing literature reveals that NRS are an increasingly popular research topic that can be studied through various methodological and theoretical lenses. Yet, despite, or perhaps because of, the rapidly increasing scholarly interest, the field remains divided along the fault lines of supply and demand-sides perspectives. Our review uncovered important research gaps that affirm that more research on NRS specifically is needed. Additionally, to aid future research in closing the identified political news-related research gaps, we would like to conclude our scoping study with five recommendations covering five aspects: the integration of the supply and demand sides, theory-explicated work, comparative research, interdisciplinarity, and industry-academia collaboration.

Recommendation 1: more research integrating the supply and demand sides
In today's high-choice information environment, the demand and supply sides of news are inextricably linked (Van Aelst et al., 2017). This is particularly true for NRS as their outputs both affect and are necessarily affected by user behaviour (Guzman, 2019;Abdollahpouri et al., 2020). As audiences' engagement with such systems steer their output, they can, in turn, also affect journalistic distribution practices and, ultimately, possibly decisions regarding future story selection and creation Carlson, 2018;. Therefore, similarly to political communication research in general (Van Aelst et al., 2017), future NRS scholarship should aim to consider the interconnectedness between the supply and demand sides.
One growing research strain that aims to do so theoretically analyses how NRS should be programmed to satisfy normative and democratically relevant journalistic criteria (e.g. . Such theoretical scholarly endeavours offer suggestions for how the supply of NRS should ideally be configured by taking into consideration users' demands as well as democratically deduced journalistic ideals. Specifically, scholars have argued that NRS should expose users to diverse information regarding topics, sources, and viewpoints to ensure an informed democratic citizenry . What is expected of an algorithm, however, differs according to the normative perspective and democratic model to which one adheres . Thus, the practical implementation of concepts such as diversity must be defined with a view of the requirements each democratic paradigm imposes on its media and citizens .
Transparency, accountability and explicability of the employed algorithms have also mostly been discussed theoretically (e.g. Bastian et al., 2019;Leerssen, 2020;. Studies havefurther pointed to the importance of enabling user agency through features such as customisability and the ability to turn off algorithmic recommendations . Additionally, individual studies have developed news recommender systems which can be calibrated with different democratically relevant features in mind (e.g. .
However, research has yet to empirically investigate the extent to which these considerations are present in current NRS design and how the realisation of concepts such as diversity, transparency, and user control (or, conversely, the lack thereof) affects user demand (but see e.g. . Such an empirical approach is crucial to study the interdependence between the supply and demand sides, as well as possible discrepancies in user expectations and actual NRS design. Additionally, studies should consider the reciprocal feedback loops that result from user signals and the outputs generated by NRS Guzman, 2019). Simulations, specifically agentbased approaches, may prove especially fruitful to capture these feedback loops for several reasons (Gilbert & Troitzsch, 2005). First, by emulating human behaviour, simulations can capture the dynamic effects of algorithmic recommender systems on recommendation outcomes and subsequent user engagement with the recommended content (Haim, 2020;. Relatedly, they also allow researchers to predict how such systems evolve in the long term, even if empirical data on their filtering mechanisms or user engagement with such systems is not available yet ; see also Bokányi & Hannák, 2020). Thus, in the case of NRS, simulations can help model how specific filtering mechanisms and algorithmic interventions affect both the diversity of the recommended news content as well as the diversity of users' subsequent news reading behaviour (see also Recommendation 4 below).

Recommendation 2: more theory-explicated work
Our review revealed that research on NRS has consulted different theoretical frameworks which focus on various micro and, to a lesser extent, meso-level or macro-level factors. However, our engagement with the literature also indicated that scholarship on the supply side has, in some cases,relied on a descriptive or a case-study approach. On the demand side, several studies follow an inductive, empirical rather than a deductive, theory-driven approach (see also Ahva & Steensen, 2020;Steensen & Westlund, 2021 for a similar interpretation of existing digital communication research). To model the effects on journalistic work, news consumption, as well as political attitudes and subsequent behaviour, future studies can profit from a stronger consideration of theories of journalism, media effects, and political psychology.
On the supply side, besides classical journalism theories such as journalistic role theory (Hanitzsch, 2007), Deuze's (2005) work on journalism as an ideology (for an empirical investigation, see e.g. , and gatekeeping theory (Shoemaker & Vos, 2009; for an empirical examination see e.g. , additional theoretical frameworks can also adequately model how the introduction of NRS affects communication flows and journalistic practices. As  argues, classical media theories, such as the two-step flow of mass communication (Katz & Lazarsfeld, 2017), can also be adapted to model the role that personalisation algorithms play in digital environments. Alluding to algorithms' contribution in personalising users' news experience,  delineates how algorithms, like opinion leaders who traditionally disseminated mass media content, 'can compensate for the lack of interpersonal conversation in the mass media era' (p. 304).
Another potentially fruitful framework might be institutional theory (Napoli, 2014;Scott et al., 1994) or ,more prominently, new institutionalism (Powell & DiMaggio, 2012;Ryfe, 2006), as it can shed light on the processes of legitimisation, imitation, and homogenisation of NRS usage within and across organisations (Caplan & Boyd, 2018). Studies have already discovered imitation processes in the implementation of data-driven solutions, such as audience analytics across different types of media organisations (e.g. Christin, 2017;Zamith et al., 2020; for NRS see . Other branches of institutional theory that focus on endogenous factors rather than on environmental pressures, such as the institutional logics perspective (Thornton et al., 2012;Greenwood et al., 2017), can also uncover emerging tensions between organisational divisions that adhere to different commercial and societal values but are jointly involved in the implementation and maintenance of NRS (for an empirical investigation see e.g. . To account for the multiplicity of actors who take part in the design of NRS, scholars have also argued in favour of a multistakeholder approach, which considers the interplay between the various stakeholders on the demand and supply sides who affect or are affected by the delivery of recommendations to users (Abdollahpouri et al., 2020;. To capture in a comprehensive way the cognitive thought and deliberation processes underlying the usage and effects of NRS on the demand side, i.e. for audiences, scholars could consult additional theories besides selective exposure. Media-effects theories, such as uses and gratifications theory (UGT) (Sundar & Limperos, 2013), can be used to model both user expectations towards NRS and perceived gratifications of personalised news consumption (for an empirical investigation see e.g. . Broader psychology theories, such as the elaboration likelihood model (Petty & Cacioppo, 1986) or the theory of motivated reasoning (Taber & Lodge, 2006), might also be employedto capture the effects of NRS on phenomena such as the breadth and depth of political information reception or political decision-making . For example, Beam (2014)'s experimental study draws on the elaboration-likelihood-model to explain why readers are more likely to process news content that isrecommended to them based on their political preferences more deeply than attitude-inconsistent messages. Similarly, the theory of motivated reasoning can be consulted to examine the motivational influences guiding political information consumption in personalised news environments, owing, for instance, to individuals' inclination to denigrate arguments that challenge their beliefs or to their political sophistication and knowledge (see also p. 397;Strickland et al., 2011, p. 937).
These theoretical frameworks can further be enriched with approaches that model attitudes towards algorithmic technologies, such as the Modality-Agency-Interactivity-Navigability (MAIN) model (Sundar, 2008;Sundar & Kim, 2019), which examines the role of different mental shortcuts such as the machine heuristic in decisions about the credibility of machine-driven communication (for an empirical investigation see e.g. Tandoc et al., 2020).
Other potentially fruitful theories include the technology acceptance model (Venkatesh & Davis, 2000), which predicts acceptance or rejection of algorithmic technologies based on factors such as their perceived ease of use (for an empirical investigation see e.g. , or the newly developed algorithmic persuasion framework, which models the persuasive effects of algorithmmediated communication on beliefs, attitudes, and behaviour .

Recommendation 3: more comparative research
Our review indicates that research on NRS has so far focused mainly on a few Western countries, with most empirical investigations being single-country studiesa deficit often found in political communication research (Van Aelst et al., 2017). However, contextual factors of different political information environments, such as the media and political systems, legal frameworks, or overall technological development, may influence the implementation of NRS and its effects on news use and user behaviour. Therefore, future research on NRS should more often pursue a comparative approach.
Especially in political and media systems that are less stable or non-democratic, NRS could potentially have important repercussions for news consumption and political knowledge if they are biased towards certain political opinions or if news portals are not transparent about their data practices . This highlights the need for more de-Westernized work on digital technologies such as NRS .
In addition, few studies on the supply side have investigated the implementation of NRS across different types of news organisations (but see e.g. . Comparing public service and private mediaor quality and tabloid media, might uncover important differences in NRS usage owing to divergent financial resources, target audiences, and overall journalistic missions.

Recommendation 4: more interdisciplinary work
The era of big data has undeniably changed the way we study social science phenomena (Bonenfant & Meurs, 2020). Consequently, communication and computer science now share more research interests than ever before. This has inspired various calls for stronger interdisciplinary collaboration between them (Bonenfant & Meurs, 2020). The study of NRS is a perfect example of a research field that is of great interest to both disciplines and can thus particularly benefit from a stronger interdisciplinary approach. To date, however, there have been few attempts at an integration of the different perspectives (but see e.g. .
Future NRS research should continue to focus on the technological advancement of algorithmic solutions (e.g. . However, it should also be mindful of the special characteristics and normative dimensions of political news media and what these imply for the technical implementation of NRS.
To this end, computer science can inform communication researchers about innovative and technologically advanced methodological approaches to NRS research. Simulations can help model aggregate effects that are difficult to capture in the field with real user data (Haim, 2020). Algorithmic audits can uncover important information about the filtering mechanisms employed by current NRS, to which communication scholars would not otherwise be privy . Additionally, computer scientists can shed light on the technical limitations and impact of current implementation efforts towards transparency, privacy, usability, or user-friendly platform interfaces and offer suggestions on how best to overcome them (e.g. .
Vice versa, communication scholars can enrich computer science research on NRS with insights from media effects and journalism theories on the role and impact of NRS for journalism and audiences. Moreover, political communication scientists can point to democratically and normatively relevant criteria, such as transparency and diversity, which need to be implemented in NRS design to ensure an informed democratic citizenry. From a democratic perspective, the consideration of normative and democratically relevant NRS features is particularly important .
In addition, interdisciplinary collaboration with law scholars can be fruitful, as examples have proven (e.g. Eskens, 2019;. Such collaborations, on the one hand, allow scholars to account for the implications of legal frameworks for the implementation and use of NRS. On the other hand, it seems crucial that insights from social science and computer science inform the regulation and governance of NRS.

Recommendation 5: more industry-academia collaboration
Many researchers have noted the importance and advantages of industry-academia collaboration in media and communication research (Freeman, 2016;Grubenmann, 2016;Jensen, 2020). NRS research can particularly benefit from such partnerships. To understand NRS' effect on news workers and audiences, systematic knowledge of the underlying algorithms is crucial. To date, however, most research on news personalisation has relied on incomplete information, agent-based testing, and anecdotal evidence about the inner workings of recommender systems. This is partly due to research focusing on intermediaries owned by international tech companies, whose personalisation algorithms constitute trade secrets (DeVito, 2017; Thorson et al., 2021).
Through collaborations with news organisations, extensive testing of existing NRS can be conducted using, amongst other approaches, simulations based on real audience data (e.g. . This can inform researchers about the real-life, possibly long-term implications of NRS for news consumption, user satisfaction, and political information behaviour. Moreover, insights from the newsroom can inform researchers about the current state of NRS implementation as well as news organisations' hopes and reservations related to the future of news personalisation. This knowledge can be used to inspire scholarly investigations of important, hitherto unexplored phenomena.
Findings can subsequently be relayed to news organisations in such a way that supports them in future NRS endeavours. Thus, collaborations between academia and the news industry can prove equally beneficial to both sides and create pathways to industry impacts with democratic ramifications.
Afterthought NRS can carry important implications for journalism, audiences, and democracy. The societal relevance of such algorithmic systems in the domain of political news is further reflected in the increasing amount of scholarly work on the topic. As the number of studies analysing NRS grows, however, so does the risk of the research field becoming fragmented across demand and supply perspectives.
As Van Aelst et al. (2017) argue, 'a full understanding of political information environments needs to take not only the supply side but also the demand side into account ' (p. 4). This is particularly true for NRS on news websites because their outputs affect and are affected by anticipated as well as actual user behaviour (Guzman, 2019). Thus, if scholarship is to fully understand the implications of algorithmically driven technologies that can play an incremental role in the dissemination and acquisition of political information, future research endeavours need to be mindful of phenomena both on the supply and demand sides. To lay the groundwork for such a holistic approach to NRS, we used our scoping study to present a programmatic research review on the topic of news recommender systems. Our goal was threefold: In a first step, we reviewed the literature on NRS on the supply and demand sides that has accumulated in the past twelve years, paying particular attention to the political news domain. In doing so, we were able to synthesise the key focus areas to date as well as identify conflicting empirical evidence and underexplored areas on both sides. Following this evaluation, we advanced five recommendations on how future NRS research can continue to evolve from a political communication perspective. We believe that our review can provide an important impetus for the maturing of NRS research and hope that this contribution will help scholars find inspiration and orientation in this exciting, rapidly growing field of research. Notes 1. In political communication research, the demand side traditionally 'encompasses how various segments within a society make use of political news and information' (Van Aelst et al., 2017, p. 4). Demand for and subsequently use of specific content or digital services can in turn affect audience behaviour. For example, in their article, Van Aelst et al. (2017) jointly examine 'the demand for and the effects of polarized news' (p. 13, emphasis in original). Therefore, in the following, we also subsume effects of NRS on political attitudes and political information behaviour under the demand side. 2. Although news personalisation does not necessarily equate with NRS, as recommendations constitute only one possible form of personalisation, some studies use these terms interchangeably (e.g. . 3. Additionally, a search of the relevant databases for studies prior to 2011 revealed only a limited number of studies that matched our eligibility criteria. Upon closer inspection, these studies examined less technologically advanced NRS solutions (e.g. Knobloch-Westerwick et al., 2005), which we believe cannot be treated as equivalents to more sophisticated NRS as examined in research published between 2011 and 2022. 4. Of the 18 'other' studies, 15 had a theoretical or normative focus, and three (3) were literary reviews with a computer science focus. It is further noteworthy that of the 24 demand side studies, six (6) measured or simulated users' interaction with self-programmed NRS, thereby at least partially accounting for the connection between the supply and demand sides.

Disclosure statement
No potential conflict of interest was reported by the author(s).