Your own personal Jesus
Someone to hear your prayers
Someone who cares
Your own personal Jesus
Someone to hear your prayers
Someone who’s there

Feeling unknown and you’re all alone
Flesh and bone by the telephone
Lift up the receiver, I’ll make you a believer
Take second best, put me to the test
Things on your chest you need to confess
I will deliver, you know I’m a forgiver

Reach out and touch faith (Depeche Mode - Personal Jesus)

Some time ago, I was preparing material for a seminar on societal and ethics issues related to AI. The target audience was lower secondary school teachers. Due to the current context and the desiderata legitimately expressed by the organizers, where you find AI, you should think chatbots employing some LLM. Looking at interesting online material to warn these educators about risks of an unaware adoption of these tools, I found a report on “How People are Really Using Generative AI Now” by Marc Zao-Sanders. I did not have the time and even the competence to review the adopted methods— that are rooted in qualitative analysis— so it should be clear that I’m not proposing it as a definitive response to the question you would get adding a question mark at the end of the title. I certainly think that there are cultural differences that influence the use of such tools in different countries, but still seeing uses labeled as “Therapy / companionship”, “Organise my life”, “Find purpose” above utilitarian usages is striking.

This immediately reminded me about a long article titled “AI Is a False God” by Navneet Alang on The Walrus, and specifically a quote:

If you find yourself asking AI about the meaning of life, it isn’t the answer that’s wrong. It’s the question.

However, I don’t want to look down at people actually using chatbots for reasons that I find completely unreasonable (for both technical and philosophical reasons). I’d like to try and understand how it is that someone ends up looking for companionship or asking questions about purpose in life to a computer program. Reasons are not just to be looked for in the users but also in the technology.

Bearing this in mind, now it is probably clearer the initial quotation from the song by Depeche Mode. It is striking how frequently statements crafted as hyperboles fail to withstand the test of time, as they are invariably surpassed by the realities that emerge. Back to the point, what if AI becomes not a false god but a personal Jesus?

As you probably know, any modern LLM has a very long and quite complicated training process. In addition to a phase in which the training process elaborates data as it is, in an essentially unsupervised way, there are subsequent phases in which the resulting model is aligned to fit the intentions or preferences of their developers and the needs of users. OpenAI, in particular, adopted a Reinforcement Learning from Human Feedback approach, in which the model was steered (Reinforcement Learning) to produce responses in tune with a model trained to generalize results of user annotations on sample dialogues (Human Feedback). Other LLM producers employ supervised approaches, but I feel reasonably certain that all training processes for modern LLMs include an alignment phase in which user evaluations and preferences are considered.

To which extent this phase influences the responses of an LLM-based chatbot, it is difficult to say: not just due to the complex training process (not just complicated: the overall training process typically includes results of different and mutually influencing activities carried out by humans and machines, with lots of implications of local choices and actions), but also because it is not the last phase before the delivery of the chatbot.

Chatbots, like ChatGPT, Claude, and many others, are offered “as a service”, typically in a freemium model, but the services are still largely operating at a loss. To avoid various types of issues, such as spitting out copyrighted material, supporting wannabe terrorists, providing responses that might be considered controversial by users, LLMs used by these chatbots are provided with a system prompt, a set of behavioral guidelines, and specific rules to follow, hidden to the user but always added before the start of any dialogue as input to the LLM. A recent post of Ars Technica comments on Claude’s system prompt, following an analysis of information shared by Anthropic in public release notes about the chatbot.

What’s interesting about Claude’s system prompt is that it is trying to avoid a relentlessly positive tone and excessive flattery in its responses.

“Claude never starts its response by saying a question or idea or observation was good, great, fascinating, profound, excellent, or any other positive adjective,” Anthropic writes in the prompt. “It skips the flattery and responds directly.”

If you use these systems even on an occasional basis, you know that sometimes it feels like being Harry Potter talking to Dobby, just to drop another pop culture reference.

However, we now probably have some elements, some of the reasons for which users feel that an internet service accessible through the web can act as a companion: these services can definitely feel like one. They are extremely good at using language; they processed in their training a quantity of text that no human could really read in a lifetime. They are trained to produce answers that are appreciated by human users. They are instructed that their goals are to satisfy users’ requests, although with specific limits, and even to care about users’ well-being. It does not come as a surprise that a user can see such a service as a personal Jesus.

Let us, however, get back to the business model of chatbots, and to some extent of the overall industry behind them. Again, they are operating at a loss. Allow me a rhetorical question: why? Well, I think it has to do with marketing, with the idea that a need can be created by manipulating customers. What scares me is the recurrent trajectory that many internet-based services and platforms especially have followed in recent years. An unpleasant trajectory that Cory Doctorow called enshittification. Maybe we won’t get there, but I am not optimistic.

Stability AI had trained Stable Diffusion on “more than 12 million photographs from Getty Images’ collection, along with the associated captions and metadata, without permission from or compensation to Getty Images, as part of its efforts to build a competing business.”

The sad news - or at least what I interpret as a sad news - is that Getty Images CEO has declared that it’s simply too expensive to fight every copyright battle, these days.

My immediate reaction to reading this news was “well, so much to establishing a fair use discipline in the GenAI era”. Quite simply, since so much money has flowed funding initiatives in the GenAI sector, how can even established and legitimate businesses like Getty Images, or the New York Times or even Wikipedia (OK, this is not a business, it’s a crowdsourced initiative offering digital contents through open licenses, but it is suffering from the implications of the rise of AI crawlers) think of defending themselves when even lawsuits are ineffective?

Moreover, part of the strategy of offering some of these chatbots and services basically without entrance fees is aimed at generating a shared feeling that these tools are not just useful, they are necessary! Maybe not the addiction cases that some are reporting, but a more diffused form of dependency, closer to those associated to smartphones, social media, pervasive communication. There’s a reasonably interesting book I can suggest on this topic: “Stolen Focus” by Johann Hari.

Finally, as per a previous comment a few months ago, the GenAI industry is leveraging nationalist feelings against rulings limiting the possibility to do essentially whatever they please with any digital content they can grab, legally or not, in the internet.

The overall combined effect of these points is somewhat clear. Even if some of these lawsuits would find guilty some GenAI company, many could end up being angry at the judge:

• for putting at risk our beloved assistants;
• for helping the Chinese attaining GenAI supremacy.

I’d love to be able to say something enlightening to elaborate why and how it is that we still don’t have a clear fair use discipline in the GenAI era. However, it is a lot better to suggest you reading The Code of Capital - by Katharina Pistor.

My interest in football is mild at best. I am fan of Inter Milano, but often I don’t know the players of the team or what is the result of the last game. This year, Inter had a very good run in the Champion’s League, and in particular it won against Barcelona in the semi-finals. These last two matches between Inter and Barcelona were particularly rich of goals, very spectacular; the return match was won at the extra time by Inter in the home stadium. Near the end of the regular time, Inter was one goal down, after being ahead in the first half, but a defender scored and tied the game in extremis. So, the game was particularly emotional, and it has certainly generated strong feelings, not just in the fans of the two teams.

Italians know something about winning very tight matches at the extra time: in the 1970 world cup in Mexico, the national team beat Germany 4-3 in the semi-finals, before loosing the final against Pelé’s Brazil. It was another long match, very emotional, that seemed endless to those living that moment.

If you talk about this match of Italian national football team to an Italian - if you find someone old enough to know something about it (I was born later, but this match was used as a theme for movies, and it is quite legendary, at least in Italy) - the general description will be that of an heroic feat.

The opinions about Inter’s victory against Barcelona, instead, will mostly fall in one among three categories: Inter fans will immediately draw a similarity between the two matches, while fans of other teams will vigorously reject this opinion (and they will cast shadows on quality of the referee of the semi-final against Barcelona), and of course there are those that do not know or even care about the topic of the discussion. Football can be quite divisive, polarizing. Probably just as politics, and maybe more. Sport, after all, has an rhetorical power and a smooth ability to put us in contact with deep sense of belonging. Something that we are probably designed by evolution to look for.

These recent events reminded me of a book I read that discusses about some of the causes of this phenomenon. The book is The Knowledge Illusion, by Steven Sloman and Philip Fernbach.

The book is about the fundamentally communal, collective nature of intelligence and knowledge. It is about the fact that, due to this feature, humans were able to bring about extremely complex technologies, whose realization spectacularly exceeds the abilities and knowledge of a single individual, by leveraging what a community was collectively able to produce collaboratively. These times put a lot of emphasis on competition, probably too much. We tend to neglect that even when we invoke competition as a mechanism to stimulate ourselves to do our best, it is often with the aim of a shared, common good, granting an individual prize as well to the winners.

Most of us don’t really know that the Earth orbits around the Sun, but we rely on commonly agreed upon knowledge, and at least most of us would argue supporting this statement against alternative positions. We can and do stand on the shoulders of giants, as per the famous metaphor. But this mechanism also makes us vulnerable to a form of hacking of our intelligence.

The subtitle of the book includes the statement “we never think alone”, something that - as some book reviewer said - is at the same time obvious as well as profound. We live in a world of communities, groups, teams, committed to values and knowledge systems. These systems can be sound or flawed, acceptable or aberrant, but that strongly depends from the point of view. Someone inside the group is, after all, at home, and quite ready to defend it in case of an attack, physical or metaphorical. This is one of the reasons making conspiracy theories so resilient (another one is that real conspiracies do exist, so it is sometimes inherently hard to tell true from false).

In my mind this book is very close to The Enigma of Reason, by Hugo Mercier and Dan Sperber (I talked about it in a previous post). They both talk about the working of our cognitive system, and its limits. They also act as a warning to everyone of us. We should be sufficiently humble to be skeptical of our beliefs and convictions. We should be more used to taking a walk in the shoes of the others.

But the book, and its implications, suggest me an organizational consideration. When we create groups, departments, areas, committees within an organization, we should be very careful. In particular when one such bodies has some form of power over another. First of all, members of a group, at least a group that works - not just a set of people in which an individual forces positions, decisions, duties unto the others, will eventually develop a sense of belonging. If the group value system is not well aligned with the one of the overall organization some form of tension will develop, sooner or later. Any group, in time, can develop dysfunctional dynamics due to groupthink: when harmony or conformity prevail over reason, collective decisions can even be worse than individual ones.

Unless there is a level in which tensions and even contrasts are acknowledged, recognized, in which parts, and their positions are represented, with dignity and citizenship, given room for discussion, the organization might end up developing dysfunctional dynamics. Sense of belonging might drop in many members, turnover can rise, and somehow the recent phenomenon called quiet quitting can be one of the consequences of the eventual feeling of powerlessness outside the most powerful groups. Quiet quitters don’t believe anymore in what they’re doing: to them, work is not part of their actual life, it’s just a duty somehow allowing them to live, to do things that actually matter.

At an individual level, allow me a few suggestions:

• if you are the go-to person for a certain kind of organizational task, if you have been in charge of something for a very long time, consider stepping down. Or, at least, consider paving the way for someone else to take up your role.
• if you are in charge of selecting members for a group, or have some influence on the selection process, bring on board someone that you know has a different point of view on some relevant part of the job the group needs to carry out, or some well known troublemaker. Friction, might be annoying, but it is not necessarily a problem: it might be actually helping you keeping away from groupthink and bad decisions.
• preserve and pursue doubt: some decisions can be obvious, but when tackling complex issues, after having listened to all relevant positions, should we always strive for complete agreement? Perhaps not. But keep the door open to getting back to the point in the future.

Let’s start with OpenAI’s recent cry for more latitude in their operations to preserve the supposed lead in the AI race against China. Basically, they are saying:

• if we take the position that training on copyrighted works isn’t fair use we are basically out of the game;
• even though we might respect this ruling, there are bad boys out there that will not, and they’re foreigners;
• anyway, we have a lot of mechanisms in action that will prevent our chatbots to basically spit out copyrighted material in the original form. Trust us on that, and be flexible on how to judge if we are right. Even better, let us decide.

If you want to know more about fair use, or if you know about it but you want to understand if generative AI qualifies for fair use read what the sadly deceased Suchir Balaji thought about GenAI and fair use.

OpenAI is not alone in this crusade against copyright rules that favor content creators and intellectual property owners. Google recently joined them, and I suspect that most big tech companies are completely aligned with them.

Amazon, for instance, is making a very clear and bold move: Echo devices were (to some extent) able to process Alexa requests locally, and avoid sending voice recordings to Amazon’s cloud. It looks like this will no longer be the case as of March 28.

I don’t have an Echo device, I never wanted something constantly listening to me all the time as a primary function (remember, it has been proven that if you’re not careful your mobile phone can surely do that as well). If you think that Amazon was surely looking at all the generated results of the local processing, and that it does not seem a huge change, maybe you’re right. But Amazon does have a track record for mismanaging Alexa’s voice recordings, and deepfake frauds are becoming easier to perform every day, and you might want to minimize your voice and video recordings widely available in the dark (or, what, white?) web.

I think this scenario is the best possible advertisement for a book titled The Age of Surveillance Capitalism, by Shoshana Zuboff.

It is really a massive book, and certainly not even an easy read. Mainly it is not easy due to its message, because it is scrubbing in our faces an unpleasant truth: the optimism of the Internet’s early days was naivete, and it is always bitter to realize that you were naive.

Quoting Shoshana Zuboff:

“Surveillance capitalism is best described as a coup from above, not an overthrow of the state but rather an overthrow of the people’s sovereignty and a prominent force in the perilous drift towards democratic de-consolidation that now threatens Western liberal democracies,” - The new tech totalitarianism by John Gray

It is also pretty disturbing that the just started second Donald Trump’s administration includes someone like Elon Musk, that seems to me to be evoked by Shoshana Zuboff’s words, as well as fitting the Spider Man’s meme.

Consider, for instance, that he’s the owner of an automotive company, very much involved autonomous driving, and now he’s in the position to influence the National Highway Traffic Safety Administration… it is not exactly reassuring, considering data reported by George Mason’s University Professor Missy Cumminngs, showing that self-driving technology is far from safe as of this moment.

Just as OpenAI’s and Google’s arguments serve as perfect marketing for Shoshana Zuboff’s book, Trump’s administration itself is a stark reminder that the concerns raised in The Age of Surveillance Capitalism are not some outdated Marxist ghost from the past. They are terrifyingly real—and possibly even worse than the book’s most pessimistic readers anticipated.

It’s hard to summarize the book, that is very rich and thought provoking, but I would say that it is an investigation on rationality, not unlike the probably more famous Thinking, Fast and Slow, by Daniel Kahneman. Where Kahneman postulates the existence of two systems in our mind (System 1 being the fast, intuitive, and emotional, System 2 being the slower, more deliberative, and more logical), Mercier and Sperber propose a highly modular cognitive system, in which different specialized modules, typically of bayesian nature, have developed throughout our evolution. Reason is one of such modules, that developed late in our evolution, not so much to enable us solving abstract, logical problems, but rather to solve problems, and leverage possibilities, posed by living in groups. As a consequence, we also developed the possibility to make abstractions, develop formal mathematical and logical frameworks, even discuss rationality describing it in a quantitative way. But that is probably an abstraction, that does not really fit well the working of our cognitive system.

I don’t know if I was able to provide a reasonable summary of the book, but anyway, consider reading it. It’s really worth the effort and time.

In Computer Science and in Artificial Intelligence, the term reasoning (and probably I should talk about automated reasoning) also has a long history, with roots that definitely predate both the disciplines. Automated reasoning was probably an inevitable development of formal logic, and a significant portion of AI researches and results, those we refer to when we talk about symbolic AI or GOFAI, are essentially based in this line of work. Although probably no one would go as far as claiming that we reason similarly as a logic programming system, there surely are situations in which logic programming systems work very well, although they are not currently very trendy. It is no surprise that Knowledge Representation and Reasoning is a typical topic of interest for AI conferences, as well as a topic for sessions in their programmes.

LLMs and LLM based chatbots are now a much more popular topic of research, experimental application development, attempts to develop new systems, applications, services. We still need to understand the economical and energetic sustainability of LLM based products and services, and this will require some time. The simple, but extremely powerful, mechanism of operation of an LLM is sometimes summarized as “autoregressive token generation”: essentially, given the textual prompt, the LLM iteratively (i.e. word by word, or token by token) predicts the next word, based on the information internalized during the training (I am honestly upset by the proliferation of new terms such as pre-training and… what, post-training?). Early experiments with LLM based systems revealed, although LLMs are mostly great at producing plausible continuations of phrases, and responses to straight questions (also called zero-shot prompts), they were not an oracle. They often made (and sometimes still make, despite the advancements from the earlier generations) factual mistakes, despite the linguistic plausibility of the overall result. It was also found out that, in particular in cases in which some mathematical calculation or simple logical (or commonsense) reasoning was necessary, it was often effective to add an expression such as “Let’s think step by step” to the prompt. This sentence asks the model to “reason out loud” and to go through all the required steps to carry out the task. This discovery was instrumental in the definition of the so-called “prompt engineering”, which is from my point of view a practice of defining and refining prompts to acquire the desired results from an LLM (engineering is probably excessive). The “chain-of-thought” prompt technique also led to a plethora of works trying to improve the LLM reasoning process, in a lot of different ways (some of which of “agentic” nature, something relating to a recent post in my blog).

I’ll tell you a secret: I felt upset in using the term “reasoning” to describe the LLM elaboration of a response to a prompt. At the same time, I eventually used the term anyway.

I usually pay attention and try not to anthropomorphize AI techniques or results.

However, it’s probably part of the human nature to do that. I can surely remember pareidolia, our tendency (as humans) to give a meaningful interpretation on a possibly random stimulus, usually visual, so that one detects an object, pattern, or even meaning where there is none. And I often see faces in a lot of stuff.

Moreover, LLMs are terribly good at having you forget that you’re interacting with a computer program. Ethan Mollick’s book is full of examples in which this happens, and the author even suggests actively forgetting that behind the chatbot there’s an LLM and think of being interacting with an intelligent, sentient entity.

Still, I’m upset with myself.

I’ve been doing research in AI, although not specifically in Knowledge Representation and Reasoning, for a long time. I am serving in the organization of scientific conferences on AI. It’s not that I consider the term reasoning to be copyrighted, and that one needs to pay a royalty to the Knowledge Representation and Reasoning community. Certainly, we are going to have a problem in organizing the programme of conferences in which different papers might attribute extremely different semantics to the term reasoning… we should get ready to discuss as a community long standing traditions, implicit definitions, habits, and maybe tear them down. We should also somehow think back at the schema in Russell and Norvig’s AIMA book: I used to think I was working in a quadrant where people were mostly trying to make agents acting rationally, but it seems that investigating how to build agents that think as humans might be the next frontier of AI.

I internalized that the term reasoning is stronger than a simple processing, that reasoning was generally a process preserving truth along its unfolding, assuring some property. It seems ironic that we get back to something that, at the same time, looks closer to human reasoning, being potentially prone to error, but also (from my point of view, and based on what I remember of Mercier and Sperber’s book) only good at representing human post-hoc analysis of how a decision has been taken, for sake of justification or communication.

This week has been particularly lively and eventful, with a new kid on the block of commercial LLM based chatbot services: Deepseek launched an “open weights Reasoning model” (I feel the urge to talk about the term reasoning for LLMs, but that will need to wait) and made it available through web and mobile phone applications. What made this launch particularly intriguing were:

• The astonishing cost efficiency: Deepseek disclosed that its operational costs are orders of magnitude lower than OpenAI’s estimates for running its larger ChatGPT models;
• The use of non-cutting-edge hardware: Deepseek reportedly did not rely on the latest and most expensive Nvidia GPUs.
• Accusations of service misuse: OpenAI swiftly accused Deepseek of improperly using its services to extract and distill information, violating OpenAI’s terms of service;
• the fact that the company is Chinese.

The first two points caused an understandable shock in the stock market. A company’s stock value is basically a concrete representation of the current collective expectations about the revenues that company will provide to shareholders in a relatively limited time frame. It’s therefore really clear why the stock value of many companies dropped as a result of this news. After all, the value of many of those stocks had grown incredibly in the last few years, to the point of being considered by several analysts the next bubble. Ironically, there is no independent verification of Deepseek’s cost estimates or its claim of using less expensive hardware. But that’s the market, and no, the market is not always right.

The third point - the accusation of having distilled information from OpenAI’s models - is the most fascinating to me. It highlights at least two types of ideological stances.

For one, OpenAI and other major LLM developers have trained their models using massive amounts of data scraped from the internet, often disregarding terms of service and possibly even breaching copyright laws, not just terms of service in a contract among a client and a provider. There are ongoing and very relevant lawsuits questioning the legality of such practices, particularly regarding fair use (by the way, do yourself a favor and read what the sadly deceased Suchir Balaji thought about GenAI and fair use). I suspect that the comparably larger outrage in this case is due to the fact that the Deepseek is a Chinese company (the fourth point in my list), whereas OpenAI is an American one. With reference to the “moving fast and breaking things” tendency, these companies don’t seem to be acting in a fundamentally different way, though. Someone also commented that “karma is a bitch”, or (using a quote of Chinese roots) “Don’t do unto others what you don’t want done unto you”.

Now, you probably read about these events already, and with more interesting/detailed/intelligent comments than mine… but I want to connect and an additional bit, that you probably missed in all this commotion. Just few hours before the Deepseek news, ArsTechnica posted about a sort of server-side malware targeting crawlers that ignore robots.txt, a tool (and approach) to trap and trick AI scrapers that don’t respect the request not to index areas of a website, ignoring the robots.txt file in the server. The post is very interesting, and reports parts of an interview with the person that developed the so called “tarpit” trick. The approach does not just cause inconveniences to the crawler but it also injects information that could poison the training process. Of course, OpenAI said that they have already developed countermeasures to these poisoning attempts, but it is clear that response to their MO (due to the grey area in which they are operating the acronym seems particularly appropriate) has moved beyond courts of law, some people are taking the matter in their hands directly. So, the general sentiment has changed, and besides hopes, enthusiasm, and fear, AI (or, more precisely, AI companies) has also started to generate genuine hostility, even hatred. This is a bitter consideration for me, and for the portion of the academic world that is working in AI, that will need to carry part of this burden.

More broadly, these conflicts reflect unresolved tensions in the digital economy and in the development of the complex socio-technical phenomenon that is the Internet. Once again, these events are somewhat reminiscent of the controversies that had arisen between content producers and social media companies (something that had a role in the rise of the “click bait” phenomenon). The Internet is a very complex ecosystem, and due to its size, spread, and penetration in human activities it has reached a huge relevance, although - paraphrasing Morozov - it did not save everything. It did change it deeply. Within this process, sometime curious, bizarre, sometimes grotesque statements have been uttered very seriously, but often instrumentally, like “ad-blockers will kill the Internet”.

We are still grappling with unresolved issues, and time continues to bring them back to our doorstep. The real question remains: will we finally address them, or just wait for the storm to pass—again?

I think there are motivations to this, and they are related to the fact we are in a phase in which we have this new promising approach and technology and we are understandably trying to understand it, its limits, and potential applications. It’s a pretty wild phase, potentially a revolutionary moment in time, deeply changing the way things are done in a large set of areas in ICT, and in many human activities. It’s also understandable that people working in those areas, that maybe have spent years building an awareness of the problems and a competence in their solutions, are cautious, conservative, and maybe even angry to see a context in which they though to dominate change so radically, so fast.

However, I also think it’s not just a psychological and quite human reaction that however has no real reason, no rational or scientific motivation. Let us consider one of the recent concepts that has been put forward in LLM related developments, that deals with one of the limits of this novel technology. LLMs have been developed with huge investments because of their generality, because they were showing capabilities that were not initially intended; well, maybe someone was hopeful, but there was not guarantee that you could equally ask an LLM to correct a text you’ve written from typos and maybe change its style to make it less colloquial, more formal, and more business ready, and also ask the LLM to be the dungeon master in a role playing session. Despite their generality, LLMs lack intention. This has been a topic of discussion at the intersection of philosophy and technology, with some arguing that LLMs cannot be considered agents because action is understood as autonomous, intentional behavior.

Agents, and multi-agent systems, are not a new concept even in computer science and engineering, though. The concept of agent is basically the starting point of Russel and Norvig for the “Artificial Intelligence - A modern approach” book, but they are also instrumental in Genesereth and Nilssen’s “Logical foundations of Artificial Intelligence”, and certainly many more, before and after the ones I mentioned. The mentioned books include reactive agents architectures, with or without an internal state, that are not necessarily provided with a formal explicit representation of a goal. Think of boids, for instance: very specialized “social” agents exhibiting a nice form of overall resulting system level behavior, achieved as a consequence of very simple reaction rules. We are, after all, seeking the knowledge and capacity to build artificial systems implementing our goals as modelers, designers, engineers, users.

It does not represent a surprise, at least to me, that the trend in LLM related developments has come to the point of including a notion of agency. It is a very obvious development: since LLMs are so good in responding questions, in preparing texts of different kind, and even computer programming code, for us, why not delegating more complicated tasks to them? Well, in this case, we naturally need to provide LLMs with an architecture granting them a (to a certain extent) persistent existence, possibility to interact with external tools, maybe with other LLMs (other agents), and maybe also spontaneous behavior (technically maybe achieved as a reaction to the passage of time). The surprise is that people doing this, even in the academia, are essentially neglecting the body of work that has been carried out by the autonomous agents and multi-agent systems research community in over thirty years of work. I made a comment on someone’s post in LinkedIn about this point, basically saying that the current trend in computer science and engineering is the opposite of the punk rock movement: punk rock was about “no future”, whereas nowadays developments in (at least certain areas of) computer science and engineering are “no past”, since they do not even bother considering past research and experiences.

What’s funny is that this area of research is not exactly a small one. There is an International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), that sponsors the annual International Conference on Autonomous Agents and Multiagent Systems (AAMAS) (reaching the 24th edition in 2025). This conference is quite recognized in the computer science community, for what it’s worth it even has a very good ranking in the CORE Conference Rankings. My personal point of view is that, regrettably, research on agents and multi-agent system has brought results whose impact has not been as deep as what has been achieved by other fields of AI. I remember a provocative Jim Heldler’s editorial titled “Where Are All the Intelligent Agents?”, and that was quite some time ago (very good researchers proposed replies to that editorial highlighting achieved results): to some extent, I think that he had a point, although he was of course provocative. On the other hand, I also think that this is scientific research, and that this is what we think now. Research is a risky enterprise, there is always the chance that some line of work turns out to reveal unexpected or unsolvable issues, or the motivations behind it change and then the whole effort becomes not motivated anymore. In the mid nineties some very serious and skilled software engineers were investigating code mobility; ten years later their perspective was obviously quite different. Researchers in the area of autonomous agents and multi-agent systems can be angry right now, they can be thinking that LLMs and LLM-based system need to integrate additional concepts, mechanisms, results from their own field and also acknowledge that what is going on is not completely new. Similar considerations could be done for semantic technologies, although knowledge graphs are generally much more recognized by LLM researches and developments (see, events seem to show that maybe Jim Hendler had a point after all). So, my final point is: I think people using terms and concepts related to autonomous agents for LLM based developments, especially in the academia, are plain wrong in not considering and acknowledging past researches, results, and even failures to deliver expected results and impact (we don’t want history to repeat in these failures, do we?). On the other hand, they also need to consider that there’s this niche, there’s a chance to give a contribution to the overall scientific process, to maybe change its direction for the better (hopefully). It would be childish not to consider this possibility, as a research community, just because we are being ignored.

I was thus thrilled to read a post on ArsTechnica about an experiment comparing human and ants on solving a problem requiring cooperation. Essentially, a group of individuals must move a large object through bottlenecks resembling a maze. The object’s weight (resembling a piano, hence the ‘piano-mover puzzle’ name) makes it impossible for a single individual to complete the task. Its peculiar shape further complicates determining a trajectory for movement.

An intriguing aspect of this experiment (which might even be considered for an Ig Nobel Prize, potentially more so than the paper on the phase behavior of the cacio and pepe sauce) is that when humans are restricted from verbal or non-verbal communication, the group’s overall performance becomes worse than that of an individual.

The paper is published on PNAS, and if you’re curious - or if you think that my description is unclear, which is probably true - you can take a peek there.

Results, by the way, seem very much in tune with a conjecture described in The Enigma of Reason, by Hugo Mercier and Dan Sperber: their theory is that reason has appeared relatively late in our evolution and it enabled communication, cooperation, and also justification of our behavior. They also show that in different situations problems hardly solved by an individual are more tractable for a group of people, provided that they can communicate, discuss, also argue about things like the best line of work to follow in a given situation.

This does not negate the so called wisdom of crowds (from a book by James Surowiecki), the capacity of large groups of individuals not necessarily communicating with each other or however coordinating to come up with a joint solution, to collectively perform better than the best individual among them… it just shows that this is not always the case. To be honest, Surowiecki mentions a few cases in which crowds (and markets) fail, the book (at least in my memory of it) is not a universal praise for markets as mechanisms for decision making… but others often tended to simplify the message.

One of our collective decisions as human group was that research funding should be allocated by means of competitive schemes. I had the chance to discuss recently the news about a non competitive research funding scheme in a post in which I tried to look for papers trying to evaluate different types of research funding schemes.

More recently I was suggested by one of the authors to read another PNAS paper about the costs of competition in distributing research funds. I was quite happy to receive this suggestion, since it is a topic of interest to me, and also given the quality of the work. Authors analyze the different types of costs associated to competition, not just the economic ones, that are not trivial to evaluate, but that are certainly not the only ones.

There are also epistemic costs, for instance related to the fact that high-risk (and high-impact) research is generally not favored in funding decisions. Research project proposal always must claim to be leading towards groundbreaking results… they just need to show a credible operational plan, sometimes including risk analyses and contingency plans. These biases in proposal evaluations influence researchers’ behaviors, much like the predominantly bibliometric approach to research evaluation has, in my opinion, contributed to the inflation of published papers (I talked a bit about it in several posts, also dealing with public goods game) and I like to remind you to read an important work on this topic.

There are also additional social and ethical costs of competition, intertwined with epistemic ones, such as the the sometimes unbearable pressure to win funding that has a significant impact on mental health and work-life balance of researchers, especially in the early stages of their careers. The competitive nature of the overall system also affects the local scale of the system, with work environments like departments in which colleagues are certainly not encouraged to collaborate, reducing collegiality.

Authors explicitly say that they do not suggest that more non-competitive, block funding, and a reduction of competition in the system would certainly reduce these problems. Nonetheless, they certainly propose a wide set of references to literature works that do build a case for a causality between at least the current level of competition and at least some of these problems, but this is my point of view. The authors make a crucial point: we are not even attempting to evaluate whether our current approach to managing research funding achieves its intended effects or works effectively. I add, metaphorically speaking, that we are acting like the humans in the piano-mover experiment that are not allowed to coordinate with each other.

I often had the impression that academics have often an excellent understanding of some portion of reality, its internal working, mechanisms and techniques… but they are often pretty bad at putting their knowledge in action just inches (or centimeters, if you prefer) outside their specific area of expertise. I am not exception to this, despite all my efforts. This probably has to do with the way our performance is evaluated, and/or to the way different forms of knowledge are organized and compartmentalized, sometimes in hard to escape silos.

So, I join the authors in urging the overall scientific community to do like them, and like the authors of the paper on “the strain on scientific publishing”, and start considering more the possibility to apply scientific research to scientific research, and to ask that scientific approaches are applied by funding agencies. We must ask funding agencies to self-evaluate their mechanisms much more substantially and systematically. We are being asked to comply to recommendations and good practices proposed by the Open Science movement, starting from data availability. I want to undersign a simple statement by the authors that is “Nonsensitive data that do not raise privacy or data protection concerns should be accessible to the public without restrictions” (and I would extend this consideration to states, local authorities, municipalities, public administrations, etc.).

Authors do propose additional recommendations, that are also very interesting (e.g. systematically try to evaluate alternative ways to fund research), but this post is getting a tad too long, so I stop by strongly suggesting all readers to do themselves a favor: read the paper yourselves, think it over, and share it to other colleagues.

Bias refers to a systematic deviation from rationality or neutral thinking. Sometimes it can be a consequence of repeated experiences that lead to an often ineradicable set of beliefs and attitudes. Bias can therefore be the result of an overgeneralization, leading to erronoeus position. Bias, however, can also arise due to various forms of racism and forms of discrimination of gender, beliefs, political positions, and so on. In the latter case, bias could be fueled by what is called (ironically) confirmation bias: I have a prejudice and I actively seek information (not very important if it’s true or false) supporting my opinion, discarding any conflicting information as wrong, caused by prejudice or bias. If you’re thinking about social media, well, you’re right: echo chambers, sometimes fueled by recommender systems that are doing what they’re designed for (proposing appreciated contents), certainly have non trivial relationships with bias.

One problem with bias is that it can sometimes be (to a certain extent) related to actual experiences.

We also have a different, although maybe non that popular, name for mental shortcuts that we use to simplify some decision making activity also related to experiences: heuristics. Heuristics are pragmatic methods, often based in induction or analogical thinking, and they generally support us in taking decisions based on the idea that finding optimal solutions is sometimes impractical or even impossible. On the other hand, it can be relatively simple to find satisfactory or acceptable solutions. Of course, heuristics can fail or anyway produce suboptimal results, but nonetheless we generally project a more neutral intent to the term.

But how can we decide if we are facing a fallible but neutral heuristic rule, a rather neutral bias, or a bias rooted on prejudice, created and fueled like a self-fulfilling prophecy? And even when it is backed up by data somehow justifying the rule, that is therefore not exactly a prejudice, is it right to keep applying it or could it be more just, fair, transformative, and potentially even useful (in the long run) to bring some change in the state of affairs?

That seems to me a serious problem, and I think a problem that is not technical, is not a disciplinary problem for AI, or psychology and cognitive sciences. It seems to me a societal problem, a political problem, an issue pertaining the moral and ethical domain. I like to remind anyone taking the time to read these words, and wanting to go deeper in the subject, to take the chance to read a book by Cathy O’Neil that has sometimes been violently overtaken by events but that, in its essence, still holds most of the initial value.

I am not completely sure about what I think about the article: I appreciate the reference to Morozov and an anti pan-optimistic perspective on the implications of technological development. However, since it’s from a person whose cultural background is quite different from mine, and since it deals with topics I don’t have a deep knowledge of, but that I find particularly important, I have the decency to avoid trying to summarize it. I will, though, report a quote from the article that connected with some intuitions I had considering the recent trends in AI literature and some experiences from attending ECAI 2024. Here’s the quote:

If you find yourself asking AI about the meaning of life, it isn’t the answer that’s wrong. It’s the question.

I don’t want to talk about topics that, again, are out of the reach of my knowledge and competencies, like the question if AI could become a conscious, sentient being or (like in Navneet Alang’s article) if AI could be a “shortcut to objectivity or ultimate meaning” (if such a thing could even exist outside Douglas Adams’ book or a Prolog interpreter).

What really struck a nerve was the “wrong question” expression. As a researcher, I think it’s very important to periodically ask myself “Am I asking the right (research) questions?”.

I have always been a modeler. I always tried to come up with ideas about models that either mimic some form of decision making activity or that can plausibly generate, as implications of some internal mechanism regulating portions of a composite system, an overall observable behavior that is object of study. Models I came up with were interesting because they could, to a certain extent, be used as a proxy of a reality, to support some form of analysis or decision making activity. The models I usually built were largely manually defined, based on some understanding of the system object of study, coming from relevant literature. I generally tried to improve my knowledge about some reality to make better models, and sometimes models helped me understand better that reality. Some colleagues also more formally defined the concept of a “learning-by-modeling” approach.

Of course, this process has deep relationships with data, with empirical evidences, but they were used to inspire creative human work on models (either new or modified/integrated to better reflect some evidence), not to substitute it by actually learning a model.

Don’t get me wrong, it makes sense to try doing that, to investigate to which extent it is possible and what are the limits to this approach. I also started experimenting this kind of approach, learning decision making models for agents instead of defining them manually. I am not so sure this will be as useful to learn new things myself about the object of investigation, although I did start to learn something about machine learning.

Recently I have stumbled upon researches about the application of Large Language Models (LLM) for managing Non-Playing Characters in interactive systems (such as video games, Virtual Reality (VR) applications, and such), or some other application in which, basically, the LLM acts as an oracle, solving some problem of a connected system. I often asked myself: what’s the value of this research? What’s the complexity of this research? Well, in a sense, the work necessary to perform this kind of study is probably not that much, it basically leverages the work carried out to conceive, construct and train an LLM. However, despite the fact that maybe our work as researchers is not much, we can achieve an advancement of our knowledge as an outcome of this kind of work, if it is not carried out simplistically. LLMs are complex instruments, one could legitimately think that, as of this moment, some of the raised expectations will not be reached, but LLMs and their actual capabilities are a more than legitimate object of investigation. The question “can an LLM be used to carry out this task”, maybe adding “and how does this approach compare to alternatives ones”, generally makes sense, as long the research is carried out properly. In some time, I suspect we will see fewer researches like these, when we will have a better understanding of what can we do with LLMs, with which results, if and when it is convenient to use them instead of alternative, and typically more focused approaches. It will take time and honest, non biased, research.

I suspect that, sometimes, the feeling that it is a somewhat lesser kind of research is due to the fact that we don’t see the how the LLM solves the problem, we don’t really learn much about solving the problem, we delegate the solution to an oracle… we just try to see how good the oracle is.

Whatever is your point of view on this topic, I hope you agree that it is a good thing to stop and, maybe without necessarily taking the theatrical pose of Rodin’s statue, take the time to ask ourselves: “Am I asking the right (research) question?”