During the last three years, LLMs in the form of conversational AI have managed to penetrate deeply into the everyday lives of many people with a working computer and an internet connection.

Importantly, we as users of such tools must not forget one extremely important fact. That is, any LLM-powered tool is not merely a result of a technological breakthrough that has magically made its way into our homes. The availability of such tools emerges as a commercial product, developed by for-profit organisations that have the ability to harness this technology and the objective to gain the advantage over their competitors. The main reason for its wide availability is because it enables large scale profit, not because of its potential at leaving a positive and lasting impact for humanity.

Like all products that are catered for the masses, an LLM chatbot needs to be marketed as a simple, specialisation-free tool that can be used by anyone, for anything. But unlike prior historical examples, the case of LLMs carries some unique attributes and differentiating factors. These are:

  • The mere usage of the word “intelligence” already sets a significant bias, as it leads casual users to expect one specific thing, a product that is actually intelligent.
  • LLMs, are largely “black-box” mechanisms. If domain experts have large gaps in their understanding of how LLMs develop specific behaviors, how can anyone expect them to responsibly educate others on safely using AI.

Given the above as some of my considerations, one could come to form a number of questions that must urgently be addressed. We list some of them one-by-one while also providing my argumentation on why these are condiderations that we must focus on:

  1. With the en-masse rollout of commodified “intelligence”, a company has the responsibility to educate the intended users on what exactly their product does and how it should safely be used. On the other hand, governments should require such companies to provide the best and most suitable form of transparency and guidelines through strict policy. Marketing an AI powered product as a companion, assistant or conversation partner comes with risks that non-specialised users must and should be aware of. The main blockage point for the adoption of this practice is of course the fact that it comes as a hinderance to simplicity. And lack of simplicity makes a product commercially unatractive and commercial unatractiveness is a direct impediment to maximised profit. It’s true that overloading the usage of AI products with a bunch of notices, do’s and dont’s, will make its integration harder for simple users , but the fact is, these simple, casual, non-technically trained users are the ones most vulnerable to the misuse of any intelligent tool. It is the tradeoff of Greater transparency and education against consumer objectives that the companies providing these services must face. It is therefore almost certain that without correct intervention and policy, these companies will keep offering such sophisticated tools without training for better use, in order to ensure their self-preservation and financial well-being.
  1. Providers need to be held accountable for their choice to roll out a powerful, black-box tool to the public. How can it be normal, that the same state of the art models are being freely and openly provided for household use while also being used for intelligence or defence operations? How can it be accepted that someone can organise their daily agenda using an LLM while someone in the US Department of Defence can use the same model to automate intelligence processes for usage in intelligence or even ground warfare? In general, how is it acceptable that OpenAI can give access to their latest, most advanced model for household use (without proper guidelines and precuations) while also providing the same models for defense and intelligence purposes (even if usage is limited to administrative documentation/reporting)?

To my defense, and certainly not to my surprise, real time events while writing this blogpost help to further strengthen the relevance of the above questions. During the last couple of weeks, Anthropic has revoked access to its new frontier model called Fable, after pressure from the US government citing important security considerations. Specifically, it is worth stating that apart from commercial access, Anthropic also prohibited internal use of the model for all employees who are foreign nationals. It is therefore interesting to think that the general public, for a very short period, had access to a tool that the US government itself regards as a potential risk for the country’s national security. Fable is now accessible again, with potential further safeguards now integrated. The US government might feel safer for itself, thus allowing the model’s re-entry into public access. What changes were made, and how these changes have made the usage of the model safer for the everyday consumer, is still unknown.

Human and AI: LLMs effect on mental health

Setting the boundary between personal level interactions of humans with an intelligent-like agent should theoretically be as easy as simply accepting the fact that a conciousless computer algorithm cannot ever act as an emotionally intelligent companion. The emergence of LLMs has given some and continues to provide insights that prove that these boundaries are far more difficult to define and understand in practice than theory.

Recent research publications like this one from MIT media lab attempts to examine psychosocial effects of AI-chatbot usage.

The headlining correlational finding was that higher daily ChatGPT usage correlated with higher loneliness, dependence, problematic use and lower socialization. Heavy users were more likely to consider the chatbot a “friend” or attribute human-like emotions to it. While acknowledging the proprietary nature of such studies and the need for longer-running and more well-informed research on the topic, these initial indicative findings can and should be considered in our cause for building a safer AI usage culture.

The Psychology today article called “The Emotional Implications of the AI Risk Report 2026”, references the above publication to intelligently state the following:

“A controlled study revealed that people with stronger attachment tendencies and those who viewed AI as potential friends experienced worse psychosocial outcomes from extended daily chatbot use. The participants couldn’t predict their own negative outcomes.

Neither can you.

This reveals an unsettling irony: We’re building systems that exploit our cognitive biases and the very psychological vulnerabilities that make us poor judges of AI risk. Our loneliness, attachment patterns, and need for validation aren’t bugs AI accidentally triggers—they’re features driving engagement, whether or not developers consciously design for them.”

Anyone who has used any prominent LLM for conversational purposes can observe at least one of their consistent behavioral characterestics. This is none other than the LLM’s proclivity to generating affirming responses that agree or even encourage the user’s thoughts without actually logically challenging them. Also known as sycophantic behaviour, this frequently exhibited characterestic has been the focus of several recent research efforts, whose findings raise an alert which can’t and shouldn’t be ignored. Indications tying sycophant AI systems to the user’s hindered social judgement are shown in this Article from March 2026, which concludes that this is a prevalent behavior with broad downstream consequences.

Illustration of a sycophant
Source: wordinfo.info/results/sycophants
Depiction of sycophantic behaviour

Inspired by these observations, and as a precaution for avoiding even larger and potentially catastrophic consequences, it is of utmost importance to adress the right questions sooner rather than later. It is also important to realise that the responsibility of adressing these issues is not entirely concentrated towards the CEOs or the owners. A corporate executive is an expendable, replacable and mostly predictable link of the chain. The collective bodies of scientists and engineers are the true foundations of the advancements we see today, and the collective behaviour and ethical stance of these communities when facing these issues holds much more weight than the signature of any big tech executive. Being aware of the responsibility and value we carry is therefore a step much needed towards developing ethical, human centered AI.