ChatGPT in Finance: Reliable Analyses or Hallucinations?
In short:
- ChatGPT to reach $1 billion in revenue by 2024
- ChatGPT’s speed in news and market analysis may allow it to consistently beat the market
- A ChatGPT-selected stock portfolio outperforms the top 10 UK funds
- Its current imprecision and general privacy concerns affirm it has a long way to go until it becomes the go-to tool in banking
ChatGPT is forecasted to reach 1 billion users by the fourth quarter of 2023, less than one year since its launch in November 2022. This now infamous AI chatbot has become the fastest-growing app to exist, reaching 1 million users in only 5 days and 100 million users in less than 2 months.
Figure 1: Speed of achieving 1 million users
Source: Statista
As Figure 1 shows, its rapid growth has conquered even the leading social media giants. OpenAI, the company behind ChatGPT, has existed for 7 years and enjoyed a generous $20 billion valuation after its launch. Having recently introduced the premium version, priced at $20 per month, OpenAI expects to generate revenue of $200 million in 2023 and $1 billion in 2024. Such projections have caused other AI stocks to soar, as shown in Figure 2.
Google’s announcement of AI chatbot Bard has led analysts to strongly recommend buying into this stock, while ChatGPT likely continues to impact Google’s bottom line. Chegg, an educational technology company, has seen its stock decline over 40% since it admitted a significant negative impact on growth prospects due to students shifting towards ChatGPT. On the other hand, Nvidia, the computer hardware manufacturer, is in luck since speculation that it will supply 10 000 graphics cards to support ChatGPT’s growth has boosted its stock by over 25% since the announcement.
Figure 2: Stock price chart of AI firms
Source: Bloomberg
It is safe to say that ChatGPT has taken the world by a storm, and its use is only expected to escalate. Students use it as a “reliable” study tool; artists argue that it’s stealing their work, and professionals in several industries raise concerns over ChatGPT taking over jobs. Linked to such worries are debates regarding the accuracy and reliability of the chatbot’s analyses.
Morgan Stanley analysts proclaim: “ChatGPT will keep hallucinating wrong answers for years to come”. The choice of words seems like merely an opinion on the chatbot’s inaccuracy but is indeed a real phenomenon in AI. Hallucinations occur when an AI model produces results that differ from expectations, are not connected to reality, and do not match data on which the AI was trained. In practice, this means the AI generates an incorrect answer with a plausible explanation to satisfy the user’s query, which implies its output cannot be fully trusted. Some examples of hallucinations include producing analyses based on fabricated data or articles with citations of non-existent sources. Before diving into the potential implications of such hallucinations, we must answer the question:
What is ChatGPT?
Without delving into technical details, ChatGPT is an AI (artificial intelligence) language model which uses natural language processing to interact with the user in a conversational way, allowing it to formulate answers in a human-like way, answer follow-up questions, admit its mistakes, challenge incorrect premises, and write answers in a format pre-specified by the user.
Anyone who has experienced a conversation with ChatGPT can admit that its vast and detailed “knowledge” of various subjects is impressive. So, where is all this information coming from? The chatbot obtains information from a wide range of books, articles, websites, and other sources on the internet and formulates it into a viable answer for the user. Even though these sources probably (hopefully) do not contain unsupervised information platforms like Wikipedia, their reliability and the accuracy of ChatGPT’s interpretation are still questionable. As OpenAI themselves admit, the chatbot’s newest version has become more convincing in writing plausible yet incorrect answers, and the company is not exactly transparent about its information sources. Considering the widespread use of ChatGPT as an information source for several applications, this may be cause for concern.
Applications in Finance – Potential Employee Threat?
Numerous potential applications in finance have been discussed with a focus on the banking sector. Alongside customer service and compliance uses, ChatGPT is expected to automate recurring tasks such as claims processing and analysing financial statements, allowing professionals to focus on the implications of the analysis. One of ChatGPT’s major advantages is its ability to source information within seconds, increasing the efficiency of tedious data collection and again allowing humans to focus on its interpretation. However, ChatGPT’s current hallucinations of plausible yet false information indicate limited reliability for this use, at least in the chatbot’s present form. Further applications in fraud detection, default prediction and determining the risk of investments make it a useful tool for assessing the creditworthiness of companies.
Even though ChatGPT’s ability to produce sophisticated charts is limited so far, it can generate R or Python code to create various data visualisations. However, not only is such code not always accurate, this hints at another major issue; ChatGPT cannot carry out the work of constructing an Excel model or data visualisation; it can simply advise on it. Yet, Microsoft stepping in with its third and largest investment into ChatGPT, this time worth $10 billion, shows the magnitude of its involvement, and the chatbot generating full Excel models or PowerPoint presentations soon would not be a massive surprise. Notably, Microsoft’s major investment at the end of January caused its stock to rise ca 28% since the announcement.
Most speculation revolves around trading and portfolio management applications, especially in the analysis of trends and market sentiment. This brings us to claims that ChatGPT will be able to beat the market. Is this chatbot going to solve, what many have tried, and tried, and miserably failed at? Is this AI superior to all the other algorithms produced for this very purpose? One US finance professor believes so on the grounds that ChatGPT is able to comprehend news, headlines, and reports in the same way a human can, which differs from past algorithms that focus on quantitative data. Since ChatGPT can pool massive amounts of information much faster than a human, it should carry out trades on such information extremely quickly, beating any human investor. If the market does not respond perfectly to the announcement of news, there is an opportunity to make above-market returns consistently.
In his study, the professor finds statistically significant accuracy in predicting stock prices when fed specific news articles. Clearly, there is some potential. Yet, ChatGPT is still facing hallucination issues, where it could easily produce plausible yet incorrect analyses, potentially based on non-existent data. Even if such issues are fixed in the future, ChatGPT is still just making predictions of future stock movements based on past information, as any human or algorithmic investor would. Therefore, as any investor, ChatGPT is bound to make mistakes simply because it cannot fully consider future events and act accordingly. Despite that, the speed advantage may be sufficient to beat the market on average, even if mistakes are made. Although, this advantage will be lost if many investors adopt the same strategy.
Since the chatbot pulls information directly from the internet in real-time when answering a query, it should easily access all financial data publicly available in a matter of seconds and produce a detailed stock analysis. Currently, ChatGPT refuses to do so due to its limited access to data up to 2021. Explicitly providing it with stock data and asking for an analysis yields the generic response that, as an AI language model, it cannot answer such financial questions. As its use grows, the development of a version for financial analyses is inevitable, whether this includes data extraction from public platforms or the use of user-provided data. Once ChatGPT gains the capability of producing full financial models, it should be able to provide price targets and ratings in a matter of minutes.
Finder, a UK-based comparison platform and information service, was able to have a premium version of ChatGPT select a portfolio of 38 stocks based on criteria determined by analysts. As seen in Figure 3, the portfolio increased by 4,6% within 45 trading days, while 10 leading UK lost 1,8% on average. The results of the graph are difficult to argue against, but we must not forget the threat of AI hallucinations in recommending stocks based on analyses of fictional data. Plus, knowledgeable analysts selected the criteria, so we must attribute at least part of the success to them.
Figure 3: ChatGPT-generated stock portfolio performance
Source: Finder
Still, questions remain, such as: Will ChatGPT choose the same portfolio when the same criteria are selected? How sensitive are its recommendations to changes in those criteria? Does it sufficiently consider investors’ risk appetite? Are recommendations on how long to hold the stocks provided? Caution is further emphasized by the fact that ChatGPT and alike tend to provide results with about 90% accuracy, which is acceptable for some applications but may not be the most effective tool in projects that require complete accuracy. Therefore, the chatbot’s output should be checked by finance professionals.
ChatGPT’s task automation and performance of investment analyses would allow institutions to become more efficient – but that’s exactly the concern. Increased efficiency and automation mean fewer employees are needed to accomplish the same tasks, which could have adverse effects on employees’ job security. Still, ChatGPT’s inaccuracies and current unreliability lead to the need for the approval of its analyses by finance experts. Therefore, ChatGPT is more of an analytical tool guided by human expertise, as opposed to a substitute for analysts, at least in the near future. For example, the head quant of Wolfe Research, an independent sell-side equity research firm, expects ChatGPT to aid sell-side analysts in writing earnings previews.
Privacy Concerns
Thanks to the easy access to ChatGPT, many people have jumped to ask it their most pressing questions, as evidenced by its over 10 million users. However, amidst the excitement, most of us forgot to stop and think – why would they offer ChatGPT for free? Especially since its tremendous popularity must have been anticipated to at least some extent.
OpenAI’s decision to make ChatGPT so accessible not only allows it to uncover and fix mistakes in its responses but it also gives the chatbot access to a whole new mountain of data. If you’re a student typing in some complex concept you do not understand, hoping for a comprehensive explanation, this is not a concern. But if you are a bank with valuable and confidential internal information, then leaking such information to an AI firm with its own interests is a big deal.
Aware of this, several banks, including JP Morgan Chase, Wells Fargo, Citi Group, Goldman Sachs, and Bank of America, outright banned any use of the chatbot by employees due to privacy concerns over their data, so analysts cannot take advantage of ChatGPT’s developing applications. Recently, OpenAI admitted to a breach in the system due to ChatGPT’s direct connection to the internet, which led to it being taken offline until the breach was handled.
Privacy concerns go beyond the financial industry, where Amazon, Verizon, Accenture, and firms alike have also announced a ban. Concerns over ChatGPT violating European GDPR rules have even led to its temporary ban in Italy. Clearly, ChatGPT still has plenty of issues to fix before corporations may use it for certain responsibilities currently being performed by analysts, especially in the banking and finance industry. Still, it is likely that at some point, there will be a secured version of the chatbot tailored to the financial or any other industry which accesses only internal company data.
Bottom Line
So where does this leave us? Although ChatGPT seems like this incredible revolutionary tool in the eyes of the public, there are still many issues that must be addressed before deploying it as a full-fledged tool in any professional environment, let alone in the finance industry. Nevertheless, improved versions of ChatGPT are expected to come soon, and other companies are launching their chatbots to compete – Google’s Bard, Duolingo’s chatbot and others. Generally, ChatGPT is seen as an aid rather than a replacement for analysts and has the potential to transform our approach to portfolio management.
