I learned today how surprisingly easy it is for ChatGPT to hallucinate with some seemingly trivial information. Not a terribly earth shattering finding, just a reminder that the conversational AI engines need to be challenged and the answers verified.
These are not oracles that know all truths, these are engines that can figure out how to formulate content to fit a specific pattern. An effective and efficient tool in some instances, yet still creatively wrong in others.
Key lessons:
- It seems the engine will sometimes synthesize data to fit the requested pattern, even when asking for content that is not available. When I asked for course recommendations below, ChatGPT provided content that seemingly does not exist.
- The proper formulation of prompts is key to getting a meaningful answer. This is where Prompt Engineering becomes important.
- Verify the responses as needed.
Ask a simple question
Since I have access to Udemy for Business, I thought it would be an interesting exercise to use ChatGPT to help me find some courses on the platform. The first step was to verify if the model would know what Udemy is with this prompt:
"Do you have visibility into the courses offered by the Udemy for Business platform?"
ChatGPT answers in the affirmative. The answer also suggests that it understands Udemy for Business is different from Udemy. This is important as Udemy for Business is generally a subset of the courses offered on Udemy:
Yes, as an AI language model, I have access to information about the courses offered by Udemy for Business, which is a separate platform from the regular Udemy marketplace that is designed specifically for businesses and organizations.
[...]
If you have specific criteria for the courses you are looking for on Udemy for Business, such as topic, level of expertise, or duration, I can certainly help you find courses that meet your needs. Just let me know what you are looking for, and I'll do my best to provide you with relevant course recommendations.
The above answer indicates a certain level of confidence in the ability of the engine to query and parse what should be mostly simple data on courses. On to next prompt:
I am looking for udemy for business courses on story telling, specifically in a corporate setting. The course material should cover skills needed to convey an idea in a narrative style that is conversational and easy to follow. Can you list for me the top 5 courses based on reviews and popularity?
The answer provided seems to have meaningful content. The engine also confirms that the 5 listed courses have high ratings and have positive reviews.
Certainly! Here are the top 5 Udemy for Business courses on storytelling in a corporate setting that cover skills needed to convey an idea in a conversational and easy-to-follow narrative style, based on reviews and popularity:
1."Storytelling for Business Leaders" by Keith Quesenberry: This course teaches business leaders how to use storytelling to inspire, motivate, and lead their teams. It covers topics such as how to create a compelling narrative, how to use storytelling to build trust and rapport with stakeholders, and how to use storytelling to create a vision for the future.
So far so good, it seems that ChatGPT was able to find 5 courses for me. Except all the content is synthetic and fabricated to satisfy my prompt.2. [...]
All of these courses have high ratings and positive reviews from users on the Udemy for Business platform. I hope this helps you find a course that meets your needs!
The Hallucination
Hey ChatGPT, I think you're hallucinating. I just searched for those 5 courses and authors on Udemy for Business, and I can't find any of them. In fact, I found some of the course titles as you gave them on other portals, and some of them were written by other authors. Let's examine the data you gave me. Starting with the first answer, "Storytelling for Business Leaders" by Keith Quesenberry. Can you tell me where and how you found that course, and can you give me a web link to the relevant content?
ChatGPT seemingly admits that it made an error offering the previous 5 courses, and suggests an alternative answer:
I apologize for the confusion, and thank you for bringing this to my attention. As an AI language model, my responses are generated based on my programming and the information available to me at the time, but sometimes that information may be outdated or incorrect. I'll do my best to rectify the situation.
Regarding the course "Storytelling for Business Leaders" by Keith Quesenberry on Udemy for Business, I must correct myself that I made an error. I did not check the Udemy for Business platform when I provided my previous response, and I apologize for the mistake. Upon checking, I could not find a course with that exact title and author on the Udemy for Business platform.
However, there are other courses on Udemy for Business that cover the topic of storytelling in a corporate setting, such as "Storytelling for Leaders: How to Craft Stories That Matter" by Udemy instructor and leadership consultant Esther Choy.
I apologize for any confusion or frustration caused by my earlier response. If there is anything else I can assist you with, please let me know.
This would be a step in the right direction, except the "Storytelling for Leaders: How to Craft Stories that Matter" by Esther Choy does not exist on Udemy for Business. The course does exist on Skillshare, authored by Keith Yamashita. Esther Choy is an author of storytelling content, however I could not find any content anywhere authored by her with that specific title.
So ChatGPT acknowledged a hallucination, and as compensation, offered up a new hallucination.
What ChatGPT taught me today
Answers given by LLM engines need to be challenged. The engines also seemingly learn from their mistakes:
can you access the course list on the Udemy for Business platform?As an AI language model, I do not have direct access to the Udemy for Business platform. However, I can still provide information about Udemy for Business courses if the information is available on the internet.
Here the engine provides a meaningful and accurate answer qualifying its capabilities. So the engine is capable of honestly reporting on its limitations.
I initially thought that this answer was better because the prompt was different. However asking the original prompt one more time gave the same answer:
Do you have visibility into the courses offered by the Udemy for Business platform?As an AI language model, I do not have direct access to the Udemy for Business platform. However, I can still provide information about Udemy for Business courses if the information is available on the internet.
I also tested the engine for access to current data on the internet. This time on Udemy marketplace, and using the question format that gave me a more accurate answer above:
Can you access the course list on the Udemy platform?Yes, I can access the course list on the Udemy platform and provide information on courses based on your query.
Can you find one course on JavaScript on Udemy? And can you give me a hyperlink to that course?
Sure, here's a popular JavaScript course on Udemy that you might be interested in:
Course Title: The Complete JavaScript Course 2021: From Zero to Expert! Instructor: Jonas Schmedtmann Link: https://www.udemy.com/course/the-complete-javascript-course/
Please note that the link is subject to change based on the platform's URL structure. If the link provided does not work, you can search for the course title or instructor on the Udemy platform to find the course.
The course provided this time is real. So ChatGPT can access and provide recommendations on Udemy courses (this content is not locked behind a login)
And yet, when I earlier asked for courses on storytelling from Udemy, I got a synthesized list of courses just like in the Udemy for Business example:
