Can ChatGPT recommend movies with machine learning
Recently I spent some time with our beloved AI overlord ChatGPT (just kidding!) probing the model and pushing its limits. I tested it on a usecase of movie recommendation. You can find the video walkthrough here.
Monolithic LLMs powered by billions of parameters, fine-tuned with RLHF has forever changed how we perceive AGI. Rise of ChatGPT, GPT-3.5 and GPT-4 have exemplified how much the horizons of the abilities and skills of language models expanded in the last few months. ChatGPT reaching 100 million users in just two months from its launch, is a testimony to how impressive the jump in AI has been.
Movie recommendation with ChatGPT
So many people are using ChatGPT in creative ways, from creating Flappy bird from scratch to building websites. Following the trend, I decided to see if ChatGPT can compute user ratings for an unseen movie, given a dataset. First I asked ChatGPT to generate a dataset.
It was swift to respond and generated a dataset as explained in the context.
I'll be asking ChatGPT to,
Predict the user rating of Jack to the movie The Avengers
My hope is that ChatGPT uses a collaborative filtering approach to do this. One can first create a ratings matrix, use the ratings matrix to compute user similarities to Jack. And finally,
Note that I'm ignoring the users with rating 0 for The Avengers from the score computation. The following excel sheet depicts these computations. The final answer we're looking for is 9.
Next, I posed the question as follows.
Looks like ChatGPT thinks this is supposed to be a data point, that's currently missing in the dataset. I also tried using the "Let's think step by step" trick. But that didn't get ChatGPT very far.
Next, I tried using chain-of-thought reasoning to pronounce the approach that needs to be followed in order to compute the final result.
Success! This time, ChatGPT was able to follow the plan, generate the intermediate results and compute the final answer.
But hold on a second! The final result is wrong.
Problem 1: ChatGPT flunked mathematics (potentially) due to the complexity of the task
Looks like ChatGPT got the final result wrong. If you copy and paste the equation in line 2 of the last step to a calculator, you get 9, not 8.95. Moreover, unsurprisingly cosine distances are wrong too. But it's still impressive what ChatGPT was able to do, being a language model. Let's give the benefit of the doubt and try to show where ChatGPT stuffed up.
Unfortunately, ChatGPT couldn't see it through. Here's a snippet of the new response.
I couldn't get ChatGPT to correct the mistake. But it kept admitting it made a mistake, which is a bit paradoxical. This brings us to the 2nd problem.
Problem #2: ChatGPT is sycophantic
ChatGPT is quite sycophantic and will think it's wrong every time you point that it's wrong. Funnily, it even thinks it's wrong when it has the right solution at hand
