Can Chat GPT Play chess?
2021 I developed an Artificial Intelligence model that works with multiple strategies and Deep Reinforcement Learning to be a realistic and expert chess player, and I tested the AI bot against many different bots. You can get more details about this journey in the article below:
Hacking Chess with Decision Making Deep Reinforcement Learning
2023I am using the same AI to test is ChatGPT can play chess and how good it can be in a real match.
The first thing before starting is to ask ChatGPT if it can play chess. ChatGPT says that it can play but it is not a perfect player (is this an excuse if it loses the game ? ).
Ok, my AI is playing with whites, let's start the match with a classical e4!
In the first moves, we can see that ChatGPT understands openings and answers with logical analysis about each move. The match is well balanced till now.
After 8 moves, my AI model made a mistake after castling and developing a pawn to h3 it preferred to move the Bishop to f4 instead of taking d4 with the Knight, which would be the best move for this position.
This mistake opened a huge advantage to ChatGPT but …
ChatGPT did a nonsense analysis saying that my AI model pinned its knight to the queen and decided to move a pawn to a6 instead of taking the Knight in c3, which would be the best move for this position.
It looks like ChatGPT is not able to keep track of the board and all its pieces and it is confusing this match with some other matches from the trained dataset.
ChatGPT didn't get the opportunity and my AI decided to move Qe2 then another nonsense analysis by ChatGPT, said that I moved the queen to connect my rooks and support d4, however, I don't even have a pawn in d4. This weird analysis confirms the fact that it is not tracking any board position.
Even with some errors ChatGPT still had the advantage, and I thought for a moment that my AI would lose this game.
Then … ChatGPT decided to play Ba6, a mistake that cost the entire match and reverted the advantage to my AI.
2 moves ahead, after some nonsense analysis, ChatGPT lost complete track of the board positions and entered an infinite loop of illegal moves. Here is a demonstration of the first illegal move:
After entering this infinite loop I wasn't able to play anymore because ChatGPT was not able to find a single legal move after move 15, and the game ended when I considered a resignation from ChatGPT.
Victory of ChessAI! by knowing what it is doing more than ChatGPT!
Conclusion
In conclusion, ChatGPT 3.5 cannot play a chess match, the Large Language Model is not tracking the board and checking for legal/illegal moves. ChatGPT uses the trained dataset to infer the next move from previous matches but it is confusing these matches and does not analyze the actual board position and how to play this position. We can assume that ChatGPT doesn't have the knowledge about the game to produce logical decisions, even after saying in the beginning that it was trained with chess rules, information, and strategies.
Chess is a Sequential Decision problem with a high state of possibilities and a strong set of rules that GPT 3.5 was not able to learn yet, and RL models or search models still being better for this type of problem. For now my ChessAI model, a chess expert model, still doing better!
In the next article, I will play a match with the same AI but now against GPT 4.0. GPT 4.0 is better than 3.5 on chess? It will be able to play a whole match? Let's see in the next article!
If you want to evaluate the whole game:
FEN: r1bqkbnr/pp2pppp/2np4/2p5/2B1P3/5N2/PPPP1PPP/RNBQK2R w KQkq – 0 4
PGN: 1. e4 c5 2. Bc4 Nc6 3. Nf3 d6 4. Nc3 Nf6 5. O-O e6 6. h3 Be7 7. d4 cxd4 8. Bf4 a6 9. Qe2 b5 10. Nxb5 axb5 11. Bxb5 Ba6 12. Bxa6 Rxa6 13. Qxa6 Qb6 14. Qd3 Qb7 15. b4 Ne5
Any questions or suggestions can contact me via LinkedIn: https://www.linkedin.com/in/octavio-b-santiago/
