ChatGPT just rolled out plugins to all paying users, so I tried out the Wolfram Plugin. One can now ask it arbitrary questions and it can figure out how to formulate the question for WolframAlpha / Mathematica and then return the answer. It can easily answer many straightforward questions. For example, it can calculate the odds of getting heads on coin flips or the weight of a lead sphere.
For more complicated questions, it can sometimes struggle. In some cases it will enter syntax into Wolfram that Wolfram doesn’t understand and then it will try again until it gets an actual answer. Often the final answer is correct but sometimes ChatGPT will make a mistake that isn’t always obvious. It’s like a human in a way.
For example, I previously asked ChatGPT about the odds of getting a flush in Texas Hold’em when dealt two cards of the same suit, and it gave an answer that was much too high, although it did a little better more recently.
What are the odds of getting a flush in Texas Holdem, if dealt two cards of the same suit, and no other cards have been dealt yet?
Re-formulating the question slightly and using the Wolfram plugin, it was able to derive the correct answer. In fact, it could even answer the question in this format without the plugin! I tried the previous format of the question with the plugin, and it gave an answer that was much too low. It seems that subtle changes in the text can trip it up.
I'm playing Texas Hold'em and have been dealt two cards of the same suit and none of the face-up cards have been dealt yet. What are the odds of getting a flush by the time all the face-up cards have been dealt?
For now it seems like humans will still need to review its work, but I can imagine it improving quickly over time and being able to solve more problems correctly. I used to say that math education should focus more on teaching people how to formulate problems for tools like WolframAlpha and focus less on how to solve things with a pen and paper. But if computers can formulate the problem as well, what will people need to do?
