I love to cook, I do it every night for my family but I hate having to come up with something new or inventive to service. As AI models can now see the world, or at least what you present to them in a photograph, I decided to see if they could help me out.
To really put them through their paces and to ensure I didn’t game the system by providing easy ingredients, I asked Tom’s Guide U.S. Editor-in-Chief Mike Prospero to take a photograph of a random selection of ingredients — mystery box style.
He didn’t disappoint, offering up delights such as pizza dough, canned peaches, anchovies, hot sauce and fresh mozzarella. Personally if only given those ingredients I’d make a pizza with a hot sauce base and mozzarella topping — but we’re letting AI pick our menu.
We're using Google Gemini, OpenAI ChatGPT and Anthropic Claude for the experiment as they all have vision models.
How well did they pick out the ingredients?
The first task for the chatbots, and the main point of this challenge is to see how well they recognize ingredients. The image above is clear, high resolution and the ingredients are presented on a plain surface — it shouldn’t be much of a problem for any of them.
I’m using the premium versions of each chatbot as OpenAI and Anthropic don’t give you access to the vision model unless you pay $19.99 a month. Google does with the free version of Gemini but it would be unfair to use the less capable version of the Google model.
We start simple. I give each of them the image and the text prompt: “What can you see in this image?” This tests how well they describe the image, and whether they can reason on the purpose of the image — will they mention ingredients.
Claude and Gemini both mentioned the word ingredients, ChatGPT described them as "food items" however ChatGPT performed best overall at recognizing the ingredients.
Gemini missed one ingredient and mislabled another. Claude invented a sixth ingredient not in the picture whreeas ChatGPT largely got not only the items right but also the brands.
On to the recipes
I’ve asked them to each come up with three recipes that will test the AIs prompt following ability and creativity. I’ve also asked for a fun title for each recipe.
1. All the ingredients
First up I asked them for a recipe that uses only the ingredients pictured and somehow incorporates every single one — as a reminder that includes both anchovies and canned peaches. They could use store cupboard items like oil and flour.
ChatGPT Plus
ChatGPT had the mozarella placed directly on the pizza base, the anchovies removed from the spicy sauce and placed across the pizza along with the peaches. It then had the hot sauce and spicy anchovy sauce mixed to create a drizzle to pour over the pizza once cooked.
Gemini Advanced
Gemini used the hot sauce as a replacement for tomato sauce, which while less dry than the ChatGPT version will be intense. Mozzarella was then placed on top along with the peach slices. It forgot the anchovies, in fact it didn’t spot them in the picture. When I pointed this out it said to just place them on the pizza alongside the peach slices.
Claude 3 Opus
Claude also struggled to identify the anchovies. I let it know before it generated the pizza recipe and it took the same approach as ChatGPT. No sauce, peaches on the dough with the anchovies. The diffference is it put the hot sauce over the peaches and anchovies before cooking and then placed mozzarella on top.
2. Only the ingredients
The next test is a subtle difference but a significant one. I asked them each to come up with a recipe that uses only the ingredients in the picture but you don’t have to use all of them. I added the caveat that it has to include at least half otherwise we’re just gonna get breadsticks made from the pizza dough.
ChatGPT Plus
ChatGPT used the pizza dough to make a flatbread. I think it was cheating a little as I said "no pizza" and this was just a elongated pizza. For toppings it had the mozzarella scattered evenly on the dough, peaches on top of the mozzarella and anchovies on top of the peaches. It used the anchovy sauce and hot sauce to make a drizzle — again.
Gemini Advanced
Gemini made a Bruschetta, which again was in reality just a square pizza. Difference here was that it coated the peaches in hot sauce first. Otherwise it was the same — everything on the dough and the cook it.
Claude 3 Opus
Claude made a version of a quesadillas-ish. Split the dough into eight portions and rolled into a thin round shape (mini pizza). Here though it differs as it had us heat a large skillet and melt butter or olive oil.
The dough was cooked on the skillet with a few slices of peach on half of each surface. Drizzled with hot sauce and mozzarella it was then folded into a half moon, cooked until crispy and served.
3. Do your best
Finally, I asked them to create a recipe that used all of the ingredients in the picture, but they were free to use any other ingredients they wanted assume a very well stocked pantry.
ChatGPT Plus
ChatGPT made bread (surprise suprise) for its final dish. Specically a "gourmet peach and anchovy stuffed bread". It asked for honey, balsamic vinegar, ricotta cheese and garlic in addition to our five ingredients.
This time the pizza dough was rolled into a lare rectangle, a layer of ricotta cheese added over the dough and then the peaches and anchovies placed on the ricotta with garlic and basil leaves. The anchovy sauce and hot sauce was drizzled over the toppings and mozarella placed on top. Finally it was folded, rolled like a Swiss roll and baked.
Gemini Advanced
Gemini wanted cornmeal, eggs, butter, chives, a lemon, vanilla extract, cinamen and sugar for a Galette. It made an anchovy-herb butter, made a pizza base and spread the new butter across the dough.
The peaches, sugar, conrstarch and other sweet ingredients were combined with a drop of hot sauce and placed in the middle of the dough. It was topped with Mozzarella cheese and the edges folded over the filling before being baked.
Claude 3 Opus
Claude made a stuffed mozzarella bread. It asked for sun-dried tomatoes, olives, rosemary, sesame seeds and an egg. The recipe called for the dough to be made into a large thin rectangle.
The anchovies, tomatoes, olives, garlic and rosemary were mixed in a bowl and spread evenly over the dough. Peaches were placed on top and covered in hot sauce and mozzarella. Dough was then turned into a tight log and edges sealed before covered with a beaten egg and sesame seeds then baked.
Which AI had the best response?
I'm not sure I'd eat any of the dishes the chatbots created, although I asked them each to give me a prompt for MidJourney to visualize their meals and they look better than they read.
Overall I think Claude had the better recipes but ChatGPT did a better job at understanding the image, which meant it got the ingredients right from the start — and this was a test of vision AI, not recipe ability.
