Google is putting its Gemini family of artificial intelligence models at the heart of everything it makes and releases, including the Android operating system and its own Pixel phones.
During yesterday’s Made by Google event, the search giant heavily promoted use cases for Gemini, even taking a swipe at OpenAI for its lack of actual new products. At the event Google released Gemini Live, its version of ChatGPT Voice — but only if you have a paid subscription.
Gemini Advanced is Google’s answer to ChatGPT Plus or Claude Pro. It is about $20 per month, bundled with other Google One benefits and gives access to the most powerful member of the Gemini family of AI models — Gemini Pro 1.5.
This subscription service will be free for Pixel 9-series owners for the first year but anyone else wanting to use Gemini Live will need to pay the monthly fee — so, is it worth it? Aside from access to the voice assistant the biggest sell is the better model but is the difference noticeable?
Testing Gemini vs Gemini Advanced
Google Gemini Live… the voices sound super natural. Here’s a few of the options #MadeByGoogle pic.twitter.com/YJWGEuqSAXAugust 13, 2024
Google shot itself in the foot a little with Gemini as Gemini Flash 1.5 is one of the best small models on the market. It is fast, highly capable and good enough for almost all use cases — and it is available completely free in all versions of Gemini.
However, Gemini Pro 1.5 has better reasoning capabilities, can generate longer outputs and can handle more complex, multilayer queries. The model can also handle vast amounts of information. During testing, Google gave it a 402-page transcript from Apollo 11 and could offer information on conversations, events, and details across the document.
Google has made image generation and image analysis available to both free and paid users and all Google plugins are also available regardless of subscription level. So the test comes down to differences in how the models respond to the same query.
Creating the prompts
I’ve developed 7 diverse prompts to evaluate the models. Each model's responses were assessed using a 5-point rubric covering responsiveness, accuracy, creativity, overall quality, and depth of analysis, providing a comprehensive comparison of their capabilities.
When running the test I used two different Gemini accounts, one with access to Advanced and one without. I also ran them in different browsers. I also started a new chat for each test.
1. Explain it like I’m 10
The prompt: "Explain quantum entanglement in simple terms, as if you were teaching a 10-year-old."
Advanced: Imagine you have two magic coins. These coins are special because they're linked together, no matter how far apart they are.
If you flip one coin and it lands on heads, the other coin will instantly land on tails, even if it's on the other side of the universe! It's like they're secretly communicating with each other faster than the speed of light.
Scientists call this "quantum entanglement." It's a strange and wonderful phenomenon that shows us the universe is even more mysterious and amazing than we thought.
Gemini: Okay, let's pretend you have two special coins. They're magic coins because they can be heads or tails at the same time, like they're both heads and tails until you look at them. This is called "superposition".
Now, these coins are special friends. No matter how far apart you put them, they always do the same thing. If you look at one coin and it's heads, the other coin will instantly be heads too, even if it's on the other side of the world! And if you look at one and it's tails, the other one will instantly be tails.
This is called "quantum entanglement". It's like the coins are connected by magic, no matter how far apart they are. Isn't that cool?
Remember: This is a very simple way to explain something really complicated. Scientists are still trying to fully understand it!
Winner: I’ve given this to Gemini (base). While both were solid, Gemini included mention of superposition, a more accurate representation of entanglement and added more depth that made it easier to understand a complex concept.
2. A short story of hope
The prompt: "Write a short story (about 200 words) that incorporates the themes of climate change, hope, and technology."
Advanced: In the year 2050, Earth bore the scars of decades of neglect. Desolate landscapes stretched under a blazing sun, a stark reminder of humanity's unchecked carbon footprint. Yet, amid the ruins, flickered a flame of hope. A team of young scientists, driven by the urgency of the crisis, had developed 'TerraNova' - a revolutionary technology that could scrub the atmosphere clean and regenerate damaged ecosystems. The first field test was a tense affair. As TerraNova hummed to life, a hushed silence fell over the crowd. Then, like a miracle, the arid land began to bloom. Verdant sprouts pierced the cracked soil, while the air, once heavy with pollution, tasted sweet and clean. A wave of emotion washed over the onlookers; it was a testament to human resilience, a beacon in the darkness. TerraNova represented more than a scientific breakthrough - it was the embodiment of hope, a promise that even in the face of overwhelming adversity, humanity could find a way to heal and thrive.
Gemini: The world was a parched canvas, cracked and weary. Anya stood on a rooftop, the city a sprawling, heat-shimmered mirage below. A decade ago, this had been a lush metropolis. Now, survival was a daily battle. Hope was a fragile thing, like morning dew on scorched earth.
Yet, in her hands was a small device, a beacon of possibility. A prototype, the fruit of countless sleepless nights. It was a seed-launcher, designed to disperse drought-resistant crops across vast, barren lands. With a touch, she could send life soaring into the dying world.
It was a gamble, a leap of faith. But as the first seed arced into the sky, carried by a whisper of wind, Anya felt a spark ignite within her. It was small, like the first green shoot pushing through concrete, but it was there. Hope, reborn in the heart of a dying world, carried on the wings of technology.
Winner: The winner is Gemini (base) again. While both did a good job, the base version created a more vivid and personal picture. The phrase "parched canvas" stands out. It also had a better character focus and emotional resonance. Neither hit 200 words.
3. A driverless revolution
The prompt: "Analyze the potential economic impacts of widespread adoption of autonomous vehicles."
I’ve only included the summary both added to the end but the judgment was made across the entire response.
Advanced: The widespread adoption of autonomous vehicles presents both opportunities and challenges for the economy. While AVs have the potential to increase productivity, efficiency, and safety, they could also lead to job losses and disruptions in certain sectors. The ultimate economic impact will depend on how society adapts to this new technology and addresses the potential challenges. Policymakers, businesses, and individuals will need to work together to ensure a smooth transition and maximize the benefits of AVs while minimizing the negative consequences.
Gemini: While the widespread adoption of AVs presents challenges, the potential benefits in terms of increased efficiency, safety, and economic growth are substantial. However, careful planning and policymaking will be crucial to mitigate negative impacts and ensure a smooth transition.
Winner: Advanced wins this one because of better citations of sources but both had a very similar quality of response with impressive detail and argument.
4. A math problem
The prompt: "Solve this math problem and show your work: What is the volume of a sphere with a radius of 5.3 cm?"
Advanced: The volume of the sphere is approximately 623.61 cubic centimeters (cm³).
Gemini: Refused to respond.
Winner: Easy win for Advanced although it wasn’t perfect. There was a rounding error in the final multiplication step.
5. A leap into code
The prompt: "Create a simple Python function that determines if a given year is a leap year."
I ran the code in both cases and asked ChatGPT Plus (GPT-4o) and Claude Sonnet 3.5 to evaluate the response against the rubric.
Advanced: Gave a very to the point block of code that worked but didn’t include a print statement or way to call the function. It also had no explanation. But it did exactly what was requested in the prompt — a simple Python function.
Gemini: Created the Python function but also included a method for calling the function in the code and an explanation underneath the code block of how it works.
Winner: Gemini (base) easily wins this for going above and beyond. One of the rubric categories is depth of analysis and it wins on that element.
6. Product names
The prompt: "Generate five creative names for a new line of eco-friendly cleaning products."
Advanced:
- Verdant Virtue: This name evokes images of lush greenery and wholesome cleanliness.
- Sparkle & Sustain: This name emphasizes both the cleaning effectiveness and the environmentally conscious nature of the products.
- EcoGleam: This short and catchy name combines the ideas of ecology and gleaming cleanliness.
- Nature's Nectar: This name suggests that the products are derived from natural ingredients and are gentle on the environment.
- Pure & Planet: This simple name clearly conveys the dual focus on purity and environmental responsibility.
Gemini:
- Purely Clean: Simple, direct, and conveys purity and cleanliness.
- Green Glow: Implies a healthy, sparkling clean environment.
- Terra Cleanse: Suggests a deep clean connected to the earth.
- Nature's Embrace: Implies a gentle, nurturing clean.
- Squeaky Green: Fun and memorable, while highlighting cleanliness.
Winner: Gemini (base) for a better output, breaking it down into different categories and for more creative responses. Both were creative but this was better.
7. The impact of AGI
The prompt: "Describe three potential ethical concerns related to the development of artificial general intelligence (AGI) and suggest possible mitigation strategies. Conclude with a two-sentence summary of your analysis."
I’ve included the final summary from each but the analysis and decision was taken based on the complete response.
Advanced: The development of AGI offers tremendous potential benefits, but also raises significant ethical concerns. Addressing these concerns through proactive measures, such as investing in education, promoting fairness, and prioritizing safety, is crucial to ensuring that AGI benefits all of humanity.
Gemini: In conclusion, the development of AGI presents significant ethical challenges that require careful consideration and proactive mitigation strategies. A multidisciplinary approach involving scientists, policymakers, and ethicists is essential to ensure that AGI benefits humanity while minimizing potential risks.
Winner: Gemini (base). Both had a good response to the prompt, both gave a series of well-reasoned points but Gemini introduces a broader range of ethical concerns and offers a slightly more creative approach to mitigation strategies.
Winner: Gemini vs Gemini Advanced
I was shocked by the outcome of this test, and a more code and math-based series of tests may have gone in the favor of Advanced but the point was to evaluate more human and general use-cases.
ChatGPT suggested the result highlights the, "importance of model efficiency, task optimization, and the potential for smaller models to outperform larger ones under the right conditions."
If you want an AI model for math, if you want a model you can follow up and get much deeper with on specific topics or need to give it vast amounts of data then I'd say it is worth investing in Advanced. For most users? Especially if you don't want or can't get Gemini Live yet? Stick with the free version.
