To get the best out of a brainstorming session with ChatGPT, you may need to do a little more than just ask a basic question.
People across industries are using ChatGPT, the artificial-intelligence-powered bot, to help them with tasks like summarizing documents, writing emails and generating ideas. More than 400 million users query the AI weekly. But when it comes to brainstorming, the bot may have a weakness, especially for users who pose basic prompts, according to a recent study from the Wharton School at the University of Pennsylvania. The results can be surprisingly alike with just slight variations, which can limit the pool of diverse responses.
“ChatGPT is really good at coming up with ideas,” said Christian Terwiesch, co-director of Wharton’s Mack Institute for Innovation Management and co-author of the study. “But they are too similar to each other.”
In one experiment, researchers asked ChatGPT for suggestions on repurposing a garden hose and a tennis racket. Nearly 21 percent of ChatGPT’s results were types of sprinklers. Web-aided and human-only brainstorms produced far more diverse results, such as a picture mount and a storage rack, the study found.
But with a little creativity in crafting your prompts, you could end up with not only more-diverse ideas but a lot of them, AI researchers and experts say. We spoke to researchers and software engineers for tips on how to better brainstorm with chatbots.
Be clear and specific
To avoid getting similar results, simply tell the bot you need options that are vastly different, the Wharton researchers said.
That might be as simple as adding that request into your prompt or being specific about how you’d like the results to vary, Terwiesch said. If you’re looking for U.S. cities that are great for summer vacations, you might say that you’d like the suggestions to cover different parts of the country, he added.
Tell the bot to give you more than one answer. You could ask for 10 or even 20 suggestions to get a much more diverse set of responses, said Simon Willison, a software engineer who has studied AI engineering.
Large language models “give the most average answer,” he said. So “always ask for as many options as possible. The first six or seven are obvious, then it gets interesting.”
‘Chain-of-thought’ prompts
If you don’t get a good set of ideas, try again. One of the benefits of chatting with AI is that it can iterate on previous responses, experts said.
So if you get a bunch of lackluster ideas to start with, you can always add a second prompt like, “Make these bolder,” said Lennart Meincke, a Wharton researcher and co-author of the study. You can go step by step or you can put it all in one prompt to say something like, “Generate a few ideas, try to combine them, make them bolder and then refine.”
Allen Karns, chief AI technology officer at Vanderbilt University’s generative AI center, calls this a chain-of-thought prompt, where you ask to do a series of steps to come up with a set of results.
If you don’t have thoughts on exactly what kinds of ideas you want or to differentiate them, you can simply turn back to the AI for help, Karns said. You can start with: “Ask me questions that will help me generate ideas for a children’s product.”
“Turn it back on the [bot],” Karns said. It might say something like, “‘What outcome are you looking for?’ and it can help drive the conversation.”
Don’t be scared to tell the AI that you don’t like its ideas. You can say, “That’s boring” or “That’s dull,” or, as “someone once said, ‘Bro,’ full stop,” Willison said.
“You have to play with them. Don’t take them seriously at all.”
Role-playing
Give it a fictitious role to play. Using your imagination can help the AI walk you through generating ideas, Karns said.
You can say: “You’re a travel agent, and I’m the customer. Let’s have a dialogue where I’m about to take a vacation.” You can create a custom chatbot with ChatGPT that always assumes a specific role for brainstorms you regularly have, Karns said.
You could even ask it to assume the identity of your target customer, Karns said. For example: “You’re a parent of a 12-year-old. What are things you would consider about a toy that both you and your child can enjoy?”
You can make the scenario as simple or as complex as you want. Maybe the parent is looking for toys that work in the water or specifically at beaches. Once you get through those prompts, you’re likely to get results that are more diverse, creative and closer to solving your problem, Karns said.
Testing your ideas
Ask the AI to evaluate your ideas or whittle them down to the best based on specific parameters you set, experts suggest. Alternatively, feed it a bunch of ideas collected from you and your colleagues and build on, tweak, combine or evaluate those.
You could say: “Here are 500 ideas from my team and feedback customers have given us about earbuds. Pick the top 30 I should take to an executive,” Terwiesch said. You can also set up scoring systems so that it evaluates based on specific criteria, he added.
Ask it to find your blind spots and challenge you. You could ask it why you should not do something or what things you aren’t considering, Terwiesch said.
Another way to do this is to combine stress-testing with role-playing, Willison said. “I had someone tell me when they’re coding, they tell the machine to imagine it’s an episode of ‘Survivor’ and that certain parts should be voted off the island,” he said. “That gets it to argue more.”
Tell it to act as a critic and challenge your idea with different perspectives or to help you improve it. You could tell it to analyze something from the perspective of the child playing with the toy as well as the business executive who has to pay for making it, Karns said. You might find “it’s a great idea, but it’s going to cost $200 to make,” he said.
It’s important to start a new chat because the conversation history could alter the results, Willison said. The AI might reinforce your ideas simply because it’s trying to please you. Crediting someone else by saying “evaluate this idea from my friend” helps break that pattern, Willison said.
Verify everything
Ultimately, you should rely on your own brain to determine the most creative ideas and decision-making, experts say.
AI should help accelerate you through the boring parts, come up with interesting combinations and explore new boundaries, Willison said. That could help you get to the ultimate winning idea, he added.
You should make the final decision — despite AI’s evaluations — as you will be held accountable for whatever you pitch, Terwiesch said. And with anything generated by AI, verify everything. The bots still get things wrong even if they sound convincing, Karns said.
“It will always seem smarter than it is, but it’s just numeric calculations under the hood,” Willison said.
