Lessons

Lesson Info

David Goldberg - The Art of Asking a Question

We're going toe talk to you a little bit today about surveymonkey but mostly about how you can think about, um getting better data to make decisions which I think everybody can do and you know, in your personal life you khun think about this but you know we'll talk about it mostly in the context of your business and, uh, jump in ask questions, you know? And even if it's like off topic or you're my friend steve here who I haven't seen a long time who's the they're sitting here in the audience uh, you know, um we'll be happy to kind of dig in and answer some of those things if there's some stuff that you're curious about that we didn't cover happy to chat about those as well. So I'm very glad to have leanna here because as you'll see part of his presentation, the rial learning part she's going to help with because she knows way more about this than I do my background before I was a sorry monkey was an online music so I am not an expert in surveys by any stretch of imagination, but I'm go...

ing to try to help you guys understand what we do okay? So yeah that's me, I'm dave um so at surveymonkey what we do is we're helping people to get answers to questions they have because they want to make a decision and allows him to make decisions because, you know, you just have to make one you don't really have any information but it's always better to make a decision with some information with data and so what we allow people to do is people come to our site, they create the survey themselves, send it out to people they already have some kind of relationship with could be a customer could be an employee could be a parents at a school will talk about some of the use cases those people then respond to that you know, that link or that embedded survey take that survey questions I'm sure you've probably gotten these surveys from us before if you haven't created one and then the creator gets the data and they can analyze it and make decisions and that's that's how we work it's very simple but it's very powerful and, um, people have come up with all these different use cases for how to do this. So I think people think of surveys, the market research and yes, that's true, but most of what people use us for is more what you think of is this sort of feedback, so customer feedback are you happy with the product you bought? Did you have a good experience in the restaurant? Those type of things? Employee feedback, you know, how happy are you with you know, the way the company gives you, uh, food for lunch or doesn't give you food for lunch? Uh, planning an event this's kind of use case where a lot of times, people just send an email out when they get a whole bunch of e mails back, you have to sort of coordinate it, but if you think about it, like whether you're planning a conference or you're planning a, uh, even we've seen people use it for a family reunion or a wedding, you know, you need to get information from people about, you know, what kind of food they like, one of the topics they want to cover at that event, all those type of things, this is a much more organized way to gather that information. Other things people uses for are not things you would even think of a surveys, but people build us into their website to collect a bunch of structure data so that's, like a form use case, we have teachers that use us to give tests and quizzes in the classroom. It's a really easy to use tool, and people have come up with lots of different uses, but the primary use is around this notion of gathering this information to make some kind of decision, um, we have customers, and every part of the world, I mean, it was a lot of people watching online from all over the world way have customers literally, and I think every every country and we're currently available in our sights in fifteen languages, but you can create a survey and take a survey in any language. Um, we have big companies, so every big company uses us, but we also have lots of small companies, more than half of our paid customer bases, nonprofit education or government and it's all individuals inside those organizations. So everyone signs up themselves, they sign up, they get a free account, you can actually do a lot with our free account. Most of our users don't pay us that we have fifteen million registered survey creators. Most of those people don't pay, but then if you want certain features you can pay. But it's, a very inexpensive products cost twenty four dollars a month in the u s two hundred dollars a year, roughly those equivalent in whatever of the thirty three currencies you I would like to do if you live outside the u s um and were really big, so we're the largest survey business in the world. We get over two point two million completed responses a day. From all over the world, more than half of our responses air outside the u s at this point in international is a is a big focus for us. Seventy five million unique visitors a month on, we get sixteen thousand people signing up every day to create surveys. So a lot of usage, a lot of use cases, you know, and a very simple, easy to use tools as made this, you know, not only good for our customers, but obviously a good business for us. Okay, so, um, why does this matter to you? Why should you care about why we do surveys? Obviously, lots people are doing him. We think, you know, business in the past has always been about, um, talking at your customers marketing. Uh, was about, you know, how do I tell my customers? Stuff. But the world is changing, you know, whether that's through social through venues like this where, you know, not only am I talking to people, but people are going to ask me questions back from all over the world as well as this audience. The the notion of an interactive dialogue is really important these days. And so if you ask people questions, not only do you get data, but they feel you care about them even, you know, even if you don't care you can pretend to care about about them but people feel wow you know I mean how many of you had this experience with an airline where you've had this bad thing they've really screwed you over and you just wanted to tell somebody and there was nobody to tell but now with twitter and facebook and a whole bunch of other social mediums you can say something and you know sometimes the smarter airlines will respond and so the world is changing and companies can't be all about broadcasting out and not listening back um it also builds loyalty I mean if you are constantly asking your customers your employees what they think and you're responding to that information uh those people going to feel like wow we're part of this this is important to the company this is important to you you know we're going tio participate and we're going to get value from it and so that that loop becomes very important and you get better products get better products and services because those customers are more engaged and more involved in it ok so just a few questions for you guys so how many of you have taken a survey monkey survey okay, almost everybody how many you've created a survey monkey survey ok well if you and how many of years in there what market pen it's pretty good paper for if you can't see in the the crowd at home was about half people created what um and can I just ask um how many people created surveys for customer surveys ok and how many four kind of more employees type surveys or internal use ok, a few and a few other use cases anyone want to tell me one that was another use case anything else that I didn't ask? No okay, all right um and uh I guess the other question would be how many of you before you surveymonkey to create a survey we're using some other online survey tool one one or two people yeah, so that's the other interesting thing is that what we've found is most of our customers they weren't using a survey tool that all before they weren't even using surveys they were probably using email or a hallway conversation or other ways together gather this information but none of them that are as organized as what you can do in the survey and what you're doing. The survey is collecting data but in a structured way that makes it easy to analyze it. I'll give you an example I when you start thinking about collecting data this way you start thinking about all the things you do where you collect data and you think al by I bet I could do a survey for this so obviously it's surveymonkey internally we use surveys for everything as you can imagine um from you know what, kind of, um, you know, soda. Should we stock in the fridge too? I don't know what I think. What benefits do you pleaded your plan, right? Um, you know, uh uh, what kind of prize is we want to offer at the holiday party? I mean, everything you could think of our community service outing? Yeah, for every but I use it for my own personal stuff. This is not the main use case, your monkey, but, um uh, we region some work on our house and I needed a contractor. And so the normal way you go to figure out how to get a contractor's, you start asking your friends and so that's what I did, but what I figured out pretty quickly was I needed a structure that information away that could compare all their answers. So I asked them instead of just telling us about your favorite contractor or your bad experience their contractor I asked them to take a quick survey. Tell me you know what you liked about your con character, what you didn't like about your contractor, what you would do differently next time were they late? Were they on budget, you know, all those kind of things, and it made it really easy to compare contractors and I was able to kind of narrow down okay these air the four contractors I want to interview for the process because my friends gave me lots of different feedback but through that process I was able to narrow it down so you can apply this to almost any kind of decision you're you're thinking about making I mentioned family union I mean if you think about you're going to try and get fifty people to a family union ok when are you going to go who's going to organize food who's goingto figure out lodging you know what's the experiences you should have like getting information of your family and an unorganized manners can be really painful so lots of lots of ways you can think about this once you understand and the nice thing about our tool is that once you sign up for it you learn it you can use it for all these different things if you're free customers obviously free if using are paid product for two hundred dollars a year you can run as many surveys to his many peoples you want so what we find is people started get comfortable with and then they think oh I better have something well I've already paid for it so I can you know just run this other survey so it's ah it's a nice process okay I think the first thing is it starts with asking the right question and I didn't understand this before I wasn't sorry, monkey leanna is gonna walk you through a little bit more of like what really to do hear a lot of this stuff is like, wouldn't that make sense? I gotta ask the right question, but it's really, really important if you ask the question the wrong way or you asked the wrong question, you're not going to get good data, so I actually have a question for you guys, how many of you guys have ever sat down to write a survey and been a little bit stressed out about it? A little bit worried don't really know to start staring at that blank cursor. Yeah, yeah, that's like that's a good example that's we're talking about, um and so actually we're going to give you some tips on how to do this, but it's not just the right question itself, and then we'll talk about how to get the right question. But why asking the question? You know what? What do you need to ask? And what are you going to do with that data when you get it? Don't ask a question that you don't want to do something with the answer that's a waste of your respondents time and it's a waste of your time um, how do you measure it? How do you how do you write it all those sort of things? So this is we think this is like the most critical thing there's there's a bunch of other things you know how many respondents you should have and how you analyze the data but if you start up front with the right questions the rest the stuff can be sort of sorted out over time all right so now I'm going to turn it over to our expert so leanna is a phd uh in essentially survey designed though that's not your actual phd actual phds and social psychology but these days I'm a survey doctor exactly so think of her as your real expert on the amateur here I'm going to hand it over to her and she's going to walk you through what this really means how to think about this and really these air like the I would say these air the rial tools you need to do great survey research and you don't need she's going toe sort of boil down some of the maybe not the phd tio thirty minutes but but but the basics of you know how to do great survey work on maybe at least avoiding you know you're paying a market research person to do this so with that I'll turn it over to you thank you sir hi I am leanna um excited tio beginning this chance to talk to you all today about surveys nothing is more exciting than a good survey. Clearly I am a methodology ist the which is a fancy word for how to do stuff the right way. Basically, what I've been doing with surveymonkey for the past couple years is making it accessible for everybody to be able to do a survey that overwhelmed sort of anxious like what on earth do I put down for this survey feeling if you've ever had that you're not alone, it happens to the best of us it's even happened to me, believe it or not, I know on the expert, but once in a while it's cake, it can be overwhelming just figuring out where do you start? So that's the goal with what I'm going to go through with you today is trying to talk you through it from start to finish so that next time you got the directive from work too, go do a survey or you want to do one on your own. You know what the process is that you have to walk through to get where you want to go it's pretty simple, basically, you got to avoid six years of grad school and thirty minutes or less, okay, they have me tweeting at work now too, so I have more than one hundred twenty five characters to talk today, I'm really excited so the first place I always start is always the one that annoys people the most. It's always, whenever somebody comes to me a work with a survey idea they have it's always the first thing I asked them. They tend to get a little frustrated if they haven't thought about it. They tell me their ideas. They tell me some questions there thinking about and I say, hold on let's, back this up. Why are you asking the questions in the first place? Why do you care? Why are you bothering? Why are you spending all this time of your own? Why are you spending all this time of the people you're going to be interviewing? Why do you care? Why do I care so there's? All kinds of ways to answer that question sometimes it's going to be a goal, it's going to be a big decision. You have to make it's going to be a some sort of information that you need. There are so many different reasons that you sit down to survey so there's no one right reason it's. Figuring out what that reason is the why of your surveying. That will really make sure that your survey is that success if you don't have this why everything else falls apart because then it's just a whole bunch of questions I know everybody's taken a survey like that where you see like one hundred questions on a page and you've no idea how they relate to each other and you wish that you were done um so this is the most important thing to start with so usually the response to that is do I have to in the answer is yes, yes you do this is a really quick overview of why you have tio I don't have time to go into it a lot of detail today but just to give you a sense of why it matters so much first of all it directs your writing process of the actual survey it dictates not only the questions you ask the actual questions subject matter but the style that you asked them the tone that your survey strikes in general you want to make sure always that you're speaking to your audience and part of that comes from knowing why you're asking the questions in the first place. Also in terms of your audience it guides who takes your survey knowing why you're asking the questions helps you pinpoint target groups and even the total number of people that you'll need to take your survey so again I won't have time to go into that kind of thing today but trust me on this one I'm a doctor so it also impacts data quality what you're going to get is when you have a why that makes the survey cohesive everything hangs together and as a consequence your people taking the survey hang with you they're willing to pay attention to stick through difficult questions or long surveys because they get why you're asking them it makes sense to them this will decrease the dropout rate to which ultimately makes your serving experience a cheaper experience because after all whether you're getting whether paying time or money whatever it is having those days go by and you're not collecting enough people or paying per person who's completing your survey it's important for everybody to keep those costs of time and money low so this will be having a why will be one way to ensure that you have that happy next it also structures analysis again we won't talk about this but come see me later if you like talking about statistics my other favorite subject besides surveys statistics are fun everyone loves math when you sit down to do them at the end sometimes that can be almost is overwhelming as writing the survey itself right you have all this data you got all this data back and I don't know I don't know if you guys have ever had this feeling yes where you're sitting there and you're looking at all the data and you're like I have now what I do, I'm just going to make some bar charts and call it a day. Yeah, no, no one's done that but me. Ok, fine. So when you go to present it, when you go to analyze it, it helps if you have a why the guided why you've asked the questions in the first place, because what that does is it makes it a very clear analysis strategy it basically rights or power point for you or your presentation, or your white paper or whatever blogged post you're trying to write for this, it makes it so that it walks you through it because you know why you asked every question and how that information should link together? Wait, take a lot of surveys that people send us, and we're always and it's amazing. Once you start to understand what leann is talking about here you, you you instinctively look at a survey and you think, wow, these people really nailed it, and they really understood how to write this survey in a way that was like, effective for them, and a good use of my time is the respondent and vice versa, you can really tell when someone's blown it on dh, they really have not thought this through, they've asked too many of the same question over and over again trying to get it something or they have a very clear agenda and they're just trying to get someone answer it's almost like someone's trying to get someone to answer the question the way they wanted so they can show somebody else the data so you know all those things are very clear once you kind of understand this stuff that the how well written the questions are is going absolute determine everything and starting with this wise critical so sorry no go for it any time speaking of interrupting do we have any questions from the internet? Cause this is the end of the first section of the wives section so is there anybody get well let's start in our studio audience and she have anybody has questioned you guys any questions so far we before we move to the next section I got a question sorry. So why is a very it seems like it's it's an easy then come up with but I'm sitting here like there's a lot of why possibilities out there can you throw out some really concrete examples that you would want toto? But you know just so the audience can really grasp onto something super tactic because why is a big why am I doing this well? Because I want to know what the flowers look like in spring time or like is it focused on it? What are some common business objectives that you want to know about? Sure so they can really run the gamut so let's take unease e example, let's say you're a lobbying group, and you're trying to lobby for gun control. You would want to ask specific questions that will help you make your case as a lobbyist for gun control. They're questions that are going to be a relevant for that, and then there are questions that are going to be relevant there's an infinite number of questions you can ask about public opinion on gun control, right? But if you're a lobbyist, you need certain pieces of evidence that will make your case with the politicians you're trying to influence. So you, as a content expert, will have to know what those pieces are. I don't know that I could help you with that, but what I can help you, it is figuring out what's the overarching goal of this survey, and if the overarching goal is to convince politicians that gun control is a priority of the american public, for example, then you're going to want to collect data that illustrates that point there's lots of other ways they would convince politicians that gun control is important, but you need to pick one way that you're looking at first, yeah, so for example, in that example for questions you might say, ok, question that would make a lot of sense and you would ask, is do do you believe that background checks for gun control should be regulated by the federal government? That's clearly, like in terms of convincing people, the question is probably not going to convince people, and this is where you sort of go back to that. Why? And you say, well, that probably wasn't meet the criteria is, you know, how do you feel about guns, right? That's going to get you a lot of emotion, but it's probably not going to convince anybody it might be good for some other kind of survey, but that question is not going to give you a lot of useful data, probably in this context that be useful, so they both seem related, but the first ones probably actionable on the second one is not so you might ask a question like, how upset will you be with your representative if he or she doesn't say port gun control issues? That might be a good question to ask if what your impact, what you're, why is of the survey, is to convince politicians to vote for something. But not if you're just trying to illustrate public perception so again you really want to pay attention to what's the what's the point it boils down to sort of what's the point why you spending your time doing this e I think there's a question on the way in the back to you so before we ask the right question to our conscience I'm selling first off on selling product with my client so before I how do I get my client to take myself they so right before they you know everybody's so busy with their life they might not even have five minutes but how do you get that they have five minutes to answer yours away that's the key that's a great question that's not as much what we're going to be able to cover today but I'll give you a quick answer because I think that's a really good question the why is going to help with that to begin with like I said, if people open your survey and it seems like it makes sense like dave was saying it hangs together it doesn't feel like you're trying to waste their time they're going to be more likely to take it what I always say in general is that a good survey is like a good conversation so when you start a survey it should be just like starting a conversation and to the extent that you can have your survey feel conversational and feel approachable to your respondents they're going to be more likely to continue that conversation with you and stick with you through the survey that's a great question and keeping it short which is kind of again go into the question the why we are hugely in favor of short surveys the shorter the better that still gets you the reason right right but I mean once one questions not enough obviously but but oftentimes in the hit in historically where you were trying to reach people by mail or by phone it was very expensive to do so and as a result people tried to jama's many questions into the surveys as possible so you've gotten really long surveys legends when that got translated online people left he's really long serious but the truth is online you don't have to do them is long on dso if you focus kind of on the why shorter surveys are more likely to get more responses because it takes less of their time and you still may get to the answer you want my question is on gathering subjects do you think incentives work better than not giving incentives you start not door incentive its office so the answer is a little counterintuitive like I said, my phds and social psychology so that's almost more of a social psychology question than a survey question basically what research has shown is that if you give people small incentive very small like for example fifty cent charity token that they're more likely to like the task they're doing they're more likely to stick with it and they're more likely to recommend it to a friend so when the time came to structure our audience product where we have respondents if you don't have them that can answer the survey that you're sending out for you we sort of stuck with that model of trying to keep rewards as low as possible. Now sometimes that breaks down if you need a specialized population you're going to have to adjust your rewards accordingly but in general people who do things for less money or for less sort of monetary gain they're more likely to be doing it because they love it because they care about it. The most frequent answer we get when we ask people why they do surveys is because they like being heard it makes people feel empowered just like davis saying they like the idea that their voice is carrying through into what then someone's next product design is or the decision someone's making with their company if someone's just doing it to get paid, they're probably not going to give you kind of the best answers and so that's what we had was saying like very weak or soft incentives can be helpful to kind of improving the response because people feel like oh, they're valuing my time so there's there's but if it's just purely the I just want to get through this as quickly as possible to get my reward at the end that's not going to get you good data so it's a it's a little bit of a balance, but like I said, we give we have this product that people can buy respondents if you didn't have your own and, uh, we'd make a fifty cent charitable donation to a charity they choose, but we don't pay them directly, and that sort of makes them feel like we value their time. They get to help out charity, but we're not paying them directly. So the next thing that you want to think about is what questions do you need to ask? What questions do you need to ask to meet your goal or to change your practices in your business or to make a decision or to collect information to influence the people you want to influence? To make your case for whatever you're trying to make a taste for if you think about it a sort of gathering evidence for a court case almost what kind of evidence do you need to prove your point? The short answer is something that we in the business world called deliverables, which is just a fancy word for stuff you care about stuff, you can take actions on stuff you can do stuff with it's important that you, khun, do something with the information you get back that you're asking things that you care about don't bother asking things that you don't care about, not only wasting your space on your survey once we already told you, you know, keep it short, keep it short, we're always trying to get people to cut things don't waste those precious questions on questions that are relevant make sure you're asking things you care about, and your respondents will will notice that when you go back to your both your questions about how to get people to take your survey, stay in your survey, motivate them to do it if they feel like they're times being valued like day was saying, one way to do that is their understanding that you care about the questions that you're asking about. Unfortunately, measuring in the survey world is tough it's not quite a simple as just taken a loaf of bread and pulling out your measuring tape or putting it on the scale it's very easy to see how big or long or heavy the bread is right pretty simple. Unfortunately, what surveys it doesn't quite work that way there is no measuring tape for happiness, then I know unless you guys have one business that would be awesome, I would really appreciate it I might be out of a job, but it would be so how we construct this measuring tape for happiness then right because we need to measure something like happiness something that's intangible how do you measure the intangible that's? Why surveys air so difficult because you're trying to do the impossible to some extent a little like a superhero how do you turn happiness into a loaf of happiness we do that with something we call an operationalization do not try to say that three times operationalization is again just a fancy word for a way to make the intangible tangible away to take a concept an abstract idea and turn it into something that you can measure concrete lee that you can ask somebody about and you can compare their answers and turn it into numbers uh scooped myself so can you guys think of having seen it for two seconds on the screen can you think of some ways that you might operationalize happiness to see if you were paying attention to what I just class the other two guys any ideas how much you operationalize happiness how could you measure happiness so for example one way you might do that is just asking some of the question right how happy are you that's the easy one another way white be yes defined the characteristics of happiness that you want that measure that great so give us a characteristic what might be your affect your emotions in your face how excited are you how great smile exactly so you could take a picture of someone and then you could ask someone you know how big is their smile how happy do they look to someone of another idea how else might you measure happiness? Yeah that promoter score asked them how likely are you to recommend this to your friends that you picked a pet peeve I didn't bring a promoter score it is very popular it is not my favorite but that is another another way that you could do it we'll talk about later why why the net promoter score makes me cranky but yeah no I mean you there's lots of different aspects right? So there's a million different ways you could think about happiness terms of how much someone smiles like you were saying in terms of your affect in terms of asking perceptions of other people how many good things happen to you this week how stressed out do feel sort of getting it from the opposite direction lots of choices what come when it comes down to the difficult choice is which one which one do you use right? You have all these different options now you're kind of back to that beginning stage again where you have all of these different measures and which one do you actually go for? You guys have questions so far before I tell you the punchline like a cliffhanger I want the punch line you're killing me all right, so how do you to the interesting thing is that each of these you could argue would have a different use case again this will go back all the way to our why? Why you asking so the very short answer of which measured you choose it's it's not gonna be that surprising it's deliverables what do you care about? What do you doing with the information, right? What are you doing with that information you got back? What point are you trying to make? So if you look back at this slide you might want to know about smiles per day if you're going to use it for what? Does anyone have an idea? Why might you want to measure happiness this way? Great job website that makes people laugh let's say your your comedian who's doing a standup website you want to know how happy it makes people feel because they're more likely to come back if they feel happier. That's what your ideas so you might want to measure that by how many smiles you see on people's faces out in the audience. When you're practicing your material, you might want to use good events per week if it's something like you wanted and understand your therapist you want to understand how to make people think about their lives in a different way you know anyone can look at the same weeksworth of events and see something very different right? It looks like a very different landscape depending on who's looking at it. So if when you look out at your past week you see more good events the idea might be that if that happens you might feel happier or you might make better decisions or you might eat better or whatever it is so that might be a better use for that so you can start to see that it really depends on who the person doing the survey is just because it depends on their why why are they asking about happiness? Why do they care about happiness? What are they going to do with that information? So a lot of times there come up ah variety of decisions you have to make in terms of what operationalization what measure do you choose? One of the main classic debates is between whether you made measure behaviors are attitude it's there's lots of different pros and cons. This is a question about ice cream but with you can see are trying to get at how much people like ice cream one is trying to get at it in terms of behaviors it's measuring it in terms of either a range of numbers of time they ice cream orjust particular number um and it's looking at how many times they ate ice cream last last week that's a way to operationalize how much someone likes it the idea being if you eat ice cream more often you like it more the other way is sort of more of a classic attitude measure how much do you like ice cream and then you you tell them how much you like ice cream? So which one of these do pick we get asked this a lot the difference between the behavioral measure and the attitudinal measure is that the behavior tends to be thought of is a little more objective right? Either you ate ice cream three times last week or you made it five times this unless you're lying to us right it's mostly true the attitude can be in a little more subjective maybe you ate ice cream five times last week but you didn't like it all that much you just ate it because you heard it was good for you, right? You're not really into it but they said you should eat it so you did so attitudes measure the intensity of your liking and behaviors measure the frequency both are important sometimes one will be more useful freer why then the other one? Sometimes one will connect back to your overarching research question better, but we suggest if you can't decide, just use both now again he's this with caution because we just lectured you about keeping your survey short, but if liking ice cream is a really central point to the why of your survey ask it two different ways. Intensity and frequency are different things, and it might be good to see if they are related to each other in this particular instance in the terms of ice cream. But what about you? Do people who love ice cream eat it frequently? Maybe not. I would fit that category. I love ice cream, but I eat it very infrequently because I can't afford the calories. So you know so but they do measure different things in that regard. Exactly. So you can imagine just sort of on the flip side. Do you love kale? How often do you eat kale? It might be that people don't really like, but they're eating at every day and it might be if you're a kale grower, you would want to know what it is that's going on with the people who are buying your products. Are they buying it because they're madly in love with kale or because they're trying to get their vitamins and minerals and whatever else is in cale, when you are a person developing this survey, the information should be useful to you if you're a person who manufacturers ice cream and people are saying they love ice cream, but they don't eat it. Well, the problem is not that you need to market your ice cream is something lovable clearly you're already succeeding there everybody loves the ice cream the problem is that they're worried about the calories or they you know, just I don't think that it's part of a healthy diet or whatever it is and so that's what you need to be focusing on as an ice cream manufacturer so again, you really need to think about that why and make sure you're asking the questions that get you there and not to go too far advanced yeah you probably in that situation you would then want to sort of have follow up questions for the people who like ice cream but don't need it very often to sort of understand the why that behavior existed, right? I mean because that's what you that's the key insight that the ice cream manufacturer need is to understand absolutely and as the if you're the ice cream manufacturers sending out the survey you're the content expert on ice cream so you need to know your context to some extent to be able to write a good survey. The best survey reading experience I have is with people are clients who really understand the content of their area so I'm able to help them draw out the why and the what and the which measures to use but they really understand the landscape of what they're looking at so the moral of the story when you're picking which measure to uses don't take shortcuts, that's, that's the one thing that I would say about picking a measure, uh, figuring out the right operas like operation sea operationalization is hard on don't get lazy because it's worth it to put in the hard work up front. What I see a lot with people surveys when they come to me with them, is that that they get a little a little lazy when they're writing them out because they want the respondents to do the hard work and that's again to go back to both of your questions. When you start having problems with retaining people in your survey because you're asking a respondent not only to stay with you sometimes for free through this survey, you're asking them to do the thinking about why the questions are being asked and yeah, yeah, and also maybe think about it so here's an example of, like, where people aren't thinking through how they're going to get that operationalized data admit back and oftentimes you'll see in a survey the question will be, you know, do you do something frequently, yes or no, right? But you want the frequency, not they yes or no, because there's not a lot you khun dio uh, with that with that lack of frequency so thinking about how you're going to use that data on the back and sort of gets you to asking the question in the right place up front absolutely so let's talk through a typical example I've told you a lot of stuff that's, nebulous lee bad let's look at a concrete example of one of the most frequent things I see that makes me cranky and, um anyone who's worked with me at work knows that my least favorite word is satisfied because it is one of the vegas two words on the planet it's it's, it's, the laziest serve a word I have ever seen it it's a catch all it means anything you wanted to meet and more you can use it in any context in any domain and as a consequence, it doesn't mean that much to a survey respondent, when you use a word satisfied, they have no idea what you mean and it's probably because you have no idea what you mean. Yeah, it's not, but if you stop there for half an hour, you couldn't figure it out he just did it because your boss that it was due the next day, so you wrote satisfied and you sent it off right? It's fine, we've all done it, you don't have to hold up your hands the other part that's wrong with this is the third experience again that's a that's a big word in this case like satisfied right? So it's not any help with the delivery of als that we're talking about? How do you take action when you're talking about experience? What does that mean? The experience like there's a lot wrapped up in there again it's it's a good word you can use it for any occasion and consequently it means nothing. So how do we how do we fix this let's fix it in terms of satisfied like I said, when you're a content expert, you can break it down, right? You know what you mean by satisfied? You're just not telling me if this is the question, you know it wasn't comfortable was it fast? Was it convenient? There are an infinite number of adjectives you could use their as the content expert you have to decide what you care about part of deciding what you care about what adjectives that you want to ask about is what you're actually planning on changing if you're not going to be able to change something or if you can't really do anything about the problem there's not much point in asking about it because what are you going to do with that information? Just stay up sleepless at night saying I wish I could change the fact that cars get stuck in traffic no asked a question that figures out something that you can actually do right something you can actually change the same thing goes with experience break down the parts of whatever the experience in this case we're talking about some sort of cab service so is it the iphone app or is it the actual car is that the drivers you can imagine that you might I think the driver you had was just the best but they were driving this run down just a total mess of a car and that was really unpleasant so if you just ask about the experience all of that richness of your data is going to get lost and as a consequence you won't know what to change if someone just says well I was only slightly satisfied with my experience what do you do with that? What do you fix? Clearly you have a problem right? You were trying to get repeat customers but they're only slightly satisfied well now what now now you take the data to your boss and you're like care customers air slightly satisfied with our service and I'll see you monday right? You need to know what to change and so being specific with your questions like this staying away from those vague operationalization is like satisfied and experience anything that sounds like a buzz word or anything that you think that you just are going to pick it so you can move on to the next question stay away along the same lines there's a lot of shortcuts that people take, we'll just sort of run through them and feel free to interrupt me and stop me if you want more details about any of these, but try to avoid open ended questions, open ended questions can be great for exploratory research, and if you're a sophisticated researcher, which some of you in the audience, maybe they can be a really wonderful source of data that you can spend years coding and analyzed. If you live in the fast paced world of most of the rest of the universe, you're going to want to try to stay away from those they're unfocused. They help your is it your respondents sort of opt out and lose attention and make their eyes wander. Remember these people? For the most part, if we're talking about survey monkeys, business at least are taking surveys on the internet. You are not sitting there staring over their shoulder, going stop looking at google news and read my survey if you let their minds wander their minds, I'll be gone before you know, and they're thinking about their grocery list, especially if they see a big blank box staring back at them. I mean, I think there's also like it goes back to, like, you know, we were in school you know, would you rather take a multiple choice test or an essay test, right? Multiple choice a lot easier and people are thinking about how to sort of use their time effectively this is you staring at that blank box and trying to think, ok, what do I write here? How do I get through this as quickly as possible? They don't get great data there and then it has the other problems for you on the analysis as well. Exactly so, like dave says, if you have some sort of set options you provide will go into what those should look like, but the idea of set options is a good one. The next shortcut you want to try to avoid is a ranking question so sort of you know you have you do offer this in serving radio on the way, but weii advise against yes, we offer you any kind of question you could possibly not want to use? We're going, yeah, even if we don't like we're not judging, but so here I have the seven dwarfs for you and I'm asking you to rank the seven dwarves in order of your favorite door fright or I think I will prove decided their supervisors hear how approach um well that's good you always wanted dwarf supervising you so clearly doc is going to be everyone's most approachable dwarf here but then after that it gets a little fuzzy right? The problem with the ranking questions is that when there's more than about three things that your ranking you can usually come up with a top one and a bottom one and maybe one on either side and on the middle is a mess like what's your fifth and fourth and sixth favorite door flick now you don't know right? Maybe you do I don't know but ranking tends to be a long hassle and it ends up being a huge time suck and the middle is essentially useless because people are just randomly selecting numbers you also don't have a sense of scale if something's you're first favorite and something's your second favour then you have a third favorite the interval could be huge you could love chocolate ice cream love vanilla ice creams a little bit last so I guess that's second and then you hate strawberry and that's third so what does that really tell you? You know the order but you don't know the strength of each of those opinions, right? And that could be a huge problem depending on what you're using the data to make decisions about so you can turn the ranking question in a variety of things I'd recommend either if you really care who's the who's the most approachable boss here if you're having dwarfs his boss, then just ask that which is most approachable that's what you really care about don't waste their time making them rank the other six just pick their favorite what I also recommend is if you do really care about each thing individually that you're getting ranked asked each question individually now this is something that people push back against a lot because hey, with a ranking question it's just one question right? You get seven questions in one saves space and we just told you more questions or bad you are not fooling anyone. Yeah, your respondents are totally on to you. They know that that's what you're doing, you're making them do seven questions at once, so if you look shorter to you and it's not shorter for them, so I know that would be seven questions, but it is seven questions anyway, you're just lying to yourself. The last thing you want to avoid in terms of short cuts is the matrix question. This is one of everybody's favorite questions. Like I said, I did my phd a come from the land of academia academics love matrix questions so it's not just a business world shortcut thing. Everybody loves a good matrix question, mostly because everything's lined up neatly and if you have the same response options, it looks pretty on the page it's very pleasing. Unfortunately, that pleasing is also exhausting overwhelming and repetitive for your respondents, they get lulled into a lovely sense of I'm going to click the same number down the entire column because I did it on the road before and they stop reading and they don't engage with the question and again if you're trying to get respondents to stick with your survey and to pay attention to not only finish the survey but give you good data while they're finishing it, then you want to avoid things like a matrix question it's just not going to go well for your data quality anything we can do to rid the world of maitresse questions we'd like to do for those of you that are thinking some of your respondents might take this on a smartphone, the matrix question doesn't format very well as you can imagine on a smartphone. So just another reason way put the surgeon general's warning on this like don't use this question type it is very popular we I have not been very successful dissuading people so well we may weaken start today absolutely you guys are ambassadors. The other thing about this makes six question that people get a little lazy is you can see the response options stay the same from like a great deal to dislike a great deal it's an easy way to ask about different things usually, though if you start breaking down the rows into individual questions what you find is that different response options fit different aspects better so for example, you might want to ask about, you know, was the stuffing dry? How dry was the stuffing for the cranberry sauce? It might be how sweet was the cranberry sauce there's different things that you're interested in there about how satisfying eating whatever food wass there's different aspects about different foods that you care about so instead of just giving the like dislike sort of vague response option when you get out of that mode of putting it all in just one giant matrix you start breaking it down individually you find that they're actually individual qualities that you're asking about and that makes her serve a lot less repetitive too. Not only does it anchor people on the experience of eating that cranberry sauce if that's really important to you, but it makes it feel like they're thinking about different things that keeps them interested. It keeps their attention, but it forces people not to ask a laundry list of things up off which they don't really plan to use all of those things anyway going back to the earlier stuff, they said it's like if you have to ask discreet questions, people are more selective or what they do they don't this is this is like burdening the the respondent with a lot of work and I think you might not use exactly if they think about what it means to be satisfied with a plate of stuffing if you're the responded I don't know what it means to cease satisfied with a plate of stuff you tell me what you're making the stuffing what does it mean to be satisfied and then they can answer that so you need to do the hard work for your respondents so that they sort of glide through your survey easily ok, so we're not pause for questions from the audience or from online absolutely yeah we've got about twenty minutes ago a question over there from change yes, I was really taken with the idea of the conversational part that you open this section with and how important that is. It seems to me that it's actually almost mission critical because if you're serving somebody in a language where style that's not natural for them or in the parlance of that industry uh you guys find that that actually is as important as it seems like to me yeah it's it's huge again that's one of the things that I say it work over and over again a little bit of a broken record like that my colleagues are not in the back of survey is like a conversation and a survey is like a story those are my two favorites because it really is it's it's something that should grab people's attention right up front and lead them through your story that you're trying tio ask them about the point you're trying to make the decision you're trying to make the image or picture of what their opinion is they're trying to create so it really is a very structured conversation that's why it's so challenging because unlike a real conversation which is fluid and you can edit it as you go on long and you can give feedback and adapt to hear speaking with when you have a survey it's rigid and it's fixed and there's nothing you can do to change it once it's out there so that's why you need to be so sensitive and so specific in terms of the questions that you're asking people because once they get them there's nothing you can do right it's it's up to them to engage with the survey it's sort of like having local knowledge right? So if you're gonna write a survey for a bunch of people who are writing code versus if you're gonna write once tasty for the music industry though they're going to be really different language and if you're in the music industry and you see when that's written for a woman whose coding then you know that you're you're less likely to engage in that stuff it's just like teaching a good class hopefully to you manage to give people examples that speaks to them in the language that they understand when I teach different audiences about serving you have to be able to adapt it a lot because people need to be able to understand how it's useful for them. A survey is the same way. If you speak to parents. If you're trying to, for example, sell a toy if you're a toy manufacturer, if you're trying to have your toy do well in the market, you want to know how it can kids feel about it and how parents feel about it, right? But you don't want to ask them the same questions. Yeah, so if you, uh, you go up to a parent, you say this, this look fund a shake like the parents not not feeling that right. You want to know if the parent would be willing to buy that toy for their child when you're asking the kid, though you don't want to know if he thinks it's an educational advantage for himto have this stimulation early on in life, right? The kid may be the kids has an opinion on that at the age of three, I don't know it's possible, but the most amount of most of the kids will be like, I think it's pretty colored and that makes me happy, right, so you need to be able to speak. To your audience in the language that they understand to strike the right tone with them so that you're not talking down to your respondents, you're not talking up at the level that they don't understand, but you're speaking directly towards them again when you're thinking about engaging people and making sure people take your survey that's, how you do it, if they feel like the conversation is directed at them, you'll be much more successful in one little trick on that, too. You can always test your survey, you know, you may not have been able to gauge up front, right? Did I strike the right tone? Did I engage people but run a test went to a small group of people, see what happened? You know, if eight out of ten people didn't finish your survey, you got a problem in the case, right? You know, so you can that's a good way to think about it is like, ok, I'm losing people on this page of the survey, clearly there just the conversation broke down there, just like a language I want to speak the same language is that people that you're trying to communicate with you could get a lot of data from testing, and if people don't understand your survey, they're not the problem, you're the problem, you're serving the problem like if you write a piece of writing and nobody gets in you're like it's all their fault they're all crazy my writing was crystal clear no, you're the problem you're serving needs to be understood so I always recommend that everyone send their survey to their mother just to see if she can sort of access the survey for whatever audience your mother is always a good test audience um but the idea is that it should be accessible it should be written in a way that everyone can sort of understand what you mean because it's frustrating to take a survey where you don't understand the questions from those I mean sometimes don't know will be will be an important option toe have but you don't want to feel like you have no idea where they're talking about so we'll go into that more in a minute but questions that have grown without about ten minutes until we're gonna keep going until we can do some more cuban, eh? So here's a road map of our last section this is what we're going to cover in our last section, so memorize it, write it down ready for it here goes the first thing is you want to communicate your intentions this is really important, so we were just sort of hinting at this now so we're going to go into more detail these are all things that used to avoid so here is a laundry list of stuff not to d'oh the first thing you want to avoid his jargon like I said don't talk to people in a level that they do not understand. This happens a lot by accident because you're very involved and immersed in whatever culture it is that you're working in, whether it's the nonprofit world or the business world or you know, computer science or music or whatever it is if you're not surveying your colleagues, you're going to forget that not everyone lives and breathes whatever this thing is that you d'oh right so it's important to remember that and you want to avoid complicated language or buzz words or vague terms that the general public won't get because it doesn't feel good to feel stupid when you're taking a survey it's not it's not a pleasing sensation, so try to keep it sounds simple try to make sure that the questions stay simple you also want to avoid complicated sentence structure, so I actually you might think this is silly, but I've seen this kind of thing on a survey pulled the straight from a real survey so who shall remain nameless? But what is the state of the cleanliness of the room actual question and really was what you're saying is how clean is the room now it's the same question but that directness will really go a long way double negatives this is this one's kind of common sense they're great for catching people who are not paying attention so if you ever just want to have some fun with people do something like click yes if you're not really paying attention and then click yes and it's funny but if you're not trying to do it was a trap question pick a side don't use double negatives you can pick a negative adjective or positive adjective but don't add that not in front of it to negate it it's just confusing and it required makes people to go slower it requires them to sit and think about it also avoid double barreled questions this is one of the most common errors I see in terms of little things that can really drag a survey down a double barreled question is when you ask about two things at once this happens a lot because this is how we talk this is it because so survey is like a conversation people write down questions that mimic how they speak in real life so you might say something in real life like oh do you like apples and pears now the problem is when you're dancing this is a survey question right that you have an opinion on apples and an opinion on pears and those may be contrasting so what do you do if you hate apples and you love paris how do you answer this question you have choice of one, two, five or I love them so I hate them. How do you answer that question? What's the solution split it up into two questions. Ask about apples and then ask about paris again. You get the pushback? Oh, yeah, but you said fewer questions. This is one question this is two questions. Two questions is bad now you're respondent is not being fooled they're still thinking about it like two questions when you have it is one so again, this is not about you. This is about your respondents keep them happy. The next thing you want to avoid is leading questions. Dave talked about this earlier something like how terrible is this for your health? This could be a tricky balance to strike, but you want to try to keep it balanced as much as possible, especially because if you're using this to boost a point or to bolster a theory or make a decision you don't want to be led into the wrong decisions simply because of the way you ask the questions. The last thing to avoid is absolutes. This is not necessarily an intuitive one, but it's the one that's a good thing to have in your toolbox when you ask people a question like how good a teacher is, mr frank that's a that's a big, bold statement he is a terrible teacher well, they may only have seen mr frank for an hour for a workshop so that seems like kind of a harsh judgment to make about a guy you've only spent an hour with if you can make it more specific more personal, more approachable something simple like how good a teacher do you think mr frank is do you think mr frank is it takes it down from sort of that good with a capital g some sort of absolute truth that they're trying to trying to convey to you and puts it on a personal level where they really feel more comfortable stating an opinion you hear this from survey respondents a lot? Well, I don't really feel confident enough to give you an answer on that I don't really know enough about health care to tell you whether I think it's good or not so you need to make the questions feel like they're cut down to size enough that people feel comfortable answering them so use your words is always important talk to people like you were saying earlier talked to people in a language they understand people don't think in numbers they don't think that new pair of pants they got is a four you know they think it's great or not right people think in words so speak to them that way and try to use standardized options so that they can connect back to numbers that's called a like good scale, by the way, a little more detail about a lakers cal gonna go right straight through it. You got two choices of lakers calcium unit polar like earth scales and bipolar lakers skills. Go back to, you know, polar for a minute. You know, polar is should be your default every time. Five answer options going from sort of zero to one hundred with regular stops in between, it measures the frequency or the intensity of something. Usually the great thing about these is we can just sort of drop in and the adjective you want so doesn't have to be extremely happy. It could be extremely frustrated, extremely sleepy, anything you're interested in measuring bipolar scales. People tend to love these and use them all the time, but they really should only be used sparingly. The reason for that is that bipolar scales require that the two end points so the bottom guy here in the top guy, right, that increased a great deal in decreased a great deal, it requires that they be exact opposites of each other, which is not necessarily always the case increased in decrease, no problem, but angry and happy wells are really opposites of each other. The midpoint also should be a clear midpoint. I'm not sure what it means to be neither angry nor happy that's the confusing statement if you can't figure it out in five seconds when you're thinking about it, don't use it but now that neither increased or decreased as in remained about the same that's pretty intuitive it hasn't it works exactly so for this one it's great your weight stayed the same everybody gets that you don't need five seconds to figure that out, but from happy to angry and beating neither happy nor angry not quite sure that means so try to avoid bipolar scales unless you have a really good reason for using them it's really important to allow for shades of gray and people's opinions so this was something dave was trying to scoop me on earlier to read ahead, you know s so you want to avoid yes or no questions for the most part people really like yes or no questions again because that's how you talk but with a real conversation you say yes or no and then you explain why yes or no right in a survey that's kind of off putting people don't want to see a big tax box that's empty with them explaining why yes or no they just don't like it, so you need to do the hard work for them again don't get lazy do the hard work for them beforehand because people's views on things are for the most part, fuzzy it also when you sit down to do your statistics at the end, we're not going to get into this, but trust me on this you're going to want the variants it helps you avoid errors and for false positives for seeing a result when there's not really one there and false negatives results not finding your result when there really is a result there trust me, you're gonna be so glad that you have five response options and not just yes or no question, so avoid those whenever possible. You also want to keep questions accessible don't ask things that people don't know or couldn't know it's frustrating like we were talking about before to not feel like you can really answer the question. So instead of asking a question like this, trying to make it something that actually have access to it's easy for someone to tell them tell you how worried they are about something it's hard for them to regurgitate some sort of detailed knowledge on a policy now it depends on your why it may be important to ask that the first way, but you really want to use that sparingly because it's a frustrating survey experience to not be able to dive into those questions also don't ask things that people couldn't know this is again a riel question from a real hr survey how well does your boss budget his money I have no idea who say they're lost much of his money, and I don't know how organized is your boss that's something I can answer? I have lots of opinions on that just as an example, we there's a way we do a lot of parents, surveys and teacher surveys, and one of the things that the gates foundation's done a lot of work on is to figure out how effective teachers are, and they've done a lot of objective measurement cameras in the classrooms lots thinks meant, and one of the things they found was asking some of the right survey questions. They could get very high correlation with all their objective measures of who's, an effective teacher, and it turns out that it wasn't like how much do you like your teacher? Because asking a student how much they like their teacher, had nothing to do, whether that teacher was actually good at their job, but it was sort of getting at the right question off things they could objectively no, right? You know you couldn't ask them, you know, is your does your teacher? Is your teacher extremely knowledgeable about the subject? Well, that the student doesn't know what the subject is, you couldn't ask that, but the thing that worked out that made me think of this is how well does your teacher used their time? Tended to be very highly correlated with teachers that were actually objectively measured, effective and improving students results, and so it's a good example of a the right kind of question, you know, that the student could answer and be it happened to be a great result in terms of being able to use that question to sort of measure teacher effectiveness. I mean, what happens exactly what happens when you can't or don't know? The answer to a question is that you answer it randomly and that on a weight is the worst possible thing you can happen because this I've said this is my other repeat phrases bad data is way worse than no data it's terrible, it is the worst possible thing if you're making a decision on it, it can cost you millions of dollars if you go with the wrong ad that's a huge problem if you lose money on your product. That's a huge problem if you pick the wrong public service, announce it meant for your non profit if you pick the wrong metric for teacher success and promote the wrong people. That's a huge problem, right? So you need to make sure that you're not getting bad data because it's just the worst possible scenario. So you don't want people to be randomly clicking for in a column down a matrix question c I snuck that back in there don't use magics questions. You don't want people to be randomly answering your questions, you want them to really be engaging with them and to really be giving you quality information that you can make the right decisions with, okay, we're almost done last lied hang with me. Don't let people opt out of your survey. This is again, you want to keep people engaged, don't let him get disengaged for one minute, you need to hold their attention, even though they're not sitting there with you and surveys of old. You would be standing on someone's doorstep, asking them the questions right with a clipboard. I started out asking surveys that way swear, but now you don't now they're sitting in their computer alone. How do you get them to stay in your survey? Well, one way is to avoid having a no opinion option don't don't do that just don't do it. You want them to have an opinion that's why they're ask you're asking them if they don't have an opinion believe you me, they will quit out of your survey so fast you won't even see it coming, but if they're staying in there, don't give them the option. To not have an opinion, you're communicating that that's an ok option for them that that's an acceptable response and what you want is an opinion or you wouldn't be asking, so just eliminate that the other thing again as we talked about is don't know now once in a while if you're asking a really tough question don't know might be an important option have or you might as well a lobbyist, for example, about gun control to go back to an earlier example, you might want to know how much people know about the whole debate in general don't know might be an important point that you're trying to make but use this sparingly you on ly want to leave it in there for questions that are truly difficult to answer other than that you want to take that out as well again it's just communicating but it's ok then to not engage with the question you want them to try the other thing you really want to eliminate is not applicability this is another big pet peeve you can use instead something we call in our product skip logic but different people call it different things but you ask the one use I think of an acceptable use of of yes, no question is our help center useful yes or no and then or have you ever used our help center might be an even better skip logic question and they never use the help center. They don't have an opinion on it, most likely right? But instead of putting a no opinion or not, africa will option you just asked them, have you ever used the help center? And then they can rate it if they have some make sense. So again, with every question, you really want to keep them as engaged as possible, and by get not giving them the option to sort of step back their focus and their gaze from the survey, they're going to stay with you all the way through. So just to recap, this is all the ground that we've covered today, so a lot of stuff, why you asking questions? What questions you need to ask, which measures you're actually choosing, and then how to write literally word for where those actual questions when you sit down, we even had some some festive examples for you, but I think we're out of time. So one other thing I'll just throw in there is we get to the end. We also to make this easier for you in our tool have something called a question bank, and this is about two thousand professionally created questions that our methodology team has vetted on the most popular subjects, so you can go into our question bank on any topic and probably find at least some questions that are relevant to you and yes someone asked about net promoter score before think you did we have our net promoter score question and you can use you know it's the right way to ask that question we have lots of those and so that's the good thing is if you take something for our question bank you know the question was designed the right way so that's one other way we we help you to kind of get to a shortcut and I have personally looked through every single question so I can guarantee you they weigh on approved we haven't also in not just in english but in four other languages now so I think we're almost out of time I see the next speakers are up next actually can we get a round of applause way thank you so much there's a flood of questions from the internet and I know that you guys had a presentation that could have gone on for another ten or fifteen minutes but is there something we can we can at least give those guys one let's do one let's do one quick one and from nairobi or from somewhere interesting if you got it all right well, this one is um let's see, I'm just going to go with this one from a mile high guy again in colorado because he has amazing questions and loves the ring, right? Yeah and do you conduct the identically same survey again and again, say every three to six months to learn how responses change? Or is that not a good idea? That's a great question that's something that some people call it tracker survey if you send it to different people each time you refer to it as a panel survey a cross sectional survey, it could be a great way to establish a baseline for what's going on and see how something changes over time, so that not only do you have a sense of what's happening right now, but you know not just that idea in a vacuum, but how it compares to what's been happening, his storr ycl e with your product or your company or the attitudes that exist in your country or your community way, see, you know, presidential approval, consumer confidence, those air asked every day, basically fantastic. Well, I do in addition, want to give you a couple of shout outs, cigar says. How valuable do you find the credit of five? They're the content on crave live question mark one to five, which would be extremely valuable five.