Inside TV Audience Testing

TD00428I visited Las Vegas this week with my sister to see Britney Spears’ show at Planet Hollywood, but we had to find stuff to do the rest of our three-day stay because neither of us likes gambling. Good news! It’s fall-pilot-testing season for the broadcast networks. And CBS and NBC reps were out in force recruiting people off the Las Vegas Strip during the day to watch some shows and answer survey questions about them. Oh, and they were offering anywhere from $20 to $50 for opinions, depending on the length of the show and the intensity of the questioning. Brilliant! Now we could “win” money in Vegas and lose nothing. We couldn’t present ourselves to the recruiters fast enough.

Both of us are TV nerds, so we loved this idea. I’ve always wondered about the mythical testing process, which is an integral part of any great show’s origin story. Inevitably in these stories (please see: The Mary Tyler Moore Show, Seinfeld), the audience testing says a show is total garbage, and later it goes on to great acclaim, popularity, and otherwise legendary status. Audiences hated The Mary Tyler Moore Show because they thought she was a single loser who hung around with annoying and scary people (Rhoda and Lou). They hated Seinfeld because they preferred Jerry’s standup to the actual story, and they thought the whole thing was kind-of pointless.

I won’t get too much into the specifics of the shows I watched, because I feel some sense of duty, like I just served on a jury or something. But I can tell you that it was really interesting to see the differences in how the two networks tested shows, as well as the questions asked. Both nets had the legendary Good/Bad buttons: At NBC we had Yes, I Like This! (green button) and No, I Hate This! (red button); at CBS, we had dials from 1 (hate) to 100 (love) that we had to keep moving through the entire show. We also answered endless questions about our feelings toward every character, plotline, and relationship in the shows. We picked our favorite and least favorite shows among those already on. We, of course, specified our ages and genders and income levels and places of residence. The worst part about this system is the binary; it’s incredibly simplistic to ask people whether they “like” or “don’t like” a show at any second. What does that mean? Do people really need to “like” a show every second for it to be good? It’s easy to pick out shows that, whether or not they were tested, you can imagine testing terribly. Mad Men? Too slow. Game of Thrones? Too confusing.

If you go back to the Mary Tyler Moore example, obviously the audiences weren’t quite ready for an independent, single woman or any characters who weren’t all sunshine and happiness; the show would push those boundaries until audiences did like, or at least more readily accept, such things. Seinfeld, I think, is a little different, because anyone who loves the show and watches the pilot episode now can see that it wasn’t anywhere near what it would become. So the testing question is somewhat moot when it comes to Seinfeld, though it does prove that some shows need a chance to find themselves before random people from the street pick them apart.

My favorite part of my testing experience this week was when we finished watching a drama pilot at CBS, which was obviously targeting younger women (18-49). The workers rushed my sister and me afterwards because we fit the target demo, and asked if we wanted to come back in a few hours for a more detailed discussion (and another $50). I was lucky in that I have a red-eye flight out of Vegas tonight, so I had time to stick around; she didn’t. Listening to the more in-depth discussion with my fellow target-demo ladies and a moderator, I could imagine how a writer would go absolutely nuts listening to what we all liked and didn’t like. (I wanted more backstory! I wanted less backstory! I hated this person! I wanted more information about every single ancillary character! The opening drew me in! I loathed the opening! And was that the main character’s sister, best friend, roommate, or lesbian lover?) As someone who writes books, I can’t imagine how infuriated I would be. (Yes, I can, actually, because I’ve made the mistake of reading some of my Amazon reviews.) Obviously the creator wrestled with all of these questions for months, and he has his own vision, and suddenly a bunch of randoms wandering in between eating tacos and drinking a 45-ounce margarita have all kinds of ideas for him?

At the same time, it wasn’t as bad as I thought it might be. A lot of the women pointed out areas of real confusion and had some genuinely great insights as to how to make the show better. If the shows I watched air, I’ll have a lot more insight as to how testing affects final products. And I’m grateful to CBS and NBC for helping me win in Vegas.

Advertisements

33 comments

  1. In LA they recruit testing auds at the famed Farmers Market and in front of Grauman’s Chinese. They don’t allow show business types so I assumed the persona of my neighbor and voted like she would have voted. They also test a bunch of commercials at the same sitting. Hardest part was during the discussion b/c I didn’t want to be revealed as an industry insider. A few months later, they tested a pilot of mine and I got to watch from a one-way window. What utter bullshit. This is why shows are migrating to the web!

    1. That does sound fascinating … somehow it’s even more terrifying to think of them applying these principles to music. Music just strikes me as an even more subjective and personal experience. (Though testing would also explain all the boring music on the radio.)

      1. the repeat factor is present. years ago one musician bet another he could make a song called surfing. One word over and over again. It made money.

  2. This is really interesting insight into the behind-the-scenes process. I wonder how much more critical the audience is, knowing they are supposed to be critiquing the show. Also, I’m glad the broadcast companies could help pay for your trip to Vegas.

  3. This sounds cool. I’ve always wanted to do this, see which shows made it to my tv, and then see if I could recall what feedback was offered and if the writers actually took any of it.

    1. Yeah, I’m really looking forward to seeing the shows now in their final form. The focus group I did was for The CW’s iZombie. The NBC pilot I watched was Ellen More or Less, which unfortunately didn’t get picked up. (I’m sort-of bummed because I enjoyed it, though iZombie was something I’d genuinely watch regularly.)

  4. I’d love to be in a TV test audience someday, because a lot of what’s on TV just doesn’t appeal to me. It’s either way too out there or unpleasantly similar to real life, often both by turns. It doesn’t help that I was raised on a steady diet of PBS either.

      1. Stupid isn’t exactly the word I’d use, but it seems like there are way too many shows on just a few themes. People only want to invest in something “safe” I guess.

  5. That was really interesting. Thank you for sharing your experience. I don’t think I would be a good test audience. Maybe it is my poor attention span, or my sometimes blunt criticism of TV

  6. They do testing, but how come there’s nothing to watch on TV? News, fine, weather, great and we have 1 documentary channel. These old comedy shows may be somehow ok, but some of them are terribly boring. I would love watching more Mentalist, but that’s just once a week. I never watch reality shows, they’re just breathtakingly dumb. We have very little of educational or non-fiction stuff in Canada. It would be great to see more art, theater, writers, good music and less of these dumb Honey-Bo-boos (sorry maybe spelled wrong), Kardashians (maybe spelling wrong, too). Every third channel has this type of entertainment, so it’s nothing much to watch: repeated movies, mostly old, very few good shows if any and news. That’s why I am watching TV only 5-15 min a day while I’m getting dressed. Exception is Mentalist. That would total in 1 hour a week for shows and movies.
    Can you popularize ideas about more educational, self-development, etc. stuff if you are a critic? Or maybe there’s no demand for that?

  7. interesting stuff, as mentioned by some here, I think the danger with asking and paying for an opinion is it can make you say more than you really need to, just human nature really. That said, Im sure there are methods in the analysis that filter this out to some degree.
    At the end of the day, tv shows should be about entertainment, and what one person finds funny or interesting, another will find boring.

    1. Exactly. It’s too subjective, especially when the options are “like” and “dislike.” You can not “like” every moment of a great show … you may find it uncomfortable and confusing at times, but that’s part of the point.

  8. Testing and statistics… they want variable data. .. quantifiable. Attribute data is much harder to gauge. But still can be done. However, attribute would be your I like this but not that. Segmenting each piece. Variable is a like or not and how much. I’m guessing but would make sense. Good article.

  9. Interesting look at how TV audience testing works- thanks for sharing. It would be revealing to see the stats of the correlation of level of audience approval to the show being a hit.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s