SXSW 2005 Panel: How to Inform Design

Monday, March 14, 2005

[This is a (mostly complete) transcript of Nick Finck, Jeffrey Veen, and Kit Seeborg talking about using data and research to improve design. Official details about the panel here.]

Finck: The presentation slides are available at: nickfinck.com/presentations/sxsw2005

Finck: I’m Nick Finck. I have a company. And I run Digital Web magazine. And I’m an ambassador for the User Experience Network.

Seeborg: I’m from Oregon. And I’ve had a blast. Professional webmaster and web producer.

Veen: Founded Adaptive Path in San Francisco. User experience consulting. Founded Hotwired.com in 1994.

Finck: First point. Some of the things that inform design… Assumptions, goals, research, conclusions, recommendations, change. These impact what kind of solutions you find that affect the users and the business goals. In a nutshell, these are the topics we’ll be hitting on. Next slide.

Seeborg: We are very interested in the dialog and discussion. We’re very interested in hearing what your experiences are.

Finck: Quantitative vs. qualitative. How to prioritize? Is all of the information we get about a site valuable? Would you guys like to weigh in with any comments?

Seeborg: I’ll speak first to the challenges. With so many tools to measure traffic out there, some challlenges include a difficulty with ambiguous language. “Traffic” and “hits” can mean many different things. So before you analyse, you have to look at why you’re using these metrics.

Veen: I’ve done so much research. And I made some slides about it. I used to do the quantitative stuff. What I found over and over was that I was measuring mistakes people were making. Ultimately, that’s what that kind of research will get you. I wanted to get beyond that to design for what people were expecting, what’s intuitive. Who are these people? [Photo of random pair of people on the screen.] So I swtiched to this kind of pseudo-ethnographic research. It’s not “scientific,” though. My favorite example in the world is the US Dept of Agriculture Hay Net: “Need hay,” “Have hay.” [Laughter. It’s really funny!] It’s great! Ebay: “Need Junk,” “Have Junk.” Etc. [Now he’s showing the Alumni Network page for Berkeley Haas School of Business. And it’s all wrong for the users.] So who is the audience? Well, you talk to people and write down what they say. Highlight the interesting thing and put it on stickie notes. I’m a visual thinker so I put things on little pieces of paper and move pieces around and argue over them. So we need to map the stickie notes to an organized architecture. So our users will find it intuitive — they’ll get it. Ah ha! [Wall of stickies photo.] How many of you saw Malcolm Gladwell speak? I read the book “Blink,” and it made me think “Man, that’s exactly how I do design.” In my office at Adaptive Path there’s paper all over the walls and sometimes it synthesizes and we’re like, “ah, that’s what we’re going to do!” It’s a process of seeing through theeyes of our users. And now it’s kind of self-centered design, with me acting the role of the user like DeNiro being a taxi driver before starring in the movie. Immersion beforehand instead of going back to fix later.

Finck: Similar to a process I go to for Digital Web magazine. I use digital stickie notes. You don’t understand the contesxt of the user until you put yourself in their situation. You learn much by just watching users use the site. Maybe even task them, but not even that. Just review how they use it, where they’re going. Maybe if they’re a friend you know a bit about their background. If they’re not getting where you want and you’re not sure why, sometmes you learn when you’re forced to use the website. Like searching for an old article. That how I learned about some of these problems. So we made a task-based navigation. Took a long time to redefine the site to get there, but that’s what we did. It was kind of controversial, some people didn’t like it at all. But as soon a people started using the site, people started understanding that it worked quite a bit better. They could find stuff they didn’t know existed on the site.

Veen: What was interesting was that you could make the change because you thought it was a good idea. I seldom have that opportunity with my clients because they’restake-holders and these decisions affect the business. They need financial justification. You were telling me about following the money.

Seeborg: Oh yeah. One of the first things I ask when people are telling me about their project is “tell me where the money trail is.” That determines the bottom line and who the ultimate decision-maker is. If you don’t know that, you might be blindsided with it later on. I had a meeting this week and asked just that question. That’s important. And about quantitative analysis: don’t get hung up on the datapoints. Look at your patterns, your trends, what’s going on over time. Site traffic status is imperfect. Back in the late-nineties it was only to show if server burped. So it’s informative to track those numbers over time. So spikes and other anamalies can tell you a lot.

Veen: Server stats are performance numbers. An analytic report just shows you how much the server is being utilized — nothing about the users. It’s frustrating. We’ve worked with Blogger, Technorati, CC, etc. We try and figure out, why are people loggin in? What makes a blog good? Is there a number? Or not? My wife gets 100 page views per month and she considers it wildly successful. Compare to Kottke.org. So what value are we looking at? I have 80% of my potential market. And that’s good.

Seeborg: Go home and baseline all your sites. Pageviews. Visitors per day. Get a sense of where you are.

Veen: A friend has a shopping blog. She has questions like, “should I post five things in the morning or post five things over the course of the day?” I don’t know, but it’s a fascinating question. Do you want to be the top, bold thing in people’s aggregators in the morning? It’s interesting because it’s such a limited domain. What you learn about blogs will not apply to e-commerce or a search engine.

Seeborg: Genre matters.

Veen: Let’s go back to talking about following the money. [Chart on screen: “Why consider research?”] I got this from Aaron Murchison associates. They got it from everyone else. Chart over time. As you go along through the project, the number of design options decrease over time. Your first brainstorming meeting, you’ll cut down half the possibilities. You keep doing that until launch, when you have one thing. So the more research you do up front, the faster those options go away (which is good). And the cost of going back afterwards and changing the design is much higher. It’s cheaper to argue over stickie notes. That’s when the alternatives should be eliminated. Without reserach, you’ll make expensive mistakes.

Seeborg: I’m a big fan of Survey Monkey. If you can sqeeze it in, it’s a great tool. Puts survey results in a nice little report. [Goes to the site.] Get anything you can up front.

Finck: I have a point about that. You may have seen the surveys on Digital Web. I didn’t have the best analytic tools. So I just posted a question every so often. It blew me away, the response. I still today get responses that are different from the simple survey data. So you have to continually do it and compare your data. It informs you where you need to be making changes on your site.

Veen: It was so easy to do the quantitative wrong, whereas it was so easy to do the qualitative surveys right.

Seeborg: Sometimes the quantitative info is sometimes trivial minutia. Get rid of the trivia. Be aware of why you’re doing this.

Finck: User needs and business needs. That’s the foundation of any web project. There’s a gap between the needs of the users and the needs of the business. I’m curious what your processes are for identifying and prioritizing these.

Veen: Well, identifying user needs. I ask them. But there’s a specific way to ask them. I’m more interested in how they do things in the real world. Generally I don’t tell people I’ll doing the web when I talk to them. “How do you buy the software?” Not: “How do you buy the software through the website?” […]

Finck: How do you associate value with all of that? You’ve figured out the needs and have a prototype or beta or something. How do you put a value to that information?

Veen: Sometimes easy, sometimes difficult. Never the same for any business. It’s a discovery process, right? Here’s a registration system for Blogger.com. [On screen.] So we analyised what people were doing on the site. Our goal for a user first coming the site was having them post to their own weblog, even if they didn’t yet have a blog. Goal #1: Stop the [user] bleeding. Goal #2: Lose the geekiness. Goal #3: First post ASAP. Because then they’ll come back. So here’s an entire project plan based on metrics. our tool? Excel. That’s it.

Seeborg: I want to take a side-road. Not all websites sell products. Some are service sites. That’s the road I’ve been heading down. […]

Veen: Amazon’s product pages are a disaster. They use AB testing. At any given time they’re doing 500 of these. 50% of users see one thing, and 50% use another. So they’ve been interating like this for ten years. And the result is a disaster! It’s ugly. No narrative going through the page at all. I don’t know how to rationalize this — we used to do AB testing at Hotbot all the time.

Seeborg: So you have to think about all of these sorts of information sources. So whatever amount of time you think you’ll spend: double it. It’s not just the data that’s valuable, it’s the recommendation of what to do next.

Finck: So I also wanted to talk about the tools at this panel. Is there a perfect tool for this kind of media? Will a product out there solve their problems? Is this even important.

Veen: No, it’s not important at all. It’s all about design process when determining process. But then I’m not doing click-path analysis through retail sites. And there are tools to help out with this.

Seeborg: Some people’s jobs hang in the balance based on conversion rates, and these people do need quantitative measures. They can’t screw up. [… She’s giving an example.] Once we found the technical problem on the site, our users sales skyrocketed once again. And to do this we needed to have the numbers after the fact. This wasn’t a planning thing. This was afterwards. And that’s where the quantitative stuff in helpful.

Veen: And I like that stuff because you get two weeks: What can we fix in two weeks. And that’s what the Blogger project was like. Quick returns. So, we know the problem. Let’s fix that. Move this around, poof — we’re up three points. So the team has something to feel good about and can show real progress. “Give us some money.”

Seeborg: Huge credibility-builder. It’ll impress the CFO and finance people.

Veen: One other note about eye-tracking testing. Hotwired was acquired by Lycos. And they bought one of those machines. Fascinating! It didn’t really solve the major problems, so I still think that the simple solutions will solve 80% of the problems. The major problems. After that,, get the eye-tracker thingie. I feel the same way about web stats packages. They try to provide every possible tool and what that ends up being is a tool where you have to build everything for the site. I’m surprised that there aren’t really genre-based web traffic tools.

Seeborg: None of these tools will synthesize information. A person has to put it together and think it through.

Question: You’re talking about using tool and analysis. Is there a case in your experience where you’ve done this on a project and then your gut tells you to do something totally different? Something differrent from the case-studies and data?

Seeborg: Intuitive response.

Veen: I try to breed that by surrounding myself with the project all of the time. It’s hard for me to bill by the hour because I’m thinking about it while biking. If that happens, it means I’ve had a disconnect somewhere. It means more work, generally, on my part.

Seeborg: You’re building a system for people. And that intuitive connection is very important and I encourage you to follow that and build it into your process. It has value.

Veen: The numbers that I was showing you for Blogger were people who made it through the page without error. An easy number to track.

Question: How do you bill your questions, then?

Seeborg: When you live, eat, and breathe it.

Veen: I build it into my hourly rate.

Seeborg: Absolutely I’ve gotten to the point where people pay me to think and I build it into my estimate.

Veen: The other thing is the notion of research as an event. I don’t like that. I like research as a culture. Not something you have to do by next Friday. Usabiltiy testing can be really cheap. Just a digital camera.

Question: Do you always test the same few people or do you refresh the pool over time.

Veen: No, I don’t do that long-term testing. We do screen for people who have done this sort of research in the past. I don’t want people who are good at being research subjects. And I would not do user research if it were not for screeners. I work with a focus-group search firm in SF that help me find good and useful people. Your audience may be easier to come by. And is SF it costs about $200 per person. So doing eight people costs about $1600. That’s a rounding error for most organizations.

Seeborg: We were in Montrael listening to Jared Spool. He paid $100,000 on research. And he was doing research on how eCommerce works. And there was a question last night from Stu about RSS tracking.

Finck: There are tools that claim to do this, but like any other analytic tool it’s not exactly scientific. So it’s only partway there. I learn more by doing surveys and more traditional user analysis.

Veen: Does anyone track their RSS feeds? Not really. I’m interested in the balance and the diffferences in usage patterns. Bloglines kind of hosts your feed and gathers traffic information.

Finck: So we’re out of time. This is all online at my site. Thanks!

[Note: I might alter this transcript in the coming days to remove typos and clarify the content. I post it now just to make it available as soon as possible.]