Free Resources

November, 2007 – Who's Keeping Score? The Value of Usability Scorecards and Metrics

Dr. Susan Weinschenk: Hello I am Dr. Susan Weinschenk, Chief of Technical Staff at HFI. Welcome to this live webcast presented by HFI's Usability Broadcast Network. The title of today's webcast is "Who's keeping Score? The Value of Usability Scorecards and Metrics." I am very excited to have with me here today, my dear friend and colleague, Dr. Phil Goddard. Welcome Phil. Dr. Phil Goddard: Thank you. Dr. Susan Weinschenk: Phil has been travelling around the US, various locations and giving a talk of this version this talk and I am really excited to hear his talk today. Before we get started, I want to remind everyone that you can download a white paper from the HFI website on the topic and it's called "The Business of UX Metrics" and you can also at the website, view a complete schedule of upcoming webcasts and I'll put a little plug-in for our 2008 schedule will be out in a couple weeks and we have many, many exciting topics for 2008 as well. Don't forget that we are going to have a Q&A session at the end of Phil's presentation and you can submit questions all through the broadcast as we go along, using the link in the lower right-hand corner. So to get us started, I actually have a question for our audience. Have you ever been in a situation where you've had to present the results of a heuristic evaluation? You have evaluated a website or a software application for usability issues and you present – there you are at the meeting and you are presenting the results and you have in the audience some mid-level managers, some high-level managers, you get to the very end and they say, "Okay, did we pass? Did we fail? What's our grade?" And you go, "Well I don't have a grade. I have this nice list of usability improvements but I don't have a grade" and they want a grade. So that's what we are going to be talking about here today. Dr. Phil Goddard: Awesome, you did so well. (Laughter) Dr. Susan Weinschenk: Did I pass or did I fail? (Laughs). Dr. Phil Goddard: So thank you Susan, you are my favorite Chief. I'm so happy to be here and I feel very lucky. This is a really fun topic – Metrics, it's very concrete and I'd like to actually tell it as a story because in fact, it has evolved over time and so there are three threads to this talk. One is the evolution of the expert review process which is a simple assessment framework that we have been using for years. Some people call it heuristic evaluation, some call it a cognitive walk through, I think we have a different form where we combine all these different things, but we do quite a bit of them and as we evolve the expert review, surprisingly the scorecard has emerged from that assessment. You know we have usability testing and that's got metrics around it, we have web analytics and that has metrics around it. The one thing that hasn't had metrics around it is the expert review which tends to be very qualitative in nature. So in our attempts to quantify what good design is we've actually sort of practiced with these scorecards and so the story unfolds by virtue of how these scorecards have impacted the people that communicate - our deliverables rep. The third and sort of exciting outcome of this is the opportunity to think in a new way about assessments and using the scorecard as a way of integrating all the assessment streams and when you start to think this way you sort of, the realization dawns on you that we could have an integrated metrics framework and so we are going to leave you I think, with a very powerful opportunity. A new way of thinking about assessments as an integrated approach. So Susan, we do a lot of expert reviews. I would say that from my part of the world, we probably do twenty or thirty a year, as a company how many do you think we can do? Dr. Susan Weinschenk: Oh yeah, I mean you probably do twenty or thirty with your team out in the West Coast and we have got teams all over the world. I mean, I'm guessing we do seventy or eighty a year or more a year. Dr. Phil Goddard: So when you do that many expert reviews, you look for ways to structure your feedback and effectively what people are asking you when they hand you their designs is (and we laugh about this internally) is – "Is our baby ugly?" you know... Dr. Susan Weinschenk: Right. (Laughs) Dr. Phil Goddard: And here we have a cute little baby but often we have the design work (Laughs) Dr. Susan Weinschenk: It doesn't look that way. Dr. Phil Goddard: It's not very pretty and so when we do these critiques, we have to be diplomatic in a way that we present our thinking. We have to have a compelling way in presenting these results and we have to have an approach that is defensible, we can defend what good design is. So over the years, I think we have tried different things, different frameworks in presenting what's good design? What's a good design and what's a bad design? In the past life at HFI, I was Director of Training and in our training programs obviously, we were teaching what good design is and we would come up with different ways of organizing what good design is in our teaching materials and one of the more powerful constructs that emerged from that was what I affectionately call NCPI, Navigation, Content, Presentation and Interaction. It's just one of the frameworks that we have adopted over the years but when you look at design you have to break it down. You have to articulate the various aspects of the design in different ways because you can actually be very strong in navigation framework but very weak in your IA. I think of navigation as the roadway systems in a site application and the information architecture as the signage along the way. So you can have really good roadways and really bad signage and have a really bad design, right? So when it comes back to the baby essentially you can say, "Look the baby is quite cute but you know it bites." (Laughter). Dr. Phil Goddard: You know you have to be able to analyze behavior and separate that. Dr. Susan Weinschenk: So sort of parse it out? Dr. Phil Goddard: Right, so over the years the bottom line here is we've created these compelling decks of recommendations and breaking down these designs under these categories. And so here is an example of a deck that I just happened to pull from one of my research projects and here you see a nice slide that calls out the different navigation problems. In this case, it is a B2B site from mortgage brokers and we are looking at the major link groups and recognizing that they are organizing consistently as you navigate across the site. So navigation problems are all grouped together by issue. Then we have – did I skip one? Let me see. Then we have presentation issues, the actual visual presentation of the site. Dr. Susan Weinschenk: Now these that you are showing us here, before a scorecard concept... Dr. Phil Goddard: Exactly. Dr. Susan Weinschenk: You are saying this is what we usually do, right? Dr. Phil Goddard: Yes, so in this example we are talking about density verses clutter. Actually frequent users like dense screens but density can move into a clutter if you don't follow good alignment principles and so forth. We are talking about the principles of presentation here. Content, in this case, the site was not using the user-centered taxonomy or terminology. So again, an example of a slide that's articulating problems in the content design area. And then finally interaction and these are all just to show you samples of what we would do in a formal expert review of an interaction problem. In this case, the forms were loading with misleading defaults and if people didn't change those defaults, it led them into error situations and it makes no sense, from an interaction perspective. So wonderful, we have these 100 slides...150 slides. Dr. Susan Weinschenk: Right with all this information. Dr. Phil Goddard: We've organized it around these principles. Dr. Susan Weinschenk: What's wrong with that? Dr. Phil Goddard: Here's the question, the nagging question. After you do many of these reviews and you have an impact on the team and on the designs and they go away and then act on their designs, you start to ask yourself, "What happened to those brilliant reports that we created?" you know, "What corporate memory is there of that review?" and the same thing you could say for usability testing right, you do usability tests, you create a compelling report, it's acted upon... Dr. Susan Weinschenk: You're saying that even if they act upon it, which is good and they have taken your recommendations - then it is kind of gone. Dr. Phil Goddard: It's gone. Dr. Susan Weinschenk: It kind of disappears. Dr. Phil Goddard: How many times have we gone back into an organization a year later and it may be a new team? Dr. Susan Weinschenk: They don't even know if the work was even done. Dr. Phil Goddard: They don't even know anything, right? Dr. Susan Weinschenk: Right. Dr. Phil Goddard: So this is sort of a nagging question, so enter the scorecard. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: Okay and this is what we started to do about five or six or seven years ago, actually in our training program, we were trying to teach people good design and so we were using exercises around these scorecards and we'd have them sit in pairs and they'd compare Home Depot and Lowe's or Federal Express and UPS or you know, a couple of e-commerce sites and we'd show them how to analyze what a good design is and then score them. Well we started to put these on the front of our decks-our expert review decks -and we created a simple 100-point scorecard and as you see here, it's articulated by the different navigation content presentation areas and that's broken down and articulated so you could say for example, in this case the color, the use of color on this site is not useful for grabbing attention. It's not being used well for grouping and it's not even used well for highlighting. So a team could look to this report card and say, "Well, my presentation score is 6 out of 20 and looking into it, looks like I'm doing some pretty good things here with layout with my use of color and my use of grapics, graphics (Laughs) is poor, type use is good" right? So it became, all of a sudden, something that everybody looked at and what was happening when we started to put these scorecards is, all of a sudden, the business people were getting it. They were saying, "35 out of a 100? We're in trouble." Dr. Susan Weinschenk: So you're saying that when you just presented those results even though it was categorized the usability people got it, but the business people, sometimes it didn't have the impact but having the score changes that? Dr. Phil Goddard: The score made a huge difference and I think that's why we go with -well actually, let me get there. Let me tell you how this is done in practice in the next few slides. We would have of course, an expert review. We'd have two independent expert reviewers review the site and John would do it and Sally would do it and they'd both use the same scorecard and we'd set up some scenarios that we used to have experienced in this site in various ways and they'd fill up this general scorecard and then we'd present them to the client as actually independent scores and then weight it out. Dr. Susan Weinschenk: Now why do you use more than one person? Dr. Phil Goddard: It's just best practice in terms of expert review. It's a sampling issue actually you know, you can use even more than two expert reviewers but you will get good feedback from two experts reviewers. It becomes a pragmatic decision, but one is obviously... Dr. Susan Weinschenk: One is better than nothing, Dr. Phil Goddard: One is better than nothing... Dr. Susan Weinschenk: Right. Two is better than one, okay. Dr. Phil Goddard: Right. So in this case John scored a 35 and Sally scored a 38 and obviously that's going to be kind of bad you know, because we are talking out of a 100 here. So they present this back to the client and say, and I've been in meetings with the scorecard before where the VPs of e-commerce are sitting in a meeting and they look at the scores and they go, "Good Lord! Is that a D+ or is that, is that a F?" you know, actually they're kind of chuckling about it and actually it's an amazing thing because you have just delivered the "baby is quite ugly"... (Laughter) Dr. Phil Goddard: And they are taking this feedback and they are processing it very quickly and they get it. Dr. Susan Weinschenk: Well, I know that one time when I was in a situation like this and had the VP of the firm there and he said, "Well now I know that this is not just nice to know and I'll let my team go off and fix things a little. I know that I have to mobilize you know, several teams to fix this, this is serious." So that's what he wanted to know was, how serious is this and you know, when the score was a 30 something out of a 100, it was serious. Dr. Phil Goddard: That's it and we will talk about the scale itself... Dr. Susan Weinschenk: Right. Dr. Phil Goddard: And what the meaning of the scale is... Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: But it's a magnitude, it's you know, "which way are we trending?" And often they come to us as they don't know anything and they don't know where the problems are. So the scorecard immediately becomes an impact on that. It has an immediate impact on the behavior because it's actionable. We'll sit around with the team and the graphic design team and the IAs and the Marketing groups and say, "Okay, let's talk about how we are going to divide and conquer and fix this" right? The third thing here is that very interestingly, when it becomes a number or a grade, it becomes a quantitative benchmark that can be tracked. So now you are out of the qualitative world, you are into the quantitative world and you're able to say, "Okay, here's where we are right now. Let's look at where we are going to be in 3 months, in 6 months, 9 months, and 12 months and let's do the same process again." So now it becomes a benchmark. Dr. Susan Weinschenk: That's huge. Dr. Phil Goddard: That's huge. The other thing that is part of that is that now you are in an arena where you can do it in a competitive framework, a competitive or comparative environment. So you can look at your site against your competition or best practice in the industry that you want to strive to be like in a comparative situation. Dr. Susan Weinschenk: Or as in the benchmark you know, the current one to the new version? Dr. Phil Goddard: Yes within a comparative design, absolutely. So the great score has been used a lot by us now in this 100-point scale and recognize that this was done, we started this about 5 years ago and we've started to do this as a part of all reviews and we've done many of them. So we have a lot of experience using this channel, the scorecard and I'll elaborate on that more as we go. Here's an example of what you can do as a tracking opportunity. Let's say that your initial benchmark is done on the first part of the year on the legacy site and you scored a 44. You did a tactical redesign. Maybe you found a piece of the site where the pinging was high maybe it was in your product area or it was in your checkout area whatever you know, maybe it was in your application flow and you decided, let's fix that piece. So you do a focus piece, a focus group about redesign and you release that. Now you can benchmark that part of the site and now you've shown a score of 68 and application drop-off decreases coming in at a 60% less, right? So then you do a major redesign in the year January 07. You do a benchmarking then. You've got an 87. You're comparing that or correlating that with your inquirers, the transactioners and you find that you've got an increase of people who simply inquire to transact, obviously banking, this is obviously a banking scenario but you get the idea it becomes something, that it's a program rather than just a one-off point review. So over time, we've recognized that there are different sites that have different business goals and we've developed these general scorecards for large scale information sites and corporate sites. We've developed them for e-commerce sites. Dr. Susan Weinschenk: So you are saying you have a different flavor of the scorecard if it's an e-commerce site, the scorecard is slightly different than if it's a web application that kind of thing? Dr. Phil Goddard: Exactly, exactly. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: You know we've got web applications that are served up in browser, productivity tools and... Dr. Susan Weinschenk: So some of the things that you'll be evaluating are a little bit different. Dr. Phil Goddard: Yeah. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: Software applications in a window, now you have a menu in your system and large scale information that's about content organization. I mean, we did a library of Congress, I mean that's a huge – Dr. Susan Weinschenk: But it's different... Dr. Phil Goddard: Where as e-commerce is about encouraging the sales. Dr. Susan Weinschenk: Right, right. Dr. Phil Goddard: And financial services. So over the years we have developed these general scorecards that's target-specific in the industry. Let me reflect then back on what we said a little bit because there's running threads going through this and I just want to summarize that the expert review as a tool in the UX teams toolbox of assessments now becomes a quantifiable tool, right? You've got usability testing going obviously but this is not a replacement to that. You've got web analytics happening, page-level activity; you've got survey data on customer service, customer satisfaction. What you haven't had the ability and this is kind of cool is that in design time you can have reviewers doing a metric approach to your designs and add that to your metric streams. Usability tests tend to be phases a little bit more, a little bit more longer term in gauging them when you do major milestones on them. Web analytics are coming in everyday. What you don't really have is something that is coming in, in design time. By that I mean that your team is doing pages, they are designing pages or collections of pages, and they want feedback, fast. Dr. Susan Weinschenk: Right there and as they design. Dr. Phil Goddard: And they are going to want to quantify. So this becomes another metric tool. Dr. Susan Weinschenk: So this is not, you're saying you wouldn't just do this once and then site goes out and then you know 6 months later you might do it again? You're saying you could actually use this during the design, interface design process? Dr. Phil Goddard: Absolutely. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: And I will qualify that, I'll talk about that more. Okay, so over the years, we've focused on a score carding process and we've come up with new ways and better ways to do this and I'm just going to share a couple of newer things we've introduced in the expert review process. One of the things that we realized is the expert reviewers tend to be scenario-based when they do the reviews. They actually - we define tasks upfront, we define who the users are, and quick persona workshop and they go through the site as a flow... Dr. Susan Weinschenk: So they're not just looking at one screen and saying, "Oh this screen is cluttered" or "There is poor use of color" or "The terminology is hard to use." But actually using a user scenario typical – here's what this persona, this group of users would do and evaluating a whole flow of screens. Dr. Phil Goddard: Right. Dr. Susan Weinschenk: In the scorecard? Dr. Phil Goddard: Because then you get the context of someone trying to achieve a goal of interaction and their experience as they go through. So, we've now sort of re-engineered the scorecard to be a scenario-based scorecard. Something that a reviewer can do on a scenario by scenario basis and every time they go through this scenario they fill out this scorecard. That's a powerful approach because it's actually a much more user-centered approach. You can have your reviewers evaluate a presentation of the system, you can have them evaluate... Dr. Susan Weinschenk: This thing that we talked about previously? Dr. Phil Goddard: Exactly. Dr. Susan Weinschenk: Of navigation? Dr. Phil Goddard: But this becomes more user-centered. Dr. Susan Weinschenk: Can you give an example of something that's on some of the scenarios of scorecard that might you know not be on just a generic scorecard? Dr. Phil Goddard: Yeah, in fact that's actually the next slide. Dr. Susan Weinschenk: I didn't know that. Dr. Phil Goddard: It's a perfect question because at the same time we've kept all the things at the best practice scorecard and put them in the scenario. Dr. Susan Weinschenk: Okay, so there are all the usual things that are there. Dr. Phil Goddard: They are all there. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So...but we've recognized that there's the usability of a site and there's also the usefulness of the site or the power. The experience that you give the user in terms of meaningful content and interaction. There's also the emotional experience and the persuasion architecture of the site so we are recognizing now and HFI is working to have this deeper dimensions of human – or rather the user experience. So usability is actually one slice of the review. It's the amount of effort that user is taking to browse, search or find... Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: To comprehend, consume and interact with content and that covers our navigation content too. Dr. Susan Weinschenk: So that's kind of our typical usability issues spelt. Dr. Phil Goddard: Right but then the question is power. Do, when I get to my destination, is the content there that answers my question that provides the value that I was looking for? Or is the interaction there to accomplish the work that I came to do? Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So it's about content and features and power and obviously you can have high power and high effort and therefore you have a bad usability design, you have to focus on your usability. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: But a lot of the wars in the past have been around power and we've been there to say, "Hey it's not all about features and functions it's all about you know, getting there too." Dr. Susan Weinschenk: Right. Dr. Phil Goddard: If you can find it you know it doesn't exist. Dr. Susan Weinschenk: It's all there. Dr. Phil Goddard: Right and then recently we've introduced ways of looking at capturing the emotional experience of a user and also evaluating the persuasion architecture of a site. So this is about finding out upfront what the brand goals are for the site and understanding what the persuasion architecture is and actually measuring whether it is reaching its goal or not. Dr. Susan Weinschenk: So that's one of the new areas that as you know, I'm very excited about working with Mona Patel on and Mona likes to talk about the "can do" versus "will do", yeah. Dr. Phil Goddard: Right, so this, if you do this all right together, you have a compelling user experience. So we just added these dimensions and this is where we get the experiences of "This product is desirable, I want to have it and I'm going to stick around for a long time." Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: Okay? So these are the two things that have come up in the last year or so into our metrics. Dr. Susan Weinschenk: So you've built some of these into the scorecards? Dr. Phil Goddard: Yes and so this is an example of a roll-up for example, of someone doing a scenario review using the scorecard. You see there's five dimensions of the user experience browsing, searching and finding is all about movement to content , comprehension and interaction is about when you are at the content, is it easy to consume and interact with? Again usability dimensions and these have sub-dimensions or sub... Dr. Susan Weinschenk: So you say this is a roll-up and that actually there will be another part of the scorecard that will have details on it? Dr. Phil Goddard: Exactly. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: And then empowering and delivering is all about power usefulness. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: Experience of brand and responding to value, the persuasion architecture. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So this is an example of the way that we present our usability and expert review findings now. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: The next slide basically just shows an example of how we approach this in a very simplistic way as we spend a little time upfront of finding who the personas are, the target users of this site and this is an over elaboration but it has to have the critical kind of background on the user that differentiates them from another target audience. Dr. Susan Weinschenk: So, and personas or something that you know, we've been doing in usability for many years, is there anything different about defining personas before you do scorecards or is it you know basically the same as we do when we use personas for any other type of usability work besides scorecards? Dr. Phil Goddard: I think it is the same, the only thing is, my belief is that a persona is only useful if you are able to define what the motivations are that the person has and what the goals are that they are trying to accomplish in here, in your site or product. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So it has to be actionable. Dr. Susan Weinschenk: It has to be actionable in order to do scorecards but you are saying that it should be that no matter how you are using the personas in scorecards. Dr. Phil Goddard: Right. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So in this case I'm just giving you example of Samantha Jones, is a young woman who is excited about the convenience of shopping online as long as it's easy, fast and informative. In this case we created this, this is actually not a project that we did, but for purposes of showing you, we created a scenario where she's been given some goldfish... (Laughter) Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: As a gift and she wants to buy a goldfish bowl online... Dr. Susan Weinschenk: That would be good if you have a goldfish. Dr. Phil Goddard: Right and she is going to go to a pet supplies store and she wants it delivered to her apartment. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So we defined the context she's in... Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: The goldfish she has... Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: And the expectation she comes to with a website and so basically what the reviewer does is take that scenario... Dr. Susan Weinschenk: And uses that... Dr. Phil Goddard: Scores it... Dr. Susan Weinschenk: Looks at the website... Dr. Phil Goddard: Overall and then presents it. In this case, the user - reviewer gave this scenario a score of 46.25 of a 100. Dr. Susan Weinschenk: And this is the roll-up? Dr. Phil Goddard: This is the roll-up. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So looking at this, you can see that the browse, search and find experience was quite low. 13.75 out of 40. Dr. Susan Weinschenk: Okay, that's not good. Dr. Phil Goddard: 17.5 out of 30 Dr. Susan Weinschenk: That's not good either. Dr. Phil Goddard: So this is the way again, of focusing on your priorities. Dr. Susan Weinschenk: So if it had turned out you know, that one of these was relatively high, like if the browsing, searching and finding was between a 30 or 32 out of 40, then I could say, "Well, there's a little tweaking I need to do there but actually I should really concentrate on the others." Dr. Phil Goddard: You got, you got it. Now when you do this, this next score card shows you even a higher level roll-up and it's by the personas across the top and I'm switching back to our financial services environment, potential customer, inquirer and a transactor. Dr. Susan Weinschenk: Oh, so this is a different kind of roll-up? Dr. Phil Goddard: This is the power of doing a persona definition with a collection of scenarios that you're reviewing. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: And then rolling them up by UX dimension, you see down the left-hand side... Dr. Susan Weinschenk: That's really interesting, yeah. Dr. Phil Goddard: Across the top you can see by scenario. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So this snapshot right here you can say, "Hey you know what? I'm doing very well with my inquirers and I'm not doing very well with my transactors or actually in this case, I'm not doing very well with my potential customers." Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: The things that are flagged in orange here... Dr. Susan Weinschenk: Right. Dr. Phil Goddard: They're below the levels that you set. Dr. Susan Weinschenk: Alright. Dr. Phil Goddard: And these numbers are just there to show you that I mean, this is the kind of thing you can do when you take this metrics approach. So it's a very powerful roll-up going by scenario and by UX dimension. So there are issues, obviously. Dr. Susan Weinschenk: This isn't like "easy, improved", right? Dr. Phil Goddard: Well, sure, yeah and we have to do this with both eyes open. Alright? Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So I want to get those out to everyone basically the role of expertise is the first one and that's to say that this metrics scorecard in the hands of someone who is not very knowledgeable, not very skilled about what good design is or what the principles are that go behind it could be detrimental to it. So a metric is only as good as the power of the... Dr. Susan Weinschenk: Skills and knowledge of the person who used it. Dr. Phil Goddard: Of the person who used it, right, yeah. Inter-rater reliability has to do with making sure that you have a scorecard framework that's internally reliable and that people will appreciate the metrics in the same way and review the site using a common understanding of what those metrics are and ultimately their ratings will not reflect a bias in their interpretation of what that metric really is. Dr. Susan Weinschenk: What do you do if you get – you know you have two people like in the example you gave early on you know, where someone scored a 35 and the other scored a 38? That's pretty close and you could find out what that discrepancy is but what is someone scored a 35 and someone scored a 72? You know what would you... Dr. Phil Goddard: That would be a problem. Dr. Susan Weinschenk: (Laughs) Okay. Dr. Phil Goddard: And we don't have that actually. We do have times when one score on the score card will deviate significantly and actually we use that as an opportunity to discuss with the client. Dr. Susan Weinschenk: To discuss why – why it's this part but you're saying in our practice, we're not finding this discrepancy? Dr. Phil Goddard: I thought I'd have a lot more inter-rater reliability problems than I'm having. Dr. Susan Weinschenk: So the expert reviewers are maybe 5 or maybe 5 or 8 points apart. Dr. Susan Weinschenk: So can we our internal training program is good at HFI? (Laughs) Dr. Phil Goddard: It could be, it could be. Dr. Susan Weinschenk: But we do have people with a similar level of experience and training in our company doing this. Dr. Phil Goddard: Well, we have to spend time with the people who haven't done it before to make sure that they understand all the metrics. Dr. Susan Weinschenk: Okay, so that's important too. Make sure that whoever is using the score card really understands each item on the score card and what's behind that because I would guess that if you didn't understand it, you might give a different score just because you were interpreting that item on the score card differently than others would have. Dr. Phil Goddard: Exactly. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: But you know what? It's making lemonade out of lemons actually. I mean it's actually turned out to be a good thing to communicate that with the client when John and Sally deviate by 15 points on "sense of place" you know that's a metric for navigation, and there's a good sense of place and if they're disagreeing on that, then you say, "Sally why? Why is you sense of place metric so low?" and she says, "Over here, I really felt lost." Oh, you know what? That's an interesting point. Let's focus on that. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: But anyway, it's all about what is the result of the metrics? Dr. Susan Weinschenk: But you're saying that this inter-rater reliability is something that you might want to track... Dr. Phil Goddard: Yes. Dr. Susan Weinschenk: As you were to implement score cards, if you were doing work organization, you would want to keep an eye on that and see if you had that reliability or not. Dr. Phil Goddard: Exactly. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: And I'm sure we'll get questions from the audience on that too. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: And I'd be happy to answer that. The third thing is people will say, "Well, what does a 60 mean? We got a 60." Dr. Susan Weinschenk: Is that good or bad? Dr. Phil Goddard: Any absolute value doesn't really mean much but think of it as a directional indicator. If you had a 40 before and now you have a 60 you know you're moving in the right direction. Dr. Susan Weinschenk: That's good, right? You're in the right direction. Interesting so sometimes a 60 might be good and sometimes also, a 60 might be bad. Dr. Phil Goddard: It's true and I think right now, obviously a 60 means, "Hey, you're not excelling. You're not excelling in your design work but you're not flunking either." Dr. Susan Weinschenk: So do people get 100? Dr. Phil Goddard: We've – I've never seen a review give a 100 yet. Dr. Susan Weinschenk: But of course that's because they're calling us to evaluate something that we didn't design, right? (Laughs) Dr. Phil Goddard: You know, some people would actually really enjoy a 100. Dr. Susan Weinschenk: Because if we designed it, it would be a 100. (Laughter) Dr. Phil Goddard: But I will say this that some scenarios score in the 90s. Dr. Susan Weinschenk: Really? Dr. Phil Goddard: And some scenarios, with the same user, will score and not reviewer – user, and by user you mean persona. Dr. Susan Weinschenk: Oh by persona... Dr. Phil Goddard: Will score in the 40s or 50s. Dr. Susan Weinschenk: Right. Dr. Phil Goddard: So the power is when you aggregate all this up and you look at the data, it is one thing to say "I'm getting 80" but can you drill down into it Dr. Susan Weinschenk: It's important to drill down... Dr. Phil Goddard: And say, "You know what? These three scenarios are scoring high but this one is very low." Dr. Susan Weinschenk: This one is low so it's important to drill down because an 80 overall sounds pretty good but if some of your most, you know - your most important user group actually was down much lower, you'd want to catch that. Dr. Phil Goddard: You're getting pulled down. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: And in fact, you can go to the extent to show standard deviations in these. Dr. Susan Weinschenk: Oh, I love that statistics stuff you know, because that's my background. Dr. Phil Goddard: Yeah, well if you've got a lot of standard deviation around one metric that means there is a lot of variance inside it and you can look into that. Dr. Susan Weinschenk: Right, okay. Dr. Phil Goddard: The scaly issue of scales... Dr. Susan Weinschenk: (Laughs) What is that? Dr. Phil Goddard: It's to say that, I mean I started off with these general score cards with different scales for different categories and you know, I would give presentation a scale of 1 to 3 and navigation a scale of 1 to 7... Dr. Susan Weinschenk: A scale of 1 to 5, okay. Dr. Phil Goddard: And over time, we realized that no, you have to standardize on these scales. Dr. Susan Weinschenk: It's confusing, yeah. Dr. Phil Goddard: And you also have to weight them... Dr. Susan Weinschenk: Yeah. Dr. Phil Goddard: In a way that's meaningful to you. Dr. Susan Weinschenk: Yeah. Dr. Phil Goddard: For years, we've been saying you know, navigation is 80% usability... Dr. Susan Weinschenk: Right. Dr. Phil Goddard: Well, that may be true for some environments... Dr. Susan Weinschenk: But not all. Dr. Phil Goddard: But it may not be true for others. Dr. Susan Weinschenk: So do you change the weighting depending on the client? Dr. Phil Goddard: The weights are free for you to set, alright. Dr. Susan Weinschenk: Okay. Then once you've set the weights, do you want to keep them for...so that you can do benchmarking going forward? That's probably good, right? Dr. Phil Goddard: Yeah. Dr. Susan Weinschenk: Set the weights then as you use it over the next year, you can reevaluate? Dr. Phil Goddard: Yeah but you can create versions of these score cards... Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: Like for example, the public side versus the secure side, you may scale up the brand experience... Dr. Susan Weinschenk: Right. Dr. Phil Goddard: On the public side and the persuasion architecture higher than on the secure side... Dr. Susan Weinschenk: Exactly. Dr. Phil Goddard: When you're in a productivity environment. Dr. Susan Weinschenk: Right. Dr. Phil Goddard: Experts aren't users, so I just want to make sure that everyone knows that we're not saying that this is a substitute for usability. Dr. Susan Weinschenk: Right. There's always a case for expert reviews, heuristic evaluations, and so that's always an issue. Dr. Phil Goddard: And the examples I have on this slide, I don't know if you guys saw the movie "The Comebacks" but it rated an "F" and it definitely deserved an "F". Dr. Susan Weinschenk: (Laughs) Dr. Phil Goddard: But you know, here we have a great scale A to F and stock rating scales rating stocks from 1 to 10, in a consumer report, we have a scale of 1 to 10 so I think we've agreed that you know, people are used to a grade scale which can also be easily converted to a percentile based scale – 1 to 100... Dr. Susan Weinschenk: You mean A, B, C, yeah. Dr. Phil Goddard: And also, a scale that's either on a scale of 1 to 10 or 1 to 100 seems the thing, the way that we've sort of the way we've been indoctrinated over our great experience as kids. Dr. Susan Weinschenk: Kind of like the metrics system, I guess. Dr. Phil Goddard: So now I'm going to shift quickly because I'm not sure how we're doing on time. Dr. Susan Weinschenk: We have maybe 5 minutes. Dr. Phil Goddard: 5 minutes? Dr. Susan Weinschenk: Yeah. Dr. Phil Goddard: Wow, okay, okay. Dr. Susan Weinschenk: (Laughs) Dr. Phil Goddard: There is another way to approach metrics. Dr. Susan Weinschenk: 5 minutes before the questions and answer period, I should say. Dr. Phil Goddard: Okay, alright, that's still only 5 minutes. Dr. Susan Weinschenk: Right. (Laughs) Dr. Phil Goddard: So another way to approach this is the page view. So far I have been talking about analyzing a site as a sequence... Dr. Susan Weinschenk: Yeah. Dr. Phil Goddard: Taking a persona view... Dr. Susan Weinschenk: Right. Dr. Phil Goddard: Having a scenario view. We can also you know, there are certain types of sites you know, like an e-commerce site, where we know what a good product page should be. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: It has a purpose. Dr. Susan Weinschenk: Right. Dr. Phil Goddard: You should design a product page in this way to accomplish the goals the users have and your business goals. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So what we have here is an example of a Meta-architecture of a site where we've defined the tiers of the architecture here. Dr. Susan Weinschenk: The different kinds of pages you'd use. Dr. Phil Goddard: Each of these pages can have an individual score card. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So now you have another way of scoring. Dr. Susan Weinschenk: So you might have a product page score card and a checkout page score card? Dr. Phil Goddard: Exactly and that's what I'm showing on the next slide. Here's an example of – and we've done this for an e-commerce companies and we're doing it for banking environments as well. We've got a product page score card here and a search results page score card. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So now you have the ability for the business to actually send you a page and say, "Evaluate this page for best practice in terms of what the purpose of the page is." Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: From the user perspective and from the business perspective. So we've integrated those and now you get a chance to do a page-level score cards in a competitive environment. Dr. Susan Weinschenk: Oh, okay. Dr. Phil Goddard: So this is an example of showing the product page review of 3 online computer manufacturers. Dr. Susan Weinschenk: So the 1, 2 and 3 here would be... Dr. Phil Goddard: The competitors. Competitor 1, 2 and 3. Dr. Susan Weinschenk: The 3 different sites. Dr. Phil Goddard: Exactly. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: And in terms of the way they present their product page, you'll see here that site 3 in this case, is scoring the highest and we can articulate now on why. So page-level and now you have the ability to track by page over time... Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: For best practice. Dr. Susan Weinschenk: Okay, okay. Dr. Phil Goddard: So you can do changes to your home page, you can do changes to your index pages, changes to your checkout pages – without affecting any other pages... Dr. Susan Weinschenk: Oh, yeah. Dr. Phil Goddard: You can actually track the reviews of those scores over time. Dr. Susan Weinschenk: Oh, interesting, you can track that one, okay. Dr. Phil Goddard: And companies that we're working for with a metrics program are doing this kind of work now. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: Alright, so to this point what I've covered is, you have metric indicators that you can view at a scenario level, a persona level, a design level and a page level. Dr. Susan Weinschenk: Right. Dr. Phil Goddard: They're very powerful architecture checks... Dr. Susan Weinschenk: Right. Dr. Phil Goddard: All coming from well, best practice reviews but the exciting thing Susan, for me and this is where we end is to say, "Okay we're score carding expert reviews. Why not develop a common score card that we can use for all of our assessment practices? Why not create a score card that when you're done with your usability test, you can translate that data into an overall score card?" Dr. Susan Weinschenk: So you mean, not just a score card for heuristic evaluation? Dr. Phil Goddard: Exactly. A common score card. Dr. Susan Weinschenk: The same score card would be used for heuristic evaluation as would be used for usability testing? Dr. Phil Goddard: Exactly. See now what you have to do is to set up an assessment protocol that expert reviewers can follow and that your usability testers can follow and in fact, they can both use the same card-score card, to score the success of that user experience one from a persona perspective for the experts and another, just watching the users accomplish their tasks. So now what we have is the ability to put expert reviews and usability data side by side. The other thing is if you're doing a page-level metrics; why not correlate that with your web analytics metrics? So now you can tie-in your score card and your best practice at a page-level to your web analytics. What's happening at a page-level? Why not create your surveys in such a way that you can capture your survey data expressly on your score card and put it side by side to your expert review, usability tests and web analytic data. And if you do this in a consistent way, you can show a view on a channel for example, your corporate site or your e-commerce site, your call center, your IVRs, that all how on a common reporting framework. Dr. Susan Weinschenk: You're saying we can integrate the site together? Dr. Phil Goddard: Now what happens when you think that way is all of a sudden start to think of this capability of the digital dash board and this is the vision that I wanted to end today with is if you did this right, you've structured your assessments in an organized way, you could create an overall digital dash board by channel, in this case the online channel, that shows how are we doing by customer and that customer data could be derived from your expert reviews, from the usability testing, from the surveys. Dr. Susan Weinschenk: From all these pieces. Dr. Phil Goddard: How are we doing by scenario? What tasks are we supporting well and not well? What dimensions of design are we struggling with? Okay? And then over here on the right, what pages are we weak on? What pages are we appearing to get really good results and what pages don't? Dr. Susan Weinschenk: I would think this would be very powerful if, you know in the cases where the website is changing, not because of user experience issues but you know there's new technology and so – or new content or different groups have changed the website and if you were tracking this constantly, you could catch a problem right away and say, "Oh, wait a minute. They made some changes and actually we have a little problem here as a result of that update of the website" right? Dr. Phil Goddard: Yeah absolutely. And yet, we have companies who are asking us to build these metrics programs and they have – they're global. And so they will come to us and say, "Can you do a study for us in Germany?" and "Can you do a study for us in Japan?" and we're saying, "Yes, absolutely" but what we should actually be doing is setting up a regular assessment program so you can do regular studies on your Germany site, regular studies on your Japan site... Dr. Susan Weinschenk: Right. Dr. Phil Goddard: And put that right next to your U.S. site. Dr. Susan Weinschenk: And kind of compare...? Dr. Phil Goddard: And show them in a dash board. It's empowering. So what I think is exciting about this if we go back one slide – what you need to do to get there is clearly define who your target audience is through your persona work. Standardize on the way you want to characterize the most important scenarios or goals that you're trying to get your customers to achieve successfully. That's the foundational work. And then build the score carding process that you can use consistently across these assessment streams. From there, it's just a matter of how do I want to – how do I want to present this? And that's where the excitement of doing the dash board and if you think of it, there are just a number of ways to present that data but nowhere have I seen this kind of framework. Dr. Susan Weinschenk: Yet? Dr. Phil Goddard: Yet. This is an entry point for a business person. This is an entry point for a UX team member. This is an entry point for a developer. This is an entry point for a designer. This is where we should all be going to see what do we need to do? We only have limited resources. Where should we focus our attention? Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So it's about empowering ourselves with knowledge. Dr. Susan Weinschenk: And you came up with the idea... Dr. Phil Goddard: All this came out of the work we started just by doing expert reviews and thinking about what good design is and then the score cards unfolded out of there and now this vision. Dr. Susan Weinschenk: So it's an evolution as you said at the beginning. Dr. Phil Goddard: It is. Dr. Susan Weinschenk: Okay, you're ready...? Dr. Phil Goddard: So I'm ready. That was... Dr. Susan Weinschenk: Ready for Q&A? Dr. Phil Goddard: Yes, that's it. Dr. Susan Weinschenk: Alright, wonderful. Thanks Phil. So as I mentioned before, we're going to have a question and answer time. We've got some questions that have come in already and we'll just go to those first but as we're continuing the talk, please remember that you can still submit questions with that button on the corner of your screens. So let's see what people want to know. QUESTIONS & ANSWERS:- Dr. Susan Weinschenk: Our first question is, is it possible to purchase or access these score cards? Dr. Phil Goddard: Yes, absolutely. There's one in the back of the white paper. Dr. Susan Weinschenk: So you've got one example on the white paper and as we mentioned before, you can download the white paper from the website if you haven't already and there's one example there. Dr. Phil Goddard: Yeah and that's actually not a bad score card. It's a general score card for a website. Dr. Susan Weinschenk: What if they want more... Dr. Phil Goddard: I would suggest that they purchase an expert review. That would be my recommendation. Dr. Susan Weinschenk: Okay, so if they purchased an expert review from HFI then through the process of that we're going to help them customize the score card for them? Dr. Phil Goddard: Understand what this whole process is. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: I think what you want to do is you really want to be trained on how to use this score card. Dr. Susan Weinschenk: Because we really don't want to like, just sell a score card with you know, "Here's a score card, go use it." It would be best for them to go through the process with us so they really know how to use it moving forward. Dr. Phil Goddard: That's a really, really good point. Dr. Susan Weinschenk: Okay. Do the – you kind of touched on this a moment ago when you were talking about Germany and Japan - but it says do the grade and number scales that you mentioned work globally? I think this is more is the 100-point scale – is that global? Dr. Phil Goddard: You know, what's perfect about this is that the important thing is to allow a local person in the local language in the local culture do the expert review. That is the uniqueness. Right? Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: The score card is the standard. If we've designed it well, which represents the customer experience in a complete way, then what you're going to find out when you have a reviewer in Germany do it and a reviewer in Japan do it is whatever nuances they are experiencing trying to accomplish their goals are going to come out in these score cards and you're going to see that in the use of a common score card. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: But the important thing there is that the person who does it be in the culture right there. Dr. Susan Weinschenk: Be in the culture, okay. Alright, good point. Here's another one. Where do I start? I'm a staff of one in an organization that doesn't understand usability. It doesn't look as though they don't appreciate the concept; it's just that it's never been done here before. What should step 1 be for the beginning? Dr. Phil Goddard: Well, I would suggest that we get together and talk about presenting this idea to the people that can help make a decision to commit to something to demonstrate something of value. It sounds like what you're looking for is you need to build some awareness and demonstrate value around this process. What do you think, Susan? I would suggest they can maybe have a small review of some sort. Dr. Susan Weinschenk: Well you know, I think the idea of – because they're saying – you know it's interesting, a lot of times score cards really help if – if the people in your company don't appreciate the idea of usability. In this case, the person is saying, "Oh, no they appreciate; they just don't understand what it means." And I think that they score card is really useful there because even perhaps step 1 maybe, you know they could look and try out one of the sample ones in the back because it's got each item delineated on the score card. Dr. Phil Goddard: Right. Dr. Susan Weinschenk: I think it's a wonderful training tool for you to educate the people in the company that don't understand usability. You can say, "Look, I've scored you - you know, this many out of this many on navigation and here's what that means and here's why I did that." It actually is a great learning tool. Dr. Phil Goddard: It is. It's a communication vehicle. Dr. Susan Weinschenk: It provides education. Right. So step 1 is maybe for them to call us and have us help walk them through or try one out on their own. Do you have specific training on this score card method? Or is this training included for this in one of your courses? This is a very interesting question, I think because you mentioned that when you started that you know, previous life, you were training and who did you hand that over to, Phil? Dr. Phil Goddard: Dr. Susan Weinschenk. Dr. Susan Weinschenk: (Laughs) Me! So I'm in charge of training but I'm really lucky now because I also have a wonderful training director who works for me now, Michael Calvilla. But I don't know, I mean as you said it, it actually evolved from our class when we had-so we do have people in our class, "The Science and Art of Effective Web and Application Design". There's a score card that people fill out there so there's some you know, a little bit of training in it in the design class but we don't go into all of these nuances here and right now, I guess the training would be through a consulting engagement but it's an interesting question about whether we should perhaps think about... Dr. Phil Goddard: Well you know, it's funny because I've met so many people who have done a certification program... Dr. Susan Weinschenk: Yeah. Dr. Phil Goddard: And they would be the first person who I would say, is in a good position to do score carding. Dr. Susan Weinschenk: As they've got the basics of...yeah. Dr. Phil Goddard: The only problem and it's a very subtle nuance is that they don't want to be in a position to say the baby is ugly. I mean, I've actually talked to a lot of CUAs and they say, "You know, we could do the score carding but we probably shouldn't be the ones to do it because..." Dr. Susan Weinschenk: Really? Dr. Phil Goddard: "...we're inside the organization and there is a tendency to say, we need an objective and effective third party." Dr. Susan Weinschenk: So we will come in and say the baby is ugly. (Laughs) (Laughter) Dr. Phil Goddard: But back to this question, I would say that the training really does capture a lot of the essence of the research behind the score card. Dr. Susan Weinschenk: Right and they do get practice on it in the design class. Dr. Phil Goddard: And it certainly would help them create effective scenarios and personas as well which is critical. Dr. Susan Weinschenk: Right and in the classes and that would be the user-centered analysis and contextual design class. Okay, next question. How do you measure beyond usability? For example, what's meaningful and creates a pleasant experience? So, rather than just speak about usability issues, now you did touch on this before, right? Dr. Phil Goddard: Right and I would say that the dimensions that we have characterized are beyond usability which is about, did you "can do"? (Laughter) Dr. Susan Weinschenk: Yes, I can. Dr. Phil Goddard: I "can do" it. Dr. Susan Weinschenk: I can do it. Dr. Phil Goddard: In other words, we're not putting up any obstacles for users to get to the content and consume the content. Dr. Susan Weinschenk: Right, that's typical of usability. Dr. Phil Goddard: But the question is - is the content, is the interaction meaningful? Have I delivered the right content in the right way or the right interaction? And that creates a pleasant experience around that of course, is the idea of doing it in a way that's natural for that type of user. So power, the dimension of power is important. Understanding how to rate a meaningful content and interaction experience and also about understanding what's the brand goals of the company or what are they trying to accomplish from an experience point of view? And understand them enough to be able to evaluate the design and whether it's hitting that goal. Dr. Susan Weinschenk: But, but it is possible-it's a matter then of course of building these other measures – the power and the brand goals and emotion and persuasion, you're building that into your score card. Dr. Phil Goddard: Right. Dr. Susan Weinschenk: I mean if it's in your score card then you can measure it, right? Dr. Phil Goddard: And in this case, building into the design. (Laughs) Dr. Susan Weinschenk: Okay. If it's not there it doesn't get scored well. Dr. Phil Goddard: If it's not there, it doesn't exist. Dr. Susan Weinschenk: Okay. Regarding the persona based reviews, are the experts imagining that they are the person in the persona and then performing the review repeatedly? And if so, why not just run a quick evaluation with real users? Dr. Phil Goddard: Right, and that's a perfectly legitimate question. Certainly you can do many, many, many, many usability tests. This is a complement to that. In the case where you know, you're usability testing is – is perhaps your team actually the scenario I am encountering, they've got an internal usability group. They just don't have time to do usability testing you know, all the time. And you really need quick feedback on what's happening and perhaps it's a small set of screens that don't require you know a bunch of recruiting and getting users and so again, this is a great question. Yes, usability people can answer these questions however what's interesting is still-what I'm suggesting is the usability test data itself needs to be thought of in this new score carding way to be able to communicate it... Dr. Susan Weinschenk: Okay, right. So you can integrate that matter. Dr. Phil Goddard: To the business. So even if you have a very, very good usability operation, I suggest that you consider presenting your usability data in a new way so it communicates the impacts for business. And again, the expert review is a faster design frame so it can be done quicker, it can be done faster and often it can be done cheaper. So those are the strengths of the expert review. Dr. Susan Weinschenk: Okay, we had a question come in before – earlier today via e-mail and I wanted to answer that one too. It's one of our clients who said you know sometimes – his concern was if we use a score card, we've done user-centered design and we use a score card, it makes us look bad because you know, we did the design and followed user-centered design practice and if we do a score card and if it's not a 100 you know, some of the people in the organization are going to say well you know, "Why isn't it a 100? You did user-centered design, what's wrong with you?" So how do you answer something like that? Dr. Phil Goddard: Well, you know, translating user requirements to a good, sound architectural design is one of the hardest steps and that's why people come to us usually, is they don't know how to go from an abstract requirements definition to a good site structure and architecture. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So - and truthfully, we don't always get that right the first time anyway. So I don't think that they should be ashamed that... Dr. Susan Weinschenk: They didn't get a 100... Dr. Phil Goddard: A hundred percent, I mean, we never get it right the first time. Dr. Susan Weinschenk: So is it partly an education issue? Of educating the people within the organization that this is an iterative process and the score card helps it to be iterative? Dr. Phil Goddard: I think yeah, I think there's an appreciation that you're doing the right methodology and if you do it right, you're going to catch these issues early. Dr. Susan Weinschenk: Early before it's released. Dr. Phil Goddard: There are also things that the team that's doing this design work is missing in terms of what their solution is. Dr. Susan Weinschenk: What do you mean? Dr. Phil Goddard: Maybe either, maybe they don't have a real good handle on creating a good navigation system. Dr. Susan Weinschenk: Okay, so maybe the fact that they're not getting a higher score means maybe they should look – go back and look at their user-centered design process and see if they're some holes or gaps in it. Dr. Phil Goddard: Yeah. Dr. Susan Weinschenk: Okay, alright. Dr. Phil Goddard: And then you inform them about it. Dr. Susan Weinschenk: Here's another question. What are some criteria – this weighting idea – to set weights heavier than others on the score card? I mean, how do you know to say that this should have a heavier weight than that should? Dr. Phil Goddard: Well let's go – do we have the ability to go through the slide back again? Dr. Susan Weinschenk: I don't know if we can. Dr. Phil Goddard: Or can we position ourselves on a specific slide? Let's go to slide 16. And it's actually very difficult to see, there are actually 5 overall UX dimensions and in those, there are actually 9 sub-categories. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: One way to present your scores is just a flat percentage of the total points available in that group... Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: Without any weighting. So you're going to navigate, your navigation, your search... Dr. Susan Weinschenk: Yeah right, you could say 15% of the total... Dr. Phil Goddard: Your IA, your content. They're all going to be on a common scale. Dr. Susan Weinschenk: They're all the same, okay. Dr. Phil Goddard: Starting from there, historically we have given more precedence in certain areas to the navigation and IA just because that's how important it is. Dr. Susan Weinschenk: This was just kind of best practice in usability that these are the issues that tend to be more important in terms of how usable a product is. Dr. Phil Goddard: Exactly, so you really have the freedom and I get back to the question, what are some of the criteria to set weights? It's really what the priority – what the business goals are for the website, right? Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So you're going to want to rate your persuasion architecture very high on an e-commerce site. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: That's your calls to action. It's got to be done, no doubt about it. Dr. Susan Weinschenk: So there's not a specific criteria. It's really – it's not like it's a science that says weight this you know, 1.4 times as that. It's just a matter of all the different categories then thinking about the particular site or product and the industry you're in and the group agreeing that we are now going to give more weight to this than that and in fact, this one is twice as important or a third more important and you just do it based on that. Dr. Phil Goddard: That's very true. I think that that's absolutely right. Dr. Susan Weinschenk: Do some clients – these are great questions, right? Dr. Phil Goddard: Yes, they are. Dr. Susan Weinschenk: Do some clients use these cards-these score cards to just verify that their work is okay but then they don't ask for any help improving their work? What sort of qualitative feedback should accompany the delivery of these score cards? Dr. Phil Goddard: Oh, that's really good, yeah. What we do typically is we put the score card on the front of the deck. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: You know, the deck is still there. I guess that's the bottom line. Dr. Susan Weinschenk: You mean like, I have my usual heuristic evaluation; here are all the recommendations I have for changes to the site, that's still there. Dr. Phil Goddard: Right. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: Right, I mean what gets thrown away usually is the deck after it's been acted on. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: What I think should stick around is the score card simply as a benchmark and something that you can track. But the question is - yes, some people do ask us, they want to be quantified but they always want recommendations for how to improve the site based on those metrics. So what we'll do is we'll pinpoint the areas where the site is weak based on the metrics that we turn around and we'll focus on areas that they can improve on. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: And again, this can be done in a competitive way so you can look at your site relative to another, a competitor, and focus on the things that you're doing poorly and also the things that they're doing extremely well. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: But there is yes, there's a qualitative piece of the review that's important. It's the commentary on them and it's very important. Dr. Susan Weinschenk: Okay. You don't want to just say, "You have a 60...bye!" (Laughs) Dr. Phil Goddard: Not the score, yeah. Dr. Susan Weinschenk: Okay. How much additional time will it take to do the score card? So if you had – you know, you're used to doing heuristic evaluation and it usually takes you this many hours, once everybody is trained on how to use the score card, does this add time and how much time should I expect you know, a heuristic evaluation will now take to do? Dr. Phil Goddard: It – it really shouldn't take you much more time. Dr. Susan Weinschenk: Like you... Dr. Phil Goddard: Once you're trained on it... Dr. Susan Weinschenk: So can you give a 10% more or 50% more...? Dr. Phil Goddard: Well, think about usability testing. When you're doing usability testing, you've got a performance rating sheet right in front of you. The users perform in this scenario. Dr. Susan Weinschenk: Right. Dr. Phil Goddard: You're making notes on it. Dr. Susan Weinschenk: Right. Dr. Phil Goddard: At the end, you score it. Dr. Susan Weinschenk: Yeah. Dr. Phil Goddard: Right, you move onto the next scenario. Dr. Susan Weinschenk: Oh, okay, okay. Dr. Phil Goddard: So what the score card is, really something once – once you... Dr. Susan Weinschenk: Once it's in place. Dr. Phil Goddard: Once you get used to using it... Dr. Susan Weinschenk: Right, right. Dr. Phil Goddard: You're doing the flows; you're analyzing the flows... Dr. Susan Weinschenk: Okay, I see. Dr. Phil Goddard: You're scoring and you're making notes as you go. Dr. Susan Weinschenk: You're doing it as you go along, oh. Dr. Phil Goddard: It does add a little bit of time but I would say... Dr. Susan Weinschenk: We're talking hours, right? Dr. Phil Goddard: Not a ton, you just need to be trained upon it. Dr. Susan Weinschenk: So once the training is done and you're used to using it, it maybe it would add you know, minutes to an hour really, right? Dr. Phil Goddard: Right. Dr. Susan Weinschenk: That's interesting. Dr. Phil Goddard: And if you get the whole digital dash board idea set up, then it just rolls up automatically. Dr. Susan Weinschenk: Well... Dr. Phil Goddard: Then really, all you're doing again is you're entering your score cards... Dr. Susan Weinschenk: Yeah. Dr. Phil Goddard: Into the system... Dr. Susan Weinschenk: And it just... Dr. Phil Goddard: And you're doing common tasks. Dr. Susan Weinschenk: Why not let your computer do it? (Laughs) Okay, here's a question that we may actually want to – that we can answer this; I guess or refer them to someone else in the company. What is the range for an expert review in terms of costs? One slide rated it as less expensive than user testing. Dr. Phil Goddard: Yeah, yeah, that's a really good question. Well, the parameters that you're playing with in an expert review in terms of costs, is how many personas am I going to take? Dr. Susan Weinschenk: Right. Dr. Phil Goddard: How many scenarios? So let's say we take 3 personas. Dr. Susan Weinschenk: Alright, typical. Dr. Phil Goddard: And 5 scenarios each. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: We're doing 15 scenarios. Dr. Susan Weinschenk: 15 scenarios – 3 personas, 5 scenarios for each persona? Dr. Phil Goddard: Yeah. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: And then let's say you have 3 competitors. Dr. Susan Weinschenk: Oh, plus 3 competitors. Dr. Phil Goddard: And you're doing 5 scenarios of course and then you're doing 60 scenarios. Dr. Susan Weinschenk: Wow, that's a lot. Dr. Phil Goddard: So these are the kinds of things that you know, make it cheap... Dr. Susan Weinschenk: Make it grow. Dr. Phil Goddard: Or make it expensive and a full review... Dr. Susan Weinschenk: Right, right. Dr. Phil Goddard: A full competitor review can be – well, it's still a very good investment in terms of cost. Dr. Susan Weinschenk: Right. Dr. Phil Goddard: But the range I would say, is anywhere from $25,000 to nearly $45,000. Dr. Susan Weinschenk: $25,000 to $45,000 for us to come in and work on this? Dr. Phil Goddard: Yeah. Dr. Susan Weinschenk: Now how does that compare to user testing? Dr. Phil Goddard: Well again, the parameters for usability testing are how many users? How many locations? Dr. Susan Weinschenk: Right, how many people and locations, but if we can also, can we do some kind of you know, ball park like we did for the – for the expert review? Dr. Phil Goddard: You know, a lot of the costs for the usability tests come in direct costs. What do I have to pay for a facility? What do I have to pay for travel? Dr. Susan Weinschenk: Do I have to recruit the users? Dr. Phil Goddard: What do I have to pay for recruiting? What we've learned over time is that the more you go remote testing, you're removing the direct costs. All of a sudden, Dr. Susan Weinschenk: You're removing some of it but still you're doing the... Dr. Phil Goddard: You're in pretty good shape. Dr. Susan Weinschenk: So again, as an example, well- what do – you know, what do we say in terms of usability testing? Dr. Phil Goddard: I would say traditionally the UT is more than an expert review... Dr. Susan Weinschenk: Yeah so with this guide... Dr. Phil Goddard: But when you remove all the direct costs you're getting into... Dr. Susan Weinschenk: It might be similar. Dr. Phil Goddard: It might be similar. Dr. Susan Weinschenk: So I'll put my neck on the line here... Dr. Phil Goddard: Okay. (Laughs) Dr. Susan Weinschenk: And all my staff will call me up and say, "Why did you tell them that?" Alright, so we're saying again, as long as everything this is "typical", $25,000 to $45,000 for an expert review and I'll put out there $35,000 to $70,000 for a usability test. Again, depending on you know, is it-how many sites is it? Is it remote or not remote? So it is possible that an expert review with a score card could be less or more, right, than a usability test depending on these variables but in general, I think that we tend to think that in most of our scenarios, the expert reviews are a little less expensive than a usability test and then I also want to point out because this is something we do a lot of, you know, we do them together, right? There's no reason why you can't do an expert review and a usability test, right? And then something that we're very fond of doing called the 360, right, full circle – which is the expert review, the usability test and the stakeholders' interviews and you get that whole picture. Dr. Phil Goddard: Absolutely and I would suggest as a matter practice, that if you ran a design cycle that you do your expert review first so that you can act on that data and then you do your usability test afterwards because you get a chance to vent out a lot of the problems that the experts can see and not take those problems into the UT then you're in a stronger position in the UT actually... Dr. Susan Weinschenk: Right. Dr. Phil Goddard: To get whatever issues that you're bringing to the table so... Dr. Susan Weinschenk: It will be the most pragmatic. Dr. Phil Goddard: In terms of a sequence, I would do a small expert review, followed by a UT, if I were going sequence them. Dr. Susan Weinschenk: Okay. And kind of on a similar vein, would you recommend using score cards on mockups? How about rating the navigation part on a static mockup? Dr. Phil Goddard: Yes, absolutely. I mean we've been able to work on fairly low fidelity prototypes and do the score carding on them. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: Obviously the more detail that you have in the design, the more you're going to be able to capture some of the subtle... Dr. Susan Weinschenk: So there might be some items you can't even rate, right? You know some of the interaction items if it's just an early mockup. Dr. Phil Goddard: Perhaps. And if there's not a visual treatment of that... Dr. Susan Weinschenk: Right, right. Dr. Phil Goddard: That represents the brand, the interaction... Dr. Susan Weinschenk: You're not going to do that. Dr. Phil Goddard: You know the presentation... Dr. Susan Weinschenk: Right, you are not going to do that. So you just modify your score card before you rate. Dr. Phil Goddard: Yeah, but... Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: Yes, you can do them on mockups just like you can do usability testing on mockups. Dr. Susan Weinschenk: On mockups, right, okay. Do you feel there's a risk in missing a key observation by having a common score card for reviews, surveys, usability testing? Could your usability experts become robotic in their approach to looking at a design? Like they're just you know, everything starts to look the same. Dr. Phil Goddard: I think the score card – it's like standards. You know, there's the fear that "Gee, standards are going to constrain." But in fact if it's a good standard,... Dr. Susan Weinschenk: Yeah. Dr. Phil Goddard: And the score card is a good measuring stick, then there's the freedom to design and evaluate creative design that the score cards will only reflect good creative work so the fun and joy obviously, is in evaluating good designs that are truly meeting the customer's goals and are creative. These are just measuring sticks. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: So it's just a – it's a- it's a tool but it's not actually the final – the expert reviewer need to take this and interpret from it what you should do with your design. So they always have to be on the creative side of the world and the creative edge. Dr. Susan Weinschenk: And so it's the art part in a way. Dr. Phil Goddard: Yeah, think of it that way. This is the science... Dr. Susan Weinschenk: This is the science and we need that expertise... Dr. Phil Goddard: And there are the actionable recommendations that come from this which is the interpretation. Dr. Susan Weinschenk: And that's not robotic. Dr. Phil Goddard: That's not robotic. Dr. Susan Weinschenk: Okay. Dr. Phil Goddard: That's where the life breath is of all we do. Dr. Susan Weinschenk: Yeah. That's all of our questions. Dr. Phil Goddard: Okay. Dr. Susan Weinschenk: Yeah, so before we sign off, let me remind everyone that there will be an archived version of this webcast available on the website between 2 to 3 weeks. Don't forget, you can always download our white paper. Our next webcast is on December 6th and the topic is "How to Create a Captivating User Experience: User-Centered Design Inside and Out" and that's with our CXO, Jerome Nadel and our President, Jay More. Please again; check our website in a couple weeks for our 2008 webcast schedule. We look forward to seeing you at future webcasts and I want to thank you very much, Phil, for joining us. Dr. Phil Goddard: Thank you, Susan. It was a pleasure. Dr. Susan Weinschenk: Thank you. Top