Add-On to a usability evaluation

Depending on the situation, your big company has an in-house UX lab facility with the respondent database with the team of the facilitator, or moderator.   However there is another situations, you simply don’t have the these resources.  Maybe in some cases, you have to go to the foreign countries to conduct research.  In these cases, it is common practice to use the field research or market research agency for the recruiting and facilities.

I would like to share some experience about these processes using the agencies.  There are many firms specializing in these job.  For example, there is WatchLab in Chicago.  Typically they are located in the prime central place with convenience access and has luxurious interior.  For it is basically real-estate business like hotel business, location is very important.  As you can imagine, the reason is that many people has to visit this place for the first time, and the time & cost to visit this place should be optimal.  So it makes sense to have an office at the down-town prime space.

For the first timer, price is very important for budget planning.  However there is nothing like average cost.  It really depends on the situation.  For example, pilot or medical doctors are expensive.  Smart phone users will be relatively cheap.   However as an example, an incentive for the consumer electronic product can be $100 for two hour.  And you have to pay additional $100 for the recruiting to the agency.  Sometimes you will want a professional facilitator specializing in UX evaluation, which can cost $1200 a day for 4 session of 1 hour long.  The nice rooms with the mirror-window which only researchers can see the participant with lots of AV equipment costs $1000 a day.  And maybe there will be a transcript service for $100 for an hour.  It is extremely busy, so catering maybe a good idea.  Add the sum yourself, and you can find it is not cheap process.

So you know the price, how can make it valuable process?  In my opinion recruiting( and analysis for next time) is very important.  For example, if you are conducting UX evaluation about the smartphone. You would like to include two groups of people, say novice users and advanced users. It will be very tempting to say I would like to recruit novice and advanced users in the meeting with the professional research recruiter. However it is very subjective what constitutes novice and advanced user group. So the busy research recruiter may have an entirely different point of view about the each group, and prepares inadequate samples for the each groups. So it is designer’s role to prepare more numeric and quantitative guideline for the recruiting, which characteristic each group has. Say novice group has less than one year of smartphone usage experience, and uses less than 5 apps daily, with the age group of 40~50, with the equal distribution of the male/female, and does not use social media apps using their smartphone,… and so on. And during the recruiting process, involve with the recruiter actively during the process, and requesting participant survey for confirmation is good idea.

Also in my opinion, determining the right number of participants are not so important than it sounds. More often than not, it is determined by the budget and resources to conduct the research. Remember more energy is required in the analysis of the result. So it is very practical that having a number of the users, designers can personally identify and “View” the entire session (which can be really boring after say 3 person), find the design improvement point, and do an analysis over the session.  I would like to also remark that it is physically demanding task.  Part of the reason is that it may start after say 6pm due to the working participant.

Finally I would like to briefly mention 2 services, which can help during the process.

The first is eye tracking.  For the software review, knowing the pattern of the user’s eye movement can help detect the problematic area, which maybe not available without the use.  It shows the inner working logic of the user mind by showing which area got attention and didn’t.  It is possible to track eyes for the desktop app and mobile app.  Also there is a rental firm of equipment and they also provide total service including the tech personel during the process and analysis report.

The second service is the streaming service.  Sometimes designers or stakeholders are far-away.  By streaming data real-time and providing video archive for the later reference, it is possible to participate during the process.  It also provides stereo channel where left side can be original sound, and right side can be real time translation.

Bad UI: Which side is cooler?

I moved recently to the university housing.  One hot day, I turned on the air conditioner, and spontaneously turned temperature knob to the lowest setting.  But soon I found something is wrong.  It was not cool.  What was problem? It was the large title saying “Temperature” that caused mismatch between my mind and the designer’s mind.  When we say temperature, isn’t low temperature is cooler?  However in this case there is a small caption saying “Max Cool” at the highest temperature setting.  It apparently shows the conflict between the working control and the working model in user’s mind.  It is also sad that a usability test in the real world setting can easily detect this problem and the fix was easy to implement because it is just change of the silk screen print.

RAA1: Beauty is in the eye of beholder. Cultural difference about usability.

Improving Performance, Perceived Usability, and Aesthetics with Culturally Adaptive User Interfaces

Katharina Reinecke and Abraham Bernstein, University of Zurich

Purpose of the research 

We all know the aesthetics plays important role in the usability and performance.  Using pretty interface, the user likes it better, performs better.  However the problem is beauty is really subjective thing.  And certainly culture is one of the factors deciding the standards for the beauty.  I remember doing class presentation with team about excise, and Eddy showed Baroque-like chair from china and modern zen chair as an example of different perception of aesthetics. Then how about showing the users interfaces built according to their cultural norms, instead of showing same interfaces for the all global users?  Will the customers like it better and perform better?  This paper tries to answer that question.


First, how we can decide the one’s culture? It is certainly different from the country of origin.  The paper made a average of the countries where one user stayed.  So the more one user stayed one country, the more he/she is adapted to that culture.  So that is user model.

How about the cultural difference in the design of the interface.  There is very famous work by cultural anthropologist Hofstede*, where he studied the characteristics of IBM employees from different culture.  According to it, there is 5 dimension in the culture, like Power distance, Individualism, Masculinity, Uncertainty Avoidance, Long Term Orientation.  According to the this dimension, they made a specific design rule according to the high/low value.   That becomes a rule for the that culture.  So that is the culture/UI rule.

And finally we need a test platform to do this.  The author made a service website devoted to the study called MOCCA. It is basically to-do-list website.  They gathered 41 international participants.  Basically they ask users where have you been, and generated rule for the interface, and adapt interface according to the rule and show it to the user.

Main findings

Not surprisingly, the result shows that users were 22% faster using the culturally adapted interface, needed fewer clicks, and made fewer errors.  Also subjective methods (asking users to evaluate themselves), shows the users thought the adapted version was easier to user.


When we think about the design paradigm, the fact is that the bars are getting higher.  At first something working reliably working is welcome.  Then it should be easy to use to win the competition.  Finally at the fierce competition around matured product,  it should be pleasant or fun to use to survive in the market.  And aesthetics plays important role in the pleasant experience.   For example, Apple is one of the most successful company, for they are really good at providing beautiful experience.

However when we think about aesthetics, it is really subjective, isn’t it?  So the previous paradigm, where the omnipotent designer worked really hard to create universally absolutely beautiful product is not applicable any more.  Rather the design output can be a function of various inputs like culture, rather than a static point.  It is this papers value that shows diligently this direction.  It is easy to assume it.  But to test this assumption, they made a working sample of website and developed rule for the adaptation.  And invited 41 users to test it, and evaluated their result to show that this assumption is valid.  It is not a small task.

However I would like to extend this theme a little more.  For me,  culture is certainly one dimension in the optimal design function.  Other can be sex/age/income and so on.  At this point it becomes more complex multidimensional statistical problem.  So which dimension affects how much becomes hard to tell. Then finding a hidden theme becomes more of user clustering or machine learning.  There is many users of various taste.  Even person from same culture shows different taste.  So this concept of culturally adapted interface should be extended to the taste adapted interface.

Personally aesthetics has been always my weak point.  As you can see my motto “Intuinno” , I work to make innovation (which is science or engineering driven) intuitively (which is about usability), but that’s all.  I never approached the aesthetics with the same confidence with other two discipline.  That’s where team work shines.  If there is world class team, like in TV series in CSI, I can play the usability part (Anyway my engineering is also under average, even though I can make it barely work. )  and Jonathan Ive can do the aesthetics part.

Finally here is some interesting pictures of the beauties from all over the world.   You can see we are different.


Katharina Reinecke and Abraham Bernstein. 2011. Improving performance, perceived usability, and aesthetics with culturally adaptive user interfaces. ACM Trans. Comput.-Hum. Interact. 18, 2, Article 8 (July 2011), 29 pages. DOI=10.1145/1970378.1970382

Photos examples of Beauties.  National Geographic channel. The link is

Hofstede, G. 2001. Culture’s Consequences: Comparing Values, Behaviors and Organizations across Nations 2nd Ed. Sage Publications.

RR: Info Architecture; Prototyping

Chapter5: Information Architecture

Information architecture is organizing information so that it is easy to use.  If we apply paradigm of design we learned last week, information architecture deals with the paradigm of engineering and paradigm of usability. (You can see nice pyramid about paradigm of design here)  So it is about how to make information easy to find.   So many interesting theories like information foraging is used. It will be especially useful, when there is many information in certain places, when the user will have to guess where the related information is located.  Providing self-relevant categories and many hints(scent) about where the info may lies is key task for the information architect.

There can be a really wide variety of category over specific items.  However some of the user-centered techniques for creating such architecture includes card sorting ,closed card sorting, similarity matching.  They request users to categorize and label various items.  The reason these are user-centered is that from various categorization available, designer follows what the customer thinks naturally.  One concept useful in this approach that is not discussed in the article is the concept of user’s mental model and designer’s representation model.  By making these similar, we can achieve optimal organization.  And that’s why we designers are interested in the model in user’s mind.

There are many topology certain sites can have.  For example there is hierarchy(tree), linear (sequence), matrix, full mesh, arbitrary network and hybrid.  Even though there is no absolute answer to the topology, a topology is adequate at certain situation.  One thing I would like to note is the horizontal vs vertical organization.  If there is many contents, horizontal organization with many links at certain level which means short required level.  Also the vertical organization means at certain level there is small number of links, which means deep required level.  It seems horizontal organization performs generally better than the vertical organization.  It seems because there are many choices, the probability of error also increases.  So in vertical organization where the user has to make many choices, there can be more errors.  So heuristically sublevel below 3 is now recommended.

Chapter 11: Prototyping

Let me begin with the different types and fidelity levels of prototypes.  There is horizontal and vertical prototype. Also there is variations like T and local prototype.  In terms of fidelity there is low, middle, high-fidelity prototypes.  Also if we consider the interactivity, there is scripted and click-through, a fully programmed, Wizard of Oz type, physical mockup and animated prototype.  So there is many types of prototype, and finding a adequate one to try seems daunting task.  Or is it?  Well, it seems common sense and reality check with time, budget and man power can reduce many choices into a few feasible rational selection.

1. First, matching the level of fidelity with the stage of progress is good idea.  That means start with low fidelity, proceed to the high quality pixel perfect prototype.

2. You should know using programming makes it complex, because nature of program is that it should be perfect to work even for the prototype.  Pen and paper are easy to create variations.  So begin with the pen and paper and proceed to the programming if it is required.  Please keep in mind that even though the prototyping code looks functional, development will prefer starting fresh.

3. Vertical prototyping is nice to test a certain feature which requires sequential interaction like check out. Horizontal prototyping is good for the general look and feel and feature set.

4. There is times when high fidelity prototype is required.  When the time comes, you will probably know.  Such event includes VC pitch, Big boss meeting, request from technical sales.

However there are pitfalls when using prototyping.  I already mentioned pitfalls with the programming in the previous blog post. There is a few more pitfalls.  Interestingly, there is a pitfall with low fidelity prototype and pitfalls with high fidelity prototype.

1. Low fidelity prototype is nice if it is used with the people who understands the design process.  However there are many cases where people don’t know about it.  If the people is your boss, client or stakeholder, it can be problematic.  For example, they may think you are an inexperienced amatuer (Maybe it is true that you are inexperienced amateur.  Not in the sense, your design skill is not good enough but you are not experienced with clients, if you showed it to someone who can think like that. )

2. High fidelity prototype also has its own pitfall.  For example, the clients sees the prototype, they think it’s done and may even try to save money by declaring the job done.  Other than that if you have a nice slick look and feel, most of your feedback can be on the topic of detail about look and feel not the interaction flow.  Also people tend to be less inclined to criticize work if it looks done, for it is too late.

By the way, the topics about the paper mockup was interesting.  I wonder maybe our next class will be an art and craft with pen scissor glue developing paper mockup for the mini-hub.

Seduction for the mortgage finance

Yesterday a few graduate student led by the Dr. Mihaela Vorvoreanu met Mr. John Yuda at the nice restaurant in the indianapolis. Mr. Yuda has been doing interaction design for for the government agency which helps normal people makes an informed decision for the important financial matters like college education or housing mortgage. Mr. Yuda kindly explained his experience and answered questions.

His project was making a website where people can learn more about mortgage product and make wise decision. I was fascinated by the difficulty of the problem. There is two reason why it was difficult. First it try to change the thinking of grownups. You know that adult people is really hard change. For example, smoker will continue smoking no matter what gross image of their lungs is shown. Or how about obesity? Healing obesity is difficult because it requires life style change. Surgery or drug can heal the obesity even though they can help temporarily. Using emotional devices like curiosity and attachment seems relatively effective than the intellectual approach. There is a discipline called persuasive technology, which deals with the issues mainly about this. Games, humor and aesthetics are frequently used device to make change.

However the second reason it was difficult was that they try to teach them very dry subject like financial education. It makes the use of the emotional approach very hard, because it is usually hard for normal people to feel the economics sexy. And it is hard because the end goal is not a pleasant experience but a meaningful experience. This website is not for the entertainment but education. There is a very nice book I am reading recently called “Seductive Interaction Design”. I will blog more about it later. But there is a nice chart showing the hierarchy of the design. It is similar to the paradigms of the design we learned in the class, but has more detail. It shows that achieving meaningfulness is harder than achieving pleasant experience.

So if we suppose this website is human, the tone of the interaction is tricky and important. Also what kind of emotional state we are designing for the readers? I asked Mr. Yuda about it. And he answered even though the team is considering game mechanism also, it should not focus on the fun. It should not be also too erudite. So he proposed a voice of spiritual leader. That seems great.

I always love the interesting problem. So I happen to spend sometime thinking myself, how I would do if I was given the task. I am thinking a metaphor of “economist at the cocktail party”. It’s first time you met him. He seems a plausible guy and try to use the plain language. So you tell him that you are looking for a house to buy. Or considering refinancing the existing contract with excitement which is typical if you buy house. He congratulates and says like this.

“Congratulation!  I have been conducting the case study of housing mortgage for years. There is some interesting story I would like to share, if you don’t mind.

There was a myfriend named Jim. He is a good guy and family man. After he got a baby, he bought a house. He earns about $2000 weekly. But he has to pay $1200 for the mortgage. Even though it was manageable, he couldn’t save any money for later after paying his bill and mortgage. However one day Jim’s car is broken and it costed $800 to fix it.

……… (Here goes all the sad story of Jim)

So it would be wise for the Jim, if he bought a cheaper house with smaller montly payment and save a little for the emergency. It could save him a lot of trouble”

How I translate this experience? Well, I prepared a very simple mockup of website. For I am not a visual designer, and has bad aesthetics, it is ugly. But the thing I would like to express is a “friendly economist at the party” who tells you “story that is interesting and meaningful” with “lots of graphics and fewer text”. I considered using cartoons also, but the government has some style of its own. So a realistic picture seems better.


So that’s the end of the today’s story.  I would like to thank Mr. Yuda and Dr. Mihaela Vorvoreanu for their kind effort for poor graduate student.  Mr. Yuda’s remark about paying forward is venerable.  And I would like to thank CGT also.  If somebody from the department read this, I would like to remark that after this meeting I began to consider Ph.D in CGT department.  And I would like to mention also that Dr.V was educational all the way and responsible driver also.  I was happy that we had insurance anyway.

RR Cooper Ch 7 : Know Thy Enemy

Knowing who is an enemy makes the difference between the survival and death during the battlefield.  At the design process, the developers are your enemy.  I regret I expressed it too strongly.  It makes it vulnerable to the counter-argument.  However I express it that way because it is true.

Why they are your enemy?  Because once they write a single line of the code, they will stick to it.  And worse, they can implement most of the user requirements in a few hours or days.  So they think interaction design takes too long, and are not sure if it is necessary at all.  While waiting, they got tempted to write a few line of codes to implement user requirement.  So how about beginning implementation while waiting?  Oops it started working.  And I finished 90% of function already.  So how about bringing a designer and ask him/her to make the existing product look better and declare done!  Nobody likes the delaying project.  And every management will love the guy who delivers project is done before schedule.   Oh, I forgot the guys interviewing people and sketching rough drawings on the wall.  They bring something to the table finally.  But who cares?  The project seems to be done, and no one in the management will like the news the existing product should be made again to reflect the change interaction designer brings.

That’s why those innocent looking, nice and kind developers are enemy in a nutshell.  For this Cooper warns strongly do not let the coding begin before the interaction design process.  Also that is why I kept interrupting Dr.V about the idea of two team (developer and interaction designer) working simultaneously.

So you know the identity of the enemy.  It’s time to learn their characteristics so that we can defend ourselves.  In this purpose, I recommend reading the Joel on software blog.  He is very smart like every other developers.  But what makes him really stands is that he has the rare talent of explaining simply with the humor using written language.