Frequently asked questions

1. Is it true that measure from users can only be subjective?
2. How can user measures be objective?
3. What is the difference between qualitative and quantitative measures?
4. Can you calculate an average of a MOS scale?
5. How can subjective measures be quantitative?
6. Are there manuals available about user tests?

Is it true that measure from users can only be subjective?

No. It is possible to take objective measures in laboratory experiments and field studies. In fact it can be argued that the most important measures from users are objective measures because they should be the most valid and reliable measures. More information and examples are provided in the chapters here on ‘Objective measures’ and ‘Why objective measures are important’.

How can user measures be objective?

User measures can be objective if they record actual user performance such as number of errors and time to complete a task.

What is the difference between qualitative and quantitative measures?

Qualitative measures are measures at the nominal or ordinal level of data. Nominal data is categorical without any order implied between the categories, such as whether people are left or right handed. Ordinal data have rank-order relations, such as ‘Bad’, ‘Poor’, ‘Fair’, ‘Good’ and ‘Excellent’, but conclusions about the distance between the scale positions cannot be made. For example, we cannot necessarily assume that the distance between ‘Poor’ and ‘Fair’ is the same as ‘Fair’ and ‘Good’. Even if the numbers 1 to 5 were assigned to these categories, numerical calculations such as an average value would be invalid if calculated.

Quantitative measures have a known ratio between any two intervals. An example is time taken to communicate, in which case 4 minutes is twice as fast as 2 minutes.

Can you calculate an average of a MOS scale?

There is a risk that calculating the average value of a MOS scale is invalid. This is because typically the MOS quality scale is an ordinal qualitative scale and not a quantitative scale. Even if the five scale positions of ‘Bad’, ‘Poor’, ‘Fair’, ‘Good’ and ‘Excellent’ are numbered from 1 to 5 or from 0 to 100, the numbers may not accurately reflect the distance between the scale positions.

How can subjective measures be quantitative?

A subjective measure can be quantitative if the user opinion is measured on a rating scale that has no labels or that only has a label at each end of the scale. In this way the users are expected to use a rating scale as if it has numerical properties with ratios between the intervals. If the intervals have labels such as ‘Bad’, ‘Poor’, ‘Fair’, ‘Good’ and ‘Excellent’ the user is expected to think in terms of ordered categories without a known ratio between the intervals.

Are there manuals available about user tests?

Yes, there are two guides on user measures that are particularly recommended for more information:

• ETSI EG 201 472 is concerned with usability evaluation for the design of telecommunication systems, services and terminals
• The Eye-2-Eye Evaluation Toolkit (Heim et al., 2003) focuses on real-time person-to-person communication services and was used to derive some of the empirical results contributing to the guidelines available at this site.