Part
01
of one
Part
01
What's the difference between experts and non-experts (i.e., novices or quasi-experts) when it comes to evaluating creative products?
Hello! Thanks for your question, "What's the difference between experts and non-experts (i.e., novices or quasi-experts) when it comes to evaluating creative products? "
The short version is that there are ten major differences between experts and non-experts (i.e., novices or quasi-experts) when it comes to evaluating creative products. They are:
- their differing knowledge;
- their ways of approaching the problem, i.e. their problem-decomposing strategies;
- their ways of adapting to the ready availability or lack of a decision criterion;
- their having different internalized standards of what is creative;
- their different requirements of information needs and information sources;
- their differences in the structure of their concurrent cognitive actions;
- their degree of originality and practicality while undertaking a task;
- their giving different weights for the cognitive foundations of confusion and interest in appraising art;
- their rating novelty and own intuition; and
- their understanding of creativity in terms of novelty, appropriateness (relevance and effectiveness), elegance, and genesis.
Below you will find a deep dive of my findings.
METHODOLOGY
A comprehensive search was undertaken to answer the research question. The search yielded several secondary sources. The sources were filtered to find only empirical studies or peer-reviewed psychology or neuroscience journal articles, which revealed meaningful information about the differences between experts and non-experts, with respect to their responses elicited by creative things.
However, many of the empirical studies were not directly related to the subject of creativity. For example, studies of web search behavior, entrepreneurial decision, mobile eye tracking, etc., and these were not considered, although they contained expert-novice differences.
Ten studies, which were directly related to the subject of creativity were selected and these were analyzed. The analysis was directed towards outlining a key difference as stated in each study, regarding how experts and non-experts perceived creative things differently. Suitable texts from the studies were chosen and these were presented as excerpts in block quotes with relevant page numbers, to focus on their evaluating methods, results obtained, and how exactly did the expertise allowed the expert to do better than a non-expert.
This analysis begins with the paragraph, "List", wherein ten studies are analyzed. This is followed by the paragraph "Suggested Further Readings", wherein links of 4 studies are given. These studies answer the question - creativity and how creativity is judged by experts - in greater detail. Lastly, the paragraph "Conclusion" is presented, which concludes this study with a summary of this analysis.
LIST
[ Plucker, J. A., Kaufman, J. C., Temple, J. S., & Qian, M. (2009). Do experts and novices evaluate movies the same way?. Psychology & Marketing, 26(5), 470-478. ]
* Key difference in how experts and non-experts (i.e., novices or quasi-experts) perceive creative things differently was:
- their differing knowledge. " 'True' novices—college students who are very unlikely to work in the film industry—were not only significantly different from professional critics (a clear example of 'expert' raters) but were also different from supposed novices from movie Web sites (p.476)."
* Excerpts in block quotes with relevant page numbers
- See page 470: "Do experts and novices evaluate creativity the same way? This question is particularly relevant to the study of critical and public response to movies. How do public opinions differ from movie critic opinions? This study assessed college student (i.e., novice) ratings on movies released from 2001 to 2005 and compared them to expert opinions and those of self-declared novices on major movie rating Web sites. Results suggest that the student ratings overlapped considerably—but not overwhelmingly—with the self-described novices, student ratings correlated at a lower magnitude with critic ratings, and the ratings of students who saw the most movies correlated more highly with both critics and self-described novices than those of students who saw the least movies. The results suggest a continuum of creative evaluation in which the distinctions between categories such as “novice,” “amateur,” and “expert” are blurry and often overlap—yet the categories of expertise are not without importance. "
2. Some phenomena of problem decomposition strategy for design thinking: differences between novices and experts
[ Ho, C. H. (2001). Some phenomena of problem decomposition strategy for design thinking: differences between novices and experts. Design Studies, 22(1), 27-45. ]
* Key difference in how experts and non-experts (i.e., novices or quasi-experts) perceive creative things differently was:
- their ways of approaching the problem, i.e. their problem-decomposing strategies.
* Excerpts in block quotes with relevant page numbers
- See page 27: "Previous studies indicate that expert scientists use working-forward strategies to solve well-structured scientific problems, while novices use working-backward ones. Although design problems are mostly ill-structured, it was found that designers often decompose an ill-structured design problem into well-structured subproblems. However, little study has focused on the designer’s search strategies in dealing with well-structured design subproblems. The research method adopted is protocol
analysis. The results suggest that an obvious difference between experts and novices is their problem-decomposing strategies. Experts’ explicit problem-decomposing strategies are important factors in design efficiency."
- See page 43: "The results of this study indicate that an obvious difference between experts and novices is the way they approach the problem; this difference might be due to their different problem-decomposing strategies. Experts tend to establish problem structure at the beginning in order to represent the problem in their own way. [...] While novice designers tend to eliminate the problem when they fail to handle it. "
3. Effects of framing on evaluation of comparable and noncomparable alternatives by expert and novice consumers
[ Bettman, J. R., & Sujan, M. (1987). Effects of framing on evaluation of comparable and noncomparable alternatives by expert and novice consumers. Journal of Consumer Research, 14(2), 141-154. ]
* Key difference in how experts and non-experts (i.e., novices or quasi-experts) perceive creative things differently was:
- their ways of adapting to the ready availability or lack of a decision criterion.
* Excerpts in block quotes with relevant page numbers
- See page 147: "All dependent measures were analyzed in a 3 (creativity criterion/ reliability criterion/ no criterion) x 2 (comparable/ noncomparable alternative set) x 2 (expert/ novice) between-subjects analysis of variance design. Since a priori hypotheses were made, these were directly tested with complex contrasts using the mean square error from the overall analysis of variance table. The degrees of freedom for the t-statistic of the contrasts were 1 and 174 except where otherwise reported."
- See page 141: "The study finds that framing, i.e., priming different decision criteria, influences evaluation outcomes for both expert and novice consumers when the alternatives are noncomparable and influences evaluation outcomes for novices when the alternatives are comparable. The ready availability of a decision criterion, as opposed to the lack of one, also alters consumers' cognitive responses for noncomparable sets to make these responses appear more like cognitive responses typical of comparable sets. One fundamental distinction between sets of noncomparable and comparable alternatives may be the ready availability of decision criteria versus the need to construct them, rather than any inherent differences in category types."
[ Kaufman, J. C., Baer, J., Cole, J. C., & Sexton, J. D. (2008). A comparison of expert and nonexpert raters using the consensual assessment technique. Creativity Research Journal, 20(2), 171-178.]
* Key difference in how experts and non-experts (i.e., novices or quasi-experts) perceive creative things differently was:
- their having different internalized standards of what is creative.
* Excerpts in block quotes with relevant page numbers
- See page 171: "The Consensual Assessment Technique (CAT) is one of the most highly regarded assessment tools in creativity, but it is often difficult and/or expensive to assemble the teams of experts required by the CAT. Some researchers have tried using non-expert raters in their place, but the validity of replacing experts with non-experts has not been adequately tested. Expert (n=10) and non-expert (n=106) creativity ratings of 205 poems were compared and found to be quite different, making the simple replacement of experts by non-expert raters suspect. Non-expert raters’ judgments of creativity were inconsistent (showing low inter-rater reliability) and did not match those of the expert raters."
- See page 175-176: "The poets were judging the poems, one might assume, based on internalized standards of what is creative in poetry. [...] It depends on the goal of the evaluation. Indeed, if a studio wants a gauge of how well a movie will perform at the box office, an assessment from novices may be quite preferable to an assessment from traditional experts. [...] If the goal is to find the most accurate assessment of creativity of products in a given domain, based on the current standards and values of that domain, then experts and other gatekeepers seem to be the group with the most face validity. Indeed, from a logical standpoint,
they would seem to be most valid judges. In some fields, experts may be the only reasonable judges for most purposes. Asking the general public to evaluate the creativity of a new theory in nuclear engineering would make little sense."
[ Björklund, T. A. (2013). Initial mental representations of design problems: Differences between experts and novices. Design Studies, 34(2), 135-160. ISSN 0142-694X (printed). DOI: 10.1016/j.destud.2012.08.005.]
* Key difference in how experts and non-experts (i.e., novices or quasi-experts) perceive creative things differently was:
- their different requirements of information needs and information sources. "Experts seem to perceive both more information needs and more information sources relevant to the problem at hand (p.28)" than novices.
* Excerpts in block quotes with relevant page numbers
- See page 6: "Given the recent demonstrations of the importance of proactivity for successful work behavior (Baer & Frese, 2003; Frese, et al., 2007; Koop, De Reu & Frese, 2000; Seibert, Crant & Kraimer, 1999) and innovativeness (Binnewies, Ohly & Sonnentag, 2007; Frese, Teng & Wijnen, 1999; Ohly, Sonnentag & Pluntke, 2006; Seibert, Kraimer & Crant,
2001), proactiveness presents itself as a critical ingredient for success, especially in creative domains such as product development. Thus whether proactivity differences between experts and novices can be traced back to the initial mental representations or problem structuring and framing stage of product development problem solving becomes an interesting question. "
- See page 28: "This study addresses the gap in knowledge by exploring the differences in initial mental problem representations and reflections on real-life product development problems between advanced product development students and
recommended, professional product development experts. The results reveal that experts have superior extent, depth and detail in their representations, accommodate for more interconnections both within the problem information and between the problem and previous knowledge, and approach the problem in a more proactive manner. Design experts seem to perceive both more information needs and more information sources relevant to the problem at hand. Thus the results of this study indicate that forming comprehensive mental representations based on wide relevancy perceptions should be considered as a performance and proactivity enhancer in design along with previously identified cognitive-motivational issues."
[ Kavakli, M., & Gero, J. S. (2002). The structure of concurrent cognitive actions: a case study on novice and expert designers. Design studies, 23(1), 25-40.]
* Key difference in how experts and non-experts (i.e., novices or quasi-experts) perceive creative things differently was:
- their differences in the structure of their concurrent cognitive actions.
* Excerpts in block quotes with relevant page numbers
- See Abstract: "This paper presents a case study of concurrent cognitive actions of a novice and an expert designer. We analyzed cognitive actions of designers using the retrospective protocol analysis method and found evidence of coexistence of certain types of cognitive actions in both the novice and expert designers’ protocols. The main difference between the two designers’ protocols is the structure of concurrent cognitive
actions. While the expert's cognitive actions are clearly organized and structured, there are many concurrent actions that are hard to categorize in the novice’s protocol. We also found that the expert’s cognitive activity and productivity in the design process were three times as high as the novice's. Based on these results, we discuss the possibility that the expert’s structured and organized cognitive actions lead the expert to a more efficient performance than the novice."
7. Interactive processes between mental and external operations in creative activity: a comparison of experts' and novices' performance
[ Ishii, N., & Miwa, K. (2002, October). Interactive processes between mental and external operations in creative activity: a comparison of experts' and novices' performance. In Proceedings of the 4th conference on Creativity & cognition (pp. 178-185). ACM.]
* Key difference in how experts and non-experts (i.e., novices or quasi-experts) perceive creative things differently was:
- their degree of originality and practicality while undertaking a task.
* Excerpts in block quotes with relevant page numbers
- See Abstract: "Prior studies of creativity, in the field of cognitive psychology, have mainly dealt with only the process of mentally thinking of ideas. We investigated, through a cognitive psychological experiment and its protocol analysis, experts' and novices' interactive process between the mental operation by which participants considered their ideas and the external operations by which they actually produced physical objects in creative activity. In our experiment, the participants were required to build a creative robot with LEGO Mindstorms. The experimental results showed that the experts could create work that fulfilled both high originality and practicality simultaneously. Moreover, the following four points were confirmed as characteristics of the experts' creative process: (1) the experts globally considered their initial ideas, (2) the experts reconstructed their ideas by considering comprehensively the relationship among the elements constructing their plans, (3) the experts reconstructed their ideas more actively, and (4) the experts were able to embody successfully their ideas by focusing on various aspects of important viewpoints."
[ Silvia, P. J. (2013). Interested experts, confused novices: Art expertise and the knowledge emotions. Empirical Studies of the Arts, 31(1), 107-115.]
* Key difference in how experts and non-experts (i.e., novices or quasi-experts) perceive creative things differently was:
- their giving different weights for the cognitive foundations of confusion and interest in appraising art.
* Excerpts in block quotes with relevant page numbers
- See Abstract: "Experts and novices differ in how they experience and understand art, and expertise in the arts moderates a wide range of aesthetic processes. The present research examined the role of expertise in the experience of interest and confusion, two knowledge emotions, particularly whether expertise moderated the cognitive appraisals that bring about the emotions. A sample of 174 people viewed and rated a series of abstract images taken from small-press publications. Expertise was measured with the aesthetic fluency scale (Smith & Smith, 2006), and multivariate multilevel models were used to estimate the effects of expertise and appraisals. As expected, people high in expertise found the images significantly more interesting and less confusing. The main effects of appraisals on interest and confusion replicated past work, and expertise interacted with most of the appraisals—the cognitive foundations of interest and confusion have different weights for experts and novices. "
- See page 113: "The present research expands evidence for confusion and interest as aesthetic states and illustrates several roles for expertise. [...] expertise in the arts had a main effect on both interest and confusion: people with higher aesthetic fluency scores found the images more interesting and less confusing. Finally, expertise interacted with appraisals consistently, indicating that the cognitive bases of interest and confusion differ between experts and novices. A central goal was to explore the conflicting visions of how experts vary in the underlying roots of aesthetic experience. The findings for interest and confusion were more supportive of an “expert consensus” approach (Hekkert & van Wieringen, 1996) than an “idiosyncratic experts” approach (Leder et al., 2012). As expertise increased,
1. novelty more strongly predicted interest;
2. comprehensibility less strongly predicted interest; and
3. comprehensibility more strongly predicted confusion.
Because only one within-person relationship decayed as expertise increased, the findings offer less support for the view that experts’ preferences are idiosyncratic. Instead, experts appeared to lean more on appraisals of novelty for interest and comprehensibility for confusion, so some dimensions were more important for
experts than novices."
9. Learning to judge creativity: The underlying mechanisms in creativity training for non-expert judges
[ Storme, M., Myszkowski, N., Çelik, P., & Lubart, T. (2014). Learning to judge creativity: The underlying mechanisms in creativity training for non-expert judges. Learning and Individual Differences, 32, 19-25. ]
* Key difference in how experts and non-experts (i.e., novices or quasi-experts) perceive creative things differently was:
- their rating novelty and own intuition. "It is more likely that the relatively more analytical cognitive investment involved in rating novelty and elaboration (prior to rating creativity) went at the cost of the relatively more intuitive process of judging creativity (p.25)." The differential was significantly reduced by training novices, to judge creativity more like experts in the area of children's drawings .
* Excerpts in block quotes with relevant page numbers
-See Abstract: " Evaluating individual creativity is an important challenge in creativity research. We developed a training module for non-expert judges in which participants learned the definitions of components of creativity and received expert feedback in an interactive creativity judgment exercise. We aimed to test whether and how the training module would increase the reliability and validity of non-expert ratings. Study 1 (N = 79)
showed that the training had a positive effect on the test–retest reliability and validity of creativity ratings. Study 2 (N = 126) replicated the results on test–retest reliability and validity but with low absolute values, indicating that trained participants cannot substitute experts. In addition, Study 2 showed that the effect of the training module on the validity of creativity ratings was mediated by increased validity of ratings of novelty and elaboration."
- See page 25: "As a conclusion, the results of the present paper show that in principle it is possible to teach lay people to judge creativity more like experts, at least in the area of children's drawings, and suggest ways to design a training module based on the subcomponents of creativity judgments. However, our results also call for more research to investigate the boundary conditions and generalizability of this effect, and perhaps more importantly whether it is possible to turn novices into experts quicker than the time that we need to find experts."
[ Kaufman, J. C., Baer, J., Cropley, D. H., Reiter-Palmon, R., & Sinnett, S. (2013). Furious activity vs. understanding: How much expertise is needed to evaluate creative work?. Psychology of Aesthetics, Creativity, and the Arts, 7(4), 332.]
* Key difference in how experts and non-experts (i.e., novices or quasi-experts) perceive creative things differently was:
- their understanding of creativity in terms of novelty, appropriateness (relevance and effectiveness), elegance, and genesis.
* Excerpts in block quotes with relevant page numbers
- See Abstract: " What is the role of expertise in evaluating creative products? Novices and experts do not assess creativity similarly, indicating domain-specific knowledge’s role in judging creativity. We describe two studies that examined how quasi-experts (people who have more experience in a domain than novices but also lack recognized standing as experts) compared
to novices and experts in rating creative work. In Study One, we compared different types of quasi-experts with novices and experts in rating short stories. In Study Two, we compared experts, quasi-experts, and novices in evaluating an engineering product (a mousetrap design). Quasi-experts (regardless of type) seemed to be appropriate raters for short stories, yet results were mixed for the engineer quasi-experts. Some domains may require more expertise than others to properly evaluate creative work."
- See page 23: "However, when creativity is broken down into more highly differentiated components: novelty, relevance and effectiveness, elegance and genesis, a more complex picture emerges. It appears that the core criteria that defines creativity (novelty and appropriateness), may be more independent of the level of expertise of the observer, while more sophisticated (and domain-dependent) criteria such as elegance and genesis remain strongly linked to the level of expertise of the observer."
- See page 24-25: "On a practical level, expense of getting expert raters to look at creative work has largely limited the Consensual Assessment Technique to research use. Finding a happy medium might encourage more work with this technique, which allows a domain-specific perspective on creativity. On a theoretical level,
exploring the question of how differing levels of expertise leads to similar or divergent perceptions of creative work can yield insight into many aspects of cognition. This study compared novice, quasi-expert, and expert ratings of creativity in two domains, creative writing (short stories) and engineering (product design). Novices, consistent with extensive past work (see, e.g., Kaufman & Baer, 2012) only showed acceptable levels of reliability and expert
agreement when used in excessively large numbers. When the level of analysis was lowered to groups of 10, reliability and agreement was drastically reduced. The utility of quasi-experts varied by domain. For creative writing, quasi-experts showed strong reliability and expert agreement; in engineering, the results were inconsistent and generally did not support the use of quasi-experts as raters."
SUGGESTED FURTHER READINGS
someone is an expert or not
4. Creative Expertise as Superior Reproducible Performance: Innovative and Flexible Aspects of Expert Performance
CONCLUSION
To wrap it up, this analysis successfully:
1. lists 10 academic papers related to creativity;
2. outlines 10 key differences in how experts and non-experts (i.e., novices or quasi-experts) perceive creative things differently;
3. provides suitable texts from the papers as excerpts in block quotes with relevant page numbers, to focus on their evaluating methods, results obtained, and how exactly did the expertise allowed the expert to do better than a non-expert; and
4. provides 4 links as suggested further reading material for detailed examination of the research question.
Thanks for using Wonder! Please let us know if we can help with anything else!