How to Do Things with Pictures: The Experience of Challenging Imagery in Visual Media

Abstract

Some imagery poses an interpretational challenge that must be met in order to fully process it in the context of goal-oriented visual media. Evaluations of such imagery can only be considered in terms of a reader’s experience, an interpretational sequence that plays out in time. Pictures that are “challenging,” as discussed here, qualify as “integral experiences” according to Dewey: they have meaning, they are best understood as processes rather than elements, and they promote conceptual change in a reader. Four kinds of pictures that represent challenges to the reader are identified: problematic, linguistic and creative imagery, and visual metaphor. The experiential impacts of text–picture relationships are discussed. Boundary examples of imagery are addressed to distinguish challenging imagery from other types and to establish the distinction of “performative” imagery, that which instigates and targets particular interpretational processes. The types of challenging pictures presented here qualify as performative in that they do more than repeat textual content or offer a moment of distraction

Keywords: image function, interpretation, picture processing, semiotics, visual challenge, visual metaphor, visual rhetoric.

The Image as Activity in Visual Media

When we “read” a worthwhile picture, or make sense of the challenges it puts before us, we are engaged with interaction design as surely as if we were tapping or swiping on a trackpad. I mean to suggest that both activities engender engagements which occur over time and contribute to what pragmatist philosopher John Dewey called an integral experience[1] There is a sense of consummation of an interaction in an integral experience. It has meaning in a person’s life. It does not go unnoticed. In an integral experience, the reader (or user, or person) undergoes a process until she and the object (picture, text, etc.) are mutually adapted: the reader changes through learning, and the object, which effectively exists through her relation to it, is changed as well. The designer who approaches her work with a user-centered focus knows she is designing for experiences. Ideally she is concerned that these experiences will be productive for users, that she has created conditions under which it is likely that something will be learned, or that a positive memory will remain.

Not every picture is experiential; far from it. Herein I will outline a number of ways in which pictures can be designed to produce integral experiences by modelling interpretational processes. In considering imagery, I do not isolate it from its everyday expressions on web sites, in magazines, on billboards, or anywhere else that people routinely encounter pictures. Rather, whatever universality I intend for my assertions, I address imagery with a sensitivity to situation and context. I define situation as the conditions under which a user encounters design (a student reading a textbook in order to pass a coming test), and context as physical features surrounding any element under scrutiny (a textual passage near a picture, an enclosing textbook chapter, and even a classroom itself). Indeed, pictures often function in relation both to text in a localized visual system and to a particular genre in terms of a reader’s assumptions about and approaches to them.

I will here outline some of the ways in which pictures involve readers in integral experiences, especially as challenges that beg resolution, through an entirely user-centered perspective. I believe such a perspective is most productive for designers, design students, and scholars alike, to understand visual media in terms consistent with how it actually functions according to its intended use. It is one thing to consciously and reflectively interpret visual media in a formal critique, and another entirely to use it in order to learn or construct knowledge about something. Especially as it affects the practice of graphic design, a user-centered conceptualization of design should extend to the image. It is best to frame this line of inquiry in relation to philosophical precedents before the analysis of some examples of imagery as experience.

Precedents for the Study of Image Function

Literary theory and linguistics are precursors to the study of pictorial communication and meaning. I take a constructivist approach to the study of visual media. A constructivist literary theory of representation considers meaning to arise only out of symbolic practices (the act of interpretation)[2] A reader constructs knowledge; she is not given it, nor does she discover it as if it were waiting there for her. The constructivist approach stands in opposition to intentional and reflective approaches. The intentional approach, which generally preceded the constructivist approach, considers meaning in relation to authorial intent: a thing means what an author intended it to mean. Any other interpretation is wrong. The generally earlier reflective approach assumes that objects have inherent meaning irrespective of authors, readers, or the act of reading[3] In this way, relative to design and visual media, a reflective theory is media-centric and an intentional theory designer-centric, while a constructivist theory is user-centric, or perhaps more accurately use-centric.

Semiotics, the study of signs (which includes sounds, the written word, photographs, etc.), is a constructivist theory that largely addresses representation—how entities and concepts are related to indirect portrayals of them. Ferdinand de Saussure’s early scholarship on semiotics asserted that signs don’t have fixed meanings. Also, they are best understood in terms of contrasts: the essence of red lies in its distinction from green, while father has meaning only in relation to mother, daughter, etc[4] While Saussure was concerned exclusively with linguistics, soon after Charles Sanders Peirce offered an epistemological framework for semiotics that explicitly addressed pictorial representation. Roland Barthes built on Saussure’s work and addressed pictorial advertisements in terms of cultural negotiations. His deep analyses of print ads revealed layers of cultural production, as representations support and even construct culturally-dependent myths. Saussure and Peirce’s concerns are morerepresentational than experiential. Barthes’ analyses are complementary to the experiential analysis I am occupied with here, but his concerns are more broadly cultural than acutely interactive.

Literary critic Stanley Fish, though he addresses text and not image, operates with a mostly experiential approach, which serves as an example of constructivist reader-response theory[a] Fish conceives of the reader as the one who actually writes a text. “The text” in this sense is the reader’s unique (but not altogether unpredictable) construction, while the words on the page act as a kind of instigator. By his reckoning, “reading is not a matter of discovering what the text means, but a process of experiencing what it does to you.”[5]

These are loose philosophical precedents for the present work. Gunther Kress and Theo Van Leeuwen’s analysis of pictures and schematic arrangement[6] describes internal features of imagery as vectors while connecting representational strategies to the kinds of cultural negotiations that Barthes described. However, their focus isn’t squarely on reader activity, which differentiates their approach from Fish’s purely textual analysis.

Visual rhetoric is concerned with how pictorial features can persuade readers to think or even behave differently than they did before they read or viewed a given array of forms[b] This presupposes visual media as communication, while my concern with media in everyday use suggests situations that don’t fall under the generally accepted definition for communication. For instance, a reader can use a diagram to construct a holistic sense of a complex entity or phenomenon (e.g. multiple views of mushrooms over a life cycle), where the interaction involves her working through a resource more than receiving and then interpreting a message. Furthermore, proposals for models of visual rhetoric are unified in their basis in traditional verbal rhetoric. From working memory theory[7] and dual coding theory,[8] we know that humans have complementary but independent cognitive resources for processing information encoded linguistically and pictorially. The linguistic code is fundamentally distinct from the pictorial code: the former is serial, with grammar constrained by linear directionality and a proper “reading order,” while the latter is synchronous, with schematic relationships that enable more dynamic information access. Suffice it to say: imagery is not text. Many visual rhetoric proposals begin by seeking pictorial analogs to the schemes and tropes that were originally applied to describe language usage[c] Such visual rhetoric descriptions often strain to fit pictorial stimuli to established schemes and tropes. This method not only results in forced identification but is also likely to overlook possibilities unique to imagery.

Educational psychologist Joel Levin initiated a body of literature on “picture function” that is cognitively informed and serves as the most direct precedent for an experiential analysis of imagery[d] His original contribution presents a typology of functions,[9] and subsequent authors have either modified that typology or offered equivalent lists. Levin’s types are described exclusively in relation to text: they describe and locate “prose learning” functions. This limits their efficacy here, as pictures can also serve to structure integral experiences alone. Furthermore, his types tend to describe outcomes as opposed to activities. For instance, his remunerative function relates to a publisher’s sales, independent of the specific reader activity that might reside in or result from that outcome.

These philosophical precedents help to frame the reading of imagery, but there is work to be done regarding imagery as integral experience.

The Image in Relation to Text

Figure 1: Interactive picture–caption system. From T. E. Hill’s Hill’s Manual of Social and Business Forms: A Guide to Correct Writing (1879). Illustrator unknown. Public domain.

Before focusing on the image’s specific role in visual media, I want to demonstrate how text and image can become experiential by virtue of their intimate structuring. Children’s textbooks too often fail to achieve this effectively due to a production model that separates content creation from graphic design[9e] As a result, a textbook’s design has traditionally taken on a continuous prose format, with pictures that appear marginal to the text and are therefore likely to be ignored[f] In contrast to this lack of intimate structuring is Figure 1, Bad Manners at the Table, a picture–caption system that functions as interface despite being created circa 1878 for a book. What follows is a possible reading experience:

Note that nowhere within this construct did the visual media (here an illustrated page in a book) explicitly ask you to play a game. It merely provided conditions under which you were likely to engage in this type of activity, or a semblance of it. It modelled a kind of interaction by virtue of its structure. Consider how useful employing such a method might be for a textbook designer. Children typically read their textbooks under duress. A textual prompt to engage in gameplay is likely met with resistance by a child not invested in what the textbook is meant to offer, or who has developed a particular, narrowly informed mindset about what constitutes gameplay and what does not, and what gameplay “looks like.” But a structure akin to Bad Manners might increase involvement despite the child’s attitude.

The account described above must be understood as a possibility only, contingent on situational factors, including the psychological and sociocultural characteristics of the reader herself. All assertions regarding image function are tentative, and at best any function is probabilistic—it is a likelihood rather than a certainty. All imagery is inert without a reader’s investment, be it intentional or unwitting.

I will henceforth focus on the image’s contribution to such rich inter-actions (and more specifically with regard to interpretative difficulties). But first, in order to clarify through contrast, I will discriminate amongst pictures based on the presence or absence of such contribution.

The Boundary Conditions of Performative Imagery

I call pictures that demonstrably contribute to an integral experience by providing for rich user–media interactions performative—in the sense that the reader’s behavior is staged by a picture, as if imagery and reader were acting collaboratively. What the performative image accomplishes or models in an integrated visual system could be called interpretational processing, cognitive work, or, fundamentally, the construction of meaning.

I take my sense of performance loosely from linguistic philosopher J. L. Austin, who distinguished performative language from its more common counterpart, constative language[10] A constative utterance is a statement of fact, intended to reflect or report on reality. A performative utterance, in contrast, accomplishes something itself. For instance, in a socially negotiated situation, an inheritance is created when somebody declares to whom their possessions will pass, by virtue of the declaration itself. I propose that select pictures likewise accomplish something demonstrable.

Figure 2: Decorative and reiterative imagery. (a) Originals: O’Reilly’s Using Drupal and Programming PHP. (c) Reiterative and (d) decorative (in relation to book-context) section title imagery. Originals from A. R. Veri, W. W. Jenna, and D. E. Bergamaschi’s Environmental Quality by Design: South Florida (1975). Illustration by Sander Weeks, after the originals.

I want to define performativity in imagery partly through accrual, suggesting a category with its members (i.e. examples of performative imagery). But much imagery you encounter in your mediated life is at or beyond the boundary of this category. Consider, for instance, the book covers illustrated in Figure 2a[g] They feature a squirrel (Using Drupal) and a songbird (Programming PHP), though neither backyard animal bears any relation to its corresponding book’s content. While a free association exercise might “discover” a Drupal–squirrel relationship, such activity cannot be expected from most readers faced with a book cover. The same person who would get nothing out of Using Drupal’s imagery on Amazon might have a dissimilar experience with it in another situation. If this book were exhibited at an art museum, she would take on another role—let’s call her a critic in this case—and might actively seek out a resolution to the implicit question, “why are this picture and this book on display here?,” because she is acting in that more inquisitive mode. We approach media with agendas in part based on roles, and this delimits how imagery functions. In practice, though, while Using Drupal and Programming PHP pictures may help to sell the books, there is no discernable reader activity modelled here.

What defines such decorative imagery[h] is that it is irrelevant given the visual system within which it is placed. Such imagery, by design, fails to promote significant cognitive work because it does not reward it. This can be the case even when imagery is relevant. The two book section titles in Figure 2b and Figure 2c differ slightly: the illustration for Mangrove and Coastal Marsh depicts just that, while the Summary page is illustrated with a raptor. The latter picture is like the examples of Figure 2a, though it is less extreme in its irrelevance. But though the former is relevant, it does little more than its Summary counterpart. In this example, the picture functions as a background, something to accompany nearby text while adding nothing to it. If a designer creates or selects a picture based largely on appropriateness, she is likely producing reiterative imagery, which merely repeats something already asserted in its context (here in textual form). While this is not harmful, it is especially limited. The reiterative image is akin to Austin’s “constative utterance” in that it is entirely subservient to other components in its context. It “reflects the reality” of other elements, likely text. (By definition, a picture presented in isolation cannot be reiterative—without something to reiterate, expressed through other components in a context, a picture must function in some other manner.)

Imagery that is neither decorative nor reiterative performs.

When Imagery Is a Challenging Partner in the Reader’s Endeavor

When a reader constructs meaning with a picture (when they perform together), she is, in one way or another, removing an incongruity presented to her. I am not referring to perceptual processes, where we recognize scenes holistically and pull figures forth from a cacophony of color and line with immediacy. Meaning construction happens closer to the surface, at or near the level of conscious thought. When faced with a book titled “Taking Care,” illustrated with a beach scene featuring a palm tree laden with snow above proper sand, and a white rabbit in the foreground, perception is not enough (Figure 3a). The components—“taking care,” beach scene, snow, and rabbit—are catalogued adequately through the perceptual apparatus, but the challenge they pose in aggregate is another matter. Whereas Using Drupal seemed straightforward enough for its squirrel to be dismissed, Taking Care begs interpretation. The reader senses a gesture from the designer or illustrator. The illustration, then, is provisionally considered to be nonrandom, and thus promises some kind of meaning. (Where is that meaning? Our folk conceptualization would suggest it is in the object, waiting to be extracted. But we know better. Meaning is constructed on the spot by a person, and media simply mediates.)

Figure 3: Problematic imagery. (a) Cover of Joy Williams’ Taking Care. Illustration by Sander Weeks, after the original. (b) Dennis the Menace cartoon. Original, with different caption, by Hank Ketcham, © 1989 North America Syndicate, Inc., used with permission. This caption was written by Gary Larson as intended for his Far Side cartoon (featuring a taxonomist).

In the present example, the incongruity exists at the threshold of perception and cognition. We are able to process scenes quickly in part because we have schemata for them[11] A schema is an organized mental representation that is learned and called to mind most efficiently as we continually make sense of the world’s endless complexity. We have schemata for any number of scenes: the office, the classroom, the beach. Both snow and rabbit are “violations” of our beach schema, i so the reader is taken aback as a perceptual hiccup suddenly excites cognition. Does it matter if there is no resolution? I myself cannot “solve” the image except to assume that it is a dream or is otherwise surreal and thus inscrutable. Basically, I give up; but at least I tried. The point, perhaps, is not the outcome but the attempt. This image, precisely because it is problematic, inspires deep consideration from the reader.

This process is quite like humor, even when, as in the case of Taking Care, the discourse isn’t funny. According to psychologist Jerry Suls’ two-stage model of incongruity resolution (a theory of humor), j a joke builds expectations and then presents a punchline, one that is initially incompatible with those expectations. Consummation occurs in a denouement stage of processing and reconciliation, where the reader recognizes a new rule that permits her to remove any internal contradiction when all aspects are considered together[12] Taking Care’s illustration is problematic in the sense that all humor is problematic—as something to be solved or resolved.

Another example of problematic imagery is the cartoon of Figure 3b, which, if you know the social and cultural conventions of Dennis the Menace, is entirely disorienting. Dennis looks at his mom and says something sinister, and the reader struggles to discover the rule that will resolve the incongruity between picture and caption. There is no definitive resolution, however, because this particular cartoon was actually a newspaper typesetter’s error: the caption was intended for that day’s Far Side cartoon, a cartoon with entirely different social and cultural conventions (through which the Far Side’s readership has developed specific expectations). While all single-frame cartoons could be considered problematic according to Suls’ two-stage theory of humor, this erroneous example of Dennis the Menace is explicitly so.

Figure 4: Linguistic and creative imagery. (a) Original from I. Thomas’s The Hieroglyphick Bible (1814). Pictures are positioned as syntactical interruptions to text and thus relate to it. Translation equalizes the representational system. (b) Original from AARP the Magazine, Oct ./ Nov. 2012 (photography by Adam Voorhes). (c) Original Comedy Central ad by Grey (advertising agency, Argentina), Diego Medvedocky (executive creative director), Antoine Quairiat (illustrator). Illustrations by Brian Lorenz, Scott Durand, and Sander Weeks, respectively, after the originals.

In both of the problematic examples depicted above, language helps define the context in which the picture operates. Language can also be internal to imagery. Both Figure 4a, from The Hieroglyphick Bible, and Figure 4b, from AARP the Magazine, ask the reader to produce language directly from imagery. In the latter, the linguistically-derived concept of “business incubator” must be activated to resolve the incongruity. These are examples of linguistic imagery.

In addition to language, readers will develop their own imagery in response to supplied pictures. The Comedy Central ad of Figure 4c, like problematic and linguistic imagery, is incomplete until a reader provides resolution. A physical gag awaits the reader who projects the image of the ostrich head into each of two holes. This is thus creative imagery. Here the active role of the reader is most pronounced. The illustrator or designer essentially prepares an incomplete picture. In a constructivist sense, this is a powerful method. It is not enough here to be passive; the content is not given to you. A constructivist view of education emphasizes the learner’s active role in constructing knowledge for herself. Constructivists contend that we learn best when we do something.

In problematic, linguistic, and creative imagery an experience takes shape through the reader’s active engagement in a process of interpretation. The performance ends (alas with no applause) when the reader feels her experience is naturally and satisfactorily fulfilled, or when she otherwise simply abandons it.

Visual Metaphor

Another form of incongruity in imagery is visual metaphor, which has been described in rhetoric as an “artful deviation” from expectations[13] When a reader engages with metaphorical construction in visual media, she must recognize imagery as a form of communication—not always a given in everyday reading—in order to reach a resolution of understanding. This means she perceives the designer’s gesture. Gesture as a variable is explored in Figure 5, which presents two versions of a metaphorical illustration. Though this picture of a pipeline-wielding titan bursting forth from a pasture is shown here out of context, it nevertheless seems obvious that it is not to be taken literally. The titan represents industry, the cattle and their pasture represent agriculture, and industry thus overtakes agriculture through the depicted action. In the original version (Figure 5a) the cattle are not responsive to the titan’s interruption. Their inert appearance reads as symbolic more than iconic. In the altered version (Figure 5b) some cows are tossed while others flee (as they should), and this more natural depiction makes it slightly less likely that the picture will read as metaphor, and slightly more likely that it will read as science fiction. The designer’s gesture makes an overt request of the reader.

Figure 5: Metaphorical imagery with (a) greater and (b) lesser degrees of gesture. Original from Time Magazine, 4/11/55 (original illustrator unknown). Illustration by Scott Durand, after the original.

Of course, to recognize a gesture and process imagery metaphorically requires an existing apparatus. Though known as a rhetorical trope, metaphor is not simply some manipulation of language and imagery. It is a fundamental way in which we humans make sense of the world, even when we are not involved in any form of communication. Metaphor plays a major role in our basic conceptual system[14] As we humans develop in and from infancy, we relate new phenomena to past experiences, finding similarities with which we can work to help us better understand the workings of the world around us[15] Any time a person attempts to communicate to another through metaphor, she is relying on a basic but accrued human proficiency.

In any metaphor, one concept or entity is related to another. (Visual metaphor, in practice, is mostly limited to comparing entities, as concepts cannot be shown directly.) A target entity is understood in terms of a source entity. The designer might use metaphor to help a reader understand an unfamiliar target in terms of a familiar source. But metaphor can also be used to cast a known entity in a new light. The comparison of metaphor is achieved through a mapping process, where select attributes of the source are mapped onto the target. (Structure can also be mapped instead of attributes[16] ) Not all attributes are mapped—the reader determines which amongst all possible attributes are germane based on her perception of contextual cues. For instance, if I say “he is a bear of a man,” you will not assume the person to which I refer hibernates for the winter or has fearsome claws. (This is a familiar metaphor, but we readily handle novelty as well.)

These basic components—source, target, and attributes or structure—are part of any metaphor. But because language and imagery are so different, and are handled through separate but complementary components in the brain,[17] we should not expect particular aspects of verbal metaphor to correspond directly to visual metaphor. That is because each is an expression of an underlying capacity for conceptual metaphor—verbal metaphor arises from conceptual metaphor independently, as visual metaphor arises from conceptual metaphor.

The differences in such expressions are particular to the dual linguistic and pictorial codes themselves. The serial nature of language and the directionality of referents means that source–target relationships are explicit: the form of “this is a that” as a verbal utterance positions target ahead of source in a conventional and explicit presentation. In imagery, where positional relations are manifold, assigning source and target roles to entities requires greater contextual awareness.

Figure 6: Metaphorical imagery with varied source–target relationships. (a) Original Saab USA ad by unknown agency. (b) Original Katfa ad by SIVA (agency, Cairo), Amr Ossama (creative and art director, photographer). (c) Original Welch’s ad by unknown agency, found in: Phillips and McQuarrie, Beyond Visual Metaphor. Illustrations by Scott Durand (a & b) and Eric Pryor, after the originals.

Figure 6 presents three types of source–target relationships in visual metaphor that have no direct verbal corollaries, as proposed by advertising researchers Barbara J. Phillips and Edward McQuarrie[k] In Figure 6a, both source and target are pictured, and are related by the reader through contextual factors. This is a juxtaposition structure. Note that this particular example is a natural scene, yet familiarity with the advertising genre and a propensity to find order in stimuli together help the reader render the picture as a form of communication. In Figure 6b, a new hybrid entity is created that explicitly combines source and target. This is a fusion structure. Here the designer’s gesture is most overt, and less is required of the reader to apprehend the relationship. The fusion structure renders the source–target relationship unequivocal, but it complicates the perceptual process of isolating the two entities. In contrast, the juxtaposition structure in no way interferes with the perceptual process—car and aircraft are both complete. However, the source–target relationship is subtle and the reader must work harder to apprehend it.

In Figure 6c the target is entirely absent, but the reader has a sense of it nonetheless from environmental features. This is a replacement structure. In a capacity not dissimilar from that required of creative imagery, the reader must imagine an entity that is never pictured. In juxtaposition and fusion structures, apprehension of the second entity is a perceptual process. Here it is a cognitive process. As the reader imagines the missing entity, she does so already with a sense of a source–target relationship. These structural distinctions thus have implications for processing stages. This is but a hint at the formal issues at play in visual metaphor.

When the interest is in a reader’s interpretational experience—a learning experience, always—such subtle distinctions may very well matter. In an advertising context, it may be desirable to design metaphorical imagery that is processed as quickly as possible. In an educational context, it may be desirable to do just the opposite, to more deeply engage a learner in extended interpretative activity.

As with humor, the reader derives pleasure when she solves a metaphor. Bad Manners was a game, and so is every picture that offers the potential for metaphorical elaboration. With that stated, it should be understood that visual metaphor is not simply a means to please readers. Metaphorical imagery is a way to learn and can thus be qualified as utilitarian and, ultimately, performative.

Conclusion

The functions described above (problematic, linguistic, creative, and metaphorical) articulate how imagery can present its reader with a challenge. Imagery performs in other ways as well, be it as an external workspace in which the reader builds conceptualizations and makes comparisons,[l] or as evidence of the passage of time within which the reader constructs a narrative. These are some of the other things the illustrator and designer can stage with pictures.

I have tried here to demonstrate how imagery in media can model interpretational processes, and how much the efficacious design of pictures depends upon an honest emphasis on the reader as an active constructor of knowledge. Who is this reader? We must first recognize that she is a person, a person, a person: it bears repeating. I am skeptical of what her demographic might suggest. It is certainly important to recognize that she belongs to a culture and any number of subcultures, and that this contextualizes and even defines her world[m] She takes on roles. When I refer to this person as a reader, I mean to emphasize her active participation in realizing imagery and texts. (She does the work; the designer merely provides resources.) But she also behaves as a user when she makes a tool of a product and cares not who designed it, a critic when she walks into a gallery and hopes to perceive an artist’s way of thinking, a learner when she is suddenly aware of her ongoing construction of meaning through interactions with the world, and any number of other things in her extensive repertoire based on her present situation.

We humans have been making performative imagery to communicate with and to convey meaning to one another for eons. Our earliest writing systems emerged out of imagery. Advertisers use imagery’s performative capacity to convince us of something, and sometimes (or too often) to mislead us. Educators use it (too infrequently) to empower us. It is all too possible to overlook the potential of the image as a designer, especially when we are preoccupied with the appropriateness of selections. Appropriateness leads us to specify imagery that merely reiterates content. I mean to stress that the reader’s activity is what matters, and that this is the metric by which visual media should be assessed. While such a stance is perhaps common enough, the details of this exposition may help to actualize it where pictures are concerned.

Imagery is experience—at least at its best, and only with a consenting reader.

References

• Austin, J.L. How to Do Things with Words. Cambridge, MA, USA: Harvard University Press, 1955 / 1975.
• Baddeley, A. “Working Memory.” Comptes Rendus de l’Académie des Sciences 321.2–3 (1998): pgs. 167–173.
• Boyce, S.J., & Pollatsek, A. “An Exploration of the Effects of Scene Context on Object Identification,” in Eye Movements and Visual Cognition: Scene Perception and Reading, edited by Keith Rayner, pgs. 227–42. New York, NY, USA: Springer-Verlag, 1992.
• Carney, R.N., & Levin, J.R. “Pictorial Illustrations Still Improve Students’ Learning from Text.” Educational Psychology Review 14.1 (2002): pgs. 5–26.
• Carroll, P.J., Young, J.R., & Guertin, M.S. “Visual Analysis of Cartoons: A View from the Far Side,” in Eye Movements and Visual Cognition: Scene Perception and Reading, edited by Keith Rayner, pgs. 444–71. New York, NY, USA: Springer-Verlag, 1992.
• Dewey, J. Art as Experience. New York, NY, USA: Capricorn Books, 1934 / 1958.
• DiGiuseppe, M. “Representing Nature of Science in a Science Textbook: Exploring Author-Editor-Publisher Interactions.” International Journal of Science Education 36. 7 (2014): pgs. 1061–1082.
• Eagleton, T. Literary Theory: An Introduction. Minneapolis, MN, USA: University of Minnesota Press, 1996.
• Fish, S. “Interpreting the Variorum.” Critical Inquiry 2.3 (1976): pgs. 465–85.
• Gentner, D., & Gentner, D.R. “Flowing Waters or Teeming Crowds: Mental Models of Electricity,” in Mental Models, edited by Dedre Gentner and Albert L. Stevens, pgs. 99–129. Hillsdale, NJ, USA: Lawrence Erlbaum Associates, Publishers, 1983.
• Hall, S. “The Work of Representation,” in Representation: Cultural Representation and Signifying Practices, edited by Stuart Hall, pgs. 15–64. London, UK: Sage Publications, 1997.
• Johnson, M. The Body in the Mind. Chicago, IL, USA: University of Chicago Press, 1987.
• Kress, G., & Leeuwen, T.V. “Narrative Representations: Designing Social Action,” in Reading Images: The Grammar of Visual Design, edited by Gunther Kress and Theo Van Leeuwen, pgs. 45–73. New York, NY, USA: Routledge, 2006.
• Lakoff, G., & Johnson, M. Metaphors We Live By. Chicago, IL, USA: University of Chicago Press, 2003.
• Levin, J.R. “On Functions of Pictures in Prose,” in Report from the Project on Studies in Language: Reading and Communication. Madison, WI, USA: Wisconsin Research and Development Center for Individualized Schooling, 1979.
• Levin, J.R., & Mayer, R.E. “Understanding Illustrations in Text,” in Learning from Textbooks: Theory and Practice, edited by Bruce K. Britton, Arthur Woodward, and Marilyn Binkley, pgs. 95–113. Hillsdale, NJ, USA: Lawrence Erlbaum Associates, Publishers, 1993.
• Lupton, E., & Ehses, H. Rhetorical Handbook: An Illustrated Manual for Graphic Designers. Halifax, Nova Scotia, Canada: Nova Scotia College of Art and Design, 1988.
• Mandler, J.M. Stories, Scripts and Scenes: Aspects of Schema Theory. Hillsday, NJ, USA: Lawrence Erlbaum Associates, Publishers, 1984.
• McQuarrie, E.F., & Mick, D.G. “Figures of Rhetoric in Advertising Language.” Journal of Consumer Research 22.4 (1996): pgs. 424–438.
• McQuarrie, E.F., & Phillips, B.J. “Indirect Persuasion in Advertising: How Consumers Process Metaphors Presented in Pictures and Words.” Journal of Advertising 34.2 (2005): pgs. 7–20.
• Peterson, M.O. “The Integration of Text and Image and Its Impact on Reader Interest.” Visible Language 48.1 (2014): pgs. 22–39.
• Pettersson, R. “Image Functions in Information Design,” in The 30th Annual Conference of the International Visual Literacy Association. Athens, GA, USA: University of Georgia, 1998.
• Phillips, B.J. “Understanding Visual Metaphor in Advertising,” in Persuasive Imagery: A Consumer Response Perspective, edited by Linda M. Scott, and Rajeev Batra, pgs. 297–310. Mahwah, NJ, USA: Lawrence Erlbaum Associates, Inc., 2003.
• Phillips, B.J., & McQuarrie, E.F. “Beyond Visual Metaphor: A New Typology of Visual Rhetoric in Advertising.” Marketing Theory 4 (2004): pgs. 113–36.
• Rosenblatt, L.M. The Reader, the Text, the Poem: The Transactional Theory of the Literary Work. London, UK: Sage Publications, 1978.
• Sadoski, M., & Paivio, A. Imagery and Text: A Dual Coding Theory of Reading and Writing. Mahwah, NJ, USA: Lawrence Erlbaum Associates, 2001.
• Scott, L.M. “Images in Advertising: The Need for a Theory of Visual Rhetoric.” Journal of Consumer Research 21.2 (1994): pgs. 252–273.
• Scott, L.M. “The Bridge from Text to Mind: Adapting Reader-Response Theory to Consumer Research.” Journal of Consumer Research 21.3 (1994): pgs. 461–80.
• Plass, J.L., Moreno, R., & Brünken, R. Cognitive Load Theory. Cambridge, UK: Cambridge University Press, 2010.
• Vekiri, I. “What is the Value of Graphical Displays in Learning?” Educational Psychology Review 14.3 (2002): pgs. 261–312.
• Woodward, A. “Do Illustrations Serve an Instructional Purpose in U. S. Textbooks?” in Learning from Textbooks: Theory and Practice, edited by Bruce K. Britton, Arthur Woodward, and Marilyn Brinkley, pgs. 115–134. Hillsdale, NJ, USA: Lawrence Erlbaum Associates, Publishers, 1993.

Biography

Matthew Peterson (mopeters@ncsu.edu) is Assistant Professor of Graphic Design at North Carolina State University, where he received his Ph.D. in Design in 2011. Prior to his graduate studies, he was a professional web designer in Raleigh, NC, was Design Fellow at the Walker Art Center in Minneapolis, and ran the design studio Field Study in partnership in Chicago. Matthew’s research investigates the interpretation and use of integrated text and image in visual and interactive media. His dissertation addressed issues of comprehension in the science textbook according to the degree to which text and image are integrated. He continues to develop a comprehensive taxonomy of performative image function, which tracks the methods available to designers to produce specific cognitive outcomes. His recent work has focused on instructional design in science education (including visualization strategies for size and scale cognition) and the examination of visual metaphor in advertising. He is also exploring speculative interface design, which he’s using to investigate how information presentation and structure might change with new technologies, such as using eye-tracking as an input device. This work is oft theoretical, oft empirical. Matthew also developed undergraduate and graduate curricula in graphic design while teaching at the University of Illinois.

Notes

• a

  Eagleton, T. Literary Theory: An Introduction. Minneapolis, MN, USA: University of Minnesota Press, 1996, pgs. 74–77. For an example of Fish’s program, see: Stanley E. Fish, “Interpreting the Variorum,” Critical Inquiry 2.3 (1976): pgs. 465–85. For an explication of reader-response theory, see: Louise M. Rosenblatt, The Reader, the Text, the Poem: The Transactional Theory of the Literary Work. London, UK: Sage Publications, 1978. For its application in advertising, see: Linda M. Scott, “The Bridge from Text to Mind: Adapting Reader-Response Theory to Consumer Research,” Journal of Consumer Research 21. 3 (1994): pgs. 461–80.

• b

  As persuasion, visual rhetoric has enjoyed much attention in advertising. See: Linda M. Scott, “Images in Advertising: The Need for a Theory of Visual Rhetoric,” Journal of Consumer Research 21.2 (1994): pgs. 252–273. See also: Edward F. McQuarrie and Barbara J. Phillips, “Indirect Persuasion in Advertising: How Consumers Process Metaphors Presented in Pictures and Words,” Journal of Advertising 34.2 (2005): pgs. 7–20.

• c

  For an example in advertising, see: Edward F. McQuarrie and David Glen Mick, “Figures of Rhetoric in Advertising Language,” Journal of Consumer Research 22.4 (1996): pgs. 424–438. For an example directed at graphic designers, see: Ellen Lupton and Hanno Ehses, Rhetorical Handbook: An Illustrated Manual for Graphic Designers. Halifax, Nova Scotia, Canada: Nova Scotia College of Art and Design, 1988.

• d

  Levin, J.R. “On Functions of Pictures in Prose,” in Report from the Project on Studies in Language: Reading and Communication. Madison, WI, USA: Wisconsin Research and Development Center for Individualized Schooling, 1979. For a continuation of this work, see: Joel R. Levin and Richard E. Mayer, “Understanding Illustrations in Text,” in Learning from Textbooks: Theory and Practice, edited by Bruce K. Britton et al, pgs. 95–113. Hillsdale, NJ, USA: Lawrence Erlbaum Associates, Publishers, 1993,. See also: Russell N. Carney and Joel R. Levin, “Pictorial Illustrations Still Improve Students’ Learning from Text,” Educational Psychology Review 14.1 (2002): pgs. 5–26.

• e

  DiGiuseppe, M. “Representing Nature of Science in a Science Textbook: Exploring Author-Editor-Publisher Interactions,” International Journal of Science Education 36.7 (2014): pgs. 1061–1082. For implications on the utility of pictures in textbooks, see: Rune Pettersson, “Image Functions in Information Design,” in The 30th Annual Conference of the International Visual Literacy Association. Athens, GA, USA: University of Georgia, 1998. See also: Arthur Woodward, “Do Illustrations Serve an Instructional Purpose in US Textbooks?,” in Learning from Textbooks: Theory and Practice, edited by Bruce K. Britton et al, pgs. 115–134. Hillsdale, NJ, USA: Lawrence Erlbaum Associates, Publishers, 1993.

• f

  For a response to the marginalization of imagery in textbooks by defining alternate layout models, see: Matthew O. Peterson, “The Integration of Text and Image and Its Impact on Reader Interest,” Visible Language 48.1 (2014): pgs. 22–39.

• g

  These and subsequent illustrations were supported with a Creative Research Award from the College of Fine and Applied Arts at the University of Illinois at Urbana-Champaign. Special thanks to Xuxa Rodriguez for her help with image search and documentation.

• h

   Levin identifies both decorative and reiterative image functions. Levin, “On Functions of Pictures.”

• i

   It is easier to process an object in its “natural” setting than in an unnatural one. Susan J. Boyce and Alexander Pollatsek, “An Exploration of the Effects of Scene Context on Object Identification,” in Eye Movements and Visual Cognition: Scene Perception and Reading, edited by Keith Rayner, pgs. 227–42. New York, NY, USA: Springer-Verlag, 1992.

• j

   Cited in: Patrick J. Carroll, Jason R. Young, and Michael S. Guertin, “Visual Analysis of Cartoons: A View from the Far Side,” in Eye Movements and Visual Cognition: Scene Perception and Reading, edited by Keith Rayner, pgs. 444–71. New York, NY, USA: Springer-Verlag, 1992. The example of Figure 3b was discovered by Carroll et al. and served as the inspiration for their reported study.

• k

  Phillips, B.J. and McQuarrie, E.F. “Beyond Visual Metaphor: A New Typology of Visual Rhetoric in Advertising,” Marketing Theory 4 (2004): pgs. 113–136. Research in advertising over two decades has resulted in this typology of visual metaphor structure, though others continue to propose alternatives.

• l

  Exploratory, constitutive, comparative, and computational functions fall under this particular description.

• m

  Stanley Fish invokes interpretive strategies and interpretive communities to salvage our ability to discuss a text despite an acknowledgment of individual differences (that is, as an escape from pure subjectivism, which would allow that every reader's experience is unique and thus disable any true criticism or analysis). See:: Fish, "Interpreting the Variorum."

1. Dewey, J. Art as Experience. New York, NY, USA: Capricorn Books, 1934 / 1958.

2. Hall, S. “The Work of Representation,” in Representation: Cultural Representations and Signifying Practices, ed. by Stuart Hall, pgs.24-26. London: Sage Publications, 1997.

3. Hall, “Work of Representation,” pgs. 24–26.

4. Hall, “Work of Representation,” pgs. 31–32.

5. Eagleton, Literary Theory, p. 74.

6. Kress, G. and Leeuwen, T.V. “Narrative Representations: Designing Social Action,” in Reading Images: The Grammar of Visual Design, ed. by Gunther Kress and Theo Van Leeuwen. New York, NY, USA: Routledge, 2006, pgs. 45–73.

7. Baddeley, A. “Working Memory,” Comptes Rendus de l’Académie des Sciences 321.2–3 (1998): pgs. 167–173.

8. Mark Sadoski, M. and Paivio, A. Imagery and Text: A Dual Coding Theory of Reading and Writing. Mahwah, NJ, USA: Lawrence Erlbaum Associates, Publishers, 2001.

9. Levin, “On Functions of Pictures.”

10. Austin, J.L. Edited by Urmson, J.O. & Sbisá, M. How to Do Things with Words, 2nd Edition. Cambridge, MA, USA: Harvard University Press, 1975.

11. Mandler, J.M. Stories, Scripts and Scenes: Aspects of Schema Theory. Hillsday, NJ, USA: Lawrence Erlbaum Associates, Publishers, 1984.

12. Carroll et al., “Visual Analysis of Cartoons,” p. 445.

13. Phillips, B.J. “Understanding Visual Metaphor in Advertising.” In Persuasive Imagery: A Consumer Response Perspective, edited by L.M. Scott and R. Batra, pgs. 297–310. Mahwah, NJ, USA: Lawrence Erlbaum Associates, Inc., 2003.

14. Lakoff, G. and Johnson, M. Metaphors We Live By. Chicago, IL, USA: University of Chicago Press, 2003.

15. Johnson, M. The Body in the Mind. Chicago, IL, USA: University of Chicago Press, 1987.

16. Gentner, D. and Gentner, D.R. “Flowing Waters or Teeming Crowds: Mental Models of Electricity,” in Mental Models, edited by D. Gentner and A. L. Stevens, pgs. 99–129. Hillsdale, NJ, USA: Lawrence Erlbaum Associates, Publishers, 1983.

17. Sadoski and Paivio, Imagery and Text.