Kitt Healy, University of Wisconsin-Madison
Lane Selman, Oregon State University
Alex Stone, Oregon State University
Julie Dawson, University of Wisconsin-Madison
Consumers expect superior flavor, texture, and appearance from fresh produce. This is especially true of produce grown organically and purchased from local farms (Bonti-Ankomah and Yiridoe, 2006). Farmers selling direct to local markets value these qualities when choosing varieties, but they may not have a way to systematically evaluate them. This article discusses simple methods for doing culinary quality evaluations, so farmers can make informed decisions about the best varieties for their farms and markets. In this article, we define culinary quality as flavor, texture, appearance, and any additional desirable attributes for cooking or fresh eating, which may vary by crop.
Although many vegetable breeders consider flavor when developing vegetable varieties, end users are not often engaged in the evaluation process. This means that breeders may miss out on particular traits of interest to Community Supported Agriculture (CSA) customers, chefs, or other local food consumers. Gathering information on which varieties customers prefer not only helps farmers choose varieties, it also generates important feedback for plant breeders working in organic systems. More farmers trialing and evaluating the culinary quality of vegetables means more information to guide breeding decisions. The information you gain from a quality evaluation can be shared with other farmers and your customers to help you choose varieties. It can also be shared with your seed sources, particularly if the seed companies you work with are connected to the breeders of those crops. Many of the independent seed companies serving organic systems are either involved in plant breeding or are in close connection to the breeders of the seed they are selling. They are very interested in getting feedback on culinary quality!
The methods in this bulletin were developed by two programs working with end users to evaluate the culinary quality of vegetables and small grains: The Seed to Kitchen Collaborative (SKC) and the Culinary Breeding Network (CBN). Both organizations draw on recent research in rapid sensory evaluation, reviewed in Dawson and Healy (2018) and several years of experience testing and refining methodology with chefs and farmers.
The Seed to Kitchen Collaborative is based in Madison, Wisconsin and is a program of the Urban and Regional Food Systems Group in the Horticulture Department at the University of Wisconsin, Madison. Founded in 2014, SKC conducts vegetable variety trials under organic management on research stations in Wisconsin and works with farmers in the Upper Midwest to evaluate varieties in on-farm trials. As part of the trials, SKC staff evaluate flavor in all varieties and coordinate monthly tasting events with a group of 7 to 10 chefs. Each event focuses on one or two crops, from fresh tomatoes in August, to storage potatoes in March. Criteria for the evaluations are developed collaboratively with the chefs and plant breeders or the seed companies who contributed varieties.. All the produce for the evaluations comes from university-led variety trials conducted at the West Madison Agricultural Research Station. Culinary quality and agronomic information from the trials are distributed to plant breeders, seed companies and local farmers to help farmers choose the best varieties for their farms. Each year in the late summer or early fall, SKC hosts an annual event, called Farm to Flavor, which highlights outstanding varieties from the program. More information on SKC and Farm to Flavor can be found at https://seedtokitchen.horticulture.wisc.edu/.
The Culinary Breeding Network is based in Portland, Oregon and was founded by Lane Selman of Oregon State University in 2010. Selman noticed that while plant breeders often evaluate in-progress varieties for flavor, they do not usually involve end users in the process. After conducting an inaugural culinary quality evaluation of sweet peppers involving Portland-area chefs, farmers and plant breeders, she launched CBN. Her goal is to ensure that breeders benefit from the perspectives of farmers, chefs and eaters, and farmers and culinary professionals benefit from varieties selected with them in mind. CBN hosts an annual event called the Variety Showcase at which chefs, bakers and brewers partner with plant breeders to highlight new and novel varieties in dishes designed for the event, which is open to the public. More information on CBN can be found at https://www.culinarybreedingnetwork.com/.
Raw and pureed samples of winter squash at a Culinary Breeding Network event. Photo credit: Shawn Linehan.
How to Set Up an Evaluation and Who to Involve
There are many ways to evaluate culinary quality, from informal taste tests with your crew in the field, to more formal evaluations with a group of culinary professionals. No matter where your evaluations fall on this spectrum, controlling for certain variables will help ensure reliable results.
Many of the same environmental factors that influence the productivity of crop plants can also influence culinary quality. These include temperature, rainfall, soil quality, wind direction and pest or disease pressure. It is important to design culinary quality evaluations to account for these factors. First, try to evaluate crops at peak maturity and during a stretch of average weather. If you evaluate sweet corn that is slightly past the harvest window, or tomatoes after a big rainstorm, the flavor and quality will not be representative of how that crop performs under ideal or typical conditions.
Second, it is important to account for variation within the field. At their research station, SKC uses a randomized complete block design in our trials to account for field variation. If you are interested in doing a variety trial on your farm in addition to a culinary quality evaluation, there is more information in the The Growers’ Guide to Conducting On-Farm Variety Trials (2018). Culinary quality evaluations can be done on varieties you are growing for market that are not part of a variety trial on your farm; however, to ensure that you are evaluating flavor differences due to variety and not due to soil type or management, all the varieties should come from the same field. Pick samples of the same variety from multiple parts of the field and combine them to ensure that no single fruit, root, or leaf is considered singularly representative of the variety being evaluated. Take note of the parts of the field where water tends to collect, where the soil may be more compact, or where pest or disease pressure is particularly bad. Be sure to avoid focusing your evaluation on these spots.
Once samples are collected, the evaluation process should be designed to be as objective as possible. There are several ways to do this. First, it is a good idea to disguise the names of the varieties you are evaluating. Variety names are often catchy, so evaluators may subconsciously favor a variety if they like the name. Labeling varieties with a three-letter code instead of the variety name can prevent name bias. Just be sure not to lose the key linking codes to variety names! It’s best to avoid common three letter acronyms like IRS or OMG. These can also induce unconscious bias. Second, have evaluators try samples in a random order. People often have stronger reactions to the first variety they taste than to subsequent varieties. Having all the evaluators try samples in a different order reduces the influence of tasting order on results. Third, ask evaluators not to express their opinions in any way until the evaluation is over. Hearing how a friend feels about a variety or seeing someone scrunch up their face can easily sway an evaluator’s opinion. Offering earplugs to evaluators is a good way to keep them from talking to each other or getting distracted by ambient noise.
In general, it is a good idea to make the evaluation environment as comfortable as possible, so evaluators can focus on each sample. Evaluators should be in good health on evaluation day; a stuffy nose will greatly affect a person’s ability to taste.
Pepper samples are prepared and presented uniformly to minimize evaluator bias. Photo credit: Shawn Linehan.
Involving the End User
Flavor evaluations are a great way to generate valuable information while also conducting outreach to current and potential customers. Doing a taste test at a farm tour or farmers market can generate enthusiasm among general customers. Engaging culinary professionals in an evaluation can help farmers select particular varieties for high-end markets and potentially build new sales relationships. Who participates in an evaluation will influence how the evaluation is structured. As a general rule, a baseline of reliable information can be obtained either from a large group of untrained evaluators, or a smaller group of professional evaluators (Frost et al., 2015). In a large public group, extreme data points will cancel each other out to reveal general trends. In a small professional group, much more precision can be expected from tasters, so fewer data points are needed to make trends apparent. A large group of professional evaluators is, of course, ideal! The Culinary Breeding Network often involves chefs, plant breeders and experienced produce growers together in evaluations.
Chefs are highly experienced tasters, as well as potential local food customers. Take advantage of chefs’ expertise by evaluating traits important to them. For example, the general public might be interested in how a winter squash tastes roasted, but chefs might also be interested in the taste and texture of a raw, shaved squash for salads. Chefs might also be interested in single serving sized fruits, or more powerful flavors (like bitter greens) and their preferences may influence culinary trends.
Chefs, farmers and plant breeders evaluate tomato and pepper varieties at a Culinary Breeding Network tasting event for the Northern Organic Vegetable Improvement Collaborative. Photo credit: Shawn Linehan.
Farm to table restaurants and chefs interested in supporting local farmers have had a large impact on the local food movement. Understanding the end users’ priorities will help make evaluation criteria maximally applicable to real world scenarios. The chefs involved in SKC and CBN are interested in finding varieties that work for both their restaurants and the local farmers they purchase from. They do not want a variety that has excellent flavor but is difficult or expensive to grow locally. They have been key stakeholders supporting varieties that combine productivity and culinary quality.
Regardless of the audience, it is important to make the evaluation setting clean, streamlined and attractive with as little variation between samples as possible. Using identical bowls or plates (white or colorless/clear is best; avoid colors or patterns) to display samples and cutting samples in to bite-size pieces in the same way for each variety will reduce bias. Providing clean toothpicks for picking up samples, and offering water and unsalted crackers as a palate cleanser will also enhance evaluators’ abilities to focus on the task at hand. See the reference list for helpful guidelines on food safety considerations when preparing samples.
It is important to taste the crop in the manner in which people are accustomed to eating it. For example, sweet corn tastes great in the field raw, but most consumers eat it boiled a few days after harvest. Varieties can taste and perform quite differently in different applications. When tasting red bell peppers, for example, you would ideally taste them raw, sautéed, and roasted, as all three preparations are common. However, consider the amount of labor required to prepare samples and whether your evaluators can handle tasting that many samples. “Palate exhaustion” is a real concern, especially when working with the public rather than culinary professionals. You may want to focus on just one preparation at a tasting. In addition to the potential for palate exhaustion, any type of cooking takes a great deal of care to ensure that all samples are treated uniformly. Slightly different times in boiling or roasting can drastically change the texture and overall qualities of prepared samples. If cooking, it is best to have someone experience do the preparation, or practice ahead of time. A rule of thumb is to evaluate 6 varieties at a time, with no more than 5 questions about each, in 1 to 2 preparations.
Description vs. Preference
At the beginning of an evaluation event, the host will provide each participant with an evaluation sheet that includes questions aimed at obtaining the desired information about the varieties in question. If the evaluation event is a one-off, it may not be necessary to collect evaluators’ names, but it is necessary to be able to link evaluation sheets from each individual if there are separate sheets for different samples. If the same evaluators will be involved in recurring evaluations, it is important to have the names or some form of identification on the evaluation sheets so you can see whether one evaluator consistently differs from the rest of the group. This may indicate that the one evaluator tastes things differently from the general populace or doesn’t fully understand what is being asked. If it becomes clear that one evaluator is dramatically shifting the trends of the group, consider removing this person’s data from the final analysis. If two or three people differ from the group, they may represent a segment of the population that tastes differently or has different preferences. This can be the case if there are different cultural backgrounds or ages of evaluators. In this case it is important to consider the preferences of the different groups rather than just taking an average across all evaluators.
Culinary quality evaluations fall into two general categories: descriptive and preference-based or “hedonic.” Descriptive evaluations ask the evaluator to objectively rate certain qualities present in the sample. Hedonic evaluations ask the evaluator whether, how much, and why they like the sample. In more scientific sensory analysis, descriptive and hedonic evaluations are generally kept separate, since evaluators may be biased in their descriptions based on whether or not they like something. In practice, for culinary quality evaluations, it is possible to combine these two types of evaluation by asking descriptive questions first, and hedonic questions second. This is not ideal, but it is a balance between minimizing bias and getting enough relevant information. Both types of information are often needed for making decisions, and having separate tastings is not practical.
If you are only trying to find out whether your target audience likes a certain variety, hedonic information is sufficient. If you would like to know what types of varieties your customers like and why, then both types of information are useful. For variety selection it is important to know something about the culinary attributes of each variety and whether that combination is desirable. Over time, if it is clear that certain characteristics are always preferred, you may be able to do just one type of evaluation or the other. In general, when working with the general public, preference data is the most reliable, and easiest to get. When working with professionals, both types of evaluation are possible.
If you are asking both types of questions at the same time, is important to separate the descriptive and hedonic questions in evaluator’s minds. For example, an evaluation sheet might begin by asking for a 1 to 5 rating of sweetness, acidity, bitterness and flavor intensity for each variety. After evaluators have rated all varieties, they may be asked to go back and end with a 1-5 rating of overall liking for each variety. This effectively separates descriptive from hedonic evaluation of culinary quality but does require evaluators to taste samples twice and may increase the time commitment or decrease the number of samples you can evaluate. A secondary option is to do a descriptive evaluation with professionals such as chefs and then using the same varieties, do a hedonic evaluation with the general public at a farmers’ market stand or CSA pickup. How much the evaluation focuses on description or preference depends on the goals of the evaluation and the interests of the end users.
Different Methods: Rating, Description and Comparison
There are many ways to design an evaluation questionnaire. The simplest is to use a rating system. The Seed to Kitchen Collaborative uses 1 – 5 or 1- 9 (1 is typically “low”) rating scales to describe culinary quality elements such as sweetness, acidity, bitterness, umami (savory flavor), and flavor intensity (Figure1). The Culinary Breeding Network also uses a 1-9 scale, advising evaluators to use only the odd numbers and reserve the evens for truly intermediate scores. Rating scales are also used to describe hedonic perceptions such as overall liking.
1) For appearance, rate how appealing each variety looks: what is the likelihood you would purchase this variety at a market?
2) For texture, this is a subjective score, 1= poor, 5 = excellent
3) For each flavor category, note the strength of that particular flavor component. Try to be objective, without thinking about whether you like the strength of that component.
4) Umami is a savory flavor component. It is present in foods like mushrooms, soy sauce and tomatoes. Only score for tomatoes.
5) For intensity, 1=flavorless, 5 = very intense. This should relate to the degree of the characteristic vegetable flavor of this crop, not to any off flavors. Try to be objective, without thinking about whether you like the degree of intensity.
6) For overall category, give your global appreciation (1= poor, 5 = excellent) of the flavor of each variety, excluding the appearance of the variety. Do this at the end, once you have filled out the rest of the datasheet.
7) For unusual flavors, note any particularly strong flavors or anything that tastes “off”. Write in what you are tasting, and leave blank if you do not taste anything unusual.
1=low/poor 2=moderately low 3=moderate 4=moderately high 5=high/excellent
Figure 1. Each 3-letter code corresponds to a different sample on the SKC culinary quality evaluation form
CBN and SKC also use sets of dyads to pair flavors or textures perceived as opposites and ask evaluators to place samples on the continuum between the opposites. The dyad approach to evaluating can provide a nuanced picture of the overall flavor experience, akin to a description of wine or coffee. Some flavor and texture traits such as sweetness, acidity and crunchiness may be applicable to multiple crop species while others are more crop-specific (such as earthiness for beets). Normally, a set of 4 to 5 traits is suitable for an evaluation. An example of a dyad approach is given in Figure 2. CBN has worked with culinary professionals to develop highly nuanced lists of flavor descriptors, and used those to create details dyads for evaluation. See Figure 3 below for Lane's tips on creating a flavor vocabulary for the crop you are interested in evaluating.
Kale Quality Rating
Please circle where each sample falls on each continuum
| || |
* * * * *
* * * * *
* * * * *
* * * * *
* * * * *
Figure 2. Seed to Kitchen Collaborative evaluators rank kale samples along each continuum presented here. The words in the dyads were chosen in consultation with farmers to describe the most prominent flavors and textures in kale. CBN has used a similar score sheet with different words to describe the culinary attributes of winter squash.
Lane's tips on creating a flavor vocabulary*
- Gather a group of professionals to develop a flavor vocabulary. When working with chefs, we often select participants with a minimum qualification of a culinary school degree, or five years of experience as a chef or culinary educator. Wine (sommeliers and importers) and coffee professionals are perfect candidates since they have a lot of practice identifying flavors.
- Offer recipes or prep guidelines. If evaluators are working in their own kitchens, provide details about how samples should be prepared to ensure that the flavors detected are the result of variety differences rather than preparation differences.
- Minimize distractions. Noise and ambient temperature are the factors most likely to impede an evaluator’s ability to taste accurately. Also be sure there are no strong aromas in the air. Tasters should close their eyes and wear earplugs to minimize sensory distraction.
- Ask the right questions. Rather than simply asking participants to describe flavor, texture and aroma, ask “what does this remind you of?” and “what else does it taste like?” Then, using that simple prompt, ask people to brainstorm flavor words freely without additional guidance.
- Look for varietal differences. Once the vocabulary brainstorm is complete go through the results and identify words that were common across varieties, and words that could be used to differentiate varieties. Focus on the words that differentiate varieties for future flavor evaluations.
* Drawn from experience developing a comprehensive sensory vocabulary for winter squash, in collaboration with Dr. Alex Stone and Chef Timothy Wastell. More information is available at https://www.eatwintersquash.com/
Figure 3. Lane's Tips on Creating a Flavor Vocabulary
Sometimes, a comparison of sample varieties is more useful than a description and rating of each variety in its own right. Comparing varieties helps determine which characteristics most differentiate the varieties. These methods can sometimes be easier for tasters, since they focus on differences and similarities, rather than on each attribute of each sample. Classifying samples based on their degree of similarities and differences is a method that is fairly intuitive to most people.
For comparison methods, tasters sort the samples into groups based on the characteristics that are most relevant to them. After sorting samples into groups, tasters should label each of the groups to explain how samples in that group are similar to each other and different from the other groups.
To make the process more straightforward, if something about the important flavor characteristics is known, sorting can also be done in a hierarchical manner. This involves putting samples into groups based on the characteristic that most differentiates them, then sorting within groups by secondary attributes. For example, if sweetness is known to be a major contributor to differences in flavor, tasters could group varieties into sweet and not sweet or less-sweet categories, then within categories group samples together based on their degree of acidity or based on their savory flavor. Sorting can also be done for a limited set of characteristics, to get more detail on those particular factors. For example, tasters could be asked to sort based on texture only, regardless of other flavor attributes.
For comparison methods, record the samples that were grouped together and terms that were used to describe each of the groups for each taster. Then calculate how many times a pair of samples were in the same group or in different groups, and create a list of terms used to describe/differentiate each sample (Figure 4.).
|Taster||Group 1||Group 2||Group 3|
|1||Sample 1, 2; sweet||Sample 3, 4 rich, savory||Sample 4 bitter|
|2||Sample 1, 2; sweeter||Sample 3 stronger||none|
|3||Sample 1, 2; mild||Sample 3, 4 savory||Sample 4 metallic|
|4||Sample 1, 3; sweet/savory||Sample 2 acidic||none|
|5||Sample 1, 2, 3; nice flavor||Sample 4 strong, bitter||none|
|6||Sample 1, 2; sweet, bland||Sample 3, 4 strong, harsh||none|
| || |
| || |
| || |
| || |
Figure 4: Top. Example of hypothetical data sheet for a grouping exercize with four samples and 6 evaluators. The groupings are first listed, either by the evaluator after finishing the tasting, or by the researcher if the evaluators do the grouping on a piece of paper or table top and leave them for the researcher to tabulate. Bottom: The times the same samples appear together are then counted and put in a table. The number of times samples grouped together is above the diagonal, and the number of times they grouped separately below the diagonal.
For the hypothetical comparison example above, it appears that samples 1 and 2 are very similar and are described as sweeter or milder. Sample 3 changes groupings the most, and might be similar to other samples for different aspects of flavor. It appears to have a stronger, more savory flavor. Sample 4 is the most different, and may have some undesirable flavor components.
No matter which quantitative approach is employed, it is often useful to also include a few qualitative questions, or a space for evaluators to write notes. Evaluators often have important insights that are not captured in the questions on the evaluation form, such as unique culinary applications for particular varieties, or additional descriptive vocabulary for the crop in question. Having a space for write-in comments also avoids evaluators trying to fit their observation into one of the other categories, where it can skew responses. For example, if there is a metallic aftertaste to a variety, this may get incorporated into the ‘bitter’ category if you don’t provide space for comments, even though it is distinct from a true bitter flavor. Capturing these details can help differentiate between trial varieties and can help the trial leaders develop better questions for future events.
Making Sense of the Results
At the end of an evaluation, you may be left with a lot of numbers. There are many ways to turn these numbers into usable information. One option is to simply read through each of the evaluation forms and summarize the most important points from each. Which samples had the highest or lowest numbers in multiple categories, and which were most strongly preferred? If there are a lot of evaluators, however, this approach is likely unfeasible. The easiest way to begin making sense of evaluation data is to enter the data into an electronic spreadsheet using Microsoft Excel or an equivalent program. Check the spreadsheet for numbers that fall outside of the expected range, and do not include information from tasters that did not follow the instructions.
Once all the data is entered summarize the data by calculating averages for each trait evaluated on each variety. Though averages may be swayed by extreme responses, they can provide a general picture of how each variety performed for each quality being evaluated. Using the spreadsheet to calculate standard deviation will also help you see if there is a wide range of responses to a particular question, or if the evaluators were relatively aligned in their perceptions. If there appears to be a wide range of responses (standard deviation is high), check for whether the evaluators fell into two distinct groups, or whether there are single tasters that don’t align with the majority. Outliers may be caused by individual evaluators who do not understand what they are being asked to do, or who taste things differently from the rest of the group. Knowing why an outlier is present can help you decide whether to keep that response in the data set or not.
Once you have an idea of the general responses to each question, think about what information will be most useful to the target audience of the evaluation. If the purpose of the evaluation is to help farmers select varieties that will appeal to culinary customers, then highlight results demonstrating which varieties chefs favor, or which qualities tend to be present in chef favorites. Findings can be presented anecdotally (a short written description) or visually (using graphs, tables or more creative graphics). How you display results should be determined by how you expect the results to be used. Once you’ve finished displaying the results, be sure to store the data in a place where you can find it again!
The Organic Seed Alliance offers a useful online tool for keeping track of variety trial evaluations and analyzing results. Though not all culinary quality evaluations stem from variety trials, this tool can be useful for organizing, storing and interpreting culinary evaluation data. It allows you to generate a data sheet in excel and analyze your data including means, standard deviations and pairwise comparisons of varieties. The tool can be found in the On Farm Variety Trials Toolkit for Risk Management for Organic and Specialty Crop Producers.
Evaluating culinary quality of vegetable, grain, and other specialty crops can help growers determine which varieties are best suited to their customers’ needs. As with any other project or experiment, it is easy to over-complicate things when the goals of a culinary quality evaluation are not clear. Being extremely specific about the type of information you are hoping to obtain, and the intended audience of that final information will help keep the evaluation focused on the desired outcomes.
Sharing the results of your culinary quality evaluation will help make the seed system and local food system stronger. Successful evaluations provide seed companies, farmers and end users with meaningful information on what they can expect from a variety. This information in turn helps farmers equip chefs and other consumers with a better understanding of what a variety will taste like or how it can be used in a dish. One of the most commonly asked questions at a farmers’ market is “Which tomato is best?”. The answer is invariably “it depends”. What does the consumer like in a tomato? What are they using it for? If a farmer has an accurate description of the variety, they will be able to more accurately advise their customers. When results are shared, one afternoon’s worth of work conducting a culinary quality evaluation can ripple into multiple season’s worth of benefits for farmers engaged in the difficult work of feeding communities and stewarding their land.
- Bonti-Ankomah, S., and E. K. Yiridoe. 2006. Organic and conventional food: A literature review of the economics of consumer perceptions and preferences. Final report. Organic Agriculture Centre of Canada.
- Dawson, J.C. and Healy, G.K. 2018. Flavor Evaluation for Plant Breeders. Plant Breeding Reviews 41: 215-262. (Available online at: https://doi.org/10.1002/9781119414735.ch5) (verified 2 December 2018).
- Frost, M.B., D. Giacalone and K.K. Rasmussen. 2015. Alternative methods of sensory testing: working with chefs culinary professionals and brewmasters. In: Rapid sensory profiling techniques. J. Delarue ed. Woodhead Publishing: 363-282. (Available online at: https://doi.org/10.1533/9781782422587.3.363) (verified 2 December 2018).
- Michigan Department of Agriculture and Rural Development. Guidelines for providing safe food samples. 2015. (Available online at: https://www.michigan.gov/documents/mda/MDA_FdSmplngFarmMkts5-09_279110_7.pdf) (verified 2 December 2018).
- Colley, M., Dawson, J.C., Zystro, J. Healy, G.K., Myers, J., Behar, H., and Becker, K. 2018. The growers’ guide to conducting on-farm variety trials. (Available online at:https://seedalliance.org/wp-content/uploads/2018/02/Growers-guide-on-farm-variety-trials_FINAL_Digital.pdf) (verified 2 December 2018).
- Culinary Breeding Network [Online]. Available at: https://www.culinarybreedingnetwork.com/) (verified 2 December 2018).
- Eat Winter Squash [Online]. Available at: https://www.eatwintersquash.com/ (verified 2 December 2018).
- On Farm Variety Trials: Toolkit for Organic and Specialty Crop Producers. [Online]. Available at: https://articles.extension.org/pages/74613 (verified 2 December 2018).
- Seed to Kitchen Collaborative [Online]. Available at: https://seedtokitchen.horticulture.wisc.edu (verified 2 December 2018).