USING EVALUATION TO ASSESS THE RESEARCH NEEDS OF HEMP FARMERS: LESSONS IN PREPARING AND IMPLEMENTING SURVEYS A Project Paper Presented to the Faculty of the Graduate School of Cornell University in Partial Fulfillment of the Requirements for the Degree of Master of Professional Studies in Agriculture and Life Sciences Field of Horticulture by Emily Paul August 2021 © 2021 Emily Paul ABSTRACT Due to the newness of the cannabis industry, there is a lack of published research available to farmers. After meeting with Erik Carbone, a hemp farmer from Berkshire, New York, it was apparent that there is a disconnect that exists between the research that is being done and hemp farmers. This disconnect can lead to the suffering of farmers and can be detrimental to the success of the industry. A survey was created to assess the research needs of hemp farmers while also aiming to understand the accessibility of information that Cornell Cooperative Extension is producing. The goal of the survey was to inform Cornell Cooperative extension on the research needs of farmers and how they may prefer to access resources on hemp, whether it be through blog posts, the CCE website, or email communications. Within the survey, specific questions were asked to determine whether disease, pests, weeds, legal, or marketing topics were most desired by farmers in the industry. The survey questions were developed in collaboration with Dr. Carlyn Buckler, Dr. Chris Smart, and Maire Ulrich, a vegetable specialist from Cornell Cooperative Extension. Many resources were utilized in order to create an effective survey with limited bias and accurate responses. The surveys were then distributed at the Hemp Expo on 11 January 2021, and the Spring CBD Update on 22 April 2021 by Cornell Cooperative Extension. The results demonstrated that legal topics were most concerning for our participants and that email communication of hemp research updates would be preferred in the future. In addition, the results demonstrated what can be improved in future surveys. BIOGRAPHICAL SKETCH Emily Paul grew up in Buffalo, New York where her passion for the natural world began at a young age. She recalls the start of her appreciate for plants and animals on frequent nature walks with her grandfather, where they spent their time identifying and observing plants and animals. This passion and a desire to make the world a better place led to her undergraduate degree from the SUNY College of Environmental Science and Forestry. While at SUNY ESF, Emily studied conservation biology, where she focused on sustainable agriculture and designed a project that maximized space of a greenhouse for optimal productivity. After a summer internship on a hemp farm, Emily gained a strong appreciation for the plant and its many uses. This appreciation and admiration led to the decision to attend Cornell in pursuit of a Master of Professional Studies in Horticulture with a concentration in hemp science. While completing her degree, Emily reached out to local farmers in order to learn more about the current state of the industry. After talking to several farmers, she realized that many felt there was a lack of resources available to them, making it difficult for them to succeed. After learning this, Emily centered her research project on bridging the gap between the researchers at Cornell, with the goal of making information more relevant and accessible to hemp farmers. iii ACKNOWLEDGMENTS I would like to thank Dr. Carlyn Buckler and Dr. Heather Grab for facilitating the hemp science MPS program. Despite a global pandemic, they created classes that were engaging and facilitated valuable connections to the cannabis industry for their students. They supported the hemp science cohort emotionally and academically with weekly meetings and check ins. I am truly grateful for their dedication and love for their students. Second, I would like to thank Dr. Chris Smart for her dedication to the hemp science MPS program, as well as her mentorship to me as her advisee. Her knowledge of plant science and experience working with farmers was invaluable to my project and success at Cornell. I am thankful for her wisdom and support throughout the creation of this project. Third, I would like to thank Maire Ulrich and Cornell Cooperative Extension for helping me develop my survey, as well as giving me a platform to distribute it and collect data. I am so thankful for the time she took to help me with my project and her insight on the industry. Fourth, I am thankful for Erik and Brittany Carbone, who own Tricolla Farms and Bardo Labs. My first conversation with Erik sparked the idea of this research project and he has been a great mentor ever since. I am thankful to them both for employing me at Bardo Labs during my last summer in Ithaca and am lucky to have worked for two people who are genuinely passionate about what they do. I would also like to thank Emmy and Sarah, my coworkers who have taught me so much about working in the cannabis industry and became my “sisters” during my time at Bardo. I am also thankful for Sue, Haylli and Tia for making me feel welcome into the Tonic/Bardo family. iv Fifth, I would like to thank Jenna Switzer, the owner of Vital You Products who facilitated one of my first internship experiences in the cannabis industry. I am extremely thankful to have her as a mentor and a best friend. Finally, I would like to thank my parents and loved ones for supporting me through my unconventional interests and career path. I am forever thankful for their love and support throughout this journey—I couldn’t have done it without them. v TABLE OF CONTENTS BIOGRAPHICAL SKETCH……………………………………………………………………..iii ACKNOWLEDGEMENTS…………………………………………………………………........iv TABLES OF CONTENTS………………………………………………………………….........vi LIST OF FIGURES……………………………………………………………………….......…vii INTRODUCTION………………………………………………………………………………...1 METHODS…………………………………………………………………………………..........4 RESULTS………..…………………………………………………………………………..........6 DISCUSSION………………………………………………………………………………........10 CONCLUSION……………………………………………………………………………..........16 APPENDIX…………………………………………………………………………………........17 REFERENCES…………………………………………………………………………………..28 vi LIST OF FIGURES Figure 1 Answers to the question “How do you most prefer to get information on hemp?”….……8 Figure 2 Responses for the cropping concern “legal”………………………………………………8 vii INTRODUCTION During a discussion with Erik Carbone, owner of the hemp farm, Tricolla Farms in Berkshire, New York, I realized that, although Cornell is doing very important work in hemp genetics, pest and diseases, it was not the only information hemp farmers wanted. This inspired the creation of my research project, which is a survey that offers insight into the needs of hemp farmers. In cooperation with the Cornell Cooperative Extension, I developed a survey which included questions about the challenges and concerns farmers have, areas of research they would like to see in the future, social media usage, and platforms used to obtain information about farming best practices. The survey was intended to ultimately evaluate the research needs of farmers, while also assessing the accessibility of information. Thus, this research project was developed to help increase the number of farmers that Cornell Cooperative Extension is reaching, and in turn benefit the farmers by giving them the information they need to help their businesses succeed. Due to a lack of research in the cannabis industry in general at the time of this publication, there has only been one other survey from a university attempting to assess the needs of hemp farmers. Illinois Extension published a needs assessment survey to help “shape future research efforts” which can be found on their website (https://go.illinois.edu/HempNA). The goal of their survey was similar; to evaluate research needs from farmers to inform future cooperative extension programming. The survey including questions about what would be useful to farmers in a hemp variety trial (flowering dates, disease, and pest information, etc.) as well as gauging interest in being connected with CBD processing facilities. The Illinois evaluation was in the form of a Qualtrics survey and was published on the Illinois extension website in 2019 (Harbach et al., 2019). There has not been a hemp survey of this kind conducted at the Cornell 1 Hemp Expo in past, therefore the goal of my project is to create a survey framework that can be used by other universities to inform hemp research. This project consisted of a survey conducted during the Hemp Expo sessions on 11 January 2021 and a second survey at a CBD update held by Cornell Cooperative Extension on 22 April 2021. I am interested in a possible career in outreach and education in the cannabis industry after graduation, which is why this project was so appealing to me. I have a strong interest in working with plants and people; and this project will help me gain experience in the world of science communication and research. My goal while at Cornell was to gain a strong knowledge and scientific background on hemp. In my career, I would like to utilize this strong foundation in a field where I can work with people and communicate efficiently the knowledge that they want and need. The creation of an effective evaluation ensures that research and project goals are being met, examines strengths and weaknesses of programs, and can even be used to demonstrate the importance of a project to funders (Foster, 2008). Evaluations can also assess whether we are reaching a diverse group of audiences, and how we can make our programs more accessible if we are not. A lack of training in the creation of evaluations can influence data, and proper training is necessary for accurate results. Unfortunately, many organizations are not properly trained for creating an effective evaluation and understanding the statistical data that comes from the survey, therefore results may be skewed. Question ambiguity, question order, failure of the respondent to understand the question, respondents trying to influence the outcome of the study or impress the organization, can all impact the results of a survey (Brace, 2004). These factors are important to keep in mind when developing any evaluation. 2 Fortunately, there is plenty of literature on how to create an effective survey. For example, questions should be clear and specific, there should be no more than 5 answer choices, and only one question should be asked at a time, meaning that two concepts should not be addressed at once (Pew Research Center, 2021). It has been found that question order can affect responses, therefore engaging questions should be at the beginning of the survey and questions should be grouped by topic in a logical order (Pew Research Center, 2021). Since there is plenty of information available on this topic, proper training should not be a barrier to creating an effective evaluation. 3 METHODS Before creating my survey, I researched the methods of creating an effective evaluation, with the help of Dr. Carlyn Buckler and her publication “Developing Evaluation” (Buckler, 2017), “Framework for Evaluating Impacts of Informal Science Education Projects” (Allen et al., 2008) and “Writing Survey Questions” (Pew Research Center, 2021). The survey created by Illinois Extension was also used as a resource for the creation of my survey (Harbach, et al., 2019). The final evaluation was created in collaboration with Cornell Cooperative Extension, cannabis industry professionals, and my advisors at Cornell University. The questions were developed after discussing relevant issues in the hemp industry with Erik Carbone, the owner of Tricolla Farms, a hemp farm in New York State. My conversations with him gave me a perspective on what the needs of hemp farmers are, which helped inform the questions that I developed. In addition, I worked in collaboration with Dr. Carlyn Buckler, Dr. Chris Smart, and Maire Ulrich, a vegetable specialist from Cornell Cooperative Extension to develop the survey questions. Their knowledge of the cannabis industry, as well as understanding the problems that farmers are facing and learning techniques for interacting with the public, were of great value to my project. The evaluation was distributed in the form of a Qualtrics survey during two zoom events, the Cornell Hemp Expo on January 11th, and the Cornell Extension Spring CBD Update on April 22nd. After the first survey on January 11th, edits were made in collaboration with Dr. Carlyn Buckler and Dr. Chris Smart to improve the survey before the second distribution. The surveys that I created were combined with questions that Cornell Cooperative Extension developed to evaluate the success of their programming and distributed during breaks in the Zoom chat. 4 Results from the survey questions I created were used in my project, while results from Cornell Cooperative Extension’s questions were omitted from this analysis. All versions of the survey can be found in the Appendix. The results from the two surveys were compiled into an excel spreadsheet, summarized, and analyzed. 5 RESULTS During our first survey, and in collaboration with CCE, on 11 January 2021 at the Hemp Expo, there were 26 respondents. We collected demographic information on gender, race, and ethnicity, but we did not ask whether they were in farming, education, or outreach. After realizing how valuable this information would be, we adjusted our next survey accordingly to include that piece. Questions on gender, race and state of residence had a 100% response rate, while ethnicity had 25 out of 26 participants respond. Throughout the survey, response rates continued to vary. The cropping concerns question where participants ranked their concerns on a scale of 1 through 5, had an average response rate of 16 out of 26. When asked “How long have you grown hemp?” most participants (15 out of 26) answered that they are either new to producing hemp or had less than 2 years of experience. Eleven participants answered that they are growing hemp for CBD as opposed to fiber in 2021, while they also indicated that they are mostly producing field crops as opposed to greenhouse cultivation. For the first survey, when asked to rank their cropping concerns, 1 being the highest and 5 being the lowest, the average response was that weeds were ranked 5, disease was ranked 3, insects were ranked 2, legislative/legal changes/complications were ranked 1, and marketing and sales were tied for 2 and 4. When asked to rank their interest in hemp research, 1 being the highest and 7 being the lowest, most participants responded that their interest in research pertaining to weeds was ranked 7, interest in disease research was ranked 6, and the average interest in pest research was ranked 5. Their average interest in agronomy/nutrition research was ranked 4, greenhouse production was ranked 3, genetics was ranked 2, and marketing was ranked 1. When participants were asked how they prefer to get their farming information, all of the participants responded and 15 said that they prefer email communication. The question “other 6 than Cornell/CCE where do you get information on hemp?’ as a write-in question had 11 out of 26 participants respond. “How do you prefer to get your farming information” had a 100% response rate while “Are you active on social media, if so, which platforms?” had a response rate of 11 out of 26. At the second survey on 22 April 2021, there were 23 respondents. When asked how often participants use CCE as a resource, 9 out of 18 respondents answered always. When participants were asked how they prefer to get information, 17 out of 23 answered that e-mail was their preferred method of communication, responses to this question are demonstrated in Figure 1. Ten out of 23 participants responded that their primary resource for getting information on hemp, other than Cornell/CCE was the internet. When asked which social media platforms participants are most active on, the most popular answer was Facebook, although the response rate on this question was 12 out of 23. Participants were asked to rank their cropping concerns (1 being the highest, and 5 being the lowest). Most participants answered that disease was ranked 2, weeds were ranked 5, insects were tied at 2 and 4, legal issues were ranked number 1. As seen in Figure 2, most participants chose that legal issues were their biggest cropping concern. 7 "How do you most prefer to get information on hemp?" Friends/other 4% Blog posts 0% Paper mail 0% Websites 22% Social media 0% E-mail 74% Figure 1. Responses to the question “How do you most prefer to get information on hemp?” at the April 22nd distribution of the survey. 74% of the participants responded “E-mail.” Cropping Concerns; Legal Ranked 1-5 (1 being the highest, 5 being the lowest, 6=no response) 25% 37% 17% 8% 13% 0% 1 2 3 4 5 6 Figure 2. Responses to the question “Please rank your cropping concerns (1 being the highest and 5 being the lowest)” for the choice “Legal” on April 22nd 37% of participants choose “Legal” as their highest cropping concern. In the legend, a ranking of 6 represents no response. For the second survey, response rates were 23 out of 23 for the first demographic questions but began to drop as the survey went on. Twelve out of 23 participants responded to the question “please rank your cropping concerns.” Response rates for the questions “How do 8 you prefer to get information?” and “Other than Cornell/CCE where do you get information on hemp?” were 23 out of 23. When asked “Are you active on social media, if so, which platforms?” 12 out of 23 participants responded. 9 DISCUSSION As demonstrated in the results, response rates varied throughout the surveys. In the first distribution of the survey on 11 January 2021, we saw that simpler multiple-choice questions closer to the beginning of the survey had higher response rates than the questions that were write-in or where participants were asked to rank their cropping concerns. This information was used to inform the editing of the survey before the next distribution on April 22nd. For example, “Other than Cornell/CCE, where do you prefer to get information on hemp” as a write in question had a 42% response rate. We decided that having a few multiple-choice answers could potentially increase response rates, so we changed the question from a write-in response to multiple choice. This seemed to enhance the answer rate of this question, since in the next distribution of the survey this question had a 100% response rate. Alternatively, this could also be due to the cohort of participants in the second survey being more interested in this question. In the second distribution of the survey on 22 April 2021, simpler questions—such as demographic and straightforward multiple-choice questions—had higher response rate than more complex questions. The question that asked participants to rank their cropping concerns with six different choices had less than half (12 out of 23) participants respond. We see a similar pattern in the results of the first survey and can perhaps attribute this to the fact that the questions may have been confusing to participants. They may have also gotten “question fatigue” by the time they reached these questions, causing response rates to drop. In future surveys, we might move this question closer to the beginning of the survey to hopefully assess whether question fatigue was the reason for low response rates. We may also consider other ways to ask this question, such as “What are your major hemp cropping concerns (select all that apply)”, rather than having participants rank their choices. 10 The next two multiple choice questions - #’s 11, “How do you prefer to get information?” and 12, “Other than Cornell/CCE where do you get information on hemp?” had a 100% response rate. This suggests that the question on cropping concerns may have been too confusing or complex for participants, and/or that multiple-choice questions are more likely to receive responses. Question #13, “Are you active on social media, if so, which platforms?” had a very low response rate; almost half from the previous two questions at only 12 out of 23 responses. The reason for this is unknown, but one hypothesis may be that the participants of the event are of a demographic that is not as active on social media, so they did not respond. This is useful information since we might use this metric to determine whether social media outreach by CCE is effective and reaching the intended audiences. In order to accomplish the objectives of the survey, it was necessary to research and become familiar with survey best practices. This includes the correct way to word and distribute a survey for optimal response rates and low bias. “Developing Evaluation” by Dr. Carlyn Buckler, as well as “Evaluation toolkit for museum practitioners” by Harriet Foster were used as resources to create my survey. My research on survey best practices led to the editing of my survey after the first round of distribution on January 11th by Cornell Cooperative Extension. The first version of the survey can be found in Appendix A, which includes screenshots of the survey. The responses seen in the survey in Appendix A are present because the only screenshots of the survey available to us were already filled out. Since we were delivering the survey at a Cornell Cooperative Extension event, additional questions were added that were being used for CCE’s research purposes. Questions such as “Do you need DEC credits?” and “Please insert your DEC I.D. number” were used by CCE to give DEC credit to participants for attending the event. The first three pages and the last page of the survey were questions created 11 by CCE that were used to assess the success of their program. As seen in Appendix A, there is a question with a typo that says, “Please rate the topics…” which lists four different topics for participants to rate, from “extremely useful” to “extremely useless”. We can assume that the question was intended to be “Please rate your interest in the topics,” but a typo like this can create confusion among participants and potentially influence results. From the second survey on April 22nd, we can also see that legal issues were the participants biggest cropping concerns. In this new industry, it makes sense that many farmers are concerned by this. This information can be used to potentially provide more legal advisement to farmers. In addition, we could potentially ask another question in a future survey to further explore what issues about the legal system are of most concern. We also found that most respondents prefer e-mail communication to receive their farming information, which could be helpful in informing what sources of outreach would give the most coverage. This data can potentially be used to influence future programs, or help to understand how hemp information is should be distributed. After the first distribution, Dr. Carlyn Buckler and I met to evaluate the success of the survey, and how I could improve questions to increase response rates and accuracy of results. Changes that were made include, adding the question “What is your affiliation with the hemp industry?” to determine whether the respondent was a grower, processor, manufacturer, or brand owner. In addition, as understandings of what “sometimes” or “most of the time” may mean for different respondents, the answers to the question, “How often do you use Cornell Cooperative Extension as a resource?” were changed from a scale of “never to always”, to “daily, weekly, once a month, and once a year, never” to accurately assess the usage patterns of participants. We also decided that demographic data such as race, gender, and location were unnecessary for the 12 purposes of this survey. Research has shown that a shorter survey is more likely to receive responses than a longer survey, due to respondent fatigue which is when survey participants get bored of the survey task and their responses begin to decline (Lavrakas, 2008). This is why we decided to omit the questions that showed no benefit to our intended research purposes. The first survey included questions to assess the industry experience of participants, such as, “How long have you grown hemp?” and “How many acres do you plan on growing in 2021?”. These questions were also deemed unnecessary to the purpose of our survey and were omitted for the second survey with the intention of receiving more responses on a shorter survey. In addition, instead of listing each cropping concern and asking participants to choose a rating of 1 through 5, all cropping concerns were listed within one question and participants were asked to rank each choice within the question. A write in question, “Are there any specific resources that you would like to see?” was added to the end of the survey as an opportunity for participants to voice any concerns/needs that the survey did not address. After the edits, the final draft of the new survey was seven questions long. After edits were made, the survey was intended to be sent out to the list of authorized hemp research partners found on the New York State Department of Agriculture and Markets website. A statement of purpose was created to inform the research partners of the intention of the survey. The statement of purpose was as follows; “This brief survey, created by Cornell Cooperative Extension, was designed to evaluate the needs of hemp growers and processors in New York State. We are in the process of creating new resources and will use this data to inform what is needed most. This survey will only take a few minutes, and we appreciate your dedication to creating a thriving hemp industry in New York State.” A statement of purpose was created to demonstrate the importance of the survey, as well as why recipients should consider taking the survey. This is a necessary component of an email 13 survey, since potential respondents do not know who the surveyors are and why they are contacting them. Due to this, at the beginning of the survey, I included my name, who I was, the purpose of the survey, how it could potentially benefit them, and how long approximately it will take. Before the survey was intended to be sent out to the list of hemp research partners, Cornell Cooperative Extension hosted another event, the Spring CBD Update on April 22nd. They offered to distribute my survey at the event, and I sent them the updated version of the survey that included the edits mentioned above, as also seen in Appendix B. Unfortunately, some of the edits never made it to the final version of the survey that was distributed at the event (see Appendix C) which includes additions and rewording of questions done by CCE, highlighted in yellow for comparison. This miscommunication could be due to the fact that I was notified of the event the day before, and they may not have had the time to update the survey. In addition, due to the nature of the event being facilitated by Cornell Cooperative Extension, my survey was combined with questions from CCE in order to assess the success of their event. Twelve additional questions were added to the survey, making it 60% longer, as seen in Appendix C, which is the version of the survey that was distributed. Appendix B shows the version of the survey that I created to be distributed. In the future, the survey would be more successful if distributed on its own, since additional questions, not related to the intention of my survey may have influenced the responses and participation in the survey. The melding of two writing styles may also have influenced the participation and responses of the survey. This taught me the importance of working closely with the organization that I am creating an evaluation for in the future, and to emphasize that questions should not be edited because there is a method behind the wording and question order. 14 The survey was not sent to the list of licensed hemp research partners, since it was not feasible to send out the survey and collect the data from a possible 681 respondents within the time frame that I had. In future studies, it would be beneficial to send the survey directly to a listserv of hemp farmers, or distribute it as written at a future Cornell Cooperative Extension event. This way, we would know that we are reaching hemp farmers as our participants, not faculty members, students or researchers. Although this survey had limitations, the framework can be used to inform future evaluations within Cornell Cooperative Extension and other extension services throughout the country. In an industry that is in its early stages, it is important that we as researchers are evaluating the needs of the farmers in effort to help mitigate the barriers to successful production and processing in this nascent industry, making research efforts relevant and accessible to farmers. 15 CONCLUSION Overall, this experience taught me how important it is to be properly trained in conducting evaluations, as seen in what we learned from the two different surveys that were distributed. There is critical science behind conducting surveys and it is evident that it is worthwhile for government, academic researchers, and companies to invest in training for their employees, or use outside evaluation professionals. In many cases, companies are evaluating the needs of their customers or audience without any training on how to accurately do so. The effects of this can be that surveys are not receiving accurate responses from participants and results are skewed, which, depending on the industry, could cost thousands of dollars (or more) because of misinformation. Although this survey had its limitations, this framework and research in developing evaluations can be utilized by Cornell Cooperative Extension, other universities, and cannabis industry members to ensure that surveys are effective and accurate. The ultimate goal of this project was to bridge the gap between the research done at Cornell and hemp farmers, and that developing and using surveys can be an effective way of getting the information that is needed, if done correctly. In addition to my research project objectives, this project has facilitated valuable experience in working with Cooperative Extension Services, interacting with farmers, and problem solving. This project has helped me gain valuable experience that is necessary for my future career in the cannabis industry. Evaluations can be used to assess customer needs and desires in my future career, having a positive effect on business. They can also be used to assess training success, or employee satisfaction if I am in a management role. 16 APPENDIX Appendix A Survey 1, distributed at the Hemp Expo on January 11th, including questions added by Cornell Cooperative Extension 17 18 19 20 21 22 Appendix B Version of survey, after editing that was submitted to Cornell Cooperative Extension for distribution at the CBD Update on April 22nd, 2021 23 Appendix C Survey that was distributed by Cornell Cooperative Extension at the CBD Update on April 22nd. Yellow highlighting represents additions or rewording by CCE. 24 25 26 27 REFERENCES Allen, S., Campbell, P. B., Dierking, L. D., Flagg, B. N., Friedman, A. J., Garibay, C., & Ucko, D. A. (2008, February). Framework for evaluating impacts of informal science education projects. In Report from a National Science Foundation Workshop. The National Science Foundation, Division of Research on Learning in Formal and Informal Settings. Brace, I. (2018). Questionnaire design: How to plan, structure and write survey material for effective market research. Kogan Page Publishers. Buckler, C. (2017). Developing Evaluation. Science Learning Foster, H. (2008, February). Evaluation toolkit for museum practitioners. East of England Museum Hub Harbach, C. (2019, December 11). Industrial hemp resources and needs assessment survey or Illinois growers. University of Illinois Extension. https://extension.illinois.edu/blogs/hill-and- furrow/2019-12-11-industrial-hemp-resources-and-needs-assessment-survey-illinois. Lavrakas, P. J. (2008). Encyclopedia of survey research methods (Vols. 1-0). Thousand Oaks, CA: Sage Publications, Inc. doi: 10.4135/9781412963947 Pew Research Center. (2021, May 28). Writing Survey Questions. Pew Research Center. https://www.pewresearch.org/our-methods/u-s-surveys/writing-survey-questions/ 28