What email address or phone number would you like to use to sign in to Docs.com?
If you already have an account that you use with Office or other Microsoft services, enter it here.
Or sign in with:
Signing in allows you to download and like content, which the author will be aware of.
Embed code for: Erickson 509 Project Report - Copy
Select a size
Development of Assessment Tools to Measure Well Owner & Stakeholder Needs
A Literature Review & Best Practices Report with Recommendations to OHA’s Domestic Well Safety Program
Photo of Artesian Well By Jim Culp
Oregon Health Authority, DWSP Intern
July 22, 2016
Oregon Health Authority, Health Educator
Oregon Health Authority, Program Coordinator
Oregon Health Authority, Epidemiologist
Oregon Health Authority, Program Manager
Public Health Foundation, Quality Improvement Consultant
Centers for Disease Control, Epidemiologist
Portland State University & OHSU, Graduate Advisor
Portland State University & OHSU, Faculty Advisor
Suggested Citation: Erickson, K. (2016). Development of Assessment Tools to Measure Well Owner & Stakeholder Needs: A Literature Review & Best Practice Report. Unpublished Manuscript.
Table of Contents
file:///C:\Users\Katherine%20Erickson\Desktop\509%20Final%20Submissions\Project%20Report\FinalDraftJul22.docx.docx.docxDevelopment of Assessment Tools to Measure Well Owner & Stakeholder Needs 0
file:///C:\Users\Katherine%20Erickson\Desktop\509%20Final%20Submissions\Project%20Report\FinalDraftJul22.docx.docx.docxA Literature Review & Best Practices Report with Recommendations to OHA’s Domestic Well Safety Program 0
List of Acronyms (Alphabetical) v
Executive Summary vii
Public Health Risk of Private Wells in Oregon 1
About DWSP 1
Project Objectives 2
i. Inclusion/Exclusion Criteria 4
ii. Databases Used 5
iii. Search Terms 5
iv. Searches 5
Figure 3. Summary Table of Included Studies 7
General Evaluation & Assessment Best Practices 7
Figure 4. DWSP’s Existing Data Sources 11
1) Private Well Owners 13
Figure 5. Most Frequently Reported Best Practices across Studies 16
Figure 6. Studies by Number of Best Practices Reported 17
Well-Owner Assessment Best Practices 18
Figure 7. Health Belief Model-Oriented Survey Questions 23
Figure 8. Flanagan et al. Use of RANAS Variables 25
Figure 9. Demographic Information Fields From New Jersey Survey 28
Figure 10. Williams 2003 Recommendations for Increasing Response Rates 30
1.2) Well-Owner Assessment Best Practices (Other than Surveys) 32
2) Stakeholder Assessment Best Practices 35
Figure 11. Colorado Tracking Portal for Private Wells 39
Recommendations & Next Steps 42
List of Acronyms (Alphabetical)
AAP- American Academy of Pediatrics
BRFSS- Behavioral Risk Factor Surveillance System
CDC- Centers for Disease Control
CIPP- Context, Input, Process and Product evaluation model
CDPHE- Colorado Department of Public Health and Environment
DEQ- Department of Environmental Quality
DWSP- Domestic Well Safety Program of Oregon
DWTA- Domestic Well Testing Act of Oregon
EPA- Environmental Protection Agency
HBM- Health Belief Model
IRB- Institutional Review Board
LHD- Local Health Department
MeSH- Medical Subject Heading
OHA- Oregon Health Authority
PHF- Public Health Foundation
PICOs- Population, Intervention, Comparison, Outcome framework
PMC- PubMed Central database
RANAS- Risk, Attitude, Norm, Ability, and Self-Regulation Model
RET-Real Estate Transaction
SDWIS- Safe Drinking Water Information System
USGS- United States Geological Survey
VOCs- Volatile Organic Compounds
WRD- Water Resources Department
In Oregon about 1 in 6 residents depend on a private well for drinking water and approximately 350,000 private wells currently exist in the state. Without proper maintenance, testing and treatment, private wells are subject to contamination from biological and manmade chemicals such as bacteria, arsenic, lead, nitrates, VOCs(volatile organic compounds), road salt, pesticides, fertilizer, and radon11, 16. Moreover, being outside the public water supply, private well owners are primarily responsible for the maintenance of their well, which includes ensuring that their well is properly constructed and sited, regular well testing and monitoring, and source water protection through testing and treating for potential contaminants in the local area13.
The main piece of legislation which aims to safeguard property owners and the quality of their well water in Oregon is the Domestic Well Testing Act (DWTA), passed in 1989. However the DWTA requires only that private wells be tested at the sale of a property, and has no requirement of enforcement. In order to ensure that well owners have the resources, tools, knowledge and information to take care of their well and to evaluate Oregon’s Domestic Well Safety Program’s success in providing services for this population, a literature review was performed and described in the following report with best practices and recommendations.
The goal of the review was to examine best practices for assessing private well owners stewardship practices and need for services, as well as stakeholder (i.e. testing labs, well drillers/constructors, real estate agents, local government officials, etc.) practices and needs to provide better outreach and education and improve private well stewardship and overall population health. Several best practices emerged from private wells and program evaluation literatures, including pre-testing the assessment tool, use of surveys, using a random sample, application of a health behavior theory, and crowd-sourcing assessment questions. Surveys were found to be the most common form of assessment tool, and 11 studies were included which implemented a survey assessment design. Other methods of assessment were also observed, including large focus groups or conferences for sharing stewardship best practices, secondary data analysis of calls to the WellCare National Private Wells Hotline, conducting system analysis of state data repositories, and carrying out literature or document review of existing documents.
Literature with best practices for assessing private well stakeholders specifically was more difficult to find however. Still, several best practices for engaging and assessing stakeholders were demonstrated through several studies both related to private wells and not. Two studies were identified which examined private well stakeholder practices with regards to outreach and education as well as data storage of private well information, . Both of these also utilized surveys as an assessment method, which was highlighted as a best practice. Further literature was examined on engaging and assessing stakeholders generally, and some general best practices materialized, including conducting a stakeholder analysis, using existing contact lists and relationships, and employing mixed methods.
Based on the best practices which were observed in the literature review, several recommendations were made to DWSP for moving forward with an assessment of private well owners and stakeholders in Oregon. These included to utilize a survey design, implement and prioritize a stakeholder survey before a well owner survey, collect a representative sample of stakeholders and well owners in Oregon, prepare a data management plan, base the data analysis on the evaluation questions, and to adopt the best practices which were examined in the literature. Two separate assessment tools are proposed in the appendix as well as a budget for implementing the first assessment tool, or the stakeholder assessment
Public Health Risk of Private Wells in Oregon
Recent estimates suggest that approximately 1 in 6 Oregonians rely on private wells for drinking water . Private Wells are subject to contamination if not properly maintained, and many well owners may not know or realize the importance of protecting their well water through testing, treatment, and on-going maintenance. Moreover, private drinking water signals a risk to public health because it is not monitored as public water systems are under the Safe Drinking Water Act. While the 1989 Domestic Well Testing Act ensures that private wells are tested at the time of a real estate transaction (RET), there is no other state or federal requirement for testing of contaminants in wells, which creates both an oversight in policy and a significant public health concern for Oregonians. With 350,000 active wells in the state and 3,800 new wells drilled each year, the need to ensure the safety of these drinking water sources represents a significant public health challenge1.
The purpose of the Domestic Well Safety Program (DWSP) in Portland, Oregon is to promote domestic well safety to users across the state. Established in October of 2013 within the Environmental Public Health Division of the Oregon Health Authority and funded by the Centers for Disease Control (CDC), OHA-DWSP is the first state-level program designed specifically to improve private well stewardship (i.e. testing, treatment, and on-going maintenance) in Oregon . Four staff members including a manager, program coordinator, epidemiologist, and health educator contribute to DWSP’s success, which further contributes to OHA’s overall mission of improving the lifelong health of Oregonians.
The evidence and recommendations produced from this report are the first step toward implementing an assessment tool to collect information on service and workforce gaps for Oregon’s private wells. Moreover, this report provides an initial means of addressing the needs across the state for education, outreach, prevention and treatment services for this vulnerable population. The American Academy of Pediatrics (AAP) defines a tool in evaluation and research as “a specific mechanism used to collect information; it is also sometimes known as an instrument or a measure. Examples of types of tools include questionnaires, clinical checklists, computer programs, and observational rating scales”. Additionally, this assessment tool project contributes to DWSP’s program evaluation efforts, which address four key outcomes, two of which the assessment tool focuses on specifically. These outcomes are illustrated in the “short-term goals” in Figure 1 and the assessment tool is highlighted as an “output and associated activity” to the left.
Finally, to provide evidence of best practices and existing assessment tools in use, and as a first step toward implementing a successful survey tool, the student researcher conducted a literature review on program assessment best practices, particularly best practices for assessing the needs of well owners and other stakeholders in private wells. This paper describes the question which the literature review attempted to answer, methods used, results, limitations, and finally conclusion with recommendations for the next phase of evaluation.
Figure 1. Shortened Logic Model for Outcomes 3 & 4: Increase in Customer Satisfaction with Domestic Well Services and Increase in Willingness of Partner Organizations to Collaborate with DWSP
Erickson, K. (2016). Figure 1 in “Development of assessment tools to measure well owner and stakeholder needs: A literature review and best practice report”, p. 3. Unpublished Manuscript
To meet the continuing public health need for protecting private wells, the DWSP staff hired a student researcher to conduct a review of the literature to provide evidence of best practices for assessing well owners and private well stakeholders’ attitudes, beliefs, behaviors, and barriers to private well stewardship. A review of the literature was conducted in May-June of 2016 to examine and address several key questions and a single overarching research question related to well owners needs and community partner/stakeholder needs for improving well stewardship in Oregon. The review was completed by a student researcher, with some assistance and input from other DWSP, CDC, and PHF project staff. The overarching research question was established and agreed upon based on several meetings with DWSP, PHF, and CDC staff in which desired goals and data needs were expressed by staff persons. The research question was expressed as, “What are program assessment best practices for domestic well programs?” This question, in addition to a set of PICOs, was used to search for and select relevant studies for inclusion in the review. This acronym provides a common framework for researchers and writers of literature or systematic reviews to develop an answerable research question with the following criteria: population, intervention, comparison, and outcome. Criteria developed under this framework came from project staff input during meetings as well as a brief preliminary scanning of the wells literature:
Figure 2. “PICOs” Used in Research Question
What are program assessment best practices for domestic well programs?
domestic well programs, private well owners, partners and stakeholders
Program assessment best practices for wells, any methods for assessing needs of well owners or other stakeholders (i.e. surveys, focus groups, in-depth interviews, case studies, workshops, etc.)
No comparison, all assessment best practices were considered for potential utility
Client satisfaction, increased awareness of the program, awareness of health hazards, knowledge of policies, and willingness of partner orgs to collaborate.
Erickson, K. (2016). Figure 2 in “Development of an assessment tool to measure well owner & stakeholder needs: a literature review and best practice report”, p. 4. Unpublished Manuscript
Research articles that focused on risk assessment, water quality, septic systems, wastewater, GIS methods or data storage methods, or toxicity and contamination were excluded from the review. Articles were also excluded if they focused a behavior change intervention that did not have a substantial assessment/evaluation component. Grey literature intended for well owners was also excluded since the focus of the review is on assessment design methods, which were not addressed in depth by these sources. Studies from outsides the United States were considered for inclusion, including Canada, and Europe which have similar or comparable regulatory structures and cultures to the U.S. Studies that discussed design or implementation of a survey instrument or assessment tool for private well owners or stakeholders were included. Other guidance documents on program evaluation and assessment or questionnaire design were also considered for inclusion in this review.
Four primary databases, PubMed, PubMed Central (PMC), Google, and Google Scholar were utilized to search for relevant articles, using different slightly different key words, phrases, and MeSH terms appropriate for each database. Citation searching was also heavily employed to obtain articles which were otherwise not captured from traditional database searching. About half the articles in this review are from citation lists of other journal articles retrieved in database searching.
Search terms included “water wells”, “health education or water supply or private water well”, “program assessment best practices for domestic well water programs”, “domestic wells”, “program assessment for private well water best practices”, “private wells”, and “water program”. Anything published within the last 20 years (1996-2016), was considered eligible for inclusion.
Based on the initial search results and information needs of DWSP, the overarching research question of “what are program assessment best practices for domestic well programs?” was divided into two separate frames: 1) for private well owners and 2) for stakeholders and are discussed here accordingly. Additionally, the lack of “best practice” literature for assessment of private well owners and stakeholders in the United States demanded a separate search for data collection best practices in evaluation and assessment tool design generally, or in any field. The most frequent practices observed in the private wells studies were then compared against the best practices for program assessment generally to verify their credibility in the field.
The appendix provides a flow diagram of the searches conducted by database and the resulting studies from each search. The remainder of this paper will examine the body of evidence collected from the identified literature and then provide recommendations for development of an assessment tool for Oregon and potentially other state domestic well programs.
Many best practices were found across literatures and will be discussed first by general assessment best practices, then by private well survey study best practices, followed by a discussion of other private well assessment methods, and finally by stakeholder assessment best practices both for private well owner stakeholders as well as general organizational stakeholders. First discussed in the results are general evaluation and assessment best practices, since these apply to private well stewardship programs and are necessary to appraise the performance of other private well survey and assessment methodologies. Moreover, DWSP wanted to learn about other survey instruments already in existence in similar private well programs which have shown to be successful, which are discussed next. Surveys or questionnaires were found to be the most frequently used method of assessment in the private wells literature. Figure 3 illustrates all of the studies, including evaluation and assessment-focused, well-owner-focused, as well as stakeholder-focused which were utilized as primary evidence in this review.
Figure 3. Summary Table of Included Studies
Erickson, K. (2016). Figure 3 in “Development of assessment tools to measure well owner & stakeholder needs: a literature review & best practice report”, p. 7. Unpublished manuscript.
General Evaluation & Assessment Best Practices
i. Best Practice: Aligning Methods with Evaluation Design
Much of the evaluation literature emphasized the need to align the data collection methods employed with the type of evaluation. For example, CDC’s Program Evaluation Manual explains, “the design you select influences the timing of data collection, how you analyze the data, and the types of conclusions you can make from your findings” . Additionally, several designs and models of program evaluation exist including: formative evaluation, summative evaluation, outcome evaluation, cost-benefit analyses, and needs assessment .
For the purposes of the DWSP, the program evaluation in question can be best classified as a kind of formative evaluation or needs assessment, because the goal is to 1) reach a new population (private well owners, especially those for whom limited data is available) and 2) ensure that programs and services which DWSP engages in are appropriate and acceptable for private well owners and other stakeholders across the state.
Moreover, the Context, Input, Process, and Product model or CIPP developed by preeminent evaluation scholar Michael Scriven further categorizes an evaluation based on whether it is the context, input, process, or product that is being evaluated. Applying this model to DWSP, the subject under evaluation is the context of need across the state for private well services and programs. Moreover, a context evaluation assesses needs, problems, assets, and opportunities plus relevant contextual conditions and dynamics to provide guidance for identifying needed interventions, choosing goals, and setting priorities. The methods of data collection which are recommended for use in a context evaluation include system analysis, surveys, document review, secondary data analysis, hearings, interviews, focus groups, diagnostic tests, case studies, site visits, epidemiologic studies, and the Delphi technique2.
ii. Best Practice: Evaluate Pros and Cons of Each Assessment Method
Moreover, the CDC advises evaluators to consider the purpose of the evaluation, the evaluation users, respondents, resources available, degree of intrusiveness, type of information, and the pros and cons of each data collection method when choosing a data collection method . Surveys or questionnaires which are most prevalent in the private well literature, provide the least amount of control over the quality of data when compared to in-depth interviews or telephone surveys. One private well survey study author however points out that they chose a “mail back questionnaire that could be completed anonymously” over a telephone interview, given the length of the questionnaire, the kinds of questions asked, and “the scale of the sample that was desired”. VanGelder et al however notes that web-based questionnaires can improve data quality since “validation checks can be incorporated with prompts that alert respondents when they enter implausible or incomplete answers”.
iii. Use Existing Data Sources
Another consideration when choosing data collection methods is to determine if existing data sources can assist in providing evaluative or assessment information, such as the Behavioral Risk Factor Surveillance System (BRFSS), the US Census, state vital statistics, and other health surveillance databases.The advantage of these sources, CDC points out is that “they have been pretested and designed to produce valid and reliable data”. Moreover, AAP emphasizes that an existing tool is valuable when “you want to assess a construct that is not directly observable, such as self-esteem”. To find existing tools, the Academy recommends looking to the professional literature or contact with other programs and professionals from programs similar to yours in goals and objectives3.
In Oregon, data sources include the RET database, well logs filed with the Water Resources Department (WRD) at the time of well construction, data from the Department of Environmental Quality’s (DEQ) private well testing programs, and US Geological Survey’s (USGS) Willamette Basin Water Quality Assessment dataset2. Other potential data sources lie in Intergovernmental Agreements (IGAs) which DWSP has and continues to develop with local health departments (LHDs). Of these data sources, the RET database is the most accessible and most used source of data by DWSP2. Figure 11 illustrates how each of these databases, (WRD, RET, USGS, and DEQ) are entered into a funnel to provide DWSP within information on private wells in Oregon.
Meetings with the DWSP staff however emphasize the sentiment that data from these sources do not capture the full picture of private wells. Thus, considering alternative sources to supplement these existing databases is essential. Guidance documents from the CDC further affirm this gap, noting that while the databases like the BRFSS are important for formative evaluation in “identifying key burden and risk factors- the descriptive and analytic epidemiology of the public health problem”, more population-specific assessment data, through targeted surveys and or a data collection plan is necessary to meet DWSP’s objectives.
Figure 4. DWSP’s Existing Data Sources
Oregon Health Authority Domestic Wells Safety Program (2014). CDC/PWI Grantee meeting, Grants and partnerships presentation. Retrieved from S:\offices\portland(800 ne oregon st\eph\healthy waters\private dw safety\dwsp\outreach events\presentations\cdc site visit
Moreover, a review of DWSP’s internal files related to past assessment efforts indicate that attempts at assessment have been made, and may be useful for developing a future assessment tool. Before the DWSP was established, the Healthy Waters Program of OHA in 2011 implemented a small survey of well owners and stakeholders to examine the impact of Oregon’s Domestic Well Testing Act (DWTA). Six key stakeholder groups were identified as desired populations to sample from and private well owners were not directly solicited 2 Thirty-seven respondents were invited to participate, and 30 completed and returned the survey between March 2011 and June 2011. The survey asked respondents to indicate how information collected from DWTA should be used, suggestions for improving the DWTA law, the how to improve the DWTA data collection process, and user experience with the DWTA law1Results from the survey suggested that Oregon’s DWSP collaborate with other state and federal agencies which maintain datasets to enhance the utility of the RET dataset, and that the program should make efforts to understand the social, economic, and demographic characteristics of communities potentially affected118. Additionally the 2011 DWTA survey report authors recommend securing adequate and sustainable funding to promote well water safety across the state118.
iv.Best Practice: Comply with Ethics Requirements
Furthermore, approval from an Institutional Review Board may be required if the survey or assessment tool falls under the classification of research, as opposed to public health practice 115 . Research ethics also proscribe that assessment tools make transparent: risks/benefits of participation (i.e. informed consent), privacy and confidentiality, rationale for selection of subjects, and any additional safeguards so that participants do not feel forced to participate and are protected from harm. Six of the 11 private well survey studies examined in the literature obtained IRB or ethics board approval. Another ethical consideration is how to make the assessment tool accessible to all populations, particularly non-English speaking people. Williams provides a suggestion for addressing language barriers to include a cover letter that has been translated into common languages for the target population with a contact number for an interpreter.
v.Prepare a Data Management Plan
AAP recommends setting up a system for storing information before beginning data collection. Moreover, because the survey data is assessing at a single time-point and via a single form, a simple excel spreadsheet or workbook is likely sufficient to store the data. If we wanted to conduct a follow-up test however and compare time points, we might consider a relational database such as Microsoft Access3. The Academy also recommends starting data entry immediately141. Additionally, Williams suggests deciding on a coding scheme that fits with the questions selected for the survey, before data collection begins. For example, one might number each option in a multiple choice question and assign variables (i.e. does test for arsenic, does not) within a database or excel spreadsheet2.
Data Analysis Based on Evaluation Questions
A further recommendation is to base the data analysis on the original evaluation questions3, or DWSP’s logic model. Essentially the evaluation questions for the assessment tool are: What are service and knowledge gaps among the private well community, both stakeholders and well owners, and who are private well owners in Oregon and what are their needs? These kinds of who and what questions address more descriptive statistics, derived from various summary measures of the data3.
Private Well Owners
Studies which utilized surveys to assess well owner behavior, knowledge, and or attitudes included Chappells 2015, Flanagan 2015 (Dissemination), Flanagan 2015 (Influences), Flanagan 2015 (Arsenic), Hexemer 2008, Hynds 2013, Jones 2006, Kreutzwiser 2010 (Main Report), Liukonnen 2009 , Straub 2014, and Summers 2010. The findings from analysis of these studies support Outcome 4 of Oregon DWSP’s logic model: increasing customer satisfaction with DWSP services and stewardship behavior through development of an assessment tool directed towards well owners. The survey instruments from each study were examined in-depth and abstracted for information such as length of survey, types of questions, content, and key findings or questions of significance. Additionally, best practices from the literature search for general data collection in program evaluation were highlighted in the private well studies and tabulated, both by study and by practice, to report on data collection best practices used in the private well literature. Figure 5 illustrates a column chart of the 14 best practices identified across the private well survey studies and their relative frequency across all studies.
As illustrated in Figure 5 (Most Frequently Reported Best Practices across Studies), several best practices were observed across both the private well survey studies and the program evaluation/assessment development literature. Moreover, these were best practices both recommended by the experts in the evaluation and survey development literature and implemented by the private well researchers in the included studies. The best practices repeatedly found throughout the both literatures included: use of a random sample, obtaining IRB approval, offering the instrument online and in hard-copy, using mixed methods, conducting pre- and post- tests, using a contact method strategy, and crowd-sourcing the survey or assessment questions. Other practices which were not necessarily highlighted by the evaluation and assessment literature but implemented in the private wells studies included: use of knowledge brokers, using a health behavior theory to guide the assessment questions, selecting the study area based on a high private well density and groundwater contamination, offering an incentive, and using state GIS data to identify respondent addresses.
The use of incentives for the purposes of medical or healthcare research is typically frowned upon, especially when the risks of the experimental intervention to the participant are high and therefore pose a significant ethical problem. However, Grant and Sugarman point out that the incentives only become problematic in cases where the risks are in fact high for participants or in instances where the study subject develops a dependent relationship on the researcher. A study design such as a survey on the other hand, particularly regarding non-controversial information, poses little risk to the participant, and would therefore be an acceptable setting in which to implement an incentive42. Therefore, providing incentives was considered a best practice for assessment of private well owners, since it is not only ethically appropriate under the circumstances but also was observed in several of the private well studies as well as suggested in Dillman’s Tailored Design strategy for improving response to mail surveys
Figure 5. Most Frequently Reported Best Practices across Studies
Erickson, K. (2016). Figure 5 in “Development of assessment tools to measure well owner & stakeholder needs: A literature Review & Best Practice Report”, p. 9. Unpublished Manuscript
The application of health behavior theories to environmental health and private well studies is also considered a best practice because as Straub remarks, “models of health behavior change, including the HBM, have been applied with great success to health behaviors”. The employment of knowledge brokers was also unique to the private well survey literature. Given that the study population is hard-to-reach, knowledge brokers are a trusted source or ally in the community who can help relay risk information and the importance of completing an assessment or test to local respondents Additionally, the selection of the study area based on private well density was a practice implemented across all studies and is an obvious best practice for measuring the beliefs and behaviors of private well owners. Further, several studies utilized GIS technology to identify respondents which is a best practice because of the precision of these information systems and their capacity to improve the response rate for the assessment and further strategize a contact plan.
Across the private well survey studies the most frequently implemented best practices included application of a health behavior theory to the design of the study and selection of the study area based on high private well density and groundwater contamination, both of which all of the studies adopted. Less common, though frequently talked about in the program evaluation literature was the implementation of an online option to complete the survey or adoption of more than one method for dissemination of the survey. Moreover, as Figure 6 (Studies by Number of Best Practices Reported) below illustrates, the private well studies were comparable with respect to the number of assessment best practices which they each implemented. Studies which had the highest number of best practices in their methodologies included Flanagan et al (all three studies), Chappells et al, and Summers et al. What follows is a description of the best practices identified across both the private well surveys and the program evaluation/assessment literature for assessing well-owner beliefs, behaviors, and attitudes through surveys or questionnaires.
Figure 6. Studies by Number of Best Practices Reported
Erickson, K. (2016) Figure 6 in “Development of assessment tools to measure well owner and stakeholder needs: a literature review & best practice report”, p. 12. Unpublished manuscript
Well-Owner Assessment Best Practices
Best Practice: Using Mixed Methods
According to the literature, the use of mixed methods is a best practice if not an essential element in survey and assessment design because as the CDC’s Program Evaluation Manual points out, “different methods reveal different aspects of the program” and “employing multiple methods helps increase the accuracy of the measurement and the certainty of your conclusions” . However, while utilization of mixed methods approach may be ideal when designing a survey, author of the Tailored Design Method for government-sponsored establishment surveys Don Dillman adds the caveat that mixed-mode designs may be impractical, particularly if the organization already has a complete list of possible contacts, as increasing the number of methods will not necessarily improve the quality of the data. When a master contact like this is not available, the use of mixed methods (i.e. phone, mail, and or online) can provide potential respondents with alternative ways to get in touch and increase their likelihood of responding37. Furthermore, mixed methods can also mean not just providing additional ways to complete the survey, but also (or instead) offering alternative ways to communicate with the respondent through telephone reminders and or thank you postcards37. On the other hand, increasing the number of methods also adds to the cost of the study as well as to the statistical complexity of the assessment during the analysis stage. Nevertheless, mixed method approaches to assessment and evaluation are increasingly used in lieu of strictly quantitative or qualitative designs. Moreover, in spite of the limitations and additional resources they require, the ultimate goal of using mixed methods is to actually reduce error and costs and expand the possible number of people who can be included in the sample frame by offering additional opportunities or means to participate”. For these reasons the use of mixed methods for administering surveys or questionnaires was considered a “best practice”.
Of the 11 private well survey studies examined, only one used telephone as the primary method of administering the questionnaire. According to the study author, Hexemer, “the survey had 15 questions and took approximately 10 minutes to complete by telephone”51. Dillman suggests avoiding telephone surveys to reduce administrative burden and maximize efficiency except as a “last ditch effort” to reach respondents. Other program evaluation sources however, such as the CDC program evaluation manual suggest that telephone interviews may have a better response rate than mailed surveys for public health programs. Dillman’s most recent book Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method (2014) contradicts this assertion however, stating that “mail has moved from being the lowest response rate mode for many survey designs to now having response rates that are significantly higher than telephone”2. Moreover, mail surveys, once considered the lowest cost option, are now relatively higher cost compared with e-mail and web surveys2.
The conclusion Dillman makes given these changes in survey trends is to triangulate methods of contact to enhance response rates and reach as many respondents as possible54.Of the eleven surveys examined in the well literature however only one used a mixed method approach. Chappels et al developed a questionnaire survey for private well users and conducted semi-structured interviews with stakeholders to investigate factors influencing stewardship behavior.
ii. Best Practice: Offering the Survey Online and in Hard Copy
As the number of devices and forms of communication increases, so do the possibilities for survey administration. VanGelder et al remark that while the use of online surveys or questionnaires may become a more attractive alternative avenue for administering surveys, the option remains scarce in epidemiologic research due to concerns about nonresponse bias and reliability of the data obtained Still, vanGelder mentions research which indicates that many respondents prefer web-based questionnaires over telephone or postal surveys, and that provision of a PDF version of the survey instrument has shown success in improving response rates. Additionally, three of the 11 private wells survey studies provided a version of the survey online.
The advantage of having an online option is the same for a mixed-methods approach: it increases the possible sample size by capturing more audience preferences in communication. For example, Chappells et al designed their survey as a “postal survey to be completed in participants’ homes with an online survey option also made available to maximize participation”. Additionally, Summers et al report that they also provided an online version of their private well owner survey, and used the website SurveyGizmo to administer the online version. Flanagan et al also offered an online version of their survey to New Jersey private well owners via a URL with a unique code for each participant. By providing an online option in addition to a paper questionnaire, survey administrators are able to capture the greatest possible sample population and enhance the response rate, making this a best practice.
iii. Best Practice: Crowd-Sourcing Survey/Assessment Questions
For development of survey questions, Flanagan et al utilize small community meetings in affected towns in order to create their survey instrument. The questions developed from these meetings were also used and modified in Flanagan et al’s other two studies, . Williams explains succinctly why exactly crowd-sourcing survey questions is considered a best practice:
“…people are more likely to respond to questionnaires that cover issues that are relevant to them. Qualitative methods, including focus groups and unstructured interviews are increasingly being used to identify issues of importance to patients as a first stage in questionnaire studies”.
By conducting these preliminary forums or informal discussions with key stakeholders and affected populations, survey administrators are able to gather insider information that may be useful for writing more targeted questions. Jones et al implement this practice by conducting three focus groups with local residents, which then formed “the content and vocabulary of the questionnaire, as well as question categories and answer choices. Additionally, Kreutzwiser et al conducted a literature review which highlighted key issues and questions which formed the background of a workshop for private well stewardship experts. The information obtained from both the literature review and the workshop were then utilized to develop the survey questions5. Similarly, Hynds et al also conducted a literature review to highlight issues of importance to private well owners which were then developed into survey questions. Of the 11 survey studies examined in the literature, six studies total, including three by Flanagan et al, implemented this best practice.
iv. Best Practice: Using Health Behavior Theories to Guide Questions
All of the private well surveys asked questions related to testing behavior, such as frequency of testing and treatment. Moreover, they asked for the well owners’ thoughts and beliefs about certain behaviors or practices related to well stewardship, which health behavior theorists believe can predict health behavior action. Ellen Taylor-Powell, Program Development and Evaluation Specialist at the University of Wisconsin Cooperative Extension points out that “it is essential to know what kind of evidence you need” to fulfill the goal of the assessment, and that there are four different types of information which can be assessed in a questionnaire: 1) knowledge, 2) beliefs, attitudes, and opinions, 3) behavior and 4) attributes.
One theory found in the literature on domestic wells is the Health Belief Model (HBM). The theory is focused on individual behavior and identifies four constructs which may predict the adoption or non-adoption of a health behavior, such as well stewardship. The four constructs include perceived threat, perceived severity, perceived benefit, and perceived cost or barriers. Recent iterations of the model also include self-efficacy and cues to action as constructs. Straub and Leahy utilized the HBM to develop a survey which they administered to parents and children who participated in a groundwater education program through several New England schools. Based on logistic regression, the survey results showed that perceived barriers, income, and education significantly influenced past well water testing1. Additionally, compared with several other variables and HBM constructs, the survey questions which assessed cues to action were significantly associated through an ordered logistic regression with parent decision making of future well water testing1. Examples of these kinds of “cues to action” questions include: “Would the following prompt you to do a well water test--- getting a reminder, if a well testing program was available, getting a discount, etc.”1. Ten items or statements assessed this particular criteria of the HBM on the survey. Figure 7 is an excerpt from the survey administered to parents in the study, with a series of statements focused on perceived severity, perceived threat, perceived cost, and perceived benefits.
Figure 7. Health Belief Model-Oriented Survey Questions
Straub & Leahy (2014).Environmental education in New England: Parent Survey. p 1, from “Application of a modified health belief model to the pro-environmental behavior of private well water testing”. JAWRA Journal of American Water Resources Association
Further, three of the eleven surveys (all by the same author), utilized a Risk, Attitude, Norm, Ability, and Self-Regulation (RANAS) model , as a means of gathering information regarding motivators for behavior change. According to the study authors, the RANAS model outlines the five “blocks of determinants that must be favorable in order for a behavior, such as regular well testing, to take root”. The authors further explain that targeted, more effective interventions can be designed once the motivating factors for behaviors have been identified2. The model’s original author, Mosler1 further states that the model “includes several theories from social and health psychology”, including the Health Belief Model.
To measure the RANAS constructs, the surveys each include a series of statements in which respondents indicate whether they agree or disagree on a scale from 1 to 6. One of the surveys utilizing the RANAS model found that perceived personal risk, communicated through the statement, “my untreated water is not safe”, was one of the most significant predictors of mitigation behavior. Another predictor this study found for well stewardship was the belief that treatment can improve home value3. Figure 8 illustrates the RANAS variables as applied to the information or questions assessed on the survey instrument. Questions or factors which were statistically associated with mitigation behavior at the .01 level are highlighted in yellow by the student researcher.
Figure 8. Flanagan et al. Use of RANAS Variables
Flanagan et al (2015) Dissemination of well water arsenic results to homeowners in central Maine: Influences on Mitigation behavior and continued risks for exposure. Science of the Total Environment, 505, p. 1287
v. Best Practice: Format and Order of Questions according to Importance
Question placement and formatting is a further consideration when designing surveys and questionnaires. All of the surveys provided some kind of instructions at the beginning of the questionnaire and all surveys requested that the respondent verify at the beginning of the survey or questionnaire that they own or live on a property with one or more private wells and that they obtain their drinking water from that well. Powell confirms this as a best practice by suggesting that survey designers, “begin with an introduction that includes the questionnaire’s purpose, how the information obtained will be used, and assures respondents of confidentiality”. Williams also notes that it’s important to consider how to capture people’s attention and make respondents interested in completing the questionnaire when designing the layout. All of the surveys examined used a variety of question formats, including Likert scales, agree/disagree, yes/no, checkboxes, ranking of choices, multiple choice, selection of frequency for certain behaviors, true/false statements, fill in the blank, and short answer. Moreover, Powell points out that open-ended questions, or fill in the blank and short answers, are “often the easiest way to ask for information, but the responses are not easy to analyze”3. Meanwhile, a two-option response (i.e. yes/no question), Powell informs readers is often used to begin a series of questions on a topic. This is demonstrated by many of the private well surveys, which begin by asking some form of “does your property rely on a private well for drinking water?”--> answer: yes/no).
Further, Powell suggests placing demographic questions such as age, sex, and income level toward the end of the questionnaire and have the first questions be “easy, avoiding controversial topics”. All of the private well surveys reviewed in the literature which provided the full text documents of their surveys did this. Moreover, all six surveys asked for demographic information such as educational attainment and age. This information can be useful, especially since many of these studies also find significant associations between well stewardship and higher socioeconomic status. For example Flanagan et al’s follow-up survey for the dissemination of arsenic test results in Maine showed that use of a treatment system was associated with higher income and higher education. In the same study conversely, drinking bottled water was associated with lower income and lower education. Figure 9 is an excerpt from a survey given to New Jersey residents illustrating some of the kinds of demographic information requested of respondents as well as the format and order of those questions.
Surveys ranged in length from 8 to 15 pages, and from 12 to 38 questions. Dillman writes that questionnaires should have as few pages as possible to keep response rates from going down. Moreover, “the shorter the questionnaire, the lower cost of mailing and retrieving it”1
Figure 9. Demographic Information Fields From New Jersey Survey
Flanagan et al ( 2015). New Jersey household water and testing treatment survey, p. 8 from “Arsenic in private well water part 1 of 3: Impact of New Jersey Private Well Testing Act on household testing and mitigation behavior”. Science of the Total Environment, 562, p. 999-1009.
v. Best Practice: Having a Contact Strategy
Other highlights of best practice methods employed by the private wells studies include having a formal contact strategy such as Dillman’s Tailored Design Method of repeated contact, which is a set of procedures intended for government-sponsored surveys designed to increase response rate designed by Washington State Sociology Professor Don Dillman . These coordinated methods of outreach include: ‘four to five carefully-timed contacts that support one another through their wording and timing’, ‘personalized correspondence that includes real letterhead stationery, dates, and is addressed to individuals’, ‘at least one mail contact that is special’, ‘inclusion of a return envelope that contains real stamps’, and ‘switching to another mode such as telephone or e-mail for non-respondents’31.
In the New Jersey study by Flanagan et al for example, the researchers utilized Dillman’s method by mailing to participants: “a pre-survey letter, a survey with a cover letter and postage-paid return envelope, and thank-you/reminder postcard about 1-2 weeks later”. Both of Flanagan’s other two studies, also use the same contact method, one of which refers to Dillman specifically3. Straub and Leahy meanwhile do not mail the surveys to respondents but provide it to teachers in classrooms to send home with children to parents. In Minnesota, Liukkonen et al used Dillman’s contact method to survey private well owners and characterize the factors that influence private well testing.
Moreover, an “advance letter”, or “pre-survey letter” such as used by Flanagan et al in both their New Jersey and Maine surveys is a critical element in a survey or assessment contact strategy. Dillman points out that these letters sent ahead of the actual survey are commonly used to alert the respondent to the upcoming questionnaire. Dillman further suggests that advance letters should include information such as a personal contact, assurance of anonymity or confidentiality, discussion of the survey organizer’s responsibility, a discussion of benefits for completing the survey, and any direct or indirect costs to the respondent in completing the survey5. Williams also notes that Ethics Committees or Institutional Review Boards often require that these letters be part of the research plan as part of “informed consent”. Other information which an advance letter or pre-survey letter should disclose include the goal of the survey as well as any relevant or basic information about the survey organization31. Figure 10 shows an excerpt from A. Williams’ article6 on best practices for writing and analyzing a questionnaire which mentions many of the same elements of the Dillman method.
Figure 10. Williams 2003 Recommendations for Increasing Response Rates
Williams (2003). How to…Write and analyse a questionnaire. Journal of Orthodontics, 30(3), 245-252.
vii. Best Practice: Using Knowledge Brokers to Reach Respondents
Identification of potential participants is another key step in the data collection process, and may be the most challenging step in a private well survey study due to the often remote location of properties with a private well. To identify possible study participants in Central Maine, Flanagan et al contacted town offices and department of motor vehicles for lists of well owners. Alternatively, Straub and Leahy selected participants from schools in the area that were identified as having at least 60 students on well water, and subsequently administered the survey to parents and students. Yet another study obtained participants through volunteers from a previous well testing program. Moreover, Chappells et al point out that while most information about wells is disseminated at the transfer of property, key knowledge brokers or gatekeepers exist such as realtors, developers, and home builders and “represent important risk communication channels that could be better utilized”. Of the 11 private well survey studies, 6 used knowledge brokers of these kinds to identify participants.
viii. Best Practice: Pretesting the Survey or Assessment Tool
The CDC and other assessment experts recommend finding people similar to the study population to test the survey on once the questions have been drafted, or conducting a “pre-test”, which nine of the eleven private well studies did. Draugalis et al moreover categorize pre-testing as a process “often conducted with a focus group to identify ambiguous questions or wording, unclear instructions, or other problems with the instrument prior to wide-spread dissemination”. Focus groups work well for collecting qualitative information by encouraging participants to talk candidly about the research question or intervention, but often require a skilled facilitator to moderate the discussion which can add to the cost of the research project. Another option which may be more accessible for less resource-rich organizations may be cognitive testing. Draugalis describes cognitive testing as an alternative to a traditional pilot-test, in which the survey researcher has the opportunity to get immediate and direct feedback from a sub-sample of respondents through one-on-one interviews by the survey developers or other members of the survey research team81. During the interview the respondent is able to explain how they interpreted items on the questionnaire and what particular words may mean to them, which gives the research team the ability to correct their survey instrument to better communicate the survey’s objectives and assessment items81.
A. Williams from the University of Bristol Dental School adds that ideally, every questionnaire should undergo a formal pilot study, or pre-test, during which the acceptability, validity, and reliability of the tool is tested . Finally, Hynds et al note further that data from the pilot study, as a best practice, should not be included in the final analysis.
ix. Best Practice: Post/Follow-up Testing
In addition to pre-testing, another best practice demonstrated in the private well literature was the implementation of post or follow-up testing to measures the change in behavior over time, particularly when an intervention is conducted. Straub and Leahy collected post-survey information for their survey of children and parents in schools with high well density, but it was not discussed in their published article12. CDC’s Program Evaluation Manual describes follow-up as “the support that users need throughout the evaluation [or assessment] process”. The American Academy of Pediatrics (AAP) Manual for Evaluating Community-Based Programs recommends conducting post-tests to provide comparison information and reduce potential for bias or confounding. The AAP manual also suggests the usefulness of post-tests in assessing changes in knowledge, either based on an intervention, outreach program, or simply information provided or attached with the previous survey2.
1.2) Well-Owner Assessment Best Practices (Other than Surveys)
Other Best Practice for Assessment: Focus Groups/Hearings
The first model outreach program found in the literature was an expert panel convened at Johns Hopkins School of Public Health, Baltimore in 2015, to inform strategic planning of the CDC’s Clean Water for Health Program, which works to protect private wells. The panel developed several recommendations, covering domains of outreach, policy development, assurance and more to promote protection of private wells and advance the practice. As a data collection method, this could be considered a large-scale forum or focus group where needs are addressed. The panel identified several recommendations for collecting data on private wells and their stakeholders, two of which speak to program assessment best practices beyond traditional surveys or questionnaires. These include 1) “promoting strategic use of existing data” and 2) “conducting research to better characterize users of unregulated water with regard to demographics, socioeconomics, knowledge, attitudes, and practices”. Researchers at the Northwest Center for Public Health Practice point out that focus groups are useful for instances when “you want to collect in-depth information from a group of people about their experiences and perceptions related to a specific issue”. Focus groups and other forums however also require a skilled facilitator and may not be appropriate for all populations and topics. Still, several private well studies utilized community meetings, expert workshops, and city focus groups to elicit feedback from well owners and stakeholders as an initial data collection to highlight needs and concerns which were then quantitatively assessed through a survey.
Other Best Practice for Assessment: System Analyses
Another notable practice from a state health agency towards improving education within the state involved Louisiana’s Department of Public Health, which recently took on an evaluation of available data sets for private wells within the state to utilize for surveillance of arsenic hazards . The researchers determined that the EPA’s Safe Drinking Water Information System (SDWIS), and the USGS Water Use data set were “the most informative nationally available databases” with respect to measures of arsenic in groundwater and soil, private well water use, and biomonitoring4. These methods represent a kind of system analysis data collection method, by bringing together different systems and comparing their assets.
Other Best Practice for Assessment: Secondary Data Analysis
At the national level and an example of secondary data analysis, staff from CDC conducted analysis of 1,100 calls from private well owners to the national WellCare Hotline for the year 2013. Nearly 80 percent of the calls analyzed were about well water testing, which the researchers believed indicated an important information gap1. Records from the hotline also indicated that callers were not engaging in adequate well maintenance activities, with only 38% documented as conducting well stewardship practices1. The study’s authors suggest that further research (i.e. surveys, assessment tools) should examine well owners’ knowledge, attitudes, and practices to improve outreach and enhance stewardship behavior1.
Other Best Practice for Assessment: Literature/Document Review
Additionally, a recent literature review of outreach methods to private well owners by Morris et al (2015) further supports the utility of health behavior theories and models to the development of survey questions discussed in Sub-question 1. Health behavior and health psychology are fields which seek to understand the underlying reasons why individuals or groups of people choose or choose not to engage in healthy behaviors such as maintaining their well107. Possible reasons for behavior change or lack thereof can include lack of knowledge or information, social norms, perception of risk, inconvenience, or cost. Morris et al argue that understanding the underlying factors of behavior can improve educational efforts and outreach programs by allowing program managers to target their messages and programs to specific audiences and maximize potential for positive behavior change. For example, if survey results indicate that the biggest barrier for the population of interest is that well owners do not feel that contamination of their well is a real risk, then the outreach response to this would be to design social marketing and community outreach that emphasizes the perception of risk and the dangers of drinking untested well water1. By examining existing external documents this review represents a kind of document review for the purpose of data collection. Moreover the review of existing data sources discussed under Sub-Question 1 also represents a document review, by going through reports and documents internal to DWSP.
Stakeholder Assessment Best Practices
Most of the literature addressed private well owners as the assessment audience. Few studies provided best practices specific to private well programs for assessing stakeholder needs and concerns. This final question provides evidence of other assessment tools available which will help DWSP achieve outcome 3 in their evaluation, or increasing the willingness of partner organizations to collaborate in data collection, surveillance, outreach, and best practices. Stakeholder Research Associates of Canada define stakeholder engagement as “an umbrella term that covers the full range of an organization’s efforts to understand and involve stakeholders in its activities and decisions . Some of the benefits of this engagement include building public support, increasing public awareness, leveraging additional funding sources, and reducing the need for oversight and enforcement. A variety of other stakeholders and partner organizations play a key role in ensuring that the water from private wells is safe to drink, including testing and treatment labs, well drilling and construction companies, and water delivery businesses. As such, it is critical that these stakeholders who interface directly with well water consumers have the necessary resources and personnel contacts with government entities in order to ensure and promote and public health.
Stakeholder Assessment Best Practice: Surveys
Foremost in the literature for assessments of other stakeholders was a report of two surveys disseminated to water testing labs in Southeastern New York State in 2011 and 2012 by an Extension Associate from Cornell University. Questions which labs were asked on the surveys included questions about water sample submission instructions, discounts available for multiple tests, tests offered in packages, providing recommendations for certain tests, provision of sample collection services, reasons why clients test, how labs report results, educational materials provided, certification, and testing capabilities2. Additionally, the Community Water Center (CWC) in California offers several example questions which should be included in an initial assessment of stakeholder interests and priorities 113. Some of these questions include: 1) what are their interests, concerns, and priorities? 2) What are the best tools for communicating with them? What barriers might they face in participating in well stewardship promotion, testing, treatment, or collaboration, and 3) Who else do they believe should be involved?.
Stakeholder Assessment Best Practice: Mixed Methods
While surveys are a popular form of assessment in practice and in the literature, other methods previously discussed for well owners are also available for obtaining feedback from private well stakeholders and partners, including: focus groups, public meetings, workshops, and public comment. Gina Abudi, an experienced project management consultant suggests several best practices for gathering information from stakeholders, regardless of the industry or project. First, Abudi suggests using a variety of tools to assess stakeholder engagement, including online surveys, focus groups, one-on-one interviews in person, conference call interviews, and small group facilitated workshops. The California non-profit environmental group, CWC also provides recommendations for tools which can be used to develop “shared understanding on technical subject matter” such as well testing, treatment, and maintenance. Some of the methods identified in the report for sharing information include joint fact finding, establishing a technical advisory committee, web-based tools (i.e. searchable databases or well inventories), third-party neutral researchers, and collaborative models and decision-support tools 115.
Stakeholder Assessment Best Practice: Stakeholder Analysis
Abudi also points out that while it may not be possible to include all relevant stakeholders in this assessment process, it’s important to make sure that the “right people” are present. One way to ensure that the right people are at the decision-making table and being considered is by conducting a stakeholder analysis, for which many methods are available. After stakeholders have been categorized and prioritized according to a chosen criteria, a priority list can be created for whom to continually engage and include in the program development or decision-making process.
Stakeholder Assessment Best Practice: Use Existing Data/Contacts
Finally, the project management consultant suggests reviewing data on stakeholders that may already be available. For example a previous project from DWSP maintains a list of contacts under the “Private DW Safety, Survey” folder on its hard-drive which may still be relevant for use currently. The Community Water Center also recommends “identifying and leveraging existing community forums”.
v. Stakeholder Assessment Best Practice: System Analyses
One example of a successful implementation of an assessment tool intended for well stakeholders comes from the Colorado Department of Public Health and Environment (CDPHE)120. A collaborative effort between the CDC’s Private Well Initiative, the Environmental Public Health Tracking Network, and the CDPHE developed a survey tool to assess other state partner’s data collection capacity120. Based on preliminary data collection from the state’s Behavioral Risk Factor Surveillance Survey (BRFSS), an inventory tool to gather metadata from private well data stewards was created and developed by the researchers and collected information related to water quality and quantity as well as data steward information and accessibility of the data. Agencies engaged in collecting private well water data were identified by the researchers, including private laboratories, the Colorado Department of Public Health and Environment Laboratory Services Division, the Colorado Department of Agriculture, the Colorado State University Extension, the USGS, the Colorado Geological Survey, the State Engineer’s Office, and several local health departments and were asked to fill out the web-based inventory tool regarding the kinds of specific information they collect. The survey/inventory tool covers several domains including general information about the agency's dataset, geographic details related to the dataset, well information within the dataset, water property details, contaminant details, and a comment box. Based on the information collected from this inventory tool, the collaboration between the CDC and Colorado state department of health was able to develop a standard set of well water measures to be displayed on Colorado’s Tracking portal and aggregate and summarize the approximately 18 datasets identified across the state on their web portal120. Figure 11 shows the homepage for this data portal as well as a portion of the inventory tool used to create the data portal which was obtained from the study’s primary author.
Figure 11. Colorado Tracking Portal for Private Wells
Brown et al (2015). Survey Example Provided with Permission from Study Author, originally reported on in: “Private Well Water in Colorado: Collaboration, Data Use, and Public Health Outreach”. Journal of Public Health Management and Practice, 21.
Regardless of the end user or assessment method, the overall goal of the chosen assessment method for the intended audience is the same: to engage outside individuals into the organizational decision-making process and to ensure satisfaction of those individuals related to their interactions with organization and the services which it provides. Because of the unique focus of this project, a variety of different sources were considered to answer the question of: “What are program assessment best practices for domestic well programs?” Articles were drawn from a variety of disciplines, including business, environmental science, and public health program evaluation, demonstrating the complexity of the research topic.
In spite of this multi-disciplinary scope of literature, many of the program assessment best practices were the same for both private wells, public health programs, other programs, as well as across different users, for both stakeholders and well owners. For example, for either well owners or stakeholders and regardless of the program, best practice dictates the survey administrator should conduct a pre-test to make sure that questions on the survey are relevant for the respondent, . Moreover, both stakeholder and participant, assessment experts recommend utilizing mixed methods to increase response rates. Additionally, use of existing data is both recommended and highly utilized for both stakeholder and well owner assessments and for any kind of program assessment. For example in the private well literature, Flanagan et al rely on data from the Superfund Research Group of Columbia University and the Maine Geological Survey while Chappells et al developed their sample based on a longitudinal health cohort called the Atlantic Partnership for Tomorrow’s Health (Atlantic PATH).
By identifying best practices for assessing well owners and stakeholders, DWSP moves closer towards characterizing this hard-to-reach population. The evidence and findings from this report have subsequently informed the development of two assessment tools designed specifically for DWSP to implement in Oregon amongst private well owners and stakeholders (see appendix).
Some limitations to this literature review of best practices include the fact that the student researcher worked primarily alone and without much background knowledge of the subject or expertise. An additional limitation to this report is that the researcher did not extensively contact or consult other experts personally (i.e. via e-mail, or in-person), to find evidence or literature which may have not been captured by database searching. Further, more specific and informed searching of databases could result in a greater and more relevant number of studies. For example, little evidence was found on the issue of assessment of stakeholders, whereas much of the private wells literature focused on well owners. An updated literature review might look more in-depth and targeted towards private well stakeholders, through creative searching and greater expertise in the field of groundwater and private water supply quality.
The evidence of best practices from this literature review support both the objectives of DWSP’s overall evaluation plan as well as implementation of two different assessment tools for well owners and stakeholders (see appendix). Significant similarities were found between the private wells literature and the program evaluation/assessment literature with respect to best practices for designing assessment tools. Moreover, several best practices for designing survey assessment tools were identified across both literatures, including: usage of a random sample, offering the survey online and in paper, obtaining IRB approval, using mixed methods, crowdsourcing question items or issues, pre-testing and post-testing the audience/tool, and use of a contact method or strategy.
Additionally, other best practices observed in the private wells and program evaluation literatures for assessment tools beyond just surveys included conducting focus groups or large expert workshops, running system analyses by examining, weighing, and compiling different data sources to develop a single source or improve understanding of information houses, implementing secondary data analysis, or performing a literature or document review. Finally, several best practices for administering assessments of stakeholders were observed, which included disseminating surveys, using mixed methods, carrying out a stakeholder analysis, and finally using existing data or lists of contacts.
With these best practices in mind the student researcher has provided recommendations and next steps for DWSP’s implementation of an assessment of private well owners and stakeholders in Oregon. The recommendations are discussed along with their application to the 10 Essential Services of Environmental Public Health. DWSP’s overall evaluation plan is guided by the CDC’s Ten Essential Environmental Public Health Services, which were developed in 1994 following the Institute of Medicine’s publication of the Future of Public Health. They are a set of activities modified from the ten Essential Public Health Services, and are based on the three core functions of public health: assurance, assessment, and policy development.
Recommendations & Next Steps
i. Use of Survey Method for Assessment Tools
Because surveys were noted as the most frequently and successfully-used methods of assessment across the private wells literature, we recommend incorporating two surveys as part of the assessment design, one for private well owners and one for stakeholders. Further, given the evidence of best practice, a mail-back survey and online version are recommended as methods for administering the assessment tool. By conducting a survey of both private well owners and stakeholder in Oregon, DWSP will monitor the health and environmental status of residents and be better equipped to address health hazards. Moreover, information and conclusions drawn from these surveys can be used to update DWSP’s services and programs as well as referrals to resources via their website. Finally, implementation of the assessment tool via two surveys to private well owners and stakeholders respectively allows DWSP to fulfill its task of evaluation and complements other elements in the overall evaluation plan underway.
ii. Oregon DWSP 2016 Well Owner Survey Design
The first survey tool recommended is a modified version of Flanagan et al’s New Jersey Private Wells Survey. The student researcher will request the original author’s permission to use the questions once DWSP has approved the document. The questions from this particular study were selected because as a whole, this study was one with the greatest number of best practices and also because the regulatory climate in New Jersey is similar to that of Oregon, in that they both have private well Real Estate Transaction laws. Additionally, the questions from the New Jersey study had been pre-tested and validated in two other private wells studies and were drawn from community meetings and a model of behavior change originally developed for the Water, Sanitation, and Hygiene sector in developing countries. Consequently, this particular study and survey represented the best fit with DWSP’s needs.
iii. Oregon DWSP 2016 Private Well Stakeholder Survey Design
The stakeholder assessment tool should also utilize a survey design, which was demonstrated successfully in the literature by other private wells researchers,. By law, Oregon well owners are required to test their well water at the sale of a property. In order to assure enforcement of this law, OHA must verify that labs, real estate agents, drillers, and other stakeholders are aware of and capable of providing the necessary services to well owners when this time comes. By conducting a survey of these stakeholders OHA is both increasing awareness and identifying knowledge gaps to better enforce the Oregon Real Estate Transaction Law, and fulfill its public health duty to enforce laws and regulations that protect and promote public health.
The questions from the instrument should assess stakeholder’s practices with regards to testing and treatment, as well as identification of their role in private well stewardship, education, and outreach. To apply best practice as well as accomplish DWSP’s program objectives, questions for the stakeholder survey should borrow from questions used in Brown139 et al and Galford et al140, which previously assessed private well testing labs as well as government and other stakeholders’ datasets and databases. Permission from the study authors will be requested once the survey document is approved by DWSP. By assessing stakeholder confidence in private well stewardship and providing services to well owners, implementation of a survey can show service gaps and areas where stakeholders may need assistance in providing resources to well owners. Moreover, enquiry about stakeholders practices adds an extra check of surveillance and encourages their use of best practices, such as providing written instructions with a sampling kit, helps to assure that those who work to maintain private wells are capable and qualified.
For both stakeholder and well owner surveys, the evidence suggests providing an online version of the survey through web software such as SurveyGizmo, a free survey creation website, used also by the Summers et al’s Alberta Well Water Survey. To contact participants for the surveys, DWSP should employ Dillman’s Tailored Design Method as used in many of the other private well surveys and as widely acknowledged to be a best practice.
iv. Pilot Test both Surveys
To comply with best practices found in the private wells and evaluation literature, , DWSP should pilot test both survey instruments to ensure that the questions are valid, reliable, and applicable to Oregon’s well owners and private wells stakeholders. Additionally individuals selected for the pilot test should be representative and similar to the overall sample frame for each study population and can be either a random sample or convenience sample. To conduct each pilot test, a version of the survey will be distributed to approximately 15 respondents identified to participate, through mail and electronic means a week before an informal meeting to discuss the survey instrument. This will allow pilot survey participants to review the survey instrument, complete and submit it, and come to a meeting with a DWSP staff member(s) with feedback, revisions, and comments for how to improve the assessment tool. Further, the pilot survey offers an opportunity to test the data management plan with respect to data entry and the coding scheme chosen3. By improving accuracy of the assessment, the pilot tests will enhance DWSP’s ability to “diagnose and investigate environmental health problems in the community”.
v. Implement Stakeholder Survey First
As previously mentioned, one of the greatest challenges in assessing private well owners is their remote location and the general lack of knowledge about their demographics and how to reach them. Further, because of the potential for existing data to improve the scope and response rate of a well owner survey, conducting a stakeholder survey first is advisable. We need stakeholder input and collaboration in order to cast a wider net and capture as many private well owners as possible.
Moreover, responses from the stakeholder survey may provide insight as to whom we should survey for the well owner survey. In the event that a mail and online stakeholder questionnaire results in a low response-rate, the DWSP staff person assigned to the project should take extra efforts to make personal contact with stakeholders via e-mail, one-on-one meetings, site visits, and phone calls, as the survey is open to submission. The staff person should probe the stakeholders for any information about data they may have on where private well owners are located and best ways to reach them.
vi. Sample Frame for Well Owner Survey
Because the 2011 survey only examined a small sample of 30 respondents, and no other survey on record could be found for private well owners or stakeholders in Oregon, a larger sample size should be attempted. Moreover, given a total sample size of 350,000private wells in Oregon, we could sample 2,385 of those wells and their owners and be 95% confident with a margin of error of 2. This would be a long-term, or year-end goal, but a response-rate of about 1000 well owners in Oregon will still provide us with a strong sample with a margin of error of only 3. These sample sizes are comparable to the ones examined in other previous private well surveys, . DWSP will need to discuss as a team how to further focus or refine the scope of their own private well owner sample. Additionally, the sample should be random in accordance with best practice. Furthermore, the sample frame for well owners should take into consideration those most affected by potential health risks from private well water. Specifically, the list of potential respondents should include residents from counties with the highest number of private wells with elevated nitrate concentrations, as determined by Hoppe et al.
vii. Sample Frame for Stakeholder Survey
To develop a sample frame for the Stakeholder Survey, the DWSP staff member should first consult existing contact lists in DWSP’s records and lists of existing partners. Stakeholders should include major well drillers, constructors, real estate agents, testing labs, and agricultural businesses or farms across the state, given these groups’ dominant roles in the education and support of private well owners with regard to their wells. Other stakeholders to include in the sample frame may include local and county water quality departments, water non-profits, and universities or schools with testing programs. Once a master list or sampling frame is developed or during the process of its development, a stakeholder analysis should be performed to identify the “right mix” of stakeholders to include in the survey, which would be those who have a high amount of influence of private well owners as well as those who have a high amount of interest in assisting with education and outreach efforts to improve private well stewardship. The Oregon Water Resources Department lists approximately 160 licensed well constructors in its database. Because of the number of real estate agents that exist in the state, a more reasonable sample frame would be to sample from real estate firms or groups in the state, with special attention to areas with high private well density.
viii. Use Knowledge Brokers during Data Collection
On the stakeholder survey, we will ask stakeholders if they would be willing to help us disseminate a similar survey to well owners whom they interact with and serve. If they answer yes to this question, we will follow up with them and mail them copies of the survey to deliver to respondents whom they personally know are well owners. By asking this question DWSP is recognizing that stakeholders can be of use in helping DWSP to reach private well owners and in sharing knowledge and information about possible well contamination and resources. Ensuring that stakeholders have the necessary factsheets and website information is necessary to protect well owner health
ix. Develop OHA Advisory Committee for Private Well Stewardship in Oregon
Advisory groups offer organizations an opportunity to obtain low-cost specialized experience and hear outside perspectives. Moreover, the topic of maintaining and ensuring the health of private wells is multidisciplinary and requires expertise from a variety of fields, including business, science, and medicine. By establishing an advisory committee, DWSP can enhance its program capacity which it is currently lacking due to funding and staffing. Moreover, through involvement of various stakeholders and private citizens on an advisory committee, DWSP can increase the likelihood that its future programs will be well-received and also enhance awareness of its programs. Furthermore, an advisory committee would mobilize community partnerships and allow DWSP to collaborate with local health departments and community organizations. An advisory committee may also be able to undertake a system analyses of existing sources of well data and would further extend DWSP’s network of contacts and ability to reach stakeholders and well owners. Additionally, the advisory committee could contain members of the legislature or local officials who may have ability to lobby for stronger private well regulation and enforcement. Finally, advisory committee tasks could involve research into new innovative techniques for ensuring private well stewardship, such as development of an app, or implementation of periodic inspection service.
x. Implement other Best Practices as Feasible and Necessary
Finally the author recommends implementing focus groups, community hearings, document review and secondary data analysis, as DWSP is able. The recommendations listed above acknowledge DWSP’s existing capacity constraints but encourages, as future opportunities arise, that DWSP staff consider other best practices to collect input from stakeholders and well owners.
Flow Diagram of Searches
Framework for Assessment Tools
Assessment Tool for Well Owners
Assessment Tool for Stakeholders
Budget for Proposed Assessment Plan
Visual Summary Table of Well Owner & Stakeholder Assessment Best Practices Coded by CDC Ten Essential Environmental Health Services
11 testing labs, and agricultural businesses or farms across the state, given these groups’ dominant roles in the education and support of private well owners with regard to their wells. Other stakeholders to include in the sample frame may include local and county water quality departments, water non-profits, and universities or schools with testing programs. Once a master list or sampling frame is developed or during the process of its development, a stakeholder analysis should be performed to identify the “right mix” of stakeholders to include in the survey, w