Outreach Evaluation Resource Center
Annual Report
Year One May 2001-April 2002

Training and Continuing Education

The OERC staff conducted nine evaluation training sessions during Year 1 of the contract:

• Full-day workshop on outreach planning and evaluation, Health Libraries of British Columbia in Vancouver, B.C., a member group of the Pacific Northwest Chapter of the Medical Library Association, conducted by Catherine Burroughs and Maryanne Blake. (June 2001)
• Presentation about outreach evaluation to NN/LM GMR's executive committee members at their Regional Advisory Committee meeting, conducted by C. Burroughs. (July 2001)
• RML staff evaluation workshop at NLM for new and current staff in Regions 1, 2, and 8, and several NLM program staff, conducted by C. Burroughs. (October 2001)
• RML staff evaluation workshop for new and current staff in Regions 3 and 5, and a guest network librarian who has RML outreach funding, in Chicago, conducted by C. Burroughs and M. Blake. (October 2001)
• 1.5 hour presentation about program evaluation at the meeting of NN/LM-SCR Regional Advisory Committee, conducted by C. Burroughs. Her attendance at the 2-day meeting was very helpful in understanding the unique and overlapping concerns of various regions in the NN/LM. (December 2001)
• Full-day workshop for library staff at University of Texas Health Sciences Center and several libraries from other UT campuses, held in San Antonio by C. Burroughs. (February 2002)
• Full-day workshop for staff of the NN/LM MCR conducted by C. Burroughs and M. Blake. (February 2002)
• Full-day workshop for librarians and faculty at E. Carolina University, hosted by William E. Laupus Health Sciences Library, East Carolina University, conducted by C. Burroughs and M. Blake. (March 2002)
• Full day evaluation workshop for the NN/LM PSR and several members of their network and their Regional Advisory Council, conducted by C. Burroughs .(April 2002)

The following table shows that the workshops received very high ratings on the MLA Continuing Education evaluation forms. The workshop'; contribution to participants' knowledge and skills, the clarity and relevance of the objectives, and the knowledge, effectiveness, preparation, and responsiveness to questions of the instructors received 100% positive ratings in the eight workshops that were evaluated. The content and instructional materials received 90% to 100% positive ratings. (Seldom did any item on the MLA evaluation forms received more than 1 negative rating per workshop.

Participant Feedback for OERC Evaluation Workshops
	Percentage of Positive Ratings (8 Workshops Total)
	Number of Workshop with ...
	100% Positive Ratings	90-99% Positive Ratings	<90% Positive Ratings
[Participant] Acquired useful knowledge and skills*	8
Objectives [clearly stated; achieved]*	8
Content [relevant; well organized]*	5	3
Instructional materials [relevant; supported content; included effective AV/Demos and hands-on exercises]*	6	2
Instructor(s) [knowledgeable, well-prepared, effective, responsive to questions]*	8
Facility [conducive to learning]*	5	1	2
Global rating**	5	3
* Positive rating="Agree" or "Somewhat Agree"** ** Positive rating="Excellent" or "Above Average"

To increase her expertise in various areas of program evaluation, the assistant director of the OERC completed several professional development workshops:

• C. Burroughs attended the American Evaluation Association national conference in St. Louis, MO and completed two full-day continuing education courses on qualitative research and participatory research methods. (November 2001)
• C. Burroughs attended a 2-day continuing education course on practical program evaluation at the Evaluator's Institute in San Francisco, taught by evaluation expert Dr. Michael Scriven. (January 2002)

Web site development

• A description of the center, the evaluation workshop and additional resources for evaluation planning were added to the OERC website.
• C. Burroughs and consultant Dr. Zoe Stavri began work on a stand alone web-based toolkit for the evaluation guide, with an emphasis on developing research objectives and methodology for decision-based evaluation.

Consultation with Network members

OERC director Catherine Burroughs provided consultation on the following projects:

• PNR Outreach Awards. C. Burroughs assisted with revision of guidelines for evaluation planning in the RFP for the awards, especially in sections on the Identification of Need; Goals, Outcomes and Objectives; and Evaluation Planning. The revised RFP encourages applicants to demonstrate thoughtful planning for a feasible project, and acknowledges that the PNR will expect more detailed objectives and evaluation plans after completing a thorough community assessment (post funding). The RFP now links applicants to the OERC website and to articles about aspects of evaluation published in the RML newsletter. [Dragonfly can be viewed at http://nnlm.gov/pnr/funding/outproj2001.html.]
• University of Texas Health Science Center at San Antonio (UTHSCSA) Library's Texas Lower Rio Grande Valley Health Information Hispanic Outreach (HI HO) project. C. Burroughs provided consultation at various stages of this project, including the following:
  • Interviewing their prospective evaluation consultant to understand her vision of the project evaluation and securing the consultant's agreement to use Measuring the Difference: Guide to Health Information Outreach Planning and Evaluation in the project and provide feedback to OERC and NLM regarding its usefulness.
  • Reviewing audience analysis and goals/objectives for the outreach project at Med High, one of 4 sites selected for HI HO project
  • Visiting potential outreach sites for the project
  • Meeting with several project staff for this project, reviewing the outreach plans, and meeting several contacts to be interviewed during the community assessment phase of the project.
• Needs assessment of public health nurses (a project proposed by Kathy Murray, University of Alaska). C. Burroughs contributed to this project by
  • Providing consultation to Ms. Murray about the project
  • Recommending an evaluation consultant (Susan Usitalo) who had applicable evaluation experience through her work on a Tribal Connections outcomes study
  • Provided S. Usitalo with information about the project and design plans for the assessment
  • Reviewed the interview guide developed by S.Usitalo for a follow-up evaluation of a previous PNR resource library project assessing Alaska public health nurses' internet use.
• Program-wide RML evaluation plan. C. Burroughs participated in
  • A preliminary meeting with Angela Ruffin about planning the program-wide RML evaluation
  • Development of a concept memo to outline issues and considerations that was presented first in a meeting with A. Ruffin, Sherrilynne Fuller, and Neil Rambo at the PNC meeting in Salishan, Oregon, then revised for a subsequent planning meeting at NLM with A. Ruffin, Betsy Humphries, and Becky Lyon
  • Submission of a memo to A. Ruffin clarifying NNO's suggested goals for an RML program evaluation, with a suggestion that these evaluation topics be discussed at the next RML Director's meeting
  • A presentation at the RML Director's meeting March 21 in Washington D.C. C. Burroughs also facilitated the discussion following her presentation about cross-regional evaluation of the NN/LM program. Based on this meeting, 2 areas of focus were identified for evaluation: outreach to public libraries and outreach to public health departments
  • Teleconference with A. Ruffin and B. Humphries to plan a process for including the NN/LM staff in further refining the evaluation focus. C. Burroughs suggested asking each RML to submit up to 5 outcomes that might be evaluated to assess progress in outreach to public libraries and public health departments.
• Tribal Connections Project, Phase 3 (TCP 3) C. Burroughs assisted in
  • Helping the NN/LM PNR develop a logic model and theory of action statement for Tribal Connections Project, Phase 3 (TCP 3). To facilitate a focused and collaborative approach to TC3 planning, Ms. Burroughs arranged to have an experienced evaluator conduct a mapping session with the TC3 staff.
  • Participated in the hiring process to identify and recruit an evaluation consultant for Tribal Connections Project, Phase 3 (TCP 3).
• C. Burroughs met with Anne Turner, NLM Informatics Fellow, to share the qualitative analysis done on evaluation reports submitted by public health outreach projects for the Public Health Outreach Forum in March 2001 and talk about the fellow's upcoming study of public health information needs

Assistance with analysis, synthesis, and reporting of evaluation results

• C. Burroughs oversaw completion of an outcomes evaluation of the Tribal Connections Project. She assisted with analysis of the findings, and wrote several sections of the final report, including the executive summary, background and implications.
• C. Burroughs assisted with a pilot study conducted by Z. Stavri, in collaboration with NLM, to investigate personal health information seeking (PHIS) behavior, including use of Internet resources such as MedlinePlus. C. Burroughs helped develop the research methodology and instruments. She assisted with intensive interviews of 10 participants in Honolulu, and wrote up the results.
• C. Burroughs assisted in developing research methodology to review the PNR website and information needs of targeted users. C. Burroughs assisted with conducting interviews, analyzing results, and using the information to develop a prototype revision of the website. She and the PNR staff met to conceptualize an article about the results of the user-informed PNR website study.

Communications and collaboration

• C. Burroughs participated with R. Sahali on a panel at the mid-year meeting of the Society of Public Health Educators, held in Seattle (May 2001). They presented about the Tribal Connections project and evaluation of health information outreach.
• C. Burroughs presented an overview of the Outreach Evaluation Resource Center to the RML Director's meeting in Orlando (May 2001).
• Marlo Young, an NLM Associate Fellow, asked C. Burroughs to review a draft of a 37-page report "Hispanic Health Information Outreach: Culturally Sensitive Tactics." The report addressed the importance of community-based efforts for effective outreach outcomes and included several references to theories and principles in Measuring the Difference: Guide to Health Information Outreach Planning and Evaluation.
• Elaine Graham, associate director of PSR, asked C. Burroughs to review the final project report submitted by Z. Stavri for the MedlinePlus evaluation funded through the PSR.
• C. Burroughs contributed sections about community assessment and evaluation for a draft article describing the process and results of Tribal Connections project thus far. PNR and NLM staff met to discuss the article and ideas about Tribal Connections Phase 3 that will be a more in-depth effort at applying community-based approaches to health skills training in tribal communities. C. Burroughs investigated key contacts in the fields of health education and health communications with demonstrable expertise in evaluation of community-based health activities who might assist or collaborate on the TC3 project.
• As co-author, C. Burroughs helped draft an article written by NLM featuring an evaluative analysis of Tribal Connections, Phase1, to be submitted to the Journal of the Medical Library Association.
• C. Burroughs assisted project staff with evaluation planning for an NLM-funded consumer health project at Harborview Medical Center in Seattle. This project targets seven immigrant and refugee communities in the Seattle area: Oromo, Somali, Ethiopian, Tigrean, Cambodian, Vietnamese, and Latino. Her participation included attendance at a mid-project meeting with key contacts from each community and provided input about evaluation planning for the project.
• C. Burroughs, R.Sahali, and N. Press co-facilitated the organization and delivery of a virtual Web-based teleconference about community-based outreach for all NN/LM staff.

Utilization and promotion of the evaluation guide

C. Burroughs and PNR staff promoted the evaluation guide to various groups encountered in projects, presentations and classes, including

• Evaluation workshops
• The PNR Consumer Health Advisory Members
• The GMR Regional Advisory Council, the CE Course Teaching the Adult Learner in the Library Setting
• The Society for Public Health Educators annual scientific meeting
• Exhibit at the National Indian Education Association in Billings, MT
• The Washington State Joint Conference on Health in Yakima, WA.

There were more than 210 domestic and international individual requests for the guide via the OERC website from

• Health and medical libraries.
• State, county, and public libraries
• Public health departments
• Schools of dentistry and pharmacy
• Patient and health educators
• Government agencies
• Consultants
• A preventive medicine residency program
• An aging and long-term care center
• A cultural competency resource center
• A faculty member of a health care research class
• Christine Owens for use in planning and evaluating the NLM-funded project "Accessing Online Information for Immigrant and Refugee Health"

Administrative

• C. Burroughs developed and submitted a budget and justification for Year 2 of the OERC contract.
• With N. Rambo, C. Burroughs reviewed the proposed subcontract proposal from the Indiana University for the National Outreach Mapping Center and participated in a teleconference with staff from the Indiana University for the National Outreach Mapping Center.

Publications from OERC staff

• Catherine Burroughs, "Library program evaluation," MLA News. Jan 2001; 332: 10.
• Zenan JS, Rambo N, Burroughs CM, Alpi KM, Cahn MA, Rankin J. Public Health Outreach Forum: report. Bulletin of the Medical Library Association. Oct. 2001; 89(4); 400-3.
• Rambo N, Zenan JS, Alpi KM, Burroughs CM, Cahn MA, Rankin J. Public Health Outreach Forum: lessons learned. Bulletin of the Medical Library Association. Oct. 2001; 89(4); 403-6.