Evaluation is the process of determining value and identifying what has been learned. While data collection is part of evaluation, it is only a piece of the process. Evaluation is a required part of most grants.

Evaluation can be used in several ways:

• It provides accountability to the funder
  • Proof the project implemented as promised
  • Evidence of how SPPS upheld its obligations to the grant contract
• It can be used to inform leaders of lessons learned
  • SPPS round table
  • Publication online or in SPPS communications
  • Shared with the board and cabinet
• The results can be disseminated widely, contributing to literature on a topic
  • Articles, peer-reviewed academic journals and other publications
  • Presentation at conferences and other academic venues

There are many types of evaluations, each having its own specific focus or goal. If the grant doesn’t designate which type or types of evaluation will be used, the program evaluator will make a determination. The most commonly used evaluations focus on project implementation and/or outcomes.

Implementation Evaluation

• Primary focus: How successfully was the program implemented?
• Secondary focus: What happened in the program?
• Outcomes can’t be measured if the program isn’t fully implemented

Outcomes Evaluation

• Measures the effect of the implemented program
• Questions will vary depending on the project

A lot can be learned even when all planned outcomes aren’t achieved. Evaluation can help articulate what was learned, what barriers prevented full attainment, what methods or techniques showed more positive results, etc.

SPPS uses both internal and external evaluators. Internal evaluators are SPPS employees from Research, Evaluation and Assessment (REA). External evaluators are non-SPPS employees contracted to fulfill this need. REA will determine what evaluators will be used based on type of project, evaluation burden, available REA staff time, and project content needs.

Internal Evaluator Advantages

• Familiarity with SPPS history, resources, personnel, and common practices

External Evaluator Advantages

• Specialized skills
• Strength in a specific subject area
• Content knowledge that would benefit the project (eg: familiarity with arts projects and programs, expertise with STEM grants and activities, etc.)

Evaluation is a continuous part of the project, which allows the evaluator to help ensure a grant gets implemented completely. If it isn’t fully implemented, they can ensure documentation is clear on why.

Dyad or Team Approach

The evaluator should be part of the leadership dyad or team.

• Develop a relationship with the evaluator
• Include the evaluator in the kick off meeting if possible
• Integrate the evaluator in ongoing activities of the program
  • Considered a valued member of the leadership team, not an outsider or monitor
  • Involved in developing meeting schedule to ensure attendance
  • Present at meetings to provide regular updates
  • Include meetings in evaluator work plan
• Thoroughly read the grant to determine what needs to be learned from the evaluation
  • The grant itself informs the evaluation
  • Look at the budget, determine what needs to be spent and when
  • Look for the timeline; work with evaluator if there isn’t one
  • Timelines are guidelines and they may change, so revisit them and revise as needed
• If the project is longer than one year, revisit the approved grant at least once a year
  • Verify the program is on track

Trust the Evaluator

Whether the evaluator is internal or external, they will be a skilled professional. Most evaluators are members of the American Evaluation Association (AEA).

• Evaluators will keep survey and evaluation questions focused on the grant
  • Ensuring the project stays focused on the approved grant project
• All evaluations, surveys and interviews are voluntary
• Evaluators follow the AEA’s guiding principles
  • Surveys need a large enough size to ensure anonymity
  • Will redirect requests that violate standards of ethics or conduct

Summary of the AEA’s Guiding Principles

• Evaluators conduct systematic, data-based inquiries
  • Adhere to high technological standards
  • Explore the strengths and weaknesses of questions and evaluation approaches
  • Explain the approaches in sufficient detail that others can understand

• Evaluators are competent
  • Ensure evaluation team has the necessary education, skills and experience
  • Ensure the team demonstrates cultural competence and uses appropriate strategies
  • Practice within the scope of their competence, making any limitations clear
  • Seek to improve their competence to provide highest level performance

• Evaluators display honesty and integrity in their behavior and the evaluation process
  • Negotiate honestly regarding tasks, costs, and limitations
  • Disclose any potential conflicts of interest prior to accepting assignments
  • Document changes to original plan including the reasons
  • Clear about all interests and values related to the evaluation
  • Accurately represent their procedures, data and findings and correct misuse
  • Resolve concerns regarding activities likely to yield misleading information
  • Disclose all sources of funding and the source of the request for evaluation

• Evaluators respect the security, dignity and self-worth of those they interact with
  • Seek full understanding of context
  • Abide by current professional standards of ethics and regulations
  • Seek to maximize benefits while reducing any harm
  • Conduct the evaluation and share results in a respectful fashion
  • Foster social equality in evaluation so participants can benefit

• Evaluators consider the diversity of general and public interests and values
  • Include the full range of relevant perspectives
  • Consider long term impacts in addition to immediate responses and outcomes
  • Present evaluation information in an understandable way
  • Balance client and stakeholder needs and interests
  • Consider public benefit in addition to stakeholder interest

For a program to be meaningful and sustainable, there needs to be clear documentation.

• Document what was done and how
• Record what worked and what didn’t work
• Identify barriers and gaps to successful ability to sustain the work

 
The evaluator can help the program identify what and how to document. They may not do the documentation themselves, but they can help guide the process to ensures the continuity necessary to:

• Truly know what happened even if the original actors are gone
• Be able to sustain the program
• Be able to successfully implement it elsewhere

Modify

Sometimes the project needs to change due to discoveries made over the course of the project, especially with multi-year grants. Go to the grants management coordinator (GMC) to discuss what has changed from the original plan. Determine if the change is okay or if the grant needs to retain the original plan.

Changing the project may involve discussion with the funder. If a grant modification is needed, work with GMC to complete and submit that paperwork.

Failure to Implement

When a program or project fails to fully implement or complete a project as put forward in the grant document, SPPS may be required to:

• Return some or all of the funds
• Take corrective action guided by funder
• Amend the grant work plan, with funder approval
• Some other option or combination of options

The consequences of a grant being terminated early vary, depending on the circumstances and the funder. If the termination is due to congress choosing not to appropriate funding, SPPS is informed months in advance and grant managers receive instructions on how to shut the project down. If the termination is for cause, there will be written guidance on what to do.

If a grant is shut down for cause, SPPS can be categorized as a high-risk grantee, which will limit the chances of getting future grants.

It is REA’s role to bring alignment, efficiency and accuracy to program research, evaluation, and assessment. When grant projects are strategically aligned, the evaluation of those projects informs district priorities. The data collected and reports written about a grant are ultimately for the district to use, not just to inform the program or the funder.

REA reviews and approves all evaluation reports to ensure grantor requirements and requirements for use of data are met. REA will review reports for use of best practices in data collection, analysis and evaluation. REA will assist in identifying potential issues, successes and best practices for full district implementation.

In partnership with REA, the grant manager will examine the goals of the evaluation and negotiate a working style regarding how the project will work with REA (closely, loosely, regular schedule, ad hoc schedule, etc.) Inquiries are sent out by REA to selected evaluators that would be a good fit for the project based on project subject area, existing projects and working style. 

The grant manager assembles the complete contract and monitors that expectations are being met, both on the part of the district and that of the evaluator.

A data sharing agreement is necessary for all projects involving student data. REA can help with this.

Surveys

It is strongly recommended that surveys be approved by REA or created by the external evaluator. Without expertise in the evaluation field, surveys can yield no information or a lot of information that isn’t useful to the project. REA has experts who will help construct surveys with the right questions to get the information that is needed.

• Health and Human Services Administration for Children and Families guide, The Program Manager’s Guide to Evaluation.
• Better Evaluation – Utilization-Focused Evaluation