Teaching and Learning

DigitalWorks Evaluation and Results

Formative and Summative findings for students and teachers are summarized below:

• Teachers were initially motivated to participate in DigitalWorks because they wanted to enhance their teaching, they liked working with their teaching team at school, or they wanted to improve their use of technology. 
• Teachers value the DigitalWorks support team to help them create arts integrated lessons that their engage students, meet their learning needs, and incorporate useful assessments to help them understand what students are learning.
• Teachers in their second or third year of the project are feeling more confident about independently creating arts integrated lessons as compared to teachers in their first year of the project.  Also, longer-term teachers have more confidence when it comes to incorporating iPads with their arts integrated lessons compared to newer project teachers. 
• Teachers are slightly less confident about their capacity to design useful assessments to accompany their arts-integrated lessons.
• Most teachers feel their principals are supportive and knowledgeable of the project. 
• Teachers’ units and lessons are quality examples of arts integrated learning.
• Students are achieving the proficiency goals set for each year of the project.
• Students are meeting desired levels of proficiency in math but not in reading.

Results from the quasi-experimental analysis using hierarchical linear modeling revealed significant interactions between the curriculum, cohort, and grade level and the results were consistent across mathematics and reading growth scores. With cohort one, DigitalWorks students were much less likely to meet yearly growth targets, however in cohort two, these new DigitalWorks students in grade four outperformed their control group peers in average mathematics and reading growth scores. This indicates these students were more likely to meet growth targets.

Evaluation Approach

The DigitalWorks evaluation plan employs qualitative and quantitative methods to collect formative and summative data in order to measure the achievement of several project objectives:

• Teachers increasing their understanding of appropriate content standards and arts integration;
• Teachers improving their ability to design and deliver arts integrated lessons,
• Teachers improving their ability to design arts integrated assessments to measure student learning,
• Teachers improving their ability to incorporate digital technology into arts integrated lessons,
• Teachers enhancing their capacity to design arts-integrated units and lessons,
• Students demonstrating proficiency on their classroom arts integrated assessments, and
• Students demonstrating improvements in reading and math on the MCA-III.

In addition to the evaluation plan described above, a quasi-experimental design was implemented in the second and third years of the grant period.  Briefly, the relationship between arts integrated instruction and student achievement in reading and mathematics has been explored using a growth model with two cohorts and two replications.  One cohort consisted of fourth grade students and the other is comprised of sixth grade students with both cohorts including classrooms from two schools.  A comparison group consisting of students in the same grade but from other schools within the Saint Paul Public School district not receiving the treatment was followed.  Statistical controls (e.g. student demographic data and prior test scores) were used to help limit bias due to non-equivalent groups.  Lastly, a replication was completed in each year for each cohort (e.g. fourth grade and sixth grade students receiving the treatment and were followed) allowing the ability to test if the implementation improved over time and this also improved the sample size due to attrition.

The sample of students receiving the treatment consisted of intact classrooms from four schools within the Saint Paul Public School district.  The DigitalWorks arts integration curriculum was introduced into two of the schools in fourth grade and two of the schools in sixth grade.  Student achievement for both reading and mathematics was measured with the Minnesota Comprehensive Assessment (MCA).  These assessments are given in the spring of the school year.  The benefit of using the MCA is that the pre-measurement was free of effect from the DigitalWorks arts integration curriculum.

The growth model used consisted of two measurements in both math and reading; each subject was assessed pre and post treatment.  The pre-measurement was used to set expected growth targets.  Growth was measured in a z-score calculated by the Minnesota Department of Education (MDE).  Average z-scores were compared between the treatment and control groups to determine if there was more growth due to the treatment.  Lastly, differences among the treatment and control group were explored statistically.