Evaluation Report of Big Brainz' Timez Attack

Prepared for

Big Brainz, Educational Gaming Company

Prepared By

The IP&T 661: Intro to Evaluation in Education Class, Spring 2010
Taught by Dr. Richard E. West
Instructional Psychology and Technology Department
David O. McKay School of Education
Brigham Young University, Provo

June 15, 2010

TABLE OF CONTENTS

EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Evaluation Purpos

Limitations of the Evaluation and Disclaimers

Description of Timez Attack

LITERATURE REVIEW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . .

Multiplication Mastery and Math Anxiety

Computer Assisted Instruction (CAI) Benefits

Recommendations from the National Mathematics Advisory Panel

Conclusion

METHODOLOGY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Research Design

Participants

Research Methods

Quantitative Research Measures

Qualitative Research Methods

Data Collection Procedures

Data Analysis

Quantitative Data Analysis

Qualitative Data Analysis

PRESENTATION OF RESULTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Summary of Finding

Timez Attack and Student Mastery of Multiplication Facts

Timez Attack and Student Self-Efficacy

DISCUSSION OF FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Achievement Comparison

Mixed Methods Benefits

Age Differences

Anxiety and Self-efficacy

Peer Interaction

Learning vs. Gameplay

CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Conclusions

Recommendations

Evaluation Questions

Evaluation Methods

Future Planning for TA

APPENDICES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Appendix A – Data Collection Instruments

Multiplication Attitudes Survey Pre-test

Multiplication Attitude Survey Post-test

Multiplication with 12s Pre/Post-test

Appendix B – Student Comments

Student Comments Recorded by Observers During TA Play

Student Responses to Post-Survey Question 11

Student Responses to Post-Survey Question 12

REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

METAEVALUATION: Evaluation Criteria, Standards, Objectives and Judgments . . . . . . . . . .

Utility

Feasibility

Accuracy

Propriety

Conflicts of Interest

Executive Summary

Big Brainz, an educational gaming company, developed Timez Attack (TA) to help students learn basic multiplication facts through stimulating, game-based learning. With keyboard and mouse controls, students navigate various three-dimensional environments and encounter "assessment stations" to demonstrate their mastery of multiplication facts.

Big Brainz requested an evaluation of TA to identify changes in student mastery of multiplication facts and student self-efficacy regarding mathematics. In preparation for the evaluation our evaluation team--most of which are graduate IP&T students--conducted a literature review researching multiplication mastery and math anxiety; computer assisted instruction (CAI) benefits including motivation, achievement, and self-efficacy; and recommendations from the National Mathematics Advisory Panel.

Our evaluation team, in cooperation with administrators and faculty at Wasatch Elementary School, completed a controlled exercise using a variety of methods in order to collect and code the data. We administered a 12s times table test to two groups of students and collected data from pre/post-test surveys. We also collected data through observations and informal interviews throughout the experiment and from the participants' written responses to open-ended questions on the surveys.

Our data suggests that TA compares with other instructional methods for teaching times-tables. The outcome between a session of playing Timez Attack and a session of classroom instruction showed no significant differences. Our preliminary findings also suggest that students who experienced both teacher-instruction and TA gameplay attempted more multiplication problems on their post multiplication tests and made fewer errors than students without the additional instruction.

Our data shows a positive change in students' mathematics self-efficacy, or their belief in their own capacity to solve multiplication problems. Responses indicate that exposure to TA resulted in generally decreased math anxiety and an increased confidence toward learning multiplication facts.

It is recommended that the findings in this report be utilized for marketing purposes to help consumers understand the positive ways the product contributes to students' experiences and complements in-class instruction. While it was observed that Timez Attack can be engaging and beneficial for the students, the benefits the game provides may not be readily understood by teachers, administrators, and parents.

The narrow time frame of this project resulted in limiting the scope of this evaluation. As a final recommendation, this project should not be viewed as conclusive but rather serve as an initial step toward a more comprehensive and thorough evaluation with more controlled conditions and larger subject pool.

Introduction

“I did it!” Joe exclaimed with excitement. The rest of his class had begun filling out a survey about Timez Attack – the game they’d recently finished playing – just as the evaluation team had asked them to. “I just have to beat the sixes and I'm done,” pleaded Joe. “Can I play the game?”

The evaluator whom Joe had asked felt a mix of frustration and pleasure. They needed the questionnaires filled out so they could evaluate the game, but Joe was so excited about mastering his times tables!

Joe began trying to multitask, switching back and forth between the game and the questionnaire. “Yes! I got it done!” Joe raised his arms in victory.

The evaluator was satisfied because Joe still got the questionnaire finished before his class had to leave the lab. Joe and the evaluator both got what they wanted. And for Joe, at the very least, Timez Attack seems to be a great success.

Timez Attack (TA) is a computer game developed by Big Brainz Incorporated. The game was developed to help students master basic multiplication facts. The design allows students to practice multiplication facts while overcoming challenges presented by the game. Students engage in learning and mastering the facts as they progress through the game.

Big Brainz designed TA to reduce or eliminate math-related anxiety that can, according to Scarpello (2007), decrease students' "confidence in their ability to do math." To achieve this they implemented high quality game-effects to engage and motivate the students to master their times-tables in a less stressful self-evaluative setting as opposed to a stress-inducing peer setting often found in other instructional methods.

This evaluation of Timez Attack was conducted with forty-eight 2^nd and 3^rd grade students and their instructors from two separate classes at Wasatch Elementary School in Provo, Utah. The report that follows outlines the planning, execution, and the results of this evaluation project.

Evaluation Purpose

This evaluation assesses the effectiveness of TA in helping students achieve multiplication mastery and self-efficacy (one’s belief in one’s ability to succeed in specific situations). The principle evaluation questions were:

Does playing TA improve multiplication fact mastery?

Does playing TA contribute to positive changes in students’ multiplication self-efficacy?

Limitations of the Evaluation and Disclaimers

Several variables impacted the effectiveness of this evaluation to draw inferences about the questions. These variables included the following:

The time frame for the evaluation project was compressed. The evaluation was a class project conducted within a short academic period that limited the scope of the work that could be completed.

The studies were conducted at the conclusion of the school year when many of the students had already mastered the math facts in the game. In addition, an Eagle Scout project focused on math instruction helped provide math facts support to the students in one of the groups (Study 1).
The evaluation included a small population of students not randomly selected. This does not necessarily invalidate the evaluation observations, but it does limit inferences that may be drawn to larger populations.

Limitations in the physical environment where the observations were conducted may have affected the students’ attention, particularly in the administration of the post-tests.

Novice (graduate student) data collectors were used for the evaluation.

The students’ time on task was not monitored. The time on task in the control group in study 2 was not compared to the time on task of the experimental group. The time on task may account for some of the observed differences.

The experimental groups were not consistent. For example, one study (study one) did not have a control group.
The control group received cutting edge math instruction from their teacher that did not represent the average teacher-led math instruction that students typically receive.

Description of Timez Attack

Figure 1. Throwing creatures as multiplicands.

TA is a computer-based video game set in a three-dimensional graphical environment wherein a single player controls the movements of an alien-like character in the third-person through fantasy worlds including a dungeon, machine world, a lava-filled underworld, and a jungle. The objective of the game is to guide this character through various regions or levels using the multiplication skills the game teaches them. To reach higher levels they must overcome frequent encounters with opposing computer characters who challenge the player with multiplication problems, or tests.

Encounters typically begin with the player arriving at an impassable obstacle, such as a locked door. The game then presents a multiplication problem with a multiplicand and multiplier and provides the player with a number of creatures (e.g. snails, robot spiders) equal to the multiplier, each with the value of the multiplicand. The player must gather all of these creatures and throw them at the obstacle (Figure 1). This action visually adds up each creature as a copy of the multiplicand until the sum equals the correct answer. The player is then tasked with typing in the correct answer, which was just revealed by throwing the creatures.

Figure 2. An encounter with an opposing non-player character.

The obstacle, or door, opens if the player types the correct answer. An opposing computer character (e.g. a troll, a dragon) then emerges to challenge the player. The player must rapidly answer multiplication problems that appear on the opposing character’s body by typing the correct answer within a short period of time. Each time the player provides a correct answer the opposing character’s “health” bar depletes by a degrees (Figure 2). Often, when the player provides an incorrect answer their character is punished by an attack which depletes part of his or her health bar, and the opposing character’s health bar recovers some degree of health. The player defeats the opposing computer character after delivering a certain number of correct answers. The computer character disappears, allowing the player to pass to the next area or obtain a key by which the player may then advance past a locked door.

Each world is composed of twelve levels, which culminate in a final “boss” challenge wherein the player must answer all the variations of multiplication problems encountered previously in that particular world. Unlike regular computer character encounters, the final boss does not respond to incorrect answers. However, the player must have mastered their times-tables and achieve 100% accuracy to defeat the final boss.

Beyond encounters with opposing computer characters, the player must evade environmental hazards such as falls from cliffs or bridges, falling rocks, etc. Failure to do so may result in depletion of the player character’s “health”, or in resetting the character to an earlier position in that level.

The player’s character is controlled via the computer’s keyboard and/or mouse. While the game may be played with one hand, optimal in-game performance depends on using both hands: one on the keyboard, the other on the mouse.

TA software is available for both Windows and Mac, and is playable on Linux through a Windows emulator such as Wine. Additional details on hardware requirements and support are available at http://www.bigbrainz.com/Support.php.

Literature Review

Educational researchers have debated the effectiveness and worth of educational computer and video game investments, through many studies and experiments. For brevity's sake, this literature review will not address whether or not high-tech educational games compete well with other recreational computer and video games. Rather, it will provide research-based evidence that multiplication math anxiety is a problem worth addressing (Berk, 2009; Jackson and Leffingwell, 1999; Meece et al, 1990; NRC, 1989; Sarason, 1980; Scarpello, 2007; Swetman, 1994; Tankersley, 1993) and that computer assisted instruction (CAI) can lessen students' math anxiety by increasing their motivation to achieve—resulting in higher self-efficacy (Berk, 2009, pgs. 644-5; Kebritchi, 2008; Ke & Grabowski, 2007; Klawe, 1998; Moreno, 2002; National Mathematics Advisory Panel, 2008; NCTM online, 2010; Programme for International Student Assessment, 2005; Rosas, 2003; Sedighian & Sedighian, 1996; Shaffer et al., 2005). It also discusses mathematic automaticity improvement induced by CAI, and addresses the impact of real-world game aspects (Carter & Norwood, 1997; Federation of American Scientists, 2006; Jackson, 1999; Meece et al., 1990; NMAP, 2008; NRC, 1989, 2001; Scarpello, 2007; Shaffer et al., 2005).

Multiplication Mastery and Math Anxiety

According to the National Council of Teachers of Mathematics's (NCTM) standards, third to fifth grade students need to develop multiplication fluency. Memorizing times tables 1-12 constitutes a fundamental basic portion of children's expected mathematical learning and understanding (NCTM online, 2010). Metaphorically speaking, one cannot understand a conversation spoken in a language one does not know. Students need to gain proficiency with the times tables language to gain fluency in higher mathematics.

Comparison studies between U.S. children and children in nations with higher mathematics achievement, in addition to studies tracking cross-generational changes within the U.S., reveal that in the earliest stages of learning math, many U.S. children today are slower and less efficient with solving whole number multiplication problems (NMAP, 2008). Some never attain proficiency. The panel attributes the deficiencies to a lack of “quantity and quality” practice, curricula emphases, and parental educational involvement (pg. 26). As a result, students are less capable when facing more complex mathematical problems. This in turn increases the students’ math anxiety and lowers their self-efficacy because, according to Meece et al. (1990), past performance is a leading factor in anxiety.

For example, Carol Jackson and Jon Leffingwell (1999) carried out a study to determine when students first experience mathematics anxiety, and to determine what instructor behaviors create or exacerbate the anxiety. 93% of their study's participants—college seniors—reported experiencing math anxiety of some kind, the first occurrences beginning mostly in 3rd and 4th grade (16%). Other research also indicates 4th grade as a key time when more students begin to experience anxiety (Swetman, 1994; Tankersley, 1993).

Computer Assisted Instruction (CAI) Benefits

Educational computer and video games offer an approach to learning that can potentially reduce negative anxiety by improving achievement and self-efficacy in mathematics. They do this by motivating students to complete and win the game through mastering the necessary educational skills (Kebritchi, 2008; Ke & Grabowski, 2007; Klawe, 1998; Moreno, 2002; Rosas, 2003; Sedighian & Sedighian, 1996).

Kebritchi, Hirumi, and Bai's study (2008) demonstrated the effectiveness educational games can have at increasing motivation and achievement. These researchers tested the effectiveness of the pre-algebra and algebra games Evolver™ and Dimenxian™ in improving achievement scores and motivation for a sample of 193 students. They also tested the games to see if their impact varied according to the students’ prior knowledge, computer experience, and language background.

Kebritchi et al. randomly assigned the teachers to experimental and control groups; teachers in the experimental groups were trained and encouraged to use the video games in class. The researchers found a significant achievement difference between the experimental and control group on exam scores. Most interviewed students said they preferred the game to other school activities. In addition, student and teacher interviews revealed that both teachers and students agreed that the game reinforced the students’ mastery and achievement by motivating them to succeed and was overall a more effective, motivating, and differentiating approach to learning math.

Recommendations from the National Mathematics Advisory Panel

The National Mathematics Advisory Panel (2008) recommends that mathematics teachers consider high-quality, well-designed and implemented CAI programs such as Evolver™ and Dimenxian™ that incorporate drill and practice to developing students' mathematical automaticity. By automaticity they mean effortless, accurate, and quick computation performance that frees the working memory so the students can focus their attention on more complicated tasks. However, they also recommend that students develop computational fluency simultaneously with conceptual understanding and problem-solving skills because they are mutually supportive and facilitative of learning each.

Many CAI games address these issues by implementing a real-world aspect making it possible for the players to “inhabit roles that are otherwise inaccessible to them” (Shaffer et al., 2005, pg. 105). They teach students how to think, act, and learn like the professionals in order to survive, or be competitive in the real world. The National Mathematics Advisory Panel advocates mathematics instruction involving real-world problems because it improves the children's assessment performance involving similar problems (NMAP, 2008). However, it does not seem to affect computation of simple word problems or equation solving if they are not aspects of the real-world problems taught.

It is not evident in the Panel's report of whether or not the Panel is aware of the real-world benefits of educational games. Even without this knowledge, they encourage the use and research of CAI, while cautioning teachers to “critically inspect individual software packages and the studies that evaluate them” (pg. 51). They also advise teachers to consider the practicalities of using the software—the necessary software, hardware and technical support, curriculum integration and professional development.

Conclusion

Students whose teachers take the Panel's recommendations can benefit from increased motivation and mathematical achievement (Berk, 2009, pg 644-5; Kebritchi, 2008; Ke & Grabowski, 2007; Klawe, 1998; Moreno, 2002; NCTM online; NMAP, 2008; Programme for International Student Assessment, 2005; Rosas, 2003; Sedighian & Sedighian, 1996; Shaffer et al., 2005). This will, in turn, increase students’ self-efficacy, thereby reducing their math anxiety (Berk, 2009; Jackson, 1999; Jackson & Leffingwell, 1999; Meece et al, 1990; NRC, 1989; Sarason, 1980; Scarpello, 2007; Swetman, 1994; Tankersley, 1993). They will also gain greater mathematical automaticity enabling them to learn more complex math with less effort (NMAP, 2008; Shaffer et al., 2005). However, how they will benefit from these positive effects depends, in part, on the real-world aspect of the games they use (Carter & Norwood, 1997; Federation of American Scientists, 2006; Jackson, 1999; Meece et al., 1990; NMAP, 2008; NRC, 1989, 2001; Scarpello, 2007; Shaffer et al., 2005).

Methodology

Research Design

Our evaluation team collected data using two studies that addressed TA's potential influence on mastery of the 12s time tables and self-efficacy toward learning multiplication facts. Multiplication with 12s was chosen for observation and analysis because the teachers had already taught times tables 1-11 to all of the participants. We employed a mixed methods approach in both studies in order to collect, code, and analyze the data. We collected the data using a pre/post 12s times test, pre/post Multiplications Attitude Survey (MAS) and field notes including observations and informal interviews.

Study 1 included 3rd-grade participants whose teacher had also previously taught the 12s multiplication facts. Due to limited computer access participants were randomized into two separate groups. However, since both groups had the same experience this study had no control group. Pre/post-test instruments were used to measure changes in ability and self-efficacy of participants after a 30-minute intervention with Timez Attack. As there was no control group we employed a single group pre/post-test research design. This design is illustrated in Figure 1 below where PRE=pre-test, POST=post-test, and X= Observed Intervention (30-minutes of playing Timez Attack).

Figure 1 -- Research Design for Study 1

All Participants

PRE

POST

Study 2 included 2nd and 3rd-grade participants whose teacher had not yet taught them the 12s multiplication facts. Students were randomly assigned to an experimental or control group by generating a randomized table. Study 2 used an experimental pre/post-test research design, as illustrated in Figure 2, where R=Randomization, E=Experimental Group, C=Control Group, PRE=pre-test, POST=post-test, Y=Traditional Teacher-Led Instruction, and X=Observed Intervention (30-minutes of playing Timez Attack).

Figure 2 -- Research Design for Study 2

Group 1 (R,E)

PRE

POST

Group 2 (R,C)

PRE

POST

POST₂

Participants

Participants in Study 1 included 26 3rd-grade students. Participants in Study 2 included 12 2nd-grade and 10 3rd-grade students all from the same elementary school. Prior to data collection, participants from both studies used Timez Attack during computer lab time. Study 1 participants' teacher had previously taught them the 12s times-tables while Study 2's teacher had not.

Research Methods

Quantitative Research Measures

We used the teachers' existing instrument to calculate and asses student proficiency of multiplication facts, consisting of a 100-item multiplication facts test limited to 12s. It included both 12 X (a number from 1-12) and (a number from 1-12) X 12 problems. It was distributed before and after the TA playing session, or teacher-led instruction in the Study 2 control group. The same instrument was used as both a pre- and post-test and can be found in Appendix A.

A pre/post-Multiplication Attitudes Survey (MAS) was developed to access (1) attitudes towards learning multiplication facts and (2) attitudes towards and perceptions of Timez Attack. The pre-MAS contained eight questions. The post-MAS contained twelve questions. Questions 1-6 on both the pre-MAS and post-MAS were designed to measure students’ self-efficacy, along with physiological and emotional states involved in multiplication math fact mastery. Questions 1, 2, 4 and 5 were identical while 3 and 6 were only slightly different on the pre- and post-tests. Questions 7 and 8 on the pre-test were designed to provide information regarding home learning of multiplication facts and previous exposure to Timez Attack. Questions 7-12 on the post-tests elicited information regarding perceived utility of and motivation to use Timez Attack.

Response options for the first seven questions on the pre-test and first ten questions on the post-test involved a five-point Likert-type scale. Five possible responses were provided -- “NO, no, ?, yes, YES” -- with the two extreme options being in bolder type to make it easier for students to interpret the rating choices. Instructions on how to interpret the scale were verbally explained to students in advance by saying, “Circle the big ‘YES’ if you think what the statement says is completely true and you very much want to say yes, circle the little ‘yes’ if you think it is basically true,” and so forth. Similar methods of administration were used by Nichols et al. (1990). Response options for question 8 on the pre-test included four options: “never, a few times, many times, almost every day.” (See Qualitative Research Methods regarding open-ended questions 11 and 12 on the post-MAS)

Likert-type questions measuring self-efficacy and psychological and emotional states regarding multiplication fact mastery were adapted from the Fennema-Sherman Survey (Fennema & Sherman, 1976). This is one of the most widely used math attitude surveys in math education. Changes to Fennema-Sherman survey questions included changing the word “math” to “multiplication math facts.” Other adaptations were done in order to make the questions age-appropriate (see Table 1). The complete pre- and post-MAS can be found in Appendix A.

Table 1 -- Likert-type Questions Pre/Post MAS

Index	Statements	Pre/Post

Self-Efficacy	I can learn new multiplication math facts easily. I am good at memorizing multiplication math facts. Remembering new multiplication facts is hard for me	Pre/Post Pre Post
Psychological/Emotional states	I like learning multiplication math facts. Learning new multiplication math facts scares me. I usually do not worry about remembering multiplication math facts. Learning new multiplication math facts makes me feel bored Learning new multiplication math facts makes me feel uneasy and confused.	Pre/Post Pre/Post Pre Post Pre
Home influence	I study my multiplication math facts at home.	Pre
Exposure to Timez Attack	I have played Timez Attack.	Post
Perception of TA Utility and Motivation to Use	I will play Timez Attack at home. I think playing Timez Attack would help me know my multiplication math facts. Learning to play Timez Attack was easy. Playing Timez Attack is a fun way to learn my multiplication math facts.	Post Post Post Post

Qualitative Research Methods

Qualitative data collection methods included participant responses to open ended questions 11 and 12 on the post-MAS (see Table 2), field note observations, and short informal interviews. Field notes were taken during all stages of the studies. There was no specific focus for the observations, and observers did not have a specific checklist of items to watch for. Rather, observers were instructed to note anything that seemed interesting about the students’ actions and statements while participating in our evaluation. So, though we shared knowledge of the evaluation questions, our observations may be particular to the individual observers.

During Timez Attack gameplay, however, observers did particularly note participants, enjoyment, frustrations, improved self-efficacy and learning (including conceptual understanding) of multiplication facts. Interview questions, also not specifically predetermined, were intended to illicit participants’ feelings and attitudes toward learning multiplication facts by playing Timez Attack, along with their feelings toward Timez Attack game play in general.

Some questions regarding learning with Timez Attack included, “Why do you throw the balls at the gate?” “Why do you pick up the spiders?” and “Do you think Timez Attack is useful?” Other questions meant to determine Timez Attack game play satisfaction included, “If you could change anything about Timez Attack what would you change?” and “What did you like most about playing Timez Attack?” All student responses to open-ended questions as well as field note observations and interviews can be found in Appendix B.

Table 2 -- Short Response Indices and Statements Post-MAS

Index	Statements	Pre/Post

Timez Attack Utility/Attitude	What I love about Timez Attack is…	Post
	What I hate about Timez Attack is…	Post

Data Collection Procedures

All participants were given a three-minute test on the 12s times table in their regular classroom, followed by the MAS pre-test. All Study 1 participants then played thirty minutes of Timez Attack on the 12s level. After playing the game, Study 1 students again took the 12s timed test followed by the MAS post-test. During all stages of data collection, researchers took field notes including observations of and short interviews with students. Table 3 illustrates data collection procedures for Study 1.

Table 3 -- Study 1 data collection procedures

Time

Study 1 (Prior 12s learning)

minutes

All Participants -

Times-test and Pre-MAS

Observations and Interviews

30 minutes

All Participants -

Timez Attack (12s level)

Observations and Interviews

15 minutes

All Participants -

Times-test and Post-MAS

Observations and Interviews

In Study 2, after administrating the pre-test to all study participants, Group 1 (the experimental group consisting of six 2nd graders and five 3rd graders) played the multiplication by 12s level of Timez Attack for thirty minutes. During this same time Group 2, the control group (consisting of six 2nd graders and five 3rd graders), were taught the 12s time table through teacher-led instruction.

Teacher instruction was carried out in the students’ regular classroom by their classroom teacher, and consisted of multiple representations of multiplication. The teacher helped students access prior knowledge about arrays, applied that knowledge to the new fact (12s), discussed the problem-solving strategy of breaking up 12s facts into the sum of 10s and 2s, and drilled using a kinetic game involving rhythm.

At the end of thirty minutes Group 1 and Group 2 were again given a three-minute 12s times test. The post-MAS was then administered only to Group 1. At this point in Study 2, Group 1 went back to their classroom for teacher-led instruction and Group 2 was also allowed to play the 12s level of Timez Attack. As with Group 1, Group 2 played Timez Attack at level 12 for thirty minutes and was given the 12s times test followed by the post-MAS. Similar to Study 1, researchers recorded observations of and short informal interviews with Study 2 participants throughout the various stages of this experiment. Table 4 illustrates data collection procedures for Study 2.

Table 4 -- Study 2 data collection procedures

Time

Study 2 (No prior 12s learning)

minutes

Group 1 and 2 participants -

Times-test and Pre-MAS

Observations and Interviews

30 minutes

Group 1 (experimental group)-

Timez Attack (Level 12)

Observations and interviews

Group 2 (control group)-

Teacher instruction of 12s

Observations and Interviews

15 minutes

Group 1 - Times-test and Post-MAS

Group 2 - Times-test

minutes

Group 1- Teacher instruction of 12s

Observations and interviews

Group 2- Timez Attack (Level 12)

Observations and interviews.

15 minutes

Group 2 - Times-test and Post-MAS

Data Analysis

Quantitative Data Analysis

In scoring the times tests, evaluators determined students’ overall correctness and the correctness of each individual 12s math fact (i.e. 12 x 1, 12 x 2 …12 x 12) by considering any written answer to a problem as an "attempt," every correct answer as a "correct response," and every incorrect attempt as an "error." These definitions allowed evaluators to measure changes in attempts, changes in the number of correct responses, changes in the number of errors, and ratio of errors to attempts, or the "error rate." Changes in correct responses, errors, and the error rate were calculated and also analyzed using Excel in order to measure the achievement of mastery by the students. Changes in attempts may reflect not only mastery, but also changes in student self-efficacy.

The coding of Likert-type questions (YES, yes, ?, no, NO) from the pre/post-MAS ranged from numbers 1 to 5 with 1 coinciding with “NO” and 5 coinciding with “YES.” Coding of “never, a few times, many times and almost every day” response question ranged from 1-4 with 1 being “never” and 4 being “almost every day. ”

All Likert-type responses and changes in identical Likert-type student pre/post-MAS responses were then calculated, summarized, compared and analyzed using Excel tables and graphs. Information resulting from pre/post-times test analysis and Likert-type pre/post-MAS can be found in written, table and graph form in the Presentation of Results section of this report.

Qualitative Data Analysis

At least two evaluators coded each open-ended question response, observation, and interview, and sought agreement on the interpretation. Codes were not pre-determined but emerged during data collection and analysis. A complete list of categories used in coding qualitative data, along with other tables and graphs used in qualitative data analysis can be found in the Presentation of Results section of this report.

Presentation of Results

Summary of Findings

The primary questions to be answered in this evaluation were

Does playing TA improve multiplication fact mastery?

Does playing TA contribute to positive changes in students’ multiplication self-efficacy?

Our evaluation showed an improved proficiency in multiplication of 12s amongst students who played Timez Attack. The tables and discussion that follow will explain the observed differences. On average, students answered 11-13 more questions correctly on the post-test after they played the game than they scored on the pre-test.

Our evaluation also showed that there was a positive change in students' self-efficacy toward multiplication. The tables and discussion that follow will explain the observed differences in the self-efficacy measures as well.

Timez Attack and Student Mastery of Multiplication Facts

As discussed in the methodology section, the evaluation took the form of two separate studies.

Study One

The first study was an experimental group consisting entirely of students in the third grade and did not have a corresponding control group. The only treatment applied to the experimental group between the pre and post tests was playing Timez Attack. This group had already studied the 12s and many had "passed off" these facts in normal classroom instruction. The results of the evaluation study on this group showed students attempted an average of 10.3 more problems on the post-test than on the pre-test. The students answered an average of 11.1 more problems correctly, made an average of .8 fewer errors, and reduced their error rate from 2.8% of attempts to .06% of attempts.

Table 5 -- Students in Study #1

Description	Pre-Test	Post-Test	Change	Avg Change
# of Students in Class	24	24
# Questions Attempted	968	1,215	+247	+10.3
Attempted (% of Possible¹)	40.3%	50.6%
Correct Answers	941	1,207	+266	+11.1
Correct % (of Attempts)	97.2%	99.3%
Errors	27	8	-19	-.8
Error Rate (of Attempts)	2.8%	.06%

¹ Total Attempts possible is 100 problem per test multiplied by the number of students

Study Two

The second study conducted for this evaluation differed from the first study in three critical aspects:

The students are in a class composed students in the 2nd and 3rd grades.
The students had not yet studied the 12s multiplication facts.
The study students were divided into an experimental group and a control group.

The experimental group was taught and practiced the 12s facts using Timez Attack. The control group was taught and practiced the 12s facts through instruction from the classroom teacher. Students received pre- and post-tests to evaluate the affect of the different modes of instruction and practice.

Experimental Group

The results from the experimental group (Table 6) showed that students attempted an average of 11.5 more problems while answering an average of 12.9 more problems correctly. The students also made an average of 1.5 fewer errors and reduced their error rate from 21.7% to 6.6% of attempts following their Timez Attack experience.

Table 6 -- Students in Study #2 - Experimental Group

Description	Pre-Test	Post-Test	Change	Avg Change
# of Students in Class	11	11
# Questions Attempted	161	287	+126	+11.5
Attempted (% of Possible¹)	14.6%	26.1%
Correct Answers	126	268	+142	+12.9
Correct % (of Attempts)	78.3%	93.4%
Errors	35	19	-16	-1.5
Error Rate (of Attempts)	21.7%	6.6%

¹ Total Attempts possible is 100 problem per test multiplied by the number of students

Control Group

The results from the control group (Table 7) showed that students attempted an average of 10.5 more problems while answering an average of 15.1 more problems correctly. The students made an average of 1.5 fewer errors and reduced their error rate from 35.2% to 4.4% of attempts following the classroom instruction received from their teacher.

Table 7 -- Students in Study #2 - Control Group

Description	Pre-Test	Post-Test	Change	Avg Change
# of Students in Class	11	11
# Questions Attempted	179	295	+116	+10.5
Attempted (% of Possible¹)	16.3%	26.8%
Correct Answers	116	282	+166	+15.1
Correct % (of Attempts)	64.8%	95.6%
Errors	63	13	-50	-4.5
Error Rate (of Attempts)	35.2%	4.4%

¹ Total Attempts possible is 100 problem per test multiplied by the number of students

Additional Question of Interest

Study group two consisted of one class that was a combined class of 2nd and 3rd grade students. The random assignment of students into the experimental and control groups resulted in groups that were approximately the same of 2nd and 3rd grade students in each group. When combined, there were 11 students in each grade who were part of the evaluation study.

The evaluation team considered the possibility that grade level may influence the performance of students. In order to address this question, further analysis segregated Study Two's groups into 2nd or 3rd grade students in order to examine the results more granularly (Table 8 and Table 9).

These data show 3rd grade students achieved significantly higher improvement than the 2nd grade students in both the experimental and control groups.

3rd grade students in the experimental group attempted an average of 16.7 more problems, answered an average of 17.2 more problems correctly, and reduced their error rate from 18.7% of attempts to 6.9% of attempts following game play. This compares to the results for 2nd grade students in the experimental group who attempted an average of 5.8 more questions, answered and average of 7.8 more questions correctly, and reduced their error rate from 27.8% to 6.0%.

Date for the control group show similar results. The 3rd grade students attempted an average of 12.2 more problems, answered an average of 19.0 more problems correctly, and reduced their error rate from 38.0% of attempts to 2.5% of attempts following teacher instruction. This compares to the results for 2nd grade students who attempted an average of 9.2 more questions, answered and average of 11.8 more questions correctly, and reduced their error rate from 30.1% to 2.1%.

The data held consistent in grade level comparisons following the final assessment where 3rd grade students attempted 14.6 more problems on average while answering and average of 19.5 more problems correctly and reducing the error rate from 28.0% to .03%. The 2nd grade students attempted an average of 9.5 more problems, answering and average of 12.6 more problem correctly and reducing their error rate from 30.1% to 2.1%.

Table 8 -- Students in Study #2 - 3rd Grade Students Only

Description

Experimental Group

Control Group

Combined Treatments

Pre-Test

Post-Test

Chng

Avg Chng

Pre-Test

Post-Test

Chng

Avg Chng

Pre-Test

Post-Test

Chng

Avg Chng

# of Students in Class

# of Attempts

107

204

+103

+16.7

100

161

+61

+12.2

207

368

+161

+14.6

Attempted (% of Possible¹)

17.8%

34.0%

20.0%

32.2%

18.8%

33.5%

Correct

190

+103

+17.2

157

+95

+19.0

149

367

+218

+19.5

Correct % (of Attempts)

81.3%

93.1%

62.0%

97.5%

72.0%

99.7%

Error

-6

-1.0

-34

-6.8

-57

-5.18

Error Rate (% of Attempts

18.7%

6.9%

38.0%

2.5%

28.0%

.03%

¹ Total Attempts possible is 100 problem per test multiplied by the number of students

Table 9 -- Students in Study #2 - 2nd Grade Students Only

Description

Experimental Group

Control Group

Combined Treatments

Pre-Test

Post-Test

Chng

Avg Chng

Pre-Test

Post-Test

Chng

Avg Chng

Pre-Test

Post-Test

Chng

Avg Chng

# of Students in Class

# of Attempts

+29

+5.8

134

+55

+9.2

133

237

+104

+9.5

Attempted (% of Possible¹)

10.8%

16.6%

13.2%

22.3%

12.1%

21.5%

Correct

+39

+7.8

125

+71

+11.8

232

+139

+12.6

Correct % (of Attempts)

72.2%

94.0%

68.4%

93.3%

69.9%

97.9%

Error

-10

-2.0

-16

-2.7

-35

-3.2

Error Rate (% of Attempts

27.8%

6.0%

31.6%

6.7%

30.1%

2.1%

¹ Total Attempts possible is 100 problem per test multiplied by the number of students

The data was further analyzed to aggregate the two experimental groups and report the impact of Timez Attack after both studies. Table 10 reports the results for the experimental groups.

The final data analysis viewed the experimental group by grade level. That data is summarized in Table 11.

Table 10 -- Combined Report - Both Studies Experimental Group

		Attempts				Correct				Errors
Description	Count	Pre	Post	Change	Avg	Pre	Post	Change	Avg	Pre	Post	Change	Avg
Study #1 (all participants)	24	968	1,215	247	+10.3	941	1,207	266	+11.1	27	8	-19	-.8
Study #2 (experimental group)	11	161	287	126	+11.5	126	268	142	+12.9	35	19	-16	-1.5
Totals	35	1,129	1,502	373	+10.7	1,067	1,475	408	+11.7	62	27	-35	-1.0

Table 11 -- Combined Report - Both Studies Experimental Group By Gradet

		Attempts				Correct				Errors
Description	Count	Pre	Post	Change	Avg	Pre	Post	Change	Avg	Pre	Post	Change	Avg
3rd Grade	30	1,075	1,419	344	+11.5	1,028	1,397	369	+12.3	47	22	-25	-.8
2nd Grade	5	54	83	29	+5.8	39	78	39	+7.8	15	5	-10	-2.0
Totals	35	1,129	1,502	373	+10.7	1,067	1,475	408	+11.7	62	27	-35	-1.0

All of the data analyzed suggest that Timez Attack (both instead of and combined with teacher-led instruction) supports significant positive results for students learning multiplication facts, similar to results from teacher-led classroom instruction.

Timez Attack and Student Self-Efficacy

In addition to the pre- and post-tests of multiplication facts, we employed survey instruments, observation, and student interviews to assess students' experiences playing TA. Our focus in this section is on students' mathematics self-efficacy, which is their sense of their abilities to master math facts and their general learning capabilities.

The surveys were distributed to Study 1 as well as Study 2 (both experimental and control groups). Table 12 summarizes the analysis of the survey responses to these questions.

The student responses to question 1 suggests an increased positive feeling that "I can learn new multiplication math facts easily" for all participants. Study 2's experimental group showed a much greater increase in this sense of self-efficacy than the control group, and significantly higher than students exposed to TA in Study 1.

Students who played Timez Attack--both in Study 1 and in Study 2's experimental group--agreed more with the statement "I like learning multiplication facts" after playing Timez Attack, whereas student agreement with question 2 declined in Study 2's control group.

Responses to question 4, "Remembering new multiplication facts is hard for me", showed nearly identical increased disagreement in both control and experimental groups, suggesting that self-efficacy in terms of memory simply improved with practice.

Question 5 attempted to measure anxiety about math, stating "I usually do not worry about remembering multiplication math facts". Agreement with this statement went up in both Study 1 and Study 2's experimental group, but down slightly in Study 2's control group.

Table 12 -- Student Responses to Likert-type Self-Efficacy Survey Questions

Description	Question #2			Question #2			Question #4			Question #5
Description	Pre-Test	Post-Test	Diff	Pre-Test	Post-Test	Diff	Pre-Test	Post-Test	Diff	Pre-Test	Post-Test	Diff
Study #1 (all participants)	3.913	4.478	0.565	4.000	4.565	0.565	1.500	1.409	-0.087	3.136	3.136	0.000
Study #2 (experimental group)	3.182	4.182	1.000	4.091	4.545	.0455	1.182	1.000	-0.182	3.091	3.273	0.182
Study #2 (control group)	3.727	3.818	0.091	3.818	3.545	-0.273	1.364	1.300	-0.182	2.000	2.182	0.182
Survey Questions: Question 1: I can learn new multiplication facts easily. Question 2: I like learning multiplication facts. Question 4: Remembering new multiplication facts is hard for me. Question 5: I usually do not worry about remembering multiplication math facts.

Findings from Qualitative Data:

Seven observers recorded their impressions watching students engage with TA, while one observer recorded her observation of the control group. Students also responded to two open-ended post-survey questions about their experience with TA. We coded all statements and, in the course of coding, 14 main codes emerged. They are described below, with any relevant sub-codes and examples taken from the data:

Table 13 -- Descriptions and Examples of Codes

Code	Description	Examples
Comparison	The statement represents a students’ comparison of Timez Attack to another game or another type of mathematics learning activity. Such statements were sub-coded as either Positive or Negative, with Positive indicating a preference for Timez Attack, and Negative indicating a preference for another activity.	Positive: "I play a few other math video games. It’s pretty good. Better. … I like it more.” Negative: "It was pretty good. It’s a little bit less good because there’s this [other] game I really like.”
Conceptual Understanding	These observation notes give evidence that the students might be developing a conceptual understanding of the process behind or meaning of multiplication as repeated addition.	“It is 7X12 so since there is 7 balls in each [and] I count by sevens to get the answer.“
Emotional Response	These statements include any action or declaration by students that give insight into what emotional response they may be having to playing the game. Such statements were sub-coded as either Positive or Negative. Positive sub-codes correspond with positive emotions (such as enjoyment or happiness), while Negative sub-codes correspond with negative emotions (such as frustration or confusion).	Positive: Student shouted, “This is fun!” Negative: It was obvious he knew the math facts and was looking to use them in the game, but the pace of the game and confusing instructions frustrated him.
Engagement	Statements with this code reflect how engaged the students were while playing the Timez Attack program. Statements were sub-coded as Positive or Negative. Positive sub-codes indicate active engagement with the game, and Negative sub-codes indicate disengagement from the game.	Positive: He was engaged in the game throughout the entire session. Negative: Looked very bored pushing the button to throw the balls. Not looking at screen when doing this.
Game Construct	This code focuses on the basic appearance and design of the game. Sub-codes are Graphics, Audio, Timing, and Characters.	Graphics: When the colors shine out at one point in a fantastical way, she "ooohs." Audio: The students interacted differently with the game. [Some students] used the headphones. [Two others] did not. Timing: “Umm... how when the spiders come out, there are blue things... it like turns blue near them and then you can't get them--it takes a long time to get them.” Characters: Commenting about the characters while playing it.
Gameplay	Statements given this code reflect the dynamic and interactive nature of gaming as students play Timez Attack.	Picked up the spider characters and threw them over the edge of the walkway opposite the wall where they were supposed to be thrown for the game. Played the game instead of the math.
Hawthorne Effect	These statements reflect an influence of the evaluator/observer on the student trying to take the test or play the game.	Very nervous about being watched. Edgy each time someone is looking over his shoulder. Looks up at person watching and has a difficult time proceeding with game.
Learning	These observation statements contain a reference to students learning multiplication facts using Timez Attack.	"I like how it's a fun game and you can learn in a fun game."
Problem-Solving Strategy	These statements are about students demonstrating any type of problem-solving strategy to solve multiplication problems.	Student was using his fingers as manipulatives as soon as he encountered a multiplication fact. He actually took his fingers off of the controls to figure the answer.
Self-Efficacy	Statements regarding students’ self-efficacy reflect how the student viewed himself as a mathematician and how competent he feels in his mathematical abilities. Such statements might include insights into a student’s confidence, or lack thereof, in doing mathematics. Sub-codes were either Positive or Negative, where Positive indicates a high level of self-efficacy and Negative indicates a lower level of self-efficacy.	Positive: “The game makes me feel better at math.” Negative: “This is hard… I can’t think of it!”
Social	These statements involve an interaction between two or more students or observers while playing Timez Attack. Such statements give insight into what students are saying/doing to each other, how they interact with those around them, and how their interactions are influencing their use of Timez Attack.	Asked her neighbor for help. He showed her what to do.
Technical Issue	Statements indicating issues with the game not working due to problems with the hardware/software.	Froze a bit, then started working.
Usability	These statements indicate how easily the students are able to use the game and achieve their goals. Sub-codes were Positive or Negative, with Positive indicating student facility with the game and Negative indicating difficulties had by the students.	Positive: Used mouse combined with direction arrows on keyboard to move more smoothly. Negative: Still confused by game controls and movements.
Usage	These observation notes deal with the extent to which the student has used Timez Attack in the past.	"I played it a lot last year at school.”

There were 335 total coded statements from these observation protocols, informal interviews, and survey responses. Because multiple codes were allowed for some statements, a total of 486 codes were assigned. The following table and accompanying chart display the frequency of code occurrences:

Table 14 -- Frequency of Codes

Code		Frequency
Emotional Response	Positive	61	103
	Negative	42
Gameplay			84
Usability	Positive	26	58
	Negative	32
Learning			41
Engagement	Positive	22	35
	Negative	13
Social			31
Game Construct	Graphics	6	29
	Audio	6
	Timing	9
	Characters	8
Self-Efficacy	Positive	14	24
	Negative	10
Usage			21
Conceptual Understanding			18
Comparison	Positive	16	17
	Negative	1
Technical Issue			12
Problem-Solving Strategy			8
Hawthorne Effect			5
TOTAL:		486

Three tables listing all of the student comments noted during evaluator observations or recorded as responses to the open-ended post-survey questions can be found in Appendix B.

Discussion of Findings

Based on the data above, we found the following:

Achievement Comparison between TA and Classroom Instruction

Was there a difference in students' abilities to memorize multiplication facts? First, we split a 3^rd grade class in half and gave them all a pretest. Then, for 30 minutes, one half experienced classroom instruction while the other half played TA. Last, all the students took a post test. The data from pre- and post-tests show no significant difference between the two groups. On the bright side, students in both of these groups attempted more problems and made fewer errors on the post-test than other groups that experienced only one of these.

Grade/Age Differences

Not surprisingly, 3^rd graders who played TA made larger gains than 2^nd graders who played TA for the same amount of time. It might follow to reason that the young tender minds of 2^nd graders should thus be protected from TA. However, this same reasoning would also require that scrawny individuals should be disallowed from gyms, sick people should be barred from hospitals, and ignorant children should be prevented from attending school at all. In the face of all this, the ambitious educator will see tremendous opportunity. It is likely that if 2^nd graders practice more than 3^rd graders, those 2^nd graders will eventually exceed 3^rd grade performance. We highly recommend further evaluation to provide evidence for this assertion.

Anxiety and Self-efficacy

In the group that experienced 30 minutes of direct classroom instruction, there was no significant difference in reported self-efficacy from before to after. However, when we compared pre- to post- testing of the group that played TA for 30 minutes, there was a significant positive change in self-efficacy. Students who only played TA also reported much less anxiety toward learning multiplication facts after they had played for half an hour.

Peer Interaction

Students frequently engaged in social interactions with their peers during game play. This suggests that social networking applications within the TA game play structure could lead to greater usage and deeper effects on student self-efficacy.

Learning vs. Gameplay

Students enjoyed playing the game. When asked what they disliked about the game, they most often mentioned the robot, falling off of bridges, or dying and having to restart. Although these items were intentionally built into the game, some math teachers (who we talked to on the side) were concerned that it took too much time away from student learning.

Marketing

We found that TA helps students learn their multiplication facts about as well as traditional classroom approaches. Therefore, it is a good supplement to traditional classroom teaching for visually and conceptually teaching the times tables. We also found that playing TA improves students' self-efficacy of multiplication skills more than traditional classroom approaches to teaching multiplication facts. These two facts suggest that in order to effectively market TA to school districts, teachers, and administrators, TA ought to be advertised as a powerful supplement to classroom instruction that helps to build student confidence and mastery. This marketing approach will also bring with it the idea that these benefits can be accessed outside of the time and social constraints inherent in traditional classroom approaches.

Conclusions

The focus of this evaluation project was to determine if playing TA improves multiplication fact mastery and/or contributes to positive changes in students’ multiplication self-efficacy. According to the data collected, playing TA does improve mastery of multiplication facts, but this improvement was roughly equal to that of the control group. This suggests that TA is equally as effective as classroom instruction in helping students achieve multiplication mastery. Additionally, students who received teacher instruction first and then played TA exhibited the highest improvement overall. While this improvement could be attributed to time on task, it shows that time spent playing TA is time "on task" and its use extends beyond the simple entertaining nature of other educational games.

According to the results of the multiplication tests and the student math attitude surveys, it appears that playing TA results in a positive change in students' multiplication self-efficacy. The pretest/post-test results showed that in all of the study groups, students attempted more questions on the post-tests. This indicated a higher level of self-efficacy (as students appear to be more confident in attempting more test items). This increase in multiplication self-efficacy was roughly equivalent to the increase that was experienced by those who received only classroom instruction. This indicates that playing TA has the positive effects of improving students’ multiplication self-efficacy beyond the classroom setting.

The results from our study indicate that TA is useful as a supplement to classroom instruction. Teachers can only devote a limited amount of individualized attention to each student, while playing TA allows for an unlimited practice experience with multiplication facts outside of classroom instruction. For those students who may feel anxiety or experience difficulty in learning multiplication facts in classroom settings, TA may help provide an alternative environment for students to learn by trial and error without time or social pressures. Games like TA can offer alternative modes of learning that may reach students who are not as prepared to learn in traditional classroom settings. As more is required from students and teachers, it will become critically important to leverage technology to help everyone use time more efficiently in the classroom and in the home.

Possible Confounding Variables in Findings

The summary of findings is supported by the quantitative and qualitative data collected for this evaluation. However, there are some confounding variables that may be influencing these reported results that are unrelated to specific research questions. Some of the variables that we consider pertinent to the analysis include the following:

The students took the same test three different times within a compressed time period. The frequency of testing may have allowed students to experience greater confidence unrelated to the methods of instruction. This may have also allowed the students to incorporate strategies in the taking of the test that build upon the prior test experience to short-cut answering without considering each problem.
The students in Study 2 were separated into two different groups with one group led by a teacher and the other group using Timez Attack for instruction. There may be variations in time-on-task spent learning the math facts based on whether the student was participating in the classroom instruction or the game. This may be one reason for the measures of improvement in the quantitative analysis of these two groups.
The students were immersed in math-related activities throughout the duration of the study. This focus may have influenced performance on the assessments. A controlled follow-up with some interval of time between assessments and engagement in the evaluation exercise would inform whether the students had really improved in proficiency by retaining the facts over a longer term.

Recommendations

The results and findings of this evaluation report can be helpful in two major ways. First, the findings can be utilized to drive the enhancement of Timez Attack game programming and marketing strategies. Timez Attack can be really engaging for students, but teachers and other school personnel are largely unaware of how the game will provide a benefit for the students. Parents may also experience this confusion about the learning benefits of Timez Attack game play. We suggest that Big Brainz gear its marketing toward helping teachers, administrators, and parents understand that Timez Attack has been shown to provide a fun, confidence building way to complement the teacher in the classroom by producing higher times-table competencies than would otherwise be attained. It could be helpful to provide information on the website addressed to the different stakeholder groups such as parents, teachers, school administrators, etc. Building rapport with these stakeholders can build market penetration and client satisfaction.

Another recommendation is making the information of students' progress more readily available to teachers. The inclusion of more granular data reporting for the teacher could include measurements of time spent on encounters, time spent moving around, number of correct and failed attempts for each problem, sortable by multiplier/multiplicand, etc. Additional reporting metrics will add value to the game when a buying decision is considered.

Future Evaluation Plan

Our initial findings can provide direction for a more comprehensive and thorough evaluation of the Timez attack software with a more robust environment, conditions, and subject pool. In an effort to promote continuous improvement, we submit here some suggestions for future evaluations.

We recommend that a future evaluation work with students at the beginning of their 3rd-grade year before they have been exposed to the multiplication tables. This will reduce the lurking variable of students' previous exposure to the multiplication facts. Also, we recommend that the evaluation be more longitudinal in nature with a greater number of students. This will allow for more concrete, trustworthy results, increasing the strength of the marketing strategy.

Additionally as part of our data analysis, we discovered that for the 12s multiplication facts, students improved on some numbers much more than they did on others. 3,4,5,11,12 seem to be the numbers that most students got wrong, and interestingly enough, 3,4,10, and 12 were the numbers that students attempted more. Also, these seem to be the numbers where students made the most improvement. If we look between the lines, we see that 5,7,8,9, and 11 were the least attempted, and 6-9 were the numbers where the least improvement was made. We did not include these results in the evaluation because there are too many lurking variables, such as how often the questions show up on the test, where they are placed on the test, and students' individual test-taking strategies. However, we do recommend that a future evaluation examine which questions are more problematic for students (controlling for the lurking variables by having students use the same strategy and having the questions more evenly distributed). This information could affect how the game could focus more/less on certain multiplication facts, which would be a significant marketing advantage.

Below are listed other possible questions and methods for future evaluations.

Evaluation Questions:

1. What effect does game play have on the way students feel about their ability to learn and use math functions?

2. How do different groups use the game?

Public school students
Public school teachers
Other school personnel (librarians, lab technicians, SPED, etc.)
School Administrators
Home schoolers

3. What influence does the teacher have on the attitudes of the students with respect to the game?

4. Does the confidence developed by game play transfer to other math areas? In what ways?

5. Does the confidence developed by game play transfer to other curricular subjects? Which and in what ways?

6. What other methods are used to teach the multiplication tables in traditional classrooms? How much time do they take? How do their effects

compare with Timez Attack?

7. How do teachers feel about Timez Attack?

8. What do students remember the most after playing Timez Attack?

9. What is the effect of Timez Attack as a remediation intervention?

Evaluation Methods:

In this evaluation, two classes of second and third grade students were observed before and after they played Timez Attack. There were some serious factors that reduced the effectiveness of this evaluation, which should be remedied in future evaluations. It is suggested that:

1. Students have no previous experience with the Timez Attack game.

2. Students are observed who have not begun to learn their multiplication facts.

3. Feedback from teachers be intentionally solicited as part of the evaluation.

4. Only online versions of the game be used so that usage analytics can be acquired.

5. More than two classrooms be used in the study.

6. More than one school demographic be used in the study.

7. Multiple observations be conducted over a period of time.

8. The study starts with the beginning and not the end of the school year.

9. More interviews and case studies are generated to provide a thick description of the experience of the clients.

10. Students be chosen at random to participate in the study to enhance validity of the study.

11. Student achievement be assessed over time to analyze long-term recall of math facts.

These suggestions can help to drive a deeper and more meaningful exploration of the effects that Timez Attack is achieving on the various stakeholder groups.

Appendices

Appendix A - Data Collection Instruments

Pre-Multiplication Attitudes Survey

Read each sentence.

Circle “NO” if you think what is says is not true AT ALL

Circle “no” if you think the sentence is mostly not true

Circle the “?” if you don’t know or are not sure

Circle “yes” if you think it is mostly true

Circle “YES” if you think it is VERY true

1. I can learn new multiplication math facts easily

yes

YES

2. I like learning multiplication math facts

yes

YES

3. I am good at memorizing multiplication math facts

yes

YES

4. Learning new multiplication math facts scares me.

yes

YES

5. I usually do not worry about remembering multiplication math facts.

yes

YES

6. Learning new multiplication math facts makes me feel bored.

yes

YES

7. I study my multiplication math facts at home.

yes

YES

8. I have played Timez Attack…(Circle the correct answer.)

yes

YES

Post-Multiplication Attitudes Survey

Read each sentence.

Circle “NO” if you think what is says is not true AT ALL

Circle “no” if you think the sentence is mostly not true

Circle the “?” if you don’t know or are not sure

Circle “yes” if you think it is mostly true

Circle “YES” if you think it is VERY true

1. I can learn new multiplication math facts easily.

yes

YES

2. I like learning multiplication math facts.

yes

YES

3. Remembering new multiplication facts is hard for me.

yes

YES

4. Learning new multiplication math facts scares me.

yes

YES

5. I usually do not worry about remembering multiplication math facts.

yes

YES

6. Learning new multiplication math facts makes me feel uneasy and confused.

yes

YES

7. I will play Timez Attack at home.

yes

YES

8. I think playing Timez Attack would help me know my multiplication math facts.

yes

YES

9. Learning to play Timez Attack was easy.

yes

YES

10. Playing Timez Attack is a fun way to learn my multiplication math facts.

yes

YES

11. What I love about Timez Attack is… (Write you answer below.)

12. What I hate about Timez Attack is… (Write your answer below.)

Multiplication with 12s Pre/Post-test

Appendix B - Student Comments

Student Comments Recorded by Observers During TA Play

Shouted “This is fun!”

Said she had played the game “a few times."

Exclaimed “I’m Good!” when he defeated the robot after answering all of the facts in an end of level checkpoint. “Yes!”

“Finally” when completing the end of level checkpoint after she had lost to the robot on two previous attempts

“When do we get the dvd so we can take it home? I wish I could load it today at home and play it now.”

"This is hard." "I can't think of it."

"Watch this, it's funny!" she says when she was at a challenge door.

"I played it a lot last year at school. I think it's really cool. It's, I don't know.” (shrugs)

"It's pretty fun. I don't know, it's just fun."

"I knew what it was, but I can't find the keys. It's hard."

"What the heck?" one student says when a challenge door is missing the second bottom numeral. Another girl said the same thing had happened to her.

"I really like it. I just enjoy doing my times tables. This is one of my favorite ways. It's times tables made into a game so double the fun." He said he had played it a long time at school, and liked the lava world because "it's the hardest one."

"Do you know if you wait, the spiders just walk to you?"

"It was fun. It was pretty good (compared with other games he has played). It's a little bit less good because there's this game I really like. You go around catching stuff and stuff. "

"Yes I did it" a girl pumps her fist when she defeats the robot.

"I hate this part. I always fall off the edge" when she's at the bridge, trying to cross.

"Finally I'm going to the next level. Are you on the third level? I'm on the third level. I passed you. Have you been keeping track of your levels?"

"Look at me!" she says this as she falls off the cliff.

"I've played the older version at school. Don't we get to download this at home? My mom will probably let me do that. I play a lot [of games]. I play Wii. I play a few other math video games. It's pretty good. Better (Timez Attack is vs. those others). I like it more. I like video games where you run around. The running around and fighting the bad guys. Do we get to play again? I want to play right now.”

"I thought it was fun because you can do challenges and stuff but math is involved. I've never gotten past the dungeon" "It is pretty much the same fun [as other games]."

"It was awesome! It was really fun to play." She most liked going against the robot but said it was " a little hard to learn the keys.”

"You can't backspace [when at a challenge spot]. It sucks."

Interview- Q: What did you like the most about Timez Attack? “Helps us learn times tables. Parents can't tell us not to use it. I like it the most cause it's long.”

Q: What did you NOT like about Timez Attack? “When you really want to finish it, try to go too fast, and make dumb mistakes.”

"Yes" when he beats final boss

"I can't see. It's too dark on the thing."

"Yes" "This is really cool"

"I don't know how to work it."

"It's angrifying" the thing in the game doesn't com back if you fall

"My computer's frozen."

"I can't think of it!"

"Ah, 12s are hard!"

Interview – Q: What do you like about Timez Attack? “It's not like having a piece of paper or flashcards. It's like a game. It's fun. And it teaches you and makes it so you can memorize.”

Q: What don't you like about Timez Attack? “ Umm... how when the spiders come out, there are blue things... it like turns blue near them and then you can't get them--it takes a long time to get them.”

Q: Do you think playing Timez Attack is useful? “It's good practice.”

Q: if you could change anything about Timez Attack, what would you change? “On the last level, make it easier to get the dragon.”

Q: Do you play at home? “Not much. I play one or two times in three or two weeks.”

Q: Do you have any siblings? Do they play at home? “I have three younger siblings, but they don't play. My older brother used to play when he was in third grade, but he passed all of them, so he stopped.”

"I don't know how to get out.”

"Did I just come up from there?"

Interview - Q: Why do you throw the snails at the wall? “ Well, they represent each group in the array.”

Q: So, they help you count up? “Yeah.”

Interview – Q: Why do they have those dots? “Because they... it's sort of like an array."

Interview - Q: Did you know your 12s coming in? “Sort of. A few of them--that would have been useful to know!"

"No!" he exclaimed quietly when he missed another question.

"Now I'm fighting the big boss, I'm guessing."

"No!" "This is so fun!"

''Holy cow you are good”

“Not another one of these guys”(robot) “This is fun”

Interview - Q: Is it easy to play the game? “Kind of easy (the multiplication can be hard).”

Interview - Q: Have you played the game before? “Yea, at school.“

Shouting “yea” when she is correct, shouting “oh no” when wrong.

Interview - Q: Why do you throw balls at the gate? “It helps you figure out the right answer so you can get in.”

Interview - Q: Why do you throw balls at the gate? “You have to throw the balls to get in the gate.“

Interview - Q: Why do you throw balls at the gate? “Counts until last one then that is the answer .”

Interview - Q: Have you play this before “Yes.”

Q: Where? “At school and home. At home it has slugs. I like slugs better.”

Q: Why are there groups with 5 balls on the gate? “Not sure.”

Interview - Q: Why do you hit the spider? “Helps you count to get answer.”

Q: What do you like about the game? “You do not have to ask anyone the answer if you forget.”

Q: Anything else? “It makes me get me answers faster.”

Q: What do you not like about the game? “When I get the wrong answer.”

Claps when right, says “ah sweet”

Interview - Q: Why is it counting by two when you throw the ball? “Because it is 2 x 12, so you have to count by 2’s.”

“I do not like battling the robot. He beats me up.” “Oh no, I killed him” “Not sure how I get to next level. Am I in it?”

Says “phew” when he gets them right.

Interview - Q: Why are there groups of balls on the gate? “They help you figure out the answer.”

Q: How? “Since it is 9X12 you have 9 groups. Wait, I guess I have 9 in every group and then I would have to count twelve of them.”

Interview - Q: Why are there groups of balls on the gate? “To get the answer “

Q: Why is there 7 in each group? “It is 7X12 so since there is 7 balls in each I count by sevens to get the answer.“

“YES!” at checkpoint, with handpump.

Interview - Q: Why do you throw the balls? “Because it counts up and gives the answer if you need it. “

Likes the game “because it's a mix between a game and times tables.”

Likes the game “better than memory cards.”

Game “makes me feel better at math. “

“I just did it!”

“Ah, 12s are hard!”

“Oh, it's . . .; I can't do it . . .” “Yes!"

“Okay, think.”

“Yes!” when he found it was right.

“I just have to beat the sixes and I'm done. Can I play the game?”

“Yes! I got it done!”

“Oh! Oh my Gosh!”

“Ah!”

“Turn the other way!”

“Door, another door, weird.” “Yes! Oh, bam bam ...”

“Oh!”

To his friend- “It shows you the ones you need to practice to beat the game.”

“Okay.” “Ah.” “Ch, ch.”

“Oh no!”

“Ah! I forgot that one!”

Gives her friend suggestions- “Just walk off the edge.”

Singing- “I'm going backward; I'm going backward.”

“Oh!” -goes back and laughs. “When is it gonna come and get me?”

“How do I do this part?” “Ahh! I did it!”

“And I was doing good and just figured out how . . .”

“Only got two.”

“When does this guy [robot] end? When does this guy end? I'm getting annoyed at how long he's taking! Do you know how much longer?” “I couldn't finish it.” -He told me. “How many lives do I have?” -He asked me. “I had like twenty.”-He replied. I asked him if he was glad to be doing something else rather than facing the robot. “Yeah, but I'm gonna have to face him again.” -He said.

“That was hard!”

“It took me 25 minutes just to beat the robot!”

“Sixty.” (Speaking right answer as typing.)

“Oh come on, no!”

One student, while filling out pre-survey, "Nobody would do it almost everyday!"

Student leaving to play, "I'm not good at Timez Attack. I always get stuck in the dungeon."

"It's a good game."

"Burst all the people"

"I like how it's a fun game and you can learn in a fun game."

"They made it more difficult. Like you can go in alleyways."

"I like how they made a videogame for while you're learning."

"I didn't hate any of it."

"I liked it. It was a fun game."

"There's just SO much spiders! You have to get ALL of them and it takes a long time."

"I liked every bit of it!"

"It kind of takes a long time to figure them out..."

"I LOVED all of it!"

"Well, some of the class didn't like it."

Student Responses to Post-Survey Question 11

The following responses are taken verbatim from the open-ended, post-survey question, "What I love about Timez Attack is..."

It helps you remember timz tables

Asoum

It is a game you can play and learn lots of new stuff

It is fun.

you can learn multiplication in a fun way.

It helps you learn your math facts.

It's a fun game to play and help's you learn multiplication

It make me feel smarter.

they made multiekatshin easy

Everything

how you can chang levels!

I love the green guy.

you had to fite

It helped me learn my math facts

It's a good way to learn.

the same

That you can learn about the math facts!

That you have challenges and match is envolved

It teaches you multiplication facts.

It helps you learn more.

I can learn math while I'm playing computer games

That it teaches you something when you play. So it's a good way to learn.

It's a fun way to learn times tables.

the math.

mutipulcation

it teches me times

it’s a fun way to learn math facts

It helps kid learn times tables.

nothing

its fun and it teachs you x tables

It help me think

You can learn and have fun at the same time!

Lrning math facts

it gives you a challage.

It's so fun. It helped me with my multiplucation 12's.

killing the robotst.

nothing

you lern your multiplication

when you colecet the spiders

Because its fun and a good way to learn your times quistian

You get to go on the moving elivater thing

the ashin

It's fun and easy and you do not get lost. P.S. the green dude

it is math but it is a fun game

Student Responses to Post-Survey Question 12

The following responses are taken verbatim from the open-ended, post-survey question, "What I hate about Timez Attack is..."

starting over

nothing!

It is hard.

you have to get the spiders & slugs.

I fall off when I'm trying to go really really fast.

that you don't just get to write the anser then

the spiders

I hatted nothing

They tell you the answer when you get it wrong

nothing!

The robot!

nuthing

the graphikiks

nothing.

the robot

That you fall off when you miss going on the blocks!

nothing

You have to type in the answer before a certain time

Ansers to hard

If you died, you would have to restart the level

I don't hate it but it challenging to go on the transportation you might fall.

the green guy doesn’t have much clothes on.

you die.

doesn't give you enough time to answer

nuthing

NONE

nothing!

nothing

nothing!

nothing

It doesn’t give me the time I need to find the numbers on the keybord!

nathing

nothing

I keep pressing rong buttons

when the brige was really skinny.

That once I beat the boss it made me start over

Nothing

the robot!

its hard to get from qustion

You have to throw the dimends.

dus not let me think

Nothing

they make you go back if the robot kills you

References

Berk, L. (2009). Child Development (8^th ed.). Boston, MA: Pearson Education, Inc.

Carter, G. S., & Norwood, K. S. (1997). The relationship between teacher and student beliefs about mathematics. School Science and Mathematics, 97, 62-67.

Fennema, E., & Sherman, J.A. (1976). Fennema-Sherman mathematics attitudes scales: Instruments designed to measure attitudes towards the learning of mathematics by males and females. JSAS Catalog of Selected Documents in Psychology, 6(1), 3b.

Federation of American Scientists. (2006). Harnessing the power of video game for learning. Retrieved from http://fas.org/gamesummit/.

Jackson, C. D., Leffingwell, J. R. (1999). The role of instructions in creating math anxiety in students from kindergarten through college. The Mathematics Teacher, 92(7), 583-586.

Ke, F. & Grabowski, B. (2007). Gameplaying for maths learning: Cooperative or not? British Journal of Educational Technology, 38(2), 249-259.

Kebritchi, M., Hirumi, A., and Bai, H. (2008). The effects of modern math computer games on learner's math achievement and math course motivation in a public high school setting. [Research brief based on dissertation research for Doctor of Philosophy in Education with an Instructional Technology Specialization from the College of Education at the University of Central Florida (UCF)].

Klawe, M. M. (1998). When does the use of computer games and other interactive mult-imedia software help students learn mathematics? Retrieved from http://www.cs.ubc.ca/nest/egems/reports/NCTM.doc.

Meece, J. L., Wigfield, Al, & Eccles, J. (1990). Predictors of math anxiety and its influence on young adolescents' course enrollment intentions and performance in mathematics. Journal of Educational Psycholoy, 1, 60-70.

Moreno, R. (2002). Who learns best with multiple representations? Cognitive theory implications for individual differences in multimedia learning. Paper presented at World Conference on Educational Multimedia, Hypermedia, & Telecommunications. Denver, CO.

National Council of Teachers of Mathematics. (1995). Assessment standards. Retrieved from http://www.standards.nctm.org/.

National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National Mathematics Advisory Panel. Washington, DC: U.S. Department of Education.

National Research Council. (1989). Everybody counts: A report to the nation on the future of mathematics education. Washington, DC: National Academy Press.

Nichols, J., Cobb, P., Wood, T., Yackel, E., & Patashnick, M. (1990). Assessing students' theories of success in mathematics: Individual and classroom differences. Journal for Research in Mathematics Education, 21(2), 109-122.

Programme for International Student Assessment. (2003). Learning for tomorrow's world: First results from Programme for International Student Assessment 2003. Retrieved from www.pisa.oecd.org.

Rosas, R., Nussbaum, M., Cumsille, P., Marianov, V., Correa, M., Flores, P., et al. (2003). Beyond nintendo: design and assessment of educational video games for first and second grade students. Computers & Education, 40(1), 71-24.

Scarpello, G. (2007). Helping students get past math anxiety. Connecting Education & Careers, 82(6), 34-35.

Sedighian, K. & Sedighian, A. S. (1996). Can educational computer games help educators learn about the psychology of learning mathematics in children? 18th Annual Meeting of the International Group for the Psychology of Mathematics Education. Florida, USA.

Shaffer, David W., Squire, K. R., Halverson, R. and Gee, J. P. (2005). Video games and the future of learning. The Phi Delta Kappan, 87(2): 104-111.

Swetman, Daniel. (1994). Fourth grade math: The beginning of the end? Reading Improvement, 31, 173-76.

Tankersley, Karen. (1993). Teaching math their way. Educational Leadership, 50, 12-13.

Metaevaluation

The meta evaluation was performed by the evaluation team using the following peer reviewed instrument:

*Program Evaluations Metaevaluation Checklist (Based on The Program Evaluation Standards),created by Daniel L. Stufflebeam (1999).

Utility:

On the subject of Utility the overall score was 71%. Report clarity and Information Scope were particularly strong. The educational nature of this evaluation encouraged us to properly identify and clarify stakeholder needs and circumstances. Due to the limited scope of the evaluation, Stakeholder identification and Timeliness/Dissemination achieved slightly lower (Good) results. Stakeholder involvement was identified as a main weakness in the evaluation, again due to the restricted timeline of the project. Data was collected from only two classes and at only one point in time. Because of these limitations, we were unable maintain a more satisfactory level of interaction with the stakeholders. In summary, while we feel we gave adequate attention to the stakeholder’s needs there was simply not enough time to properly address the questions.

Feasibility:

Overall, the feasibility of this evaluation was rated “very good” using Stufflebeam's Program Evaluations Metaevaluation Checklist (1999). Our overall rating was a score of 8, which equates to 67%--the low end of the "very good" category.

This rating is a result of averaging three areas: Practical procedures, political viability, and cost effectiveness.

Practical procedures was our strongest area. While we question our appointment of competent staff (all evaluators were student novices), we believe we made a strong showing in other areas. For example, we tailored our methods and instruments to the client's information requirements, choosing procedures in light of known constraints. We also made use of local resources, and designed the observation to minimize disruption to the teachers and classrooms.

Political viability was our weakest area. We failed to maintain communication or involvement with the stakeholders, particularly our client, throughout the evaluation. We did not agree on, or even extensively consider, editorial and dissemination authority. Further, the lack of formality in planning and contracting for the engagement with the client created concern in understanding expectations. On the other hand, we did spend sufficient time considering positions of different interest groups. We also immediately terminated any corrupted evaluation data as it was encountered. While we were consistently aware of what our client wanted, we maintained objectivity and worked to consistently avoid bias.

Considering the cost and value of the evaluators time was null, it is not surprising that our analysis of our evaluation project was high in terms of cost-effectiveness. However, there were examples of inefficiency. Particularly, we had seven observers present during the evaluation—nearly one observer for every three students. Also, we employed all 12 evaluators in the authoring of the evaluation report. This was because this was largely an academic exercise for our course, and our instructor wanted each student to have the experience of participating in every step of the evaluation. Our strengths in this area included making use of in-kind services, providing a report that may inform decisions, and minimizing both disruption and time demands on client/program personnel.

Accuracy:

Our overall accuracy score was 68%, or “very good” based on Daniel L. Stufflebeam’s “Program Evaluations Metaevaluation Checklist” accuracy standards. In regards to program documentation, a technical report was produced and a copy was provided for the stakeholders. Limitations lie in that we did not collect descriptions of the intended program from various written sources, and we did not analyze discrepancies between the various descriptions of how the function was intended to function. Context analysis was also a strong area of our evaluation, as we noted and reported important contextual influences and effects. However, we did not analyze how the program’s context is similar to or different from contexts where the program might be adopted. Furthermore, data collection procedures were specifically identified and tied to the evaluation’s purposes. Information was collected from a variety of sources from documented populations, and all data collection instruments were documented, justified, and reported in the evaluation report (in a technical appendix). Key questions for the evaluation were identified, data analysis procedures were identified to address these questions, and information was collected in a valid and reliable manner. Although consistency between multiple observers was not checked during data collection, methods of triangulation were employed during data analysis. Our biggest weakness lied in aspects of impartial reporting. The client was not engaged during the process to ensure impartial reporting, and perspectives of all involved stakeholders were not reported. We did not obtain any outside audits of the report, and we did not report alternative plausible conclusions for our results.

Propriety:

Service Orientation: We identified the strengths and weaknesses of the program. Because of the brief nature nature of this report, we chose to focus on the needs of Big Brainz.

Advanced Written Agreements: While we did not include written agreements, we did have verbal agreements.

Rights of human Subjects: We were clear to the stake holders that the evaluation would protect the participants human rights.

Potential Conflicts of Interest:

James West (of Big Brainz) is a first cousin of Rick West (the Instructor of the BYU course that undertook the evaluation). Our evaluation team took precautions to avoid a conflict of interest by choosing a Project Manager other than Rick West to lead the interactions with Big Brainz.
Rick West's daughter attends the same school and class where we did the evaluation. Again, our evaluation team took care to minimize this potential conflict of interest.

IPT 564

Sunday, October 10, 2010

Attack on ISD

Tuesday, October 5, 2010

Evaluation Report -- Timez Attack

Evaluation Report of Big Brainz' Timez Attack

Big Brainz, Educational Gaming Company

Introduction

Evaluation Purpose

This evaluation assesses the effectiveness of TA in helping students achieve multiplication mastery and self-efficacy (one’s belief in one’s ability to succeed in specific situations). The principle evaluation questions were:

Limitations of the Evaluation and Disclaimers

Description of Timez Attack

Literature Review

Methodology

Research Design

Research Methods

Data Collection Procedures

Presentation of Results

Summary of Findings

Study One

Study Two

Possible Confounding Variables in Findings

Recommendations

Evaluation Questions:

Evaluation Methods:

Appendices

Appendix A - Data Collection Instruments

Pre-Multiplication Attitudes Survey

Post-Multiplication Attitudes Survey

Read each sentence.

Multiplication with 12s Pre/Post-test

Appendix B - Student Comments

Student Comments Recorded by Observers During TA Play

References

Metaevaluation