Bill Evers on TV on Standardized Testing

I joined Sean Reardon, an associate professor at the Stanford School of Education, for a half-hour discussion on statewide testing of K-12 students in California.

Within the over-arching topic of testing, Sean & I discuss:

-- Can you test such matters as civic understanding and ...
creativity?
-- Can you have effective school reform in the absence of
data from student test scores?
-- Is money per se the key to effective
school reform
-- Is the racial achievement gap narrowing?
-- Is there a "stereotype threat" that leads girls and African-American and Latino
students to underperform on tests? And if so, what should be done about it?
-- Should California renew its accountability system in 2010? Should it retain its statewide grade-by-grade tests? Its high-school exit exam?

The program is entitled "Class Action." The show aired on Sunday, November 8, NBC Bay Area (KNTV, Channel 11), and the video has now been posted online. The host is KNTV news anchor Jessica Aguirre.

Complaining About Testing -- Can You Guess in What Year This Complaint Was Made?

"[T]oday's schools are driven by assessment to a degree that surprised us....[T]ests [are being] used as ends in themselves, rather than as one means to assist in intellectual development of the individual student, or even as a reasonable method of public accountability.

"Tests -- usually standardized tests, but even including tests that teachers devise to monitor and judge week-to-week progress -- are omnipresent and constantly on everyone's minds. Teachers, administrators and students are preoccupied with scores and grades -- often against their better judgment, and often against the values that parents, students and they themselves say that the schools should reflect....

"More than one researcher returned from the school sites with stories about how schools elsewhere had boosted their scores by encouraging low-performing students not to attend school on the day of the [state-developed] examinations.

"In another unnamed school, according to another rumor, twelfth-graders took revenge on the school's administrators, who had cancelled the traditional Senior Skip Day, by deliberately scoring poorly on the tests....

"[T]hese [standardized] tests have encroached powerfully on the schools and on teaching....

"Many teachers and school administrators agree that the emphasis on and uses of testing reflect an educational practice out of control. Points, credits and scores are pursued and accumulated as if they represented the core values of the high school....

"[T]eachers have articulate, idealistic educational goals for their students, but amidst the pressures they face, they proceed to test in predictable ways, often modeling their approaches on the externally developed examinations they see most often, the standardized achievement test....

"In many cases, review for a test meant the teacher coaching the youngsters about the specific pieces of information to be included on the exam, along with the appropriate answer."

[Source: Inside Schools: A Collaborative Approach, by J. Myron Atkin, Donald Kennedy, and Cynthia L. Patrick (Falmer Press) -- published 20 years ago, in 1989.]

Must We Give Up High School Exit Exams?

Education school researchers at U.C. Davis and Stanford assert that California's high school exit exam is keeping girls and non-whites from graduating from high school.

The study speculates that girls and non-whites have been failing the exit exam in high numbers because of a "stereotype threat." "Stereotype threat," as suggested by psychologist Claude Steele, is a situation in which blacks, women, etc. are put off their pace by awareness that others expect them to fail.

The Los Angeles Times story on the study paraphrases Sean F. Reardon, one of the study's co-authors:

Reardon said there was no other apparent reason [than stereotype threat] why girls and non-whites fail the exam more often than [those] white boys who are their equals in other, lower-stress academic assessments.

Here's what the executive summary of the study says:

Among students in the lowest quartile of achievement, the CAHSEE [exit exam] requirement has no effect on the graduation rate of white students, but a large negative effect on graduation rates of black, Hispanic, and Asian students.

On average, among students in the bottom quartile of achievement, graduation rates were 19 percentage points lower among Black students, 15 points lower among Hispanic students, and 17 points lower among Asian students who were subject to the CAHSEE requirement than among similar students not subject to the requirement.

Likewise, graduation rates were 19 percentage points lower among female students, but only 12 points lower among male students who were subject to the CAHSEE requirement than among similar students not subject to the requirement.

I wonder about several things:

• In theory, the stereotype threat should not be lowering scores for Asians.
• It should not be lowering scores for boys in the lowest quartile almost as much as for girls.
  
• Why is the stereotype threat working only on the lowest-quartile students? In theory, it should be lowering scores for blacks and girls in all quartiles on this high-stakes test.

There would seem to be many difficulties for stereotype threat as the explanation for the results.

(The authors anticipate these objections and attempt, not very persuasively, to respond to them in their paper.)

Without getting into my skepticism about some aspects of the stereotype-threat effect, let's assume that it's true or can sometimes be true. The need then is to accustom blacks and women to competition and challenges, and the potential would seem to be there for greater success when people have instead high, demanding expectations about blacks and women.

The solution cannot be to cushion blacks and women from all challenges (like passing the not-very-tough high school exit exam). [This is a point that has been made about the stereotype threat by John McWhorter both in interviews and in his book Losing the Race.] Life is full of challenges.

Getting rid of the high school exit exam cannot be the solution. The solution has to be preparing low-performing students to pass the exam and telling them that their teachers, parents, ministers, and other community leaders expect them to succeed and will accept no excuses.

California State Supt. Jack O'Connell and others put the exit exam in place in order to have a meaningful high school diploma. It will help the students in life if they reach at least the foundational levels of knowledge that the exit exam measures. The Los Angeles Times describes those foundational levels:

The exit exam, which students can take multiple times beginning in their sophomore year, includes math and English tests, with the math aligned to eighth-grade standards and English to 10th-grade standards.

Abolishing the test and handing out meaningless diplomas to uneducated students will not really help them.

High achievement on tests stems from diligent effort and effective study habits. These need to be persevered in over time, so that knowledge, skills, and understanding can accumulate and be stored in long-term memory.

Blaming all problems on a stereotype threat and then giving up all high-stakes tests (as Reardon and others have suggested) will solve nothing.

[Thanks to Ze'ev Wurman for his advice and counsel on this post.]

UPDATE (4/23): A better place to look for advice is the 2008 report of the Public Policy Institute of California (PPIC) called "Predicting Success, Preventing Failure: An Investigation of the California High School Exit Exam," by Andrew Zau and Julian Betts. This report says that after-school reading programs and the use of peer coaches in professional development -- both instituted in San Diego during the Bersin-Alvarado years -- could help students pass the exam. It also says that the evidence suggests that intensive tutoring focused on academic content is likely to be more effective that less-intensive tutoring focused on test-taking strategies.

The Zau-Betts report also proposes spotting students early who are at risk of failing the exit exam. Then schools should provide them with academic tutoring in elementary and middle school -- rather than waiting until high school when making up the academic deficit is much more difficult.

There is room for improvement on the particulars of what San Diego did, but the Zau-Betts report points out an evidence-based path to better exit exam results: early detection, supplemental reading instruction, and intensive tutoring on academic content.

You Can't Escape If You Don't Know Soon Enough

Chad Aldeman has a post on The Quick & the Ed today (Apr. 1) about an Obama administration retreat on options for children to escape failing schools. As Aldeman puts it:

Secretary of Education [Arne Duncan's] letter rolls back a regulation that could have helped provide parents of children enrolled in unsuccessful schools the option of choosing a better one.

Aldeman explains the retreat by asking us to use our imaginations to consider a parent in the following situations:

1. [I]f you're a parent, imagine going through the school registration process in the fall, buying supplies for your child, and believing that your child will begin attending school X. Then, on the first day of school, your child brings home a letter that says her school failed to make adequate yearly progress last spring, and she now has the option to transfer to another school.
2. Instead, imagine you, as a parent, were notified at least two weeks in advance of the new school year.
   

The second scenario -- adequate notice -- is what the Bush administration's regulations called for. With adequate notice, you as a parent would be in a much better situation:

You would have time to consider your options, visit new schools (maybe even new teachers), and plan transportation. You might be altogether more interested in exercising your right to choose [a better school for your child].

The Obama administration's retreat traps parents (and their ill-served children) in scenario #1. This retreat also exacerbates another important problem: late reporting of test results.

[Secretary Arne Duncan's letter] makes it less urgent for states to turn around test results promptly, which has implications beyond just an under-used school choice provision. Late data results also penalize schools labeled in need of improvement, because it gives them little time to implement a real school improvement plan.

There you have it: less school improvement and less opportunity to escape failing schools.

We Should Escape the Babble of K-16 Standards, If We Can

Michael Kirst blogging at College Puzzle has an insightful post (Mar. 12) on the lack of alignment between (1) K-12 standards and tests and (2) college admissions and placement tests.

Kirst points out that the two testing systems only overlap in Advanced Placement courses. He says that:

[Forty-nine] states (all but Iowa) set K-12 standards and assessments...without talking with higher education institutions and state boards for higher education....

Higher education, Kirst says, pays attention to "the upward trajectory of pupils" -- for example, an admissions test’s capacity "to predict student performance in the first year of college."

The K-12 systems pay attention to high school graduation and meeting annual state and federal growth goals.

In the past, Kirst has pointed out that a lack of alignment between K-12 standards-and-testing and college admissions deprives the K-12 accountability system of support from a civicly-engaged constituency: the parents of college-bound children. In this post, he points out that the lack of alignment leads to college students having to take remediation course on subject matter that they should have learned in high school.

Kirst points out that university officials say that K–12 tests "have not been evaluated to see how well they predict freshman grades." But he says "such evaluations are not difficult to conduct."

I would be interested in hearing good reasons why we can't have grades (advanced, proficient, basic, below basic, and the like) on K-12 tests aligned to college admissions and placement levels and why we can't have admissions achievement tests aligned to placement levels . The existing "babble of standards," as Kirst terms it, sends unclear signals to students (particularly, Kirst notes, those from low-SES families).

NAEP Tries Interactive Assessment That Tests Students' Ability to Formulate & Perform Experiments

Sean Cavanagh, who blogs at Curriculum Matters, reports today (Mar. 12) that the National Assessment of Educational Progress (NAEP) science test, which has been administered across the nation in recent weeks, includes an interactive computer feature, given to a subset of children. This section of the test aims to assess student skills and understanding, such as "the ability to formulate and and perform experiments," that cannot easily be tested in other ways.

The new NAEP test has students making decisions and drawing conclusions based on data stemming from experiments. This, test writers hope, allows for "a better measure of depth of knowledge."

This may prove to be a way to combine skills, conceptual understanding, and objectivity.

Head Start, 21st-C Skills, Multiple Choice, Productivity -- Is Pres. Obama Missing What's Important?

Russ Whitehurst of the Brookings Institution responded briefly to President Obama' speech on education policy. But he also pointed back to his (Whitehurst's) Feb. 4 analysis of the President's stimulus plan. In this earlier analysis, Whitehurst made several important points on preschool education, 21st-century skills, testing formats, and school productivity:

Early Childhood

On early childhood, we should not be satisfied with more programs, “like Head Start.” A rigorous national study of Head Start found positive effects in some performance domains at the end of a year in the program, e.g., naming letters of the alphabet, but not in others, e.g., vocabulary. Most children in the study were still far below average in school preparation skills on exit from Head Start. We can do better, and there is a lot of evidence on effective preschool programs to lead the way.

21st-Century Skills

The president’s call for assessments that measure 21st century skills such as creativity and entrepreneurship is fraught with danger and difficulty. There are serious questions about whether it is possible to teach someone to be creative in a general sense, disconnected from expert knowledge in particular domains. In any case, the definitional and measurement challenges are huge.

Instead of chasing abstractions, it would be more productive to construct standards that reflect what the best experts in particular fields agree that students need to know and be able to do to be competent in those fields. That will surely involve problem solving, creativity, and the like, but anchored in content rather than floating free.

Pres. Obama's Opposition to Multiple-Choice Tests

[P]lease, Mr. President, no more talk about needing to teach students more than how to bubble in answers on tests. Any assessment expert will tell you that very complex skills can be measured with multiple choice test questions. The issue is what we are expecting students to learn, not the structure of the test item that is used to determine whether they’ve learned it.

Efficiency & Productivity

What about efficiency and productivity? We spend more on education per student than any other nation. Shouldn’t we be pursuing ways to get more out of our current investments before we double down?

Whitehurst is asking the questions that the education-beat journalists should be asking.

Should 2nd graders be tested (in Calif.)?

California state Sen. Loni Hancock has once again introduced legislation to end the testing of second graders.

Hancock cites too much "teaching to the test" and wants, she says, to reduce bureaucracy and to "let teachers teach."

But, Sen. Hancock, if I may address you directly, the principal purpose of state accountability tests is to see where schools are not performing well. We need to know this about how schools are doing with 2nd graders.

Los Angeles Teachers' Union vs. Diagnostic Tests

David Lehrer and Joe Hicks (President and Vice-President of Community Advocates, respectively) posted an incandescent piece today (Mar. 9) on the Fox & Hounds blog on the opposition of United Teachers of Los Angeles to the school district’s diagnostic tests scheduled to be administered to students in two weeks.

The union has told the public that its members will “boycott a series of unnecessary assessment tests” that would result in “the loss of valuable instruction time spent prepping students for the tests instead of teaching.”

But union leaders are telling their members something quite different (namely, that the boycott is about protecting teachers' jobs, not about classroom teaching time for students). According to Lehrer and Hicks:

In a February letter to UTLA’s members its president struck a very different theme, clearly not about “education being a top priority.” He told teachers that the boycott was really about “everyone’s health care, class sizes” and that “jobs are on the line…now is the time for unity among members—not division.” Not surprisingly, the claimed pedagogic problems with the assessment tests aren’t mentioned in his teacher-directed communication.

Lehrer and Hicks go on to say that these tests have proven to serve valuable diagnostic purposes and have been shown to help children learn:

In reality, the “periodic assessment” tests which UTLA finds so objectionable facilitate teaching the curriculum that these teachers are hired to instruct. The tests’ goal, as the [Los Angeles] Times has written, is “to give teachers insight into what students need to learn while there remains time in the current school year to adjust instruction.”

A statistical analysis by the [Los Angeles Unified School District] has found that the assessments “contribute to higher student achievement.” The District’s study found that, especially at the elementary level, students who take the full battery of four assessment tests during the course of the year are “much more likely to score proficient or advanced on the California Standards Test than similar students that either do not take the tests or take them only once or twice.”

The tests are designed to provide individualized data to the teachers, within 48 hours of having been administered, of what each student has mastered in English, language arts, math, science and social studies---core academic subjects. If teachers “prep” students for this test they are teaching the curriculum that it is their job to deliver, not teaching obscure skills that are only relevant to a test.

Checker Finn's Seven Worries about National Standards

Checker Finn, one of the nation's leading proponents of national standards, writes in Gadfly today (Mar. 5) that he has accumulated seven worries.

First -- National standards may well (because of the involvement of unions and other special interests) focus on resources, instead of focusing on mastery of academic content:

...I cannot be the only person whose heart sank when Dennis Van Roekel announced that the National Education Association [NEA] was also joining the "partnership."...What really formed icicles on my toes was his declaration that this move is perfectly compatible with the NEA's adoration of "21st Century skills" and "comprehensive" standards that include "accountability for child well-being, facilities and supplies." (Who else remembers the brouhaha over "opportunity to learn standards" in the early '90s?)

Second -- National standards may well be heavily influenced by the wooly "21-Century Skills" movement:

...Speaking of 21st Century skills, the more I learn about this woolly notion, the clearer it becomes that this infatuation is bad for liberal learning; a ploy to sidestep results-based accountability; somewhere between disingenuous and naive regarding its impact on serious academic content; and both psychologically questionable and pedagogically unsound. (For a terrific exposition of these problems, see here.)

Third -- National standards may well be heavily influenced by the unsound and ideologically-freighted PISA test:

...[S]ome [in the current national-standards effort] are...overly fond of PISA--that's the Paris-based OECD's international math/science/literacy testing program for fifteen year olds--and view it as the surest path to "international benchmarking" and multi-national comparisons. Yet Tom Loveless of the Brown Center at Brookings has recently unmasked PISA's ideological bias and misguided notions about what young people should know and be able to do.

Fourth -- National standards may well not include the great works of literature:

...[A]s revisions are made in Achieve's respected " American Diploma Project" (ADP) benchmarks--these are at the core of the common standards project--one hears reports of a major tussle over whether English should continue to include literature and list important literary works.

Fifth -- National standards may well narrow the curriculum:

...[I]f the common standards enterprise remains confined, like NCLB (and ADP), to English and math, it may further narrow what's seriously taught in school--with a malign effect on states that have a decently rounded curriculum that gives due weight to science, history, even art. (Picture what happens to history education in a state that joins a national project that wants no part of history.)

Sixth -- National standards may well be ignored, because they are not tested:

...[S]uppose that the emerging standards are sound. Yet nobody is talking about common assessments to accompany them, at least not in this lifetime. But without an agreed-upon test and "cut points" for passing it (or, if you prefer, demonstrating "proficiency") these standards will have no traction in the real world of NCLB and discrepant state accountability systems.

Seventh -- Given the existing American political landscape, national standards may well change at any (unpredictable) time:

...[M]ost troubling...is institutional instability. The United States of America in 2009 lacks a suitable place to house national standards and tests over the long haul.

Who will "own" them? Who will be responsible for revising them? Correcting their errors? Ensuring that assessment results are reported in timely fashion? Nobody wants the Education Department to do this. There's reason to keep it separate from the National Assessment of Educational Progress and its governing board.

Yet the awkward ad hoc "partnership" now assembling to pursue this process could fall apart tomorrow if key individuals retire, die, or defect; if election results change the makeup of participating organizations; if the money runs out; or if their working draft runs into political headwinds like the "voluntary national standards" of the early 90s.

Finn remains cautiously optimistic about national standards, at the same time, he has given us all a list of what to watch out for.

I would add that No Child Left Behind has not led to significant narrowing of the curriculum. Schools have added a few minutes more a day for reading and subtracted a few minutes from other subjects. For National Center for Education Statistics data on this, please see:

http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2007305