A teachable moment

I do believe in serendipity.

Earlier I posted a comment to this post responding to Bob Moffitt of the American Lung Association. Bob once again ducked questions about statistical analysis applied to studies of secondhand smoke in favor of the “appeal to authority” logical fallacy -- if some organization with a title says something is true, it must be. Because I agree with the analysis of this just-arrived press release regarding reading test scores, it would be tempting for me to use Bob’s methodology and simply swallow the content of the press release.

BOB WISE, PRESIDENT OF
THE ALLIANCE FOR EXCELLENT EDUCATION,
REACTS TO MINNESOTA’S 2005 NAEP READING SCORES

Washington, D.C., October 19, 2005 – According to the 2005 “Nation’s Report Card” released by the U.S. Department of Education today, Minnesota’s 8th grade students’ reading scores average the same as those of the 8th graders who took the test two years ago. Governor Bob Wise, president of the Alliance for Excellent Education, observed:

“Reading is the heart of learning, and Minnesota and the nation are in the literacy emergency room showing a flat line on the education EKG. The state’s results clearly demonstrate that we still are not doing what is needed to help these older students build the reading skills they will need to deal with increasingly complex high school courses. Twenty percent of Minnesota’s 8th grade students are reading significantly below grade level and they are most likely to drop out of high school or graduate without the skills needed to succeed in college or the workplace.

“For the most part, we stop teaching our children how to read when they leave third grade, and expect that they’ll continue to expand vocabulary and comprehension skills on their own. That’s like a builder laying the foundation of a house and leaving the buyer to put up the walls and roof without help. The investments made in early grades to teach our kids to read are critical, but we must continue to intervene throughout their school years to assure that they are maintaining and expanding the literacy skills that are so necessary for success in life.”

The National Assessment of Educational Progress (NAEP), also known as "the Nation's Report Card," is the only nationally representative and continuing assessment of what America's students know and can do in various subject areas. Today, results of the 2005 reading and math assessments were released, with national and state-level scores for students in grades 4 and 8.

Wise makes sense, right? Especially if like I do you agree with his analysis. Reading is not emphasized nearly enough in upper grades, nor are reading assignments nearly challenging enough. However, Wise doesn’t provide enough data to statistically support his claim that test scores have not improved.

Simpson’s Paradox is a statistical anomaly that posits when a large data set is made of smaller subsets, each subset can show improvement even when the large data set show a decline or flat-line improvement. The reason has to do with weighting of the subgroups.

For example, white, upper class students tend to score higher on reading tests than minorities. If year-over-year more minority students are added to the mix, it is possible, even likely, that both white student and minority student scores can improve, but because the total population is now weighted more to the lower scoring group, the overall improvement line is flat.

So while I agree with Wise’s analysis, in fairness, the point must be made that statistically, he doesn’t make his case. That's how professional communicators work. We know stuff; when we don't, we admit it, and we do our homework.