"I am not now and have never been a constructionist"

(This post also appears at Observational Epidemiology.)

After my last post thought I should run this titular disclaimer. For those of you not up on the subject, here's a definition from the well-written Wikipedia entry on the subject:

Constructivist teaching is based on constructivist learning theory. This theoretical framework holds that learning always builds upon knowledge that a student already knows; this prior knowledge is called a schema. Because all learning is filtered through pre-existing schemata, constructivists suggest that learning is more effective when a student is actively engaged in the learning process rather than attempting to receive knowledge passively. A wide variety of methods claim to be based on constructivist learning theory. Most of these methods rely on some form of guided discovery where the teacher avoids most direct instruction and attempts to lead the student through questions and activities to discover, discuss, appreciate and verbalize the new knowledge.

Don't get me wrong. For the right topic, executed the right way with the right teacher and class, this can be a great, wonderful, spectacular and really good approach to education. Unfortunately, education reformers (particularly the current crop), are not good at conditional problems. They tend instead to fall into the new tool camp (you know the saying, "to a man with a new hammer, the whole world is a nail.").

Worse yet, (and I'm afraid there's no nice way to say this) many of the educational theorists don't have a firm grasp on the subjects they are working with. This is never more plain than in constructionist science classes that almost entirely eschew lectures and traditional reading assignments and instead have the students spend their time conducting paint-by-numbers experiments, recording the results and performing a few simple calculations.

To most laymen, that's what science is: stuff you do while wearing a lab coat. Most people don't associate science with forming hypotheses, designing experiments, analyzing results and writing papers and, based on my limited but first hand experience, many science educators don't give those things much thought either.

The shining exception to the those-who-can't-teach-teach-teachers rule is George Polya. Though best known as an educational theorist, Polya was a major Twentieth Century mathematician (among his other claims to fame, he coined the term "central limit theorem") so he certainly fell in the those-who-can camp.

But it it important to note that Polya advocated guided discovery specifically as a way of teaching the problem solving process. I suspect that when it came to simply acquiring information, he would have told his students to go home and read their textbooks.

Reasons to teach what we teach

[note: this is a math-centric post but most of the concepts can, on some level, be generalized to other subjects]

There's a curiously inverted quality to the education debate. We spend a great deal of time discussing revolutionary changes to the educational system and almost no time talking about what we should be teaching, as if the proper combination of reforms and incentives can somehow overcome the rule of garbage in, garbage out.

I spent a lot of my time as a teacher thinking about which parts of the mathematics curriculum were good and which parts were garbage and I came up with a list of reasons why a topic might be worth the student's time. The list isn't in order (I'm not sure it's even orderable) but it is meant to be comprehensive -- everything that belongs in the curriculum should qualify under one or (generally) more of these criteria.


1. Students are likely to need frequent and immediate access to this for jobs and daily life.

and

2. Students are likely to need to know how to find this (Samuel Johnson level knowledge).

(These are the only two mutually exclusive reasons on the list.)


3. This illustrates an important mathematical concept


4. This helps develop transferable skills in reasoning, pattern-recognition and problem solving skills


5. Students need to know this in order to understand an upcoming lesson


6. A culturally literate person needs to know this

Most topics can be justified under multiple reasons. Some, like the Pythagorean Theorem can be justified under any of the six (though not, of course, under one and two simultaneously).

Where a topic appears on this list affects the way it should be taught and tested. Memorizing algorithms is an entirely appropriate approach to problems that fall primarily under number one. Take long division. We would like it if all our students understood the underlying concepts behind each step but we'll settle for all of them being able to get the right answer.

If, however, a problem falls primarily under four, this same approach is disastrous. One of my favorite examples of this comes from a high school GT text that was supposed to develop logic skills. The lesson was built around those puzzles where you have to reason out which traits go with which person (the man in the red house owns a dog, drives a Lincoln and smokes Camels -- back when people in puzzles smoked). These puzzles require some surprisingly advanced problem solving techniques but they really can be enjoyable, as demonstrated by the millions of people who have done them just for fun. (as an added bonus, problems very similar to this frequently appear on the SAT.)

The trick to doing these puzzles is figuring out an effective way of diagramming the conditions and, of course, this ability (graphically depicting information) is absolutely vital for most high level problem solving. Even though the problem itself was trivial, the skill required to find the right approach to solve it was readily transferable to any number of high value areas. The key to teaching this type of lesson is to provide as little guidance as possible while still keeping the frustration level manageable (one way to do this is to let the students work in groups or do the problem as a class, limiting the teacher's participation to leading questions and vague hints).

What you don't want to do is spell everything out and that was, unfortunately, the exact approach the book took. It presented the students with a step-by-step guide to solving this specific kind of logic problem, even providing out the ready-to-fill-in chart. It was like taking the students to the gym then lifting the weights for them.

Long division and logic puzzles are, of course, extreme cases, but the same issues show up across the curriculum. Take factoring trinomials. A friend and former boss of mine wrote a successful college algebra text book that omitted the topic entirely. I had mixed feelings about the decision but I understood his reasoning: this is one of those things you will almost certainly never have to do outside of a math class (what fraction of trinomials are even factorable?).

You can justify teaching the factoring of trinomials because it illustrates important mathematical concepts and because it gives students practice manipulating algebraic expressions, but the way you teach this concept has got to reflect the reasons for teaching it. Having students memorize a step-by-step algorithm would be the easiest way to teach the students to answer these questions (and improve their standardized test scores) but it completely miss the point of the lesson.

The point about standardized test scores is significant and needs to be revisited a post of its own. By evaluating teachers and schools on standardized test scores, we put pressure on teachers to treat all subjects as if they fell solely under reason one. This is not a good outcome.

Even more important than how we should teach something is the question of what we should be teaching. Current curricula tend to be broad and shallow with a tragic evenhandedness that often grants the same amount of time to trivial techniques as it does to fundamental concepts. This is bad enough when a class on grade level and everything is going well but it's disastrous when a large part of the class is struggling. There is tremendous pressure under those circumstances to leave the stragglers behind (a pressure that actually increases under many proposed reforms).

In addition to being overstuffed, the current curriculum omits subjects that are arguably more important than most of what we cover. The obvious example here is statistics, a topic that everyone actually does need on a daily basis (as informed citizens and consumers if nothing else). Perhaps even more relevant is what we might call spreadsheet math (customized worksheets, recursive functions, graphs, macro programming). You could also make a case for discrete mathematics, particularly graph theory (I might even put this one up there with statistics and spreadsheets but that's a subject for another post).

"Counseling out"

(This post also appears at Observational Epidemiology.)

Paul Tough writing in Slate recounts the following:

In Whatever It Takes, in one of the chapters on the Promise Academy middle school, I describe the impact of the KIPP schools in the Bronx and Harlem on the Promise Academy’s leaders and staff. This was during the first few years of the Harlem Children Zone’s middle school, which were a struggle, and those KIPP schools, which had very good test results, were for the Promise Academy administrators both a standard to be aspired to and a frustrating reminder that their own students weren’t performing at the same high level as KIPP’s students.

Terri Grey, the Promise Academy principal at the time, believed the attrition issue was part of what was holding her school back. As she put it to me in one conversation, “At most charter schools, if the school is not a good fit for their child, the school finds a way to counsel parents out”—to firmly suggest, in other words, that their child might be happier elsewhere. “Whereas Promise Academy is taking the most disengaged families and students and saying, ‘No, we want you, and we’re trying to keep you here, and we don’t want to counsel you out.” That policy made it impossible, she believed, for the Promise Academy to achieve KIPP-like results.

I’m not entirely convinced that that was the real problem at Promise Academy—or that the KIPP schools in New York were actually “counseling out” a significant number of students. But I do think it’s true that Geoffrey Canada’s guiding ethic has always been to go out of his way to attract and retain the most troubled parents and students. And that makes running a school, or any program, more difficult, even if it makes the mission purer and, in the end, more important.

For reasons I'll get to later, I suspect that the number of students you have to "counsel out" to have a significant effect on a school's test scores is lower than Mr. Tough realizes, but there are a couple of more important points.

The first is that selective attrition is recognized as a serious issue not just by critics of the reform movement but by responsible people within the charter school community.

The second is that all charter schools and charter school administrators are not interchangeable. There are some gifted educators with great ideas in that system. We've spent almost two decades overlooking the flaws in charter schools. It would be a serious mistake to try to compensate by overlooking the strengths.

"they are purging nonperforming students at an alarming rate"

(This post also appears at Education and Statistics.)

Mike at ScienceBlogs has some thoughts about selection by attrition:

A letter to Diane Ravitch from a Los Angeles school prinicipal documents just how dishonest and harmful this practice is (italics [Mike's]):

I received an email from Dr. DeWayne Davis, the principal of Audubon Middle School in Los Angeles, which was sent to several public officials. Dr. Davis said that local charter schools were sending their low-performing students to his school in the middle of the year. He wrote:

"Since school began, we enrolled 159 new students (grades 7 and 8). Of the 159 new students, 147 of them are far below basic (FBB)!!! Of the 147 students who are FBB, 142 are from charter schools. It is ridiculous that they can pick and choose kids and pretend that they are raising scores when, in fact, they are purging nonperforming students at an alarming rate--that is how they are raising their scores, not by improving the performance of students. Such a large number of FBB students will handicap the growth that the Audubon staff initiated this year, and further, will negatively impact the school's overall scores as we continue to receive a recurring tide of low-performing students."

Ravitch concludes:

Doing better than an under-resourced neighborhood school is not the same as getting "amazing results." Very few charters do. Probably less than 5 percent. Charters are not a silver bullet. They are a lead bullet. Their target is American public education.

This is just par for the course for modern conservatism: have private systems skim the cream, and leave the public sector to clean up an impossible mess. When they can't, this supposedly shows the inability of government to solve problems.

I have a few points to add:

1. This is a brutal way to treat these kids. You build their hopes up, then crush them, then dump the kids in a new school in the middle of the year;

2. We are talking about getting an influx of students who are badly behind and who are ready to give up and/or act out. This will disrupt classes slightly less than having a nearby car alarm go off at random times once or twice an hour;

3. But I think Ravitch overstates the case against charter schools. I've dealt with some small, independent schools that have impressed the hell out of me and I can see them playing an important role in our system, though a radically different role than Arne Duncan sees.

Topologists at play -- the game of Sprouts

It's important to have students think deeply about math in both structured and unstructured ways (I have a guilty feeling that I ought to say more about this, but that will have to wait for a future post). It's the unstructured part that tends to cause problems. That's one of the reasons I liked to make games part of my lessons when I was a teacher.

Games (at least the kind I recommend) require a great deal of focus -- you have to think about what you're doing or you won't do well -- and they encourage exploration and a playful attitude to the material. All of these things help build mathematical intuition.

On the subject of topology, my game of choice is Sprouts, invented by mathematicians John Horton Conway and Michael S. Paterson at Cambridge University in 1967 (as a general rule, you can't go wrong with a game if Conway had anything to do with it).

The rules are simple:

1. Start with some dots on the paper. The more dots you have the longer the game takes so you will probably just want to start with two or three.

2. Players take turns either connecting two of the dots with lines or drawing a line that loops back and connects a dot with itself.

3. The lines can be straight or curved but they can’t cross themselves or any other lines.

4. Each dot can have at most three lines connecting it.

5. When you draw a line put a new dot in the middle.

6. The first player who can’t draw a line loses.

You can find a couple of sample games here.

Benoît Mandelbrot (1924 - 2010) and Education Reform

From A maverick's apprenticeship:

It was then and there that a gift was revealed. During high school and the wandering year and a half that followed, I became intimately familiar with a myriad of geometric shapes that I could instantly identify when even a hint of their presence occurred in a problem. “Le Père Coissard,” our marvelous mathematics professor, would read a list of questions in algebra and analytic geometry. I was not only listening to him but also to another voice. Having made a drawing, I nearly always felt that it missed something, was aesthetically incomplete. For example, it would improve by some projection or inversion with respect to some circle. After a few transformations of this sort, almost every shape became more harmonious. The Ancient Greeks would have called the new shape “symmetric” and in due time symmetry was to become central to my work. Completing this playful activity made impossibly difficult problems become obvious by inspection. The needed algebra could always be filled in later. I could also evaluate complicated integrals by relating them to familiar shapes.

I was cheating but my strange performance never broke any written rule. Everyone else was training towards speed and accuracy in algebra and reduction of complicated integrals; I managed to be examined on the basis of speed and good taste in translating algebra back into geometry and then thinking in terms of geometrical shapes.

Where did my gift come from? One cannot unscramble nature from nurture but there are clues. My uncle lived a double life as weekday mathematician and Sunday painter. My gift for shapes might have been destroyed, were it not for the unplanned complication of my life during childhood and the War. Becoming more fluent at manipulating formulas might have harmed this gift. And the absence of regular schooling influenced many life choices, but ended up not as a handicap but as a boon.

I realize we can't have an educational system focused entirely on the occasional Mandelbrot, but I can't help but wonder how the great man would have fared in the rigid, metric-driven system we're headed toward.

(also posted at Observational Epidemiology)

Wolfmeat -- games edition

(As I mentioned before, games and puzzles have always been a big part of my approach to teaching, partially because they helped make school interesting but also because they often conceal some extraordinarily sophisticated mathematical concepts. The following is a starter set of classroom games I put together a few years ago.)

Czarist-era Russians who had to take long trips by sled, particularly at night, would often gather up several large chunks of meat in a sack before starting out. If the sled happened across a pack of wolves, the driver would throw out the meat a piece at a time in the hope that the wolves would stop for a few moments to fight over the food.

Even the best teachers will have a Russian sled moment now and then, when the wolves are circling your desk and searching diligently for your last nerve, so it's always a good idea to keep a few sacks of wolfmeat on hand just in case.

Hex – Probably my first choice for a "here, do this" moment. A fast, simple strategy game with a great pedigree. You can easily fit two hex boards on one side of a sheet of letter paper.

Checkers – Don't disrespect the lowly checker. Players who have mastered both chess and checkers often argue that checkers is the more challenging game, particularly the Spanish version with long jumps. A few cheap chess/checkers sets are a great classroom investment.

Chess – There are two contenders for the world's most popular game, Chess and Go, but only one is available for five dollars at your local discount store.

Chess Board Games – A chessboard is probably the most versatile playing surface ever invented. There are countless games that can be played on all or part of a chessboard. Here are a few good ones to start with:
Dodgem
Nine Hole
Chomp

Pencil and Paper Games – The only thing wrong with pencil and paper games is that people usually play the wrong one. Tic Tac Toe is the least interesting member of a distinguished family of row games (click here, here, and here for a few examples) many of which can be played with pencil and paper. In addition to row games there are Nim, Tac Tix, Dots and Boxes, Sprouts, Hangman and the very entertaining Racetrack.

Teaching across the Curriculum -- Yes, it's a buzzword, but as buzzwords go it's not bad. Here's a table to help you get started.

The Secret of Doublets

I've always been a big fan of games and puzzles as teaching tools. They develop logical thinking and problem solving, they remind us that learning is supposed to be an enjoyable activity, and more often than not, they have a dirty, little secret.

Doublets are a great example. Developed by Charles Dodgson (writing as Lewis Carroll), they were, for a while, the rage of the Victorian party scene. The rules were elegantly simple: take two words with the same number of letters; change the the first word to the second one letter at a time with the condition that each transition is also a word (think Scrabble rules -- no slang, no proper names). Though not required, the two words would usually have some logical connection.

Scores are determined by how many steps it takes. As in golf, low score wins.

Here's how a doublet player might go from FOOT to BALL:

FOOT

FOOL

TOOL

TOLL

TALL

BALL

Of course, FOOT BALL is an easy one. the doublets Carroll created tended to be far more challenging. Here are some examples from Carroll's Doublets: a word puzzle (available in cut-and-paste friendly plain text here)

Change OAT to RYE.
Get WOOD from TREE.
Prove GRASS to be GREEN.
Change CAIN into ABEL.
Make FLOUR into BREAD.
Evolve MAN from APE.

Now we get to the secret of doublets.

After you've used them as time fillers at the end of class and handed out the puzzle sheets and maybe even given some bonus points to the first student to solve a particularly challenging example, only then do you reveal the dirty little secret:

It's mathematics.

Specifically, it's graph theory.

That's right, you've tricked all of those poor, innocent kids into doing math and, worse yet, thinking they enjoyed it. You've introduced a sophisticated mathematical concept, reinforced it with a memorable example and laid the groundwork for future lessons.

Naturally, the patron saint of math teachers, Martin Gardner, was here first.

Teacher's Bookshelf -- The Play of Words

Perhaps more than any other subject, English should be playful. If you can't enjoy playing with words in exactly the same way a that a small child plays with a box of toys (or with words through rhymes and puns and silly strings of syllables), then you have largely missed the point.

That makes books like Richard Lederer's The Play of Words useful verging on essential. This book, a Xerox machine and a loose reading of intellectual property laws has gotten me through countless classes and has made those classes a bit more bearable for the students.

Lederer explores metaphors, clichés, rhyme and alliteration, etymologies and logic puzzles in a way that's thoughtful and addictive. I'll admit there's an art to introducing word games with names like 'inky pinkies' to a class of inner city high school students (never, at any time, use the word 'fun'), but it can be done and it's well worth the effort.

"Shape of the earth -- opinions still differ"

(The following post originally appeared in Observational Epidemiology.)

I was reminded of Paul Krugman's parody of a New York Times headline when I came to this NYT headline:

"Despite Push, Success at Charter Schools Is Mixed By TRIP GABRIEL"

Followed a few paragraphs later by the money shot:

But for all their support and cultural cachet, the majority of the 5,000 or so charter schools nationwide appear to be no better, and in many cases worse, than local public schools when measured by achievement on standardized tests, according to experts citing years of research. Last year one of the most comprehensive studies, by researchers from Stanford University, found that fewer than one-fifth of charter schools nationally offered a better education than comparable local schools, almost half offered an equivalent education and more than a third, 37 percent, were “significantly worse.”

Although “charter schools have become a rallying cry for education reformers,” the report, by the Center for Research on Education Outcomes, warned, “this study reveals in unmistakable terms that, in the aggregate, charter students are not faring as well” as students in traditional schools.

As I mentioned before, there is reason to believe that this research is biased in favor of charter schools.

If you showed me test results for a new cholesterol-controlling drug in which 20% of the subjects had lower LDL levels than when they started taking the drugs, 51% stayed the same and 37% were "significantly worse," I don't think I would describe the results as 'mixed.'

But, of course, I'm not writing for the New York Times.

Career paths, educational reform and unintended consequences

(The following post originally appeared in Observational Epidemiology.)

This post by Joseph about career paths for researchers reminded me of some disturbing trends in career paths on the other side of academia, teaching.

In primary and secondary education, you can have a completely successful career by any conceivable standard and never get a single promotion. You can easily spend your entire professional life doing the same job with the same title. That might even be the ideal.

For a field requiring a degree, additional coursework and certification, this is an extraordinarily limited career path. To make up for that we have traditionally offered the following:

1. Reliable income that increases at an agreed-upon rate annually.

2. A high level of job security after a certain number of years (though this is somewhat offset by low job security before reaching tenure).

As well as creating a career path that didn't depend on promotion (and therefore largely avoided the Peter Principle), this emphasis on deferred, but steady compensation meant schools could minimize their investment in new, untested personnel (most really disastrous teachers do not make it to the tenure mark).

Two of the central tenets of the current move for education reform are elimination of tenure and replacing raises based on experience and education with merit pay. This leaves us with a career path that offers no real chance of promotion, no job security and wildly variable pay based on metrics that are largely out of the teacher's control and can easily be gamed by a biased administrator (see here).

Some context on schools and the magic of the markets

(The following post originally appeared in Observational Epidemiology.)

One reason emotions run so hot in the current debate is that the always heated controversies of education have somehow become intertwined with sensitive points of economic philosophy. The discussion over child welfare and opportunity has been rewritten as an epic struggle between big government and unions on one hand and markets and entrepreneurs on the other. (insert Lord of the Rings reference here)

When Ben Wildavsky said "Perhaps most striking to me as I read Death and Life was Ravitch’s odd aversion to, even contempt for, market economics and business as they relate to education" he wasn't wasting his time on a minor aspect of the book; he was focusing on the fundamental principle of the debate.

The success or even the applicability of business metrics and mission statements in education is a topic for another post, but the subject does remind me of a presentation the head of the education department gave when I was getting my certification in the late Eighties. He showed us a video of Tom Peter's discussing In Search of Excellence then spent about an hour extolling Peters ideas.

(on a related note, I don't recall any of my education classes mentioning George Polya)

I can't say exactly when but by 1987 business-based approaches were the big thing in education and had been for quite a while, a movement that led to the introduction of charter schools at the end of the decade. And the movement has continued to this day.

In other words, American schools have been trying a free market/business school approach for between twenty-five and thirty years.

I'm not going to say anything here about the success or failure of those efforts, but it is worth putting in context.

Harlem Children's Zero Sum Game

(The following post originally appeared in Observational Epidemiology.)

I used to work in the marketing side of large corporation (I don't think they'd like me to use their name so let's just say you've heard of it and leave the matter at that). We frequently discussed the dangers of adverse selection: the possibility that a marketing campaign might bring in customers we didn't want, particularly those we couldn't legally refuse. We also spent a lot of time talking about how to maximize the ratio of perceived value to real value.

On a completely unrelated note, here's an interesting article from the New York Times:

Pressed by Charters, Public Schools Try Marketing
By JENNIFER MEDINA

Rafaela Espinal held her first poolside chat last summer, offering cheese, crackers and apple cider to draw people to hear her pitch.

She keeps a handful of brochures in her purse, and also gives a few to her daughter before she leaves for school each morning. She painted signs on the windows of her Chrysler minivan, turning it into a mobile advertisement.

It is all an effort to build awareness for her product, which is not new, but is in need of an image makeover: a public school in Harlem.

As charter schools have grown around the country, both in number and in popularity, public school principals like Ms. Espinal are being forced to compete for bodies or risk having their schools closed. So among their many challenges, some of these principals, who had never given much thought to attracting students, have been spending considerable time toiling over ways to market their schools. They are revamping school logos, encouraging students and teachers to wear T-shirts emblazoned with the new designs. They emphasize their after-school programs as an alternative to the extended days at many charter schools. A few have worked with professional marketing firms to create sophisticated Web sites and blogs.
...

For most schools, the marketing amounts to less than $500, raised by parents and teachers to print up full color postcards or brochures. Typically, principals rely on staff members with a creative bent to draw up whatever they can.

Student recruitment has always been necessary for charter schools, which are privately run but receive public money based on their enrollment, supplemented by whatever private donations they can corral.

The Harlem Success Academy network, run by the former City Council member Eva Moskowitz, is widely regarded, with admiration by some and scorn by others, as having the biggest marketing effort. Their bright orange advertisements pepper the bus stops in the neighborhood, and prospective parents receive full color mailings almost monthly.

Ms. Moskowitz said the extensive outreach was necessary to make sure they were drawing from a broad spectrum of parents. Ms. Moskowitz said they spent roughly $90 per applicant for recruitment. With about 3,600 applicants last year for the four schools in the network, she said, the total amounted to $325,000.

Charter schools, social norming and zero-sum games

(The following post originally appeared in Observational Epidemiology.)

You've probably heard about the Harlem Children's Zone, an impressive, even inspiring initiative to improve the lives of poor inner-city children through charter schools and community programs. Having taught in Watts and the Mississippi Delta in my pre-statistician days, this is an area of long-standing interest to me and I like a lot of the things I'm hearing about HCZ. What I don't like nearly as much is the reaction I'm seeing to the research study by Will Dobbie and Roland G. Fryer, Jr. of Harvard. Here's Alex Tabarrok at Marginal Revolution with a representative sample, "I don't know why anyone interested in the welfare of children would want to discourage this kind of experimentation."

Maybe I can provide some reasons.

First off, this is an observational study, not a randomized experiment. I think we may be reaching the limits of what analysis of observational data can do in the education debate and, given the importance and complexity of the questions, I don't understand why we aren't employing randomized trials to answer some of these questions once and for all.

More significantly I'm also troubled by the aliasing of data on the Promise Academies and by the fact that the authors draw a conclusion ("HCZ is enormously successful at boosting achievement in math and ELA in elementary school and math in middle school. The impact of being offered admission into the HCZ middle school on ELA achievement is positive, but less dramatic. High-quality schools or community investments coupled with high-quality schools drive these results, but community investments alone cannot.") that the data can't support.

In statistics, aliasing means combining treatments in such a way that you can't tell which treatment or combination of treatments caused the effect you observed. In this case the first treatment is the educational environment of the Promise Academies. The second is something called social norming.

When you isolate a group of students, they will quickly arrive at a consensus of what constitutes normal behavior. It is a complex and somewhat unpredictable process driven by personalities and random connections and any number of outside factors. You can however, exercise a great deal of control over the outcome by restricting the make-up of the group.

If we restricted students via an application process, how would we expect that group to differ from the general population and how would that affect the norms the group would settle on? For starters, all the parents would have taken a direct interest in their children's schooling.

Compared to the general population, the applicants will be much more likely to see working hard, making good grades, not getting into trouble as normal behaviors. The applicants (particularly older applicants) would be more likely to be interested in school and to see academic and professional success as a reasonable possibility because they would have made an active choice to move to a new and more demanding school. Having the older students committed to the program is particularly important because older children play a disproportionate role in the setting of social norms.

Dobbie and Fryer address the question of self-selection, "[R]esults from any lottery sample may lack external validity. The counterfactual we identify is for students who are already interested in charter schools. The effect of being offered admission to HCZ for these students may be different than for other types of students." In other words, they can't conclude from the data how well students would do at the Promise Academies if, for instance, their parents weren't engaged and supportive (a group effective eliminated by the application process).

But there's another question, one with tremendous policy implications, that the paper does not address: how well would the students who were accepted to HCZ have done if they were given the same amount of instruction * as they would have received from HCZ using public school teachers while being isolated from the general population? (There was a control group of lottery losers but there is no evidence that they were kept together as a group.)

Why is this question so important? Because we are thinking about spending an enormous amount of time, effort and money on a major overhaul of the education system when we don't have the data to tell us if what we'll spend will wasted or, worse yet, if we are to some extent playing a zero sum game.

Social norming can work both ways. If you remove all of the students whose parents are willing and able to go through the application process, the norms of acceptable behavior for those left behind will move in an ugly direction and the kids who started out with the greatest disadvantages would be left to bear the burden.

But we can answer these questions and make decisions based on solid, statistically sound data. Educational reform is not like climate change where observational data is our only reasonable option. Randomized trials are an option in most cases; they are not that difficult or expensive.

Until we get good data, how can we expect to make good decisions?

* Correction: There should have been a link here to this post by Andrew Gelman.

Perils of Convergence

(The following post originally appeared in Observational Epidemiology.)

This article ("Building the Better Teacher") in the New York Times Magazine is generating a lot of blog posts about education reform and talk of education reform always makes me deeply nervous. Part of the anxiety comes having spent a number of years behind the podium and having seen the disparity between the claims and the reality of previous reforms. The rest comes from being a statistician and knowing what things like convergence can do to data.

Convergent behavior violates the assumption of independent observations used in most simple analyses, but educational studies commonly, perhaps even routinely ignore the complex ways that social norming can cause the nesting of student performance data.

In other words, educational research is often based of the idea that teenagers do not respond to peer pressure.

Since most teenagers are looking for someone else to take the lead, social norming can be extremely sensitive to small changes in initial conditions, particularly in the make-up of the group. This makes it easy for administrators to play favorites -- when a disruptive or under-performing student is reassigned from a favored to an unfavored teacher, the student lowers the average of the second class and often resets the standards of normal behavior for his or her peers.

If we were to adopt the proposed Jack-Welch model (big financial incentitves at the top; pink slips at the bottom), an administrator could, just by moving three or four students, arrange for one teacher to be put in line for for achievement bonuses while another teacher of equal ability would be in danger of dismissal.

Worse yet, social norming can greatly magnify the bias caused by self-selection and self-selection biases are rampant in educational research. Any kind of application process automatically removes almost all of the students that either don't want to go to school or aren't interested in academic achievement or know that their parents won't care what they do.

If you can get a class consisting entirely of ambitious, engaged students with supportive parents, social norming is your best friend. These classes are almost (but not quite) idiot proof and teachers lucky enough to have these classes will see their metrics go through the roof (and their stress levels plummet -- those are fun classes to teach). If you can get an entire school filled with these students, the effect will be even stronger.

This effect is often stated in terms of the difference in performance between the charter schools and the schools the charter students were drawn from which adds another level of bias (not to mention insult to injury).

Ethically, this raises a number of tough questions about our obligations to all students (even the difficult and at-risk) and what kind of sacrifices we can reasonably ask most students to make for a few of their peers.

Statistically, though, the situation is remarkably clear: if this effect is present in a study and is not accounted for, the results are at best questionable and at worst meaningless.

(this is the first in a series of posts about education. Later this week, I'll take a look at the errors in the influential paper on Harlem's Promise Academy.)

Just what the world needs, another blog

But here I go anyway.

If you're coming here via Observational Epidemiology, a lot of these posts will look familiar (particularly as this is getting started), one of the reasons behind Education and Statistics is to archive some posts on the topic in a site with a much less scary name.

But there will E&S-only posts so even if you're a regular OE reader, drop by here now and then.

Welcome to the blog,
Mark