How my school gamed the stats

I was reading the Slate Star Codex review of The Cult of Smart. Part of it discusses charter vs state schools and the allegations of fraud of various kinds undermining charter schools record of better achievement. Reading it, I realized that I took for granted that public schools engage in systematic fraud in a variety of ways. I don’t think this is something everyone understands, hence this post.

I went to a state school in the UK. State schools are rated on a 1 – 4 scale from unsatisfactory to outstanding. My school was rated good, meaning a 3. A few memories which stand out. During my first week I saw one of the boys in my class who was 11 at the time held up against the wall in a corridor while a 16 year old put a shiv to his throat and robbed him. He handed over his wallet and keys. A year or two later and I remember seeing a small boy who struggled with depression held up by the throat against a locker and slapped in the face by a troublemaker from the same class in front of everyone just before we went in to the classroom. I remember classes which were filled start to finish with people shouting and talking. Neither of the first two events were common but they also weren’t uncommon. No one was surprised to witness them. It’s worth emphasizing again that my school was above average, in fact quite far above average, and in a middle class area. It’s also worth noting that I was mostly in top ability streamed classes, meaning my classroom experience was likely far better than average.

There were many ways in which the school and teachers gamed the system to boost their measured performance. One way was to do exams for students. I was on a bottom set language class for French. After two years I literally couldn’t speak a single sentence in french and maybe knew 20 words in total. I still passed my exams. How? We did the tests in class. Often the teacher would go through them with us. Literally giving us the test and then going through each question on the whiteboard and telling us what to write. A different year and a different teacher, this time the teacher would sit next to us and write the answers down. Why sit next to us? It was the bottom set so people often wouldn’t even bother to write down the answer if they were told it. This kind of thing was normal, so much so that I, and I think most people there, didn’t realize anything unusual was happening.

Another way schools game metrics is to cheat inspections. A major component of how schools are judged in the UK is through independent inspections carried out by an independent quasi-governmental organization called Ofsted. Now, you may imagine that these inspections would be unannounced, so as to best get a real image of how a school works. Not the case. They’re scheduled well in advance. Before every inspection, a few things would happen in my school:

  • The worst troublemaker kids would be taken aside and put in a special room where inspectors wouldn’t see them. Either that or they would just be told not to come into school at all on that day.
  • All of us were told in assembly that an inspection was coming and to be on our best behavior on that day. Often teachers would have conversations with less serious troublemakers and impress on them that they would behave on that day or face consequences afterwards.
  • Teachers would put a great deal more effort into their lesson plans than was normal. Classroom behavior management would also be far stricter. Because of these and other measures my school during an inspection was utterly different than my school on a normal day. On some level this isn’t surprising. If teachers’ promotions and management’s jobs depend on good inspection results and inspections are easy to game, people will game them. Incentives drive behavior. But it’s still sad.

Another way the stats were gamed was by not recording bad behavior. When a school gives a detention or suspends/expels a student, there’s a record of it. This is especially true of suspensions, students being sent home or expulsions. The more of these you have, the worse you look as a school. The solution then is obvious, don’t punish people or punish them in non-recorded ways. Again, in my school it was completely normal for students in lower sets to swear at the teacher, talk over them or disrupt the class for everyone else. It was normal for someone to be aggressive and abusive towards others and to face at most a 40 minute detention, but even getting a detention would be unusual.

I realize that one data point is not enough to draw solid general conclusions. My own perception is that this kind of fraud wasn’t specific to my school. My cousin went to a state school fairly nearby. He’s 4 years younger than me. During one of my winters back from undergrad we discussed his school and his experiences mirrored mine. His exact words regarding inspections were "I learned 4 times more that day than any other day that year. It was amazing". I talked to a few British students at university, although specifically the not middle/upper class ones who would have gone to public schools. They had gone to schools similar to mine in different parts of the country and their stories were similar and often worse. Two particularly funny examples from my friends’ experiences stand out. A teacher in year 9 walked up to a student who was talking, picked them up and threw them out of an (open) first floor window. My friend sitting in class noticed two boys making fun of him and then proceeded to get up in the middle of class while the teacher was talking, walk to their table, flip the table upwards to hit them in the face before going to sit down again when the teacher told him to. (Remember, my friend was a studious, sporty Asian kid and not a troublemaker. This kind of thing is normal in that environment). Comedic stories aside, my experiences in school, while not universal, seem fairly common in the UK and from what I’ve read of the statistics, bad US schools are far, far worse.

I’m unsure what my point here is. I think I have two:

  • Charters may cook their books in various ways. In the UK, State schools do too. I would be surprised if it wasn’t also the case in the US.
  • I think that I feel like a lot of commentators on places like SSC have fairly middle class experiences of fairly good schools and that bleeds into how their comparison between state vs charter schools. It’s just good to remember that it’s not those nice middle class schools that charters typically replace.

2020 in Review

N.B: This is my annual retrospective post. It contains no special insights and is 100% skippable.

Summary

In terms of my material life, this has been a good year. My social skills continue to advance. My career continues to improve. My relationship with my girlfriend has also progressed. My physical health has also substantially improved.

Intellectually, the year has been poor. I’ve written far less than in previous years. I’ve read far fewer books and slightly more articles as a result of a gradually expanding RSS feed. I’ve also picked up a few good new podcasts, Ideas of India being the best.

Writing

Total posts this year: 13 (-26 vs 2019)

Decent Posts

Looking back, I think my writing voice is improving. Writing has also become easier since I’ve started using Obsidian, which is a non-hierarchical, bidirectional knowledge graph like system similar to Roam Research. My quantity of writing has decreased markedly. I think the main cause of this has been less time spent commuting. Whereas before I would write or read on the train, now I do not.

Reading/Listening

Total Books Read This Year: 5 (-21 vs 2018)

I continue to read three serials: A Practical Guide To Evil, Pale and Delve.

Books:

Money/Career

My financial situation has improved markedly. I began the year making £3.77k net per month. I ended it making £3.94k plus £0.5k – £5k of equity, depending on how the firm does. My saving rate has risen from 43% to 53% and my overall wealth has risen from £25k to roughly £54k. I currently have 26 months of runway.

My career is going well. I’ve learned a great deal about backend development and picked up or refined a few specific technologies and languages. My social skills have also continued to improve. I can increasingly pass for normal in most work interactions.

Misc

I began tracking macros for everything I eat. I’ve done this intermittently in the past, but this year I’ve done it more often and for longer periods of time. I’ve found it highly effective, both in improving the quality of the food I eat and making me aware of when I’m overeating.

Large Gains from Small Choices

Looking at my average day, I often think that I’m lazy and wasting most of my time. Usually I work for 8 hours, although probably only half of that time is productive. I go on a short walk, buy food and eat. After work, I mostly play videogames, do household chores and sometimes spend an hour or two talking to my girlfriend. I’m by no means a paragon of productivity. Still there are a few small things I do regularly that over a decade have compounded into substantial gains. These include:

  • Reading a good book a month (or trying to)
  • Using an RSS feed to stream high quality writing to myself daily
  • Subscribing to various good podcasts and listening to them as I go on walks. Usually summing up to 30 mins per day.
  • Going for a walk a day and doing weightlifting at least twice a week for 30 mins or so.

The more I look back, the more I think that being productive is not a matter of using every minute well or being busy all or even most of the time. Rather, I think productivity is a matter of gradually building high-impact, low-cost habits into your days. These habits don’t lead to magic changes in the short term but the small differences they do make compound over the days and years. After a decade the difference between the person with good habits and the person without them is night and day.

I don’t have anything more than plausible sounding arguments to prove this, but I think playing with a few numbers can help.

Let’s say you commit to studying one chapter of a textbook per week. I’m currently working through Modern Principles of Economics 4th Edition. It has 38 chapters. I think that’s a bit above average but let’s be conservative and assume that the average textbook has 40 chapters. Let’s also assume that you miss a few weeks a year. Instead of studying 52 chapters per year, you study 40. One year, one textbook. Over the course of a single decade you could get a grounding (a.k.a: know more than 95% of people) in economics, world history, criminology, law, philosophy, warfare, agriculture and four more topics of your choice. Would you be an expert in these topics? No. But the relevant comparison is not to the person who knows more than you but to the hypothetical alternate you who didn’t study a chapter a day and didn’t know about the green revolution in food production or that high housing prices are a result of constrained supply and increasing demand. It’s the difference between being oblivious and being able to recognize obviously wrong statements or at least somewhat understand the evidence for and against policy positions.

Let’s repeat the exercise for books. One person reads a book a quarter. Another reads a book a month. Over the course of a year the person who reads a book a month will have read 12 books. A difference of 9. Over the course of 10 years they will have read 120 books. A difference of 90. Assuming you’re choosing well and reading things that have meaning, think of the amount of knowledge those 90 books could contain. From a visions of 20 years in the gulag to a blow by blow account of the occupation of Iraq to the social dynamics of chimps to theories on technological stagnation. Whole new worlds opening to you, thousands of branching questions and thoughts. Just from a few hours a month.

These specific habits are just examples. There’s a large space of possible habits. I aim to write about a few I have found particularly useful eventually but don’t think that the habit’s I’ve stumbled on are particularly original. I think the most important point is not that habit X is effective but that self improvement is possible and really not that difficult, at least over the medium to long term.

Social media checks, bias and the presumption of innocence

A few weeks ago, stories started circulating about an IT consultant who would approach women on tinder and, after being rejected, insult them. I work at a series B fintech in London. At our weekly all-hands, I learned that a he used to work for my company. After talking about him being bad, the CEO talked about introducing stricter checks for new staff, including social media and news searches, to prevent someone like that from getting in.

Putting aside the fact that a policy like this wouldn’t do much, most bad people don’t have newspaper stories written about them, it’s interesting how quickly people were willing to destroy someones life. Of the people I talked to about this, most were left wing and yet almost none had stopped for a moment to think about the ethics of barring someone from holding a job on the basis of basis of a single article published in a right wing tabloid. It’s not that we had different views on what the burden of proof was before someone should be considered guilty/too-risky and silently denied a job at our firm, it’s that the question was never raised in the first place. Needless to say, this is wrong. As always [[A big part of ethics is getting people to admit their decision is a moral one]]

As startups grow, most start doing background on candidates at some point. There are a few risks that arise from this. The two most serious in my eyes are the increased scope for bias as the personal life/political views of a candidate leak more into the interview process and getting things wrong.

Getting people to want to address the problem of bias can be hard. Assuming people do want to address it, solving it is not that hard. Some basic process around the screening can help:

  • The people doing the screening should be different from those doing the interview
  • Screeners by default should not pass information about the candidates personal life, political beliefs or other bias vectors to interviewers/assessors
  • The things that candidates are being screened for should be well defined (e.g: criminal activity, threats of violence on social media, history of rape accusations etc…) rather than left vague. This is because, unfortunately, a significant proportion of people genuinely believe that the political tribe opposed to theirs is evil and even greater proportions have strong biases against those who are from their political outgroup. Hence leaving the screening criteria undefined will usually lead to those biases creeping in.

Mitigating the wrongness problem is harder. It’s harder because determining what is and is not true when all you have to go on are years old media articles or anonymous posts is difficult. It’s also hard because it requires that people do things that are morally right but bad for them personally. As a decision-maker, choosing to not hire someone is always safe. On the other hand hiring someone who is later revealed to have done bad things, or things other employees think are bad, can seriously damage your career. It’s because of this incentive structure that in the vast majority of situations people will advocate for not-hiring a person but no one will advocate for hiring them. I’m not sure how to solve the wrongness problem. My only two suggestions are:

  • Having a procedure for intentionally taking the accused’s side can help overcome the social stigma associated with defending a "bad" person. Just as we specifically assign lawyers to defend accused criminals in court, so as a hiring committee you can either choose one person or set aside 15 minutes as a group where you genuinely try to shoot holes in the case against a prospective candidate.
  • Rather than silently binning candidates, consider calling them and asking them about the allegations. It’s surprising how often media stories are later retracted or how common cases of mistaken identity can be.

Denying someone employment on ethical rather than economic grounds is a serious decision. It means that you’re leveraging your economic power to make political power, which is not something to be done lightly. If other firms do the same, as they often will since most large companies in a given sector tend to have similar norms and culture, your actions may make it impossible for these people to hold jobs. This kind of blacklisting is even more serious. If you’re going to do it, you should invest some time to make sure that you do it well.

Racism as a prisoners dlimma

Imagine you live in a society that is divided between three major ethnic groups. The groups help their own members, frequently fight over resources and there have been more than a few atrocities or borderline genocidal wars in the past with different groups using circumstances to kill/drive away their opponents. You are a member of one of these groups. Your personal fortune in significant part depends on that of your group. The more of the machinery of state your group controls, the more jobs and government provided goods and services you and your family can expect to receive. If another group dominates in the area you live, you can expect mild harassment your business being targeted for extortion/enforcement, the police being non-responsive to your claims and finding a job to be harder.

Now that the stage is set, let’s consider a question. Should you be a non-racist? Even assuming you have a liberal worldview, recognize that the ethnic division in your society is an illogical accident of history and recognize that there are good and bad people present in every group the answer is still not an obvious "yes". The problem is that if you stop favoring members of your group and fighting other groups, you weaken your group and comparatively strengthen others. If enough people in your group adopt your liberal norms, then your group starts to punch below it’s weight and will loose contests with other groups more and more, leading to worse and worse outcomes for it’s members. In effect, you’re trapped in a collective action problem. The costs of defecting from the existing (sub-optimal) racist equilibrium are private and fall upon the individual or group who defects. The benefits of moving towards a society free from ethnic division and strife flows to everyone. Two implications:

  • You do not want your group to be the first to adopt non-racism. The consequences of loss of group cohesion, especially in a developing nation, can be dire.
  • As a group, you have a strong incentive to punish individuals who do not contribute back to the group or challenge the legitimacy of the group as an organizing principle.

It’s a hard problem to resolve. When I did debating, I often saw people who had grown up in developed countries talk about genocide or ethnic conflict as a matter of people just being evil and the wrong beliefs. It’s not that simple. Ideology plays a part but so does the incentive structure you find yourself in. It’s easy to claim that people doing bad things are just moral aliens or brainwashed. The reality is usually more complex.

Why prostitution is immoral on the buy side

(TLDR: Because a significant % of prostitutes are forced/coerced into it, meaning when you sleep with a prostitute there’s a decent % chance you’re raping someone)

When I did debating, most arguments I would hear against prostitution were about the innate wrongness of it. Prostitution was bad because it’s morally wrong to trade sex. Prostitution was bad because it’s exploitative of sex workers as they have to engage in it to eat. Prostitution is bad because you can’t meaningfully consent to sex if money is involved etc… I never found these arguments persuasive. Logically, if you can consent to working other dangerous jobs such as coal mining or the military you can consent to the dangers/hardships of life as a prostitute. In terms of exploitation I think people should be allowed choose for themselves if they would rather be exploited or unemployed/employed in a shitty job doing manual labour. Still, my intuition was that there was still something wrong other people weren’t hitting on. In retrospect the reason why it’s wrong, at least from the buy side, is obvious. A decent percentage of prostitutes in first world countries are controlled by pimps, coerced, trafficked and generally can’t really say no. Hence purchasing sex from a prostitute means running a high risk of raping someone. Hence it’s wrong to buy sex from a prostitute.

Links 20/08/2020

Thoughts on cancel culture.

Let’s say there’s a society where people who say the wrong thing about topic X are killed by the secret police. Is this a society where people have freedom of speech in relation to topic X? The answer seems to be an obvious no.

Let’s say there’s a society where people who say the wrong thing about topic X are not killed or thrown in prison but instead loose their party membership and with it the ability to hold any high-status, well-paying or skilled job. Is this a society where people have freedom of speech in relation to X? Again the answer seems to be no.

Let’s say there’s a society where people who say the wrong thing about topic X face not retribution or action from the state. Instead they find that society seems them as being strange or immoral or holding the wrong views and consequently they are fired, struggle to find another job, can be denied education and that even their parents, spouse or family members can face similar consequences. Is this a society where people have freedom of speech in relation to X? For the third time the answer seems to be no.

My position on cancel culture is the same as my position on most forms of social coercion of individuals who’s beliefs or behavior falls outside of social norms. It’s wrong by default and requires very strong reasons to be just.

Why do the defenders of cancel culture think they’re right? I spot a few lines of argumentation:

  • People who take the wrong views on certain issues are bad and deserve to be punished
  • Cancelling people chills speech on certain topics, stopping bad ideas from spreading.

I think a good limtus test for this reasoning is the one applicable to moral reasoning in general. Is it universalizable? Does it work differently for your ingroup and outgroup or is it the same for everyone?

Three examples of cancellations:

  • A person living in a republican dominated area comes out supporting abortion. Most others think abortion is murder and that the person is evil for supporting such a position. A mob forms that pressures their employer to fire them. Their employer agrees. That person cannot find a job and is unemployed or stacks shelves.
  • A person working in an oil firm in Texas in 2003 comes out as being against the Iraq War. Most people are for the war. Most people believe that by being against the war, the person is spreading dangerous ideas which make it more likely that a dictator who literally murders, gasses and kills millions will be allowed to keep doing so. The person looses their job, their reputation and no one will hire them. Other peopl who have doubts about the war see what happens and are silent from then onwards.
  • A miner works in a coal mine in Thatchers England. He organizes strikes and agitates for industrial action. The social elite, including officials, mine owners and most newspaper owners, have decided that strikes are bad. The miner is fired and blacklisted so that he can never work again.

I believe that almost no one who supports modern cancel culture would support the above examples of cancellation. There’s no impartial basis for that difference in judgement. The only difference is that the victims in those examples are either left-wing or defending traditionally blue tribe positions.

Hence I put modern cancel culture in the category of "it’s fine to use force and coercion and restrictions of speech on our opponents but obviously not fine for them to do it to us".

Closing thoughts.

People should be free to believe what they wish. Freedom of conscience requires freedom of speech, both because ideas you silence internally enough times die and because ideas are not the product of one mind but of a network of minds.

Using economic coercion to silence people with beliefs you and your tribe deem to be wrong is almost always immoral. Using economic coercion purely to punish people for having the wrong beliefs is always wrong.

Avoiding obviously bad things

Draw one circle inside another. The inner circle is you. The outer circle is the world.

One of the main pursuits in life is being better. The stronger, wiser and more moral you are the more you can achieve.

There are things in life which are bad. Some of those bad things are innate parts of your character and psychology. They can’t be entirely avoided, only fought against and managed. Other things, like addiction, are imposed from the external world. You can’t be a drug addict if you never take drugs. This second class of bad thing is often easy to avoid. It’s clear what the bad thing is. It’s clear that the tradeoff, pleasure for a chance at addiction in the case of drugs, is not worth it. All it takes is a choice and an obvious one at that.

My impression is that for the average person born in the west today, all it takes to have a decent life is to avoid obviously bad choices.

Founding a tech startups is not a good way to maximize your own wellbeing

Why do people start a business? Two reasons:

  • Meaning
  • Money

Let’s focus only on money for now.

If you start a tech startup, you should expect to fail and make no money. Even if you think you’re super special, have an amazing idea and generally have all the signs of a successful business, including funding from a major firm in your seed round, you should still assume you’ll fail. The reality is that 99%+ of tech startups fail and even for the 1% that do succeed, most value is captured by a very small proportion of firms. The distribution of income among startup founders is a fairly extreme power one.

TLDR: If you start a tech startup, you’ll probably make no money and have a tiny chance of making lots of money.

One could take the money you could make in tech startup land and do an expected value calculation at this point. That’s what some people do who then say "You’ll probably make no money but the extreme payoffs in the marginal cases still push the expected value high enough that it’s worth doing.". That’s mistaken because money does not convert to utility at a constant rate.

If you go from having $0 income to having $600, there’s a huge utility gain. You get to not starve to death. If you go from $80’000 to $100’000’000, there’s not that much of a gain in utility. You can buy nicer things. That’s about it. You can’t buy a longer life, a happy family, meaningful friends, IQ points or anything else that would make a difference once your basic needs are met.

TLDR: Money converts to utility at a steeply declining rate.

If you’re a highly paid professional, you already make a lot of money. Probably enough to save 50% of your income and still live a fairly luxurious life. Winning the startup lottery and getting rich will make your life a bit better but not much. Loosing the lottery will mean years of stress, burn-out, potentially low income and other bad stuff. The utility calculus is less one of a small chance of extreme payoffs and more one of a large chance of halving your life quality and a very small chance of increasing it by 10 – 20%.

TLDR: If you want to start a startup because you want to be rich, you’re probably making a mistake.

Cases where this doesn’t apply:

  • You want to start a startup for other, non monetary reasons. (A good litmus test for this is whether you would do it even if you had to forgo any personal profit from it.)
  • You money/utils conversion does not suffer from diminishing marginal returns. (e.g: you donate money to AMF and want to maximize lives saved. A thousand times more money = a thousand times more lives saved.)
  • Your entry/exit costs are low. (Note that this is unlikely. Starting a firm is unforgiving. It will likely require late nights and weekends for a prolonged period of time. It takes at least half a year to know if you’re on track for success of some kind or for it to be clear that your a failure. Even if you don’t loose career capital/a cush job, you will loose time and your relationships will suffer.)