A stitch in time
Why better evaluation could have helped LA's homeless services
The Los Angeles Homeless Services Authority (LAHSA) is not popular. Every few months, it suffers a scandal devastating enough to kill a lesser agency, ranging from the ironic (2021: staff not paid enough and some are homeless) to the puzzling (2022: staff caught throwing away food meant for the hungry). This week, a federal judge held a hearing to publicly ask the city, the county, and the agency about another scandal: why had they loaned $48 million in cash to local service providers with basically no hope of ever getting it back and nothing to show for it?
The missing cash came to light during an audit by the LA Controller released last week, which you can read here, but which I will briefly summarize. The Controller discovered huge irregularities in the finances of LAHSA, including issues like not having records of who the agency currently has contracts with and not keeping track of how much money they had given out (LAHSA agreed with these verdicts). The issue of the missing cash, the auditors suggested, could have been avoided by entering into formal agreements with service providers specifying terms of the loans. LAHSA officially responded to this suggestion by stating that 1) the county doesn’t require them to get the money back and, 2) they aren’t required to enter into formal agreements with people to whom they loan money. So far, they’ve gotten about 5% of the money back.1
The episode raises a lot of questions. For one thing, why is it policy to loan cash to nonprofits (as most of the recipients are) who provide services but usually don’t have a good way to make money to pay back the loans? Theoretically, the nonprofits could ask their funders for money to cover the debt, but that is a tough pitch. “Please bail us out, we owe a million dollars to LAHSA and we thought they forgot about it but they didn’t” does not make a fantastic fundraising email. The city and county had the power to simply create grants and give away the money. Why didn’t they do that instead?
The next question that arises, I think, has to do with the culture of oversight and evaluation in Los Angeles. The independent auditor originally quoted a cost of $2.8-$4.2 million to properly conduct the audit and LA argued them down to $2.2 million, well below their lowest estimated cost. This still might seem like a high price but remember that this is an audit of a joint powers authority with an annual budget of about $875 million, a sizable proportion of which is, frankly, lost. A project like this takes a lot of very skilled people, particularly if you want it done fast, which you usually do when money or proof of work is missing.
Surprising no one, LAHSA did not comply with data requests in a timely fashion and delayed the audit, racking up about a million dollars in additional fees from the independent auditors who needed to spend more time on the case. The city’s attorneys argued in court that they shouldn’t have to pay for the incredibly cheap (given the breadth) audit, effectively trying to stiff the auditor in broad daylight. The federal judge on the case accused the city of trying to underfund the audit deliberately. “I will not allow this to be an inadequate audit,” he said.
What does all this say about the culture of evaluation and oversight in Los Angeles?
What was missing?
Here is the standout quote from LAist’s excellent coverage of the developing LAHSA story:
“Thursday’s hearing came after a scathing audit released Tuesday night by the L.A. County Auditor-Controller’s office found that contracts by LAHSA with various providers didn’t set proper standards or measure outcomes, resulting in inconsistent and poor care.”
The two key concepts here are “set proper standards” and “measure outcomes.” Whatever our pet ideas about evaluation may be, these are still the core concerns for real-world institutions. Audits are a type of evaluation in which the criteria are financial and bureaucratic. As evaluations, they must have criteria in order to frame value judgments, and these criteria are cashed out in specific standards. On the other side of the balance scale, we accumulate evidence for outcomes. LAHSA didn’t do it, and now the people of Los Angeles have a big, politically toxic mystery on our hands. Worse, we have no idea whether the funds spent on homelessness are being used for their intended purposes or what local strategies are and are not working.
A stitch
This week just before the LAHSA story broke, my friend RB and I had a conversation about why many social problems are so hard to fix. She suggested the idea that many social problems can be understood as Nash equilibria, that is, “games” in which every player is making their best move conditional on all the other players making their best move.2 My personal favorite version of a suboptimal Nash equilibrium is the Western gunfight, in which both duelists know that they would be better off turning their backs and walking away, but this only works if they both do it, so they both draw and take their chances. As long as we remember that game theory is a model and so it will only partly fit a messy reality, this perspective can get us pretty far. Organizational mismanagement seems like a good application for the suboptimal Nash equilibrium model because of the way incentives are aligned in highly dysfunctional bureaucracies. Here’s a test of this: is it actually in the interests of a single, well-intentioned person to improve the functioning of the system? In highly dysfunctional organizations (such as those that, say, have no idea who they are contracting with and lose track of millions of dollars), think about what happens to people who try to improve the organization – their ideas get shot down, they get treated like “teacher’s pet” just for trying to get people to follow standard procedure, everyone hates them, and maybe they even get scapegoated for the very problems they point out. Anyone higher up in the hierarchy who might be blamed for the problem (which, thanks to the pyramidal structure of most modern organizations, could be almost anyone high enough in the organization) gains an instant incentive to silence critics, which coincides conveniently with their power to do so.3 Once we understand the incentive structures in dysfunctional organizations, we understand why everyone puts up with the suboptimal equilibrium of bad behavior – any action to improve the system beyond its defaults is actually disadvantageous to individuals.
This is where evaluation comes in: to disturb the suboptimal Nash equilibrium of the dysfunctional organization. The evaluator says things like “Can you please show me the balance sheet for last quarter’s costs?” and “Why is this is a loan instead of a grant?” and “What are the formal terms of the loan?” and “Can I see the provider agreement?” and “Can I talk to your participants?” and so on. But, unlike the staff member, the evaluator cannot be punished for the indiscretion of asking the wrong questions. The evaluator is outside the chain of command. If the records cannot be provided or staff don’t cooperate with the evaluator, this too goes in the report for the sake of transparency so that at least everyone knows there is a potential problem. But this doesn’t happen often because the presence of the evaluator changes the incentive structure in the organization so that sharing evidence with the evaluator is the smart move. As Ry Brennan pointed out to me this week, evaluation is arguably a method for stimulating cooperation to overcome coordination problems. The equilibrium is being acted on by an outside force that has the ability to nudge it in the right direction.4
Of the several long sighs I emitted while reading the LAHSA audit, the biggest one was when I realized that the cash loans were given out in 2017 and the public just learned about them this week. Even one year of this policy would have been enough to raise eyebrows, but seven years of silence? An ongoing, cost-inclusive evaluation could have been done for a tiny fraction of the money that went missing. This evaluation would have uncovered the irregularities in the first year and made a public report. It would have been cheaper than the audit. Critically, it could have stopped the money from disappearing in the first place.
The 18th-century expression “a stitch in time may save nine” is about prevention. If you spot a problem early enough, like a sweater beginning to unravel, you can use a quick stitch to prevent it from getting much, much worse. I like the expression because it conjures the image of doing some skilled work – we don’t just get to think about fixing the problem, we have to grab our kit, focus, and take action. Sewing and weaving are likely the earliest applications of abstract math, geometry, and computer programming, so I would argue that comparing evaluation to sewing does honor to the former.
Because this is a developing story, please fact check my assertions if you are reading this post in the future. We are still finding out the details of what happened here and there are details I might not know about this situation right now but that could come out later.
If you’re having trouble with this concept, it’s important to remember that Nash equilibria refer to non-cooperative games only and that players by definition aren’t allowed to help each other. This is an assumption of the model. As a reality check, remember that regular people are often really good at cooperating when big institutions aren’t around to raise the stakes of competition.
There is a tendency among people who learn about concepts like Nash equilibria to start seeing them everywhere. Part of my point in this paragraph is that there are specific conditions in which Nash equilibria obtain. I also think that it is possible to build better organizations that engineer around most of the problems I’ve listed in this piece. People do it all the time.
Most of the organizations I’ve worked with are functional and do not meet the description of a suboptimal Nash equilibrium I’ve given here. I’m not saying that I know for sure that LAHSA meets this description either. However, my messages are open if they would like to be evaluated.