Advanced A/B Testing for Formative Evaluation

LTT version

Note: In this paid post, I'm going to introduce a useful program evaluation method, explain when you would use it, work an example, and give you code you can copy to do the analysis yourself. I’m starting paid posts for a couple of reasons: 1) because certain posts are highly technical and take me a long time to write, and 2) I want to support Substack in its current form as an ad-free platform.

If you haven’t already read my first lesson on A/B testing, I’d suggest you start with that one because this one picks up where we left off last time.

As a reminder, I’m going to repost the facts of our worked example.

Worked Example

Suppose that a community health program is testing a new appointment reminder system to improve patient follow-through on specialist referrals. Patients needing specialist care are randomly assigned to:

• Treatment as Usual (TAU): Standard referral process with a written referral slip.

• Novel Approach: An enhanced referral system where patients receive personalized text reminders and a call from a case manager before their appointment.

• The outcome variable is successful attendance at the referral appointment.

After recruiting 300 participants, we observe the following outcomes:

• TAU group: 90 out of 150 patients (60%) attended.

• Novel Approach group: 108 out of 150 patients (72%) attended.

Logit Transformation Testing

Let’s start by recalling what we did previously. In our first analysis (Beta-Binomial Model with IBE), we placed independent priors on the success probabilities for each group.

This means we assumed separate priors for TAU and Novel treatment. Formally:

\(p_1 ​ ∼Beta(α_1 ​ ,β_1 ​ ),\ p_2 ​ ∼Beta(α_2 ​ ,β_2 ​ )\)