date = 2023-04-26 tags = ["deno", "statistics"]
A hypothesis test is an experiment designed to determine whether to confirm or reject a hypothesis. An A/B test is an experiment to establish which of two treatments is superior. One group of data is called 'control' and the other is called 'treatment'.
One way of determining if the difference between groups is real or just by chance is permutation test.
I wrote a Deno script that does a permutation test on given data. The input data file is expected to be in the following format:
group value A 1 A 1.1 A 0.5 B 0.9 B 0.4 B 1.4
The following is how to run the script and an example run:
% deno run https://jaeyoung.se/toys/hypothesis.js A/B test Enter the path to the TSV file containing A/B test data (default _local/data.tsv) _local/data.tsv ✅ Granted read access to "_local/data.tsv". Pick the type of null hypothesis * 0) No difference between the means of group A and group B * 1) A <= B * 2) B is not X% greater than A > 0 Resampling 1000 times p-value: 0.737000