Is that lucky streak real - or did you just look hard enough?

Call it m, the number of tests. At a 5% threshold, the expected count of false alarms is m times 0.05. For m = 62 that is about 3 - so finding three “lucky” contexts is exactly what noise produces. Counting the tests you never formally ran (the garden of forking paths) is the hard, honest part.

For a context with mean score x-bar over n observations and spread s:

More observations shrink the denominator, so the same real edge shows up as a bigger z. That is the grain of truth in “collect more data” - but it does nothing about the 62 looks in the next step.

Three small datasets, three shapes of data - and the same recipe every time: (observed average − average expected under no luck), divided by the standard error.

For each dataset we walk the path an analyst would walk at the keyboard: look at a few raw rows, sanity-check the file as a whole, group by the context, then turn each group into one z. The highlighted rows are the numbers the rest of the page keeps reusing.

Example 1 - a continuous score with a known expectation. Six months of the asker's simulated-lottery log: 1,800 rows, one per play, with columns date, hour, weekday and score - the outcome already normalized against the game's known expectation, so “no luck” means an average score of 0, and single plays scatter with spread s = 1. Context to test: “Tuesdays”, which holds n = 260 plays averaging x-bar = +0.16. Standard error = s/√n = 1/√260 ≈ 0.062, so z = 0.16/0.062 ≈ 2.6 - the z we carry through this whole page.

What the log actually looks like - the first rows of the file:

Micro-EDA before any pattern hunting: the mean of all 1,800 scores is +0.004 with spread 1.00 - the normalization behaves, and a broken score column would show up right here. Now group by weekday and summarise each group: how many plays it holds, their average, and that average divided by its own standard error:

Six weekdays hover where noise lives; Tuesday pokes out at z = 2.6. Repeat the same group-and-summarise for the 24 hours and the 31 days of the month, and you hold all 62 z's - the column that steps 3 to 5 will judge.

Example 2 - win/lose outcomes. A game where each try wins with a known probability of 20% - the expectation is known by design, as in the asker's lottery simulations. 2,100 rows: timestamp, hour, win (1 or 0). Context: “the 3pm hour”, with n = 150 tries and 41 wins - a rate of 27.3% against the expected 20%. For a proportion the standard error is √(p(1−p)/n) with p the expected rate: √(0.20 × 0.80/150) ≈ 0.033, so z = (0.273 − 0.200)/0.033 ≈ 2.2. Same recipe - only the standard-error formula changes with the data type.

The raw log:

Micro-EDA: 424 wins out of 2,100 tries - an overall rate of 20.2%, sitting right on the designed 20%, so the game is fair overall and any luck must hide in the slices. Note the unequal group sizes: plays cluster in waking hours, so busy hours hold about 150 tries and early-morning ones far fewer - which is exactly why the summary keeps n visible. Group by hour:

Only 15:00 leaves the pack - and the rest of this page is about deciding whether even that is anything.

Example 3 - sparse subjective ratings. The asker's second stream: 400 evening self-assessments, columns date and rating - −1 (worse than usual), 0 (as expected), +1 (better). Here no external expectation exists, so “no luck” is the person's own long-run average - subtract it first; say the centered ratings scatter with s = 0.75. Context: “the 13th of the month” - only n = 13 evenings, but they average x-bar = +0.31, and it feels like “the 13th is my day”. Standard error = 0.75/√13 ≈ 0.21, so z = 0.31/0.21 ≈ 1.5. The lesson: rare contexts have tiny n, so even a big-looking average sits comfortably inside noise - and that is before the 62-contexts correction.

The raw log:

Micro-EDA: the 400 ratings split into 118 × (−1), 172 × 0 and 110 × (+1) - overall mean −0.02, spread s = 0.75. No designed expectation exists here, so that −0.02 is the person's own baseline: subtract it from everything before comparing contexts, otherwise a generally gloomy or cheerful rater fakes a signal in every slice. Group by day of the month:

Look at what this table quietly teaches: with n around 13 per day, the centered means swing between −0.23 and +0.31 while nothing real is going on anywhere - single days of the month are simply that noisy. The 13th's +0.31 feels like a lot; z = 1.5 says a 13-evening average wanders that far routinely.

Whatever the shape of the data, the same three ingredients: an observed average, the average you would expect under no luck, and a standard error that shrinks like the square root of n. That z is what steps 3 to 5 judge.

The whole procedure in one line, runnable on any log: group rows by context; count n and average each group; subtract the no-luck average; divide by the standard error (s/√n for scores, √(p(1−p)/n) for win/lose) - and you hold one z per context. If your data can produce tables like the ones above, you can run every step on this page yourself.

For 62 contexts, the best-of-62 z under pure noise has a median near 2.3 and a 95th percentile near 3.1. Your 2.6 sits around the 75th percentile - noise matches or beats it roughly one run in four. Not interesting. The “worth a look” line is about z = 3.1. Shortcut without simulating: Bonferroni asks each context to clear 0.05 divided by 62, about 0.0008 (z above 3.2) - your 2.6 fails. If you would rather screen many candidates than make one hard claim, control the false discovery rate (Benjamini-Hochberg) instead. Medical scale-up: genome studies test about a million sites and use a threshold of 5 in 100 million.

A close cousin, but no. The bootstrap resamples your data with replacement to ask “how much would my estimate wobble in a fresh sample from the same world?” - it measures the uncertainty around a value. What step 3 does is a permutation test: shuffle the time labels without replacement to build the world where luck definitely does not exist, then see where your real result falls in it. And taking the best of all 62 contexts in every shuffled run is the classic max-T scheme (Westfall-Young) - it handles multiplicity more honestly than Bonferroni, because the shuffles automatically respect the overlap between contexts (the same play sits in both “Tuesday” and “3pm”), where Bonferroni over-corrects.

And yes - the verdict needs no bell-curve table at all: you simply check whether your real best lands in the top 5% tail of the shuffled distribution (one-sided, because “lucky” is a directional claim). One catch stops you from dropping the z-score entirely: contexts come in very different sizes, and raw averages are not comparable across them. Take the two contexts from step 2. “Tuesdays” holds n = 260 plays, so its average wobbles with a standard error of about 0.062 - under pure noise it stays roughly within ±0.12 of zero (two standard errors). “The 13th of the month” holds only n = 13 evenings, standard error about 0.21 - an average roughly 3.4 times noisier, which the same pure chance routinely swings out to ±0.4. Rank contexts by raw average and the rare ones win almost every shuffled run, simply because they are noisier - your 260-play Tuesdays can never out-wobble a 13-evening context, no matter how real their edge is. Dividing each average by its own standard error (that is all the z-score is) puts every context on one ruler: one z unit now costs the same amount of surprise everywhere, and the maximum becomes a fair contest. Any statistic with that property works - including each context's own permutation p-value (the min-p variant).

Full disclosure: the playground below draws independent bell-curve z's rather than shuffling a real dataset - the right simplification for a lesson. On your own data, the real permutation is strictly better: it inherits your data's odd distributions and the dependence between contexts for free.

At the confirmation stage there is no multiplicity - you test one thing you named in advance, so a plain 5% is honest again. This is also what defeats peeking and regression to the mean in one move: a fluke found by looking everywhere will not repeat on data it never touched. If it replicates, it earns further study; if it evaporates, it was noise all along. Pre-registering the hypothesis, or an N-of-1 design, formalises the same idea.