Cebul.la · Writing · Brant Cebulla · · Updated
Gathering CrossFit data: the methodology
The CrossFit posts used to rest on a scrape from May 2014. They now use a later export of the same kind of athlete profile. This page is how the first scrape was gathered, and what both scrapes leave out.
On a CrossFit Games athlete profile, athletes can type in their best marks:
- One-rep maxes for the clean and jerk, snatch, deadlift, and back squat.
- Max unbroken pull-ups.
- A 400m sprint and a 5k.
- Fran, Helen, Grace, Filthy 50, and Fight Gone Bad.
They also type in a division, a height, a weight, and an age. I wanted the shape of those marks: an average, a good time, a good lift.
The 2014 scrape
I pulled the pages with a tool called FMiner. The project walked every twentieth CrossFit team, from team 1 through team 6,000. On paper that is 300 team URLs. Some of those pages were missing, and some teams had no athletes.
When a team page loaded, the scraper opened every athlete on the open roster.
On the athlete page, it copied the self-reported marks.
That produced 5,376 athletes, 3,159 men and 2,217 women. Most of them had left the marks blank. Dropping the blank profiles left 1,935 athletes, 1,301 men and 634 women. Men were 58.8% of the profiles and 67.2% of the people who had posted a mark. The sample sizes in the original posts are smaller still, because each post kept only the athletes who had filled in that one field.
The later scrape
The posts now use a later export of Games profiles, athlete IDs 1 through 299,999. That is 60,843 men and 34,241 women. Gender comes from the division field. I check for women and girls before men and boys, because the word women contains the word men. The fields are the ones listed above. The Open score columns in the export are empty, so the posts leave them unused. Lifts and body weights posted in kilograms were converted to pounds. Each post says which times and lifts it kept.
What gets left out
People who type in a mark are a fitter group than a typical class. They know the number. The Open already leans toward athletes who want a score next to their name. A posted Fran, or a posted clean and jerk, is that group.
The profile records the number and leaves out the conditions. A Fran time does not record the load on the bar. A clean and jerk does not record whether the lift was a squat clean. Where a fast time lines up with a heavy lift, the posts treat that as the standard workout and say so.
The current numbers start with What’s a good Fran (CrossFit) time?.