How it works

The census is five pipelines and this page, all Mojo. The pipelines run on orca, magmalake's pipeline orchestrator, on AWS; they write Iceberg tables to S3, and this site reads them.

• gharchive loads each day of GH Archive, every public GitHub event, into gharchive.events at 03:00 UTC for the day before. Each hour is a dista task on a worker instance, which downloads, gunzips, parses and writes its Parquet files; the step commits the hours in order.
• mojo_repos is a sensor on that table. For each new day it finds the repositories that might hold Mojo, checks them with GitHub, and keeps the Mojo ones in mojo.repos.
• mojo_history is keyed by repository: each new one is cloned and its history read, the Mojo files and lines at each commit into mojo.commits and each change of Mojo version in its lock file into mojo.migrations, and its git tags into mojo.tags.
• mojo_activity counts each repo's GH Archive events per day, with no API calls: stars, forks, releases, issues and pull requests opened, pushes and people, into mojo.activity and mojo.releases. A repo found today is read back over every day already loaded, so its history is complete.
• mojo_stats asks GitHub about every repo each Monday (one request each) and keeps the stars, forks, issues and topics week by week in mojo.repo_stats.

Every run, with its steps, logs, machines and the tables it wrote, is on orca.mlake.dev. The orca docs cover writing a pipeline, running it, the web UI and debugging a failed run; the census's own code is magmalake/mojo-census, and the getting-started guide builds it from scratch.

How the pipelines got faster, with the measurements, is on the Optimizations tab.