Why I built it
Since last summer I write most of my code together with an AI. I started experimenting with Claude after I saw a friend of mine use it, and built a number of projects with it. After a while I noticed a pattern in myself: I merged pull requests quickly, because I wanted to see the result. Reviewing everything the AI wrote was tiring, and a lot of that code was rewritten a few sessions later anyway, so the review had been wasted.
That gave me the idea to review only code that is ripe: code that has not changed for a long time. Once the AI stops touching a piece of code, it has settled, and that is the moment a human review is worth the effort.
How a block ripens
Rumination mirrors the GitHub repositories it watches, replays the history of their default branch and splits every Python file into blocks: functions, methods, class bodies and module-level code. For every block it knows when it last changed, how many commits it survived since, and whether an AI made the change (a Co-Authored-By: trailer naming an AI tool). A block is one of four things:
- Fresh: changed recently. Reviewing it now is likely wasted: the AI will change it again.
- Ripe: survived 3 commits to the same file, or 20 commits anywhere, without changing.
- Hotspot: changed 4 times or more in the last 30 commits. Look at the design, not the lines.
- Reviewed: someone approved exactly this version.
A few rules keep this honest. The fingerprint of a block is a hash of its syntax tree without docstrings, comments and positions, so a formatter run does not make code fresh again, but changing < into <= does. An approval belongs to that fingerprint, so the moment the code changes it is no longer reviewed. Code that moves to another file unchanged keeps its age, and a merged pull request counts as one commit, so an AI session that pushes twenty commits to a branch counts once. Each repository can set its own thresholds and risk tiers in a .rumination.toml.
The dashboard
The dashboard lists the repositories with ripe code, the review queue per repository, and a code view to approve a block, with the file around it, where it is used and the code it calls. A treemap shows a whole repository at a glance. Rumination reads repositories through its own read-only GitHub App, and because it shows private code, every page needs a GitHub login through the shared sign-in service of my homelab. It runs on Locus.
Guardrails before, Rumination after
Next to Rumination lives architecture-guardrails: architecture tests a repository declares for itself, each with the reason why. They catch the mistakes an AI makes quickly, such as an import where it does not belong or a dependency cycle hidden inside a function, and the failure message says why it is not allowed. Both keep AI-written code in check: the guardrails before a change, Rumination after it has settled.
Information retrieval, applied to my own code
This year I follow Information Retrieval in the Artificial Intelligence bachelor at the University of Amsterdam. Two ideas from the course turned out to fit code as well as they fit documents, and I am now using them on my own codebase.
Evaluating a detector like a search engine. The guardrails also measure code smells, such as a function more interested in another class than its own. A rule like that has a threshold, and the question is where to put it. I answer it the way information retrieval evaluates a search engine. Several systems flag candidates on a random sample of Locus: the rule itself, an old rule of thumb, and four blind AI reviews that only got the definition of the smell. As in TREC, only the union of what they found is judged, by me. From my verdicts follow precision, recall, precision@k and a curve over the threshold, so the threshold comes from that curve instead of a gut feeling.
Latent semantic indexing for cohesion. The course covers how latent semantic analysis solves vocabulary mismatch: with a singular value decomposition, words that occur together, like boat and ship, end up in the same dimension, so a document matches a query that shares none of its words. I use the same technique on code. Every function becomes a document of the words in its identifiers, docstring and comments, weighted with TF-IDF and reduced with SVD to 100 dimensions. The cohesion of a module is the mean cosine between its functions in that space: close to 1 means the module talks about one thing. The space is learned from the project itself: no model to download, deterministic, and a few seconds on a few thousand functions. I compare it with a pre-trained sentence model, all-MiniLM-L6-v2, the one I used for the search engine in the course project, and score both rankings with average precision and nDCG.
On Locus that gives 649 modules with at least four functions and a median cohesion of 0.43. The loosest modules are almost all command-line commands, and the space splits each of them into the same two topics: the messages every command repeats (failure, interrupted, loading) and the work the command actually does. Whether that means those messages belong in one shared place is exactly what I am judging now.
Where it stands
Rumination runs on my own repositories. The research is in progress: the lists are measured on a fixed commit of Locus and wait for my verdicts, and the thresholds the guardrails use will follow from those.
Sources
- Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval, Cambridge University Press, 2008: chapter 8 on evaluation and chapter 18 on latent semantic indexing.
- Scott Deerwester, Susan Dumais, George Furnas, Thomas Landauer and Richard Harshman, “Indexing by latent semantic analysis”, Journal of the American Society for Information Science, 1990.
- Andrian Marcus, Denys Poshyvanyk and Rudolf Ferenc, “Using the conceptual cohesion of classes for fault prediction in object-oriented systems”, IEEE Transactions on Software Engineering, 2008: the C3 cohesion measure.
- Kalervo Järvelin and Jaana Kekäläinen, “Cumulated gain-based evaluation of IR techniques”, ACM Transactions on Information Systems, 2002: nDCG.
- Ellen Voorhees and Donna Harman (eds.), TREC: Experiment and Evaluation in Information Retrieval, MIT Press, 2005: pooling.
- Martin Fowler, Refactoring, second edition, Addison-Wesley, 2018: the code smells.