The answer: I’m not sure, but I’m building a tool to find out!
Part of the reason I started this blog was to learn in public. Explaining what you’re learning to help others learn is apparently one of the best ways to actually…learn! (…I’m currently reading Elements of Eloquence to help with my writing, don’t @ me).
I also started this blog to share my thoughts about the state of the world. I’m a US citizen, and the direction we are headed is making my hair stand on end. I’ve always been interested in economics and finance, and it baffles me how members of the government are allowed to trade on their insider information. People should have easier access to this data, so why not a data aggregator! I did some research and…theres about 50 that already exists, check out congressstock.com, capitoltrades.com, or congresstocks.com. So, I thought more about this… How can I actually try and add value here?
For better or worse, I’m currently living in the area of AI. AI’s like claude seem to be incredible at pattern recognition. So, now I’m playing around with a tool that reads through hundreds of government trading filings on its own and reports back to me daily with what it found, and calls out possible patterns.
That’s the actual pitch. Data aggregation is a solved problem. Pattern recognition across messy, slow-moving, self-reported government data is not — or at least, not solved in a way that’s sitting on a public dashboard anywhere I’ve found. So that’s the gist. Claude does the reading, I do the deciding whether what it found is actually a pattern or just noise wearing a pattern’s clothes.
Which brings me to the two things that I am still trying to solve for.
Concern One: Where Is It Even Getting This From?
First, I’m not a data guy. I’m just a dude messing around and see what these tools can do. So, if I tasked a human to find patterns in government reports, and the news, what would I do?
Easy, I would give them a list of approved sources, and a list of rejected sources. So, thats exactly what Claude was provided. Primary sources first (House Clerk filings, actual disclosure documents). Aggregators like Capitol Trades and Quiver as the daily workhorses. News coverage cross-checked against a “least biased” outlet list, helpfully provided by https://mediabiasfactcheck.com, and anything extremely biased (also helpfully provided by https://mediabiasfactcheck.com) or conspiracy theory peddling is just off the table.
Writing those rules down was the easy part. The part I genuinely don’t have a clean answer for is: how do I know it’s following them, and how can I know that the data sources are truly minimizing the bias? I can’t watch every time Claude fetches this data, and I cant read every single news article to get a sense for its bias. I’m not going to read every source it touches before it touches it. At some point I’m trusting a system to police its own reading list, which is itself assembled by a 3rd party. There just too much data for a human to verify in a reasonable amount of time (and on a tight budget).
What I’ve landed on for now, imperfectly, is forcing receipts. Every pattern in the tracker has to cite back to a specific source, logged in one running file, so I can spot-check instead of blind-trusting. It’s not proof the rules are being followed every single time. It’s a paper trail I can actually audit instead of a black box I have to take on faith. Still figuring out if that’s enough.
Concern Two: There are three kinds of lies
Bad sourcing is at least visible — I can go check the citation. A confidently-stated pattern that isn’t real is a lot sneakier, because it reads just as convincingly as a true one. Mark Twain popularized the quote “There Are Three Kinds of Lies: Lies, Damned Lies, and Statistics”. I don’t want to use Claude to be a peddler of lies.
To account for this, I aimed to structure the pattern recognition prompt to specifically call out how certain it is in a pattern, based on how frequently its seen, how recent the data that makes up the pattern is, and to explicitly call out shifts in patterns its previously identified.
A few days into running this, the tracker flagged “a small number of members account for most trading activity” as a pattern, backed by frequency. Fine. Then two days later it came back and complicated its own finding: some members trade rarely but in huge dollar amounts, so “who dominates” depends entirely on whether you’re counting trades or counting dollars. It didn’t quietly swap the story. It kept both reads on the page and said, basically, “this is more complicated than I first said.”
That’s the behavior I actually want — a system willing to correct itself in public instead of doubling down on the first clean-sounding narrative. It doesn’t mean it’s right every time. It means when it’s wrong, it’s wrong in a way I can catch and it can walk back.
The Honest Answer: I’m not sure yet…
I don’t have this fully solved. I have a citation trail to account for my lack of data knowledge, and a series of prompts that are, so far, leaning Claude to self-correction. This still doesn’t include any guarantee against hallucination. I think that what I’ve come up with so far are bandaids to this problem, not long term fixes. I eventually want this to be under the “projects” part of this website, but I think it needs a bit of refinement before I get to that point.
I’m treating this the same way I treat everything else on this blog — build it in public, watch it closely, and be honest here when it breaks.
If you’d like to know when this goes live, give this blog post a like!
Leave a Reply