How FinSimLab Authors Country Tax Models
You may wonder how I added all these countries' tax models. I have a reasonable understanding of the Irish and the Agentinian tax systems, having lived in both. But if other countries are as different from each other as these two, there’s no way for me to hand-model more than a couple of countries. It simply doesn’t scale.
As a solo developer, AI was my only option. I know, LLMs “hallucinate” every word they say, and sometimes that happens to match reality. Asking an AI to “model the tax rules for country X for the simulator” would not work. But the same way there's techniques to get an AI to write complex software, there's also ways to achieve complex goals like this one. So let me explain how I add a new country to the simulator:
Independent research packet
The first step is to get an AI agent to research the country’s tax system and write an official-source citation packet following a template that has a section for each aspect of the tax system that we care about, with exact source quotes. This is done without any access or knowledge of the simulator, to avoid any bias. The source quotes are critical because if they match the text found in the source, it proves that the AI actually had to read the document. It’s well known that LLMs can make up sources, with likely sounding URLs and everything.
Mechanical quote verification
The next step is a mechanical verification of the actual presence of the quoted text in the linked source material. A simple script goes through the list of citations, fetches the cited document, and verifies that the quote is present. Scripts can’t hallucinate. If this gate is passed, the research agent did actually read a real document.
Semantic review
A valid quote doesn’t guarantee that a tax rule claimed to come from the quoted document is actually in the document. So now another AI agent goes through the tax rules and the cited documents, one by one, and tries to find cases where the tax rule doesn't match what the document describes. LLMs like to please the user. This user is pleased if it finds an issue, a mistake. This kind of adversarial review, while not perfect, works quite well.
Targeted addendum
If any of the last two steps produced findings (invalid quotes or unsupported claims), a new research brief is created to correct the mistake. That goes back to step 1, which produces a research addendum, which then goes through steps 2 and 3 again. And if there’s still issues, another round is made. This keeps going until there’s no more issues. In most cases this takes one or two passes.
Modeling spec
Now we have a set of tax rules for each aspect of the simulator that has been vetted. Now it’s time to bring in the knowledge about the simulator: what it can model and what it can’t. An AI agent takes the research packet and decides what can be represented directly, what should be approximated, and what is not currently representable by the tax engine. That last part becomes a “gap report” and includes a list of recommendations to expand the tax engine.
Ruleset implementation
Another AI agent takes the modeling spec and encodes the country’s tax rules as a configuration file. Part of the research packet was to find worked examples and calculators. Those are then turned into tests and used to validate the implementation.
Economic data
A script then fetches economic figures like inflation, exchange rates, cost of living and rental yield, from sources like the World Bank and Numbeo, and adds that to the country configuration file.
Audit
The final gate audits the chain from research packet to implemented tax model, looking for dropped rules, unsupported modelled claims, undocumented approximations, label problems, missing tests and remaining issues, and any findings produce a new review request that repeats all the necessary steps.
Do I blindly trust the output of this process? Of course not. I need real people checking the result. Since I can’t hire experts, I let people try it for free if they promise to give me feedback. If they do, they get a promo code.
