LANGUAGE MODEL RESEARCH / HUMAN + AI
Fractal Context
I could test the mathematics without writing the machinery.
I have long been fascinated by fractals. For this experiment, I wanted to test a specific idea: could a transformer learn something useful by applying the same simple quadratic rule to its recent context at several different scales?
I worked on the hypothesis, the mathematics, the comparisons, and what to make of the results. AI implemented the experiments, ran them, and brought back the evidence. Throughout this work, I did not inspect the implementation myself.
That division of labor let me spend my time on the question I was trying to answer, rather than on writing the code needed to ask it.
The idea needs to beat simpler explanations.
The experiment looks back over spans of 1, 2, 4, 8, and 16 tokens. It applies the same quadratic recurrence across those scales in a ReLU² transformer.
Getting that to run would not tell us much. I need to compare it with the unchanged baseline, an ordinary gate, and a matched linear recurrence. Does the quadratic part add something the simpler alternatives do not?
The controls are there to separate the idea I am interested in from other explanations: extra computation, generic gating, or chance. A promising number is only useful when I understand what could have produced it.
The conversation was about the experiment.
When a result was ambiguous, I could work out what comparison or measurement would help. AI could change the code, add the instrumentation, and run the next experiment.
I would then look at the evidence. Was the behavior interesting? Was the comparison fair? Had we ruled out the obvious alternatives? Those questions determined what we did next.
I found this more interesting than having AI write code faster. It let me pursue the mathematical question directly, with AI doing much of the work required to turn that question into an experiment.
The early results make me want to keep going.
There is enough promise in the early results to justify another experiment. I do not yet know whether the architecture improves learning.
The apparent effect could disappear in larger models, change with the random seed, or fail against the full set of matched controls. It may turn out to be noise, or something that only helps at a small scale. I need to run those tests before I can say whether the idea works.
This is a way of working I intend to keep exploring.
I see a parallel with AI-assisted work on mathematical questions such as Erdős problems. There are many things AI can help with: searching, running numerical experiments, finding relevant literature, and formalizing an argument. Someone still has to judge what the work actually establishes.
In my experiment, the useful questions came from working with the mathematics and looking critically at each result. Knowing what to ask next mattered as much as being able to run the test.
The fractal hypothesis may fail. I would still have learned something valuable about how to pursue a difficult technical idea. AI let me keep returning to the part that interested me: the conjecture, the evidence, and what we should try next.
Work with me
Tell me what you have in mind.
I build websites and apps, provide AI training, and consult with businesses and organizations getting AI into their work. Tell me what you want to do and what has been getting in the way.