Researcher proposes 'generalized belief states' to decode and steer what sequence models represent

Artemy Kolchinsky's LessWrong essay defines low-dimensional, recursively updatable statistics sufficient for prediction and maps them onto a toy transformer trained on a simple hidden Markov process, finding the KV cache plays a key role; it is a single-author post, not a peer-reviewed paper.