Andrej Karpathy posted an interesting question on X recently, asking how people used LLM chat bots. Specifically he was talking about making new chats for every question/purpose, vs. the One Thread approach. He discussed some trade-offs in both performance and fidelity when taking advantage of the large context windows now available to us. It’s really interesting and I recommend you read the whole post. But I was surprised to find that no one else who answered him described the method I use, and so I wrote it up, and it garnered a lot of interest, so I thought I’d flesh it out a bit further here:
I manage hundreds (thousands?) of conversations that fall into four groups:
- long-running, bookmarked – basically my staff
3 examples:- I have an AI personal trainer/nutritionist I always return to for training/nutrition questions.
- I have one conversation that helped me build my current home Linux box, and I return to it for any HW/OS/SW questions related to it.
- I have several AI professors I use to learn various subjects – one per subject
- useful, may return to, but not necessarily
examples:- I saw nice sweet potatoes at the grocery store – asked about sweet potato soup, made soup. A week later, I saw a nice pumpkin – wanted to make a similar soup. Remember that convo, which already knows my equipment and preferences, returned to that conversation for a different soup.
- in general, if I think I’ve asked a question before, and the context from before will save me some time now, I use search to look at previous conversations, and might continue one of them rather than start a new one
- One-off questions: I usually ask them in a fresh conversation
- Truly throwaway questions. Not only do I start a fresh conversation, but I will usually archive/delete it when I’m done. This is when the subject is pretty trivial, and I view it as clutter.
Special case for some long-running conversations: I have also noticed that sometimes overly long context can start to produce weird effects (and Andrej describes a bit of why this happens). The LLM starts to hallucinate more, is less reliable about remembering details, and so on. In situations like this I sometimes ask it to generate a detailed summary of everything we have been working on, and I may ask follow-up questions, and then I paste the results into a new conversation and continue from there basically having transplanted the essentials from the older chat to the fresh one.
It is true that, by this point, the LLM is already getting a bit squirrely, so if the summary is missing anything important, I remind the LLM, and it’s enough to help it remember and then it add a summary of that too. It’s not that it forgets, it just has lost the thread on what is most relevant. But I can help, and so I repeat until I’m satisfied and only then do I feed it to the new chat. It’s not perfect, but I’m also not deleting the old chat so I can go back to it if necessary.
One of the followup questions I got was about how I go about organizing and finding the conversations I want in the midst of the thousands I have. That works like this:
Both Grok & ChatGPT make it relatively easy. You can rename chats, so for my “staff,” that is, my long-running chats-with-a-purpose that I return to, I rename them like,
** Personal Trainer **
** Nuclear Engineering Professor **
** Productivity Coach **
which makes them easy to pick out of the list.
Grok even lets you bookmark specific chats. And, if one of my AI staff has scrolled way down because I haven’t talked to it in a while, it will pop right up when I search for it because it has a good and memorable name.
I used to create browser bookmarks too, because each chat has its own URL, but I find that’s not needed and so I stopped.
As for the remaining conversations, I’m less worried about needing to find them. Like in my soup recipe example, I just search for “soup” and 13 conversations pop up, and I reopen the one that is the most relevant to what I want to do now.
Hope you found this as helpful and let me know on X (I don’t ever look at the comments here), or if you have a better method I’d love to hear it too.
