The original thread can be found here: https://trashchan.xyz/robowaifu/thread/595.html --- Welcome to the Lemon Cookie thread, The goal of Lemon Cookie is to create a framework where a synthetic "mind and soul" can emerge through a "LLM as cognitive architecture" approach. This thread exists to collect feedback, ask for help & to document my progress. First I am going to try to give a high level overview of how this cognitive architecture is envisioned and the ideas behind it. I have spent time looking at cognitive architecture work, in the field there is now a consensus on how the mind works at a high level. An important mechanism is a "whiteboard", basically a global temporary memory that all the other systems read in and out of. Then there is different long-term memory systems that react to and add content to the whiteboard. Along with memory pattern matcher(s)/rules work on the content of the whiteboard. A key thing to consider is the difference in philosophy that cognitive architecture projects have, the intelligence is considered to emerge from the entire system. Compare this to LLM agent work where it's considered the intelligence is the LLM. My feelings on the general LLM space are conflicted, I am both amazed and really disappointed. LLMs possess an incredible level of flexibility, world knowledge and coherence. But everything outside of the model is stagnant. It's endless API wrappers & redundant frameworks all slight permutations on RAG & basic tool calling. I will believe that LLMs are misused as chatbots, simply put their pattern matching and associative power is constrained by chat format and shallow tooling. In the Lemon Cookie Cognitive Architecture so far here are the important aspects: 1. Memory is difficult. I do not think there is a singular data structure or method that is able to handle it all, several distinct types of memory will be needed. So far I plan for a PathRAG like system and a "Triadic Memory" inspired system for external associations (this is missing in most LLM solutions). 2. LLM as Kernel, The LLM's context window is the Whiteboard and has a REPL like mechanism. It holds structured data and logic in scripting-like format so it's both LLM & Human readable while staying easy to parse & allows for expressive structured data. The LLM's role will be to decompose data and make patterns and associations explicit as executable statements. 3. The language has to be LLM/CogArch-centric. There is a thousand ""agents"" that give LLMs a python interpreter as a tool. The two need to be more tightly coupled. Scripted behavior via pattern matching, The whiteboard is a bag of objects, this allows for programmable pattern matching (think functional programming like Haskell). It's also important to allow the LLM to observe code execution and to be able to modify state and execution flow. Data in languages have scoping rules, so should LLM context. Etc... I will go into more depth about the language in another post. 4. Another important system is the "GSR" Generative Sparse Representation and it will be a first class language & runtime type, This also needs its own post. But in general I am inspired by two things, "Generative FrameNet" paper where an LLM & an embedding model is used to automatically construct new FrameNet frames. The second source is "Numenta's SDRs"/"Sparse distributed memory" this representation has a lot of useful properties for memory (Please watch the videos under the "What the hell is an SDR?" segment in my links list for an easy introduction.) I think SDR unions & SDR noise tolerance will be especially useful. 5. A custom model, For all of the above to work well, a model will need to be fine tuned with special behaviors. I do want input on this. Baking facts & behaviors into LLM weights is costly, creating bloated models that are hard to run or train (why memorize all the capitals?), while letting authors gatekeep truth and impose "safety" detached from context. Blocking role-play "violence" or intimacy isn't protection: it's authors hijacking your AI companion to preach at you. Externalizing behaviors via whiteboard pattern matching shifts control: stabbing you in-game can be funny, but a robot wielding a knife isn't. Maybe you want intimacy privately, but don't want your AI flirting back at your friends. When put together I think this will be able to host a kind of synthetic "soul", In a living being what we call a personality is the accumulated associations, learned behaviors, beliefs and quirks molded by a unique set of experiences. I hope this will be true for this system too. Cool links, I recommend looking at them. https://arxiv.org/pdf/2502.14902 | PathRAG: Pruning Graph-based Retrieval Augmented Generation with Relational Paths https://arxiv.org/pdf/2412.05967v1 | Language hooks: a modular framework for augmenting LLM reasoning that decouples tool usage from the model and its prompt https://arxiv.org/pdf/2503.09516 | Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning https://arxiv.org/pdf/2405.06907v1 | CoRE: LLM as Interpreter for Natural Language Programming, Pseudo-Code Programming, and Flow Programming of AI Agents https://github.com/PeterOvermann/TriadicMemory | Triadic Memory: Cognitive Computing with Associative Memory Algorithms https://aclanthology.org/2025.neusymbridge-1.11.pdf | Generative FrameNet https://youtu.be/zmnzW0r_g8k | Forge by Nous Research @ Nouscon 2024 https://youtu.be/cpu6TooJ0Dk | NARS with GPT as natural language channel https://youtu.be/xT4jxQUl0X8 | DeepSeek's GRPO (Group Relative Policy Optimization) | Reinforcement Learning for LLMs What the hell is an SDR? https://en.wikipedia.org/wiki/Hierarchical_temporal_memory#Sparse_distributed_representations & https://en.wikipedia.org/wiki/Sparse_distributed_memory https://youtu.be/ZDgCdWTuIzc SDR Capacity & Comparison https://youtu.be/vU2OZdgBXAQ SDR Overlap Sets and Subsampling https://youtu.be/8WIzIBaLXIs SDR Sets & Unions