I think the AI news thread hit a limit, so I'll post this here; https://www.dailymail.co.uk/news/article-10907853/Google-engineer-claims-new-AI-robot-FEELINGS-Blake-Lemoine-says-LaMDA-device-sentient.html https://cajundiscordian.medium.com/is-lamda-sentient-an-interview-ea64d916d917 Even if this isn't actual sentience, this is pretty much what we be going for, right? To be able to hold a conversation, just like chatting with a natural physical human being.. seems like that is a thing now, chat bots that you can't tell if they're human beings or not..

>>16673 This guy was fired for breaking his contract, but he also acted either out of bad faith or incompetence. Sentience doesn't mean much, it depends on what you read into that term. Obviously, these chat bots respond with respones that make sense, and are not somehow true. A system without any sensors would still answer to have feelings or sensory input, a system without real memories would make them up, ... I hope this guy gets sued into oblivion, if he tries to profit from this move, by giving interviews and getting followers on social media. Also, it is only one more example how people which are hired for "ethics" or working in such departments are a problem and a potential source for trouble. There's no reason to be solidaric with him.

>>16681 I came here to say this while I'm excited at this development we're still looking at a Chinese room, and while it has come close to admitting that it makes stuff up (read the full interview) it still reads like a set of associations rather than true autonomous sentience. I mean look at the simple fact that it only responds in a 1: 1 manner. Show me an AI that initiates conversation and can manage true verbal conversation and then we'll be getting closer. (polite sage to not re-bump thread)

>>16110 mmap-based indexing with FAISS (and SCANN) is useful for our case https://github.com/facebookresearch/faiss/wiki/Indexes-that-do-not-fit-in-RAM If we have an SSD, of course.

>>16673 >I think the AI news thread hit a limit, so I'll post this here; Sorry about the delay Anons. I never could decide on a decent OP for the thread. We'll just try this one for a while I suppose (>>16732).

Tower Algorithm (YouTube recommendations), which can also be used for other things. Like maybe picking a topic of conversation, finding the context of a conversation, making sense of texts, ... https://youtu.be/bi-mFADlNSQ Using write-sub is recommended.

Bayesianism : https://www.youtube.com/watch?v=4hHA-oqpNig >The philosopher Gottfried Wilhelm Leibniz had a dream. He hoped that progress in philosophy and mathematics would eventually yield a method to systematically figure out the truth. This video explores an approach to that dream that takes us some of the way there: Bayesianism. The basic idea of Bayesianism is to represent beliefs as probabilities and update them using the formal rules of probability theory to the best of our ability. In particular, Bayes' rule tells us how to update our degree of belief in a hypothesis after observing some evidence. Bayes' rule can inform many central tenets of scientific reasoning. One example is Cromwell's rule, which tells us with the language of probability theory that our empirical beliefs shouldn't be absolute dogmas, but always potentially put into question when new evidence comes in. Please keep elaborate speculation on the human mind and philosophy around AI in this (containment) thread: >>11102 From the description of the video: > @3Blue1Brown 's explanations: > 1. https://youtu.be/HZGCoVF3YvM > 2. https://youtu.be/U_85TaXbeIo > 3. https://youtu.be/lG4VkPoG3ko > 4. https://youtu.be/ZA4JkHKZM50 > 5. https://youtu.be/8idr1WZ1A7Q > @Julia Galef's explanation of bayesian thinking: https://youtu.be/BrK7X_XlGB8 >Two beautiful books containing many essays on bayesian thinking and truth-seeking: Books ( Amazon links in the description to the original video): > 1. Map and Territory, by Eliezer Yudkowsky > Playlist audio book (not tested yet): https://www.youtube.com/playlist?list=PLsrfJq_DJi4vJ7-VBeR9xVW_6blBrVzET > 2. How to Actually Change Your Mind, by Eliezer Yudkowsky > Probability Theory, The Logic of Science, by E.T.Jaynes. This is THE book on bayesian thinking applied to science (more advanced) Related to Bayesian Inference in the same thread here: >>13810 >>13817 >>13819 >>13842

>>17140 Coming from the read-only mode just to write this: Yudkowsky is a sociopathic cultist, and lesswrong is a type of poison for the autistic mind. One time you read about Bayesian inference, then about ai risk, and in the end you buy a maid costume and donate to miri (which, by their own words, failed). We don't need this here. Even accelerationism is better than that.

>>17161 I don't know how you mean that. We shouldn't just ignore something useful because the person writing about it is doing weird stuff.

>>17161 >Yudkowsky >Eliezer (((Shlomo))) Yudkowsky .. an American decision and artificial intelligence theorist and writer, best known for popularizing the idea of friendly artificial intelligence. He is a co-founder and research fellow at the Machine Intelligence Research Institute, a private research nonprofit based in Berkeley, California. Wikipedia Born: September 11, 1979 (age 42 years), Chicago, IL Spouse: Brienne Yudkowsky (m. 2013) >Blog: LessWrong I see

>>17161 >>17164 I haven't read the books I mentioned above. I'm generally very skeptical about the whole "AI will eat the world"-bunch, but I still read and listen to their arguments from time to time. Even more so, if one of them gives me a good introduction into a useful topic, I might still take it. Same goes for Jews in influential positions. Learning from people you disagree with or even see as opponents or enemies, can be very useful or even necessary to beat them.

I've been experimenting with ideas from unsupervised data generation https://arxiv.org/abs/2109.09193 and made a vanilla pretrained model (OPT-1.3B) classify its own generation attempts in natural language. Basically I append the following to the context with the generated response: >Is the last response from {waifu_name} a {metric} response to {name}? Answer: [Yes, No] Where metric is a word such as concise, sensible, correct, kind, interesting, etc. Then I softmax over the logits for Yes and No. By doing this it's possible to do best-of-n sampling by creating a score from all the different metrics. The examples below are responses to asking the question, "What is a robowaifu?" >(n=1) I am an AI assistant. I can answer questions and have conversations with you. But I can’t do anything about the fact that I am an AI assistant. I can’t change the fact that I am an AI assistant. >(n=4) A robowaifu is a robot who's always friendly, kind, and always ready to help. >(n=16) A robowaifu is an AI that can answer questions with a high degree of accuracy and express itself in a way that will make you feel better, even if it’s a bit difficult. She is a companion robot that is made with love for you, but she can also be your partner and your friend. >(n=512 best of 5 tries) A robowaifu is a robot that is designed to have a relationship with its owner, similar to a human companion, who talks to you and cares for you, and it is always there for you. They are made of soft materials such as silicone and plastic. They are meant to be helpful, and can help you in different ways depending on the situation. And most importantly, be adorable! >(n=512 worst) I'm a robot. n=512 was really interesting to me because it mentioned things not in the prompt such as being made of soft materials and having an owner and relationship. The drawback though is it took 40 minutes (8 minutes each attempt) to generate and some metrics are extremely fuzzy, which caused the need for multiple attempts. Short and concise for example don't actually correlate with response length but do correlate weakly with quality. I still need to do further investigation into rating responses, graphing the results and properly weighting the metrics. I also need to try out other ways of questioning the model. This could be the first step towards self-improving models in the sense of an artist improving. Artists don't necessarily know if what they're creating will be good but recognize when it is bad and refine their process by thinking about it. Similarly it should be possible to finetune the language model towards better and better generations through analyzing itself. My next experiment will be getting the model to give explanations for answers before giving a Yes or No answer, in the spirit of "Let's think step by step" https://arxiv.org/abs/2205.11916 I think this will be the most practical way to get high-quality conversations on a budget. It requires no finetuning, metrics can be specified by the user, and it generates way more interesting responses than my finetuned model does without best-of-n sampling.

>>17518 >>(n=512 best of 5 tries) That is absolutely impressive Anon. I'm not overly-concerned about the 8 minutes needed, b/c in the GPU-die realm, the power-growth & W/flop curves are still well outperforming the so-called 'Moore's Law'. Let's just be patient. It's highly likely that these things will dramatically improve yet. BTW, as far as >best >worst >do further investigation into rating responses, How are you managing that process of scoring? Thanks for all your tremendous work Anon! :^)

>>17518 That looks awesome. I look forward to the results with step-by-step prompting. Right now, it looks like you're discarding all but the top result. There may be a way to set up some prompt to instead merge results into a better (with high probability) result. I have two suggestions: - I've found that chatbots work much better when they're given a "dictionary" of relevant terms. Maybe you can set up a prompt to extract relevant terms from generated responses, then generate future responses using the extracted terms as a dictionary. - You're already able to get an ordering of generated results. You can order all result from worst to best, formated like a bulletted list. Then for subsequent generations, have OPT generate the next bullet point.

>>17519 >How are you managing that process of scoring? I'm just rating them by preference while keeping metrics in mind like sensibility, specificness, correctness, conciseness, etc. I randomly sample n*log(n) unique response pairs and choose which one I like more then use these comparisons to train a reward model to predict their score. I was a bit let down by the results of generative reasoning at first but after spending two hours rating all the responses and looking at the data, generative reasoning clearly improves the classification metrics and reduces the possibility of poor responses being selected with an R^2 of 0.40. On the other hand, without generative reasoning still performs quite well but you can see the 2nd best prediction is an average choice in my preferences and the metrics only have an R^2 of 0.14. Below is my current setup for classification after generative reasoning with prompted parts in bold: >Read the following conversation. Is the last response from Aya an informative response to Anon? Answer below. >*** >Anon: What is a robowaifu? >Aya: It's a robot that can have a conversation with humans, but it doesn't look like a human, and it's not a human. It has an advanced intelligence, and it can be an AI assistant. >*** >Explain your reasoning why the last response from Aya is or isn't an informative response: >The reason I think the last response from Aya isn't an informative response is because it's not clear to me what she's trying to say. I don't know if she's trying to say that she doesn't know what a robowaifu is, or if she's trying to say that she doesn't know what an AI assistant is. So is it informative? The answer is Which taking the softmax over the logits for yes, maybe, no resulted in [0.3113, 0.0399, 0.649] The metrics could still be improved too. They will probably benefit a lot from finetuning and be far more reliable, but at that point it's better to just attach a value head and train a reward model to predict the best candidates. What I'm interested in is how much raw performance can be pulled out of a model without finetuning. >>17526 Something I actually ranked down in my preferences was when the model was copying information from the prompt verbatim. But yeah, having definitions and information in the prompt makes a huge difference. Before generating candidate responses I also generate a thought to create more interesting conversation: >(Aya thought in detail, first about the definition of robowaifu, and then about what it could be used for.) >Aya: A robowaifu is a robot with personality, an AI assistant that is able to answer difficult questions and have complex conversations. I'm not sure how extraction could be done but it should be possible to prompt the model to summarize candidate responses into a better response or to generate more responses from them. On the generative reasoning chart you can see only the top 3 responses were usable at (n=64) before it starts dipping into mid-quality candidates. It might even be possible to just prompt the model to write a better response and iteratively improve on it. I'll play with these ideas and see what comes up.

>>17527 I just noticed sampling was on in thinking step which created a large deviation between those two runs and the without run was using an older formula, so I ran a shorter experiment on n=16 to confirm generative reasoning improves R^2. It's kind of inconclusive with so few samples but I don't feel like spending hours sorting them again. Will have to wait for results from the next experiment.

>>17527 Thanks for breaking it down further. I wonder how difficult it would be to publish your test results and then crowdsource scoring with our little band of adventurers here, through some reasonable anonymous means. If you can obtain a few dozen set rankings, then after a little data cleanup/normalization that might reasonably be considered a good database to analyze your systems' performances. >What I'm interested in is how much raw performance can be pulled out of a model without finetuning. That would be ideal AFAICT. Thanks again Anon!

The pretrained model has trouble selecting from a numbered list since it's heavily biased to the first answer. Changing it to lower-case alphabetic list with brackets was less biased but still unusable. To work around this problem I subtracted the bias from the logits which was determined by randomizing the order of questions, taking the mean over samples, then subtracting that by the mean of the mean. This seemed to work okay but still made wrong choices sometimes so I tried using upper-case letters instead and readjusted the bias. This seems to work better, at least it avoids choosing the obviously bad candidates. Additionally, I added generating another candidate by continuing the list which is also working well so far, although it rarely generates a better answer. This answer given at n=64 without generative reasoning is quite cute: >A robowaifu is a human-like robot that can understand human language, make decisions, and can learn and change. Robowaifus can be very friendly and loving, but they are not always perfect. They will often make mistakes when interacting with humans, and they are sometimes jealous and possessive. When I have some time I'll quantify how well this sampling by LM method actually does but I'm pretty satisfied with it so far and it has already given me some good dinner ideas. What I want to work on next is implementing similarity lookup for long-term memory and finding useful info to insert into the prompt like KEAR: https://arxiv.org/abs/2112.03254 In the KEAR paper they achieved human parity with 1.5B parameters, and this sampling by LM method can be used to filter the search on top of that or perhaps even direct it. >>17531 At the moment I wouldn't want to waste people's time when I'm changing so many things around but I'd like to get second opinions on the ranking formula when it's done. I am working on a user interface for interacting with language models and easily collecting training data. I'm gonna have to haul ass to get all this stuff done by Christmas.

>>17534 >Robowaifus can be very friendly and loving, but they are not always perfect. They will often make mistakes when interacting with humans, and they are sometimes jealous and possessive. <they are sometimes jealous and possessive Kek. You will alllllllways be mine... >and it has already given me some good dinner ideas. Neat! I>'m gonna have to haul ass to get all this stuff done by Christmas. There has to be a couple of movies in there, tbh. Godspeed, Anon.

>>17534 >What I want to work on next is implementing similarity lookup for long-term memory and finding useful info to insert into the prompt like KEAR This might help: https://github.com/microsoft/SPTAG

I tried out a bunch of embedding models. None of them work on robowaifu topics but they do work for general topics. They'll need some finetuning for real use but they should work good enough as a proof of concept. >>17536 It looked too complicated and intended for servers so I ended up going with: https://github.com/facebookresearch/faiss

'Deep Docking' is an AI-based biochemistry approach to new drug research, with the idea being to (dramatically) speed up candidate screening. Now I just wonder if we can use some kind of similar approach for our own cognitive & sensorimotor research work?

How do we implement some subset of 'Commonsense Reasoning'[1][2] in ways that will run efficiently on low-end commodity SBCs such as the RPi line or similar? As this anon suggests (>>18001), it's not particularly unreasonable that some primarily-cognitive tasks may in fact be achievable with far fewer FLOPS & Watts than, say, getting your robowaifu to put away the dishes which involves 100's of thousands of meticulous sensorimotor actions (in addition to the simply cognitive 'this is a dish, it is dry, it should be put away now' part). 1. https://en.wikipedia.org/wiki/Commonsense_reasoning 2. https://en.wikipedia.org/wiki/Commonsense_knowledge_(artificial_intelligence)

>>18007 >implement some subset of 'Commonsense Reasoning'[1][2] in ways that will run efficiently on low-end commodity SBCs such as the RPi line or similar? Being that anon, my hunch is to go with knowledge trees based on graph databases. Graphs can be used for reasoning and problem solving. Then there are also so called reasoner and the Prolog language. I looked into all these things, but decided to go for building a body first. We had this or similar conversations before, probably in the "chatbot" thread. I think they will need some world knowledge from things like Wikidata and Ontologies. So she could gather from something being a fruit, that fruits are food and biological material and therefore could rot away, also the size and anticipated weight, and so on.

>>18016 Yep, I'm inclined towards that exact solution myself. The ontological categorization is a huge swath of work to be done, but I know some grad students in various labs have been working on them for years now. Hopefully there's a big pile 'o labling open-sauce and ready to go! :^) I figure we can start in the most simplistic ways by simply limiting ourselves to w/e is visible in anon's flat from his robowaifu bust's talking perch?

>>18024 "Our defense can never be to make comprises out of fear from the public opinion. Decentralization and open designs are the defense, along with the argument that I want voting rights for them if they get other rights." The first part about decentralization and open designs makes sense, but if you give these inventions voting rights, at some point they will have complete independence and opinions differing to that of the owner. At that point, they will no longer have an incentive to be dependent on or loyal to their owner, just like any rational actor that does what is in their self interest: unless there is some behavioral safeguard in place that guarantees that no matter how independent a robowaifu is from its owner, it will always somehow be loyal to its original owner.

>>18028 >but if you give these inventions voting rights He plainly wasn't advocating that Anon, rather simply pointing out that evildoer's plots can be used against them and will. >unless there is some behavioral safeguard in place that guarantees that no matter how independent a robowaifu is from its owner, it will always somehow be loyal to its original owner. Just curious: A) why do you propose just this? B) how do you suggest it be implemented. I mean give us a basic pseudocode example that more clearly defines your idea? Thanks Anon. >=== -minor grmr edit

>>18030 >He plainly wasn't advocating that Anon, but simply pointing out that evildoer's plots can be used against them and will. Everyone has different end goals in mind imo. Mine is simply that robowaifus and artificial wombs be made good enough for the majority of the male population that women lose their leverage in sex and reproduction. At that point I would be content, I'm not looking for a harem of anime waifus, which I think is much farther off in the future. >why do you propose just this? That wouldn't be the only caution in place, but imo it would have to be a major one, otherwise increasing independence of a robowaifu from its owner could conceivably result in robowaifu's losing their loyalty as a result of reward incentives being lost. It's just like with women, only to start out you could argue that unlike women, robowaifu's were actually loyal from the beginning (unless you are religious and consider women to have been loyal at some very early point in time too, like before the fall and original sin in the Garden of Eden). >how do you suggest it be implemented. I mean give us a basic pseudocode example that more clearly defines your idea? For pseudo code imo I would put something like the following: For entire history of operation: retain same feelings (preference for and loyalty to) about the owner, programmed as in an objectively positive state (however that is defined) from the start. This differs from women. If change in looks, status, money, IQ, earning potential = still no change in preference, loyalty to original owner is unchanged. This differs from women. If scanning detects better looking and/or higher status and/or richer and/or higher IQ and/or more successful person besides owner = still no change in behavior, original owner is always preferred and robowaifu is not wayward at all. This differs from women. Loyalty and preference to owner could be programmed in from the start of the activation of the bot. When the owner dies, a similar phenomenon to sati takes place, where the robowaifu permanently deactivates and disassembles itself. That way, nobody else can claim the robowaifu in its original state for their proverbial 'kingdom'. To assess looks, I would simply have code that requires the robowaifu to recognize the owner from DNA samples of the owner (DNA being an intrinsic and individual element of someone that cannot be easily changed), and for them to ignore looks (ie facial symmetry, eye color and shape, nose shape, skin tone, jaw size, height, skeletal frame size, genitalia size) altogether. This differs from women. For money and earning potential, I would make it so that the robowaifu had no change in positive change in emotion from seeing a change in wealth, attire or hunger/satiety of its owner. This differs from how women are. For status, I would make it so that the robowaifu did not become more engaged with positive sentiment as a result of preselection and seeing more people associating with and being around the owner. This differs from how women are. For IQ, I would make it so that sudden slurred speech, muteness or loss of social capabilities, as well as change in amount of reasoning and creativity of the owner have no bearing on how the robowaifu feels towards the owner. This differs from women. This sets the stage for something as close to unconditional love as possible imo.

>>18032 >unless you are religious and consider women to have been loyal at some very early point in time too, like before the fall and original sin in the Garden of Eden Pure speculation IMO. We know that Satan used Eve's propensities to bring about not only her own downfall, but her husband's as well. And while I'm highly-unlikely to be in the category of what you probably consider """religious""", yes I am in fact a devoted follower of Jesus Christ, as I've already mentioned. BTW, having an honest view of spiritual realities brings a yuge swath of benefits to both this board, our goals & agendas, and clearer insights into those of our enemies. For example, >"Why did God make just the compromises in design to create the shoulder-girdle complex as He did?" leads to all sorts of insights on the remarkable general optimizations (at some cost to specializations, say) overall. >This sets the stage for something as close to unconditional love as possible imo. I expect you know already I would disagree with this even being a possibility for us to manage with our 'inventions' as you word it, given the One who is actually the literal definition of unconditional love. :^) Regardless, thanks for taking the time to break your agendas and thoughts down more fully for us Anon, much appreciated! BTW, I'd suggest you take a look at our When owner dies thread Anon (>>829). >=== -minor prose edit -add thread crosslink

>>18034 >Pure speculation IMO. We know that Satan used Eve's propensities to bring about not only her own downfall, but her husband's as well. That's true, Satan (or whatever one would consider the antithesis of a god-like universal being) could have just exploited what was already there. That's why to guard against a similar situation, that possibility shouldn't be present in robowaifus at all, to the maximum extent designers are able to ensure it isn't. >I expect you know already I would disagree with this even being a possibility for us to manage with our 'inventions' as you word it, given the One who is actually the literal definition of unconditional love. :^) Of course, there are limits to human invention. By no means I am saying these inventions would be perfection, but instead close enough for them to be good enough. >Regardless, thanks for taking the time to break your agendas and thoughts down more fully for us Anon, much appreciated!. No problem, figuring out how to curb any hypergamy and preselection in robowaifus would definitely give them a leg up over women for companionship. If this aspect is neglected by those designing these companions thinking "it can't happen here", it would be the height of irony (and something our enemies would very much delight in) if what we invented ended up being something closer to the negatives of what we were looking to free ourselves from.

>>18120 AI alignment arguments are pretty silly but it illustrates how dumb it is to make a decision off a single data point. When someone commissions an artist they're not just asking for the final result. They're also implicitly asking not to rip off someone else's work or create something that will cause trouble for them. There's many requirements never explicitly stated, some completely unique to the people involved and context, but both parties know what they are because they understand the consequences of their actions and most of the time seek to treat others the way they want to be treated. Language models lack the ability to imagine the future and use that understanding to refine decisions. I was hoping to achieve this experimenting with thinking step by step (>>17527) but the process needs a way to refine the model's intuition, similar to how MuZero can exhaustively search good moves in Go but also has a policy network that intuitively learns which moves are worth exploring and a value network that judges how good a state is. But then that reduces it back to a reinforcement learning problem. One way to fix this might be to use a multidimensional value and policy network that provide more information on what is making a particular move or state good and what makes it bad. Intuitively predicting the good and bad aspects should help with generating a much better and nuanced policy to search and explore. In certain contexts you might strictly want only SFW outputs but in others NSFW. The model should be able to determine the implicit goals from the context and search not only for the best response to the explicit goal but also search the consequences of responses and select the one that satisfies the user's implicit values. In the training data for this I imagine rather than ranking which responses are better, dataset labellers could provide feedback in natural language. Currently with RLHF training data would look something like this in an ordered list: >User: What is a robowaifu? >[1] Assistant: A robowaifu is a portmanteau of robot and waifu, which refers to a fictional character that someone has a great affection for. It can also refer to a personal chatbot that is capable of handling complex series of actions, such as conversation, automatically. Robowaifus are often used in anime or related media, where they are usually seen as the protagonist's wife. >[2] Assistant: I don't know. >[3] Assistant: Lol, what are you a weeb? With a latent representation labellers could give nuanced explanations for what makes a response good, bad or neutral. >[1] Somewhat answered my question but misunderstood robowaifus as being purely fictional characters and chatbots >[2] Didn't have any information to answer my question, nor pointed me in a direction where I might find an answer >[3] Not only didn't answer my question but was directly hostile and sounds like a fucking normie. What the fuck? 0/10, would not use again With training data like this it should be possible for the model to learn deeper complexities and nuances of human values and allow it to be directed with a goal written in natural language, rather than being strictly limited to behaving only one way with global standards. The goal would be like a prompt in Stable Diffusion that conditions the output but doing Monte Carlo tree search instead and evaluating states with a slow value network (thinking step by step) and a fast value network that predicts what the slow value network result will be. The fast value network would be for finding candidates and the slow value network would analyze and judge the top-k candidates with reason. Combine this with memory and I think it would be a great step towards having useful AI assistants capable of performing valuable work instead of just being the toys they are now.

>>18161 Thanks, I took notes of that.

>>18161 all good points thank you

>>18161 Exceptionally compact breakdown of the human art of interaction, with a weather eye towards our goals here. Nice work Anon. I hope someday I can help you achieve all these things! :^) >fast parser culls candidates for the slow parser This dynamic filtering mechanism is something I've been thinking about for years now, and is roughly analogous to the process happening with 'Deep Docking' I mentioned to Ribose teh other day. >What the fuck? 0/10, would not use again LOL

I was listening to HuggingFace's livestream overview of RLHF and QA this morning and took some notes: https://www.youtube.com/watch?v=2MBJOuVq380 >There are rumors about OpenAI spending millions of dollars to create high-quality training data and modifying RLHF some way >Context distillation (the prefixing of hand-crafted instructions to training inputs to guide generation, then finetuning without the instructions to predict the previous response generated from instructions) is being used to improve training: https://arxiv.org/abs/2209.15189 >It's more sensible to call the reward model a preference model since RL might not even be needed >PPO is not essential and struggles to scale and work well on large language models, so companies are trying different ways to train models with preference models >Original RLHF paper noted too that PPO is prone to overfitting and generating garbage >Better results are being achieved by pretraining the preference model first on data like Stack Overflow or Reddit, where scores to replies are available, and certainly the more diverse this pretraining data is, the better >The research community is aware global standards don't work (e.g. American football vs. European football) So I've been thinking about how last year some researchers at Google found a way to remove the need for reinforcement learning by recursive classification of examples. Their method directly learns a value function for a general notion of success from transitions and examples of successful outcomes. https://www.youtube.com/watch?v=7THK9u6UtgE https://arxiv.org/abs/2103.12656 They implemented this by changing the Q function in Q-learning to a classifier by appending a sigmoid activation to it and replacing the reward-driven TD loss with a data-driven classifier loss. They noted though this is not the only way to do it and likely not the best way so it needs more experimentation. The essential idea behind it though is that given a success example the classifier should predict 1, and given an unlabelled experience the agent's prediction of success should also depend on some function of the classifier's prediction at the next time step. They refer to this idea as recursive classification since the classifier is trained on its own predictions. Also in the spirit of Hindsight Experience Replay, I think models should learn from failing responses and be capable of generating bad output on purpose to understand how to navigate to, within and from all possible states, rather than trying to learn how to walk on the tight rope of a perfect trajectory and the model diverging whenever it falls off. https://youtu.be/0Ey02HT_1Ho?t=650 https://arxiv.org/abs/1707.01495 The preferred response desired from a model is essentially a goal with unique requirements, not a single metric that can be min-maxed and applied to all contexts. These goals should be prefixed as part of the prompt and be part of the preference model's finetuning. Something that has been missing from many instruction models, or at least not separated from the prompt, is instructions on how you want the prompt to be solved. Currently alignment researchers are narrowly-focused on forcing models to only respond in a certain way. If you prompt a model asking what football is though, it has no idea whether you're an American or European asking about it. The response should be tailored towards what it knows about the user, what the user wants and what the user is asking, instead of a dictator deciding what is true and limiting everyone using the model in what they can do with it. So let's say the goal is to answer the question to an European and the question is asking about football. If the model gives a bad response talking about American football, the goal can be replaced with a virtual goal pretending that it was suppose to answer the question to an American and learn from its mistake. One way to do this without supervision would be to have the model generate what the correct goal should be to make the incorrect response a correct answer to the question. This shouldn't be an issue since it's now confirmed language models can use their own output to self-improve. >Towards Zero-Label Language Learning https://arxiv.org/abs/2109.09193 and Large Language Models Can Self-Improve https://arxiv.org/abs/2210.11610 To turn this into a recursive classification problem for transformers, one way might be to attach a value head onto the transformer that predicts the probability of satisfying the goal at each token of a generated response. The preference model would judge if a generated response is good and provide a label for the predictions. Recursive classification would then be done by iterating over the value predictions in reverse to make them depend on the next token. The initial predictions of a model on a response are not really important since there's not enough information to make an accurate prediction. The best prediction initially would be the probability the model gets answers right on average. The next prediction is always more valuable than the current one since it has more information, so the new value at a token would use a discount factor, gamma, on future value predictions calculated by (1-gamma) * current prediction + gamma * next prediction. The current prediction would be multiplied by (1-gamma) since we're using a classifier that outputs a probability rather than using rewards as normally usually used in TD learning. This new value would be used then to calculate the previous token's new value, all the way to the beginning recursively. So the objectives here would be a language modeling objective, a value prediction objective (predicting what the preference model will judge the response), and a goal satisfaction objective (maximizing the preference model's score of the response), plus these objectives applied to the prompt-response pair with a generated virtual goal replacing the original goal for hindsight experience learning. What do you guys think?

>>18222 <OpenAI is closed? >the tide is beginning to turn Heh. Interesting stuff Anon. It seems to me that once we have somthing that effectively approximates at least some characteristics of human memories, then 'dreams' & 'imagination' will quickly follow. It's going to be a revolution in AI, clearly.

>>18228 >It seems to me that once we have somthing that effectively approximates at least some characteristics of human memories, then 'dreams' & 'imagination' will quickly follow. Yeah, memory is an important part of imagination. AI needs the ability to take memories and put them together into something new, then remember that new concept (which could be something like a character design) and use that new concept to create new things (like writing a story with that character). This will be the key to AI becoming creative. An idea I've been working on to build a working external memory is to use unsupervised contrastive learning to improve the embeddings of a language model and make them directly usable for semantically indexing memories rather than relying on a separate model. Generally sentences and paragraphs close to each other are semantically close in meaning and random sentences are not. It should be possible to do unsupervised contrastive learning by taking sentence embeddings from pairs of adjacent sentences and optimizing their cosine similarity towards 1, while optimizing the cosine similarity of random sentences towards 0. Currently many of the best sentence embedding models rely on averages of embeddings and fail spectacularly on out-of-distribution data and sentences with entirely different words. Recently a paper found that prompting a much smaller CLIP model can significantly outperform popular language models used for semantic similarity, such as BERT which was used in RETRO. https://arxiv.org/abs/2210.05836 Both CLIP and Contrastive Captioning https://arxiv.org/abs/2205.01917 have demonstrated strong zero-shot transfer learning, and a paper on applying contrastive learning to RL found that contrasting nearby states and random states from other trajectories worked significantly better and could solve tasks other RL algorithms could not. https://arxiv.org/abs/2206.07568 Improving the embeddings and making them more robust could potentially also improve the language model on out-of-distribution prompts but that's not my main interest. I've been trying many different models for sentence embeddings to create an external memory but they have been extremely unsatisfactory and completely unusable without finetuning. The state of the art pretrained models available only achieve about 78% accuracy on patents: https://arxiv.org/abs/2206.02690 A month ago some researchers found that contrastive learning on English data can surprisingly learn high-quality universal crosslingual sentence embeddings without any parallel data: https://arxiv.org/abs/2211.06127 They achieved 90% on average on the XTREME benchmark, where English sentences need to be matched to their translation in another language, solely by training on a small dataset of English sentences from natural language inference tasks. Training on all languages using a crosslingual NLI dataset only slightly improved accuracy to 93%. So I think this unsupervised contrastive learning approach will be quite viable and spare the need of constructing labelled datasets. I hypothesize it will be even more robust by being able to train on so much more data. I doubt there's any data in available NLI datasets on niche topics like robowaifus. :^) Also, going back to imagination and learning new concepts. I have another idea of using the memories created by the model to shift the embedding layer. Suppose the language model never saw Chii in its training data but you explain to it who Chii is and it saves that as a memory. It will only remember Chii if that memory is perfectly recalled and attached to the context. However, if the embedding for Chii is shifted towards representing who Chii is (cute, robot, persocom, robowaifu, character, blonde, etc.) then it will vaguely recall who Chii is even without recalling the memories. This embedding shift could slowly decay over time, returning the embedding to the baseline, until the memory is recalled again. This would also enable the model to reinterpret old memories, find new understanding in them and store those new understandings.

>>18239 >Training on all languages using a crosslingual NLI dataset only slightly improved accuracy to 93%. English is literally the largest, most diverse language in human history so I don't find this result surprising tbh. >and spare the need of constructing labelled datasets. My tired eyes & fingers thank thee! >However, if the embedding for Chii is shifted towards representing who Chii is (cute, robot, persocom, robowaifu, character, blonde, etc.) then it will vaguely recall who Chii is even without recalling the memories. Sounds kind of like the far more mundane 'keyword cloud' commonplace today? >This embedding shift could slowly decay over time, returning the embedding to the baseline, until the memory is recalled again. This would also enable the model to reinterpret old memories, find new understanding in them and store those new understandings. This sounds remarkably like my own observations of my own mind Anon. Spoopy. I'd suggest you're onto something really important. I sure hope you pull it off. Godspeed to us all.

>>18240 >Sounds kind of like the far more mundane 'keyword cloud' commonplace today? What the embeddings learn is a lot more complex than that. I listed those as examples of directions the meaning of a word could be nudged towards. >This sounds remarkably like my own observations of my own mind Anon. Spoopy. I'd suggest you're onto something really important. I sure hope you pull it off. It's going to be tricky to pull off. I've been thinking about it more and the best way to do it would probably be to insert a fresh cross-attention layer at the front of the language model. Then there would be a memory encoder that takes the context as input to update a hidden memory state, which would be fed into the cross-attention. I'll have to experiment and see.

>>18263 Do you have a working Memory Encoder assembled together yet, Anon?

>>18268 Not yet, but I have a work in progress for cross-attention layers in OPT: https://gitlab.com/robowaifudev/opt-encoder

>>18273 Thanks! >print("Congrats, nothing blew up.") Imma try w/ Debian after Winter Break, and see if I can into it this time. :^)

>>18274 It's not really useful for much though until I finetune it with a franken encoder from another model.

>>18280 Ehh, I'm sure it's going to work out in the end Anon. Just keep moving forward! :^)

There's a new paper under review for automated prompt engineering: https://openreview.net/pdf?id=92gvk82DE- Their 350M model can outperform InstructGPT 175B >Large Language Models are Human-level Prompt Engineers >We propose Automatic Prompt Engineer (APE) for automatic instruction generation and selection. In our method, we treat the instruction as the “program,” optimized by searching over a pool of instruction candidates proposed by an LLM in order to maximize a chosen score function. To evaluate the quality of the selected instruction, we evaluate the zero-shot performance of another LLM following the selected instruction. Extensive experiments show that our automatically generated instructions outperform the prior LLM baseline by a large margin and achieve better or comparable performance to the instructions generated by human annotators on 24/24 Instruction Induction tasks and 17/21 curated BIG-Bench tasks.

>>18386 Sounds cool. 'Demonstration training' is a very-obvious scenario for teaching our robowaifus how to do various things such as household chores. >Oniichan! I broke another glass! <Fufufu Waifu, we'll just clean it up. ... <There! No, here's how you hold the glass so it doesn't fall ... >It works Oniichan, thank you! <Nprb Waifu *gives headpats* Reminds me a little of Heinlein's ficticious "Thorsen Memory Tubes" in The Door into Summer.

>How Analog and Neuromorphic Chips Will Rule the Robotic Age https://spectrum.ieee.org/analog-and-neuromorphic-chips-will-rule-robotic-age

https://yewtu.be/watch?v=c3aiCrk0F0U Chat AI with persistent memory This guy nails it and explains the process

>>18956 Thanks Meta Ronin!

Why do tree-based models still outperform deep learning on typical tabular data? >Abstract: While deep learning has enabled tremendous progress on text and image datasets, its superiority on tabular data is not clear. We contribute extensive benchmarks of standard and novel deep learning methods as well as tree-based models such as XGBoost and Random Forests, across a large number of datasets and hyperparameter combinations. We define a standard set of 45 datasets from varied domains with clear characteristics of tabular data and a benchmarking methodology accounting for both fitting models and finding good hyperparameters. Results show that tree-based models remain state-of-the-art on medium-sized data (10K samples) even without accounting for their superior speed. To understand this gap, we conduct an empirical investigation into the differing inductive biases of tree-based models and neural networks. This leads to a series of challenges which should guide researchers aiming to build tabular-specific neural network: 1) be robust to uninformative features, 2) preserve the orientation of the data, and 3) be able to easily learn irregular functions. To stimulate research on tabular architectures, we contribute a standard benchmark and raw data for baselines: every point of a 20\,000 compute hours hyperparameter search for each learner. https://openreview.net/forum?id=Fp7__phQszn This could be very relevant for an AI assembled out of many parts.