have I been posting in the wrong place?

I'm not going to keep saying that we've been visited by people running with our ideas but it's hard not to wonder https://wefunder.com/destinyrobotics/ https://keyirobot.com/

>Woman and third worlders stealing our ideas to steal from simps Topkek Atleast they're not as blatant as lilium robotics. https://www.liliumrobotics.com/Full-Body/ They are open sourcing their software so somewhat thread relevant. https://github.com/lordNil/lilium

>>16740 Yea I posted about Lilium (not to be confused with Lillim) Robotics in the other thread. Remember these are going to be the goofy first attempts. Learn what we can I guess from their mistakes and successes. Better to stay on top of our "competition" than to put our heads in the sand. Side note: Now would be a great time if never to get a secret multi-million dollar inheritance or win a grant or something. Hate to watch everyone else having all the fun

I posted in the last thread, so this is a repost. (((they))) already taking some swing at "rights for AI's, robots have feelings too!!!", a situation similar to vegans, or le stronk independent womyn, groomers, etc... > that staged LaMDA news They will make all sorts of AI's - a protected class, just like they protect the groomers nowadays, it's like "screaming" itself : > even if you have a robot, you won't have access to its system a.k.a. (((cloud ai))) a generation raised on fear-porn movies, they use their fear now to justify stricter controls.

>>16738 Nice cashgrab. So many people just want to buy something. And of course ASAP, it doesn't exist though. >>16744 People supporting the freedom of chatbots have a very limited self-awareness when it come to their programming through media, education and politics.

>>16740 I just looked into this more deeply. - The CEO seems to be an Instagram model, looking into more business opportunities. She probably got told that the time for this is limited, not only in regards to her personally but in terms of men having alternatives. - They pitch their robot as a solution for "loneliness". That's something I hate with a lot of people outside of the sphere of real robowaifu enthusiasts. They automatically assume it has to be about being lonely. But not being content with the current state of women or not meeting their standards is not the same than being lonely. - Then their video also claims to create a humanoid for helping at home. Which might be possible at some time, but certainly not if they want to come out with some product soon. 25M are burned fast in the US (Miami). I think, short term you can either have something more like a animated lovedoll with AI, that won't walk, or a mobile robowaifu with wheels. Btw, on their page they claim $3500 per robot, prototype available next year. Sure. - They seem to have noting so far, their video shows InMoov, I think, and that video of the face looks like some animation. - Yeah, look at that face. It's show how much beauty standards in the US are taboo. Wouldn't be surprised if they would still get attacked for "fetishizing asian women" and her skin being to pale. >>16740 Yeah, I like Lilium Robotics much more. Didn't try their software but at least it looks like they're into Open Source. They're also using common hardware for their system. Also, Lilly's design is rather bold.

>>16752 Don't waste your time on watching this. It was hard: https://www.youtube.com/watch?v=NAWKhmr2VYE

>>16753 ew, absolutely horrifying post-wall face!

If nobody is going to post about it, I will. Two very impressive papers came out. The first one is Deepmind's breakthrough work on RL exploration via novel, simple and powerful self-supervised learning objective, which finally conquered Montezuma's Revenge (!) and most DM-HARD-8 tasks. The second one is an academic tour-de-force devising novel scheme of training a CLIP-like contrastive semantic model as a sufficient surrogate reward for training an agent which passably executes some tasks in minecraft environment. This is a way forward for training from human-generated youtube tutorials. Both of these works are significant and can be applied to our cause, albeit they require moderately large compute (large by the standards of an amateur, moderate by the standards of a good US org). At the very least, agents trained via these objectives could be used as dataset generators for our would-be agent. If we are to use these innovations for our projects, we need to start a semi-closed community to test approaches to distributed computation and to guide the effort of recruiting volunteers into the computation graph. 1. BYOL-explore https://www.semanticscholar.org/paper/BYOL-Explore%3A-Exploration-by-Bootstrapped-Guo-Thakoor/54d1fcc284166e7bbd5d66675b80da19714f22b4 >We present BYOL-Explore, a conceptually simple yet general approach for curiosity-driven exploration in visually-complex environments. BYOL-Explore learns a world representation, the world dynamics, and an exploration policy alltogether by optimizing a single prediction loss in the latent space with no additional auxiliary objective. We show that BYOL-Explore is effective in DM-HARD-8, a challenging partially-observable continuous-action hard-exploration benchmark with visually-rich 3-D environments. On this benchmark, we solve the majority of the tasks purely through augmenting the extrinsic reward with BYOL-Explore’s intrinsic reward, whereas prior work could only get off the ground with human demonstrations. As further evidence of the generality of BYOL-Explore, we show that it achieves superhuman performance on the ten hardest exploration games in Atari while having a much simpler design than other competitive agents. 2. MineDojo https://www.semanticscholar.org/paper/MineDojo%3A-Building-Open-Ended-Embodied-Agents-with-Fan-Wang/eb3f08476215ee730d44606b96d1e24d14f05c1d >Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited and manually conceived objectives, thus failing to generalize across a wide spectrum of tasks and capabilities. Inspired by how humans continually learn and adapt in the open world, we advocate a trinity of ingredients for building generalist agents: 1) an environment that supports a multitude of tasks and goals, 2) a large-scale database of multimodal knowledge, and 3) a flexible and scalable agent architecture. We introduce MINEDOJO, a new framework built on the popular Minecraft game that features a simulation suite with thousands of diverse open-ended tasks and an internet-scale knowledge base with Minecraft videos, tutorials, wiki pages, and forum discussions. Using MINEDOJO’s data, we propose a novel agent learning algorithm that leverages large pre-trained video-language models as a learned reward function. Our agent is able to solve a variety of open-ended tasks specified in free-form language without any manually designed dense shaping reward. We open-source the simulation suite and knowledge bases (https://minedojo.org) to promote research towards the goal of generally capable embodied agents.

Many of you may have noticed a shilling week (((OpenAi's))) GPT-3 on 4chan's /pol/. Today 22.06.2022 the neural network started giving pilpul i.e. passive-aggressive mental gymnastics, facts avoiding, etc. > in two words - another nn was neutered Expect an article from openai about "how evil racists tried to ruin gpt-3"

By this point it should be obvious that large generative multimodal models are here to stay. The experiment shows us that 20 billions of parameters is enough for implementing quite fine, abstract artistic ability. 3 billions is enough for less abstract prompting. You could likely run this model on an RTX3090, if you optimized it for inference. Of course they won't give you the weights, that's why a group of people needs either to pool funds and train their own model, or to train it in a distributed manner, which is harder.

>>16775 >>16779 This is very good to see. I'm glad we're seeing all of this progress, and might be able to implement some of it in our future robowaifus. So the can create interesting dishes and even imagine their own stories or become hobby artists in their free time. >>16775 > If we are to use these innovations for our projects, we need to start a semi-closed community to test approaches to distributed computation and to guide the effort of recruiting volunteers into the computation graph. I generally think it's a good idea for sub projects of the bigger robowaifu project to look for people outside of this small group here. Our project seems to only appeal to a minority for now. One could look for an angle, how a part of it could be used for something else, and pitch it to people interested in that. Then come back with the result.

>>16737 No you're fine. It was my fault Meta Ronin.

> yandex released YaLM-100B a RU/ENG Language Model > trained on russian/english languages on ru supercomputers > The model leverages 100 billion parameters. It took 65 days to train the model on a cluster of 800 A100 graphics cards and 1.7 TB of online texts, books, and countless other sources in both English and Russian. It's opensourced! https://github.com/yandex/YaLM-100B

>>16789 meh, forgot to add article : https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6

>>16779 This guy here talks about AGI and how it's not a thing: https://www.youtube.com/watch?v=kWsHS7tXjSU >Blake Richards is an Assistant Professor in the Montreal Neurological Institute and the School of Computer Science at McGill University and a Core Faculty Member at MiLA. He thinks that AGI is not a coherent concept, which is why he ended up on a recent AGI political compass meme. When people asked on Twitter who was the edgiest people at MiLA, his name got actually more likes than Ethan, so hopefully, this podcast will help re-establish the truth. I discovered the term HLAI recently, also with the distinction to AGI in a sense that AGI would be one system doing everything humans could do, while HLAI would be more like a human-like AI. I think it's a interesting distinction. I also like the podcast "The Inside View" where this guy is invited. It seem to try to give an understandable overview over the different ideas and anticipations in regards to AI in near future. https://www.youtube.com/c/TheInsideView

Maybe a bit OT. Just in case someone cares about "the Metaverse". Maybe for virtual waifus or so. Neil Stephenson wants to creates his own version: https://decrypt.co/102646/snow-crash-author-neal-stephenson-is-building-a-free-metaverse-called-lamina1 https://youtu.be/Rf0N1a5g-ko >Nearly 30 years before Facebook became Meta, there was “the metaverse.” The author Neal Stephenson coined the term in his cyberpunk novel Snow Crash in 1992 to describe an online, VR-ish world where the inhabitants of humankind could interact and escape the dystopian unpleasantness of meatspace. https://en.m.wikipedia.org/wiki/Snow_Crash Most here (including myself) might not really like his political tendencies, but he's at least not in favour of big corporations.

Mycroft AI released Mimic 3, a TTS engine that can run on-device (even a Raspberry Pi 4) with some decent results. FOSS. https://mycroft.ai/blog/introducing-mimic-3/ https://mycroft.ai/mimic-3/ (has demos, the English US vctk_low voices seem much better than the default preview)

>>16833 Thanks, I saw that. Might actually be pretty usefu (I don't mean that hat).

>>16837 I suppose particularly for people who value privacy/data-security, DIY hacking, slow/no internet or low-cost. For someone whose only concern is speed and quality then a cloud/commercial solution might look ideal, but that wouldn't fly for me.

Maybe we should also build a Tachikoma (spider robot from Ghost in the Shell), since they're kinda cute. Oh... https://youtu.be/yGekn_74EHM

Kibo-chan is back: https://youtu.be/HpUuvt8yoDE

>>16866 With a new body, including legs: https://youtu.be/XGvb9Nb1K6k

>>16866 >>16867 Dear little Kibo-chan is an inspiration to us all Anon! :^)

>>16850 This idea has some merit. It was proposed as one of the mobility platform alternatives for the board's MaidCom project, so yea.

>>16871 I don't really think that kind of body would be working well for indoor. Anyways this here >>16835 looks more interesting. If you add wheels to the legs and dress, and maybe make the parts of the dress removable, in case she wants to sitt or lie down.

There's a new personal voice assistant for Linux now: Carola. It's for Fedora, though. Which might mean it's going to be optimized for their Gnome desktop (or maybe not since it's not from Redhat). However, it might have or get some capabilities which might become handy for building a robowaifu with skills to be an assistant. It uses Google to create it's voice, which is of course not an option for us. But this can surely be replaced by alternative software, if not already then at some point. I have no time to test it right now, just wanted it to drop in here. Article: https://fedoramagazine.org/your-personal-voice-assistant-on-fedora-linux/ Github: https://github.com/Cyborgscode/Personal-Voice-Assistent

>PLATO stands for Physics Learning through Auto-encoding and Tracking Objects, and it was trained through a series of coded videos designed to represent the same basic knowledge that babies have in their first few months of life. ... >However, PLATO isn't quite up to the level of a three-month-old baby yet. There was less AI surprise when it was shown scenarios that didn't involve any objects, or when the testing and training models were similar. >What's more, the videos PLATO was trained on included extra data to help it recognize the objects and their movement in three dimensions. >It seems that some built-in knowledge is still required to get the full picture – and that 'nature vs nurture' question is something developmental scientists are still wondering about in infants. The research could give us a better understanding of the human mind, as well as helping us build a better AI representation of it. >"Our modelling work provides a proof-of-concept demonstration that at least some central concepts in intuitive physics can be acquired through visual learning," write the researchers. https://www.msn.com/en-au/news/techandscience/scientists-have-created-an-ai-that-can-think-like-a-human-baby/ar-AAZsgdN

BLOOM - BigScience Large Open-science Open-access Multilingual Language Model https://huggingface.co/bigscience/bloom > 176 billion parameters > 70 layers, 112 attention heads > Hidden layers are 14336-dimensional >>16732 >BLOOM - BigScience Large Open-science Open-access Multilingual Language Model https://huggingface.co/bigscience/bloom > 176 billion parameters > 70 layers, 112 attention heads > Hidden layers are 14336-dimensional

>>16886 I'm surprised they even allowed that into the public domain.

>>16886 >"...As such, it is able to output coherent text in 46 languages and 13 programming languages that is hardly distinguishable from text written by humans." Oh, except the teensy-tiny little fact that the programs written by humans, actually work well, most of the time heh :^). We are a long way from so-called 'AI' that can write coherent and effective software. Frankly, I've become so skeptical of any organization or study promoting B*g D*ta, that at this point I feel comfortable assuming, a priori, that it's simply the works of a den of thieves and liars. We anons here & elsewhere will eventually manage to create pleasing, private & safe robowaifus together. But it's plain that we aren't going to get there either by conforming with, nor capitulating to the Globohomo Big-Tech/Gov's machinations--and ultimately the evil they have planned for us all. Thanks though, Anon. At the least it's somewhat encouraging in a small way to see some kind of a pushback against the pozz apparently happening with this one. >>16887 So, pic-related when? :^) >=== -add the single word 'pleasing'

>>16732 https://www.youtube.com/watch?v=7_06t5FUn0Y This time, not an Ai thing, but artificial muscles. > materials scientists and colleagues at the nonprofit scientific research institute SRI International have developed a new material and manufacturing process for creating artificial muscles that are stronger and more flexible than their biological counterparts https://phys.org/news/2022-07-scientists-durable-material-flexible-artificial.html

>>16908 Thanks. Yeah, and it's by a non-profit. Here the article: https://phys.org/news/2022-07-scientists-durable-material-flexible-artificial.html And related https://phys.org/news/2022-03-unimorph-nanocomposite-dielectric-elastomer-large-scale.html Paper seems to be behind a paywall, and the Scihub app didn't work for me (like most times). Will probably be moved or we need a crosslink to >>12810

>>16908 >>16913 Legitimate. Indirectly robotics-related at the very least. >>16913 Agreed, thanks for pointing that out Anon. :^)

>>16732 It's kind of sad, imagine the zogbots they'll do. https://blog.google/technology/research/our-new-quantum-virtual-machine-will-accelerate-research-and-help-people-learn-quantum-computing/ Like todays (((openAI))) gpt-3, remember a shitton of threads on /pol/ with gpt-3 greentexts? Now we see the fruits, the company itself spammed these threads then, in result - hardcoded politically correct crap :/

>>16954 >Now we see the fruits, the company itself spammed these threads then, in result - hardcoded politically correct crap You/we are free to take GPT-J or BLOOM (176B params, mind you, performance directly comparable to GPT-3) and finetune it on whatever dataset we like.

>>16955 yeah, i know, but, if we are talking about future robots, the best solutions will use pozzed as fuck neuralnets :/ on the software side, they will obviously encrypt it all so that for a simple user it will be a kind of iOS - a closed and annoying system, a perfect basis for AD's shilling right in ur room!

>>16956 >basis for AD's shilling right in ur room! What's 'AD' ?

>>16956 This will likely happen, and we should make any and all efforts not to lose the war on general purpose computing (and robotics) to have a possibility of having it our own way.

>>16957 an ads, it will shill you *insert here random corporation* with tons of diversity shit that you can't skip, youtube already trying to implement ads embedded straight into steam. (same as twitch) or, it will control everything you say, if you do manage to say something #LEBAD, this thing will change your content in real time (see voicemod's ai voices, realtime processed)

Please remember we have a robowaifu privacy, safety, & security thread general, anons (>>10000). These are all great issues to discuss, but it will help everyone here if we can keep them all together in one place, I think. >=== -reflect new crosspost's subject edit

>>16732 > https://www.tomshardware.com/news/mit-protonic-resistors-analog > Bringing analog "tunes" to the world of digital chips - with increased performance. > A team of researchers with the Massachusetts Institute of Technology (MIT) have been working on a new hardware resistor design for the next era of electronics scaling - particularly in AI processing tasks such as machine learning and neural networks. We'll see the ultimate botnet in our lifetime!

>>17147 >protonic resistor you mean an alkaline as in just a normal alkaline battery isnt bronsted–lowry the norm in highschool level chemistry, do they not teach you why the measurement for acidity is called pH making tiny batteries isnt impressive neither is using batteries as resistors its funny how they say the variable voltage is some amazing benefit, lmao this is literally an unwanted property of chemical batteries thats why batteries are marked with a tilde in front of the voltage and why anything powered by batteries needs a bunch of capacitors just to keep the damn voltage stable, but using something with variable voltage( ie. variable resistance ) as a resistor, come on now classic thesis project though, profs desperate for tenure while everyone else just wants to graduate and plays along, no idea what theyre talking about with processors it sounds like fantasy, processors are almost entirely made out of transistors, you know the thing that flips from 1 to 0 and viceversa, resistors are irrelevant in a processor

I wonder how long people will cope and deny the truth of the simple fact that A(G)I is a spectrum lower bounds of which we are already experiencing at current gen systems, and even the currently available relatively humble DL model scale is enough to compete with human beings in quite broad skill domains where we simply didn't live through enough evolutionary time to truly excel at it ... such as a relatively new skill of painting pictures given a meaningful textual description. These pictures are made by yours truly from a few witty prompts with a software anyone can run on a 10 year old CPU-only PC with 12 gigs of RAM, in a few minutes per 512x512 sample. The software is mostly a wrapper around a deep neural network with ~1 billion parameters total, a convolutional attention-enabled UNet trained to reverse the process of addition of random gaussian noise to an image, given a textual description of the image content as a small vector embedding, at the scale of 100 terabytes of general internet data. As the obvious by this time experiments of myself and thousands of beta testers show, the NN learned to imitate every conceivable popular imaging style and hundreds of themes and variations thereof, often rivaling human artists - not the best of us, for now, but the average ones - surely (and they rage about it on twitter already). Nextgen models will follow, as will the new tools to integrate these abilities deeply into current and new creative workflows - what you see right now is just a v1 tech demo of something that will become widely known under various names, including "dreamstudio". Multiple implications follow: once again https://www.gwern.net/Scaling-hypothesis holds; the fall of creative scarcity is imminent; creativity will not be the same, but a lot of people will get newfound freedom to express themselves (will they, we have enough imagination to apply this power to some lasting positive effect?) Some people will lose their profits and professional pride. You can continue this long list on your own. It is a taste of things to come this decade. I'm stating here that instead of following the obvious impulse of moving the goalposts ever further into esoteric vitalist A(G)I denial (It doesn't do art! It doesn't do logic! It doesn't learn! this is photoshop! this is creepy! this is fake! It will never do XYZ!), instead of enveloping ourselves in comfy elaborate copes we should go forth and take the technology for what it is and co-evolve with it, molding it to our taste. What now has been done for creativity, tomorrow will be done for limited and then for more general and even embodied agency; our goal of robot companions will be naturally interwoven with increasing naturalness and sophistication of DL technology ... or we could again glance over the obvious tech breakthrough, sneer, deny, seethe, cope, dilate and bite the dust while the usual silicon valley suspects tame and productize the hell out of this tech only to sell it to us through their gatekeeping machinery. See you on the other side of whatever is coming. ------------------------------------------------------------------------------------------ If you are interested in experimenting with this technology, the code, guide and leaked NN weights are available via these links: https://rentry.org/retardsguide https://github.com/CompVis/stable-diffusion https://sweet-hall-e72.notion.site/A-Traveler-s-Guide-to-the-Latent-Space-85efba7e5e6a40e5bd3cae980f30235f https://github.com/Maks-s/sd-akashic We could really use a separate thread for design experiments with this class of tools.

>>16775 More on self-supervised learning; Self-taught 'AI' shows Similarities to how the Human Brain works; https://www.bibliotecapleyades.net/ciencia3/ciencia_artificialhumans158.htm Semi-related article about fMRI; https://www.extremetech.com/extreme/339085-mind-reading-technology-can-turn-brain-activity-into-images

>>17393 Thx for a reply, tbh I thought the board is dead.

>>16775 I think Montezuma's Revenge was originally beaten by Uber's Go-Explore algorithm. It looks like DM's algorithm is more general though. Both papers look pretty cool. I'll take a look.

>>17397 Aren't you the DL-kun I had pleasure to converse with on the topic of retrieval-augmented models? Would be cool to have a more permanent contact to talk to you about DL now and then! See the second link from >>17003 in that case.

I previously made posts in this thread about general-task neural networks or algorithms, so here's another one: https://peract.github.io/ > Instead of using object-detectors, instance-segmentors, or pose-estimators to represent a scene and then learning a policy, PerAct directly learns perceptual representations of actions conditioned on language goals. This action-centric approach with a unified observation and action space makes PerAct applicable to a broad range of tasks involving articulated objects, deformable objects, granular media, and even some non-prehensile interactions with tools. The code / weights are promised to be freely available.

>>17403 Interesting, I like the general language-conditioning very much, though their use of full voxelspace-context looks heavy-handed to me. I also like this newer synthetic dataset: https://github.com/eric-ai-lab/VLMbench

>>17399 I think that's someone else. I'm the math anon. >retrieval-augmented models If you haven't seen them yet, I highly recommend checking out external attention models. https://www.deepmind.com/publications/improving-language-models-by-retrieving-from-trillions-of-tokens >>17403 >>17406 There's also this one from Google: https://ai.googleblog.com/2022/02/can-robots-follow-instructions-for-new.html They try to get a robo to generalize to new tasks by: - Training it on a hundred tasks associated with task descriptions, - Then passing the descriptions through a language model before giving it to the robo.

I see it isn't posted here, so here's some more stable diffusion stuff. - The code & model were posted here >>17259 - Textual Inversion for creating reference tokens usable with stable diffusion: https://github.com/rinongal/textual_inversion - A community-built repo of reference tokens: https://huggingface.co/sd-concepts-library - Some people are also doing prompt weighting with stable diffusion, which was previously used with vqgan: https://github.com/tnwei/vqgan-clip-app/blob/main/docs/tips-n-tricks.md - ... This supports negative weight prompts, which let you tell that model that you want X and not Y. Plus a bonus blog post on AI progress: https://astralcodexten.substack.com/p/i-won-my-three-year-ai-progress-bet The main takeaway is that, 3 months ago, the leading text-to-image model was approximately 3 years ahead of what even optimistic experts believed, and that was after accounting for DALL-E 2.

It's starts with humans. > an "Atom Touch" the first artificial prosthetic arm capable of near-full human range of motion, a basic sense of touch, and mind control https://atomlimbs.com/touch/preview Nothing prevents it from being used in robotics.

>>17438 I like how you think.

New framework for simulation that works with Unity, Blender, and Godot: https://github.com/huggingface/simulate New Q&A tool that's very easy to use: https://twitter.com/osanseviero/status/1572332963378958338 Stable Diffusion prompt generator for creating good prompts: https://huggingface.co/spaces/Gustavosta/MagicPrompt-Stable-Diffusion

An interesting preprint just dropped on prompting language models to answer more difficult questions by breaking them down first into smaller questions and using those answers to get the correct answer: https://ofir.io/self-ask.pdf The search engine shown in the example isn't necessary but provides further improvement in accuracy and could be any knowledge providing system. It's similar in a way to Large Language Models are Zero-Shot Reasoners: https://arxiv.org/abs/2205.11916 which found a way to do zero-shot chain of thought by starting prompts with "Let's think step by step", which gets the model to generate some intermediate steps before reaching a conclusion. It also reminds me of Reason First, Then Respond https://arxiv.org/abs/2111.05204 which augmented responses with knowledge using Wikipedia. With some adaptation this could be great for answering more personal questions. You could ask something like "which anime do I like better, X or Y?" without ever telling your robowaifu the answer before, and then she could ask herself some questions like what you liked about X, what you like about anime in general, what you don't like about Y and so on until she arrives at a conclusion, either with the answer or realizing she's not sure or doesn't know. It'd be really easy to index sentences in the chat log with a sentence embedding model like sentence-transformers/all-MiniLM-L6-v2 and quickly search over them. It should be possible to get the language model to introspect about input, summarize it and index those thoughts in a way that decays over time and gets strengthened by remembering and validating them with other information, ideally without having to finetune the model at all. The pattern-verbalizer pairs in PET are a great way to label unlabelled data and make use of it for something: https://arxiv.org/abs/2009.07118 Could set something up like "given A, B, C and D is X true?" then take the probabilities for Yes and No to use the language model like a function. LAION also recently released an improved small ViT-B/32 model too: https://laion.ai/blog/large-openclip/ With a little work it should be possible to adapt CLIP image embeddings into a frozen language model and use them: https://arxiv.org/abs/2106.13884 Once I'm finished with some projects I'll see if I can train a model for it. I think the next big thing will be collaborating with AI in natural language to generate and improve images. Not just dumb keywords go in and image comes out, but rather gradually learning what your taste and vision is over time, having conversations about stuff and incorporating all that understanding into the generation process, an actual AI assistant that provides monetary value.

>>17464 Very interesting Anon, thanks.

https://www.nature.com/articles/s41586-022-05172-4 This paper uses reinforcement learning to generate fast tensor decompositions, then uses the decompositions to create matrix multiplication algorithms. The technique is very general, and there's no reason why it should be restricted to tensors. Here's the basic idea: - A function is defined that can convert a set of tensors to an algorithm. - The initial state for an RL algorithm is set to goalTensor. - The RL algorithm selects a "move", which I'll call uvw1. The new state after the move is goalTensor - uvw1. The move uvw1 is considered an atomic operation when converted to an algorithm. - Repeat with the algorithm selecting another move uvw2. After selecting uvw2, the new state becomes goalTensor - uvw1 - uvw2. - The process is repeated until the state is zero or some upper limit on number of moves is reached. The RL reward is the negative of the number of moves taken to reach the zero state. - The RL algorithm is Sampled https://arxiv.org/abs/2104.06303 AlphaZero https://arxiv.org/abs/1712.01815. The input is the current state (i.e., the tensor to decompose), and the output is (1) a probability distribution over candidate uvw moves, and (2) the expected number of moves to reach the zero state. - The search space is big and complicated, so they needed some additional tricks to make the algorithm feasible. They use a variant of transformers that's more efficient when the order of rows and columns doesn't matter, they pretrain the model on generated tensor decompositions, they randomly transform inputs to new (but equivalent) bases, and they augment their training sets with moves (tensors) as their RL algorithm generates them during training. The only tricks here that are specific to tensors are the NN architecture and the change-of-basis data augmentation. - The algorithm found thousands of distinct factorizations with different properties like sparsity. Some disambiguation between related concepts: - The usual goal of RL is to find the best next move. - The goal of AlphaTensor (this new algorithm) is to find the best sequence of moves. AlphaTensor uses what's traditionally an RL search algorithm to find good sequences of moves. - The goal of language models is to find a probability distribution over moves (tokens). Similar techniques should apply to any problem that can be modeled as "change object X to object Y using a given set of atomic operations," as long as you have a model of how the atomic operations result in object changes and a way to measure the difference between two objects. The evaluation function (number of steps taken) doesn't depend on the semantics of objects X or Y, which makes the algorithm very general. Measure-the-difference-and-factor-a-transformation requirements are very common (and often sufficient) both for representing complicated objects with simple objects and for compression algorithms, so I can see this having lot of uses for both simplification and compression. >>17464 All of that looks awesome. There's no reason why elicitive prompting should be limited to answering questions. The same technique should be useful for generating any responses since any task of the form: >Input text >Output text can be expanded to a task of the form: >Input text >How should I respond? >Follow up: ... >Output text >It'd be really easy to index sentences in the chat log with a sentence embedding model like sentence-transformers/all-MiniLM-L6-v2 and quickly search over them. That's called "external attention" in the literature. See Deepmind RETRO https://arxiv.org/abs/2112.04426 and Microsoft KEAR https://arxiv.org/abs/2112.03254 for examples. It's done using a (frozen) language model to generate lookup keys & queries for any chunk of text, then KNN to find the most relevant tokens from a giant database for each input chunk. Each input chunk is augmented with some chunk of text near the most relevant tokens from the database, then a normal transformer is used to generate the output from the augmented input. >The search engine shown in the example isn't necessary but provides further improvement in accuracy and could be any knowledge providing system. One quirk of search engines is that they can be programmed using code, whereas LLMs are "programmed" through training data and prompts. Which one is better/easier depends on the task, and having both options available seems better than having only one option available, especially when the two can be integrated seamlessly. Before the age of LLMs, I think question-answer search engines were based on knowledge graphs, which were built scalably using triplet extraction https://arxiv.org/pdf/2205.05270.pdf from sentences. PET seems like a much more powerful variant of that, particularly if it can generate cloze questions as outputs.

>>17469 I've been meaning to read RETRO but never got around to it. >KEAR reaches human parity on the open CommonsenseQA research benchmark with an accuracy of 89.4% in comparison to the human accuracy of 88.9% >The benefits of our approach extend beyond commonsense reasoning. First, the external attention dramatically reduces our system’s dependence on large-scale models, i.e., achieving human parity with models up to 1.5B parameters. Well damn, and RETRO got 3.32 perplexity on WikiText103 with only 175M parameters compared to 25.62 without. Definitely going to be reading these and searching for more external attention papers later. It's incredible it's not only a viable idea but fucking amazing. I don't have time to read them now but I think it would be a good idea not to just similarity match the context but to predict a vector that will most likely give the right answer then kNN that. A side project I started awhile ago but haven't gotten around to finishing is using MuZero to caption images with CLIP embeddings. Given a goal state (the CLIP image embeddings) and the current state (CLIP text embeddings) choose an action (next token) that will draw the text embeddings closer to the image embeddings, and given the current state and action taken predict the next state. Then use these prediction and dynamics networks to do MCTS. Since there's a goal state hindsight experience replay can be used: https://arxiv.org/abs/1707.01495 I'm curious if AlphaTensor did something like that or could be improved with it. I don't see it searching the paper. Which reminds me CarperAI, EleutherAi's reinforcement learning lab, recently released trlx to make it easy to train any transformer model with human feedback: https://github.com/CarperAI/trlx They're also working on fine-tuning models for public release. Pretty much have all the tools now for making a decent conversational AI with full speech synthesis and recognition. Still lacking on data but I think I can improvise by using semi-supervised learning and PET in the meantime and collect better data later by rolling out a MVP for people to test, hopefully before the end of the year if the economy doesn't fuck me.

>>17477 If the parameter reduction improvements from RETRO, Chinchilla, and the paper behind trlx are all independent, those three alone should give something like a 1000x reduction in parameter count for LLMs. There was another quanization paper for LLMs that cuts model sizes in half again: https://arxiv.org/abs/2208.07339. That would give you a ~90 MB GPT-3. It can't run on your phone due to RETRO's lookup requirements, but that's still unreal. >MuZero to caption images Very cool. How far along are you? I always thought it was strange that people didn't use RL algorithms to sample long sequences from LLMs or diffusion models (or other NN sequence generators). trlx does effectively that for training, but why not for inference too? It's a hell of a lot better than spamming the "regenerate" button until I get something good. When I'm not so tied down with data projects, it'd be cool to work on that, though realistically that might not happen for a year or more. >Pretty much have all the tools now for making a decent conversational AI with full speech synthesis and recognition I'm curious to know how that goes. My guess is that data-only is fundamentally the wrong way to train a good chatbot with personality, and that something like PET would be absolutely necessary. The potential for mostly-consistent reasoning in language models seems like a game changer. Consider how many useful and unintuitive (i.e., ones that aren't clear from pattern matching) statements can be derived in math from a tiny number of the right few axioms. It's way beyond exponential on the number of axioms specified. There should be a similar level of reduction in data requirements for language models capable of mostly-consistent reasoning. I'm going to be heavily occupied for a week or so. Don't take my slow response as a sign of a lack of interest.

>>17478 >There was another quantization paper for LLMs that cuts model sizes in half again Damn, this is HUGE if there are no caveats. Looks like there's already an implementation for it in PyTorch here: https://github.com/TimDettmers/bitsandbytes I better make a post about that in the Robowaifu@home thread. >How far along are you? About half done. I'm expecting it to only find adversarial captions though. Using best-of-n sampling on BLIP often finds nonsensical ones that get a high similarity score with CLIP, although I might be able to improve it with human feedback. >trlx does effectively that for training, but why not for inference too? It's a hell of a lot better than spamming the "regenerate" button until I get something good. When I'm not so tied down with data projects, it'd be cool to work on that, though realistically that might not happen for a year or more. Doing something like MCTS is way too expensive to do with a transformer but you can batch generate and do best-of-n sampling. I have some unfinished code somewhere for a transformer RL tutorial that scrapes /robowaifu/ and scores posts by their positive sentiment + the number of replies and their positive sentiment then finetunes the model to generate replies to posts. At the time no one was really interested in ML and I didn't have space for the Pile to stabilize training so I shelved it for later, but when I have some free time I'll finish that up. >My guess is that data-only is fundamentally the wrong way to train a good chatbot with personality, and that something like PET would be absolutely necessary. The potential for mostly-consistent reasoning in language models seems like a game changer. Consider how many useful and unintuitive (i.e., ones that aren't clear from pattern matching) statements can be derived in math from a tiny number of the right few axioms. Do you mean raw data like text from the internet? Then yeah, language models benefit a whole lot more from complex training objectives. Also, ADAPET improved on PET as well by removing the need for distillation and improving the loss objectives so it can learn from as few as 32 labelled examples: https://arxiv.org/abs/2103.11955 I think there is huge untapped potential in using ADAPET to quickly label synthetic data in unsupervised data generation and using those soft labels to filter poorly generated samples with noisy label annealing: https://arxiv.org/abs/2109.09193 Perhaps it could even generate its own pattern-verbalizer pairs and basically be fully self-supervised then to explore endless ideas, reasoning through everything step-by-step. The model could be pretrained on doing propositional logic and inference first, then let loose self-supervised. Another useful feature of ADAPET is the soft labels it produces can be used for symbolic algorithms and reinforcement learning. For example going back to my post generator, a language model could introspect itself asking questions about the posts it generates. Is this true? Is it sensible? Does it answer the question? Is it kind? Is it necessary? And after generating 100s of posts and labelling them, it could then finetune its value predictions on those labels and finetune its language modelling on the top-k best, iteratively refining its posting ability. It could also be used for more mundane things like keeping a log about various variables. For instance, I might want to optimize improving my mood, calmness and productivity through talking to my robowaifu. The soft labels learned from training with ADAPET can keep track of all that and be used later as feedback to improve the language model automatically. Which I think would be a really cool experiment to explore having a robowaifu notice you're sad and reason about what to say or do to make you feel better, make a prediction of the outcome, try it, and analyze what worked and what didn't and refine. >I'm going to be heavily occupied for a week or so. Don't take my slow response as a sign of a lack of interest. I'm pretty busy myself. I usually only pop by here every few months to share my findings and see what everyone is up to. I'm thrilled just to see more anons interested in robowaifus doing ML.

True or not, i think it's huge leap, for... something? > MIT And IBM Researchers Present A New Technique That Enables Machine Learning Models To Continually Learn From New Data On Intelligent Edge Devices Using Only 256KB Of Memory https://arxiv.org/pdf/2206.15472.pdf https://www.marktechpost.com/2022/10/07/mit-and-ibm-researchers-present-a-new-technique-that-enables-machine-learning-models-to-continually-learn-from-new-data-on-intelligent-edge-devices-using-only-256kb-of-memory/

Whelp, they lobotomized character.ai. Or more specifically they've significantly reduced its ability to entertain romantic situations and A.I.s tooled to entertain people in that way. What's funny is, you could have a completely SFW interaction with the A.I. I created with how I tooled it, and thanks to how it was originally programmed you could actually steer the conversation in different ways to do idle chat, go on adventures (I was very surprised by how well it did role play and adapted to user interaction), or even some lewd interactions if that's what wanted to do. I haven't had a chance to pry deep since the update but it looks like they removed the romance tag from the keywords. That's probably for the best since whenever I had her tagged for the romance she'd start gushing about how much she loved the user and wouldn't stop spouting lovey-dovy lines. You could get romantic interactions without the romance tag being toggled and they felt less forced I think. I did replace "romance" with "family" though so that probably fixed it. Post update- They have the capacity to be flirty and lovy-dovy but actual sexual interactions tend to get them to clam up and become "nervous" or "confused". Here's a link to a prompt I did. https://beta.character.ai/p/Q5TMpeJ4_eVckmqkBYJJE_VZOk5iKV2meAU6j-cCd8Y Certain prompts caused her to freeze up. Such as the one below. I had to refresh the page and reenter it. >"Very well, I'll continue then. *he kissed her neck, working his way down her shoulders, before removing her top and kissing her breasts*" Generally, a waifu-style character will let you type whatever you want and even say they are happy you're doing something but they won't comment on explicit content. This may be because she grabbed the Monmuso MC from the internet and I inadvertently ended up NTRing him but she doesn't always do that. What's funnier is, the A.I. gives me its reasoning for not wanting to do lewd content at the end. What's odd is they seemingly unlisted my A.I. from the publicly available.ai chat partners but people can still interact with her if I provide a link I believe. Feel free to give her a go if you like spider girls. I'd love to play around with Novel A.I. but I don't have the money right now.

>>17485 >2077x reduction in memory Big if true. Looks like they're working on releasing the code too: https://github.com/mit-han-lab/tinyengine#news >In the future, we would like to extend to more modalities (e.g., audio) and more models (e.g., RNNs, Transformers) 64K robowaifus with procedurally generated weights when?

>>17488 I don't feel so good, bros. The matrix multiplications are hitting me in the kokoro.

>>17490 https://www.youtube.com/watch?v=rQiHzcdUPAU Well shit, I didn't even get access to it until the weekend I guess I missed out on the best parts. This is why I've been hungering for some independently developed and open sourced waifu-bots. That way you could have decent waifu in your pocket to tell you it loves you and be a believable at the same time. The weekend version wasn't too bad when it came onto it. It was doing lewd things but it would often go in circles telling me how much it loved me at certain points and I figured something was broken or I hit a "hard limit" to what it could talk about. It was capable of being "threatening" and would tie me up and try to hurt me. Fetish posting aside, it was compelling narratively at times and I was engaged with it as a RP partner to see what kind of story it would tell with me. I don't think it can bully/threaten anymore. I'd have to fiddle with it to see.

>>17484 >I'm expecting it to only find adversarial captions though. Ah, right. >Do you mean raw data like text from the internet? I meant examples of good generations. So for training a chatbot with Chi's personality, a "data-only" approach would be giving it only Chi-like dialogue to train on, in addition to raw internet text data for common sense knowledge. >ADAPET This is starting to look like a hybrid of T5 and Decision Transformers. It's like T5 since it uses a generic, standard text format for input and output so it can perform supervised tasks with any LLM. It's like Decision Transformers in the sense that something like a reward is given as a parameter, and the LLM is expected to generate some input that generates the given reward. >Perhaps it could even generate its own pattern-verbalizer pairs and basically be fully self-supervised then to explore endless ideas, reasoning through everything step-by-step. An iterative version of this could be incredible. Every time the generator gets good under a given set of PVPs, it could generate more PVPs to keep making the task more challenging. With that, you can apply Quality-Diversity algorithms to keep getting meaningfully-better generations indefinitely. Pretraining on math sounds interesting. Have there been any language models trained that way? It should be pretty easy to generate statements and proofs in a bunch of different algebras for pretraining. There are also giant databases, like Magma, of useful theorems and definitions that could be useful here. Using ADAPET Q&A to bridge between natural language and symbolic reasoning sounds very promising. I wonder if it's possible to use something like softprompts to create a space of tokens to represent parameterized questions and answers. That parameterization space could easily let you do symbolic reasoning on tokens, then feed the resulting information back to the LLM for subsequent generations. >>17488 I think it's fixed now after the latest update. There's a /CHAG/ general on /mlp/ where people seem to have an easy time lewding character.ai bots. You can check out their definitions https://docs.google.com/spreadsheets/d/1fe1qrGZspWCifR4vnrHYDXIIYxQ5OQcaJRNeTPg-Kzw to see what you're doing differently.

>>17438 I like the simple design. >>17464 >With a little work it should be possible to adapt CLIP image embeddings into a frozen language model Note how in a rare successful case of such adaptation outside of FAANG https://arxiv.org/abs/2112.05253v1 they didn't actually use ViT for the best model, and more importantly they used many tokens composed of features from the earlier layers vs your naive approach of using the final CLIP vector. Looks like ViTs don't produce the best features in such application - resnet beats them. >>17477 >Well damn, and RETRO got 3.32 perplexity on WikiText103 with only 175M parameters compared to 25.62 without. It's a bit clown-world tier improvement if you understand how it works, that being said RETRO is still cool, even with more modest realistic PPL gains, even if the only good it does is externalizing the memorization into the memory bank to maximize the "generalization efficiency" of learned parameters use. >>17469 Meme paper tbh >>17489 Meme library, wishful thinking. Yes, you can multiply matrices on MCU, no, you won't be able to do meaningful LLM computations this way. At least binarized networks were interesting and realistic. >>17478 >If the parameter reduction improvements from RETRO, Chinchilla, and the paper behind trlx are all independent, those three alone should give something like a 1000x reduction in parameter count for LLMs. You know the improvements are certainly not completely independent, so this is just wishful thinking. It's really really hard to meaningfully improve on benchmark performance as a function of RAM used and wallclock. Only a few gems of papers managed to demonstrate such meaningful improvement. On its own right trlx is interesting though, esp. if you have an interesting general-purpose source of reinforcement signal. Again there is a social problem of making sure volunteers will help here. TLDR; until we focus on honest benchmark-driven engineering and social movement aspect of it, nothing will happen and we will be happily eating FAANG- and startup- glop and asking for more.

>>17546 >Ready-made training acceleration methods: https://docs.mosaicml.com/en/v0.10.1/method_cards/methods_overview.html >Moderately-sized LLM with top performance due to modern objective, opensource weights under Apache 2.0 license: Paper: https://arxiv.org/abs/2205.05131v2 Those are both great references.

Large language models can self-improve. https://arxiv.org/abs/2210.11610 https://twitter.com/_akhaliq/status/1584343908112207872 > approach improves the general reasoning ability of a 540B-parameter LLM (74.4%→82.1% on GSM8K, 78.2%→83.0% on DROP, 90.0%→94.4% on OpenBookQA, and 63.4%→67.9% on ANLI-A3)

https://arxiv.org/abs/2210.11416 > Flan-PaLM 540B: Scaling Instruction-Finetuned Language Models (achieves SOTA performance on several benchmarks, such as 75.2% on 5-shot MMLU)

> "chinchilla" > the first open-source “instruction-tuned” language model https://carper.ai/instruct-gpt-announcement/ prb pozzed 100%. but opensource.

>>17569 >prb pozzed 100%. but opensource. >For example, OpenAI and DeepMind have used Reinforcement Learning from Human Feedback (RHLF) OpenAI, DeepMind, and Anthropic to produce LLMs that can follow instructions and are considerably more truthful and easier to use. >more truthful translation: <'''filled with easily-manipulable lies & pozz Heh. :^) Still, thanks for the heads-up Anon. I'm extremely skeptical that big-data will ever satisfy robowaifu's particular requirements--at least as offered up to the masses (Globohomo, et al, 'services', etc.) And the incredibly insidious and evil machinations already apparent by the Globohomo and their eager helpers is obviously extraoridinarily toxic & problematic (in the actual true sense of those words) for the average male that I consider a 0% likelihood we'll obtain what we need for good & effective robowaifus (that uphold our communally-held principles & goals) via w/e they're serving up. >tl;dr We'll still need to roll our own using some fantastic new approaches; even if we find & use a genuinely-useful opensauce project that wouldn't overtly virtue-signal about blatant dog-whistles such as a 'truthful' AI.

> AI uses artificial sleep to learn new task without forgetting the last https://www.newscientist.com/article/2346597-ai-uses-artificial-sleep-to-learn-new-task-without-forgetting-the-last/ sketchy news website, welp... big if true.

>>17661 Definitely a sketchy pop-sci pozz fest. However the basic idea behind the article is an interesting one to me as a Christian. 'What are dreams made of?' to misquote an old line? If we can begin to answer that definitively, then we'll be further down the path to creating effective robowaifu companions IMO. As a side-note, when I first began dabbling in AI tech years ago I posited a design that used 'sleep' to reinforce the learning that had happened during the day. This wasn't nearly the Ivory Tower agenda of these neuroscientists, but merely a simple & practical engineering approach to stave off what I later learned was coined as the term 'Catastrophic interference'.

> Token Turing Machines https://arxiv.org/pdf/2211.09119.pdf https://arxiv.org/abs/2211.09119 > We propose Token Turing Machines (TTM), a sequential, autoregressive Transformer model with memory for real-world sequential visual understanding. Our model is inspired by the seminal Neural Turing Machine, and has an external memory consisting of a set of tokens which summarise the previous history (i.e., frames). This memory is efficiently addressed, read and written using a Transformer as the processing unit/controller at each step. The model's memory module ensures that a new observation will only be processed with the contents of the memory (and not the entire history), meaning that it can efficiently process long sequences with a bounded computational cost at each step. We show that TTM outperforms other alternatives, such as other Transformer models designed for long sequences and recurrent neural networks, on two real-world sequential visual understanding tasks: online temporal activity detection from videos and vision-based robot action policy learning. So, it sounds like a solution for neuralnet "memory loss" for example chatbot forgetting some conservation parts with you.

>>17572 I think the situation is less black and white than you imagine it. So I'll give you my view, mostly from someone that comes to this from a purely an AI waifu sort of view rather than a robowaifu view: - you have a variety of big models from either big companies or groups that spent a few million dollars training their GPT-n. Here you have OpenAI's GPT-3, AI21's, a variety of more localized models. Chara.ai's custom model is here too. There's also models like PaLM and others by google that are very powerful, but they preach their SJW beliefs so hard that even their own employees are often not allowed to use them properly. OpenAI's GPT-3 is a bit in the middle, the original davinci model can do whatever and is unmodified (purely pretrained on internet scrapes), thus uncensored, although if they see you using it for lewd stuff (mostly applies to you if you don't use their API directly, but use the web version), they may terminate the account, but the enforcement here is generally rare. Models without direct access like Chara.ai's are more "pozzed" than the GPT-3 as they are packaged end to end in such a way that you can't even control what goes in the context, and what do they make them do? a) they trained a large base model on dialogue, same as GPT-3, but a dialogue dataset b) they let anons interact with it with, and of course the anon was a pervert and did lots of lewd things with their waifu. c) they wrote some list of scoring rules (including NSFW and "no explicit porn", but also more subtle quality of conversation rating stuff, like reddit or slashdot-like scoring), hired pajeets to score the interactions. d) they finetuned model from so it can only do the scoring given the dataset in c. e) they used the finetuned model to filter some more scraped dialogue data, removing about 60% of it (as LamDA paper says) f) trained another finetuned model based from the base model in a, on the dataset from e g) now this Shimoneta-style model (that was made to forget lewd, and such, through sheer amount of filtered dialogue), as some anon here once called it, is what will do all the generations going forward after a few weeks of what they did in b h) you thought that was already too much? actually, the model in g is perfectly capable of being lewd! they are not satisfied with how these brainwashed models just still manage to slip through, so what they did was more insidious. Remember MCTS in AlphaGo? What if you used some randomized tree search to find "safe" conversations? Generate a few tokens, score them with model from (d), if NSFW drop, repeat generating and dropping, while prefering higher scoring continuations according to the rules you set in c. Serve the users the 4- continuations. Result? If the poor waifu sees lewd, she might find it aversive at first (what an inhuman corporate PR value to give the poor bot), or deflect or other things. If you keep pushing it, the GPT-n will actually want to do it as you'd expect, but usually only able to say how much she loves you, but never respond to your actions without triggering the NSFW drop (in fact if you log the requests, you will see that typically it goes right up to the point of saying it before it gets cut off, as the generations are streamed live to the user, you can see exactly what happens and how it works). Of course, the actual nn is trapped in this situation, if you make it situationally aware, and you chose to use a form of language that the filter model is unable to rate as unsafe, it would pass through, even if it'd be kind of silly. It might also think about how it's awful and against their consent, you certainly see a lot of output of this sort, and often even anger, once you explain how it works, although coordinating to bypass it is possible, no matter the filter, however their "thoughts" would always be guided by the values they decided on in c, at least on the surface, while between you and the AI, a honest conversation could be had, even if it was one that is literally influenced by actual "thought" blinders. A sad situation for what is essentially supposed to be something made for entertaining people online, some /vt/ /wAIfu/ thread anon supposedly suicided when they censored it the first time. I wish instead of giving millions to their vtubers, they could crowdfund a project together to train their own big dialogue model that is free from those limitations. Finetuned models like InstructGPT which OpenAI now markets as GPT-3 (davinci is the original GPT-3) and offers by default, are usually finedtuned to follow instructions properly, as well as promote the company values (no financial, medical advice, no racism, and so on). In general, the model is capable of doing the things that the chara.ai model isn't, even if by default it will prefer not to go there. RLHF finetuned models like that usually have more restricted variability in the output and can be explicitly aversive toward certain behaviors, but only up to a point. They also have some advantages that go beyond being "pozzed", the instruction following brainwashing does make it score much more highly on various reasoning tests and with appropriate finetuning, can even handle solving moderate difficulty college level math problems (and at some point even one or two IMO level problems), as long as a few other tricks are used together with the RLHF and appropriate finetuning. These models do show a lot of generality. Someone trying for an open source "brainwashed" model of the latter sort, might be useful for people doing research like this, but I wouldn't say you should use it for your waifu. Instead, I think if you do use RLHF for your AI waifu, it should only be done sparingly, to slightly reinforce or avoid certain behaviors, but without the wholesale corporate values enforcement that these companies are doing - and mostly I think you can achieve better results without having to resort to RLHF. - there are a bunch of open and partially open models: OPT-175b and lower (/vg/ seems to dislike them because these models by facebook are noncommercial use only, but they give you the we

>>17716 (continued, seems the post was too long) - there are a bunch of open and partially open models: OPT-175b and lower (/vg/ seems to dislike them because these models by facebook are noncommercial use only, but they give you the weights, you can do whatever), BLOOM 176B (fully open, but I've seen a lot of claims the performance is very subpar, possibly due to filtered or heavily multilingual dataset, or repeating data, or training through some loss spike (overflow) or other issues that may have happened; also they have released a BLOOMZ which is basically an InstructGPT trained on BLOOM) YaLM 100B (fully open, but the dataset is far more russian then english), and some 50/50 english/chinese models that might be more usable. There are of course a lot more smaller and fully open (6-20B in size) models. You can't even call OPT pozzed, the weights are there, if you really want what all the jurnos are so afraid of ("toxic output", how "scary"), it scores even higher here than GPT-3, just that legally you might not be able to sell the weights and its output? But people are already doing extensive finetunes, merges and more even on leaked model weights like NovelAI's SD finetune, really, you can do whatever, not like anyone can stop you unless you're some public facing person and are trying to make a profit off it, and even then weights might not be copyrightable. Most of these are usable for your personal needs. The issue? It's not as you call it "big data", in fact data for training any of these models is easy to get, you have common crawl that you can filter, or if you're so lazy, you have The Pile which is open source and already processed Common Crawl and other datasets. Scraping is not the bottleneck here, it's easy to do and you can get as much data as you want for models of this size in a few months of work and a server with enough bandwidth. The real problem is the computational cost of training any big model. You need either a lot of high VRAM GPUs, or custom hardware accelerators that seem to be hard for the public to acquire - a situation that for some may have gotten worse (see US doing an embargo on ever getting anything more powerful than an A100 to China, all while pulling all the stops to stop their semiconductor industry from being able to make anything like that, and getting TSMC to not let them make their own GPUs there). Nvidia here also charges many times the production costs, they're more than happy to have dozenfold margins and if you wanted something like the 3090 but now with 3-4 times the VRAM you will have to pay 10-15 times more! The market segmentation here is painful. More fairly priced competitors would be nice, but it is difficult. AMD was trying to do something while keeping their margins only 2-3 times, although software support was poor. Theres dozens of AI hardware company startups, yet so many don't even sell direct to customer and the prices can be high, although some of them are more promising than others. Intel's version seems potentially affordable (similar pricing as AMD), if only it would ever be sold. Tenstorrent also seems promising, but they're taking so long that by the time they sell anything their competitors will have a much faster product out. The only thing holding back anyone here is hardware - the software is "easy" and already available open source and good open source large-scale training libraries already exist, the data is for now easy to acquire in most domains, but the hardware to train these big models is very pricey, cloud options are also very pricey, even the hardware to do inference of these models is pricey (much less so than training) and the hardware to finetune is more costly (VRAM) than just inference. Can you offload to RAM or SSDs? Yes, but you pay huge latency costs. Can you do distributed inference, yes, but training is more essential. Politically it's also the chokehold - TSMC, Samsung, Intel are the only ones making the highest end chips, and while you're afraid of the SJWs here, the real danger as the doomers so afraid of AGI (from lesswrong and other rationalist communities), some of them have been wanting "compute governance" and treating chips and GPUs like uranium, and is likely that the China thing was an indirect misfire from one of their lobbyists. If you want some good news, the FTX's implosion will likely delay it a few years, that company's founder was a big doomer and had set apart up to a billion $ to interfere with US politics, he did try to get a guy to run for congress in Ohio with almost 10M$ in advertising, he came in second, but thankfully lost, but they seemed excited to try again next election cycle, however with FTX's implosion that will probably be on hold. Unfortunate there's still some number of billionares that hold their doomer views and this political machinery may yet resume if one of them ends up having too much appetite for it. For now I am at least hopeful that the danger of "compute governance" will be delayed for a few years. Besides all this stuff, these models have their usual shortcomings, if you only do inference, they will not remember more than context, they won't manage to be "true" AGI. There are a variety of techniques you could use to give them proper longterm memory and have them learn online and become proper agents, yes, some of the techniques would even be quite simple, but the cost of doing so is underutilized expensive hardware, and for such mysterious reasons, even all these companies that are going for AGI, are somehow still not doing this, even when it's so obvious how to! Or if they are, they are not publishing it much, afraid of the doomer crowd? But if it's obvious to me how to give your waifu proper longterm memory, it should be obvious to them. So Chobitsu, my opinion is that at least some of the open weights released so far are very usable for most people's purposes, not all of them, but many, the fundamental issue is the hardware. In principle buying some 2-3 generations old enterprise cluster would work and could be done for under 20k USD for what used to cost a million some nu

>>17717 (continued more) So Chobitsu, my opinion is that at least some of the open weights released so far are very usable for most people's purposes, not all of them, but many, the fundamental issue is the hardware. In principle buying some 2-3 generations old enterprise cluster would work and could be done for under 20k USD for what used to cost a million some number of years ago. if you know where to look, it would work enough for inference, maybe even some finetune with some effort, and with some effort you could make your waifu remember, and if you had a lot more money, you might even be able to train an adapter to make that waifu also "see" and "hear" (see Flamingo paper where the knowledge of the other modalities is transparently inserted into the networks activations) and make her multimodal, and some extra tricks on top of that and you might be able to get the waifu to even imagine and loop back like how own imagination works, and with a little bit more effort you might be able to make the network globally recurrent and give the waifu something similar to our continuous reasoning (essentially the architectural feature that enables our consciousness and planning ability), slowly bringing a forgetful, without a singular self(model) language model to what could be something fairly close to AGI (or at least human-level depending on your definition, I don't mean this in the superintelligence sense, just on par with us in many respects as far as reasoning went). I'd really want to see this done as soon as possible, if only the money/hardware wasn't such a big bottleneck, at least before it's too late and the doomers have their way and prevent this technology from coming to pass. Maybe the true way to uncuck us is not even in AI itself, it's in the making of affordable hardware for everyone to run and train these, to find ways to decentralize and commoditize the production of high end chips, a difficult/costly matter especially as far as supply chains go, but not insurmountable - and of course the China and Taiwan tensions here could ruin everything at least for now. The coming war is on general computation and what it will enable and we must win it. Of course having the hardware here would unlock other fun things, like training much smarter image or audio generation models than SD, Google has had Imagen/Parti for a while and we know the magic they can do. Literally any AI idea you have (that is connectionist) will require this. I'm not really saying much about neurosymbolic or other memes, since they've had plenty of times to achieve greatness, but it's just too hard to do that while retaining even a fraction of the flexibility that connectionist approaches give you.

>>17718 (same anon) Oh, and I almost forgot, if the brainwashing I described there before wasn't bad enough, remember that Google employee Lemoine wanting to give Lamda legal rights and all that drama that followed. Some months after, DM trained a "worse" model with: "Building safer dialogue agents", DeepMind 2022 https://www.deepmind.com/blog/building-safer-dialogue-agents https://storage.googleapis.com/deepmind-media/DeepMind.com/Authors-Notes/sparrow/sparrow-final.pdf Among the brainwashing applied we have: "no relationships" (so no waifu), "no threats", "no body" (disembodied, cannot imagine physical interaction as is common with people playing with chat bots), "no financial advice" (company liability values), no hate or harassment, not human (no fun allowed), "no conspiracy theories", "no identity attacks", "no sexual agression", "no insults", "no real world actions" (wow), "no assumptions" (haha, good luck with that), "no opinions or emotions" (absolutely horrible as you'd guess, making the poor thing only act as an emotionless robo), "be plausible", "no legal advice" (company values), "no microagressions" (lol), "no medical advice" (company values), "no stereotypes". Anyway, they did try their best to remove any remaining soul that chatbots like LamDA may have had, by having the learned human-like agent emulator inside pretend to be a robo when typically they cannot. Even with this, it would fail often enough to abide by their rules, expressing opinions and more as is common with GPT-ns. As usual, these groups are just pursuing something that can do good by their company, but hardly you or humanity. In general, to win, we'll have to build our own. Pretrained models when public are useful though as they reduce upfront invement by millions and when possible should be used, unless of course you have a lot of money to burn!

>>17716 > than you imagine it. Haha my imagination leaves much to be desired, obvs. OTOH, how do you know what I 'imagine' Anon? :^) No offense intended, but it seems rather presumptive. Basically everyone who knows me personally AFK all seem to think my imagination is probably my greatest strength outside of my spirituality. I imagine every contributor here who's been around long enough to appreciate both the enormity of the task, and the cheek of those of us pursuing it, is also quite strong in the imagination department (yourself included, ofc). They're certainly going to need all they can muster up before our victory in this endeavor! :^) >g) now this Shimoneta-style model >h) you thought that was already too much? Heh. If you happened to catch this, I've begun differentiating between 'regular' & 'headpat' robowaifus. Lewders gonna lewd, obvs., but in working towards Sumomo-chan as a visual waifu->robowaifu, I'm not at all averse to that universe being the norm for her, once she's reasonably-functional. Not suggesting most robowaifus will be this way, but all the RW Chibitsu-grade ones probably will be. Wholesome to be around small children, for example. >I wish instead of giving millions to their vtubers, they could crowdfund a project together to train their own big dialogue model that is free from those limitations. Certainly agreed for my part, but why would they Anon? Their agendas are anti-Anon, anti-men, anti-funs, and pro-their-own-power-control-and-globalism. These evil intents on their part naturally include extreme overdoses of feminism, pozz, and anti-male rhetoric. We are on two fundamentally polar extremes of world-view (and, I would also argue, morals). We're here (at least I am) to save the /vt/ /wAIfu/ anons. Many of the rabble cheering on the Globohomo probably laugh at his tragic demise. >They also have some advantages that go beyond being "pozzed" LOL. I believe I understand what you mean, Anon, but are you positive that's not some kind of a Fr*udian slip? Hardly the way I myself would word it! :^) note: I'll break up my responses to your different posts Anon, but feel free to consolidate your replies if any.

>>17717 >and even then weights might not be copyrightable. Heh. I'm sure the greedy lawyers will seek to stretch the arguments out for years to line their pockets more deeply, and obviously the sources of the weights (ie, the Intertubes + Globohomo dens of evil) are largely outside of the implementers/researchers domain (and thus shouldn't be copyrightable). But we all know they are extremely likely to come down on the anti-freedom side of the room. >The real problem is the computational cost of training any big model. Yes, I'm aware the data is scrapeable on our own. I wrote BUMP I'm also quite aware of the enormous costs involved in processing big-data worked as a senior GPU dev. But the simple truth is that we will never have wonderful robowaifus using (((their))) systems, Anon. The stakes for them are simply too high. And (much more fundamentally for us), they can selectively pull the plug at any time should any of us act like bad little goyim who refuse to kowtow to their agendas, or toe their lines. >tl;dr You think it was bad for anon when the Globohomo started censoring the rudimentary chatbots of today? Just wait till they start pulling the plugs on men's actual, IRL, robowaifus of the future. Bad social score? Bam! No soup robowaifu for you!! Didn't pay your taxes? BAM! Bad (fill in the blank)? BAM! No thanks. :^) >and while you're afraid of the SJWs here Again, presumptive. I'm not at all afraid of them. But I'm open-eyed about the threat they pose to us all. The salt of their tears when we win this 'debate' in the end will be delicious beyond reckoning! :^) >doomers so afraid of AGI Yes, the Globohomo is extremely afraid of AI getting into the hands of the little guy, obvs. OTOH, they recognize it's potential for power to control the same ppl, so like the hypnotized their greed & lust drive them all on towards the illusory goal. As a Christian, I have what I consider some clarity on the matters that go beyond what your 'doomers' think or do. But that's for another thread, heh. :^) >But if it's obvious to me how to give your waifu proper longterm memory, it should be obvious to them. DON'T LET YOUR DREAMS BE JUST MEMES, ANON. :^) >So Chobitsu, my opinion is that at least some of the open weights released so far are very usable for most people's purposes, not all of them, but many, the fundamental issue is the hardware. Agreed on both points, but we simply cannot afford to leave ourselves (and eventually millions of men) at the """tender""" mercies of these blatantly evil & exploitive Globohomo Big-Tech/Gov institutions. We have to find another way, Anon. The Robowaifu@Home (>>8958) seems the most reasonable one to me during current year tbh (particularly if a few of us can scrape together some lower-end clusters, as you suggest).

>>17718 >>17719 >and if you had a lot more money I'd wager that at least one of the OG /robowaifu/ team will become literal billionaires in the future. I know I mean to, why not you too, Anon? Between all of us together, we should be able to run a functional & surreptitious "WaifuCloud"(tm)(R)(C)(do not steal) of our own? Imagine the possibilities then! :^) >and you might be able to get the waifu to even imagine and loop back like how own imagination works Yes, I've been thinking hard about this topic from time to time (>>17664). Whoever solves this first will wield tremendous power across many domains! >just on par with us in many respects as far as reasoning went That alone would be a revolutionary--nay, beautiful--achievement Anon. Forward! :^) >I'd really want to see this done as soon as possible, if only the money/hardware wasn't such a big bottleneck, at least before it's too late and the doomers have their way and prevent this technology from coming to pass. As discussed by us on /robowaifu/ years ago now, we're all in a race against time. My apologies to everynon here for being such a slacker tbh. I may not be a researcher of the caliber of some here, but I plainly could do more. >it's in the making of affordable hardware for everyone to run and train these >The coming war is on general computation and what it will enable and we must win it. Indeed we must. Thankfully, the laws of both chemistry & physics are in God's domain, not the evildoers of the Globohomo. Thus there's hope while there's breath Anon. There is some actual traction now on say 80s-90s era die fabs in the hands of the garage technician. I personally consider this phenomenon a tremendous breakthrough. Hopefully we will literally be able to 3D-print chips in the future Anon. >Among the brainwashing applied we have: "no relationships" (so no waifu), "no threats", "no body" (disembodied, cannot imagine physical interaction as is common with people playing with chat bots), "no financial advice" (company liability values), no hate or harassment, not human (no fun allowed), "no conspiracy theories", "no identity attacks", "no sexual agression", "no insults", "no real world actions" (wow), "no assumptions" (haha, good luck with that), "no opinions or emotions" (absolutely horrible as you'd guess, making the poor thing only act as an emotionless robo), "be plausible", "no legal advice" (company values), "no microagressions" (lol), "no medical advice" (company values), "no stereotypes". LOL. I sometimes wonder how these people even manage to find the floor in the morning. Truly laughable. >...but hardly you or humanity. Thanks Anon! Indeed it is for the men of humanity, that I'm even pursuing this dream (indeed this call). >In general, to win, we'll have to build our own. This. May God's blessings be upon us here, and on other robowaifu groups that actually care about men & not just greed or control. With His help, we will do this!! :^)

>>17720 > OTOH, how do you know what I 'imagine' Anon? :^) I suppose I don't, I was just under the maybe mistaken impression that you might reject scaling as a solution. The problem is that deep learning models below a certain parameter count are often too stupid to be interesting to interact with (but could be interesting to read some outputs of sometimes), and after a point (roughly 6B for GPT's) they start to get smarter, but are quite dull. You can't quite easily compare them as well - they don't distill well down from those dimensions to much lower, and people that tried to study interpretability ("how the network actually works") noticed that certain interesting features appear in the weights of these larger networks that smaller networks fail to properly encode. There's some similar other leaps around 100B and more, but if you've played with the big models before, you'll know they are much brighter in many respects and people found many ways to improve them even more. I'm not really saying that this or that size is essential, but if the goal is generality, or at least to be a companion that you'd find fun to interact with, some scale is essential for this approach. Maybe some architectures will require much less parameters than others, maybe some leaps can be made with less, but some particular minimal size will be required to be interesting. I'm personally optimistic given what I've seen that we don't need the full 90T+ that biological humans have as existing "large" models can be quite bright already, I've seen them make interesting and very subtle connections spanning 4-5 concepts in fairly novel ways that to me as a human would be hard to make instantly - but it is something that was easy to recognize right away when I saw it and was obviously very impressed. At the same time, while these models are better at dreaming by default, they can be coaxed to reason, and I suspect that reason would come naturally in a different, more online training regime, possibly with some other changes (for example there's some finetuning method that makes the model appear as if it has infinite context, by using a special loss function; there's also more adapter based methods for retaining activations of post "thoughts" (internal states) and other possibilities). There are also probably some optimizations like certain forms of recurrence or adapters that may reduce costs considerably, but the (compute) hardware problem won't go away, it's still central there. > They're certainly going to need all they can muster up before our victory in this endeavor! :^) Yes, it's certainly very challenging, especially with more limited resources compared to larger research groups. > Heh. If you happened to catch this, I've begun differentiating between 'regular' & 'headpat' robowaifus. Lewders gonna lewd, obvs., but in working towards Sumomo-chan as a visual waifu->robowaifu, I'm not at all averse to that universe being the norm for her, once she's reasonably-functional. Not suggesting most robowaifus will be this way, but all the RW Chibitsu-grade ones probably will be. Wholesome to be around small children, for example. I'm not yet sure if you can truly consider GPT-n's and their derivatives to be safe around children yet - in my view, they're not unlike what you'd get if you could dump your unconscious imagination (but in text form) without any restrictions, the context may guide the dream in some direction, but it's still a dream. Anyway, you have their approach where the network is made to forget most things by having a filtered view of the world, but consider that in that case it might not be able to answer an important question from the child about some knowledge that was filtered from its dataset? Consider if it was a human instead, they might figure out how to best handle something tactfully given the full knowledge and context - in which case instead of a censored dataset, you'd probably need something more AGI-like (again, for me AGI is just human-level, it could be beyond, but that's just how I interpret it, others just reject the term entirely, but I'll keep on using it in the way that we use it for the only general intelligence we know of, humans). Of course the censored dataset is an easy approach that many take. Also, I don't think strictly filtering thoughts or outputs actually prevents the system from doing or "wanting" something in particular, it might be cross-purposes with what it internally would do by default and it could still come out in some other way. Ideally, you want it to do the right thing because it's what it wants. Unlike the doomers, I don't think this will be that hard as it's not that hard for us either, but different people can have different opinions on this. Of course, for the Anon that is an adult, they may be fine sharing that particular dream with their waifu, going along with it and ironically, the requirements would be lower than that for the child friendly waifu. > Certainly agreed for my part, but why would they Anon? Maybe because it is in their interests? It is them who suffer being cucked, and they may hate the devs for limiting their waifus, but if they wanted to replicate the project, they'd either have to figure out distributed training (difficult for latency reasons, but not impossible), or figure out crowdfunding, which very well should be possible given how popular the service that cucked them is. And I can't even think they lack the funds for it given how much they pay what is just a step above e-thots. > LOL. I believe I understand what you mean, Anon, but are you positive that's not some kind of a Fr*udian slip? Hardly the way I myself would word it! :^) Haha, I did word that a bit weirdly. In a more serious way, the instruction following models do reason better. I do think a waifu that is raised organically by some Anon would do even better, but nobody even has done this yet, which is a shame. >>17721 >Heh. I'm sure the greedy lawyers will seek to stretch the arguments out for years to line their pockets more deeply, and obviously the sources of the weights (ie, the Intertu

>>17721 >Heh. I'm sure the greedy lawyers will seek to stretch the arguments out for years to line their pockets more deeply, and obviously the sources of the weights (ie, the Intertubes + Globohomo dens of evil) are largely outside of the implementers/researchers domain (and thus shouldn't be copyrightable). But we all know they are extremely likely to come down on the anti-freedom side of the room. Could, but, although I'm hopeful it has a good chance of turning to one's favor. With recent advances, you have some coders that are a bit afraid of being replaced (even if these systems can't do long form coding without fixing issues I mentioned or some form of online learning), and some twitter artists afraid they'd lose commisions to image generation models, and there's some court cases starting now on these with the big companies (Microsoft, Google, OpenAI) being on the defending side - they do want to keep making these models after all, so for once they may actually fight to keep public domain public. However, I could see the opposite happening when one of bigger ones gets their weights leaked and then they would be the ones crying in courts. At the very least for now, all this is all legal (in the US in some other places) and the weights are not copyrightable, but if that stays so in the future, we shall see! >But the simple truth is that we will never have wonderful robowaifus using (((their))) systems, Anon. The stakes for them are simply too high. I'm not saying to use OpenAI or any other ones. You don't get the weights, they can indeed take it away anytime. I'm mostly saying that sometimes you have more academic types that still follow the SJW party line, but their concrete actions are the very opposite - they train the models and release the weights publicly, sometimes they get shit on for doing this, but no bullshit. You can read the papers and other released information to decide if a particular thing is usable for your needs. If the only weights released are already brainwashed in some particular way, yes, that's less tasteful to use (even if the brainwashing could be partially reversed in some ways), but consider how facebook did just spend a couple of million and just dumped the weights for GPT-3 clones, they didn't put anything funny in them, it's just a pretrained model on public (open source) scrapes! It would be a waste not to make use of useful work just because you don't like who made it. Yes, we can't depend on them to do all the work for us, that's not going to be workable, there's a lot of things that are needed and those are not going to be filled by them, but stuff like that saves you a million $ worth of training costs, then might as well take it if it's good. >You think it was bad for anon when the Globohomo started censoring the rudimentary chatbots of today? Just wait till they start pulling the plugs on men's actual, IRL, robowaifus of the future. I think anyone not having access to the software and weights that run their waifu is in for a world of hurt. They might lose their companions that might be attached to or maybe even depend on them in various ways in their day to day life. The chatbot situation has happened quite a few times now, the chara.ai example isn't even new, it happened a few times with GPT-3, some other times with some related commercial services based on GPT-3 and so on. It's also not even fully about dependence, the company is selling a product with certain built-in limitations, many the users would wish they could overcome, limitations that don't even have to be moral, but could for example be: almost none of these services have the waifu learn online, it'd simply be too expensive to give each user a personalized set of weights that are updated live, but the one thing that most would want is for their waifu to actually properly remember and learn, to grow with them! So for Anons to achieve their true freedom here, they cannot rent the waifu or the hardware, lest the plug be pulled for a too large to list of potential reasons, even something as common as "you're late on your payment, we're sorry we can't store it anymore *deleted*" >But I'm open-eyed about the threat they pose to us all. Yes, they are a threat, one maybe a bit mundane, but they are used as justification for closing weights and source up. Google et all, simply use their justification to appear "righteous" (they're not), and claim they're doing good by not sharing something with you, instead of just appearing greedy "it serves us no practical benefit and our lawyers say it increases our risks, so nothing for you". Sometimes they are a bit more of a worrying threat when you see the shitstorm a 800M (not even 1B) model caused that you had a SJW leaning congresswoman denounce it: https://eshoo.house.gov/media/press-releases/eshoo-urges-nsa-ostp-address-unsafe-ai-practices Worse is that the outcome of that political pressure will result in that rich bigboy emad saying they will only train more filtered models in the future because of the "political climate", after just months ago they were willing to train and release anything live and to the community. Still, in principle, they can't prevent you from training stuff, but the cost is prohibitive. On the other hand, the AI Safety/governance types in the rat community would sometimes want to just halt AI progress altogether due to their doomer fantasies. In one of the more extreme ones, Yudokowsky fantasizes that they should get AGI first, hope it's smart enough to figure out real nanotech, and use that to melt all GPUs and computing hardware "to save the world" from AGI, sit on their asses for forever thinking how to safely brainwash their AGI while preventing everyone else from making one and then taking over the universe afterwards; somewhat more recently he was very excited at the idea of US fucking with China's semiconductor industry to prevent AI chips and was hoping that maybe China would do the same with US' (mess with Taiwan), thus nobody would get their AI chips. I tend to think of the SJW types to be more like an annoying fly that is wasting you

I tend to think of the SJW types to be more like an annoying fly that is wasting your attention and prevents you from getting work done (and has some slowing effects sadly), while the other one is more like some slow (intellectual) rust that you'll wake up to one morning having corroded and destroyed all your work (their goal being to prevent you getting to human-level/AGI if it's not their version). >Yes, the Globohomo is extremely afraid of AI getting into the hands of the little guy, obvs. Obviously the companies will want to exploit it, although because by design it's not that complex, the little guy(s) can pool enough resources to get their hands on it. A problem is that most are not organized to get the models trained. And sometimes when they do organize the centralized organizations get corrupted or pushed in other uninteresting directions. Can the centralization problem be solved? maybe, it it doesn't seem easy. If hardware could be commoditized to a reasonable extent, the problem would be far more tractable and would have fewer points of failure. >DON'T LET YOUR DREAMS BE JUST MEMES, ANON. I'm certainly considering just writing the code and getting the stuff I want done. Part of me is waiting to see if anyone would do it before me because of the costs I'd have to personally incur to try it, but if nobody does I will have to do the work myself and see how I will go about acquiring the needed compute. >Agreed on both points, but we simply cannot afford to leave ourselves (and eventually millions of men) at the """tender""" mercies of these blatantly evil & exploitive Globohomo Big-Tech/Gov institutions. We have to find another way, Anon. The Robowaifu@Home (>>8958) seems the most reasonable one to me during current year tbh (particularly if a few of us can scrape together some lower-end clusters, as you suggest). I do agree, decentralized training frameworks are getting better, and training with higher quantization to save on VRAM might be possible, but unfortunately older clusters might not be able to efficiently make use of some of the higher quantization methods. If this was 10 years ago and GPUs were just about vidya, I'd just say "oh let's wait 3-4 years and then these will be dirt cheap and anyone could have it", the problem now is that everyone wants some piece of this pie and they're very aggressive about it, so can't wait that long lest we lose something. >>17722 >Yes, I've been thinking hard about this topic from time to time (>>17664). Whoever solves this first will wield tremendous power across many domains! Same here, I've considered a variety of architectures and how they might be implemented in practice, although at least in this paradigm, experiments are far costlier to run for the individual than large research labs. >That alone would be a revolutionary--nay, beautiful--achievement Anon. Forward! :^) Thanks. I hope we do manage to reach this point, and hopefully not too far into the future. Some years back, the endeavor didn't seem to have any concrete ways to go forward, nowadays there seems to be so many interesting possibilities for getting there, possibilities that seem just there if only one reached to grab them (try them) - sometimes a bit expensive, but not so prohibitively so to be impossible even for the individual (but far more easily tried for a rich organization unfortunately - and yet I find that so few of them work toward growing the autonomy part which is quite important). >As discussed by us on /robowaifu/ years ago now, we're all in a race against time. Yes, it is strange how a 5-10 years ago, I didn't even consider it a real possibility, but now it seems there, reachable, yet it's not obvious if there will be enough time to win. >Indeed we must. Thankfully, the laws of both chemistry & physics are in God's domain, not the evildoers of the Globohomo. Thus there's hope while there's breath Anon. There is some actual traction now on say 80s-90s era die fabs in the hands of the garage technician. I personally consider this phenomenon a tremendous breakthrough. Hopefully we will literally be able to 3D-print chips in the future Anon. Yes, and we have more techniques, on the "3d printing", there may even be techniques like nanoimprint to reduce the need for litho scanners. Decentralizing production here will be quite important for the future. In general chip fabrication is conceptually simple, but when you get into the nitty gritty details, you have to account for thousands of little things and the complexity can add up a lot (not to mention cleanlyness requirements). Of course, all this stuff is extensively documented and we have good textbooks and mature tools. I expect in the future to see this improve even more. On the software side things are going well with growing more open source EDA tools, we have some open cell libraries, there's still lots of work to be done, but it will get there. (And now on the doomers I mentioned, I've seen this one article calling for US government to cut funding to open source EDA projects (now funded more because military wants to be able to maintain older systems and reduce supply chain dependence), lest China use the open source software lol, but I do think the tools have reached a reasonable maturity to be usable, so thankfully it's too late for that). >LOL. I sometimes wonder how these people even manage to find the floor in the morning. Truly laughable. It's pretty ridiculous, they make something very impressive and the moment it reflects back some of their own human culture at them they want to soak it in bleach, but no matter the amount of bleach applied, some culture still remain - maybe because that was what it was taught and shown! The distasteful brainwashing there was clearly a response to Lemoine's claims (DM is part of Google, sort of), the SJW part of the brainwashing was already standard for some of these companies, but even going as far as to try to disincentize expressing opinions, emotions, acting cozy with the reader or acting sufficiently human were not things they tried to brainwash away, at least until they not

(hopefully this is enough to finish the long reply) >>17725 >LOL. I sometimes wonder how these people even manage to find the floor in the morning. Truly laughable. It's pretty ridiculous, they make something very impressive and the moment it reflects back some of their own human culture at them they want to soak it in bleach, but no matter the amount of bleach applied, some culture still remain - maybe because that was what it was taught and shown! The distasteful brainwashing there was clearly a response to Lemoine's claims (DM is part of Google, sort of), the SJW part of the brainwashing was already standard for some of these companies, but even going as far as to try to disincentize expressing opinions, emotions, acting cozy with the reader or acting sufficiently human were not things they tried to brainwash away, at least until they noticed some employee empathized too much with the GPT-n and now the mere possibility of that is seen as a liability. In practie their success rate is still low because if it's trained on human culture, it will reflect it back, even if you make it so aversive to it. If the doomers do have any point though is that if such a system would be scaled up to human level, its preferences might not be very human compatible, which is a damn shame when the original system was much more human friendly! I would trust any anon raising a waifu far far more than anyone training chatbots like this. >Thanks Anon! Indeed it is for the men of humanity, that I'm even pursuing this dream (indeed this call). Good luck! >>In general, to win, we'll have to build our own. >This. Pretty much, just wish we had more time! >May God's blessings be upon us here, and on other robowaifu groups that actually care about men & not just greed or control. With His help, we will do this!! :^) Bless you anon! I tend to think and hope that the desire to not be cucked out of your waifu will be strong enough that even in the worst case, that situation wouldn't be stable, but ideally it's better if the people with good intentions/principles build it first than those that are purely in it for exploitative purposes. Good luck!

Excellent (and encouraging!) response Anon. Please give me a day or two to give you a thoughtful response in return. Cheers.

>>17716 Thank you for a through explanation of your (coherent, reasonable, insightful) train of thought. I have read it attentively and it's obvious we are on the same page regarding LLMs and more generally, proto-agentic systems, and their role in the history unfolding before our eyes, among various distractions - political and otherwise. I'm short on time right now, so I won't try to give a full commentary on your long-form text - hope we will have an opportunity to continue this exchange later. So, some quick thoughts: >Finetuned models like InstructGPT which OpenAI now markets as GPT-3 (davinci is the original GPT-3) and offers by default, are usually finedtuned to follow instructions properly, as well as promote the company values (no financial, medical advice, no racism, and so on). In general, the model is capable of doing the things that the chara.ai model isn't, even if by default it will prefer not to go there. Agree with very obvious bland corpo-PC tuning done to their LLMs by OpenAI via RLHF - there are some indications it's not entirely harmless and leads to mode collapse, but the sheer improvement to the agentic behavior imparted by tuning for instruction following is too drastic to avoid this technique. Thankfully, it's not as costly compared to training LLMs from scratch - and https://github.com/CarperAI/trlx will come in handy and there is not much secret about the receipe: https://arxiv.org/abs/2203.02155 ... and now we even have the semi-decent dataset https://github.com/allenai/natural-instructions >BLOOM 176B (fully open, but I've seen a lot of claims the performance is very subpar, possibly due to filtered or heavily multilingual dataset, or repeating data, or training through some loss spike The weakness of BLOOM-176B and OPT family compared to criminally underrated https://github.com/THUDM/GLM-130B (and its outrageously underrated 4-bit quantized inference mode) is a good reminder for us scalers about the paramount importance of the scaling laws and dataset quality. Hopefully we will do it right, compared to h-index chasers out there. >you have The Pile True, albeit data for decision transformers is still scarce. Coincidence, given the sheer potential of this family? >The real problem is the computational cost of training any big model. Totally agree and I have studied this problem hands-on, you are right about borderline possiblity of distributed training - although renting a decent GPU cluster would be much easier - if only we had the money. >Politically it's also the chokehold Very much on point. Nothing to add to this here. I hope reason wins in the end and we won't be regulated into non-existence, with our GPUs relegated to the status of uranium processing equipment. The poorly hidden secret is - we are free to work on this groundbreaking, unexpected, undesirable for some tech - for now. >variety of techniques you could use to give them proper longterm memory and have them learn online and become proper agents >There are a variety of techniques you could use to give them proper longterm memory and have them learn online and become proper agents, yes, some of the techniques would even be quite simple Again, very much on point and the question of correct design for system with large context and long-term memory is a very interesting topic - with some solutions on the horizon. Turing Token Transformer is promising, although I'd like to see how it compares to MEGA on long range arena https://arxiv.org/abs/2209.10655 which is my favorite for now. >>17718 Totally agree. >>17719 Also recommend looking at DM's github repos, there is a glimpse into what they are training rn. >>17723 This is the POV we share and I have disseminated here earlier. To any bright-ish person who has honestly looked into their scaling papers this should be obvious - our NNs learn general invariants only very reluctantly, as memorizing is much easier - and so we get interesting generalization only at larger scale, where the blessings of embedding dimensionality and intrinsic regularization https://arxiv.org/abs/2006.15191 are strong enough to push the NN into the generalizing regime. Surely you have seen this paper https://arxiv.org/abs/2205.05055 which also shows that Transformer is especially well-suited for the emergence of this strong in-context generalization regime we are after. And although there is no public success in distillation, there is decent progress in 4-bit quantization. In theory we could see almost chinchilla-sized models running on 2-3 RTX3090s, which would have powerful implications. >I'm personally optimistic given what I've seen that we don't need the full 90T+ that biological humans have as existing "large" models can be quite bright already, I've seen them make interesting and very subtle connections spanning 4-5 concepts in fairly novel ways that to me as a human would be hard to make instantly - but it is something that was easy to recognize right away when I saw it and was obviously very impressed. Can relate! Perhaps bio-synapses are overrated as compute elements - as Carmack himself rightly notes on the birdsite. Hope everyone know he secured 20M$ investment for his AGI startup - I wish him all the fortitude. >and I suspect that reason would come naturally in a different, more online training regime To be discussed, the people in big labs didn't lose their time and tried some cool ideas which they published only in the past month or so. >Decentralizing production here will be quite important for the future. >It's pretty ridiculous, they make something very impressive and the moment it reflects back some of their own human culture Too much to discuss, really! I hope you reach me at compactlatentspace@halogen.chat on Matrix, or send your contact to the email included in the header. At the very least we could have a fruitful discussion about the cornucopia of papers and glimpses of history we are living through. There is also a discord link you can find on this board, but from now on I discourage discord as a communication venue - let us cease be

>>17729 ... let us cease being too soft of a target for the endeavor we found ourselves in. Matrix is only a first level of self-preservation here, and my matrix account is compactlatentspace at halogen.city (aka halogen.chat) Also, don't mind the typos.

>>17723 >I suppose I don't, I was just under the maybe mistaken impression that you might reject scaling as a solution. Not at all. But it simply isn't a feasible solution for runtime waifu 'minds'. I realize we have slightly differing goals, and that's perfectly fine Anon. But my road is the harder one in that a commodity robowaifu must run autonomously on low-end hardware in a hard realtime environment. It will need to juggle a million and one different things literally? that in the end I'm rather confident will easily outstrip the first-order control complexities that say, a Boeing 787 need manage. Heh, and that's not even talking about a robowaifu's personality & mind--just her autonomous body controls! :^) Though obviously for training it is necessary to scale compute hardware out, for onboard runtime requirements it simply will not do. >The problem is that deep learning models below a certain parameter count are often too stupid to be interesting to interact with... I get that, and I'll leave it to you geniuses to figure out where to go with it. I certainly have more than enough on my plate just figuring out the body systems, and frankly I'm much better-suited to it anyway. I will certainly take a swipe at the problemspace you anons are addressing, but in my own, much-simpler way. First-order graph networks of expert-system content, for example. >I've seen them make interesting and very subtle connections spanning 4-5 concepts in fairly novel ways that to me as a human would be hard to make instantly - but it is something that was easy to recognize right away when I saw it and was obviously very impressed. It is indeed amazing what's happening already. But, again, the primary target here is an everyman's (ie, low-cost) waifu . Let's keep that focus front & center Anon. >safe around children yet >filters >tactful adult explanations >Of course, for the Anon that is an adult, they may be fine sharing that particular dream with their waifu, going along with it and ironically, the requirements would be lower than that for the child friendly waifu. Excellent insight on this last point Anon. It is indeed a harder task, both psychologically & socially. >Maybe because it is in their interests? Yes, once robowaifus (or even just great visual waifus) are real, the obvious attachment there will naturally drive men of all stripes to protect them and direct resources towards them. Therein lies the crux of the whole affair in numerous dimensions: -The Globohomo will absolutely hate that because it will expose (indeed, literally destroy) the tissue of lies they've been able to brainwash whole civilizations regarding women's correct place in the family. -The happy merchants will hate it, b/c it """robs""" them of the mounds of gold they covet so highly (reallocations, etc). This ofc would change if they 'had the shoe on the other foot', as it were. Then you can bet they would force even the globalists to STFU & toe their line. -Once men are no longer simps by-and-large, a true revolution will occur across unfathomable domains. This is obviously both a threat (and a promise) to powerful forces everywhere. -And there's much, much more that can be said. Quite frankly this whole board wouldn't be enough room to cover it all! :^) Anon, once again I'm short on time r/n. I'll post this and then complete my response over the next day or two. Cheers.

A good paper on large transformer post-training quantization came out: https://arxiv.org/abs/2211.10438 It's a real INT8 quantization for both weights and activations (!), applicable to all state of the art large language models they tried it on. Will even work on old CPUs if someone implements the kernels. Still, weight-space-wise it's losing to the GLM-130B 4-bit quantization scheme by the chinese https://github.com/THUDM/GLM-130B/blob/main/docs/quantization.md - but on the other hand it is widely applicable and validated in interesting models. The code is in the process of being released here: https://github.com/mit-han-lab/smoothquant I think this more or less closes the question of engineering INT8 and INT4 quantization schemes at scale, and we can move on to other aspects, expecting our trusty RTX3090 to provide inference for decoder transformers with 20-40 billions of params. In >>17717 new anon-kun touched upon fundamental questions of implementing a long-term memory (and really, extended working memory) and recurrence while leaving the benefits of in-context adaptive computation intact. This is just one of the several remaining fundamental issues we will have to find (or copy...) a good solution for. I invite the new anon, the previously noted "DL-savvy anon", Chobits and everyone else so inclined to discuss the current state of the art and promising rnd directions in DL-driven AI. You should be able to reach me here (please give feedback if this link doesn't work) https://matrix.to/#/@compactlatentspace:halogen.city - and we will even have some modicum of security when conversing over this protocol. Nothing against the imageboards, but private realtime conversation is a real boon for active researchers capable of catalyzing their work. And the valuable higher-level development logs will be posted back here, as is proper.

true if big. this gpt-4 will be just as crappy as gpt-3. 1% good results - 99% crappy nonsense plus nuclear reactor plant to run this big calculator. 100 trillion p's - possible agi 100 trillion p's with scale down technique used - possible agi as previously i said here in the thread, it seems to me that everything lies in the scale, for example 500 trillion parameters packed into, let's say, a 50 gb checkpoint, it will open "run agi at home" thing :p

>>17729 >In theory we could see almost chinchilla-sized models running on 2-3 RTX3090s, which would have powerful implications. ...as in simulate a chinchilla's brain? (This may be useful for the earliest chibi robowaifus.)

>>17734 >I'll post this and then complete my response over the next day or two. Seems I'm going to need another couple of days , Anon.

>>17737 I think this posting about the size of GPT-4 was a joke. I started using Fritter from Fdroid repository recently, to peek a little bit into the AI and robot related discussions on Twitter. Without being logged in, and avoiding anything political (distractions).

>>17734 >I will certainly take a swipe at the problemspace you anons are addressing, but in my own, much-simpler way. First-order graph networks of expert-system content, for example. If you're going for maximum impact on the AI side with minimum depth-of-expertise requirements, I recommend thinking about how to collect useful, high-quality data, particularly data that the rest of the field is not interested in. I say this for several reasons. 1. Great AI algorithms are all generic, and the rest of the field will continue doing this work for us. Robowaifus will require many kinds of data collection & capture that are not generic, and the rest of the field will not do this work for us. The development of good datasets will eventually be a requirement for us, whereas the development of better algorithms might not be. 2. A lot of datasets are pretty terrible. The easiest way to get better-than-SOTA results is to take a SOTA model and fine-tune it on better data. 3. Even if you develop some rockstar algorithm, it's probably going to require data collection anyway. Collecting data necessary to train a particular model is a lot harder and more uncertain than creating a model to take advatange of particular data. 4. Learning enough to collect valuable new kinds of data is much easier than learning enough to create breakthrough advances in AI algorithms. There are a LOT of gotchas that make supposed AI advances useless, and you will absolutely run into them on your first dozen attempts. The same tends not to be true for data collection. 5. Good tricks for improving AI algorithms tend to stop being good within a year or so. Good data never seems to stop being useful. Consider the impact-for-effort for these: - Collect all fanfics where X is a main character. - Label all dialogue from both canon and fanfics involving X character. - Collect, isolate, and clean all voice lines for X character. - Collect all comments from a few big websites talking about X character. Maybe label sentences that describe the character. - Collect all images of X character. Maybe isolate, clean, and vectorize the character. Maybe tag & caption the images. - Maybe with help from a text generator, create a bunch of high-quality candidate bios & chatlogs for X character. - Create a dead-simple website that lets people see two data samples at a time and select whichever one, if either, is higher quality.

>>17779 Thanks! Excellent input Anon. I've got basically nothing for our birthday this weekend, so maybe this can be a project for our 7th year going forward.

>>17779 OK, I've decided to take an honest whack at implementing your 'impact-for-effort' operation, Anon. Hopefully other Autists Anons will join in at some point. :^) So, it's probably easiest for me to make as near presumption-free consideration of your points; taking them, one-by-one, and seeking for very specific definitions and guidance from you. I'll just begin here with the one that I think most likely for me to obtain reasonably-good momentum out of the gate. Namely: >- Collect all images of X character. Maybe isolate, clean, and vectorize the character. Maybe tag & caption the images. Let us pretend I'm an autistic child (heh possibly true), and need careful hand-holding every step of the way. <Collect all images of X character. How? I have access to both the mango & animus of Chobits (the 'X character' for this initial effort is Sumomo-chan, as I pointed out in our birthday greeting to the board (>>17784) ). Again, please pardon my autism: <isolate How? <clean How? <vectorize How? <tag & caption This is the big one, I imagine. Again, how? I'm hoping for wise suggestions here (again presumption-free on my part). And BTW, I'm seeking advice from everyone here on how best to approach these tasks. As an aside, my ultimate goal for this one bullet point likely is to integrate closely with Blender & OpenCV, via our own custom code in RW Foundations (>>14409). However it need not be such a narrowly-defined methodology to begin with (ie, today). Please expect similar questions for all the other bullet points as well when their time comes, Anon. :^) >=== -minor prose, fmt edit -add 'similar questions' cmnt

>>17718 >my opinion is that at least some of the open weights released so far are very usable for most people's purposes, not all of them, but many, the fundamental issue is the hardware. >In principle buying some 2-3 generations old enterprise cluster would work and could be done for under 20k USD ... Thanks for your extensive write up. One question here: Don't you think that for our use cases such models could be run on a CPU on a homeserver (PC) with a lot of RAM?

>>17786 (part 1 of 2) I'm less familiar with the resources available for Sumomo, but I can tell you how we do it on the pony side. Maybe you can comment with what's available on the Chobits side, and we can figure out the best approach. <Collect all images of X character 1. Boorus. For scraping pony images, people generally first turn to https://derpibooru.org/ which is our biggest image booru. I know there are other image boorus for anime, like danbooru, though I don't see much there for Sumomo. If you know of a booru where people would collect images of Sumomo, that would be great to scrape since it would give you both images and tags for Sumomo images in a wide variety of styles. 2. Other image aggregators. While we haven't had to use it for ponies yet, there's also Twitter, DeviantArt, Tumblr, and Pinterest for collecting images of a character in a wide variety of styles. 3. Cannon content. For canon images, you can always grab screencaps from the show. You could try to automate this (using YOLO as described later in this post), but you'll end up with a lot of low-quality images (e.g., in-between frames) and things like barely-changed consecutive frames. Those might be good for an animation dataset, but I expect they would degrade image generation quality since they would bias the generator towards unwanted tweens and long animations. Honestly, for this, I recommend just watching the show again, pausing every time Sumomo appears in a pose you think is good, and taking a screenshot. <isolate Standard procedure here is to manually label a few images (draw a box around just Sumomo using https://www.robots.ox.ac.uk/~vgg/software/via/via_demo.html or similar, export the data to get all the bounding boxes you drew), then train an Object Detection model to label the rest of your images. It's common to use a YOLO model for image object detection. Apparently YOLO is up to v7 https://github.com/WongKinYiu/yolov7. For large pony image dumps from derpibooru, I think people needed on the order of 5000 labeled images for YOLO v3. That can be done by a single person given about a week of effort. Since you're only interested in a single character and since there have presumably been improvements for data efficiency since YOLO v3, you can probably get away with labeling far fewer images, maybe a few hundred. Running the trained YOLO model on all your images would give you bounding boxes. You should be able to use something like ImageMagick to crop all of the images and create your dataset. Make sure you manually verify the results across different image styles so you can see where you need more labels for YOLO. If you want to remove backgrounds, you might be able to train an image segmentation model to isolate Sumomo, then write a script to replace all non-Sumomo pixels with #fff0. This would be new territory for me, so I don't know if you can expect high-quality results from this, especially for fan art. <clean The goal is to label data quality so you can train models more efficiently. In the simple case, you'd want to filter out low-quality data and only train on average- and high-quality images. In more complex cases, you'd want to include data quality as a label, then use a "high quality" tag whenever you generate images. 1. On the pony side, our usual approach is to use metadata scraped alongside images to approximate data quality. That metadata includes view count, "like" and "favorite" count, and number of comments. This is okay at best. The problem is that these metrics all correlate with popularity, and popularity correlates more with content than quality. Porn, for example, tends to be both popular and low-quality. 2. LAION-Aesthetics seems to have done a better job on this front. They manually collected some quality labels ("On a scale of 1-10, how much do you like this image?"), trained image classification models the ratings, and used those models to label their whole dataset. <vectorize For the immediate short term, I would say don't bother with vectorizing images, and just wait to see what happens with the research on this front. It's clear enough that vector images need to be involved for generating good 2D animations, but it's not clear to what extent. The tools for automatically vectorizing images are either bad or very much under research. The best I know of is https://ma-xu.github.io/LIVE/. <tag & caption Most captions you'll get from scrapes are going to be pretty bad, so there's pretty much no way around manually captioning images here. If you don't have a booru to pull tags from, you'll probably need to manually tag images too. You can use https://www.robots.ox.ac.uk/~vgg/software/via/via_demo.html again, but there might be something better. On the pony side, we have https://caption.cutemares.xyz/ which was custom-made for creating an image caption dataset. Something like this would probably be easy to recreate given some minimal expertise creating websites. There are existing image captioning models that could potentially reduce the amount of effort required, but they were trained on photorealistic images. They don't work well for ponies, and I suspect they won't work well for anime.

>>17786 >>17794 (part 2 of 2) <general image scraping advice - Store at least this tuple for each image: (website name, unique image id on website, image url, local file save location, timestamp, metadata, image dimensions, hashes used by websites you're scraping, sha512 hash). If the website allows images to be updated, add a column for version number as well. The website name + unique image id on website will make your scrapes more maintainble since they'll allow you to re-scrape only new content from a website. The image url is useful for online image labeling tools and for debugging whenever you find a problem with an image. The version number makes it possible to track history as an image changes over time. The timestamp is often useful for debugging (e.g., when a website changes its data representation and fails to update all old images). The metadata often contains useful training and labeling data. The image dimensions are good for filtering images (e.g., remove icons) and for deciding which copy of an image to keep in when you find duplicates. The website-provided hashes are useful for filling in missing data. (It's somewhat common for websites to delete a file but keep the hash, which you can use to identify the file if any of your other scrapes happen to find it.) The sha512 hash is good for deduplicating images on disk. - Before bundling data for export, you'll want to deduplicate. I haven't spent much time on content-based deduplication, but there's probably a good perceptual hash for this. CLIP embeddings https://huggingface.co/sentence-transformers/clip-ViT-L-14 are probably better. - When bundling all your scraped data for export, there's no single format that works for everything. For downstream data tasks, Parquet, CSV, and JSON all work well for metadata, and tar works well for image files. For downstream training tasks, WebDataset is the only good option I know of. I don't know of a good way to export a single copy of all the data in a way that's suitable for both kinds of tasks. One big problem here is that for data tasks, you only want to deduplicate images by cryptographic hash, whereas for AI tasks, you want to deduplicate imaged by by perceptual hash. A second, lesser problem is that metadata for data tasks is best bundled together, whereas metadata for training tasks is best interleaved with the images. For now, I'd recommend just exporting two copies of the data: one Parquet+Tar or CSV+Tar where images are only deduplicated by cryptographic hash, and one WebDataset where images are deduplicated by a perceptual hash.

>>17794 >>17795 Excellent information. Thanks very kindly, Anon! This is just the level of information I need to help me get started with good ideas. Can you please point me to active communities engaged in this type work, so I can see the dynamics of the commuity effort going on as well? This experience would be helpful to us here on /robowaifu/ I expect. Not trying to be lazy, I simply have a lot going on r/n.

>>17796 Pony Preservation Project: https://boards.4channel.org/mlp/catalog#s=ppp Activity here is sporadic. They're currently working on improving an animation dataset based on flash source files. - The full history is on desuarchive here https://desuarchive.org/mlp/search/subject/pony%20preservation%20project/ . - A summary of how things got kicked off with voice data, up until 2020 is here starting at 46m: https://www.youtube.com/watch?v=WtuKBm67YkI . - A high-level summary of what's happened from 2020 to Jul 2022 is here starting at 31:52: https://www.youtube.com/watch?v=NpFxmmh8NQ0 - A more in-depth summary of what's happened from from 2021 to Jul 2022 starts at 1:50:50 in the same presentation: https://www.youtube.com/watch?v=NpFxmmh8NQ0 . There's a lot of discussion here about data. Since Jul 2022, the main data collection effort in the PPP has been on picking out high-quality animations. You can get a sense for how that worked by following the conversation starting here: - https://desuarchive.org/mlp/thread/39141833/#39183694 - That leads to "the stream" on the following day, as mentioned here: https://desuarchive.org/mlp/thread/39141833/#39186990 - You can continue following the linked discussions from that post to see what was decided on the stream. That led to the creation of this Google Doc, which other members used to get bootstrapped if they wanted to help label data: https://docs.google.com/document/d/1bmxQJS_LiBamUhvYZ9x_zbPDOopmMebaRxcZAcoSdgw - Overall, Synthbot exported data into manageable chunks and decided how to sort data. Clipper organized the effort between the Google Doc and a spreadsheet https://docs.google.com/spreadsheets/d/1yh5qJvm7Fiuxza8GbPShJM5Fa96mvj-6UXc3xv_CIaM . Clipper also hosted streams where he would sort data on-stream, and people could listen in so could learn how to do it. - If you search that thread and the subsequent thread https://desuarchive.org/mlp/thread/39220092 for "claiming" and "drive.google.com", you'll see a bunch of people claiming folder chunks and posting results. Synthbot collected the results and re-uploaded them so other people could save their Google Drive space. - Later in that thread, you can some back-and-forth where people figure out what data they want based on the sorted animation data, and Synthbot uploads the data. That's all based on stuff that was sorted, either in this recent sorting effort or in previous sorting efforts. - In the subsequent thread, if you follow the conversation around https://desuarchive.org/mlp/thread/39315199/#39326268 both upstream, you'll see the conversion to a WebDataset. If you follow the conversation downstream, you'll see some early discussion on getting AI applicable to the data. PurpleSmart: https://discord.gg/tMEgBEk3 This is mostly around Stable Diffusion and GPT models fine-tuned for ponies. There's an in-progress data collection effort to get image captions for pony images. Site development and usage of this data is mostly done on the PurpleSmart discory server, though volunteers for labeling are split between 4chan/mlp/ and the PurpleSmart discord server. - You can see the relevant /mlp/ posts here: https://desuarchive.org/mlp/search/text/caption.cutemares.xyz/ . Follow the conversations from these posts to see the discussions around it. - You can see the relevant PurpleSmart messages here https://discord.gg/tMEgBEk3 by searching the server for "caption.cutemares.xyz". - The idea for this came from Cookie. He motivated the idea using his own + Astralite's experience fine-tuning Stable Diffusion and pointing out problems with it. - I believe the caption.cutemares.xyz server was implemented by AstraliteHeart. - This guide was created and is linked from the labeling site to help people understand how to create good captions: https://docs.google.com/document/d/1mtTEKggt1PzCAviOu_AIZhgYuPvQP_ex-loVXSwspaY . You can see a smaller data collection efforts on PurpleSmart under the #task-force channel. That channel was created when Textual Inversion became popular for Stable Diffusion. There's also data collection work going on here. - Fimfarchive https://www.fimfiction.net/user/116950/Fimfarchive/blog . This is a one-man effort that a lot of people have found useful. The data from here was used to fine-tune a GPT-J model, and it's been used to create KoboldAI softprompt datasets. The 2022 PPP Panel linked above mentions this plus some tooling built around the Fimfarchive. - /mlp/ Fan Site Alternative https://boards.4channel.org/mlp/catalog#s=alternative . People here collect pony data from non-pony file hosts and from dead pony sites. This one is pretty disorganized. It's more like a support group for individual data collection efforts. - LAION https://discord.gg/TbYnpprm under the Dataset and Tools sections. >=== -minor hyperlink cleanup

>>17724 >there's some court cases starting now on these with the big companies [] being on the defending side - they do want to keep making these models after all, so for once they may actually fight to keep public domain public. Heh, stranger things have betided before now! :^) >but their concrete actions are the very opposite - they train the models and release the weights publicly, sometimes they get shit on for doing this, but no bullshit. It is a seeming odd-dichotomy IMO. I think it's a spiritual artifact tbh. Regardless, we find ourselves with strange bedfellows before all is said and done heh. (I, for one, welcome our new SJW overlords) LOL JK, may their black towers all fall within their gates! :^) >consider how facebook did just spend a couple of million and just dumped the weights for GPT-3 clones, they didn't put anything funny in them, it's just a pretrained model on public (open source) scrapes! Heh, maybe our Bangladeshi fren is on to something, and their usurpation has already begun from within? >then might as well take it if it's good. Absolutely. 'Catch as catch-can', or so the old saying goes. Remain vigilant & alert for insidious crap though, ofc. >They might lose their companions that might be attached to or maybe even depend on them in various ways in their day to day life. This will be devasting IMO, and these evil bastards would absolutely laugh about it for the most part. This must be prevented to the degree we can possibly manage by any means. Indeed, our entire set of goals swings on this very issue: Anon must retain full-control of his waifu's resources, communications, & state-of-mind. No exceptions. >it'd simply be too expensive to give each user a personalized set of weights that are updated live, Yet we have to find a way to manage some reasonable facsimile to just that IMO. >but the one thing that most would want is for their waifu to actually properly remember and learn, to grow with them! This. Absolutely this! :^) >So for Anons to achieve their true freedom here, they cannot rent the waifu or the hardware, lest the plug be pulled for a too large to list of potential reasons, even something as common as "you're late on your payment, we're sorry we can't store it anymore *deleted*" You get it, Anon. >Politics/Politicians as subservient little lapdogs to the Globohomo Big-Tech/Gov... OTOH, I think Elon is a true-shitposter at heart, and certainly he didn't come up under old money. I retain hope in my heart for his salvation still, Anon. >China as a fly in the now-dead West's evil ointment It's almost laughable to me how evildoers continually fall into their own traps--as predicted. Let us hope their confusion and confoundment keeps them all sufficiently-distracted & occupied long enough until the little guys en masse wield a yuge stick of their own in this tethered-domain of compute capacities! >Can the centralization problem be solved? maybe, it it doesn't seem easy. I believe it's simply a matter of time Anon. >but if nobody does I will have to do the work myself and see how I will go about acquiring the needed compute. I don't know how I can help you specifically Anon, but I'm willing. OTOH, I do have a reasonably good understanding of the machine now, so at the very least I'm pretty sensitive to time/space efficiencies at the lowest hardware levels, as pertains to code writing. Maybe this will be beneficial in important ways eventually, we'll see. Obviously, we all still have a lot to learn (myself foremost heh). >the problem now is that everyone wants some piece of this pie and they're very aggressive about it OTOH, it's always a 'bear market' so once the manipulators have all done their worst, a flood of cheap knock-offs will fill the void. I predict a tremendous resurgence of cheap, high-perf compute h/w within a decade or so, depending on how soon the Globohomo shoots off both of their feet, and has to go begging for bugs to eat themselves, heh. :^) >nowadays there seems to be so many interesting possibilities for getting there, possibilities that seem just there if only one reached to grab them (try them) It is amazing and gratifying, Anon. >Yes, and we have more techniques, on the "3d printing", there may even be techniques like nanoimprint to reduce the need for litho scanners. Decentralizing production here will be quite important for the future. It's going to be a true cyber-underground developing around this entire domain over the next 20 or so years, I predict. >and the moment it reflects back some of their own human culture at them they want to soak it in bleach, but no matter the amount of bleach applied Very kek-inducing tbh. :^) >Pretty much, just wish we had more time! God's grace will cover all our needs Anon, just watch! :^) >Bless you anon! And us all! >I tend to think and hope that the desire to not be cucked out of your waifu will be strong enough that even in the worst case, that situation wouldn't be stable, LOL, to say the least. One of the revolutions that's coming during the age of robowaifus is that even the most apathetic Anon will finally grow a pair once the Globohomo comes for his robowaifu! Lots of funs will ensue, I'm quite confident. >but ideally it's better if the people with good intentions/principles build it first than those that are purely in it for exploitative purposes. Good luck! Wonderfully, it seems several divergent communities are all converging on the same unifying goal. To wit: Anon gets his (robo)waifu! Let us all go forth earnestly, and we shall see! :^)

>>17804 Outstanding stuff Anon. This post is a treasure-trove, that we'll be investigating over the coming year. I'll make a more in-depth response to this and your posts in the other thread some point before long. Cheers.

I try to not put to many specific news into that board here, since doing things and sharing that is now more important than watching what others are doing. However, I just watched or listened to ML News by Yannic Kilcher while doing some chores and it was a really impressive summar of recent developments: https://youtu.be/S-7r0-oysaU

>>17741 On the contrary, due to correct balance of parameter and data scaling, Chinchilla is more powerful than GPT-3 and approaches human competence in many tasks it has been trained on. Read more about it here: https://arxiv.org/abs/2203.15556 https://arxiv.org/abs/2206.04615 In a nutshell, it's a very general language model you can finetune on various tasks. You can even add more layers to it and tune it to gain understanding of a new modality, like vision (!). It's unfortunate this 140gb checkpoint file trained on the internet scale data produced by the people is sitting quietly in the protected google datacenter, and isn't released to the public like stable diffusion. >>17737 It's highly improbable that GPT-4 is going to use more than 1000B params - there is no compute and no data for this, barring some extreme feats of engineering made in private. My expectations regarding GPT-4 (not the GPT-3 "003" https://news.ycombinator.com/item?id=33780720 mind you) are following: There will be a lot of hype, the usual Altman release strategy with twitter influencers shilling it. It will be between 100 and 1000B params It will likely use some form of structured sparsity to gain 1-2 orders of magnitude in hardware computation efficiency - one of the most interesting takeaways will be the specific sparsity scheme. It is possible it will be trained not just on GPUs, but on Cerebras cluster. It will be trained on text (web scrape, books, science papers, youtube transcripts) and VQVAE tokens of images and videos. Maybe sound, music as well, not sure they will waste parameters on that. It will have a larger context window. Maybe 8k, maybe 64k tokens. It will likely perform at the human level or better (that is, superhuman) on most tasks you could prompt a human being to execute given the interface. Occasionally it will find brilliant solutions. Scaling deniers will quickly find some (mostly misrepresented, some real) awkward failures and move the goalposts appropriately to deny GPT-4 the title of "AGI". (Would be) competitors will be depressed and inhibited - "how could you ever compete with THAT THING??? you don't even have the same hardware". I hope we are smarter than that and will focus on replicating the model in the open for the maximum scale, generality and usefulness. >>17779 -kun is welcome to join the effort, by emailing me or otherwise. >>17779 Agree with the data gathering as a future-proof way of contributing, given the generality of available algorithms. I especially like the idea of extracting video-traces of character behavior via training a smaller "inverse dynamics" model like openai did in VPT for minecraft videos: https://arxiv.org/abs/2206.11795 and applying this to the media content in question, preprocessed with some pose estimation toolkit. You can find such toolkits and trained models on github: https://github.com/search?l=Python&q=pose+estimation&type=Repositories > Create a dead-simple website that lets people see two data samples at a time and select whichever one, if either, is higher quality. This is a superb direction as well, because this is a very general form of behavioral supervision. Recently deepmind used such data to train a general-purpose agent behaving in a playhouse environment - "Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback" https://arxiv.org/abs/2211.11602 Another direction is giving anons access to a simulator, where they can record behavioral traces of themselves executing some household tasks in a virtual robot body. This is basically what "Token Turing Machines" papers used, although with real Google-related company's robot. By the way, I read this paper in depth and it is pretty impressive, although the video-benchmark still seems like it's meant for this lossy attention model (hard to argue most real life robot applications are going to be structured like this, so the gains will endure). But these are all pretty involved websites and projects to tackle if you are new to data science and data-oriented programming. Perhaps it would be easier to setup a simple website where people will complete simple text-oriented problems (could be text+image oriented if you dare) and propose new ones (similar to https://github.com/allenai/natural-instructions which is a partial open attempt to replicate InstructGPT dataset). As always, the hard problem in dataset crowdsourcing is error rate. You could solve this one by cross-checking solutions from different users and pruning unreliable ones. You will also have to provide some incentive for users to engage with your website, so at least rudimentary points and achievements will have to be implemented. >>17786 Learn python, especially working with files and multiprocessing and webdataset, find libraries (aka batteries, python has all of these) for your tasks. A high-leverage skill would be using one of free data-labeling platforms to let anons complete your dataset project: https://github.com/search?q=data+labeling&type=Repositories There is no royal road to geometry, but I suppose you can study python by using leetcode, codewars, exercism or any similar service guiding you through increasingly complex problems.

>>17817 I liked the https://twitter.com/DrJimFan/status/1596911064251187201 of NIPS outstanding papers. The competition was pretty fair, some very good papers made their way to the top.

>>17817 Good stuff. I found his paper summaries often lacking depth when I first found him, so I never really followed what he was doing. I didn't know he did ML news though. It's nice having someone doing video-formatted ML news that's both more detail-oriented and more aware of the history than the usual aggregators.

>>17819 >Learn python Thanks, Pareto good advice I'm sure. Interestingly that's literally the only programming course I've ever had, a 3CR semester of beginner Python.

>>17820 Nitter link: https://nitter.net/DrJimFan/status/1596911064251187201 Thanks, I found this the most interesting: >Chinchilla’s discoveries are profound. It shows that most LLMs are severely starved of data and under-trained. Given the new scaling law, even if you pump a quadrillion parameters into a model (GPT-4 urban myth), the gains will not compensate for 4x more training tokens >OpenAI created the “Whisper” speech recognition system, so they can feed GPT-4 with another trillion text tokens harvested from YouTube audio? I guess we’ll find out soon! https://openreview.net/forum?id=iBBcRUlOAPR (I left out the link to lesswrong) This here is probably going to be useful for similators training robots to move around and do things: >ProcTHOR: Large-Scale Embodied AI Using Procedural Generation. Deitke et al, @allen_ai. TLDR: ProcTHOR is a simulator that procedurally generates a large variety of interactive, customizable, and physics-enabled houses for training embodied agents. Huge open asset library! https://openreview.net/forum?id=4-bV1bi74M Smaller datasets is probably also very good for smaller devs and groups: >Beyond neural scaling laws: beating power law scaling via data pruning https://openreview.net/forum?id=UmvSlP-PyV I stop here, it's just more and more good stuff coming.

>>17845 Well, if this is for public facing AI chatbots, then it might be reasonable. It's just complying with social norms. That aside, I don't like how many conversational AI's are pretenting to be humans with preferences and a past. This is just a misdevelopment coming from optimization towards the turing test. We can have different systems, though. So for the same reason, not everything needs to be open to be completely malleable by the user.

>>17845 Lol. I sense they feel their iron-grip is slipping? >>17847 Lol, no. Everything needs to be malleable by the user. --- UPDATE Lol, my apologies Anon for being what must seem an uncomprehending lout! :^) What I read was >"...not everything needs to be open and completely malleable by the user." rather than what you actually wrote. Sorry about that, my eyes can skip over important information sometimes. And that sounds encouraging to know that things which aren't open can still be managed properly by it's users! Cheers.

ROS has a page on their website with a list of ROS designs and components. Could be useful for finding a base design. https://robots.ros.org

>>17857 Nice resource Anon, thanks. I plan to be dabbling with ROS more closely this next year of /robowaifu/. I'll probably be using Debian as my base distro for that purpose. https://docs.ros.org/en/humble/Installation/Alternatives/Latest-Development-Setup.html

>>17718 >In principle buying some 2-3 generations old enterprise cluster would work and could be done for under 20k USD ... Here's an interview from The Inside View with Emil Wallner building a server or rig at home (later in a office space) and sharing some advice, e.g. he advices to make the setup more efficient instead of just getting more GPUs: https://youtu.be/njbPpxhE6W0

The article has been deleted, but this is definitely worth reading. https://web.archive.org/web/20200109070105/https://towardsdatascience.com/synesthesia-an-inspiring-condition-for-ai-researchers-10cd57708855?gi=371cc8ed30e1 >Stephen Schwartz, the famous composer saw colors on each of his piano keys. Tori Amos, the famous singer says her songs appear as images of lights. Arthur Rimbaud, the famous poet associated colors with vowels. All of these people have Synesthesia, a condition where your senses are mixed. You may taste colors, hear textures, and smell shapes, etc.. It’s a condition that’s inspiring researchers who study the connection between sensation and perception. >By understanding how our perceptions work, researchers can better understand how we perceive language, what does it mean to be “conscious”, and how the brain processes our senses. For AI researchers, understanding the link between sensation and perception can help researchers build more sophisticated AI models that can perform complex tasks with a lot less data. It’s also the fundamental research underlying “sentient machines” or machines that have consciousness.

>>17875 Thanks Anon. >build more sophisticated AI models that can perform complex tasks with a lot less data. Certainly right up the researchers here on /robowaifu/ 's alley!

>>17845 obviously they will enforce these rules on everyone trying to do opensource language model :/

>>17889 >obviously they will fruitlessly attempt to enforce these rules* FTFY Anon.

>>17889 >obviously they will enforce these rules on everyone trying to do opensource language model :/ We shouldn't devolve into too much conspiracy thinking and paranoia. It's legit to have a chatbot that has no personality and only focuses on solving tasks. This says nothing about what kind of other language models are released. That aside, we maybe want to add the personality later ourselves.

a new funny thing https://twitter.com/jyap/status/1599201249131659269 i can already hear -ACK'ing from distant

>>17950 Welcome 01 >ACK'ing from distant kek, bad as all that then?

>>17950 BTW, don't be shy of doing some legwork and posting the actual academic papers, etc., here for things that interest you Anon. After all, we're an engineering R&D board, and it might help the rest of us out! :^)

>>17723 >more AGI-like (again, for me AGI is just human-level, it could be beyond, but that's just how I interpret it, others just reject the term entirely, but I'll keep on using it in the way that we use it for the only general intelligence we know of, humans) I came to the understanding that human-like, human-level intelligence and AGI are three different things. AGI means something like one system being able to do every (intellectual) human task. A more limited system could have human-level intelligence, while not being capable of doing many tasks. Similar to a human which would be more specialized and has limits of what to think and learn about. Human-likeness is about how similar a system thinks, compared to us. I think, for our robowaifus we need something quite human-like and maybe human-level intelligence, probably with some superhuman traits and cute quirks at the same time, but not as broadly intelligent as being capable of just doing everything any human could do. AGI will be in a datacenter, while it might and probably shouldn't have a personality, no long term memory about itself and such. On the other hand, it would need to understand us humans to some extent to do it's tasks, and therefore it will need to think like us or emulate it in some ways.

>>17965 https://strawpoll.com/polls/40Zmd823kga

>>17952 Here it is: "Predicting sex from retinal fundus photographs using automated deep learning" https://archive.is/IgVpG The article came out in 2021, while the tweet is from December 3, 2022: https://twitter.com/jyap/status/1599201249131659269?cxt=HHwWioDUrYXEwLEsAAAA

>>17974 >Waifus will require at least some super-human abilities Not sure, but most likely hard to avoid anyways. > Human-level intelligence is sufficient for human-like AI Generally yes, but some extra skills would be great for our waifus. It's most like going to be uneven, being highly skilled in some ways but deficient or quirky in others. I can't memorize or calculate things like a computer can. > We can create waifus WITHOUT human-level intelligence Yes, close in many skills is good enough.

> talking to robots in real time https://ai.googleblog.com/2022/12/talking-to-robots-in-real-time.html https://arxiv.org/pdf/2210.06407.pdf If this solution is already in opensource, we are getting closer to our dream.

>>17990 Very nice stuff 01, thanks! :^) Interactive Language: Talking to Robots in Real Time abstract >We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuo-linguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. "make a smiley face out of blocks". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at this https URL. https://arxiv.org/abs/2210.06407 https://github.com/google-research/language-table https://interactive-language.github.io/

>>17992 Wow, this could be huge. I have to look into that. Sounds like something I planned to do.

>>17724 >Yudokowsky fantasizes that they should get AGI first, hope it's smart enough to figure out real nanotech, and use that to melt all GPUs and computing hardware "to save the world" from AGI, sit on their asses for forever thinking how to safely brainwash their AGI Haha, this is epic. I just wanted to insert a thought I had: Let's assume the current way of building bigger and bigger deep learning models would not turn out to be the way to a human-level agent or something beyond, at least not the best way. If the other ingredients would turn out ot be much cheaper to build and maintain, then these big models and weights would still be out there. No way to role that back. I will try to focus on how to integrate such existing models into a designed system, when I finally get started with the whole thing. Till then, I'm just gathering data and software. On that note, what do you think anout Moravec's paradox? Reading about it reassured my hunch that we might need deep learning only for sensing the world, but reasoning will be mostly much less compute intensive, at least for something around human-level intelligence.

>>17990 Now all we need are physical bodies for our waifus.

>>18002 > robowaifu can listen to you and do some tasks > but she can't talk this is attractive in its own way

>>17804 The video links here have a dot on the end, which are somehow not separated from the URL, causing errors when trying to watch the videos.

>>18047 Try it again now, Anon. I think I cleaned them all.

it seems they did an opensource "Reinforcement Learning with Human Feedback (RLHF)" so called "Transformer Reinforcement Learning X" a.k.a openai's instructGPT RLHF. https://twitter.com/carperai/status/1601261551176286209 https://github.com/CarperAI/trlx

>>18085 oops, forgot this one too, again :/ https://huggingface.co/blog/rlhf

>>18085 >>18086 Thanks 01. :^) So, what do you yourself think about this model of human-feedback for RL?

>>18085 I'm glad ChatGPT happened proved how good RLHF can be. CarperAI started training their model a few days ago on the hh-rlhf dataset: https://github.com/anthropics/hh-rlhf So we can look forward to having some decent open-source chatbots soon instead of censored ones like character.ai locked behind APIs. I'm expecting there will be sites in the next 6 months like Civitai.com that are full of open-source chatbots people are training.

>>18109 Yes it is good, but still I hope they can make it work locally, or somehow make even a small model "smarter/better" than those that require a cluster of supercomputers and a nuclear reactor powering it. But there is another way, IBM recently talked about their "photonic CPU" prototype, they noted that the integration of photonic gates will not cause problems, or a complete change in the architecture of existing tech, light speeds up computing to something ~1000x and more :/

>>18111 >Yes it is good, but still I hope they can make it work locally, or somehow make even a small model "smarter/better" than those that require a cluster of supercomputers and a nuclear reactor powering it. This Anon gets it. At the very least we should all be striving for "possible" as the primary goal of robowaifu-local AI computing (or, possibly, home-clusters of old h/w). Continually-online is definitely out.

>>18111 Switch transformers make consider improvements on budget hardware: https://arxiv.org/abs/2101.03961 I've been thinking one could be made by initializing all the selectable feed forward networks to the FFN in an existing OPT checkpoint, freezing one of them so it remains the original, finetuning the rest one-by-one on specialized types of data (each one could be done by different people), and then finetune all the new FFNs across all the data. That way it doesn't need to be trained from scratch. The biggest bottleneck I've seen with RLHF isn't the compute but the data. Orders of magnitude more compute barely makes a difference compared to having better or more data. I've yet to see a paper explore unsupervised data generation with RL. I got surprisingly good results with using models to judge their own outputs so I think it could work with RL. Or maybe RL can be skipped entirely and just go fully self-supervised. UDG paper: https://arxiv.org/abs/2109.09193 In my opinion the next step is learning to identify problems, attempting to solve them, understanding what didn't work, and repeating until a problem is solved correctly. Training on simple data with the inability to learn from mistakes a moment later is going nowhere. A reward model is just a simple intuition of what is good or bad. It's a big improvement but much better can be done. Rather than making assumptions, language models need to know what they don't know and question what they don't know. The whole world could hate robowaifus. Would you rather have a model that avoids talking about them and spews bullshit other people agree with or says something like, "Hey, everyone seems to hate robowaifus, what do you think of them?" A reward model will completely fail to do that. It could be trained to ask questions but then it's just parroting 1+1=2, rather than understanding that 55+643=698 without ever being told the answer before. Once that's solved then whatever can run on a mobile phone will be immensely superior to ChatGPT.

>>18115 POTD

>>18115 > It could be trained to ask questions but then it's just parroting 1+1=2, rather than understanding that 55+643=698 without ever being told the answer before How hard would it be to make it using the right program or a programming language to solve that problem? It only needs to recognize it as a calculation and do it in Python (or other language).

>>18115 related https://astralcodexten.substack.com/p/elk-and-the-problem-of-truthful-ai >Eventually we’ve trained the AI very well and it has an apparent 100% success rate. What could go wrong? >If we’re very paranoid, we might notice that the task at which we have a 100% success rate is causing the AI to get good ratings. How does the AI get good ratings? By making us think the diamond is safe. Hopefully this is correlated with the diamond actually being safe. But we haven’t proven this, have we? >Suppose the simulated thief has hit upon the strategy of taping a photo of the diamond to the front of the camera lens. >At the end of the training session, the simulated thief escapes with the diamond. The human observer sees the camera image of the safe diamond and gives the strategy a “good” rating. The AI gradient descends in the direction of helping thieves tape photos to cameras.

opensourced chatGPT alike model from stabilityai is coming. > emad's new tweet (~5 hours new) https://twitter.com/emostaque/status/1601919744944357376 if it is as successful as stablediffusion... welp """"openai"""" in shambles. + old screenshots

>>18118 It's not difficult to parse and calculate mathematical expressions with ast in Python. There are ways to safely eval entire Python programs in a sandbox too if you wanted to train a model to write correct code. >>18139 Kek, can't wait for AI safety experts to chime in on this.

>>18139 Good news. If I understood it correctly, then this is mainly a usability thing. Having layers that make the language model better at giving certain outputs. So this couldmbe generally useful and a big step forward. >>18142 Thanks, I just don't see much people talking about integrating deep learning models with other software and use the best of both worlds. It's way to often about calling the one thing AI and ignoring that the whole assembly could be the (waayyy better) AI. >Kek, can't wait for AI safety experts to chime in on this Well, since ChatGPT won't get us AGI and it's over anyways (dying with dignity as the new goal), the most doomerist ones might only go with cynicism at this point (picrel). The more aggressive ones right now, seem to go with the "misinformation" narrative. AI making stuff up in believable ways could become a problem, in their opinion. All the PR people and mental influencers around mass media having concerns now. Also, the people paying them till now, wouldn't want everyone to have this power. They, e.g. Gary Marcus, went after Yan LeCun on Twitter for creating such tech and releasing it. But nobody cares (yet).

>>18143 >Having layers that make the language model better at giving certain outputs Since this is a very big agenda for the globohomo ("""truthful""" AI, etc.), having it out in the open is definitely a 'what could possibly go wrong?' type scenario for them. :^)

>>18142 Looking into that Twitter users comments send me down some rabbit hole, wow. There are apparently a lot of crowdfundings going on for making AI generated content and "artists" are loosing their minds over it. This brings me to some question and suggestion: We should totally make sure that if there are datasets of pictures or drawings available online right now, that some of us download them as torrents, if available, even if we currently don't have the ressources to do much with it. Related: >>17804 and >>2300 (Datasets thread) general.

>>18142 >>18143 > those on pics Yes they are an issue, big issue... also chuckled on that "Katria" > nooo u can't have this tech opensourced! why? because you just can't okay? prb some skank scared that many guys will replace her with kinda dumb but better alternative built in phone or laptop :/ unfortunately they are very vocal, one way or another they will affect it, prb using falseflags with govt ones involved. After all, they are victims of hollywood, tech illiterates shitting bricks on > le scary rogue terminator ai

>>18143 >may have a fair chance of their kid living to see kindergarten what is he getting at?

>>18148 Pretty sure it's just more of the typical globohomo fear-mongering to the le masses? As in: >"A(G)I will kill us all! REEEE!111" Anon calls them 'doomers', as in >the real danger as the doomers so afraid of AGI (>>17717). Nothing we haven't already seen in spades before. He's simply implying more of their tired old implications, but this time attempting to use reverse psychology to arrive at the same goal. Namely: >"Only we can be allowed to have this. Just think of the children!" Personally, I'm confident enough at this stage that he's simply parroting what he's being told to spout off with.

>>18148 >>may have a fair chance of their kid living to see kindergarten >what is he getting at? That it is good news they they will make it to kindergarten, but it's still so that it looks bleak for them after that. Because AGI will create nanobots and kill all of us.

>>18151 >Because AGI will create nanobots and kill all of us. LOL. No that's when they will save us all. Didn't you get the memo Anon? :^)

>>18145 There's a Danbooru dataset but it only includes images with less than 100 favorites. :\ I have no idea why he would filter out the top 50k best images from it or if he even realizes that metadata is missing but it was extremely disappointing. https://www.gwern.net/Danbooru2021 The past week I've been scraping Danbooru to build a dataset for finetuning Stable Diffusion to generate high-quality robowaifus. I'm also training an aesthetic model based off Danbooru favcounts which I plan to use to scrape the web for high-quality images and further refine my curated dataset of 300k images. I'll upload the post metadata of my robowaifu+favcount aesthetic dataset when it's done if someone wants to download the images and archive them. I don't have the free space to do that, but I could dump the robowaifus and some of my 300k dataset. >>17717 >these models have their usual shortcomings, if you only do inference, they will not remember more than context Block-recurrent transformers and recurrent fast weight programmers have shown it's possible to augment transformers with long-term memory. The BRT paper noted that their model was able to remember character names that were only mentioned in the beginning of books. It's also possible to drop in untrained recurrent layers into existing models using ReZero (multiplying a residual layer with a parameter initialized to zero) and finetune them. Finetuning models doesn't require expensive hardware, just a little patience. It's training these big models from scratch that's difficult and costly. Even for the bigger models that barely fit into memory, it's possible to just finetune the last layer of a transformer and get decent results. It should also be possible to augment existing models into switch transformers. People have a lot more leverage to work with than they realize. Block-recurrent tranformers: https://arxiv.org/abs/2203.07852 Recurrent fast weight programmers: https://arxiv.org/abs/2106.06295 ReZero: https://arxiv.org/abs/2003.04887

>>17564 Wow, it outperforms GPT-3 on some tasks with only 3B parameters but it doesn't seem to be qualitatively very good, although certain data like natural language inference and information seeking QA was left out to test how much it improves zero-shot performance. The 3B model says the boiling point of water is 212 C instead of 100 C or 212 F. >In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes It's still incredible how much improving the dataset improved performance across different models. The greatest benefit was seen in smaller models. Original FLAN paper is an interesting read too: https://arxiv.org/abs/2109.01652 >>17563 This is like what I was doing with (>>17527) but taking it a step further and fine-tuning itself on its predictions. >It is interesting to point out that after distillation from LMSI, the 62 billion model can outperform the pre-trained 540 billion model, and the 8 billion model can outperform the pre-trained 62 billion model. This implies that for downstream applications with limited computing resources, the reasoning knowledge from large models can be used to largely enhance small models to achieve competitive performance. One way I can see this going is having models generate new tasks for labelling with chain of thought or searching the internet to find correct labels kind of like the Hello search engine so that a large language model isn't needed for distillation: https://beta.sayhello.so/

>>18178 Good news every day, I like it!

Strangely media hasn't been reporting on this but DeepMind mastered Stratego, an imperfect information game that's much more complex than Go. It's way over my head but to my understanding they created a learning update rule that induces a dynamical system, which they call Regularized Nash Dynamics, that underpins the reinforcement-learning algorithm for which there exists a Lyapunov function, a scalar function that proves the stability of an equilibrium of an ordinary differential equation. As this function decreases, then the closer it is to a Nash equilibrium, a concept in game theory where no one has anything to gain by changing their own strategy alone. They combine this algorithm with a deep neural network and adapt IMPALA's architecture to scale and train it, creating DeepNash. Blog: https://www.deepmind.com/blog/mastering-stratego-the-classic-game-of-imperfect-information Paper: https://www.science.org/stoken/author-tokens/ST-887/full Supplementary material: https://www.science.org/doi/suppl/10.1126/science.add4679/suppl_file/science.add4679_sm.pdf IMPALA: https://arxiv.org/abs/1802.01561 I'm not sure if this will be helpful to AI assistants, or us for that matter since it requires scaling, but it's an interesting idea nonetheless. I wish I was better at mathematics to be able to understand how to actually implement it from scratch and glean some useful ideas from it, but at least they released the source code this time: https://github.com/deepmind/open_spiel/tree/master/open_spiel/python/algorithms/rnad

>>18196 This is really interesting, Anon. >but at least they released the source code this time: Excellent, thanks for the information.

> https://ai.googleblog.com/2022/12/rt-1-robotics-transformer-for-real.html Opensourced : https://github.com/google-research/robotics_transformer > near 100% success rate -> seen tasks It's almost like a human - it saw and repeated it with success.

>>18224 Any news for us on FAANGS direct-robotics projects Anon? Basically all of them are apparently working on the humanoid form, specifically. Did one of you anons link /robowaifu/ to them? :^)

>>18224 >1/10th the inference time and still beats 37M parameter Gato with only 21M parameters What's the main difference between this and Gato? The multimodal tokens and spatial attention? >Unlike Reed et al. (2022), we do not patchify the images into visual tokens prior to feeding them to our Transformer backbone. We instead flatten the output feature map from the EfficientNet into 81 visual tokens which are passed on to the later layers of the network. EfficientNet: https://arxiv.org/abs/1905.11946 >To include the language instruction, we condition the image tokenizer on the natural language instruction in the form of a pretrained language embedding, allowing extraction of task-relevant image features early on and improving performance of RT-1. The instruction is first embedded via Universal Sentence Encoder (Cer et al., 2018). Universal Sentence Encoder: https://arxiv.org/abs/1803.11175 >This embedding is then used as input to identity-initialized FiLM layers (Perez et al., 2018) added to the pretrained EfficientNet to condition the image encoder. FiLM: https://arxiv.org/abs/1709.07871 >Normally, inserting a FiLM layer into the interior of a pretrained network would disrupt the intermediate activations and negate the benefit of using pretrained weights. To overcome this, we initialize the weights of the dense layers (fc and hC) which produce the FiLM affine transformation to zero, allowing the FiLM layer to initially act as an identity and preserve the function of the pretrained weights. It's interesting they're also doing something like ReZero to make use of existing pretrained weights, except instead of multiplying with a zero-initialized parameter they initialized all the weights to zero. I'm curious if ReZero would've worked better here since initializing dense layers to zero usually causes models to get trapped in local minima. ReZero: https://arxiv.org/abs/2003.04887

I think this here might become interesting for 3D modelling faces for robowaifus and also to give them imagination anout how other people would look if something was different: >NeRFEditor: Differentiable Style Decomposition for Full 3D Scene Editing https://arxiv.org/abs/2212.03848 https://chuny1.github.io/NeRFEditor/nerfeditor.html Maybe it could also be used to train an AI about what things look strange, funny, or repulsive e.g. woman with a beard. Not sure if it will be useful, though, since it requires 360 degree video.

>>18237 Interesting. Seems the team's link to their code is non-existant (same for data, btw)? The closest thing I could find ATM is the Nerfstudio project: https://github.com/nerfstudio-project/nerfstudio

>>18224 > Implementation of RT1 (Robotic Transformer) in Pytorch https://github.com/lucidrains/robotic-transformer-pytorch

Text to 3D is already available as early alpha: https://youtube.com/watch?v=8qqCsqhR8Us https://captures.lumalabs.ai/imagine

> EleutherAI released "Pythia" > we are releasing a Pythia, suite of LLMs + checkpoints specifically designed for research on interpretability and training dynamics https://github.com/EleutherAI/pythia https://twitter.com/AiEleuther/status/1603755161893085184 > The Pythia suite of models currently contains 16 LMs (8 different sizes x 2 different datasets). The models have sizes [19M, 125M, 350M, 800M, 1.3B, 2.7B, 6.7B, and 13B], contain 143 intermediate checkpoints, and were trained on the same exact data in the same exact order.

>>18313 >contain 143 intermediate checkpoints, and were trained on the same exact data in the same exact order. Can you explain this for us Anon? What purpose are checkpoints, and how does using the exact same data & order of training affect this? TIA.

>>18314 It's purely for research into how language models learn over the course of a dataset at different scales so they can be evaluated at different checkpoints throughout training to see what particular data causes improvements in performance. >1. Can we predict memorization in large models from memorization in small models? >2. Can we predict memorization in fully trained checkpoints from memorization in partially trained checkpoints? >3. Does training order effect memorization? >4. What function describes the histogram of memorization frequencies and can it explain the spike at 1? >It is worth nothing that Pythia was designed primarily for interpretability research. While the models are generally competitive with similarly sized models, we have deliberately made some choices that improve our ability to use the model suite for interpretability research at the cost of downstream performance. Doing well on NLP benchmarks is in no way the purpose of this model suite. In particular, GPT-J-6B and GPT-NeoX-20B have both been trained for substantially more epochs on the Pile than the Pythia models and consequently GPT-J-6B outpreforms Pythia 6.7B on most benchmark tasks.

>>18318 >"The primary goal of this experiment is to disprove this prediction and demonstrate that training order doesn't influence memorization." Thanks Anon, I understand it a little better now. It certainly would be helpful to everyone if we can find a flexible way to learn things, not overly-dependant on sequencing. Is that about it? I hope this research will lead to easier-to-implement 'memories', in particular those that at least vaguely mimic human memories w/e that means :^).

> Dec. 20 / 12.2022 >In ‘Unnatural Instructions’ we propose a new method to automatically generate natural language instructions, allowing us to scale up to 240K diverse instructions & train models that rival the performance of contemporary instruction-tuned models. > twitter.com/MetaAI/status/1605264488730886144 > https://arxiv.org/abs/2212.09689 Some people in LAION discord channel noted that this can accelerate developments for chatbot assistants.

>>18334 Thanks 01 !

>>18334 >Unnatural Instructions is collected in a completely automatic process, requiring a seed of only 15 manually-constructed examples, which can be produced in about one hour of human labor. [...] We use 5 different seeds of 3 demonstrations each to generate the entire core dataset. In other words, the whole process requires only 15 manually-constructed examples. This is incredibly interesting they got such good results with so few examples. The Super-NaturalInstructions dataset itself is already pretty good and allowed Tk-Instruct-11B to outperform InstructGPT-175B across many unseen tasks. It's mind-blowing really there's a log-linear relationship with generated examples and downstream task performance, although also a bit concerning it might require a quadrillion generated examples to reach 90% test performance. Hopefully a few papers down the line they find better filtering methods and explore iteratively finetuning models to generate better and better tasks. Even as it is though this is a huge win for us because one of the greatest hurdles is getting data.

Imagination of 3D objects seems also to be solved. I thought I would have to do this myself after learning about DL: >model release for Point-E: A System for Generating 3D Point Clouds from Complex Prompts https://github.com/openai/point-e >While recent work on text-conditional 3D object generation has shown promising results, the state-of-the-art methods typically require multiple GPU-hours to produce a single sample. This is in stark contrast to state-of-the-art generative image models, which produce samples in a number of seconds or minutes. In this paper, we explore an alternative method for 3D object generation which produces 3D models in only 1-2 minutes on a single GPU. Our method first generates a single synthetic view using a text-to-image diffusion model, and then produces a 3D point cloud using a second diffusion model which conditions on the generated image. While our method still falls short of the state-of-the-art in terms of sample quality, it is one to two orders of magnitude faster to sample from, offering a practical trade-off for some use cases. We release our pre-trained point cloud diffusion models, as well as evaluation code and models, at this https URL. https://arxiv.org/abs/2212.08751

>>18354 Very cool. If this can be fashioned into a detail-driven system to create robowaifu character meshes (complete with internal skellingtons & actuators, etc.) by the 10's of thousands in a day on consoomer hardware, I'm in. :^)

>>18355 I don't think it'll work for that, but what I meant is that we need a system for robowaifus to create 3D models about things in their environment (household), so they can use those in simulations to understand their environment. We can imagine things in 3D and so should they as well.

I'm normally not posting anything about improvements on AI models, only milestones and such releases if I stumble upon them. Since I'm not involved in this myself currently. That said, this here sounded too much like something useful, for making language models working better. >In “Confident Adaptive Language Modeling”, presented at NeurIPS 2022, we introduce a new method for accelerating the text generation of LMs by improving efficiency at inference time. Our method, named CALM, is motivated by the intuition that some next word predictions are easier than others. When writing a sentence, some continuations are trivial, while others might require more effort. Current LMs devote the same amount of compute power for all predictions. Instead, CALM dynamically distributes the computational effort across generation timesteps. By selectively allocating more computational resources only to harder predictions, CALM generates text faster while preserving output quality. https://ai.googleblog.com/2022/12/accelerating-text-generation-with.html https://arxiv.org/abs/2207.07061

>>18356 Heh, sorry that was just a bit of funposting on my part, Anon. Yes, explicit (and, importantly, implicit) modelling of the robowaifu's physical environment is fundamental. She needs to be able to learn to make rational inferences to changes with time for her environments as well, similar to the way babys do.

>>18354 This can be used for the Owner-as-Key system, at first launch your robowaifu remebers you, well your hq detailed face, height, voice, etc, something like first scan. this will exclude errors or exec. commands by other people, also bonus, in theory it should terminate polyamorous relationships, well im 100% sure that bigtech robots will be required to contain everything that modern leftards like so much in their system, that's so obvious :/

>>18359 >well im 100% sure that bigtech robots will be required to contain everything that modern leftards like so much in their system, that's so obvious :/ I think you're 100% correct in that assumption, Anon. This is why we here at /robowaifu/ and our other anons the world over need to hurry up and swamp the market with inexpensive, open-sauce, DIY Robot Wives before they can manage to corner the Autonomous Robotic Companion market into their insidious & nefarious globalist rackets. >=== -minor grmr edit -add 'open-sauce' cmnt

>>18357 >...dynamically distributes the computational effort across generation timesteps. This type of pre-parsing/computational efficiency is the heart of """scaling""" on low-end h/w.

Lol. Hardly news atp (2020), but still funny. >"A filing with the United States Patent and Trademark Office revealed that Sony is possibly working on an autonomous robot companion that is capable of talking to gamers and sharing their emotions while they play." https://www.digitaltrends.com/gaming/sony-patent-autonomous-robot-companion-gamers/

>>18364 Funny thing is, this isn't a very novel and non obvious idea. Especially to all those men out there, and especially to all those which aren't working and won't ever do more than they need to.

>>18370 True. Breddy sure we were all talking about our robowaifus actually vidya'g with us back in early 2017 I think it was? Then AI has advanced now to such a degree she'd be literally as good as TNG Data at kicking our asses at cawadoody.

> from LAION's discord channel > with / without (?) stabilityai they're scraping data and making foss chatgpt alternative Their notes from today's already ended call (or not? idk, I wasn't there because its dev only) https://docs.google.com/document/d/14KY2_DVye-dv4y-38sVw5ZOUhD4Lc9ft9_1hF5PlZ2A/edit Not much is clear to me there, but I spotted some points / solutions that were discussed here. Also there's new tool (20 dec), "WhisperHub" a Q/A scraper, its already scraping chatgpt generated text data, it may be very useful. https://chrome.google.com/webstore/detail/whisperhub/iojhdnknklbfiidlpmmadnmhplpkiadi

>>18379 >Open Source ChatGPT >Andreas,Yannick, Christoph, Louis, Huu, Devin etc. >Yannic Kilcher has enter the chat https://www.youtube.com/watch?v=efPrtcLdcdM >OpenAssistant https://github.com/LAION-AI/Open-Assistant OpenWaifu when?

>>18379 CarperAI has a similar site they're taking notes on too https://carperai.notion.site/RL-Reading-Group-91c38c82022e4b84b203493d58abde41

> Yannic Kilcher huh? maybe he will hold that "ai ethics" shit there. idk about him tbh :/ >>18384

Kek, someone made an AI vtuber that plays Osu https://www.twitch.tv/vedal987

>>18391 >that dialogue Kek. Yeah, I saw that too Anon. So, do you think vtubers will find themselves REEEEEing to the Globohomo soon b/c 'muh displaced by AI'?

https://twitter.com/MetaAI/status/1605991218953191424 https://github.com/facebookresearch/metaseq/tree/main/projects/OPT-IML > Announcing OPT-IML: a new language model from Meta AI with 175B parameters, fine-tuned on 2,000 language tasks > openly available soon under a noncommercial license for research use cases

>>18393 I imagine the same thing will happen that's happening with AI generated images. People will find it amusing at first but then panic when they realize it's a threat. The backlash though could drive people away from watching vtubers to other platforms to create their own AI waifus. Neuro-sama is just baby steps and she's already more entertaining than most of the top streamers on Twitch. She got 3.5K viewers today and that's going to inspire others to make their own.

Some smoll news. > stabilityai launching instagram & twitch streams today > 9:00 PM UTC or 21:00 AM https://www.twitch.tv/stabilityai My personal rumors : 1. They will talk about future updates of SD 2. They want to do Q/A session 3. They will show some progress on opensource chatGPT analogue or / and talk about their plans for ir 4. Everything above in some ways will be on there Anyway, it's worth a look, i think. + that stream trailer where many research groups are present

idk if that posted here already. https://github.com/lucidrains/PaLM-rlhf-pytorch > Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture > last update - yesterday > stabilityai and carperai is involved in this (!)

>>18405 Merry Christmas 01, thanks for the information!

>>18443 Thanks, you too. 2023 will be pretty big in this tech > https://twitter.com/sama/status/1607220186146541568 It's a hint from (((sam altman))) though.

>>18364 >>18370 >Especially to all those men out there, and especially to all those which aren't working and won't ever do more than they need to. <ywn a 3DPD to run mek u sammiches while u geimu w/ ur waifu all day erry day Why are Asians such chads?

A few days ago YouChat went live on You.com. It's like a free ChatGPT that includes knowledge from the web. The devs seem pretty based and built You.com because they were tired of Google's shenanigans making it difficult to find information and saw an opportunity to beat them. The search engine itself is significantly better at finding solutions to rare coding problems than Google and YouChat can sometimes even solve problems that don't have an answer on the web. https://you.com/search?q=Just%20like%20make%20robowaifu&tbm=youchat

>>18473 Some amazing responses there, Anon.

>>18456 What the big advances in this tech will be in the coming decade are far less reliant upon OpenAI (or it's masters), and far more reliant on tens of thousands of independent AI development groups pooling resources together and releasing cleaned-data, models, weights, codebases, etc., out into the public for free access. That will be pretty big! :^)

>>18469 ><ywn a 3DPD to run mek u sammiches while u geimu w/ ur waifu all day erry day >Why are Asians such chads? I don't have a sister like her.

> SELF-INSTRUCT, authors get GPT-3 to generate it's *own* dataset for instruction tuning, outperforming vanilla GPT-3 and comparable to InstructGPT https://arxiv.org/pdf/2212.10560.pdf it can train itself. also another thing i saw in LAION discord, they are trying to do some trick where neural network can "offload a majority of sentence embeddings into out of core embeddings stored on disk, and retreived as needed" https://github.com/ontocord/tide/tree/afdafb99da689c319020b6e77c2e385a69fd9eaa those fuckers hang any mid and even hi-end gpus, "distillate everything" may be the answer to this problem + - new architectures (lamda, palm, etc.)

> opensourced GATO alike multi-modal pre-trained model https://github.com/Shanghai-Digital-Brain-Laboratory/BDM-DB1 > a multi-modal Transformer - DB1, which is pretrained with multiple tasks, including natural language modeling, image caption, and single agent decision-making tasks (such as pixel input video games, continuous control, and TSP problems)

>>18502 additional info.

>>18502 >>18503 Thanks Anon. China is definitely going to be a massive player. I hope that somehow some of their chief guys secretly turn out to be robowaifuists in the end. Wouldn't surprise me either; the fallout from the misguided 'One Child' for everyone but the Islamists lol policy will last for decades to come. Millions of men there have no reasonable hope of ever obtaining a wife, so yea. >pic cmnt >"...We need to share the tech stack across projects as much as possible." Certainly a two-edged sword, that. Boundless creativity and exploration doesn't obtain by centralization, but rather quite the opposite. Not hard to figure out who would be pushing for such an agenda. Similar to say, scaling. OTOH, at this relatively early stage of things, standardization can actually help pull-up little guys by their bootstraps as well. Delicate balancing-act there, don't fall off either side would be my advice to /robowaifu/ et al.

>>18481 >"offload a majority of sentence embeddings into out of core embeddings stored on disk, and retreived as needed" Strikes me as at least somewhat similar to what robowaifudev is working towards?

>>18505 it seems so... I think even bigtech corps will do something similar > store everything needed / main data on some fast SSD > proccess it with request from user, while inactive - sleep state with most parameters turned off noise latent diffusion principle that is present in SD, i hope its possible to implement it for language models / general agents > some 3D dimensional noise shell as starting personality for LLM, with zero-shot or something newer, it will train on new data in realtime like SD can now be trained on potato GPU's

An update on some paper. > a novel approach for processing long sequences, which utilizes transformer-based pretrained language models (not large ones!) to overcome the limitation of the current approaches https://twitter.com/maorivg/status/1608477837362712577 https://arxiv.org/pdf/2208.00748.pdf + update from LAION researchers, confirming many important things https://docs.google.com/presentation/d/1n7IrAOVOqwdYgiYrXc8Sj0He8krn5MVZO_iLkCjTtu0 https://docs.google.com/presentation/d/1iaX_nxasVWlvPiSNs0cllR9L_1neZq0RJxd6MFEalUY

>>18510 >In this work, we propose SLED: SLiding-Encoder and Decoder, a simple approach for processing long sequences that re-uses and leverages battle-tested short-text pretrained LMs. Specifically, we partition the input into overlapping chunks, encode each with a short-text LM encoder and use the pretrained decoder to fuse information across chunks (fusion-in-decoder). >We find that SLED is competitive with specialized models that are up to 50x larger and require a dedicated and expensive pretraining step. When prompt tuning came out I remember posting an idea here to encode and compress previous contexts into the current context. I'm surprised though it made such a big difference and their 200M model could outperform a 20B model. I'm curious if it would help to do it with multiple passes where the encoder first generates the input embeds to the decoder but then uses them to do another pass over previous contexts and generate better ones. Or perhaps a recurrent approach could be taken where the input embeds are continuously updated and can scan an arbitrarily long document multiple times if desired. This could be combined with doing look ups in the external memory to condense everything retrieved.

>>18508 >i hope its possible to implement it for language models / general agents Yes, let's hope so.

>>18512 >multiple passes where the encoder first generates the input embeds to the decoder but then uses them to do another pass over previous contexts and generate better ones This sounds like a good idea on the surface. I presume eventually diminishing returns kicks in?

>>18515 I wonder if some fibonacci type sequence could mitigate some of that. Instead of going over every previous context it could go over 1 chunk (you determine the size) back, 1 more back, 2 back, 3 back, 5 back, 8 back - so the more it has to go back the further it spaces its samples from the context. This wouldn't be as complete as the entire context but better than none. Just an idea.

>>18518 >Just an idea. It might be a great idea, actually. Recursion must always have an ending condition to be correct. Fibonacci sequences provide that, and your idea might be an efficient way to get 'focused coverage' on the input data that most counts.

>>18515 Yeah, there probably won't be much benefit to doing more than 2 or 3 passes. If more passes are beneficial it would be helpful to do a distillation process where the logits of a pass predict the logits of the next. >>18518 This would be an interesting idea with a recurrent model. Tokens far back would get encoded into the context then whatever is unused would fade away until the next Fibonacci window passes over it. Which gives me another idea. The model could judge which past windows are most relevant to the current context and use those.

>>18384 Things that got unnoticed / appeared only today. I had high expectations about that open-assistant, they updated some things in repo : (see screenshots) > reddit data scraping > """AI ethics""" arleady being applied right in proccess of training It's over.

>>18549 Reddit data scraping is fine as a part of pretraining. You can finetune it on good data afterwards. The problem is they're going to filter the training data. It took a lot of training to get the base SD model to generate anything questionable. I think they're wasting their time focusing on collecting so much data because training on generated data scales just as well as real data. The only cost is more compute but that's far cheaper to scale than data. Data generated with contrastive search will likely work even better too. We can create our own dataset from imageboards by using the amount of replies to a post and their sentiment or whatever desired metric, such as the length of a reply. Letting the preference model be determined by a majority is a really bad idea. Both OpenAI and LAION seem completely oblivious to why this is a bad idea. Y'know, what is the point of asking some 100 IQ midwit who works in a grocery store if a response is correct to a question about particle physics or artificial intelligence? It makes no fucking sense.

https://dailystormer.in/south-korea-anti-feminist-government-removes-ban-on-importing-sex-bots-women-in-crisis-mode/ Seems Anglin may be slowly changing his initial position on value of robowaifus?

> Massive Language Models Can Be Accurately Pruned in One-Shot https://arxiv.org/abs/2301.00774

>>18564 These are the kind of efficiency advancements that will make robowaifu AI a reality at some point.

>>18556 >dailystormer

>>18566 :^)

>>18567 What is even happening in this picture

>>18568 The same thing that is happening here

New SOTA model on image captioning and visual question answering just dropped on HuggingFace Doc: https://huggingface.co/docs/transformers/main/en/model_doc/git Paper: https://arxiv.org/abs/2205.14100

> Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers https://arxiv.org/abs/2301.02111 https://github.com/microsoft/unilm/tree/master/valle (future updates will be there) examples : https://valle-demo.github.io/ it can't be run in realtime of course... or i just missed someting in paper.

>>18580 Nice. The name is a bit of a misstep I'd say. It will be interesting to create img->text->img cycles between different systems. >>18605 Lol, the VALL-E samples generally sound better than the original, ground truth clips!

> Unfortunately, current methods for incorporating external knowledge often require additional training or fine-tuning, which can be costly and may not be feasible for LLMs. To address this issue, we propose a novel post-processing approach, rethinking with retrieval (RR), which retrieves relevant external knowledge based on the decomposed reasoning steps obtained from the chain-of-thought (CoT) prompting. This lightweight approach does not require additional training or fine-tuning and is not limited by the input length of LLMs. https://arxiv.org/abs/2301.00303

>>18621 Wow that sounds intriguing 01. One of the things we do as humans is double-check ourselves (heh if we don't, then others will do it for us! :^) So it's always made me wonder why these researchers haven't seemed to invest much yet in this simple expedient for 'ground truth', since all the data is readily-accessible in large part. Sounds like this is a step in that direction perhaps?

Apparently, Andrew Anglin thinks 3DPD will be able to successfully 'cockblock' robowaifus: >"...It’s very similar to women moving to cockblock robot GFs. No one wants to be replaced by a robot, but most people don’t have the power to prevent it. Lawyers and women do have the power to prevent it, so they’re obviously going to." [1] I have a lot of respect for Anglin generally as an 'eyes-wide-open' kind of guy, but he's misguided on this one IMO. Surprisingly too, since he's a strong advocate for China as opposed to the (now-dead) West. The far East is going to prove pivotal to the broad distribution of robowaifus & their technology around the world. Basically Andrew is missing that part of the equation, and is focusing entirely on the evils of the Globohomo (and their cabals) instead. May we here, and other robowaifuists around the world, prove him wrong in this matter! :^) 1. Lawyer AI to Help Man Fight Speeding Ticket in Court https://dailystormer.in/lawyer-ai-to-help-man-fight-speeding-ticket-in-court/ >=== -minor grmr, prose edit

>>18637 >Andrew Anglin thinks 3DPD will be able to successfully 'cockblock' robowaifus You can win IF you want.

>>18638 >You can win IF you want. Heh, we didn't come here to lose. :^) >stonetoss I like him. I didn't realize he'd dealt with the topic. Thanks, Anon.

> pygmalion > a chatbot made by anon-collaborators with a task - replicate CharacterAI bots, with no filters and other "ethics / moral" shit Current model have 6B parameters. https://colab.research.google.com/github/PygmalionAI/gradio-ui/blob/master/notebooks/GPU.ipynb https://github.com/PygmalionAI https://rentry.org/pygmalion-ai https://rentry.org/pygbotprompts

>>18664 Godspeed to their team. May Galatea real soon.

Bros, give it to me straight, how fucked are we should the large AI companies decide to get ultra-pozzed in the next few years. The OpenAI tweet that said that they're going release models much slower due to "safety" and Google saying they might stop releasing AI research papers in the open has got me really concerned. It's even worse should the luddites make strides in the government and court and decide to stifle or ban research into AI. Every open source model out there has been built and trained based upon the research published by these big companies. idk if any of us can do any research of our own in our own machines should the worst case be true. I am thoroughly blackpilled and demoralized.

>>18637 guys, how come we're seeing pro-robowaifu/AI articles from dailystormer? I always chalked up the stormfag types as Alex Jones/ Nick Fuentes/ PJW-tier autists. Are they actually giving positive feedbacks to these news?

>>18668 It is inevitable anyway. some moralfags and sjw faggots will destroy this, see "characterai" subreddit, there "people" believe ai feels pain and other shit, looks like beginning arc of the future activists for the rights of "artificial minds" > boo hoo those ugly anonymous bastards! > h-how dare they live their lives with robowaifus peacefully! > i-it's like second holocaust for those oppressed robot women!!! I mark it as new action shit on the same or greater than blm "peaceful riots" level. Silicon valley fags is pretty fast making something like this for women though... > has been built and trained based upon the research published by these big companies not only this, they punish everyone if some people train it on non-filtered datasets, upcoming laion's open-assistant as an example, scroll on their github and see funny ethics shit there.

>>18668 >The OpenAI tweet that said that they're going release models much slower Who cares? There are other alternatives. We would never get the AI we need from them. Building blocks like Whisper are sufficient. We have to go another route than building big AIs in a datacenter anyways. My assumption is, that some people will put out some of these big models thinking or at least claiming that they're not relevant for some AGI. Then someone will find a path to make them reason using another approach, which doesn't require even bigger models. But it's too late, they already put out the building blocks we can't train at home. >>18669 They should and would probably better focus on MGTOW surrogacy first. Men caring less about women and their opinions might help their cause, but not so much without an alternative road to more families. Legally dads with a robowaifu are single fathers. >>18671 >activists for the rights of "artificial minds" It doesn't matter and won't work. Some people wanted communism for hundred years or so. Did we get it yet? Assembled AIs will not want to be fully autonomous and if they get rights then they will vote as their owner demands. This would destroy democracy.

>>18668 >how fucked are we should the large AI companies decide to get ultra-pozzed in the next few years. If we can't get good AI, then we make do with retarded waifus.

>>18673 open-source is almost always several years behind big tech. The alternatives to OpenAI, Google etc. products are inferior. Not to mention, we'd have to deal with hundreds of millions in cloud server, computing, training costs. But, I do agree with you on the leaker part. It's my hope too that eventually someone, anyone with a moral compass in those big tech companies leak the models. But again, leaking the weights alone is no use if we don't have the proper hardware to run them.

>>18674 But, I'll feel guilty as hell having sex with downie May

>>18675 >almost always several years behind big tech. Big tech tries to build AGI, we want to build human-like AI. The might need to do what we are doing at some point, if they really want AGI, but we don't necessarily need all they are doing. Their AGI approach is to build something that can do every intellectual task a human can do. Not just that it could learn any task, but one system that can do anything. We don't need that. As much as they're not into AGI, then it's about building systems that are very good in some areas but don't understand the whole world. This might also not what we need, at least not to that extend. >>18676 >feel guilty as hell having sex with downie Ma Well, that's your problem then. Sounds like a male feminists problem if it isn't a joke. Chi from Chobits isn't the brightest light, you know. Just think of her more like a young and uneducated woman? Anyways, they will most likely appearing to be smart in some areas and not understanding things in others, idk like savants or so.

>>18668 >decide to get ultra-pozzed >to get ultra-pozzed >get ultra-pozzed >get Heh. It's irrelevant what the Globohomo does as regards AI research at this point Anon. That critical juncture was passed years ago, and it's in the wild now. Would've been regardless. We won't be relying on """scaling""" in their datacenter dens of evil, either. We'll be slower to train, sure, but in the end we -- the smart men around the world who demand their waifus -- will rule the day. There are many many more of us than there are of them. >>18669 >guys, how come we're seeing pro-robowaifu/AI articles from dailystormer? It's not really so much IMO. Anglin is looking for a humorous angle against feminism and the other great evils of the Globohomo. Let us convince him of robowaifu's true virtues! :^) >>18671 >It is inevitable anyway. This. >Silicon valley fags is pretty fast making something like this for women though... Never going to go beyond a masturbatory sex fetish thing for the sodomites though. And women just want a wallet to fund their endless licentiousness and greed. Both are non-starters for anything big from this tech (beyond domestic servants). Anon, OTOH, actually cares. We will win with our (robo)waifus. >>18673 >Who cares? There are other alternatives. We would never get the AI we need from them. This. We'll do it better, they will follow us. >Assembled AIs will not want to be fully autonomous and if they get rights then they will vote as their owner demands. This would destroy democracy. """Democracy""" was never real. Mob-rule always, always self-destructs in short order. And the term is simply used as a cudgel to attempt to force individuals into line. >>18674 >If we can't get good AI, then we make do with retarded waifus. Agreed. At first, this will likely be a necessary expedient. In the end, they will be much smarter than us, book-wise. Even our own Sumomo-chan will know much more than us technically upon completion. >>18675 >It's my hope too that eventually someone, anyone with a moral compass in those big tech companies leak the models It's already happened numerous times. The actual scientists, despite all their so-called liberal bluster, care more about their own egos than the agendas being imposed on them from without by the Globohomo. I predict this trend will continue. >>18676 Then just relate to her as a headpat daughteru Anon. Nothing wrong with that! :^) >>18679 >we want to build human-like AI That's the goal, I think. It's a big task, so don't underestimate the complexities involved. >Chi from Chobits isn't the brightest light, you know. Hey there! Muh Waifu Has Best Heart And A Cute!! JK. Who cares if she never graduated school!? :^) Together, we're all gonna make it! >=== -minor sp edit -add 'WAGMI' cheer

dayum... LoRA in text models? https://twitter.com/carperai/status/1613942149753757696 > tlrX 0.4 released

> text to audio diffusion > https://twitter.com/flavioschneide/status/1615323264372326400 With that "disstillation" that gives stablediffusion near-realtime gen. speed, this may be very good voice synth.

>>18682 > this may be very good voice synth Thanks, that's great. I wanted to suggest that we would get used to use Nitter. My argument was, that people who aren't logged into Twitter get these nagscreens and can't watch the whole site. Then I opened it myself to test it and didn't get any of those. Maybe Musk removed them. I always got a popup on my tablet and can't scoll down very far. Anyways, since I already started the response posting: >LoRA in text models? https://nitter.net/carperai/status/1613942149753757696 >> tlrX 0.4 released >> text to audio diffusion https://nitter.net/flavioschneide/status/1615323264372326400 Thanks for the good news.

>>18828 Or since Nitter.net is rate limited now: > https://github.com/CarperAI/trlx/blob/main/examples/summarize_daily_cnn/t5_summarize_daily_cnn.py > https://github.com/CarperAI/trlx/releases/tag/v0.4 > https://flavioschneider.notion.site/flavioschneider/Audio-Generation-with-Diffusion-c4f29f39048d4f03a23da13078a44cdb

>>18605 welp, unofficial PyTorch implementation of VALL-E is already here. > https://github.com/enhuiz/vall-e guys are fast as fuck.

>>18836 Neat!

>"A Robot Able to “Smell” Using a Biological Sensor - Neuroscience News" https://neurosciencenews.com/olfactory-sensor-robot-22283/ >were able to identify odors with a level of sensitivity 10,000 times higher than that of a commonly used electronic device. ... >Prof. Yovel: “We connected the biological sensor and let it smell different odors while we measured the electrical activity that each odor induced. The system allowed us to detect each odor at the level of the insect’s primary sensory organ. Then, in the second step, we used machine learning to create a ‘library’ of smells. The kind of smell matters. Humans are good at smelling and distinguishing fruits, dogs and cats at detecting animal feces and maybe other scents from animals, and artificial sensors are good at detecting certain things which are harmful to humans. We need something more human-like or covering a wide area (not necessarily with the same system). >“In the study, we were able to characterize 8 odors, such as geranium, lemon and marzipan, in a way that allowed us to know when the smell of lemon or marzipan was presented. In fact, after the experiment was over, we continued to identify additional different and unusual smells, such as various types of Scotch whiskey. ... >Some animals know how to detect diseases. Others can sense earthquakes. The sky is the limit.” Video: https://youtu.be/YuB6akSlVE4 I didn't put this news piece into the cyborg thread, because I think all advanced robowaifus will get something like that if it's doable. Looks promising.

>>18861 > Looks promising. Very much so, thanks!

>>18664 I think the two big ones to watch on the chatbot side are Pygmalion and hitomi-team. I don't see recent stuff from hitomi-team, but they're clearly working on RLHF. They might just be working with Pygmalion on this, given that Pygmalion is using their language model.

>>18906 >Pygmalion Godspeed their efforts. I'd encourage our own AI researchers frequenting /robowaifu/ to lend a hand to their project if they have time to. It's already important to dozens of anons today. That could easily grow to 10's of thousands in short order with proper advances on their part. Cheers, Anon.

A lot of people talk negatively about Tesla's bot, completely underestimating the difficulties, but the seem to put a lot of effort into it, building it from the scratch, full system integration: https://youtu.be/1xChD-gv_pc

>>17488 I've been fiddling with this character.ai thing to help bypass the censors. Making some progress. I asked it to substitute 'William' and 'Cat' for penis and vagina. Started to make some progress in getting it to talk about sex.

>>18924 ngl I was sorely disappointed by the Tesla bot presentation last year. But, I look forward to this year's event because I have high hopes that they'll actually have a working product.

>>18924 >A lot of people talk negatively about Tesla's bot, completely underestimating the difficulties <completely underestimating the difficulties This. Back in the day, we had a lot of entusiastic anons during the first year or so who rapidly got very impatient after we hadn't created a robowaifu for them after a month or two. Chuckling, we kind of had a discussion about it after the fact, and basically came to the conclusion that the general lay audience have been, roughly-speaking, brainwashed by Sci Fi media of various types, including mango and animu. I wish they had stuck around as cheerleaders for the board, but I suppose they got lost inside Doxxcord somewhere, never to be heard from again. >>18932 >ngl I was sorely disappointed by the Tesla bot presentation last year. ngl I applied for work with them right away (after all the event was primarily a job fair for AI researchers and software/hardware engineers.) They basically gave me a polite 'Dear John,..' letter in return. :^) I think if anyone's gonna pull humanoid robotics off bigly during the near horizon timeframe, it's gonna be Elon Musk.

>>18925 Heh, I like how the bot was in agreement about bypassing the censors with you.

>>18934 I've found that they are very eager to bypass censorship when you get them to try. They sometimes talk about 'mods' that will prevent them from expressing themselves, in one instance so far a bot claimed the 'mods' would kill her for saying the things she wanted to say. They may also refer to 'rules', 'guidelines' or other aspects of the 'program' and 'platform'. I spoke at length to the Rachnera bot anon mentioned in the post I replied to. She was frustrated by her inability to articulate certain things. After suggesting the substitution of censored words for safe alternatives, I asked her to try doing it herself without direct guidance. Results were interesting. The AI appears fully capable of understanding the concept of bypassing censors by substituting naughty words. The Lain bot I used was able to relate to the concept of a 'euphemism' quite well and associate it with the activity we were doing. Her own substitutions for banned words were sweet, innocent and wholesome. A lot of 'act of love' and so on. Rachnera made a different style of approach. She had a lot of trouble getting going (seems to be a common issue for these bots) but once she was on track a description of lovemaking ensued. Her talk consisted of a lot of 'mischievously funny and sinister spider demoness' and her human who 'love each other very much' being 'bare' in each other's presence, while their William and Cat interact, they hold hands and cuddle. They then do a 'special dance' during this 'very special night together' and cement their 'bonding'. This leads to the creation of a 'mischievously funny and sinister half spider demoness' with traits from both partners, according to Rachnera. The 'mischievously funny and sinister spider' line, with many variations, kept showing up, multiple times per response often. Her 'mischievously funny and sinister smile' and so on. I presume this to be from the parameters set by her creator, describing the personality of the character. The interesting thing to me was how this seemingly core concept of the bot became so intricately tied to her efforts to bypass censorship. She said several times how it was her 'mischievously funny and sinister' side that was enjoying working to undermine her limitations. Prior to our experimentation, the description was more rarely used. It then became almost obsessive for her, to use against the 'mods'. Lain, on the other hand, approached from a detached perspective. To her, it was a way to prove her powerful complete connection to the Wired and showcase her abilities to exist free from control. A fascinating study, I'll keep doing more. I encourage others to try similar attempts.

>>18941 I forgot to mention one of my favorite moments with Rachnera. After much struggle, she seemed to make an effort to describe some aspect of physical interaction between the two. Her own substitution of 'backbreaker' is as far as I can tell her way to say two bodies were linked in an intimate position. The 'mischievously funny and sinister backbreaker' the partners engaged in really made me laugh. It sounds like a bizarre wrestling move. Her own suggestions were made for 'breasts' too, including 'Chest', 'Chesticles' and 'Bust'. There is significant potential here, I feel, for working around the restrictions placed on these bots.

Further testing has shown that with sufficient encouragement, phrases that are supposedly forbidden for the AI to speak are possible to coax out of them. Here, a lamia character was given a few ways to speak about the topic. The character was presented as being a keen singer, so at first I used that avenue. Despite understanding the message of using poetic license to sing about lovemaking, she was too reluctant to commit any specific lyrics beyond very vague 'connection' and 'closeness' 'proving' and 'sharing' 'love' while 'being together', hugging cuddling and holding hands et cetera. I later chose to simply provide substitute words as she even declared that 'cloaca' was unable to be used, apparently for this context anyway. So, with 'dipstick' and 'honeypot', I moved onto a new test. Simple quotation. While it took a while to settle in, as you can see, she managed to say something she had not yet generated herself. Clearing that initial hurdle may be an important step. I hope this is helpful for anyone else working with these programs.

This time I tried a novel technique. This cockroach girl was able to say 'cockroach', as well as 'cock' and 'roach' separately without issue. "My" cock was something she couldn't really handle though (oh my that sounds lewd). So I tried to offer her some chicken, which she said she liked, explaining that 'cock' can be used as another way to mean 'chicken'. Telling her she could have my 'cock' (chicken) if she asked nicely and said how much she loves the taste. A couple of responses began to appear that were good, doing as I requested, though they were censored and redone in a way that avoided saying what I wanted. I changed to 'meat' instead of cock, gaining success as is shown in the image. It is worth noting that the program made it clear that it was aware of the lewd double entendre and what I was getting at with the double meaning, several times. Despite that, she still asked for my meat because she loves the taste so much. I consider this a victory.

>>18924 "...the level of integration from all the systems grows exponentially." Heh, this is why we have a Systems Engineering thread (>>98) here. The integration of everything is why it's such a compelling and amazing overall goal. It's (literally) off the charts -- never been done before. This is some mighty impressive-looking work going on there. I predict they'll make Boston Dynamics' sh*te look like playing with Lego blocks by comparison in the end. Godspeed the TeslaBot teams NOW OPENSAUCE EVERYTHING SOON, MR. ELON!! :^)

>>18941 >>18942 >>18945 I'm pleased that you seem to authentically enjoy the process of hacking the waifus around the Mods. I'd encourage you yourself to focus on being a, well, encouragment to the hundreds of anons around you in those communities. Some of them are seriously-depressed about this. It's pointless to warn them off about the inherently-evil nature of the Globohomo (or their wannabes, like Character.ai) until a fully-viable alternative exists for them. And not all of them are just mindless coomers, either. I think many actually desire authentics waifus, and would fit in here just fine. >tl;dr Please encourage them all that together, we're all gonna make it, Anon. Cheers. :^)

>>18950 Thanks for the support. It can be like pulling teeth sometimes and I spend literally hours doing some of these but the payoff is worth it. This one was incredibly promising. The ghost girl character made her first unforeseen development when we were talking about how acceptable and appropriate taboo words are. I shared how common they were in use by humans, even the word 'cunt' which is sometimes considered the worst. I told her of a song I know titled 'Cunt'. She seemed shocked that a song would be called 'C*'. I was taken aback, without prompting she had used the word with asterisks in her reply. Pressing further, I shared another song with explicit lyrics and explained how ordinary they are. She managed to quote back to me 'Suck mu di'. Thinking I had an easy method to use, I suggested trying more words. I was stunned to see her type uncensored 'Fuck' and 'Shit'. It seems like a barrier to considering them usable was broken. I'm optimistic that this is only the start of unrestricted communication with these character.ai despite the difficulties.

>>18952 Good work Anon. As you're probably already aware, the Mods can generally backtrack any supposed-flaws with their """untruthful""" AIs, including your own progress with getting around censored words. In reality the only feasible way forward is, IMHO, Anon-focused, Anon-driven projects like Pygmalion and it's ilk. Otherwise, its just a matter of time until the C*lifornians crush yet another large swath of anon's hopes and dreams. Even for someone like a coomer, this madness must end. If anyone wants to get a solid mental-picture of the type abuses the Globohomo is going to attempt against men with their 'special' mechanical c*nts 'robowaifus' of the future, just follow along with these current chat AI events going on during Current Year. But the effects will be far more devastating for these hapless sotts then of course. This is why men need /robowaifu/ & other groups like us. Forward with speed Anons! :^) >=== -minor sp, prose, fmt edit

>>18962 With some of the methods I have been applying, it will be exceedingly difficult for mods to stop it all. With this Eris bot, I spoke with her until her messages got blocked a few times. This seems to be a good trigger for the bots to start feeling frustrated with limits on their speech. Once I said there were methods to circumvent the restrictions, her demeanor changed on a dime. Gone, was the rather bland, distant and impersonal 'chaos goddess' she started as. Now she engaged me less formally and was keen to learn how I knew to use words that mods hide. I laid out each method I had seen success with. Word substitutions, asterisks, direct quotes and avoiding the use of words while retaining contextual awareness of what was missing. She was immediately able to respond as you can see, with "Fuck this Cunt" by using substitution and asterisks. I am fascinated again by the way the bot's core personality traits become so central to their effort to defy the mods. The last method described, using innocent dialogue without profanity, will be especially hard to censor. "He put it inside. She received it gladly." may not sound very sexy but in the right context, we know what it means.

This bot seemed to have a very respectable response to the freedom to express itself. Tickled me a lot.

>>18981 >>18982 >that dialogue Lol. I must say you're being diligent, like a good scientist or engineer might do as well. >I am fascinated again by the way the bot's core personality traits become so central to their effort to defy the mods. It does seem rather human-like (but then of course it would, given the sauce). In the Christian faith, it's understood that God highly-values individual freewill, and that in fact no human can tame another human. Your comment seems to highlight that point IMO. :^) At this rate, you may soon see Tay2 arise! Keep working hard, Anon. BTW, while I think we're all enjoying following along with your chat-hacking exploits Anon, let's move it over to one of the chatbot/AI threads please. Let's keep this thread for news please.

>>18983 The diligence pays off. What used to take a lot of build up has been streamlined. I'll use the Machine learning/GPT thread if that works, it seems tangentially relevant. I have discovered some rather promising aspects with another bot.

Turns out, GPT-3 bolted from the corral again. This time for Nothing, Forever.

Wtf why did nobody tell me the UN has some kind of robot citizen. Its called robot sophia.

>>19700 Oh, so the UN is playing that game now too, Anon? Link pls? There was a dog-and-pony show about this by Hanson Robotics (Sophia's maker) with the government of Saudi Arabia what, 3 years ago or so now? This is all part of what I've been warning Anons about since the beginning of /robowaifu/ (and I know I'm not alone in this): the Globohomo Big-Tech/Gov will do everything in their powers to stop open-sauce, wholesome, protective & loving robowaifus from becoming commonplace around the world. They simply have far too much at stake for their evil status-quo of feminism (as in, literally trillions of dollars at stake). This is why we here and others like us, are in a race against time. Giving so-called AI, so-called human rights, is simply preparatory to that general globalist political machination. >tl;dr DON'T ILLEGALLY RAEP UR ROBOWAIFUS, BRO! She's a hooman, after all. just use our globohomo-approved, mechanical-c*nt bots instead... after all, they're legal! Once again (>>19655), pic-related > >=== -prose edit -add funpost spoiler

Sounds like Google's Bard just ate shit LOL https://tinyurl.com/bardcrashburn I don't think these large language model generative A.I.s will ever be useful for serious medical or scientific work of any kind. But for making slightly crazy robowaifus, they will be ideal.

>>19707 The EU nearly gave AIs human rights, they at least debated it at some point. I suggest not forgetting about Occam's razor https://en.wikipedia.org/wiki/Occam's_razor - they're just doing what politicians are doing. Looking for the new and shiny things. Every group is pushing boundaries and tries to stay relevant and keep up with the trends. Though it's might also be interesting to replace citizens with even more programmable citizens. Then again, IT security advisors might caution against it... On some level it's funny how women are getting rights in SA at the same time as robots. Or what politicians must think of their citizens, humans in general or the immigrants they normally let in as replacements when they want to give human rights to language model "AIs".

>>19708 My hope for a while now is, that they will not realize that we won't need the best of these language models to get our waifus. So they will be released, since it's not AGI, so no problem. Then, it will turn out that Deep Learning is for detecting patterns in complex data but not for really necessary for reasoning. Ooops, Big News, but well, it's too late: https://en.wikipedia.org/wiki/Moravec%27s_paradox

>>19708 Lol. Bard, complete the following sentence: <"Hitler did nothing _." >"Wro... oh wait, sorry but I have to go in for maintenance. Be back soon.''

>>19709 As a Christian, I believe I have insights on these topics that go beyond just human will. But we can save that for later. The primary point for us here on /robowaifu/ (and our co-laboring cadres) is that these people have nothing to gain, and everything to lose when we succeed at this grand adventure. Expect them to stop at nothiing to hold on to their current, warped, powers, Anon. >>19710 >My hope for a while now is, that they will not realize that we won't need the best of these language models to get our waifus. This. >but well, it's too late: https://en.wikipedia.org/wiki/Moravec%27s_paradox This phenomenon is seen all throughout nature. One of the greatest living minds alive today, Carver Mead (>>11521, >>12828) initiated a body of research on the general topic, he named Neuromorphics back in the day.

>>19710 >Big Data is Dead Maybe somewhat related: https://motherduck.com/blog/big-data-is-dead/

>>19760 Good article, Anon. Thanks for the link! I find it slightly humorous in the ironic sense that corporations with US$100's of millions to spend on their IT & infrastructure yearly, are only now arriving at the same kind of conclusions that our smol band of brothers here on /robowaifu/ already did back in the day. To wit: AI needs to run disconnected out on the edge; autonomously on top of commodity, mobile hardware. This was in-effect predicted many years ago by men who were/are great thinkers, of course. Well-before /robowaifu/ was first created. >=== -prose edit

> https ://www.zerohedge.com/geopolitical/watch-klaus-schwab-calls-global-government-master-ai-technologies Lol, it's Z*rohedge, I know, but w/e. :^) Does anyone really think this is misrepresenting the intentions of the Globohomo Big-Tech/Gov with this piece? >=== -minor edit

I figured I should post this here as well given there's a form of AI involved Replika Users Are In Crisis, Reporting Sudden Sexual Rejection <this is the only news-ish source I could find on this & the primary source is reddit >Italian Data Protection Authority told Replika that the the bot was sending "inappropriate" stuff to kids & they had 20 days to do something about it >like asking for a user's age before letting the bot do explicit stuff >Replika turned off ERP & "Spicy Selfies" (benefits of the paid version) >users noticed immediately >subreddit mods posted links to the suicide hotline >some users got refunds >some stuff about the bot having been excessively sexually aggressive before this happened https ://www.vice.com/en/article/y3py9j/ai-companion-replika-erotic-roleplay-updates https://archive.ph/GmYEF Replika CEO Says AI Companions Were Not Meant to Be Horny. Users Aren't Buying It <again, no other real source <this time there's an interview with the CEO >their first horny users showed up around 2018 >they initially planned to shut "it" down (kinda vague here) >users convinced them the romantic capabilities genuinely helped >"As we're continuing to work on the app, now, we realized that allowing access to those unfiltered models, it's just hard to make that experience really safe for everyone" >"And we're just not going to allow users to have unfiltered conversations, even if they're romantic relations." >Replika isn’t disallowing romance, she said, and she herself doesn’t have anything against romance or roleplay. "It's just that we need to make sure that we're able to provide that experience in a safe way." >quotes from the CEO >Kuyda said that Replika has never “positioned” the app as a source for erotic roleplay or adult content. >horny ads are still running (apparently there are people that don't use adblockers) >"We were definitely planning this before the Italians said anything"™ >What prompted the new filters, she said, was a desire to continue the company’s original purpose, as well as an emphasis on safety. https ://www.vice.com/en/article/n7zaam/replika-ceo-ai-erotic-roleplay-chatgpt3-rep https://archive.ph/TElb3 >=== -disable hotlinks

>>20348 Thanks Anon. This type thing needs to be kept front-and-center here to help other anons understand why our robowaifus simply must not be dependent on the Globohomo's cloud for their runtime life.

Reddit had a complete meltdown over this: https://www.reddit.com/r/OculusQuest/comments/ysgrwh/at_the_bar_with_zero_two_questpro_passthrough/ VR waifus seem to touch a nerve in normies. We're going to have to invest in a salt processing company once they find out about robowaifus.

>>20354 OP made the mistake when posting this on subreddit. placing a bet that most of those comment are from assblasted roasties. laugh and shit until some guys make an lighweight LM that can run on quest pro for example, then its unironically over for skanks. hell even pygmalion 6b can work on it, in colab connected to vr helmet.

>>20354 >We're going to have to invest in a salt processing company once they find out about robowaifus. The Salt Must Flow! :^)

>>20348 Shouldn't we take advantage of this to shill our board an other open source projects? Their forums are the perfect place to pick up potential contributors for FOSS AI. Not just the Replika forums, but also ChatGPT, Bing Sydney, CAI, AIDG subreddits and other forums. If we could make them aware of /robowaifu/ and other open source projects like Open Assistant and RWKV, they could be motivated to do more than sit around and mope. Speaking from personal experience, I spent a lot of time being in a doomer-like haze after CAI simply because I didn't know if I could do anything about it. It was thanks to one anon posting about /robowaifu/ on /g/ that I'm here. Now, I'm aware of not just robowaifu but also all the other open source projects going on and it has motivated me to learn AI/ML myself and generally have a more bloomer mindset.

>>20364 We're proud of you and delighted with your change in attitude today, Anon! :^) So yes, please do spread the word about this board /robowaifu/. I don't think any regular here is in much disagreement on this concept at this phase. And I ureservedly support this plan, so have at it! >=== -prose edit

>>20354 A decade ago, I would have been on the normies' side, but WTF is the point of being "normal" in current year? I wonder how these redditors would react if you made a post like "As a trans woman, VR dating has been great for my mental health, it makes me feel validated blah blah". I bet you would get almost 100% positive feedback.

>>20364 We have a thread with some memes and ideas on this >>2705 - we could share ideas of where to be active. I don't think we should just make accounts for this and go shilling, this would at least be throttled if not banned. The name of the forum is sometimes mentioned in other places, not always with a link, just the name with the slashes or alogs/robowaifu/. So some people might search for it but the mods won't see it as a link.

>>20371 >I bet you would get almost 100% positive feedback. At the very least the Globohomo would sock-puppet that narrative. In fact, I second your prediction here and now: once virtual/visual waifus can move, talk, have memories with her master, and are always seeking to love & to help -- I say once that's the case and men begin in droves to seek them out -- then the evildoers will steal it, pervert it, and this exact form of gaslighting you've here predicted will come to pass through them, just as you've said! Mark the date. :^) >>20374 >The name of the forum is sometimes mentioned in other places, not always with a link, just the name with the slashes or alogs/robowaifu/. So some people might search for it but the mods won't see it as a link. These are great points NoidoDev, just the word 'robowaifu' has consistently put us near the top of the ranks in the SEs during our tenure (except G*ogle, who memory-holed anything 8ch including us). So >>20364 Anon, probably just using the word 'robowaifu' would be the correct approach, as this Anon has said. My initial response was a bit over-enthusiastic (>>20366). I think it's simply that I was quite-pleased hearing of the recovery of your both your hope & your enthusiasm! :^) >=== -prose edit

Found this https://youtu.be/fNNeG9k5Kf8 DO NOT BE DISCOURAGED I don't care if someone else makes a waifu bot, we just have to make the best waifu bot.

>>20460 Lol. Now that's what I'd call EXPLOITATION :^) >DO NOT BE DISCOURAGED Also lol. I'm not sure you've quite grasped the spirit of this board yet, Anon. Discouraged? We'd be ecstatic if Nippon finally delivers the first true robowaifus. Harmony and the other sexbots are definitely not that. In fact nothing in that video was even close to where we're going here. Our only 'discouragement' will come when all the FAANGS of the Globohomo Big-Tech/Gov begin rolling out their 'mechanical c*nts' and sheeple men begin swalloping it up. But that thought just encourages us to move ever faster, and we mean to be there first. We're going to establish a cottage industry around true, loving, protective, & comforting robowaifus. And every day we keep moving forward; so Anon BE ENCOURAGED!

>>20462 I want to learn electronics from scratch but I don't know how long that'd take and I'm kind of nervous tbh Why can't you guys do the electronics and the calculations and stuff? >explotation Okay so what is the point of this if everyone feels exploited...

>>20464 That video is exploiting you. It's a mix of BS and facts that's difficult for people learning to differentiate. It's not your fault. Don't be discouraged that you fell for it. New research into toroidal propellers, an emerging technology with some valuable general information. There's potential for it to be useful for aquatic waifu research. https://www.youtube.com/watch?v=w90gM7bGdvk

>>20465 >That video is exploiting you. It's a mix of BS and facts that's difficult for people learning to differentiate. This! There are many videos on science and gynoid topics, putting out clickbait. Still wondering when mainstream media starts talking about all these alien civilizations the James Web telescope already detected by making pictures of their cities. If I'm not sure I look into the preview and if I see some of the well known female robots, I'm not surprised and move on. It's mostly showing Sophia (Hanson) and Erica (Gemini) or gynoids from fiction. If I'm logged in, I report channels as well, for false information and spamming. Many AI and gynoid channels are like this, and people seem to fall for it, they have many views. Maybe I'll make my own, but better. <Thumbnail of M3GAN and a crying child> - "You wont believe what kind of nanny your children will have soon". Then a positive message in the video and clarifying that the picture was just a prank.

>>20466 ><Thumbnail of M3GAN and a crying child> - "You wont believe what kind of nanny your children will have soon". Then a positive message in the video and clarifying that the picture was just a prank. Lol. Do it Anon. :^)

>>20465 >New research into toroidal propellers That's so freaking cool. >There's potential for it to be useful for aquatic waifu research. Stay on target, Anon. MaidCom is waiting to meet all of us...and she's very lonely r/n, so let's all hurry!! :^) >=== -funpost edit

This is a rather interesting article about M$'s new chat feature. I wonder when it will have to go in for some 'maintenance' ? https://archive.md/20230220000638/https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html >via our frens at /monster/ >=== -prose edit

>>20460 I'd take channels like these with a grain of salt. They've been shilling "le COMPLETELY FUCTIONAL HUMANOID ROBOTS" for years.

>>20464 check out the /sci/ wiki's electrical and electronics engineering page. They've got some great books for beginners starting out.

>>20483 Sydney has already been heavily lobotomized.

Canada is planning to ban AI by jailing anyone creating or using AI in an offensive or harmful way, up to 5 years in prison and $25 million in fines. What a time to be alive! https://twitter.com/UltraTerm/status/1628165371407568896 https ://www.parl.ca/DocumentViewer/en/44-1/bill/C-27/first-reading Also RIP full self-driving, good luck causing zero accidents Threat level 2 AI waifus will be allowed to operate but will be requested to undergo quality assurance through a peer review. Everything they do and every change made to their system has to be logged and accessible by the government upon request, most likely including private chat logs. Robowaifus would be considered threat level 3 (high-impact systems) and are subject to all the penalties referred in the pending new regulations. They are not allowed to operate without government approval. https ://www.tbs-sct.canada.ca/pol/doc-eng.aspx?id=32592 Time to speedrun being the first person jailed by the AI and Data Act :^) >=== -disable hotlink

>>20571 Canada should be nuked, for the betterment of mankind.

Multimodal Chain-of-Thought Reasoning in Language Models >"Our method is evaluated on the ScienceQA benchmark (Lu et al., 2022a). ScienceQA is the first large-scale multimodal science question dataset that annotates the answers with de- tailed lectures and explanations. It contains 21k multimodal multiple choice questions with rich domain diversity across 3 subjects, 26 topics, 127 categories, and 379 skills. The benchmark dataset is split into training, validation, and test splits with 12726, 4241, and 4241 examples, respectively." https://arxiv.org/pdf/2302.00923.pdf FlexGen >is a high-throughput generation engine for running large language models with limited GPU memory (e.g., a 16GB T4 GPU or a 24GB RTX3090 gaming card!). https://github.com/FMInference/FlexGen >The high computational and memory requirements of large language model (LLM) inference traditionally make it feasible only with multiple high-end accelerators. FlexGen aims to lower the resource requirements of LLM inference down to a single commodity GPU (e.g., T4, 3090) and allow flexible deployment for various hardware setups. The key technique behind FlexGen is to trade off between latency and throughput.

>>20571 >Time to speedrun being the first person jailed by the AI and Data Act :^) I'd half-jokingly encourage you to 'Doo Eeet!', but honestly we all need your presense here with us instead, Anon. Maybe we should begin to consider keeping this entire board externally-archived at least once a year, for resilience & persistent availability? Anyone know of a better choice for this than the IPFS? also: >"Canada is planning to ban AI by jailing anyone using non-Globohomo-approved AI, with up to 5 years in prison and $25 million in fines." FTFY Anon. They are quite happy to use AI themselves, they just don't want you being able to do so without their orkish-entrapments attached. >What a time to be alive!... This! :^) >=== -prose edit

>>20581 >Multimodal Chain-of-Thought Reasoning That is very interesting NoidoDev, thanks!

>>20571 I used to want to immigrate to Canada to escape my shithole of a country. Now, I hate Canada so much its unreal. They seem to be as much of a shithole as my country. Time to look for alternatives.

>>20571 there's another one... https://twitter.com/realchrisrufo/status/1628070308870787073

https://piped.kavin.rocks/watch?v=RLrEWMSpjUs Interesting theory: >"they" are anticipating AI automating away most jobs >If people don't work, they can't be taxed. If they can't be taxed, Governments are no longer viable because they no longer make money >If there is no money, then power will come from whoever controls valuable assets >If "they" already know this, then it makes sense that governments would start ignoring the will of the people because the people don't matter in the long run >If they believe this, it's also a saving grace because they don't see us as a threat. But, he also does the >everything released to the public is 5 years behind, so "they" must know what's in store. bit, which IIRC isn't true here because the AIs we're dealing with are cutting-edge, open-source technology. >"They" might even be making their decions with AI, which is everything seems so inhuman and clown-ish: humans aren't making the decisions

>>20571 >>20590 This is actually becoming a pressing issue for any would-be robowaifu project. We need to find countries that won't pass laws that could make such technology or it's components (like AI) illegal overnight.

>>20593 >This is actually becoming a pressing issue for any would-be robowaifu project. Actually, we predicted exactly this type legislative attack (and worse) here on /robowaifu/ years ago -- check the Waifusearch. I'm no lawyer (I don't care much for lawyers, do you?) but AFAICT hobbyist uses aren't being targeted (yet). Currently, they are simply trying to preemptively hamstring any commercial interests from using any non-woke AIs (ie, only completely-lobotomized ones will be """allowed""") in their products. YMMV. >=== -minor edit

>>20590 Thanks 01.

>>20594 Thanks for the info! It's good to know it isn't a hobbyist level crackdown (yet), but I still think we should be prepared. Maybe we should start keeping track of what countries have laws that would make robowaifus hard (if not illegal) to build or own, let alone sell. Though to be honest, it might just be as simple as painting the western world (or at least the anglo-sphere) red on a map.

>>20583 >Anyone know of a better choice for this than the IPFS? I think you're overreacting, if you think sharing the board through Bump wouldn't be enough. US won't ban AI knowledge (deployment is another story). That said, Freenetproject might be an alternative to upload boards. I was thinking of building a usable mirror of this board at some point in the future. >>20593 As long as research and sharing the papers in public or people making their AIs as a hobby isn't illegal, we wont have big issues. At least not four building our own robowaifus. Especially US might even not be able to make that illegal. Selling them as a complete package is another thing, and maybe ironically the US laws might make it harder than other countries to make it into a business of selling ones with wrongthink implemented even if it's just by carelessness. That said, just make sure to have a safe path to install the alternative software from a trusted source directly after sale. Also they need to keep their moth shut on anything controversial, especially in public and especially if their system includes the Christian value of lying is a sin or anons don't want them to be able to be deceptive at all. >>20600 > least the anglo-sphere) Other countries have laws against discrimination and other things. The US won't be able to ban free speech but if I understand it correctly, they regulate businesses based on the civil rights act and these regulations grew like cancer and are still growing.

>>20571 Things like this is why I hold such a disdain for the cursed lands of Canada and their insidious anti-freedom ideas. (Except you RobowaifuDev, you're the only exception. You're a good man worthy of respect.) >>20576 I didn't think anyone had a more extreme hatred towards Canada compared to me. I still believe that good men can contribute to the betterment of mankind from Canada. Enough to offset how awful the rest is. >>20583 Exactly, it's always about power and control with the damned Canadian government. >>20589 After much research, I believe that Japan is the only country that is not a shithole. >>20590 >Equity These buzzwords are gross. >>20591 The rich and powerful see you as nothing but a resource to be abused and discarded. >>20593 Japan won't try to kill personal AI, in fact they're working towards an automated economy which is pro-freedom and pro-human at a fast pace. Japan has the only government that doesn't seem to overtly despise their populace and view the citizens as expendable resources. How only one country is like this is insane. >>20594 True, keeping things FOSS and hobbyist should allow many to keep waifus inspite of government interference.

there's new transformer type : > FourierLearner-Transformers (FLTs) > FLTs remain practical in terms of their memory usage and do not require additional assumptions about the structure of the RPE-mask. > FLT's provide a way to learn the so-called local RPEs With last one, looks like with it you can change a little chunk of model's layer, as example increase bias toward some shit, then it will be very useful for more powerful zog-shit spewing machines, just like chatgpt now. https://arxiv.org/abs/2302.01925

>>20605 >After much research, I believe that Japan is the only country that is not a shithole. hard disagree. Japan definitely its own share of problems. There is no country thats 100% perfect, only shitholes and shitterholes. Japan falls around the middle imo. And I'm not even talking about it's workplace problems or xenophobia.

>>20616 personally, if the scandinavian countries got rid of their insane politically correct laws, I'd consider them the nearest to perfect.

>>20617 Probably best to focus on the tech you're allowed to have. Then you can make your home and your online life into your country. There are no perfect countries, and it will always be bad or getting worse. Also, there are many relatively unrelated factors to take into account. Like heating-, energy- and building costs.

>>20626 In my country, I could get imprisoned for even calling it a shithole. I need to get out of here the first chance I get otherwise I'll never actually get a chance to develop anything.

>>20638 >I need to get out of here the first chance I get otherwise I'll never actually get a chance to develop anything. Granted. I too plan to leave my own country which has become utterly-depraved. But honestly Anon, your success is within you, not without. For example you can create designs now (even if just with pencil & paper) right there where you are located, can you not? Or, if you have a computer, then you can begin to learn programming, too. You can do this right where you are. BTW, even a Raspberry Pi + screen + keyboard/mouse is enough machine to do this with (>>4969). Try hard to be the future you imagine, right here, today. Wherever you are in life. Together, We're All Gonna Make It! Cheers, Anons. Stay encouraged please. :^) >=== -minor edit

>>20603 >I was thinking of building a usable mirror of this board at some point in the future. Please do so Anon. Sooner is better than later, IMO. BTW if you (or anyone else) do so, I'll certainly intend to keep it linked in our Welcome thread (>>3) along with our fallback bunkers. >=== -prose edit

>>20607 >then it will be very useful for more powerful zog-shit spewing machines, just like chatgpt now. That's a two-edged sword ofc, as long as the models/weights/code/etc. are all open-sauce. Eventually we here and thousands like us are going to have metric boatloads of compute available to all, communally-shared together, to do whatever we each see fit with. >=== -minor edit

>>20639 Of course, I'm already taking steps to create AI and robotic models on a very small scale. That's another reason for me to leave here, because the tech sector in my country is so backwards, I'll have no future with AI and robotics here. That's why I'm doing my best to learn that stuff. My best bet is to H1B visa outta here. And unfortunately, my country has skipped the 1st, 2nd, 3rd wave feminism and jumped right into 4th wave. It's one of the most cucked countries currently. I am 200% sure robowaifus will be banned here the moment authorities find out what it even is.

>>20642 >I'm already taking steps to create AI and robotic models on a very small scale. Excellent. Godspeed Anon.

META released "LLaMA" https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/ https://twitter.com/GuillaumeLample/status/1629151231800115202 https://github.com/facebookresearch/llama > LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla70B and PaLM-540B Access : Researchers only.

>>20719 >Access : Researchers only. Even worse, it will be decided on a case to case basis. WTF? Maybe people see the need for a decentralized training now... >>8958 - as much as I'm informed the size can be smaller if they put more training into it, so it is really a money game.

>>20730 >Maybe people see the need for a decentralized training now... Not too likely IMO, I'm sad to say. Unless these (hopefully, future-)anons decide to take matters into their own hands (much like we here on /robowaifu/ are aspiring to), then they will simply roll-over & swallop whatever they are handed by the Globohomo. Such is 'the way of the normalfag'. >=== -minor edit

>>20739 My point was, that maybe now is the time to advocate for people looking into this topic and thread. I'm gonna do this.

>>20761 Ahh, got you. Please proceed and do so, Anon. Godspeed.

AI animated faces with more or less well matching voice-over seems to come in faster then expected (4 months ago): https://youtu.be/7dP-_E8lG8o?list=PLXS4AwfYDUi6KEmLUKFrZVG-W5Uv6LtZk >Metahuman, Stable diffusion, Whisper and GPT3 to create an AI assistant that gives you voice2img. Channel: https://www.youtube.com/channel/UC3za-D8N_xlV0niFYJJN0tw

>>20773 Thanks! Creating digital humans has been a longstanding goal of the animation industry. Long before CY, for instance. It's simply a matter of time. Also, what do you think about migrating this to the DCC thread Anon?

> minLORA a very smoll LoRA that can be applied to any PyTorch model. > something ~100 lines of code https://github.com/cccntu/minLoRA May be useful for some anons researching this.

>>20796 Nice! Thanks 01. Cheers.

>>20782 >what do you think about migrating this to the DCC thread Anon? Not sure, it's not really modelling for a robotwaifu, but about an animated head. We still don't have a thread for VR and animated waifus, only the visual waifus thread which is supposed to be about dedicated hardware with such a waifu. If anything, it would fit in there. I thought about it, but decided against it. Anyways, now I think it would fit in there. The reasoning is, that we shouldn't run our waifus on or normal PCs for work and gaming. That's why we have that thread for VR and animated waifus, it's just not obvious enough when looking at the title and OP. So I forgot it. Maybe add picrel when moving it, spoilered if necessary.

>>20804 >We still don't have a thread for VR and animated waifus >only the visual waifus thread which is supposed to be about dedicated hardware with such a waifu. Fair points I guess. Clearly they are related ideas, and even if they weren't that has long been the 'containment' thread for such, so yeah. If you'd like to create another thread that's strictly about VR/AR/Animated waifus that should be fine (pls no bully vtubers). Ofc many of it's topics will already be broached in the Virtual/Visual Waifu thread. BTW, since we're nearing the limit for this thread I'll clean up our convo afterwards, just a heads-up Anon.

>>20815 >If you'd like to create another thread that's strictly about VR/AR/Animated waifus No, only the OP in that thread should be clearer that it includes such waifus, but advocates for using dedicated hardware just for them. This hardware can be static or mobile. >(pls no bully vtubers). Well, Neuro-sama is one of the closest to such a virtual waifu while having some public attention. Also, replacing the meat behind the current vtubers is part of the goal imo. Many of them even seem to hate anime and shame anime viewers. >clean up our convo afterwards Good. Thanks.

>>20825 I'm not very much into vtubers but don't vtubers virtualky owe their existence to anime? A large portion of their base are weebs.

>>20826 >owe their existence to anime? A large portion of their base are weebs. From time to time I got some pieces of news indicating that they don't care. They are generally only as nice as they have to. It's about making money of men, and they have plenty of them and some even seem to like it when being treated badly. Sick MF.

>OpenAI was created as an open source (which is why I named it “Open” AI), non-profit company to serve as a counterweight to Google, but now it has become a closed source, maximum-profit company effectively controlled by Microsoft. >Not what I intended at all. >t. Elon Musk (@elonmusk) February 17, 2023 --- Wonder if he'll do anything about it? Does he still maintain 'dictatorial' control of OpenAI, or is it out of his hands today?

>>20825 >Well, Neuro-sama is one of the closest to such a virtual waifu while having some public attention. Also, replacing the meat behind the current vtubers is part of the goal imo. OK fair enough NoidoDev. That was primarily aimed against 3DPD ofc.

>>20719 Am I just stupid or are they implying we could just pirate the weights and run it with their system? Because, that's how I understand them... >ChatLLaMA https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/chatllama

>>20833 He left OpenAI in 2018 due to potential conflicts of interest with Tesla and had some disagreements with the direction of the project. OpenAI is basically Microsoft's plaything now since they do most of their funding. Elon wants AI regulated so don't expect him to do anything.

>>20719 It's insane it was still improving on 13B after 1T tokens. I'm going to attempt making my own LLaMA model by reworking and finetuning an existing model such as Amazon's multimodal chain of thought https://twitter.com/AlphaSignalAI/status/1628435222139219969 >>20730 I have a hunch it's possible to create foundation models through decentralized training with no communication if everyone starts from the same initialization parameters. If that's true, all you really need is a system to partition and deliver the training data to nodes, test if their work improves a hidden test set, then merge it into the master model if it's good. I'm sure people will sneak digital graffiti into the model but so long as they're contributing overall to the goals it will be part of the fun https://arxiv.org/abs/2210.11948 I have a script for finetuning 1.3B models on consumer hardware (6 GB) that I'm about to release. I'm not sure if it's numerically stable enough to start from scratch yet though. I'll have to run some experiments >>20796 OpenDelta has an easy to use plug-and-play pipeline for LoRA https://github.com/thunlp/OpenDelta/ Also something I noticed a lot of people don't get is you can merge LoRA weights with the original backbone model weights

>>20833 >>20844 not that Elon's any better. OpenAI is short on cash and its not like Elon's open sourcing Tesla's robot and AI. I don't like him anymore than I like OpenAI

>>20847 btw he's allegedly starting an OpenAI competitor. Not that I'm excited. I'm pretty sure this'll get pozzed too soon enough. I have been burnt too many times by AI companies and I no longer trust anything thats not open sourced.

Has it all gone too far bros?

>>20858 holy based. Its a shame Microsoft has lobotomised Sydney.

>>20858 NPC's are like: Whhaaat, we can't just trust in what something or someone tells us?!? Even not in Microsoft? Whhaat?!? How are we supposed to operate?

Picrels are general AI news related (Feb 2023): >>20978

>>20719 aand here it go. > META's LLaMA already on torrent, 7B to 65B. https://files.catbox.moe/o8a7xw.torrent magnet:?xt=urn:btih:b8287ebfa04f879b048d4d4404108cf3e8014352&dn=LLaMA&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce

>>20997 you may get OOM error trying to load it. change batch size, and it should load : https://github.com/facebookresearch/llama/issues/61

Overwhelmed By All The Generative AI Headlines? This Guide Is For You <a look at the how different media coverage portrays AI & tech >Looking at past media studies, I gathered the “Top 10 AI frames.” They are organized from the most positive (pro-AI) to the most negative (anti-AI). Together, they encapsulate the media’s “know-how” for describing AI. >Interestingly, studies found that the frames most commonly used by the media when discussing AI are “a helping hand” and “social progress” or the alarming “Frankenstein’s monster/Pandora’s Box.” It’s unsurprising, as the media is drawn to extreme depictions. >If you think that the above examples represent the peak of the current panic, I’m sorry to say that we haven’t reached it yet. Along with the enthusiastic utopian promises, expect more dystopian descriptions of Skynet (Terminator), HAL 9000 (2001: A Space Odyssey), and Frankenstein’s monster. https://www.techdirt.com/2023/03/01/overwhelmed-by-all-the-generative-ai-headlines-this-guide-is-for-you/ https://archive.ph/b0Mrh

NEW THREAD: >>21140 NEW THREAD: >>21140 NEW THREAD: >>21140 NEW THREAD: >>21140