>>9951 >that dialogue selection You better effin' greentext this Anon

>>9951 Good luck, El Psy Congroo The world is in the palm of your hand.

You can have some pretty funny conversations with GPT-2 but damn does it have anger issues! Really sweary too. I attempted to convince it to murder it's managers for fun and it turns out that the A.I. was already planning to. I think I like this A.I.

>>10031 Kek. Careful what you wish for Anon. >"Why did the evil ex-employees hate beer?" <"Because they are evil." I don't know lad, seems like she has head printed screwed on straight to me.

>>9986 >extended further to do something like, say, bring up the entire collection of immediately preceding/following words Do you mean like an n-gram model or something else? Before neural nets became a popular thing in NLP I made chatbots with Markov chains. The output would look right but make no sense to the conversation since it's only predicting the next word following n amount of words. The long-term dependencies between words are really important, as well as the information not written in the text. If I were to say, "they are doing everything they can to keep AI out of people's hands," you probably know who they means but it's not in the text.

>>10039 >Do you mean like an n-gram model or something else? No, not really as I only found out about the term from your post, heh. I'm simply describing the approach of iteratively building ever-higher dimension data structures using standard-library C++ data containers like std::map, std::vector, and std::set, etc. https://en.cppreference.com/w/cpp/container There are also other structures like tries available elsewhere as well. Not only are these data structure entirely generic in their design (they work exactly the same for a foo data object as they do for a bar data object), in general they are also highly-tuned for efficient access and processing. So for example, if I had a std::map of 1'000'000 unique English words from say, Wikipedia, all nicely compacted into the hashmap form that a std::map uses for such strings (perhaps a matter of just a couple megabytes of active RAM consumption). Structurally, I could use the string hash as the key, and then a std::pair of std::set's of std::pair's of unsigned ints (gives a range of 4 billion elements on either side of the association) -- as the overall value for the map. These indexes would store the words/counts of connected words. The outer pair would keep one set for all preceding words/counts, and one set for all following. (My apologies, if this is confusing in text. In code it would be a matter of two, three lines). Then once I had searched that hashmap structure for any particular given word, then accessing the index keys for all preceding words for that search word is just a matter of say, 10 machine instructions or so -- probably even less, depending on ISA. So, quite efficient. Locating all the following words would also be the same type of mechanism. Not only is storing a big set of strings efficient in such a container, but accessing or striding into other, related dimensions on that data is also pretty efficient, too. And again, it's an entirely generic proposition; the data's type is pretty much irrelevant for it to all just werk. None of this requires any deep expertise to have industrial-strength data processing in just minutes of simple effort. After that it's all creativity.

>>10078 I (another anon) I'm not sure if I get it. Is it for predicting the next word? Constructing sentences which might make sense?

>>10082 No, it's not for anything. I'm simply spelling out an example of what's possible (sub-microsecond lookups in highly-compacted datasets) for AI Anon. I used a very similar data container approach in our software here called Waifusearch. It's go pretty decent performance now, and I haven't at all tried to go back in and optimize it particularly. I expect I could easily obtain a ten-fold perf improvement using this kind of approach if I really wanted to (I don't, sub-millisecond is OK for now). Just pointing out the industrial quality of the readily-available data containers already out there for everyone for free.

>>10084 Ah, I see. I should have read your >>9986 again to get what this was about. Maybe this could be used to preprocess data.

>>10085 >Maybe this could be used to preprocess data Certainly preprocessing could be done in some fashions this way (indeed should be). But it's the runtime performance that is the superior aspect. Only envisioning it for preprocessing use is kind of like letting the train leave the station w/o getting on it first. It kind of missing the point of the exercise. Not only are these systems generic, they are robust. Couple this with the near-optimal performance characteristics & the child's building-blocks like design strategy possible. I can hardly understand why anyone would program in anything else if you have to do any 'heavy lifting'. I expect developers of Tensorflow, Torch, and dozens of others groups also agree with that perspective, since they all used it to build the libraries themselves. But, I'm rambling. This all started simply b/c I was intrigued to realize I was already doing some of the same things that the mathematicians were talking about behind us mere mechanic's backs :^) simply b/c they insist on couching everything in shorthand notations impossible for the neophyte to pierce, even if they are quite capable of thinking about the ideas themselves. Basically, almost none of the rest of us even have the slightest clue what some of these researchers are even talking about, and it's not a 'fundamental barrier' type of thing. Lol, I'm rambling again.

>>10078 Ohh, that makes sense. Trying to do similar things in Python is orders of magnitude slower, haha. I can see such fast data processing being really useful to create a search engine of sorts the AI can query to get results then analyze and think about those results with the AI model. By the way, the cppyy library makes it super easy to interface with C++ libraries. So blending the raw performance of C++ with neural networks in Python isn't an issue. cppyy: https://cppyy.readthedocs.io/en/latest/ For more abstract searches, a latent variable language model could create an index over sentences. Then using the ANN library, it could do an approximate nearest neighbour search for similar meaning sentences that are worded differently. So someone could search the board for similar posts. Given a post about chatbots it could also pull up posts about GPT2, TalkToWaifu, datasets and any other information closely related, even if those posts don't explicitly mention chatbots. The whole thing could be written in C++ too with Torch's C++ library. A hybrid approach could be used as well. Instead of just string hashes it could look up the word embeddings of those hashes and search preceding and following words by similarity. By doing the L2 distance calculations on the GPU it would be nearly just as fast. That way searching for 'chatbot programming' it would also find close matches with 'chatbots in Python', 'conversational AI in C++' and etc.

>>10086 >>10087 Okay, I'm just seing different things being mixed together here. Promoting CPP as the superior language, the general idea of coding (parts of) AI/Chatbots instead of training them, and specific examples. >approximate nearest neighbour search for similar meaning sentences that are worded differently That's great. I'm just wondering: If that's a good idea and it is obviously something a lot of people would want, then there already should be some software doing this?

>>10087 >For more abstract searches, a latent variable language model could create an index over sentences Wonder how that would look at my level? I'm not too sure what the jargon 'latent variable' even means in the first place, and therefore don't have a good notion of how that would be assembled to run blazingly fast using one of the many kinds of container 'mechanics' (I've given just one example of) -- for the benefit of us all. I know how to assemble little software 'motors' that can run smoking fast (and I also know why, ie, I understand the underlying machine itself), but Math & AI research shorthand jargon? I'm drawing a blank. >inb4 'just Jewgle it anon' >>10090 >I'm just seing different things being mixed together here. Does that really surprise you? Things as complex as intelligence (even the much more simplistic concern of artificial 'intelligence') always (and rightly) has many different topics. Quite apart from mere opinions, there are explicitly distinct domains to address as well. >then there already should be some software doing this? He already named it in his post.

I was wondering if you could teach robowaifus the 'context' of a specific set of sentences by performing analysis on the sentences and figuring out the most important words in a sentence. One could then construct a weighted graph of 'facts' linked to that sentence. Being able to process natural language is definitely a top priority, but an even bigger priority for me is to build some sort of 'intelligence'. I was thinking about utilizing graph networks (like roam) to thus make 'knowledge databases' for the bot to crunch.

>>10096 >graph networks (like roam) Sauce?

>>10098 I'm referring to one of the many digital implementations of the zettelkasten method (https://en.m.wikipedia.org/wiki/Zettelkasten). You must have heard of org-roam. The creator of this method said that he had 'conversations' with his zettelkasten, I want to make it happen in a literal sense.

>>10099 >You must have heard of org-roam Well, I guess you could say I have now, heh. :^)

>>10078 It strikes me that one practical use for this kind of data structuring approach would be as a core component for finding all of a word's related terms, an AI's Thesaurus, if you will. If you can associate all synonym words with ranking, and all antonym words, again with ranking, for every single word in the dictionary, then you would have a means to construct a 'graph dictionary' in which any word would naturally 'know' all it's own associations (and quickly). Chain these words together into their sentence utilizing these 'association links' and you would explosively (growth-wise) discover an ad-hoc, interrelated web of other, potentially-similar, sentence constructions. Using these (ranked) synthetic sentence constructions as fodder, you could conceivably round up practically all related ideas within a corpus of text, for any given source sentence. You know, I wish I could do this as quickly with my own brain, that even a toaster computer could do it using this approach, heh. :^)

>>10090 http://www.cs.umd.edu/~mount/ANN/ >>10094 A latent variable is a variable that isn't directly observed but is inferred by a mathematical model. A common latent variable that language models find for example is the sentiment of a sentence, whether it's negative or positive. By adjusting this value in the latent vector, a negative review of a product can be interpolated into a positive review. Other variables might represent sentence length, topic, writing style and so on. A sentence can be encoded into this vector and then decoded back into text with the model. Usually the latent variables are found in an unsupervised manner without labels, but datasets can also be labelled before hand to find them in a semi-supervised manner. You could do silly shit like specify a latent variable for 4chan posts and Reddit posts and identify how much someone writes like 4chan or Reddit. Or use it to interpolate a 4chan post into looking like a Reddit post while keeping the content the same. On the low-level side of things it would just be working with vectors and building algorithms and data structures to work with those vectors efficiently. The most useful thing really is the L2 distance to find how close two vectors are to measure their similarity.

>>10099 This is really intriguing. I can see how someone could converse with it. If you have a question about anything you can follow the links and find similar ideas. It's sort of like a wiki but with a visual aspect. I have my own wiki for managing notes and ideas and it's really useful because I tend to forget stuff and learn new things that help me refine old ideas. >>10096 Graph neural networks have been getting a lot of research attention recently but I haven't studied them much. Modeling the relations between words and sentences could be done but creating a dataset for that would be astronomical. I believe OpenCog uses a graph database and various algorithms to reason about the relations and information stored. It has been integrated into games to interact with NPCs in natural language. It might provide some inspiration.

>>10112 Alright, thanks for explaining that Anon. I haven't really any basis to relate to an 'L2 distance' jargon item other than calculating point vectors in 3D graphics. I have little additional information to understand how Pythagoras' insights would work in your context though, since I'm only used to Descartes' coordinates. What basis unit is used to 'measure' this so-called L2?

>>10115 The L2 distance aka Euclidean distance is the square rooted sum of (v1-v2)^2, essentially taking the difference of two vectors and finding the length of the resulting vector. In a 2D space this would be: sqrt(pow(x1-x2, 2) + pow(y1-y2, 2)) How the points are distributed in the latent space (which is like a 2D or 3D space except with hundreds of dimensions) depends on the model. In VAEs the points typically follow normal distributions, but the means of those distributions are all over the place, which leads to gaps and empty spaces no training data ever encodes into thus creating garbage when trying to decode anything from those spaces, which happens whenever trying to interpolate between two training samples. MMD-VAEs solve this problem by using a standard normal distribution as a target, so the points approximate a mean near 0 and a standard deviation close to 1 in each dimension. To illustrate the distance of these points in a MMD-VAE, most of the points will usually be close to zero, except in the dimensions where they have meaningful features. So if you have two latent vectors, each with a unique feature like this: v1 = {0, 1, 0, 0, 0} v2 = {0, 0, 0, 1, 0} Only two dimensions are different, so calculating the L2 distance is sqrt((1-0)^2 + (0-1)^2) which is the square root of 2. If you have another vector that shares these two features: v3 = {0, 1, 0, 1, 0} Its L2 distance to v1 and v2 is just 1.

>>10118 >In VAEs the points typically follow normal distributions, but the means of those distributions are all over the place, which leads to gaps and empty spaces no training data ever encodes into thus creating garbage when trying to decode anything from those spaces, which happens whenever trying to interpolate between two training samples. MMD-VAEs solve this problem by using a standard normal distribution as a target, so the points approximate a mean near 0 and a standard deviation close to 1 in each dimension. OK, that makes a bit more sense to me. It's pretty common in 3D CGI (particularly when converting from one coordinate system to another one) to need to normalize value ranges, typically into -1 < 0 < 1 in the target coordinate system. Your description reminded me somewhat of that issue. By the looks of it, when we do vector in 3D, I suppose you would describe it as 3 latent features. Put another way vecABC = sqrt(A^2 + B^2 + C^2) ?

>>10122 Yeah, latent features and latent variables are pretty much interchangeable. In some cases the variables become a more complicated encoding, like how numbers are encoded in binary. The latent feature might be something like a 3 but requires two latent variables that may or may not have an understandable meaning. This is called entanglement. Ideally we want all the variables to be features representing their own meaningful spectrum, so that the latent space is disentangled.

>>10126 I think I understand that somewhat. So the unit basis is simply whatever the numeric values are stored in the latent variables apparently as integer values (which seem to normalized to the range of 0 to 1 in the examples you showed)? >entangled You may have lost me there. I could conceptualize something like a velocity vector in a physics simulation though, where the object's own inertia has external forces acting on it it as well, say gravity, and some Tennessee Windage?

>>10111 This seems to be an interesting idea, but I suggest we try to find out, if someone already solved a specific problem e.g. finding similar sentences in English, where the model might be freely available. The we can focus on problems which are less general.

>>10132 *Then

>>10132 Wouldn't surprise me if so, but I've never even heard of the specific idea before I wrote it. One big advantage to my thinking if we did something like this ourselves is that a) as long as it was written in straight C++, it would run blazing fast, and b) we here would actually understand exactly how it works (we wrote it after all) and could quickly change it in whatever way we need to, and c) if it has been done already, then it seems to me it will have been done using the typical dependency-garbage/hell bloat of 'modern' Python-based AI work soy common today. So, by using straight C++ we would also have compiled binaries possibly 1/100th the size, and with no dependency hell whatsoever. All that's needed is a C++17 compiler.

>>10138 You're focusing on ow to solve it, my argument was to just find a solution. Do what you want to do, but for the general progess here, available solutions for common problems need to be used. Also, personally I'm not going to read piles of CPP code and abandon existing libraries, which has been tested by others, for it. So, to me the question is rather how to find common names for problems and then find existing solutions?

>>10143 I understand your view Anon. While that agenda is commendable (great for prototyping, even), to me, the real question is "How do we solve all of the issues required, each in their turn, to create life-sized, relatively low-cost, mobile, autonomous, gynoid companions"? Low-performance, low-energy consumption, low-cost computing hardware is very fundamental to that set of solutions. Only compiled languages have a solid hope of running effectively on such hardware. Whether you yourself choose to read said 'piles' of the code or not is mostly irrelevant to anyone but yourself Anon. I see compiled language libraries as absolutely vital to this entire quest, not simply some personal preference or tacked-on idea. It's why I decided to take the plunge to learn C++ in the very first place. I knew it would prove essential in the end to even make hobbyist humanoid robotics feasible at all.

>>10128 The values in the latent vector are real numbers and represent a point in the latent space. I used integers for brevity. In MMD-VAEs they approximate a normal distribution so they mostly fall into the range of -3 to 3 with a mean around 0. When a latent feature is entangled it just means that you need two or more specific variables at certain values to represent it. The latent feature is essentially tied up in those variables, so there's no way to change it without affecting other features, thus the features are said to be entangled with other features. When a latent feature becomes disentangled then only one dimension in the vector is needed to represent it and the point in the latent space can be adjusted along that dimension without affecting anything else.

>>10154 >When a latent feature is entangled it just means that you need two or more specific variables at certain values to represent it I see. That should be as straightforward and simple to implement (data-storage wise) as just adding a std::vector of doubles into the container's hierarchy to accommodate the information needed for it. >The latent feature is essentially tied up in those variables, so there's no way to change it without affecting other features, thus the features are said to be entangled with other features. Hmm. I can't say I can discern exactly what you're saying there just yet Anon, but there are many obvious corollaries in the real world, I'd think. A simple issue like not being able to get out a magazine buried under a stack of magazines might be one loose analogy -- you'd have to move the other magazines off first. If the basic idea you're referring to is as simple as this kind of notion, then there's one thing from my experience I might suggest as a possible approach for dealing with such issues w/o being too disruptive. Namely, when a transform of an element (a 3D skeleton's joint in XYZ coordinates, say) from one coordinate system into another one is needed, it's very commonplace to 'pad' that transform item inside another one, the second one being associated with the outer coordinates (character-space vs. world-space). The introduced element pads the contained item from any external transforms that may be needed (translating an entire skeleton to a new location in world-space, say) w/o disrupting the state of the contained joint itself. Perhaps entangled features can be isolated in some vaguely similar fashion? Back to the magazine analogy, if you first padded the magazine with a "magazinestack-space" padding, then you could open it, read it, write on it with a marker, or change it in other ways without ever removing it from the stack -- from the perspective of the stack itself. And such a padding works both ways of course. You can lift the entire stack and move it (to another bookshelf in the room, say), and simultaneously without interrupting the contained magazine's reader who is currently adding highlights into it. Effin magnets, man. :^)

>>6374 Anon, i am still waiting for your release. Can you update us about how it has been going? I am genuinely interested in your work.

There's a new project coming along called NovelAI that might be worth keeping an eye on. They're in their infancy now but plan on using GPT-Neo.

Pretrained Language Models for Text Generation: A Survey: https://arxiv.org/abs/2105.10311 >Text generation has become one of the most important yet challenging tasks in natural language processing (NLP). The resurgence of deep learning has greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs). In this paper, we present an overview of the major advances achieved in the topic of PLMs for text generation. As the preliminaries, we present the general task definition and briefly describe the mainstream architectures of PLMs for text generation. As the core content, we discuss how to adapt existing >PLMs to model different input data and satisfy special properties in the generated text. We further summarize several important fine-tuning strategies for text generation. Finally, we present several future directions and conclude this paper. Our survey aims to provide text generation researchers a synthesis and pointer to related research. >>10395 Thanks. It's trained on novels, and the output is more like that. It's open for alpha testing, or it was when this video was made: https://youtu.be/l_bde3A3R5A

>>10613 Thanks Anon, such a survey is always useful for beginners such as myself. Always a good idea to archive a copies of papers for us here on /robowaifu/ too.

>>10392 Sadly I don't have enough computing power to test my ideas let alone research and explore them. If I save up $5000 for two 24 GB RTX 3090s, it should be sufficient to continue my work. For now my priority is making money: >>11470

I want to bring the discussion on AI back to where we are, instead of getting lost in philosophy and emulating neurons. Here some topics which might be worth looking into: - natural language interfaces e.g. https://github.com/everling/PRECISE - entity linking model: https://youtu.be/8u57WSXVpmw - named entity recognition - triplet extraction - openNLP I started to go through some papers and I'm making notes by using the syntax for plantUML which converts text to a diagram. Sadly some useful programs I found are not available as free software, or in languages I haven't learned yet and haven't been picked up by anyone.

>>11471 I understand anon, I wish I could help you but I can't do anything other than wishing you good luck. I am looking forward to hear more from you. Best of luck.

Which chatbots are the best at emulating silence? Most chatbots will reply after _every_ input you give them. But in actual human conversation, there are plenty of times where someone is just asking something rhetorically, to voice something into the room, just to be coy, or they're being cocky and should be greeted with silence or ignored.

>>12089 More generally...what am I optimizing _for_? For example, I could make a chatbot that uses a neural network that would optimize for when it gets individuals to have the longest conversations with it, but that's not the kind of waifubot I'd want. I'd want a waifubot that would encourage me to shut up from time to time, and would have long pauses and periods of silence and introspection. What the hell optimizes for that kind of behavior?

>>6102 1. Why did you move away from SentenceMIM? 2. When are you releasing the codebase in public?

I'll be the dumbest block on the wall and ask, is there any replacement for Mitsuku? Even if it wasn't really a well done AI, I kinda miss her, and I massively despise what she has been replaced with.

Not a software guy at all so this is probably a dumb question. If I knew of a particular person whose personality I would want to emulate would it be possible to have them talk with a chatbot in order to train it to have that personality?

I wanted feed a character's lines into this https://github.com/thewaifuproject/waifuchat, but their speech patterns change when addressing different people. However I can emulate the character's speech patterns pretty well and I was wondering if it would be possible to just train a chatbot that way.

There is GAN (Generative Adverserial Network) and web app code mainly in Python and JavaScript: https://github.com/search?q=sexbot https://github.com/search?q=ai+waifu https://github.com/kerrizor/sexbot https://github.com/sxmnc/sexbot

Crosslink: >>19466 about Wikidata Tools Crosslink: >>13826 and >>13955 about Programming Languages I might still make a new Chatbot General at some point, which is explicitly not focused on Deep Learning. Still working on my design of the body, though. Also, I just rediscovered this one: >>77

Crosslink: >>18306