Today, I want to talk about small generative AIs — the tiny powerhouses that are making a big impact in the world of software development.
Now, I know what you might be thinking, “Isn’t bigger always better?” Well, not exactly. When it comes to AI models, bigger usually means more accurate and impressive results. But (there is always a but) the amount of data also carries a few limitations and risks.
We all have heard about these things called large language models (LLMs) — for example, OpenAI’s ChatGPT, Google’s LaMDA and BARD, Hugging Face’s HuggingChat, Meta’s LLaMA, and Amazon’s Titan Text. It’s amazing just how fast companies got on the LLM bandwagon after the smashing success of OpenAI.
These models are impressive (at least those that are publicly available), and they couldn’t exist without millions of gigabytes of data. Not for nothing, AI products are using the number of parameters (in the billions) as a benchmark, like console manufacturers used to talk about bits in the ’90s and early 2000s.
But that size comes at a cost. The server architecture for this kind of product is massive, and that’s just the tip of the iceberg: data gathering and cleaning, engineering costs, time investment, and so much more. This isn’t the kind of product that you homebrew in your garage over a weekend.
And that’s not even taking into account some of the ethical issues related to data gathering, data mining, and other topics that have landed OpenAI in hot water in the last few months, like Italy’s ban. And trust me, we’ll get to that, but before looking at an even bigger issue with these models, the best way to explain it is by talking about a well-known classical figure: good old Socrates.
Socrates’ Woes: Episteme vs. Doxa
Socrates is an enigmatic but significant figure of the classical period; almost everything we know about him is told through the eyes of his alleged disciple, Plato. To some philosophers, Socrates wasn’t even a real person but rather a literary device to explore Plato’s philosophy through dialogues.
Leaving that debate aside, Socrates was most famous for being an obstinate man who drilled answers out of everyone who dared to have an opinion. The Socratic method is a painful process in which, through the act of questioning, alleged knowledge is dismantled, giving way to the “truth” behind the subject at hand.
He was such a zealot in the pursuit of truth that he ended up making political enemies who cost him his life. Accused of corrupting the young, he was unjustly sentenced to death. Even when he had the opportunity to escape, he decided to accept his fate since it was his responsibility as a citizen of Athens.
In other words, Socrates was against doxa, the beliefs and opinions that are held by a group of people. Instead, he tirelessly sought episteme, which refers to “true” knowledge. Back in those days, the way we understood truth was very different, but nowadays, episteme would be the equivalent of knowledge based on facts and evidence.
Here is the sad truth: when we rely too heavily on large language models, we risk prioritizing doxa over epistemology. The model might be able to generate a convincing argument or a plausible explanation, but that doesn’t necessarily make it true.
The reason is twofold. Firstly, language models don’t understand concepts the way we do (in fact, they don’t understand them at all). They are really good at understanding which words should go together but not why they should go together. That’s what leads to hallucinations when the AI makes up facts because statistically, it makes sense that those words go together.
The second reason is that most of the data out there in the world is actually doxa in the Greek sense and not episteme. The same data with which the models are built. Now, before we talk about the implications of the statement, I want to make something extremely clear: doxa isn’t the same as lies or falsehoods. Rather it would be more akin to “half-truth.” For example, most people know that vinegar prevents food from spoiling, but ask them why that happens, and odds are most people can’t explain it.
To put it in simpler terms, just because a model can talk the talk doesn’t mean it can walk the walk. It’s like if I asked my buddy who’s always spewing conspiracy theories about the moon landing, to help me with my history homework. Sure, he might be able to come up with some wild theories about what “really” happened, but that doesn’t mean he knows anything about the actual historical facts.
And that’s the real danger of relying too heavily on LLMs. We might be tempted to trust the model’s output without questioning whether it’s actually accurate or based on solid evidence. It’s like letting a Magic 8 Ball make all your business decisions — sure, you might get lucky once in a while, but answer me this: Would you fly on a plane built entirely out of ChatGPT instructions?
So, what’s the solution? As with most things in life, it’s all about balance. Large language models have their place, but we need to be careful not to let them take over completely. We should always be questioning the model’s output, checking the sources, and verifying the facts. And if we’re not sure about something, we should be willing to admit it and seek out additional information.
Now, there is another alternative. How about, instead of relying on the LLM directly, we just use it as a foundation, as a starting point to build a specialized AI with carefully selected data? Just like us human beings, we start our journey in education learning a little bit about everything, and as we grow old, we focus on specific areas.
The scope of these focused models is rather small in comparison, but at the same time, their output on specific subjects is a lot more reliable. It’s like fine-tuning stable diffusion with Velasquez paintings. From that moment on, this particular model is going to have a harder time making every kind of image, conceivable but it’s going to become really good at painting like Velasquez.
Small Generative AIs vs. Large Generative AIs: Pros and Cons
Alright, let’s talk about small generative AIs versus large generative AIs. It’s like comparing a pocket knife to a machete; both have their uses, but one may be more suitable than the other depending on the situation.
Small generative AIs can be like your trusty pocket knife: compact, efficient, and reliable for certain tasks. They are great for generating short snippets of text or images quickly and accurately. You don’t need massive computing power to get things done with them. Plus, they can be easily integrated into existing systems without causing much disruption.
On the flip side, sometimes you need that big bad machete-like AI to tackle tough issues. Large-scale models have access to vast amounts of data and processing power, which allows them to generate more complex content with greater accuracy. They might not be as easy to handle or integrate as smaller ones, but they sure pack a punch when it comes down to crunch time.
I’ve worked with both kinds of models in my time in software development, and I’ve found that choosing between small and large-scale AI really depends on what you want out of your AI system.
Sometimes all you need is a quick decision-making tool that generates simple code fragments or strings based on some predefined patterns — something relatively mundane that doesn’t require heavy computational resources — so then small models would serve better just fine.
Other times, though (like if you’re trying to make deep fakes), the big guns must come out. Larger models trained over huge amounts of data will help get us closer to achieving what we ultimately strive for: building AGI’s abilities, such as creating complete virtual worlds indistinguishable from reality.
There’s also considerations around training costs here. Running training involving vast datasets for longer periods requires immense resources and thus wastes unnecessary energy. For small problems, large models are like nuking a country just to kill a fly. Hence, it’s a tool that should be used wisely.
The Future of Small Generative AIs
First off, let me just say that I’m super stoked for what’s coming down the pipeline. As someone who loves tinkering with technology and pushing boundaries, I can’t wait to see how far we’ll take these mini marvels.
My gut feeling is that as processing power gets even faster and cheaper than it already is, we’ll see some truly astonishing things happening with small generative AIs. Think about it: right now, they’re impressing us left and right by creating stunning pieces of art or music within certain parameters set by humans (like color schemes or chord structures). But what if they had more freedom? What if they were given the creative reins?
I know some people might be worried about this idea. After all, aren’t we supposed to be the ones making art? But hear me out: I think that working alongside small generative AIs could lead to some really interesting collaborations between humans and machines. Imagine being able to bounce ideas off your own personal AI assistant until you reach something truly unique and unexpected.
As AI are fine-tuned and new models are built out of foundational models, we will see higher quality outputs out of these AI tools. The future isn’t just about gathering more data and feeding it to the big fish boys; it’s also about becoming lean and more efficient, it’s about saving energy, it’s about creating the right tools for our needs.
There are so many practical applications for this technology beyond just generating pretty pictures or tunes. Small generative AIs could help automate repetitive coding tasks or even generate entire programs from scratch using data instead of pre-written rules.
Of course, there are also potential downsides to consider when thinking about turning over parts of our creativity (and livelihoods) to machines. It’s important for us as software developers to not only explore new technologies but also evaluate their ethical implications on society at large.
Case Studies: Small Generative AIs in Action
When it comes to small generative AIs, there’s always been a bit of skepticism within the software development community. Some developers believe that these tiny algorithms are too limited in scope and can’t possibly offer any meaningful impact on their work.
But as someone who has actually implemented several small generative AIs into my own projects, I’m here to tell you: size doesn’t matter! In fact, some of the most innovative and impactful solutions I’ve created have come from smaller AI designs.
Take for example one project I worked on recently. We were developing an app that required the ability to generate personalized recommendations for users based on their behavior within the platform. Originally we considered using a larger AI pre-trained model but after experimentation decided to go with a more compact design that was better suited for our needs — taking less resources while still delivering impressive results.
Some enthusiasts have found great success in building small language models by training it with GPT outputs, for example, GPT4all. (By the way, don’t do that, it goes against OpenAI’s terms of service.)
To ensure its accuracy remained high, we trained this algorithm extensively by testing different inputs and parameters until we achieved just what we needed: swift operation and accurate results delivered consistently across all devices!
While it may not have been as robust or comprehensive as many other models out there, our compact genAI proved itself invaluable through its reliable recommendations, leading us efficiently toward new feature ideas that improved user experience without sacrificing quality or scalability in development timeframes — keeping things leaner than ever before around our Agile team!
So if you’re skeptical about implementing small generative AIs in your next project because they seem insufficiently equipped, think again! These little guys pack quite a punch where it counts — optimized performance at a lower cost of processing power!
The Ethics of Small Generative AIs
Let’s face it — AI is like a genie that can grant all our wishes but comes with its own set of terms and conditions. As developers, we have a responsibility to create smart systems without compromising on privacy or ethics. And when it comes to small generative AIs, the question arises: Does size matter?
Nowadays, we see plenty of small AI-powered apps that generate images, music, and even email content. They are handy tools for amateur artists who want to creatively express themselves while still learning the craft. But as these apps become more prevalent and powerful, they could potentially raise concerns around intellectual property.
For instance, you wouldn’t want an app that claims ownership over your generated artwork or idea because it was theoretically programmed by the algorithm within their servers — talk about robotic infringement! Similarly, imagine using an app that generates music tracks based on user input, only to face copyright issues down the road since you unknowingly crossed lines with already copyrighted pieces!
These ethical dilemmas need addressing (and are already being addressed) before such applications become mainstream, limiting any negative future impact.
Another area where small generative AIs could run into trouble would be their potential susceptibility to exploitation by rogue elements or interest groups with less than pure intentions. Making sure there isn’t abuse in certain fields could help prevent automation replacing jobs too quickly (consider how many people felt uneasy when Uber introduced self-driving cars).
I’m not saying this whole business of smaller yet powerful creative ally applications doesn’t come with cut-throat potentialities; however, likewise, any new invention has parallel possibilities associated. We just need to balance out both sides properly while endeavoring toward responsible practices.
Conclusion: Why Size Does matter in Generative AIs
The bigger your dataset and model are in generative AI development, the better chances of generating high-quality outputs. It’s like baking a cake; the more ingredients you have to choose from and mix together, the higher chance of making a delectable dessert.
But don’t let that discourage those just starting out in their AI journey. You don’t need to go big or go home right away. As they say, “small but mighty” — small generative AIs still have plenty of potential for creative outputs and learning opportunities.
And if there is anything we developers love doing, it’s sharing our code. Even small models can join forces with other models through federated learning techniques, which allow multiple algorithms across a network(s) to train collaboratively on each other without being merged into one cogent model by incorporating cross-device privacy protection mechanisms such as differential privacy (pun intended).
In any case — and whatever size model(s) you start with — remember these words of caution: always check your generated output before using it as gospel truth. There will be times when an AI creates bizarre or problematic results; always trust but verify! So go forth, experimenters! Try building different sizes of generative AI models, because why not? Play around with training datasets. See how much creativity lies beneath their layers upon layers of hyperparameters.
Just remember: “Size matters not … Look at me … Judge me by my size, do you?” — Yoda
