Disrupting the AI Scene With Open Source and Open Innovation

When I discovered OpenAI on the 23rd of December 2022, I became obsessed. I hadn't had this much fun coding since I started 40 years ago. After playing with "fine-tuning" for a month and miserably failing, I found a YouTube video by Dave Shapiro that explained how to create a Q&A chatbot using OpenAI's "embeddings" API.

Dave has since removed the video, but it explains how to use OpenAI's embeddings API and combine it with their chat/completion API to create a Q&A chatbot that knows "everything" about the problem domain. Tage's reaction best sums up my findings, as he told me one day.

OMG dad, this time you've really done it. I woke up shaking in the middle of the night because of enthusiasm, and I couldn't even sleep, so I hade to take a walk for 5 kilometres outside in the middle of the night just to calm down.

How a Q&A Chatbot Works

To understand a Q&A chatbot based upon ChatGPT, you can go to ChatGPT and find any article and copy and paste it into the prompt as follows;

Answer the following QUESTION given the specificed CONTEXT;

QUESTION; What is the meaning of life?

CONTEXT; [ ... content of some article explaining the meaning of life ... ]

What ChatGPT will do, is answer any question you might have while using the article's content as its "single source of truth." What we do, and everybody else that delivers ChatGPT chatbots, is to create a database of "context data" that might be created by uploading documents and/or scraping websites. When the user asks a question, we'll use OpenAI's embeddings API to create a "vector" of the question.

This vector is then used to perform a similarity search through our database, calculating "dot products," which becomes the "distance" between the question and the snippets from our context database. Then we order each result from our context database by this distance, take the first 4 to 5 context snippets, and send these to OpenAI as the question's "context." Before you ask, yes ...

The whole process is simply "automated prompt engineering"...

OpenAI's embeddings API again, is incredibly smart at finding similarities between questions and context data, allowing it to find the relevant data given any question you've got related context for in your database.

The Problem With AI-Based Semantic Search

The above "dot product" is the problem. To understand why, realize you have to perform a "table scan" through your entire database, extract the embedding vector for each record, and calculate the dot product for each result originating from this process. This is a CPU-intensive job, and for a context database with 2,500 records, this would take 30 to 50 seconds in our Kubernetes cluster for our systems. This is the reason why we've not been able to deliver chatbots with more than 2,500 "snippets" previously.

However, today we fixed this problem, and during the weekend, we'll hopefully be able to deploy a solution that, at least in theory, allows for 10,000+ snippets, possibly even more, while returning "context data" in 0.02 seconds instead of 5 minutes.

Ever since I realized the above process was sub-optimal, I've been periodically doing a Google search for the "sqlite vector plugin." To understand the importance of this, realize that since OpenAI went viral, at least half a dozen startups have been created with the intention to create a "vector-based database." I know about at least one such database that got 30 million dollars in VC funds earlier this year. To explain why to realize the truth of the following statement...

Whoever solves the vector database problem is destined to control the AI space and hence the world.

What's at Stake

I personally believe that AI, the way it's progressed the last year, is probably the most important thing to have occurred on Earth for the last 5 million years, seriously! When people are comparing AI to the internet, heavier than ear flight, the computer for that matter, I tend to laugh and reply with...

AI is the most significant event to have happened since we crawled down from the trees. For 5 million years we've been the smartest species on Earth, that era ends in 2023!

So, basically, if somebody can "control" innovation in this space, they basically own the future of mankind. The amount of power a company having such control could yield would inevitably make all previous power constructs become the equivalent of "a child's playground" in comparison. We cannot let this happen simply because if somebody "controls" the AI space, that person would be able to yield "God-like powers" over the rest of us!

Giving "the people" control is CRUCIAL for the above reasons!

The Solution

Even Google has publicly admitted it can't keep up with open-source AI innovation. This is because of open-source projects such as Hugging Face. However, there was always one tiny little piece missing; "vector-based database systems." A superior vector-based database can easily index millions, and even billions of database records, allowing even 14-year-old kids "to build their own Google."

As I searched for the "sqlite vector plugin," I didn't find any results before a couple of weeks ago. Two weeks ago, I found Alex's SQLite VSS plugin for SQLite. The library was an amazing piece of engineering from an "idea perspective." However, as I started playing around with it, I realized it was ipso facto like "Titanic." Beautiful and amazing, but destined to leak water and sink to the bottom of the ocean because of what we software engineers refer to as "memory leaks."

I spent a lot of time fixing the library, to the point where it could be argued: "I melted down the Titanic, cast a new boat out of its original material, ending up with a 'Battle Ship Cruiser' with perfect memory management." To put that into perspective, below you can find my pull request for Alex's amazing library.

• My pull request to SQLite VSS

It's a monster of a pull request, and it fixes some roughly 10 to 20 memory leaks. I ran the entire code through ChatGPT before submitting my PR to Alex, and even ChatGPT couldn't find any memory leaks in it and claimed every single function, class, and the construct was "perfect according to how to correctly create an SQLite database plugin." Before my PR, the thing would consume 1GB of memory in a test deployment in our Kubernetes cluster. Afterward, it consumed half and don't grow to infinity and beyond. The leaks would make the library useless for all practical concerns. Every single leak is now fixed — effectively making SQLite a better vector database than all vector database systems out there!

Once we start using the above plugin in our Kubernetes clusters, we can basically increase "model size" from 2,000 context snippets to (probably) 100,000+ context snippets for extreme cases. This allows us to scrape websites with 10,000+ pages and create Q&A chatbots out of them. Previously our maximum was roughly 500 web pages.

Further down the road, we might even be able to continue modifying the library to the point where we can, in theory, index billions of pages using this technology. This would effectively allow us "to build Google 2.0" and create chatbots with knowledge "the size of the Himalayas."

The Future Is Yours!

Magic, our platform is 100% open source. Anything else would be unfair, considering what's at stake. Within a week, we will deploy these changes into our technology for all to use, allowing you to query databases with 10,000+ records in some 0.02 seconds to extract context data.

This allows us to create chatbots for things such as CouchBase's documentation, Microsoft's website, and DEV.to, for that matter, etc., etc., etc. This allows us to deliver technology, allowing you to outperform Google and Microsoft on search. Combined with the innovation that's happening in the GPT space, with initiatives such as Hugging Face, etc. — the inevitable result becomes as follows...

The future is YOURS! Me and thousands of other open source software developers will make sure of it!

Credits

• Alex, for having built an amazing semantic search plugin for SQLite! BRAVO Alex!
• Dave Shapiro for constantly giving me good ideas for how to approach the AI space
• Facebook research for having open source licensed an amazing vector-based indexing library
• Me for pulling all the strings together, ending up with a usable product by "melting down the Titanic and creating a Battle Ship Cruiser" out of Alex's original work