I was playing around with the software on my local AI box. I know there's a ton of hardware stuff to do and a backlog of write up for my website. But, I thought let's have a play.
I had looked at Gemini Notebooks or Google NotebookLM as it once was called. I wanted to be able to upload versions of my CV so that the chat "area" would have context across all my files as well as the chat itself. ChatGPT Projects is similar. I love Gemini but as far as I can tell the only way to opt into privacy is by deleting all my chats and I like to keep my chats for future reference not only that but also because of the extended context across chats that having saved chats gives me.
I tested RAG on LM Studio and copied a test file to RAG's working directory (the view "Working Directory" option keeps disappearing so it may have said "Current Directory"). Whatever I did, mainly tweaking my prompt, several models I tried wouldn't read the test file I put in there. So I turned RAG option to off and uploaded a file to the chat window and in response LM Studio turned RAG back on. Oh, I was approaching it backwards. But, giving myself the ability to add files via the chat window using RAG didn't give me exactly what I wanted but at least I understood the mechanics of how that was supposed to work now.
I was thinking about installing AnythingLLM but left it for another day because I wanted to try to get an agent running for some other things I had in mind. I knew that having an agent physically trawl through my files was like using a sledge hammer to crack open a peanut. Especially, when AnythingLLM could probably manage my CV versions more appropriately, in a way similar to Gemini Notebooks. But hey, let's give the agent thing a go. I stumbled across Bionic when I was reading about LMLink. Yes really, I'd still not got around to reading about it on their website since installing it. Again, just like LMLink the installation was super easy, no instructions required. I downloaded the Bionic AppImage and ran it, then I connected to my local model and then checked to see if the agent could see my CV. It saw my CV by running some scripts. Later, I added the Downloads folder to the Projects option in Bionic and confirmed that the agent could read another test file. That confirmation was easy enough, I just copied another test file and asked the agent about it. The agent was able to talk about the file as soon as it was copied, without the agent running any terminal commands this time around.
At that point, I had already posted this on LinkedIn with a couple of screenshots:
I wish I had more time to investigate these types of things, a few minutes more work/fun and I'm once again left speechless. The ease, the power, the implications. Privacy-first, local agentic automation, job done?
#LMStudio #LocalAI #HomeLab #Linux #AIAgents
Did you notice one nuance that I didn't point out in the above LinkedIn post? The agent used its initiative. It opened the file to see what was in it. I didn't ask it to do that, but it's exactly what I wanted it to do. It saw beyond my prompt's instruction and asked: "what is he really asking?" If this were a helpdesk ticket would it have been closed with the response: "Yeah, your CV file is there"? With the argument that the ticket had been closed because they'd done what was asked of them. I'm just trying to highlight here the sometimes literal, robotic nature of traditional workflows versus the intuitive leap my LLM agent just made.
Let's circle back round and ask what did I mean exactly by my LinkedIn post? The baseline for IT troubleshooting is perhaps shifting. We are moving away from passive generative AI and moving toward LLM-backed proactive, autonomous digital team-mates. If an offline 27B model can demonstrate this level of investigative curiosity out of the box, and we've reached a level of maturity where an agent can be spun up this easily, and models with this capability, that would previously have only been able to be run in the cloud can now be run locally, then I can't help wonder: what now?
But why investigate all this? To expand on one of my LinkedIn post's points, maybe I don't want a human reviewer at a cloud AI provider knowing that I had a paper round once 😄 Or, future models being trained on my private information, would you? 😿
