bits on data

I've had a bunch of conversations with family, friends, and folks who avoid tech until it's necessary about the latest wave of artificial intelligence (AI) craze. A lot of these folks are curious to use AI, but are concerned about privacy or receiving biased views and misinformation. Many technologists have limited this exposure using locally running AI models which avoids sharing personal information to platforms and enables you to choose a model that has more public validation of the data it trains on and therefore why it may have a stance on a particular topic.

I realized how most blog posts on how to run local AI are written by technologists for technologists and everyone else is beholden to ChatGPT and the like. I myself have been quite excited about the potential for AI, but as a socialist who has worked in the Big Data industry and seen how information is being used against the average citizen, I want to help everyone in learning how to take ownership of your information without avoiding participating in this fun and valuable tech.

Before we even jump into that, I just want to provide a little context about the main things you need to know about AI since a tool called ChatGPT made its debut in November 2022. Rather than share the massive amounts of activity happening here, I'll cover the most interesting larger developments mixed with what you'll need to know for the tutorial below.

Generative AI in a nutshell

First, some clarity on the term “AI”... What is commonly referred to in the media in recent years as “AI” actually refers to a subset (i.e. Generative AI) of a subset (i.e. Deep Learning) of the larger field of Artificial Intelligence. Artificial intelligence prior to the year 2022 was interested in statistical models that were easier to create and computationally “small” enough to run on mobile phone chips. This enables features like autocomplete, voice-to-text, facial recognition to log into you phone or run snapchat filters to add a mustache, etc... Before those applications, artificial intelligence was more often seen in video games when a computer would generate a series of actions for another character or adversary during game play.

ChatGPT is the flagship application of the modern wave of artificial intelligence products built by a closed company ironically named OpenAI. They set a record for most downloaded application and set off the interest and investments into a lot of products that internally rely on their services. For most people ChatGPT is a list of text conversations you have in what looks like an instant messenger window, but the words replying back to you are generated by a computer.

Computers generate this text by running programs that copy as much text off the internet and as many books, magazines, YouTube transcripts, or anything a company can freely get public access to, and build a statistical prediction model called Large Language Models. When we say model, think less Heidi Klum and more like a physics model that predicts the weather. LLMs encode the patterns of speech given a context. For example, if I say “It was the best of times, it was the ____ of times” you likely can guess based on your own model and understanding of language (i.e. your intuition) that the missing word is “worst” even if you've never read Charles Dickens' A Tale of Two Cities. LLMs trained on different data sets will have different strengths, weaknesses, and biases when faced with various tasks. In a similar way our brain uses mental models to remember or recall information, you can think of LLMs as just stochastic computer-based intuition models. It should also be noted that human mental models are a shallow analogy and despite the propaganda that AI can think the same way humans do, this is utter rubbish and anthropomorphizing AI has the potential to cause a lot of harm.

One of the early open-source families of LLMs that challenged the superior performance of early ChatGPT models is called Llama. Llama brought the first equally powerful open LLM to center stage, making it possible for people to run LLMs that could even outperform ChatGPT on their own computers privately. The release of Llama was a strategic power play from Meta (aka Facebook) and by opening large scale models to the public, Meta could reap the benefits of the adoption and rapid development in open ecosystems and tooling, such as Ollama.

Despite Ollama's similar name to the Llama LLMs it is not an LLM itself, but a platform to manage and build off of any open LLM. These platforms are also called LLM providers. Ollama's name came from its initial use of the llama.cpp to customize (aka fine-tuning) Llama's LLM to new tasks. As more open LLMs were created, Ollama built its own runtime and became a platform for any open source model. Although this tool is very helpful for those in the industry, it still requires anyone with less technical experience to type all of their work in a command line and wasn't the experience you would get from ChatGPT. This is where tools like Anything LLM become handy. AnythingLLM is a versatile application that can run on your own laptop and provide the similar user interface and features that you might find on ChatGPT.

Now that I've given you all of that information, let's cover an installation of Ollama to download (aka pull) either Llama or Deepseek LLMs down that are small enough to run on the average computer. The Deepseek model should run on most modern laptops and will give a generally better output than the smaller LLama model I'm providing as backup. Once that is complete we will install Anything LLM to give you a nice sleek chat interface similar to ChatGPT which can recall all your previous chats. I've added videos to help with installation visuals as I always think just having a human explain stuff helps me with this stuff as well.

Requirements

You should have a computer that has at least a dualcore or ideally quadcore CPU and at least 4GB (ideally 8GB) of RAM. You will also need about 5GB of storage space depending on the LLM you use (more below).

Tutorial

1. Download and install Ollama and open the terminal.
2. If you have an older laptop or have less than 4GB ram, type: ollama run llama3.2 to run the LLaMa LLM (2.0GB of disk space), or ollama run deepseek-r1 to run Deepseek LLM (4.7GB of disk space).
3. Download and install Anything LLM Desktop.
4. Open Anything LLM and connect to Ollama

Once you have that set up, you can then you can use AnythingLLM application similar to ChatGPT. You can even import your local documents to the local instance of AnythingLLM. Watch this video for a nice and recent AnythingLLM overview.

If a lot of this terminology still felt confusing it has more to do with the intentional complexity and lack of user design to avoid too much democratization of running your own AI to keep you dependent on a paid service that extracts and reveals your conversations. I'll do my best to keep this article up-to-date as new tools and models come along.