Hi Rainer,

Thanks for the detailed comment. I’m glad to know that you’ve had a similar experience.

Yes, there are lots of ways to make local LLMs smarter and efficient, but most of those methods are complicated and intended for power users. My goal with this article is to give an average user with a low to mid end PC a realistic picture of what they should expect from a vanilla local LLM setup.

After all, the alternative, a cloud LLM like ChatGPT, just works out of the box. So, for most users, setting up a complex local LLM after applying all those tweaks that you mentioned to still get mixed results is not worth the time and effort.

However, I’d be really interested to hear from you if you achieve a major breakthrough that changes what we’ve both experienced so far.

Cheers!

I’m myself playing with local AI on an Intel Ultra 225H / 32 GB as well as an AMD AI 9 HX / 64 GB. So far my experiences are mixed as well. Both systems are currently in CPU only mode, running Ollama with Alpaca as front end.

Next, I like to look into utilizing their GPUs via Mesa / Vulkan stacks. The other route shall be on Intel with the OpenVino project together with an Docker image and with Ubuntu & Ollama_OV, which suppose to have direct GPU access and acceleration. They use a variety of specially tuned model for using Intel GPU and NPU.

One outlier I found with Jan, which comes with build-in agent and tuned model. It has a range of special optimize models, which can easily been downloaded from the UI. It also supports access to a local Ollama server as well as online servers. – But I have only used it on older Ryzen 7 / 32 GB ram.