project_bookworm/Local_CPU_LLM_Ollama_Mistral.ipynb

LangChain Ollama Mistral CPU

• Ollama simplifies hosting LLMs and provides a simple API to interact with them.
• https://ollama.com/
• https://ollama.com/mistral
• LangChain is a platform for building and deploying language models.
  • https://python.langchain.com/
  • https://python.langchain.com/docs/integrations/llms/
• https://python.langchain.com/docs/integrations/llms/ollama
• Mistral
  • https://mistral.ai/
  • https://mistral.ai/technology/#models

This playbook is CPU-only. Ollama runs Mistral in a Docker container with the default API endpoint

curl http://localhost:11434/api/generate -d '{
  "model": "mistral",
  "prompt":"Why is the sky blue?"
}'

The LLM is hosted locally and needs no remote API. It's on-premises self-hosted and deployed on Linux (Ubuntu Server).

Basic use with in-memory cache

In [1]:

%%time
from langchain_community.llms import Ollama
from langchain.globals import set_llm_cache
from langchain.cache import InMemoryCache

set_llm_cache(InMemoryCache())

llm = Ollama(model="mistral")

llm.invoke("Why is the sky blue?")

Out [1]:

CPU times: user 748 ms, sys: 68.3 ms, total: 817 ms
Wall time: 43.6 s

" The reason why the sky appears blue during a clear day is due to a natural phenomenon called Rayleigh scattering. When sunlight, which is made up of different wavelengths or colors, collides with molecules and particles in Earth's atmosphere, shorter wavelengths like blue get scattered more easily than longer wavelengths like red. As a result, the sky appears blue during the day because our eyes are more sensitive to blue light and we see it more readily when looking at the sky.\n\nHowever, as the sun sets, the angle of the sunlight reaching Earth changes, causing the longer wavelengths like red and orange to be scattered in greater quantities. This is why we see beautiful red, pink, and orange hues during sunrise and sunset instead of blue."

Speed-up with in-mem cache

In [2]:

%%time
llm.invoke("Why is the sky blue?")

Out [2]:

CPU times: user 276 µs, sys: 0 ns, total: 276 µs
Wall time: 279 µs

" The reason why the sky appears blue during a clear day is due to a natural phenomenon called Rayleigh scattering. When sunlight, which is made up of different wavelengths or colors, collides with molecules and particles in Earth's atmosphere, shorter wavelengths like blue get scattered more easily than longer wavelengths like red. As a result, the sky appears blue during the day because our eyes are more sensitive to blue light and we see it more readily when looking at the sky.\n\nHowever, as the sun sets, the angle of the sunlight reaching Earth changes, causing the longer wavelengths like red and orange to be scattered in greater quantities. This is why we see beautiful red, pink, and orange hues during sunrise and sunset instead of blue."

In [ ]: