LangChain, Mistral, Ollama, local LLM CPU-hosted
Marius Ciepluch
parent
8b5a88d336
commit
ad0acff4ac
|
|
@ -0,0 +1,169 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c3124e6f5d96e05f",
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"source": [
|
||||
"# LangChain Ollama Mistral CPU\n",
|
||||
"\n",
|
||||
"* Ollama simplifies hosting LLMs and provides a simple API to interact with them.\n",
|
||||
" * https://ollama.com/ \n",
|
||||
" * https://ollama.com/mistral\n",
|
||||
"* LangChain is a platform for building and deploying language models.\n",
|
||||
" * https://python.langchain.com/ \n",
|
||||
" * https://python.langchain.com/docs/integrations/llms/\n",
|
||||
"* https://python.langchain.com/docs/integrations/llms/ollama \n",
|
||||
"* Mistral \n",
|
||||
" * https://mistral.ai/\n",
|
||||
" * https://mistral.ai/technology/#models\n",
|
||||
" \n",
|
||||
"\n",
|
||||
"This playbook is CPU-only. \n",
|
||||
"Ollama runs Mistral in a Docker container with the default API endpoint\n",
|
||||
"\n",
|
||||
"```\n",
|
||||
"curl http://localhost:11434/api/generate -d '{\n",
|
||||
" \"model\": \"mistral\",\n",
|
||||
" \"prompt\":\"Why is the sky blue?\"\n",
|
||||
"}'\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"The LLM is hosted locally and needs no remote API.\n",
|
||||
"It's on-premises self-hosted and deployed on Linux (Ubuntu Server).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d566a43e6033d8d4",
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"source": [
|
||||
"## Basic use with in-memory cache"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "initial_id",
|
||||
"metadata": {
|
||||
"ExecuteTime": {
|
||||
"end_time": "2024-03-21T08:38:20.697371Z",
|
||||
"start_time": "2024-03-21T08:37:39.510775Z"
|
||||
},
|
||||
"collapsed": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"CPU times: user 30.5 ms, sys: 4.16 ms, total: 34.6 ms\n",
|
||||
"Wall time: 41.2 s\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"\" The reason why the sky appears blue during a clear day is due to a particular type of scattering called Rayleigh scattering. When the sun's rays enter Earth's atmosphere, they are made up of various wavelengths or colors of light. Short-wavelength light, such as blue and violet, gets scattered more easily because they have smaller wavelengths. This scattered blue light is what we see when we look at the sky. However, our eyes are more sensitive to blue light and less sensitive to violet light, making the sky appear more blue than violet. At sunrise and sunset, the sky can take on a variety of colors because the angles of the sunlight entering the atmosphere cause different types of scattering to occur.\""
|
||||
]
|
||||
},
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"from langchain_community.llms import Ollama\n",
|
||||
"from langchain.globals import set_llm_cache\n",
|
||||
"from langchain.cache import InMemoryCache\n",
|
||||
"\n",
|
||||
"set_llm_cache(InMemoryCache())\n",
|
||||
"\n",
|
||||
"llm = Ollama(model=\"mistral\")\n",
|
||||
"\n",
|
||||
"llm.invoke(\"Why is the sky blue?\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "609a24f85a932ee6",
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"source": [
|
||||
"### Speed-up with in-mem cache"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "33a9510c0fbbfce2",
|
||||
"metadata": {
|
||||
"ExecuteTime": {
|
||||
"end_time": "2024-03-21T08:47:29.318321Z",
|
||||
"start_time": "2024-03-21T08:47:29.313455Z"
|
||||
},
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"CPU times: user 0 ns, sys: 276 µs, total: 276 µs\n",
|
||||
"Wall time: 282 µs\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"\" The reason why the sky appears blue during a clear day is due to a particular type of scattering called Rayleigh scattering. When the sun's rays enter Earth's atmosphere, they are made up of various wavelengths or colors of light. Short-wavelength light, such as blue and violet, gets scattered more easily because they have smaller wavelengths. This scattered blue light is what we see when we look at the sky. However, our eyes are more sensitive to blue light and less sensitive to violet light, making the sky appear more blue than violet. At sunrise and sunset, the sky can take on a variety of colors because the angles of the sunlight entering the atmosphere cause different types of scattering to occur.\""
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"%%time\n",
|
||||
"llm.invoke(\"Why is the sky blue?\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"id": "aec810d53214e101",
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 2
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython2",
|
||||
"version": "2.7.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
|
|
@ -7,4 +7,5 @@ wikipedia==1.4.0
|
|||
pypdf==4.0.2
|
||||
langchain_openai==0.0.8
|
||||
langchain_experimental==0.0.53
|
||||
langchainhub==0.1.14
|
||||
langchainhub==0.1.14
|
||||
ipywidgets==8.1.2
|
||||
|
|
|
|||
Loading…
Reference in New Issue