From 5bee42f9fca9944791cfdc2425dc692d9d873caf Mon Sep 17 00:00:00 2001 From: cmungall Date: Mon, 15 Jan 2024 11:02:57 -0800 Subject: [PATCH 1/7] Added documentation on using other models via litellm. See also #15 --- README.md | 44 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 06d3ea4..d812554 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,6 @@ export OPENAI_API_KEY= You initially start with an empty database. You can load whatever you like into this database! Any JSON, YAML, or CSV is accepted. - CurateGPT comes with *wrappers* for some existing local and remote sources, including ontologies. The [Makefile](Makefile) contains some examples of how to load these. You can load any ontology using the `ont-` target, e.g.: @@ -128,6 +127,49 @@ The following are also supported: - See [notebooks](notebooks) for examples. +## Selecting models + +Currently this tool works best with the OpenAI gpt-4 model (for instruction tasks) and OpenAI `ada-text-embedding-002` for embedding. + +Curate-GPT is layered on top of [simonw/llm](https://github.com/simonw/llm) which has a plugin +architecture for using alternative models. In theory you can use any of these plugins. + +Additionally, you can set up an openai-emulating proxy using [litellm](https://github.com/BerriAI/litellm/). + +Let's say you want to run mixtral locally using ollama. You start up ollama: + +``` +ollama run mixtral +``` + +Then start up litellm: + +``` +litellm -m ollama/mixtral +``` + +Next edit your `extra-openai-models.yaml` as detailed in the llm docs: + +``` +- model_name: ollama/mixtral + model_id: litellm-mixtral + api_base: "http://0.0.0.0:8000" +``` + +You can now use this: + +```yaml +curategpt ask -m litellm-mixtral -c ont_cl "What neurotransmitter is released by the hippocampus?" +``` + +But be warned that many of the prompts in curategpt were engineered +against openai models, and they may give suboptimal results or fail +entirely on other models. As an example, `ask` seems to work quite +well with mixtral, but `complete` works horribly. We haven't yet +investigated if the issue is the model or our prompts or the overall +approach. + +Welcome to the world of AI engineering! ## Using the command line From ab810038d378829ba4b30e91c03f1f8f4cad1f3f Mon Sep 17 00:00:00 2001 From: Justin Reese Date: Wed, 17 Jan 2024 09:55:39 -0500 Subject: [PATCH 2/7] Change GH actions python version from 3.8 to 3.9 to agree with pyproject.toml requirement --- .github/workflows/qc.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/qc.yml b/.github/workflows/qc.yml index 5dbe194..24e6fd8 100644 --- a/.github/workflows/qc.yml +++ b/.github/workflows/qc.yml @@ -11,7 +11,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: [ "3.8", "3.11" ] + python-version: [ "3.9", "3.11" ] steps: - uses: actions/checkout@v3.0.2 From 613e8dd204366fb9ed9c7be7b30c0d7c6a389108 Mon Sep 17 00:00:00 2001 From: Harry Caufield Date: Wed, 17 Jan 2024 10:44:42 -0500 Subject: [PATCH 3/7] Add 3.10 to python versions --- .github/workflows/qc.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/qc.yml b/.github/workflows/qc.yml index 24e6fd8..6eb596f 100644 --- a/.github/workflows/qc.yml +++ b/.github/workflows/qc.yml @@ -11,7 +11,7 @@ jobs: runs-on: ubuntu-latest strategy: matrix: - python-version: [ "3.9", "3.11" ] + python-version: [ "3.9", "3.10", "3.11" ] steps: - uses: actions/checkout@v3.0.2 From 160065e221df4edb9a721a503abb44e41d3d0b48 Mon Sep 17 00:00:00 2001 From: Harry Caufield Date: Wed, 17 Jan 2024 13:09:37 -0500 Subject: [PATCH 4/7] Add link to point in llm docs --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d812554..b6b75e0 100644 --- a/README.md +++ b/README.md @@ -148,7 +148,7 @@ Then start up litellm: litellm -m ollama/mixtral ``` -Next edit your `extra-openai-models.yaml` as detailed in the llm docs: +Next edit your `extra-openai-models.yaml` as detailed in [the llm docs](https://llm.datasette.io/en/stable/other-models.html): ``` - model_name: ollama/mixtral From d28b8cc6954497cdcb4c742eae18bed199d1fe29 Mon Sep 17 00:00:00 2001 From: Harry Caufield Date: Wed, 17 Jan 2024 13:18:05 -0500 Subject: [PATCH 5/7] Add note about starting the ollama server --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b6b75e0..4da12e6 100644 --- a/README.md +++ b/README.md @@ -136,7 +136,7 @@ architecture for using alternative models. In theory you can use any of these pl Additionally, you can set up an openai-emulating proxy using [litellm](https://github.com/BerriAI/litellm/). -Let's say you want to run mixtral locally using ollama. You start up ollama: +Let's say you want to run mixtral locally using ollama. You start up ollama (you may have to run `ollama serve` first): ``` ollama run mixtral From f5a5d455ff684bd5af9c22d983596d4c57360ec0 Mon Sep 17 00:00:00 2001 From: Harry Caufield Date: Wed, 17 Jan 2024 14:03:45 -0500 Subject: [PATCH 6/7] Add note about installing llm with proxy extra --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 4da12e6..606ffd8 100644 --- a/README.md +++ b/README.md @@ -136,6 +136,8 @@ architecture for using alternative models. In theory you can use any of these pl Additionally, you can set up an openai-emulating proxy using [litellm](https://github.com/BerriAI/litellm/). +The `litellm` proxy may be installed with `pip` as `pip install litellm[proxy]`. + Let's say you want to run mixtral locally using ollama. You start up ollama (you may have to run `ollama serve` first): ``` From 8484f70cc7e0babb4a649c88632b676af7153ece Mon Sep 17 00:00:00 2001 From: Harry Caufield Date: Wed, 17 Jan 2024 16:24:25 -0500 Subject: [PATCH 7/7] Minor change to the cp command As-is, this command copies `stagedb` dir to `db`, which doesn't overwrite any existing chromadb. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 606ffd8..02d6efe 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ Note that by default this loads into a collection set stored at `stagedb`, where of `db`. You can copy the collection set to the db with: ``` -cp -r stagedb db +cp -r stagedb/* db/ ```