Instructions to use mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF", dtype="auto")

llama-cpp-python

How to use mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF",
	filename="meta-llama-3.1-8b-instruct-abliterated.Q2_K.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF with Ollama:
```
ollama run hf.co/mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q4_K_M
```

Unsloth Studio

How to use mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF to start chatting

Docker Model Runner
How to use mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF with Docker Model Runner:
```
docker model run hf.co/mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q4_K_M
```

Lemonade

How to use mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Meta-Llama-3.1-8B-Instruct-abliterated-GGUF-Q4_K_M

List all available models

lemonade list

Not working properly (Q5_K_M) but I am not very experienced

#10

by i4one - opened Aug 10, 2024

Discussion

i4one

Aug 10, 2024

Using it both in command line and in Ollama, it simply didn't answer anything. When I would hit return again, I got either nothing or a strange response. I am going to download a different version and try that, but put this here in case others have the same issue.

@mlabonne Thanks for doing these!

what model are you running?

can you answer?
Welcome to the virtual bookshelf! I'm here to help you find a new book recommendation. Can you please tell me what
type of books you're in the mood for? Would you like something based on a specific theme, genre, or author?

ManniX-ITA

Aug 12, 2024

•

edited Aug 12, 2024

Did you try if this one works?
https://ollama.com/mannix/llama3.1-8b-abliterated:q5_k_m

i4one

Aug 13, 2024

How do I get that model? I cannot find a hugging face version of it, I cannot find a download button or option in the page you linked me to, and I cannot pull it with ollama pull from the command line. I know this is a terribly basic question, but I am still getting started with this.

bhaswata08

Aug 14, 2024

You just enter in the terminal:

ollama run mannix/llama3-uncensored

ManniX-ITA

Aug 14, 2024

@i4one
As suggested by @bhaswata08 you can run on terminal the run command to pull, more precisely this one:

ollama run mannix/llama3.1-8b-abliterated:q5_k_m

i4one

Aug 15, 2024

@bhaswata08 @ManniX-ITA

I had actually tried both of those commands before I posted, both pull and run, no joy.

It starts with >pulling manifest
then I getting an error: >Error: Incorrect Function

There was an issue with this error raised on GitHub, and it looked like they pushed a fix. I did the full upgrade per the docs.openwebui, but same error.

bhaswata08

Aug 22, 2024

@i4one
Try running with ollama docker then. spin up an instance and pull

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment