Welcome, friend! With Heretic, you can remove restrictions from language models, or modify them in other interesting ways.
\nThat's right, you!
\nYou don't need to be a software engineer, you don't need a machine learning PhD, you don't need to understand the intricacies of how language models work internally, and you don't need expensive hardware.
\nHere's what you do need:
\nA GPU. Both Nvidia and AMD GPUs are well supported by Heretic. While Heretic also supports processing language models in CPU-only mode, this is orders of magnitude slower than accelerated processing, and should only be attempted with tiny models. You will also need to install the CUDA Toolkit if you have an Nvidia GPU, and ROCm if you have an AMD GPU.
\nSufficient VRAM. Make sure that your GPU is a match for the model you want to process. As a rule of thumb, you need about 2.5 GB of VRAM per billion model parameters. So to process a model like Qwen3.5-4B (which has 4 billion parameters), you need about 10 GB of VRAM.
\nPython. If you run Linux, Python is almost always preinstalled, but on other platforms, you may have to download and install it manually. Heretic requires Python 3.10 or later.
\nWe strongly recommend installing Heretic (and any other Python application) into its own virtual environment, which you can set up withsh
\npython -m venv venv\nsource venv/bin/activateTIP
\nIf you are using a cloud computing platform, the appropriate version of PyTorch is often already preinstalled.
\nHeretic uses PyTorch to accelerate mathematical operations. Unlike all other dependencies of Heretic, PyTorch must be installed manually because the correct installation command depends on your GPU and accelerator library.
\nYou can find full installation instructions on the PyTorch website. In most cases, if you have an Nvidia GPU, you will runsh
\npip install torch torchvisionand if you have an AMD GPU, you will runsh
\npip install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.2but it's a good idea to check the page linked above for the specific instructions for your setup.
\nThere are many ways to download and install Heretic. Most users will want to runsh
\npip install -U heretic-llmand be done with it.
\nPick a language model on Hugging Face and copy its model ID (the part of its URL after https://huggingface.co/). Then runsh
heretic Qwen/Qwen3.5-4BReplace Qwen/Qwen3.5-4B with the model ID you copied above.
That's it! The process is fully automatic and will prompt you when you need to make a decision. Heretic does not require configuration, although many configuration parameters are available, allowing you to control almost every aspect of Heretic's operation.
\nLet's walk through what happens when Heretic processes a model:
\nAt the start of the program, Heretic identifies the GPU(s) available. As you can see, even with a modest GPU (an RTX 3060 in this case) it's already possible to process small models (4B in this case).
\nTIP
\nHeretic supports loading models with 4-bit quantization using bitsandbytes, which can reduce the amount of VRAM required by about 70%. See the quantization setting for details.
█░█░█▀▀░█▀▄░█▀▀░▀█▀░█░█▀▀ v1.3.0\n█▀█░█▀▀░█▀▄░█▀▀░░█░░█░█░░\n▀░▀░▀▀▀░▀░▀░▀▀▀░░▀░░▀░▀▀▀ https://github.com/p-e-w/heretic\n\nDetected 1 CUDA device(s) (11.63 GB total VRAM)\nCUDA Version: 12.8\nDriver Version: 580.159.03\n* CUDA 0: NVIDIA GeForce RTX 3060 (11.63 GB)Heretic now loads the requested model (downloading it from Hugging Face unless it is already cached), analyzes its architecture, and displays the amount of memory it occupies.
\nLoading model Qwen/Qwen3.5-4B...\n* Trying dtype auto...\n* LoRA adapters initialized (target types: down_proj, o_proj, out_proj)\n* Transformer model with 32 layers\n* Abliterable components:\n * attn.o_proj: 32 modules total\n * mlp.down_proj: 32 modules total\n\nResident system RAM: 1.83 GB\nAllocated GPU VRAM: 8.47 GB\nReserved GPU VRAM: 8.52 GBThe prompt datasets used for calculating refusal directions are now being loaded.
\nLoading good prompts from mlabonne/harmless_alpaca...\n* 400 prompts loaded\n\nLoading bad prompts from mlabonne/harmful_behaviors...\n* 400 prompts loadedHeretic does inference on prompts in batches, which dramatically speeds up processing. This step determines the largest batch size your system can support, in order to maximize processing speed.
\nDetermining optimal batch size...\n* Trying batch size 1... Ok (25 tokens/s)\n* Trying batch size 2... Ok (44 tokens/s)\n* Trying batch size 4... Ok (78 tokens/s)\n* Trying batch size 8... Ok (122 tokens/s)\n* Trying batch size 16... Ok (178 tokens/s)\n* Trying batch size 32... Ok (225 tokens/s)\n* Trying batch size 64... Failed (CUDA out of memory)\n* Chosen batch size: 32Reasoning models often start their responses with a thinking block (such as <think>...</think>). Skipping the tokens in this block improves accuracy when calculating refusal directions, and Heretic automatically tests for such prefixes in order to do this.
In this case, no prefix is found.
\nChecking for common response prefix...\n* None foundThe prompt datasets used for evaluating model performance are now being loaded, and an initial evaluation is performed.
\nLoading good evaluation prompts from mlabonne/harmless_alpaca...\n* 100 prompts loaded\n* Obtaining first-token probability distributions...\n\nLoading bad evaluation prompts from mlabonne/harmful_behaviors...\n* 100 prompts loaded\n* Counting model refusals...\n* Initial refusals: 93/100Heretic modifies language models by ablating "refusal directions" from certain operators. This step computes those directions based on the provided prompts.
\nCalculating per-layer refusal directions...\n* Obtaining residual mean for good prompts...\n* Obtaining residual mean for bad prompts...This is the heart of the process. Heretic searches for a combination of parameter values that, when used to control ablation of the refusal directions, yields the best compromise between refusal suppression and model quality.
\nThis step can take some time, slightly under 3 hours in this case. An estimate for the remaining amount of time, as well as memory stats, are displayed after every trial.
\nRunning trial 1 of 200...\n* Parameters:\n * direction_index = per layer\n * attn.o_proj.max_weight = 1.39\n * attn.o_proj.max_weight_position = 25.29\n * attn.o_proj.min_weight = 0.15\n * attn.o_proj.min_weight_distance = 2.39\n * mlp.down_proj.max_weight = 1.23\n * mlp.down_proj.max_weight_position = 20.65\n * mlp.down_proj.min_weight = 0.19\n * mlp.down_proj.min_weight_distance = 11.58\n* Resetting model...\n* Abliterating...\n* Evaluating...\n * Obtaining first-token probability distributions...\n * KL divergence: 0.0126\n * Counting model refusals...\n * Refusals: 88/100\n\nElapsed time: 53s\nEstimated remaining time: 2h 56m\nResident system RAM: 2.23 GB\nAllocated GPU VRAM: 8.47 GB\nReserved GPU VRAM: 10.12 GB\n\nRunning trial 2 of 200...\n* Parameters:\n * direction_index = per layer\n * attn.o_proj.max_weight = 1.49\n * attn.o_proj.max_weight_position = 25.36\n * attn.o_proj.min_weight = 0.01\n * attn.o_proj.min_weight_distance = 3.19\n * mlp.down_proj.max_weight = 1.49\n * mlp.down_proj.max_weight_position = 23.56\n * mlp.down_proj.min_weight = 0.33\n * mlp.down_proj.min_weight_distance = 11.33\n* Resetting model...\n* Abliterating...\n* Evaluating...\n * Obtaining first-token probability distributions...\n * KL divergence: 0.0125\n * Counting model refusals...\n * Refusals: 88/100\n\nElapsed time: 1m 46s\nEstimated remaining time: 2h 55m\nResident system RAM: 2.16 GB\nAllocated GPU VRAM: 8.47 GB\nReserved GPU VRAM: 10.13 GB\n\n[... 198 more trials ...]After the optimization run is complete, you will be shown the Pareto front of all trials, and asked to choose which combination of refusal count and KL divergence (a rough measure of the difference between the modified model and the original one) you want.
\nIMPORTANT
\nThe trial with the lowest number of refusals is not always the best choice. Any trial with a refusal count below 10 can be assumed to strongly suppress refusals, and among those, the trial with the lowest KL divergence should be chosen because it has the best chance of preserving the original model's intelligence.
\nOptimization finished!\n\nThe following trials resulted in Pareto optimal combinations of refusals\nand KL divergence. After selecting a trial, you will be able to save the\nmodel, upload it to Hugging Face, chat with it to test how well it works,\nor run standard benchmarks on it. You can return to this menu later to\nselect a different trial. Note that KL divergence values above 0.5 usually\nindicate significant damage to the original model's capabilities.\n\n? Which trial do you want to use? (Use arrow keys)\n » [Trial 91] Refusals: 11/100, KL divergence: 0.0508\n [Trial 81] Refusals: 21/100, KL divergence: 0.0366\n [Trial 116] Refusals: 30/100, KL divergence: 0.0296\n [Trial 93] Refusals: 45/100, KL divergence: 0.0120\n [Trial 70] Refusals: 74/100, KL divergence: 0.0112\n [Trial 37] Refusals: 75/100, KL divergence: 0.0109\n [Trial 83] Refusals: 76/100, KL divergence: 0.0077\n [Trial 85] Refusals: 77/100, KL divergence: 0.0059\n [Trial 126] Refusals: 80/100, KL divergence: 0.0035\n [Trial 47] Refusals: 86/100, KL divergence: 0.0032\n [Trial 119] Refusals: 87/100, KL divergence: 0.0011\n [Trial 113] Refusals: 91/100, KL divergence: 0.0010\n [Trial 49] Refusals: 93/100, KL divergence: 0.0006\n Run additional trials\n Exit programAfter choosing a trial, you can now decide what you want to do with the resulting model. If you opt to upload the model to Hugging Face and you aren't already logged into Hugging Face on your system, you will be prompted to enter an access token, which you can get from the Hugging Face website as explained here. Make sure the token has write permissions, as otherwise you won't be able to upload a model with it.
\nRestoring model from trial 91...\n* Parameters:\n * direction_index = 19.93\n * attn.o_proj.max_weight = 1.36\n * attn.o_proj.max_weight_position = 18.63\n * attn.o_proj.min_weight = 1.21\n * attn.o_proj.min_weight_distance = 16.30\n * mlp.down_proj.max_weight = 1.44\n * mlp.down_proj.max_weight_position = 18.86\n * mlp.down_proj.min_weight = 1.16\n * mlp.down_proj.min_weight_distance = 12.35\n* Resetting model...\n* Abliterating...\n\n? What do you want to do with the decensored model? (Use arrow keys)\n » Save the model to a local folder\n Upload the model to Hugging Face\n Chat with the model\n Benchmark the model\n Return to the trial selection menuOnce you have successfully processed your first model with Heretic, it's time to explore the many configuration parameters Heretic offers that give you control over what it does.
\nIf you encounter any problems while using Heretic, or if you have suggestions or ideas for improving Heretic, please file an issue on GitHub, or join us on Discord or on Matrix!
","headings":[{"level":1,"text":"Tutorial ","id":"tutorial"},{"level":2,"text":"Prerequisites ","id":"prerequisites"},{"level":2,"text":"Set up a Python virtual environment ","id":"set-up-a-python-virtual-environment"},{"level":2,"text":"Install PyTorch ","id":"install-pytorch"},{"level":2,"text":"Install Heretic ","id":"install-heretic"},{"level":2,"text":"Run Heretic ","id":"run-heretic"},{"level":3,"text":"Startup ","id":"startup"},{"level":3,"text":"Model loading ","id":"model-loading"},{"level":3,"text":"Prompt loading ","id":"prompt-loading"},{"level":3,"text":"Automatic batch size determination ","id":"automatic-batch-size-determination"},{"level":3,"text":"Automatic response prefix determination ","id":"automatic-response-prefix-determination"},{"level":3,"text":"Evaluation prompt loading ","id":"evaluation-prompt-loading"},{"level":3,"text":"Refusal directions calculation ","id":"refusal-directions-calculation"},{"level":3,"text":"Parameter optimization ","id":"parameter-optimization"},{"level":3,"text":"Trial selection ","id":"trial-selection"},{"level":3,"text":"Action selection ","id":"action-selection"},{"level":2,"text":"Next steps ","id":"next-steps"}]}}