DCSR LLM Tools

Copyright (C) 2025 Philippe Jacquet

DCSR LLM Tools is a command-line toolkit for research workflows with large language models. It supports downloading models from Hugging Face, running chat interactions, quantization, benchmarking, and fine-tuning. It also provides inference for multiple-choice and open-ended questions, as well as information extraction from text corpora.

This toolkit is designed for already aligned instruction-tuned/chat models. It is not intended for foundation-model pre-training, continued pre-training, or initial alignment from base models.

The CLI (dcsr-llm) provides one entry point for these workflows across local models and supported API-based backends.

The toolkit is optimized for GPU-based workflows on the UNIL clusters Curnagl and Urblauna, and it also runs on MacBook Pro (M1 and later) and CPU-only systems.

It can also export local Hugging Face models to GGUF format for use with LM Studio.

Typical Workflow

This diagram shows a typical dcsr-llm workflow from setup to inference, evaluation, and fine-tuning.

Project Status (Beta)

This tool is currently in beta and under active development. You may encounter bugs or breaking changes.

Your feedback is very welcome:

Report bugs and error messages
Share confusing steps or missing explanations
Suggest improvements or new features

The goal is to reach a stable v1.0.0 release by the end of 2026. Until then, please treat the CLI and workflows as evolving.

Feedback

Please send feedback, bug reports, and suggestions as follows:

For UNIL members: use helpdesk@unil.ch (subject: DCSR-LLM)
For non-UNIL members: https://git.dcsr.unil.ch/Scientific-Computing/dcsr-llm/issues

Installation

Requirements

Python 3.11 or newer
Git
Supported platform: Linux or macOS (Apple Silicon tested)

Optional:

NVIDIA GPU and compatible CUDA environment for [cuda]
CMake for llama.cpp installation
curl for the optional fastText language model download

Choose a working directory

On Curnagl: We recommend running experiments from your scratch directory and copying any important outputs to your NAS. This repository also supports storing data, models, and results inside the project folder.

cd /scratch/username/

On your laptop:

cd <your-folder>

Clone the repository

On all devices:

git clone https://git.dcsr.unil.ch/Scientific-Computing/dcsr-llm

Clone a specific release tag

To clone a specific tagged release directly:

git clone --branch v0.1.0 --depth 1 https://git.dcsr.unil.ch/Scientific-Computing/dcsr-llm

If you already cloned the repository and want to switch to a tag:

cd dcsr-llm
git fetch --tags
git checkout v0.1.0

Replace v0.1.0 with the tag you want.

For a more reproducible setup, you can combine a tagged checkout with the pinned constraints files documented below for Mac and Curnagl.

Create and activate a virtual environment

On Curnagl

cd dcsr-llm

Sinteractive -m 20G -G 1
module load python/3.11.14

python -m venv .venv
source .venv/bin/activate

On your laptop (or DGX Spark):

cd dcsr-llm

python -m venv .venv
source .venv/bin/activate

Install the project (developer / editable mode)

Editable mode (-e) ensures that local code changes are immediately reflected when running the CLI.

On clusters (including Curnagl), set the pip cache in scratch before installing:

Curnagl home is limited to 50 GB.
For this project, the pip cache can be around 2 GB.
The project .venv can be around 10 GB, so it should be created outside home (for example in scratch).
After installation, this cache can be deleted to free space (/scratch/$USER/.cache/pip).

export PIP_CACHE_DIR=/scratch/$USER/.cache/pip

On Curnagl (A100/H100) or Linux with NVIDIA GPU (CUDA 12.8 wheels):

pip install -U pip
pip install -e ".[cuda]" --extra-index-url https://download.pytorch.org/whl/cu128

Optional: install with the tested pinned constraints for Curnagl:

pip install -U pip
pip install -e ".[cuda]" --extra-index-url https://download.pytorch.org/whl/cu128 -c requirements/requirements_cuda.txt

On DGX Spark (newer CUDA 13.0 wheels): Some systems require a newer CUDA build than the default stable PyTorch wheels. In that case, install a nightly / pre-release PyTorch build first:

pip install -U pip
pip install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu130
pip install -e ".[dgx]"

On MacBook Pro (M1+):

pip install -U pip
pip install -e ".[mac]"

Optional: install with the tested pinned constraints for Mac:

pip install -U pip
pip install -e ".[mac]" -c requirements/requirements_mac.txt

On CPU-only Linux:

pip install -U pip
pip install -e ".[cpu]" --extra-index-url https://download.pytorch.org/whl/cpu

To check that your installation was successful:

dcsr-llm check-installation

On macOS or Linux systems where matplotlib does not have a writable default cache directory, add the following line to your shell startup file (~/.zshrc or ~/.bashrc) to avoid repeated font-cache rebuilds and slow CLI startup for commands such as dcsr-llm --help:

export MPLCONFIGDIR="$HOME/.cache/matplotlib"

Then create the directory once if needed:

mkdir -p "$HOME/.cache/matplotlib"

Notes:

Editable mode (pip install -e ...) is recommended for development: you can modify the code and immediately re-run the CLI without reinstalling.
The library "bitsandbytes" cannot be installed on MacBook Pro (M1 and later) or on CPU-only systems. This is a limitation of bitsandbytes, not this tool. As a result, the dcsr-llm quantize command can only be used on Curnagl (or Linux system) with a GPU.

Install development tools (recommended)

On all devices:

pip install -e ".[dev]"

This installs the following tools: black for code formatting (run with black .), mypy for static type checking (run with mypy src), ruff for detecting unused imports and variables (run with ruff check .), and pytest for running the full test suite located in tests/ (run with pytest).

Install the tutorial extra libraries (recommended)

On all devices:

pip install -e ".[tutorial]"

This installs the additional libraries required to run the tutorial.

Download the fastText language model

If you use language detection/classifier in dcsr-llm extract with detector: fasttext, download the fastText language model weights once:

mkdir -p ~/.local/share/dcsr-llm/fasttext
curl -L -o ~/.local/share/dcsr-llm/fasttext/lid.176.ftz https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.ftz

You can then keep fasttext_model_path: null in config_extract_advanced.yaml (or in any other extract config where language.detect: true). The tool auto-detects ~/.local/share/dcsr-llm/fasttext/lid.176.ftz, and also falls back to ~/.cache/fasttext/lid.176.ftz.

Install llama.cpp

We only use llama.cpp to convert/quantize models into the GGUF format for LM Studio (Mac/Linux), so a CPU-only build is enough. Model inference is done in LM Studio, which uses its own backend (e.g. Metal on MacBook Pro M1+), independently of how llama.cpp is installed on your machine. Therefore, CUDA/Metal acceleration is not required for this tool, unless you specifically want to run inference directly with llama.cpp itself.

Note: The GGUF file produced by llama.cpp is identical whether you run the conversion/quantization on CPU or with CUDA/Metal, and whether you generate it on Linux, macOS, or another supported platform. In other words, using a CPU build does not reduce the quality of the quantized model. Also, in llama.cpp, GGUF quantization is a CPU (multi-threaded) process, so installing a CUDA/Metal-enabled build usually won’t make it meaningfully faster. In practice, CUDA/Metal mainly improves inference speed, not quantization speed.

Build llama.cpp from source as follows:

cd dcsr-llm
git clone https://github.com/ggerganov/llama.cpp.git

cd llama.cpp
cmake -B build  -DCMAKE_BUILD_TYPE=Release 
cmake --build build -j
ln -sf build/bin/llama-quantize quantize

Notes:

On Curnagl, you need to load cmake: module load cmake
If you do not install llama.cpp inside dcsr-llm folder, you need to specify the directory via the option --llama-cpp-dir (or set via LLAMA_CPP_DIR), and it must contain both:
- convert_hf_to_gguf.py
- quantize

This setup allows you to export local Hugging Face models to GGUF format for use with LM Studio.

For example, to import the local model HuggingFaceTB_SmolLM3-3B.Q4_K_M.gguf in LM Studio:

lms import HuggingFaceTB_SmolLM3-3B.Q4_K_M.gguf --copy --user-repo my-user/custom-models

Replace my-user/custom-models with your own username and repository name.

Usage

After installation, the CLI is available as:

dcsr-llm --help

To get help for a given command:

dcsr-llm command --help

Acceleration

accelerate is a Hugging Face helper library that manages where model weights run (GPU/CPU), and can help with multi-GPU and CPU offload. This helps speed up inference and training, and can reduce out-of-memory errors. In this tool, it is used behind the scenes when local models are loaded with transformers (especially with device_map).
Attention kernels (including Flash Attention) are low-level routines used to compute attention efficiently. They can speed up inference and training and reduce memory use. In this tool, they are selected automatically behind the scenes by PyTorch/Transformers when supported by the model and hardware.
torch.inference_mode() is a PyTorch context for inference-only execution. It disables training-related bookkeeping, which can reduce overhead and memory use during generation. In this tool, local model generation runs under it behind the scenes.
Mixed precision (bf16/fp16) uses lower-precision math to reduce memory use and improve speed on supported NVIDIA GPUs. In this tool, precision is selected automatically based on your device and config, with safe defaults (bf16 on CUDA in most cases).
KV-cache restore after training re-enables use_cache for the saved model so autoregressive generation can reuse past tokens efficiently. This improves inference speed, especially for longer outputs. This setting is stored in the model config files (config.json, and sometimes generation_config.json) inside the fine-tuned model folder.

Reproducibility

Small score differences across hardware backends are expected, even with the same command and seed.

Typical example: running the same benchmark on Apple Silicon (MPS) and NVIDIA (CUDA) can differ by a few questions on a 100-item MCQ benchmark.

Why this happens:

Different backend math and kernels (MPS vs CUDA)
Different precision paths (for local inference this tool uses fp16 on MPS and usually bf16 on CUDA)
Borderline questions where tiny logit differences flip one option letter

Recommended practice:

Keep protocol fixed (same model, exam file, prompt/preprompt, decoding settings, parser behavior, and seed)
Compare trends, not single-run deltas
For important claims, run repeated evaluations and report mean + spread

Model format

Local HF Transformers-compatible models using .safetensors or pytorch_model*.bin are supported.
Transformers prefers SafeTensors and falls back to PyTorch .bin when SafeTensors are unavailable.

Data format

Benchmark, predict, and extract datasets use JSON or JSONL files containing lists of question or text objects. Finetune datasets use JSONL files, with one chat conversation per line.

Input files, images, and PDFs

benchmark, predict, and extract do not take raw image or PDF files as their main input. Their main input is always a structured JSON or JSONL file.

Images can be attached only by listing image filenames or paths in the images field of each item. Supported image formats are the formats Pillow can open in the installed environment, such as common PNG/JPEG files. The tool does not guarantee support for every image file format.

PDF files are not parsed, OCRed, or split into pages automatically. To use a PDF, preprocess it first: extract the text into the question/text field, or convert pages to image files and reference those images in the JSON/JSONL input. For example, a 5-page PDF should be converted to five page images and listed in the item's images field if you want a vision-capable model to see all pages. OCR through a vision LLM is usually less controlled than using a dedicated PDF text extraction or OCR tool first. If the PDF has embedded selectable text, extracting that text directly is usually better than asking a vision model to read rendered page images.

Benchmark (MCQ)

Each item must define answer options and the correct answer:

[
  {
    "id": "1",
    "question": "What color is the sky on a clear day?",
    "options": {
      "a": "Green",
      "b": "Blue",
      "c": "Red",
      "d": "Yellow"
    },
    "correct": "b"
  }
]

Predict (MCQ)

Same format as benchmark, but without the correct field.

Predict (open-ended)

Only the id and question are required.

Images (optional)

Questions may include images by adding an images field:

{
  "question": "...",
  "images": ["image1.png", "image2.jpg"]
}

Image files must be stored in:

<exam-dir>/images/

Notes

Option keys must be single letters (a–z). Uppercase letters are accepted and normalized automatically.
The correct field (benchmark only) is case-insensitive and must match one of the option keys.
All exam files are JSON or JSONL files containing a list of question objects.
Additional fields in question objects are ignored.

Extract

Text corpus may also include images:

{
  "id": "1",
  "text": "Please prepare a quote for 2 NVIDIA H200 GPUs. We can spend up to USD 62,000 this quarter.",
  "images": ["image1.png", "image2.jpg"]
}

Image files must be stored in:

<corpus-dir>/images/

Notes (Extract)

Additional fields in corpus objects are ignored.

Finetune

Fine-tuning in this project is for task/domain adaptation of existing chat/instruct checkpoints.

All datasets must be stored as JSONL files.

Each line is one training example containing a chat conversation as a list of messages:

{"messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Write a haiku about the moon."},{"role": "assistant", "content": "Pale coin in night sky..."}]}

This corresponds to the human-friendly format:

{"messages": [
  {"role": "system", "content": "You are a helpful assistant."},
  {"role": "user", "content": "Write a haiku about the moon."},
  {"role": "assistant", "content": "Pale coin in night sky..."}
]}

Notes:

The "system" message is optional (often omitted).
The "user" and "assistant" messages are required.
Additional keys (e.g. "id", "source", "metadata") are allowed but ignored by training.

You can also store multi-turn chats:

{"messages":[
  {"role":"user","content":"What is the capital of France?"},
  {"role":"assistant","content":"Paris."},
  {"role":"user","content":"And Germany?"},
  {"role":"assistant","content":"Berlin."}
]}

By default, the training pipeline converts messages into prompt/completion pairs internally:

prompt = all messages up to the target assistant turn
completion = the assistant response being learned. This ensures loss is computed on assistant tokens only (completion-only loss), which is the standard behavior for instruction/chat fine-tuning. This behavior is controlled by the YAML field data.train_on_assistant_only (recommended: true).

In addition, configuration files must be stored as YAML:

model:
  ...
data:
  ...
training:
  ...
wandb:
  ...

See ./configs for complete examples.

Examples

Important: Run dcsr-llm from the repository root directory (dcsr-llm), not a subdirectory.

If you use SLURM scripts:

Place all SLURM scripts in the ./slurm folder.
Do not use the option #SBATCH --export=NONE.
Never write API keys directly inside SLURM scripts.
Instead, set the required environment variables in your terminal only if needed:
- export HF_TOKEN="your-hf-token"
- export OPENAI_API_KEY="your-openai-key"
- export ANTHROPIC_API_KEY="your-anthropic-key"
- export WANDB_API_KEY="your-wandb-key"

These environment variables will then be automatically available inside the SLURM scripts.

Alternatively, if you want to use the option #SBATCH --export=NONE, you can store your keys in specific files as follows:

# Create config directory
mkdir -p ~/.config/dcsr-llm

# Store secrets in local config files (one line each, no quotes, no extra spaces).
printf "%s" "your-hf-token"     > ~/.config/dcsr-llm/hf_token
printf "%s" "your-openai-key"   > ~/.config/dcsr-llm/openai_api_key
printf "%s" "your-anthropic-key"   > ~/.config/dcsr-llm/anthropic_api_key
printf "%s" "your-wandb-key"    > ~/.config/dcsr-llm/wandb_api_key

# Restrict permissions so only the user can read these files
chmod 600 ~/.config/dcsr-llm/hf_token
chmod 600 ~/.config/dcsr-llm/openai_api_key
chmod 600 ~/.config/dcsr-llm/anthropic_api_key
chmod 600 ~/.config/dcsr-llm/wandb_api_key

The CLI will first look for environment variables (e.g. OPENAI_API_KEY). If not set, it will try the corresponding file under ~/.config/dcsr-llm/.

For OpenAI-compatible API models, create a profile file and select it with --api-profile:

# Create config directory
mkdir -p ~/.config/dcsr-llm/api_profiles

# Store API profile (JSON)
cat > ~/.config/dcsr-llm/api_profiles/vllm.json <<'EOF'
{"base_url": "http://HOST:PORT/v1", "api_key": "optional", "api_mode": "chat"}
EOF

# Restrict permissions so only the user can read these files
chmod 600 ~/.config/dcsr-llm/api_profiles/vllm.json

API mode: In your API profile, set "api_mode": "chat" (default) or "responses". Use "chat" unless you know your server fully supports the Responses API; it is the safest and most widely compatible mode.

Security note: These files contain sensitive credentials. Do not place them inside the project directory or commit them to version control.

Inspect

dcsr-llm inspect --help

dcsr-llm inspect --model-name mistralai/Mistral-7B-Instruct-v0.2

For private Hugging Face repositories:

export HF_TOKEN="your-hf-token"

dcsr-llm inspect --model-name mistralai/Mistral-7B-Instruct-v0.2 --use-hf-token

Download

dcsr-llm download --help

dcsr-llm download --model-name mistralai/Mistral-7B-Instruct-v0.2

For private Hugging Face repositories:

export HF_TOKEN="your-hf-token"

dcsr-llm download --model-name mistralai/Mistral-7B-Instruct-v0.2 --use-hf-token

Chat

dcsr-llm chat --help

dcsr-llm chat --model-name HuggingFaceTB/SmolLM3-3B --think-mode off

dcsr-llm chat --model-name HuggingFaceTB/SmolLM3-3B --think-mode off --auto-checks --result-name-suffix baseline

dcsr-llm chat --model-name HuggingFaceTB/SmolLM3-3B --suffix FineTuned --think-mode off --auto-checks --baseline-report chat_checks_HuggingFaceTB_SmolLM3-3B_baseline.json

Quantize

dcsr-llm quantize --help

dcsr-llm quantize --model-name mistralai/Mistral-7B-Instruct-v0.2 --bits 4

Benchmark

dcsr-llm benchmark --help

dcsr-llm benchmark --model-name HuggingFaceTB/SmolLM3-3B --exam-name simple_quiz --think-mode off

dcsr-llm benchmark --model-name Qwen/Qwen2.5-VL-3B-Instruct --multimodal --exam-name simple_quiz_with_images

To use OpenAI models:

export OPENAI_API_KEY="your-openai-key"

dcsr-llm benchmark --model-name HuggingFaceTB/SmolLM3-3B --exam-name simple_quiz --max-new-tokens 200 --think-mode on --openai-map-to-choice --use-openai-api-key

dcsr-llm benchmark --model-name openai:gpt-4o --exam-name simple_quiz --use-openai-api-key

dcsr-llm benchmark --model-name openai:gpt-4o --multimodal --exam-name simple_quiz_with_images --use-openai-api-key

To use Anthropic models:

export ANTHROPIC_API_KEY="your-anthropic-key"

dcsr-llm benchmark --model-name anthropic:claude-sonnet-4-20250514 --exam-name simple_quiz --use-anthropic-api-key

To use OpenAI-compatible API models:

dcsr-llm benchmark --model-name api:gpt-oss-120b --api-profile vllm --exam-name simple_quiz --max-new-tokens 1000

Predict

dcsr-llm predict --help

dcsr-llm predict --mode mcq --model-name HuggingFaceTB/SmolLM3-3B --exam-name simple_quiz --think-mode off

dcsr-llm predict --mode mcq --model-name Qwen/Qwen2.5-VL-3B-Instruct --multimodal --exam-name simple_quiz_with_images

dcsr-llm predict --mode open --model-name HuggingFaceTB/SmolLM3-3B --exam-name mini_open_questions --preprompt-file-name preprompt_open --max-new-tokens 200 --think-mode on

To use OpenAI models:

export OPENAI_API_KEY="your-openai-key"

dcsr-llm predict --mode mcq --model-name openai:gpt-4o --exam-name simple_quiz --use-openai-api-key

To use Anthropic models:

export ANTHROPIC_API_KEY="your-anthropic-key"

dcsr-llm predict --mode mcq --model-name anthropic:claude-sonnet-4-20250514 --exam-name simple_quiz --use-anthropic-api-key

To use OpenAI-compatible API models:

dcsr-llm predict --mode mcq --model-name api:gpt-oss-120b --api-profile vllm --exam-name simple_quiz --max-new-tokens 1000

Extract

dcsr-llm extract --help

# 1) Basic first: use the default starter config (configs/config_extract.yaml)
dcsr-llm extract --model-name HuggingFaceTB/SmolLM3-3B --corpus-name corpus --think-mode off

# 2) Optional: add a project-specific preprompt while keeping the basic config
dcsr-llm extract --model-name Qwen/Qwen3-4B-Instruct-2507 --corpus-name corpus --config-file-name config_extract --preprompt-file-name preprompt_corpus --think-mode off

# 3) Benchmark second: compare candidate setups on a validation corpus with gold labels
dcsr-llm extract --model-name Qwen/Qwen3-4B-Instruct-2507 --corpus-name corpus_val --config-file-name config_extract_benchmark --preprompt-file-name preprompt_corpus --think-mode off

# 4) Advanced last: use the full reference config for advanced controls or multimodal extraction
dcsr-llm extract --model-name Qwen/Qwen2.5-VL-3B-Instruct --multimodal --corpus-name corpus_with_images --config-file-name config_extract_advanced --think-mode off

The multimodal sample also has a matching gold file: data/extract/corpus_with_images_gold.json. In that example, the model and budget come from the text, while the quantity comes from the image.

To use OpenAI models:

export OPENAI_API_KEY="your-openai-key"

dcsr-llm extract --model-name openai:gpt-4o --corpus-name corpus --use-openai-api-key

To use Anthropic models:

export ANTHROPIC_API_KEY="your-anthropic-key"

dcsr-llm extract --model-name anthropic:claude-sonnet-4-20250514 --corpus-name corpus --use-anthropic-api-key

To use OpenAI-compatible API models:

dcsr-llm extract --model-name api:gpt-oss-120b --api-profile vllm --corpus-name corpus

Notes on `--think-mode`, `--max-new-tokens`, and reasoning

--think-mode auto does not force anything; model defaults are used.
--think-mode on/off uses model-specific directives only for families with verified support in this tool.
Otherwise, --think-mode on/off falls back to generic best-effort prompting and may be ignored.
--max-new-tokens is a hard cap: the model does not see it, but outputs are truncated if the cap is reached.
Verbose models (e.g., some gpt-oss-* deployments) may hit the cap before emitting the final answer. Increase --max-new-tokens for those models.
OpenAI-compatible servers may return reasoning in a separate field (e.g., reasoning_content) and only put the final answer in content. Example: reasoning says “Answer is blue, so output ‘b’”, while content is just b. Local transformers runs return raw text, so you may see reasoning in raw_output even when the answer is correct.
Vision caution for API: api: with images only works if the server hosts a vision-capable model; otherwise images may be ignored and accuracy will drop.

Finetune

dcsr-llm finetune --help

dcsr-llm finetune \
  --finetune-strategy lora \
  --model-name HuggingFaceTB/SmolLM-135M-Instruct \
  --config-file-name config_lora \
  --train-data-name train --val-data-name val

dcsr-llm finetune \
   --finetune-strategy full_ddp \
   --model-name HuggingFaceTB/SmolLM-135M-Instruct \
   --config-file-name config_full_ddp \
   --train-data-name train \
   --val-data-name val

# Requires two physical GPUs, not two MIG slices from the same GPU
# So on Curnagl Sinteractive -m 100G -G 2 -c 4 will not work
# Instead you must use SLURM with #SBATCH --gres=gpu:2
torchrun --standalone --nproc_per_node 2 -m dcsr_llm finetune \
  --finetune-strategy full_fsdp \
  --model-name HuggingFaceTB/SmolLM3-3B \
  --config-file-name config_full_fsdp \
  --train-data-name train --val-data-name val

Sweep

dcsr-llm sweep --help

# Execute the full grid sequentially on the local machine
dcsr-llm sweep --grid configs/config_sweep_lora.yaml

# Execute the full grid in parallel on a cluster
dcsr-llm sweep --grid configs/config_sweep_lora.yaml \
 --index "${INDEX}" \
 --count "${ARRAY_COUNT}"

This command is useful in SLURM script to do grid search over hyper-parameters in parallel.

Terminology (for model selection):

Val benchmark: evaluation split used during hyper-parameter search and checkpoint selection.
Test benchmark: final held-out benchmark used for final reporting only.
Exam: CLI/file artifact term (for example --exam-name, --exam-dir, exam JSON/JSONL files), not the methodological split term.

Model-selection rule of thumb (step by step):

Train candidate models on the train split.
Monitor val loss during training to detect divergence, instability, or obvious overfitting.
Discard clearly bad or unstable checkpoints based on val loss behavior.
Evaluate remaining checkpoints/candidates on the val benchmark.
Rank models primarily by the main val benchmark metric appropriate for the task..
If top candidates are essentially tied on the val benchmark, prefer the one with more stable/lower val loss.
Choose final hyper-parameters and checkpoint using only validation-side information.
Evaluate once on the test benchmark (and test set) for final reporting.

What to avoid:

Do not pick sweep winners using test benchmark scores.
Do not repeatedly check test results while tuning hyper-parameters.
Avoid frequent evaluation on the val benchmark during training, as repeatedly selecting checkpoints based on it can lead to implicit overfitting to that benchmark.

After selecting the best hyper-parameters from sweep results, run test loss once on the held-out test set (do not run this during the sweep itself):

dcsr-llm finetune \
  --model-name HuggingFaceTB/SmolLM-135M-Instruct \
  --config-file-name config_test_loss \
  --test-loss \
  --test-data-name test \
  --suffix best_model

If the selected model comes from a sweep run, pass its corresponding --suffix so test loss targets the correct checkpoint.

Export to GGUF

dcsr-llm export-gguf --help

dcsr-llm export-gguf --model-name HuggingFaceTB/SmolLM3-3B --gguf-quant Q4_K_M

Note on extract

The quality of dcsr-llm extract depends on the model, the optional preprompt, and the extraction config. The optional preprompt is project-specific guidance, not a requirement for the basic starter workflow.

This repository provides three example configs with a deliberate order:

config_extract.yaml: basic first, minimal and safe to copy
config_extract_benchmark.yaml: benchmark second, for validation with gold labels
config_extract_advanced.yaml: advanced last, as a full reference for optional professional features such as evidence, language tagging, richer validation settings, or multimodal workflows

If you want reliable extraction on a real corpus, do not start with the full corpus.json. First create a small benchmark with gold answers so you can measure which setup works best.

The bundled corpus_val_gold.json is not only a happy-path toy set. It also includes teaching cases where a field must stay null because the source text is ambiguous, undecided, or gives multiple plausible values.

For normalized string fields such as gpu_model, make the canonical form explicit in your schema or preprompt. Describe the rule at the policy level, for example whether vendor prefixes, generic product words, or redundant marketing suffixes should be removed, rather than baking document-specific answers into the prompt.

Recommended workflow:

Start with config_extract.yaml and make the schema fit your task.
Test that basic setup on a few easy documents first.
If your project needs extra policy text, add a preprompt file. This is optional.
Create a small validation corpus, for example corpus_val.json.
Create the matching gold file, for example corpus_val_gold.json.
In the gold file, each document must have the same id as in the corpus, plus an output object with exactly the same keys as the fields declared in your extraction config.
Switch to config_extract_benchmark.yaml and set benchmark.filename to your gold file name.
Run extract on the validation corpus with one candidate setup:

dcsr-llm extract \
  --model-name HuggingFaceTB/SmolLM3-3B \
  --corpus-name corpus_val \
  --config-file-name config_extract_benchmark \
  --preprompt-file-name preprompt_corpus \
  --think-mode off

Read the benchmark outputs in results/extract/:
- *_benchmark.json: overall summary and per-field accuracy
- *_benchmark_failures.csv: exactly which document/field pairs failed
- *_benchmark_histogram.pdf: quick visual summary
Try a few reasonable alternatives on the validation corpus:

another model
another preprompt
another truncation or validation setting

Keep the setup that performs best on corpus_val.json. This is your selected extraction pipeline.
Evaluate that selected setup once on a separate test corpus, for example corpus_test.json with corpus_test_gold.json.
Do not choose settings based on test results. The test set is only for final confirmation.
Only after that, move to config_extract_advanced.yaml if you need evidence quotes, language tagging, extra prompt controls, or multimodal extraction.
Run the selected setup on the real corpus.json.
For the final full-corpus run, keep benchmark disabled unless you also have a gold file for that corpus.

Minimal example of a gold benchmark file:

[
  {
    "id": "1",
    "output": {
      "request_type": "quote_request",
      "gpu_model": "H200",
      "quantity": 2,
      "price_amount": 62000,
      "price_currency": "USD"
    }
  }
]

In short:

use corpus_val + corpus_val_gold to choose the best extraction setup
use corpus_test + corpus_test_gold once for final evaluation
use corpus only after the setup has been selected

Note on quantization formats

Models quantized with the quantize command (which uses bitsandbytes in 4-bit or 8-bit) and models quantized with the export-gguf command (which uses llama.cpp) rely on different quantization strategies.

As a result, even when using the same nominal bit width (for example, 8-bit), the two quantized models are not bit-identical, and small behavioral differences are expected and normal. In practice, however, you can usually expect the same overall accuracy, reasoning structure, and response style.

For this reason, if you want to use a quantized LLM as an interactive Chatbot, it is recommended to:

Benchmark bitsandbytes-quantized models when evaluating model performance, and
Export GGUF models separately for deployment (for example, in LM Studio).

Acknowledgements

This project builds on experience gained from LLM tooling developed within a UNIL project in collaboration with Kejdi Lleshi, and benefited from discussions with him during that work.

Parts of this project were developed with assistance from AI tools (e.g., OpenAI GPT models). All code was reviewed, adapted, and validated by the author.

The code in this repository was written by Philippe Jacquet and is distributed under the GPLv3-or-later license. Any prior work was used with permission, and no third-party code is included without proper authorization.

License

This project (dcsr-llm) is licensed under the GNU General Public License v3.0 or later (GPL-3.0-or-later). See the LICENSE file for the full license text.

No warranty

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU GPL for details.

Scientific use disclaimer

This disclaimer is informational and does not override the terms of the software license.

This software is intended for research and educational use. While we aim for correctness and usefulness, it has not been formally validated for any specific scientific, clinical, engineering, or operational purpose.

Results may be incomplete or incorrect. You are responsible for verifying and validating any outputs before relying on them in publications, decision-making, or other downstream uses.

Responsible use

Please use this tool responsibly:

Do not present model outputs as verified facts without independent validation.
Be especially cautious for high-stakes domains (health, legal, finance, safety, or compliance).
Respect the licenses and terms of any models, datasets, and third-party tools you download or use through this project.
When reporting results, document your settings (model, prompt, seeds, hardware, and versions) to support reproducibility.

Third-party models, datasets, and tools

This repository primarily contains code, plus small example datasets, configuration YAML files, and sample results to help you test commands and understand the expected formats. These examples are distributed under this project's license unless stated otherwise.

Model weights, datasets, and any third-party resources downloaded or used by this tool remain under their own licenses/terms. You are responsible for complying with those terms when downloading, using, or redistributing them.

Citation

If you use dcsr-llm in your research or projects, please cite it as follows:

@misc{dcsr-llm-2026,
  author = {Philippe Jacquet},
  title  = {DCSR-LLM: A Toolkit for Research},
  url    = {https://git.dcsr.unil.ch/Scientific-Computing/dcsr-llm},
  year   = {2026},
}

License

Scientific-Computing/dcsr-llm

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Languages

Packages