Local AI & ModyLM Fine-Tuning

Consumer Hardware QLoRA SFT & Autonomous Agent Runtime

Practical engineering in local model training and inference: fine-tuning Qwen2.5-7B with QLoRA on an 8GB RTX 4060, implementing an educational 33M-parameter decoder in PyTorch, and building the Bellion local agent runtime.

1. The Engineering Problem

Cloud LLM APIs introduce latency, continuous billing, and privacy exposure for sensitive personal and codebase workflows. Meanwhile, running 7B+ parameter modern models locally on consumer GPUs (e.g. mobile RTX 4060 with 8GB VRAM) frequently causes out-of-memory errors during fine-tuning. The objective was mastering the mechanics of local AI: understanding Transformer decoders from the ground up, executing VRAM-budgeted QLoRA supervised fine-tuning, and evaluating quantization tradeoffs.

2. What Was Built

ModyLM is a local AI research repository comprising three distinct pillars: (1) 4-bit QLoRA Supervised Fine-Tuning of Qwen2.5-7B base using Unsloth on consumer NVIDIA hardware, (2) an educational, from-scratch 33M-parameter Transformer decoder (ModyLM-33M) written in PyTorch to understand attention heads and residual streams, and (3) the Bellion local autonomous agent execution runtime featuring persistent mission storage and watchdog processes.

3. System Architecture & Topology

ASCII Topology

Hardware-aware local pipeline optimized for 8GB VRAM limits: tokenized dataset preparation -> Unsloth memory-efficient 4-bit QLoRA -> LoRA adapter generation -> GGUF quantization export for local Ollama serving.

+-----------------------------------------------------------------------+
|                     CURATED DATASET RESERVOIR                         |
|  Mohamed Style Chat & Systems Prompt Data (train.jsonl)               |
+------------------------------------+----------------------------------+
                                     |
                                     v
+-----------------------------------------------------------------------+
|              HARDWARE-BUDGETED QLoRA ENGINE (RTX 4060 8GB)            |
|  - Base: Qwen/Qwen2.5-7B (Base Model)                                 |
|  - Unsloth FastLanguageModel (Memory-efficient gradient checkpointing)|
|  - 4-bit NF4 Quantization + LoRA rank/alpha parameter optimization    |
|  - Strict VRAM cap: Peak allocation kept < 7.2 GB                      |
+------------------------------------+----------------------------------+
                                     | Export Trained Adapters
                                     v
+-----------------------------------------------------------------------+
|                         GGUF EXPORT & INFERENCE                       |
|  - 16-bit LoRA fusion -> GGUF format quantization (Q4_K_M / Q6_K)    |
|  - Local serving via Ollama & custom Modelfiles                       |
|  - Integrated into Bellion Local Agent Core & Tool Foundry            |
+-----------------------------------------------------------------------+

4. Mohamed's Personal Implementation

Built and curated the supervised fine-tuning dataset pipeline (`sft_reply_style`), cleaning and formatting structured instruction-response pairs.

Wrote the PyTorch architecture for the educational `ModyLM-33M` decoder (33,497,600 parameters) including Multi-Head Self-Attention, rotary embeddings, RMSNorm, and SwiGLU activation.

Configured Unsloth training scripts on an 8GB RTX 4060 GPU with gradient accumulation to prevent CUDA out-of-memory errors.

Conducted systematic quantization comparisons across Q3_K_M, Q4_K_M, and Q6_K to evaluate perplexity, memory footprint, and generation tokens-per-second.

Engineered custom Modelfiles for local agent workflows in Ollama.

5. Engineering Challenges & Solutions

Challenge: Fitting 7B Parameter Model SFT in 8GB Laptop VRAM

Solution: Combined 4-bit base weight quantization, gradient checkpointing, fused cross-entropy kernels, and 16-bit LoRA adapter matrices via Unsloth, keeping maximum VRAM consumption below 7.2 GB.

Impact: Achieved complete fine-tuning convergence on consumer laptop hardware without cloud GPU rental.

6. Technology Stack

Deep Learning Frameworks

PyTorchUnslothTransformersPEFT

Hardware Baseline

NVIDIA GeForce RTX 4060 Laptop 8GBCUDA 12Torch MSVC

Quantization & Serving

GGUFOllamaLlama.cppCustom Modelfiles

Base Architectures

Qwen2.5-7BModyLM-33M (Custom PyTorch)LFMGranite

7. Verified Outcomes

•Demonstrated practical local-AI engineering and VRAM optimization without exaggerating foundation-scale pretraining.
•Successfully trained QLoRA adapters running locally for specialized code and assistant workflows.

← Back to Portfolio