In this tutorial, we build an end-to-end GRPO training workflow that teaches Gemma-3 to reason through GSM8K math problems using Tunix, JAX, LoRA, and custom reward functions. We start by preparing the environment, authenticating with Hugging Face, loading the Gemma-3 model, and wrapping GSM8K examples into a prompt format that requires both structured reasoning and a final numeric answer. We then define reward functions that assess format adherence and mathematical correctness, attach LoRA adapters to keep training lightweight, evaluate the baseline model, and run GRPO to improve the policy via group-sampled generations. It provides a reinforcement learning tutorial in which we train only the adapter weights while keeping the workflow compact enough for a single-accelerator setup.
Installing Tunix and Configuring GRPO Training
import importlib.util, os, shutil as _sh if importlib.util.find_spec("tunix") is None: print("Installing Tunix + JAX ecosystem — this takes ~5-8 min…") %pip install -q ipywidgets tensorboardX transformers grain nest_asyncio %pip install -q datasets huggingface_hub "numpy>2" %pip install -q tensorflow tensorflow_datasets %pip install -q git+https://github.com/jax-ml/jax %pip install -q git+https://github.com/google/tunix %pip install -q git+https://github.com/google/qwix %pip uninstall -q flax -y %pip install -q git+https://github.com/google/flax %pip uninstall -q wandb -y print("nn✅ Install done. The runtime will RESTART now.") print("👉 After it restarts, RUN THIS CELL AGAIN to start training.n") os.kill(os.getpid(), 9) import getpass, functools, json, re import numpy as np os.environ["WANDB_MODE"] = "disabled" os.environ["TOKENIZERS_PARALLELISM"] = "false" os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3" HF_TOKEN = os.environ.get("HF_TOKEN") if not HF_TOKEN: try: from google.colab import userdata HF_TOKEN = userdata.get("HF_TOKEN") except Exception: HF_TOKEN = getpass.getpass("Hugging Face token (needs Gemma license access): ") os.environ["HF_TOKEN"] = HF_TOKEN or "" import nest_asyncio; nest_asyncio.apply() import tensorflow as tf; tf.config.set_visible_devices([], "GPU") import jax, jax.numpy as jnp, optax, grain, qwix from flax import nnx from orbax import checkpoint as ocp from huggingface_hub import snapshot_download, login from datasets import load_dataset from tunix.generate import sampler as sampler_lib from tunix.generate import tokenizer_adapter as tokenizer_lib from tunix.models.gemma3 import model as gemma_lib from tunix.models.gemma3 import params_safetensors as params_safetensors_lib from tunix.models.gemma3 import params as gemma_params from tunix.rl import rl_cluster as rl_cluster_lib from tunix.rl.grpo.grpo_learner import GRPOConfig, GRPOLearner from tunix.rl.rollout import base_rollout from tunix.sft import metrics_logger if HF_TOKEN: login(token=HF_TOKEN) devices = jax.devices() print("JAX backend :", jax.default_backend(), "|", len(devices), "device(s):", devices) IS_GPU = _sh.which("nvidia-smi") is not None if IS_GPU and jax.default_backend() == "cpu": print("⚠️ GPU runtime but JAX sees CPU only. Re-run `pip install -U "jax[cuda12]"` " "and restart, or switch to a TPU runtime.") MODEL_ID = "google/gemma-3-1b-it" RANK, ALPHA = 32, 32.0 MAX_PROMPT_LENGTH = 256 TOTAL_GENERATION_STEPS = 512 NUM_GENERATIONS = 2 NUM_ITERATIONS = 1 BETA = 0.08 EPSILON = 0.2 TEMPERATURE, TOP_P, TOP_K = 0.9, 1.0, 50 MAX_STEPS = 100 TRAIN_LIMIT = MAX_STEPS NUM_TEST = 16 LEARNING_RATE, B1, B2, WEIGHT_DECAY = 3e-6, 0.9, 0.99, 0.1 WARMUP_STEPS, MAX_GRAD_NORM = int(0.1 * MAX_STEPS), 0.1 CKPT_DIR, TB_DIR = "https://www.marktechpost.com/content/ckpts/", "https://www.marktechpost.com/content/tb/grpo" N = jax.device_count() MESH = [(N, 1), ("fsdp", "tp")] mesh = jax.make_mesh(*MESH, axis_types=(jax.sharding.AxisType.Auto,) * 2) We set up the complete Colab environment by installing Tunix, JAX, Flax, Qwix, TensorFlow, datasets, and the supporting notebook utilities needed for GRPO training. We configure authentication, disable unnecessary logging paths, keep TensorFlow away from the accelerator, and verify that JAX can see the available TPU or GPU devices. We also define the core training hyperparameters, LoRA settings, generation limits, checkpoint paths, and device mesh that control how the model trains across the available hardware.
Formatting GSM8K Prompts and Reward Functions
reasoning_start, reasoning_end = "", " " solution_start, solution_end = "", " " SYSTEM_PROMPT = (f"You are given a problem. First, think about the problem and provide your " f"reasoning between {reasoning_start} and {reasoning_end}. Then give the final " f"answer (just one number) between {solution_start} and {solution_end}.") TEMPLATE = ("usern{system_prompt}nn{question}nmodeln") def extract_hash_answer(text): return text.split("####")[1].strip() if "####" in text else None gsm = load_dataset("openai/gsm8k", "main") def build_grain(split, limit): rows = [{"question": r["question"], "answer": r["answer"]} for r in split.select(range(min(limit, len(split))))] return (grain.MapDataset.source(rows).shuffle(seed=42).map( lambda x: {"prompts": TEMPLATE.format(system_prompt=SYSTEM_PROMPT, question=x["question"]), "question": x["question"], "answer": extract_hash_answer(x["answer"])})) train_dataset = build_grain(gsm["train"], TRAIN_LIMIT).batch(1)[:MAX_STEPS] val_dataset = None test_rows = [{"question": r["question"], "answer": extract_hash_answer(r["answer"])} for r in gsm["test"].select(range(NUM_TEST))] print(f"Train batches: {len(train_dataset)} | Test examples: {len(test_rows)}") match_format = re.compile( rf"^[s]{{0,}}{reasoning_start}.+?{reasoning_end}.*?{solution_start}(.+?){solution_end}[s]{{0,}}$", flags=re.MULTILINE | re.DOTALL) match_numbers = re.compile(rf"{solution_start}.*?([d.]{{1,}})", flags=re.MULTILINE | re.DOTALL) def match_format_exactly(prompts, completions, **kw): return [3.0 if match_format.search(c) else 0.0 for c in completions] def match_format_approximately(prompts, completions, **kw): out = [] for c in completions: s = 0.0 s += 0.5 if c.count(reasoning_start) == 1 else -0.5 s += 0.5 if c.count(reasoning_end) == 1 else -0.5 s += 0.5 if c.count(solution_start) == 1 else -0.5 s += 0.5 if c.count(solution_end) == 1 else -0.5 out.append(s) return out def check_answer(prompts, completions, answer, **kw): guesses = [(m.group(1) if (m := match_format.search(c)) else None) for c in completions] scores = [] for g, a in zip(guesses, answer): if g is None: scores.append(0.0); continue if g == a: scores.append(3.0) elif g.strip() == a.strip(): scores.append(1.5) else: try: r = float(g) / float(a) scores.append(0.5 if 0.9 <= r <= 1.1 else 0.25 if 0.8 <= r <= 1.2 else -1.0) except Exception: scores.append(-0.5) return scores def check_numbers(prompts, completions, answer, **kw): guesses = [(m.group(1) if (m := match_numbers.search(c)) else None) for c in completions] scores = [] for g, a in zip(guesses, answer): try: scores.append(1.5 if float(g.strip()) == float(a.strip()) else 0.0) except Exception: scores.append(0.0) return scores REWARD_FNS = [match_format_exactly, match_format_approximately, check_answer, check_numbers] We define the structured reasoning format that asks the model to place its reasoning inside reasoning tags and its final numeric answer inside answer tags. We load GSM8K from Hugging Face, extract the ground-truth answers, and convert each math problem into the prompt format expected by the GRPO rollout pipeline. We then create reward functions that score exact format matching, approximate tag usage, answer correctness, and fallback numeric extraction so the model receives useful feedback from multiple signals.
Loading Gemma-3 and Attaching LoRA Adapters
print(f"Downloading {MODEL_ID} …") local_model_path = snapshot_download(repo_id=MODEL_ID, ignore_patterns=["*.pth"]) model_config = (gemma_lib.ModelConfig.gemma3_270m() if "270m" in MODEL_ID else gemma_lib.ModelConfig.gemma3_1b_it()) with mesh: base_model = params_safetensors_lib.create_model_from_safe_tensors( local_model_path, model_config, mesh) TOK = "https://www.marktechpost.com/content/tokenizer_gemma3.model" if not os.path.exists(TOK): cand = os.path.join(local_model_path, "tokenizer.model") if os.path.exists(cand): shutil.copy(cand, TOK) else: import urllib.request urllib.request.urlretrieve( "https://storage.googleapis.com/gemma-data/tokenizers/tokenizer_gemma3.model", TOK) tokenizer = tokenizer_lib.Tokenizer(tokenizer_path=TOK) EOS_TOKENS = [] gcfg = os.path.join(local_model_path, "generation_config.json") if os.path.exists(gcfg): EOS_TOKENS = list(json.load(open(gcfg)).get("eos_token_id", []) or []) if tokenizer.eos_id() not in EOS_TOKENS: EOS_TOKENS.append(tokenizer.eos_id()) print("EOS tokens:", EOS_TOKENS) def apply_lora(base, mesh): provider = qwix.LoraProvider( module_path=".*q_einsum|.*kv_einsum|.*gate_proj|.*down_proj|.*up_proj|.*attn_vec_einsum", rank=RANK, alpha=ALPHA) m = qwix.apply_lora_to_model(base, provider, **base.get_model_input()) with mesh: st = nnx.state(m) st = jax.lax.with_sharding_constraint(st, nnx.get_partition_spec(st)) nnx.update(m, st) return m lora_policy = apply_lora(base_model, mesh) def make_sampler(model): return sampler_lib.Sampler( transformer=model, tokenizer=tokenizer, cache_config=sampler_lib.CacheConfig( cache_size=MAX_PROMPT_LENGTH + TOTAL_GENERATION_STEPS + 256, num_layers=model_config.num_layers, num_kv_heads=model_config.num_kv_heads, head_dim=model_config.head_dim)) def evaluate(rows, model, tag, n_show=2): sampler = make_sampler(model) correct = fmt_ok = 0 for i in range(0, len(rows), 4): chunk = rows[i:i + 4] qs = [r["question"] for r in chunk] prompts = [TEMPLATE.format(system_prompt=SYSTEM_PROMPT, question=q) for q in qs] outs = sampler(input_strings=prompts, max_generation_steps=TOTAL_GENERATION_STEPS, temperature=None, top_k=1, top_p=None, echo=False, eos_tokens=EOS_TOKENS).text for j, (o, r) in enumerate(zip(outs, chunk)): m = match_numbers.search(o); g = m.group(1) if m else None try: correct += int(float(g) == float(r["answer"])) except Exception: pass fmt_ok += int(match_format.search(o) is not None) if i + j < n_show: print(f" Q: {qs[j][:70]}…n gold={r['answer']} pred={g}n out: {o[:150]}…n") print(f"[{tag}] accuracy = {100*correct/len(rows):.1f}% format = {100*fmt_ok/len(rows):.1f}%n") print("n════════ BASELINE (before GRPO) ════════") evaluate(test_rows, lora_policy, "baseline") We download the selected Gemma-3 checkpoint, create the base model from safetensors, and prepare the tokenizer and EOS token list for generation. We attach LoRA adapters to the attention and MLP projection modules, enabling us to train a lightweight policy without updating the full model weights. We also build a sampler-based evaluation function and run a baseline test before GRPO training to measure the model’s initial accuracy and adherence to the format.
Configuring the Tunix RL Cluster and Training
schedule = optax.schedules.warmup_cosine_decay_schedule( init_value=0.0, peak_value=LEARNING_RATE, warmup_steps=WARMUP_STEPS, decay_steps=MAX_STEPS, end_value=0.0) optimizer = optax.chain( optax.clip_by_global_norm(MAX_GRAD_NORM), optax.adamw(learning_rate=schedule, b1=B1, b2=B2, weight_decay=WEIGHT_DECAY)) cluster_config = rl_cluster_lib.ClusterConfig( role_to_mesh={rl_cluster_lib.Role.ACTOR: mesh, rl_cluster_lib.Role.REFERENCE: mesh, rl_cluster_lib.Role.ROLLOUT: mesh}, rollout_engine="vanilla", offload_to_cpu=False, training_config=rl_cluster_lib.RLTrainingConfig( actor_optimizer=optimizer, eval_every_n_steps=10**9, max_steps=MAX_STEPS, mini_batch_size=1, train_micro_batch_size=1, metrics_logging_options=metrics_logger.MetricsLoggerOptions( log_dir=TB_DIR, flush_every_n_steps=10), checkpoint_root_directory=CKPT_DIR, checkpointing_options=ocp.CheckpointManagerOptions(save_interval_steps=50, max_to_keep=2)), rollout_config=base_rollout.RolloutConfig( max_tokens_to_generate=TOTAL_GENERATION_STEPS, max_prompt_length=MAX_PROMPT_LENGTH, kv_cache_size=MAX_PROMPT_LENGTH + TOTAL_GENERATION_STEPS + 256, temperature=TEMPERATURE, top_p=TOP_P, top_k=TOP_K, eos_tokens=EOS_TOKENS)) rl_cluster = rl_cluster_lib.RLCluster( actor=lora_policy, reference=base_model, tokenizer=tokenizer, cluster_config=cluster_config) grpo_trainer = GRPOLearner( rl_cluster=rl_cluster, reward_fns=REWARD_FNS, algo_config=GRPOConfig(num_generations=NUM_GENERATIONS, num_iterations=NUM_ITERATIONS, beta=BETA, epsilon=EPSILON)) %load_ext tensorboard %tensorboard --logdir $TB_DIR --port=0 print("n════════ TRAINING (GRPO) ════════") grpo_trainer.train(train_dataset, val_dataset) We create the learning-rate schedule, optimizer, gradient clipping, and AdamW configuration that guide the LoRA policy updates during training. We define the Tunix RL cluster with actor, reference, and rollout roles, then connect the rollout configuration to the tokenizer, generation limits, sampling temperature, and EOS tokens. We initialize the GRPO learner, launch TensorBoard for live metrics, and start the GRPO training loop on the prepared GSM8K batches.
Evaluating and Exporting the Trained Model
print("n════════ AFTER GRPO ════════") evaluate(test_rows, lora_policy, "after-grpo") try: out_dir = "https://www.marktechpost.com/content/gemma3-grpo-merged" if os.path.exists(out_dir): shutil.rmtree(out_dir) gemma_params.save_lora_merged_model_as_safetensors( local_model_path=local_model_path, output_dir=out_dir, lora_model=lora_policy, rank=RANK, alpha=ALPHA) print(f"n✅ Merged model saved to {out_dir}") except Exception as e: print(f"n(Skipped merged export: {e})") print("nDone. Checkpoints are in", CKPT_DIR) We evaluate the trained LoRA policy after GRPO to compare its math accuracy and response format against the baseline result. We then try to export the LoRA-merged Gemma-3 model into a Hugging Face safetensors directory so the trained checkpoint can be reused outside the notebook. We finish by reporting the output checkpoint path, giving us a complete workflow from training setup to post-training evaluation and optional model export.
Conclusion
In conclusion, we completed a full GRPO fine-tuning loop for mathematical reasoning with Gemma-3, from dataset preparation and reward design to LoRA-based policy training and post-training evaluation. We saw how Tunix organizes the actor, reference model, rollout engine, optimizer, checkpoints, and metrics into a reusable reinforcement learning pipeline. We also compared the model before and after GRPO to observe whether its answer format and numeric accuracy improve during training. Finally, we optionally exported the merged LoRA model, providing a complete path from raw GSM8K examples to a trained, reasoning-oriented Gemma-3 checkpoint.
Check out the FULL CODES HERE. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Sana Hassan
Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.
