Fine-Tuning Time: Why the FLOP Estimate Misses by 12 Hours

Fine-Tuning Time: Why the FLOP Estimate Misses by 12 Hours

Agentic AI

The back-of-the-envelope FLOP formula most teams use to plan fine-tuning jobs missed by 12.41 hours on average across 272 production runs, on jobs that averaged 23 hours. For anyone scheduling a shared GPU cluster, that error is the gap between a queue that runs to plan and GPUs sitting idle or a deadline slipping a day.

12.41 h
mean error of the FLOP-based estimate across 272 fine-tuning runs averaging 23.4 hours

How It Works

The team at Dharma-AI published the numbers in a Hugging Face blog post previewing an upcoming paper. They took 272 supervised fine-tuning (SFT) jobs from their production pool of NVIDIA H200 GPUs, about 7,382 GPU-hours in total. The jobs covered model families up to 15 billion parameters, both LoRA and full fine-tuning, on 1, 2 or 4 GPUs shared with other training and inference work. Durations ranged from 1.6 to 172.4 hours.

The standard estimate treats training as pure compute: roughly 6 × parameters × tokens floating-point operations, divided by GPU count, peak throughput and Model FLOPs Utilization (MFU). That formula describes the optimization loop well, the authors say, but a real job is more than its loop. Before the first step there’s environment setup, model download, loading and tokenization. During training there are periodic evaluations and checkpoint uploads, and at the end the final model goes to object storage. On a multi-tenant cluster, all of it competes with other jobs for network and storage I/O.

No single MFU value fixes that. Sweeping MFU from 0.25 to 0.65, the best setting (30 percent) still left a mean error above 12 hours. A more detailed analytical model, Calculon, extended to cover LoRA and grouped-query attention, produced estimates with a median of only 11 percent of the observed wall-clock time. The authors stress that this says nothing against Calculon as a step-level tool. It shows how much of a production job happens outside the training step.

A CatBoost regressor using only information known before dispatch did far better. Its R² was 0.98 against the FLOP estimate’s 0.31, with a mean error of 1.71 hours and a median error of 0.51 hours in 5-fold cross-validation. It used 24 features, with the FLOP estimate itself as one input so the model learns the corrections. The most important feature was num_eval_data, the total samples processed in evaluation over the job’s life. Trainable parameters, evaluation batch size, total parameters and checkpoint volume followed.

What This Means for You

The authors make a careful point about the feature ranking: importance isn’t causation. Still, evaluation frequency, evaluation set size and checkpoint count all shaped wall-clock time, and those are settings people often choose out of habit when writing a training config. A team whose jobs routinely overrun can start by auditing those three before blaming the GPUs.

The results also come with limits the authors spell out. The data covers SFT only, not RLHF, DPO or RLVR, and every run used H200s, so the fitted model shouldn’t be expected to transfer numerically to other GPU generations without retraining. Errors widen above 60 hours, where there were only 13 runs. What transfers is the method: features that expose overheads, careful feature selection and the comparison protocol. The full paper, promised for arXiv, will publish the feature formulas and hyperparameters.

Up to 60 hours, the model’s median error stayed at about one hour or less. Eighty percent of predictions landed within 2.2 hours. That’s accurate enough for a scheduler to decide which jobs to run together and whether a run will finish before its deadline.

The So What

Platform teams running shared GPU clusters should stop using the FLOP formula alone to schedule fine-tuning jobs. Log every job’s configuration and actual wall-clock time now, including evaluation cadence, eval set size and checkpoint policy, so you have the history a learned predictor needs. With GPUs as scarce as our look at the 2026 compute crunch describes, an idle hour between jobs is capacity you paid for and didn’t use.

Source: Hugging Face Blog (Dharma-AI)

Content created with AI assistance and reviewed for accuracy.

💬

Join the conversation

Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.

Join Stack Insiders →

Newsletter coming soon.

Curated digests across AI, biohacking, photography, travel, and more. Be the first to know when we launch.