Measuring legal and healthcare domain adaptation of Nemotron 3.5 Super VL, checkpoint BF16-09212026

Technical report 2 of 3 on the Nemotron 3.5 Super VL study. Prepared for the NVIDIA Nemotron model team, October 1, 2026, revised October 13, 2026.

Abstract

A simulated law firm and a simulated healthcare provider want to adapt NVIDIA Nemotron 3.5 Super VL, checkpoint BF16-09212026, to their own documents, and need a reliable way to tell whether adaptation helped. Thoughtworks trained three LoRA adapters, two legal and one healthcare, on public datasets standing in for those documents, and compared each with the original model using three measurements. On held-out rows of each training corpus, the share of reference tokens the model predicts correctly rose by 9.41 and 8.04 percentage points for the two legal adapters and by 5.42 points for the healthcare adapter, each with a 95% confidence interval excluding zero. On the public CaseHOLD legal benchmark, neither legal adapter scored significantly above the original model at the 0.05 level; on the 3,407 items both models answered, the legal adapter was 1.41 points lower (p = 0.020). A comparison graded by a judge model was withdrawn after a control showed that rewording the judge's instructions changed its verdict by up to 35.1 points between two samples of the same model. Thoughtworks recommends measuring adaptation on a held-out portion of the customer's own documents, question by question. All results were measured on the pre-release checkpoint BF16-09212026, before GA (general availability). The GA checkpoints are the completed post-training release, with improvements across multimodal, verbosity, chat and instruction following, and the results have not been re-measured on them.

1. Setup

In the simulated scenario, a law firm and a healthcare provider each adopt Nemotron 3.5 Super VL and want to adapt it to their own documents. Before paying for adaptation, each needs a way to measure whether it helped. For this demonstration the study cannot use customer data, so public legal and healthcare datasets stand in for each customer's documents (Section 1.3). The report compares three ways of measuring adaptation on checkpoint BF16-09212026: accuracy on held-out rows of the customer's own domain (Section 2), public benchmarks (Section 3) and a comparison graded by a judge model (Section 4). Throughout, the original checkpoint is called the base.

Thoughtworks trained three LoRA adapters on BF16-09212026, two legal and one healthcare. LoRA (low-rank adaptation) fine-tunes a model by training small matrices added to its frozen weights.

1.1 Study design

Each experiment asks whether adapting BF16-09212026 to a domain improves it on that domain, and each measurement is judged by whether it can detect that improvement. A change outside the adapted domain is reported but does not count against an adapter, because each customer uses its adapter only in its own domain. Every comparison is between two models measured the same way on the same items. No figure is compared with NVIDIA's published scores.

1.2 Checkpoint

The base is NVIDIA's early-access post-training release of Nemotron 3.5 Super VL, downloaded as 80 files totaling about 249 GB and labeled BF16-09212026 in this study. The files were verified against the size declared in NVIDIA's NGC registry and against a checksum taken on arrival. The earlier pre-release checkpoint BF16-09112026 contributes only two things to this report: the reasoning traces of Section 1.3 and the sampling-variation estimate of Section 3.1.

1.3 Training corpora

Each training corpus mixes domain rows with general instruction data, called replay, which makes up 15% of rows (15% of target tokens for the traces corpus). The replay comes from nvidia/Nemotron-Post-Training-Dataset-v1 (math, code, STEM) and nvidia/Nemotron-SFT-Instruction-Following-Chat-v3 and preserves the model's general behavior with reasoning both on and off. One percent of each corpus was held out before training.

Table 1. Training corpora, counted from the files the adapters were trained on.

Table 2. Rows contributed by each source to the legal corpus.

The study excluded three subsets of the legal dataset: Nemotron-Pretraining-Legal-CaseHOLD and LegalBench-CUAD-v2, which are derived from evaluation benchmarks, and Nemotron-Pretraining-Legal-Case-Law-Summary, which is raw pre-training text. A hash over context and options confirms that none of the CaseHOLD training rows matches any of the 3,600 test items.

A reasoning trace is the step-by-step reasoning a model writes before its answer. To build the traces corpus, the model answered CaseHOLD training questions with reasoning on, and a reply was kept only if it finished its reasoning and then gave the correct letter. The traces were written by the earlier checkpoint BF16-09112026, so the traces adapter learned from reasoning produced by a different version of the weights it adapts.

1.4 Adapters and training

Table 3. Adapter and optimizer configuration, identical for the three experiments.

Training used the fused Mamba-2 kernels from mamba_ssm and causal_conv1d, with the Mamba-2 layers forced onto the non-split kernel path. Each run checked its corpus by sha256 and refused to start on any other file. The legal run's first logged loss was 0.6097, against 11.7835 for random guessing (the natural log of the 131,072-token vocabulary). All three runs completed their 600 steps of 32 sequences, 19,200 sequences in all, which is 0.19 epochs of the legal corpus, 0.25 of the traces corpus and 0.24 of the healthcare corpus.

Each adapter was merged into the base weights before evaluation. Re-scoring the merged model on the held-out rows matched the training run's last evaluation to within 0.0008, 0.0012 and 0.0005 for the three experiments, against a tolerance of 0.004.

1.5 Evaluation instruments

The three measurements are described in the order of Sections 2 to 4.

Held-out domain accuracy. The share of target tokens the model predicts correctly when given the preceding reference text (teacher-forced token accuracy), over the first 256 held-out rows, computed with the training code's evaluation path. The domain rows are 221 legal rows with 17,703 target tokens, or 222 healthcare rows with 102,952. Adapter and base are compared row by row, and a paired bootstrap over rows (10,000 resamples, seed 20260918) gives the 95% confidence interval. The rule, fixed before the base was scored, is that an adapter helps its domain when this interval excludes zero, with no minimum effect size.

CaseHOLD. The test split of CaseHOLD (coastalcph/lex_glue, case_hold) has 3,600 items. Each is a passage from a court decision with one cited holding masked, and five candidate holdings to choose from. The prompt lists the five lettered options and ends with "Answer with the letter alone." Generation used vLLM 0.28.1rc1.dev628+g2a02f6efe on four GPUs in bf16, with a 32,768-token context, reasoning on (the chat template's default), temperature 0.6, top-p 0.95, seed 20260922 and a limit of 16,384 generated tokens. The answer is read from the text after the reasoning block. Each item is parsed (a letter in answer position), unparsable (text but no letter) or silent (no text after the reasoning), and an unanswered item counts as wrong. Models are compared item by item with an exact two-sided McNemar test, a paired test on the items where the two models disagree.

Judged win rate. Two hundred held-out prompts from the legal corpus, 163 legal and 37 replay, taken after the 256 rows used above, were answered by both models at temperature 0.6, top-p 0.95 and a 16,384-token limit. A judge model, claude-opus-5 with a 2,048-token limit, compares the two answers in both orders. A win requires the same verdict in both orders; otherwise the pair is a tie. As a control, the judge also compares two samples of the base against each other, which should come out even.

2. Paired held-out accuracy on the customer's domain

On held-out rows of each customer's corpus, compared row by row, all three adapters score above the base. All three meet the rule fixed before scoring, and their reductions in loss also have 95% intervals excluding zero.

Table 4. Held-out teacher-forced token accuracy on each domain stratum, paired over identical rows. The base is BF16-09212026, and each adapter is trained on it.

The two legal adapters share one base measurement because their held-out files are identical. Their intervals overlap between +8.39 and +9.75 points, so this measurement does not separate them. The healthcare rows carry 5.8 times as many target tokens as the legal rows for a similar number of rows, so the healthcare gain is not ranked against the legal gains.

The rows come from each adapter's own training distribution, and the score is token prediction on reference text, with no generated answer scored. Report 1 sets this measure aside for its chart task because most target tokens there are working rather than answer. The same applies to healthcare, at about 464 target tokens a row against 80 for legal: in the healthcare training data, 89% of the target tokens of MedReason rows and 77% of medical-o1 rows are reasoning. The healthcare score therefore mostly measures how well the model reproduces reference reasoning.

2.1 General replay rows

The 15% replay is meant to preserve the base's general and reasoning behavior. On the held-out replay rows, all three adapters predict the reference text better than the base, by 1.82 to 2.40 points, with intervals excluding zero (Table 5). These rows come from the same general datasets as the replay used in training, and the score is token prediction, so the result shows the adapters kept their fit to that data; it does not measure general capability on generated answers (Section 6).

Table 5. Held-out teacher-forced token accuracy on the general replay rows, paired over identical rows against BF16-09212026.

3. Public benchmarks

A public benchmark can measure adaptation only if the model has not seen it before and it is hard enough to separate models. For legal adaptation, the adapters were compared with the base on CaseHOLD (Sections 3.1 and 3.2), and Section 3.3 describes a post-2018 benchmark built for this study, which the base answers at 91.75%. For healthcare, three benchmarks were screened against one published subset (Section 3.4).

3.1 Paired CaseHOLD comparisons

Neither legal adapter exceeds the base on CaseHOLD at p below 0.05, and the healthcare adapter, evaluated outside its domain, is 6.25 points below the base over all items (Table 7).

Table 6. CaseHOLD test split, each model scored once. Generation budget 16,384 tokens, temperature 0.6.

Table 7. CaseHOLD, each adapter paired against the base by exact McNemar. A negative delta places the adapter below the base.

The legal adapter's 0.97-point lead over all items is a net 35 items and is not significant (p = 0.129); on the 3,407 items both models answer, it is 1.41 points below the base. The traces adapter is 1.44 points below the base over all items and level with it on the 3,217 items both answer, so its deficit comes from its additional silent items.

Each model was sampled once. In a healthcare run on the earlier checkpoint BF16-09112026, 115 to 131 of 1,273 MedQA-USMLE items, about a tenth, changed answer between saved checkpoints under the same sampling settings.

3.2 Generation-budget exhaustion

All 188 silent base items, the legal adapter's 6 and the traces adapter's 275 have the same cause: the reply reaches exactly 16,384 tokens, vLLM reports the length limit as the reason it stopped, and the reasoning block never closes. None of the 469 silent items stopped for another reason.

Table 8. The 188 CaseHOLD items on which the base exhausts its generation budget, and what each adapter returns on them.

The legal adapter answers 87 items correctly where the base returns nothing, more than its net lead of 35 items over all 3,600, which is why it falls below the base on the items both answer (Table 7). Training on single-letter answers shortened its median reply from 1,400 tokens to 151 and cut budget exhaustion from 188 items to 6. The traces adapter, trained on reasoning, keeps a median reply of 1,435 tokens and exhausts the budget on 275 items, 87 more than the base. Its training examples were capped at 4,096 tokens, a quarter of the 16,384-token budget its silent replies reach.

3.3 A post-2018 legal benchmark

The case law in Nemotron-Pretraining-Legal-v1 is United States material, with citations at a median year of 1980 and a 99.9th percentile of 2018. To obtain a benchmark of cases decided after that period, Thoughtworks built one from United States federal appellate and Supreme Court opinions decided from 2019 to 2026, public domain, taken from a CourtListener bulk export, following CaseHOLD's construction. An opinion cites a case and states its holding in parentheses; that holding is masked and becomes the correct option, and the four wrong options are holdings of other cases cited in the same opinion or, failing that, from the same jurisdiction and five-year period. The build produced 78,202 items, of which 54,231 needed the fallback.

A screen for 32-word exact matches between 400 sampled items and all fourteen subsets of Nemotron-Pretraining-Legal-v1 found none. The rate of shared 8-word phrases was 8.25%, below the 9.5% measured on 2,807 post-2018 Massachusetts opinions that the dataset cannot contain by date, so it reflects common judicial phrasing.

Before any item existed, an acceptance range was fixed: the base must score above 30% and at most 60% for the benchmark to separate models. One remedy was fixed with it: if the base scored above 60%, keep only items whose wrong options come from the same opinion, and score again against the same range.

Table 9. Original BF16-09212026 on the post-2018 benchmark. 400 sampled items per pass, temperature 0.0, reasoning off, 1,024-token budget.

Both passes exceed the 60% maximum, at 91.75% and 85.00%, so the benchmark was rejected. Wrong options drawn from other cases concern other points of law, so the topic of the passage gives the answer away.

3.4 Healthcare benchmarks

Neither the healthcare adapter nor the base was scored on a healthcare benchmark. This section records which healthcare benchmarks were screened for presence in published NVIDIA data, and which data was not screened.

Of the 325 datasets published under the nvidia organization on Hugging Face at the time of the screen, none is described as a healthcare pre-training corpus, so there was no healthcare corpus to screen against. Instead, the subset closest in form to the healthcare benchmarks, Nemotron-Pretraining-Multiple-Choice in nvidia/Nemotron-Pretraining-Specialized-v1.2, 23,926,492 rows and 7.29 GB, was screened in full.

Table 10. Healthcare benchmarks screened against Nemotron-Pretraining-Multiple-Choice, 400 items sampled from each.

The positive control, 400 of 400, shows that the screen detects text present in this subset. The three benchmarks are absent from this 7.29 GB subset. About 750 GB of NVIDIA's other published synthetic, STEM and SFT data was not screened, so the healthcare benchmarks are not reported as clean.

4. Judged pairwise comparison

A judged win rate scores generated answers, which the held-out measurement of Section 2 does not. Rewording the judge's instructions changed its verdict on the same answer pairs in six of seven groups, by 4.4 to 35.1 points (Table 11).

Of the 200 judged prompts, 11 have an empty user message and are excluded, and in 23 pairs one side hit the token limit, leaving 166 pairs, 151 of them legal. In the control, the same 11 prompts are dropped, along with 75 pairs whose two base samples are identical and so say nothing about the judge, and 13 that hit the limit, leaving 101 pairs, 82 of them legal. The judge's verdict changes with the presentation order on 9 of the 101 control pairs.

Under the original instructions, the judge preferred the longer answer in 34 of 43 decided legal control pairs that differed in length, 79.1% (one-sided p = 0.0001). Both answers in the control come from BF16-09212026, so this 79.1% reflects the judge, not the model. The strengthened wording says that a short answer that resolves the question beats a longer one that resolves it equally well, and asks for the verdict the judge would give if both answers were the same length. Under that wording the judge preferred the longer answer in 10 of 44 pairs, 22.7%, that is, the shorter answer in 77.3%. The criterion fixed before the run asked for 60% or below; 22.7% meets it, but with the preference reversed.

Table 11. Judged win rate of the legal adapter, or of the second base sample in base-against-base rows, ties as half, on identical pairs under two instruction wordings. Judge claude-opus-5.

The 40 equal-length pairs do not move: every answer in them is one character, and in 33 of the 40 both sides give the same letter. Across the 166 judged pairs, the adapter's answers have a median of 12 characters against the base's 174, so every comparison involving a written explanation is also a comparison of length. The judged win rate of the legal adapter against BF16-09212026 is therefore withdrawn and reported in neither direction.

5. Limitations

• Held-out accuracy comes from a single pass over the first 256 held-out rows of each corpus, drawn from the adapter's own training distribution, and scores no generated answer.

   

• The replay result in Section 2.1 is token prediction on 35 legal and 34 healthcare rows from the training distribution. No general-capability benchmark was run on any adapter.

   

• Each CaseHOLD figure comes from one sampling draw per model at temperature 0.6.

   

• The healthcare adapter was evaluated only on held-out healthcare rows, with no healthcare benchmark and no judged comparison.

   

• The two legal adapters differ in both training data and target format, and the traces adapter's traces were written by BF16-09112026. An attempt to regenerate the traces with BF16-09212026 was stopped by its pre-set length criterion, and no adapter was trained on it.

   

• The library versions of the training environment were not recorded; the evaluation engine version is recorded with each evaluation result.

   

• The post-2018 acceptance passes used temperature 0.0, reasoning off and a 1,024-token budget, against temperature 0.6, reasoning on and 16,384 tokens for CaseHOLD, so the two accuracies are not compared.

   

• The exact-match screens detect copies, not a benchmark rewritten sentence by sentence, and a clean screen covers only the data screened.

   

• All results are on BF16-09212026 and none has been re-measured on the GA checkpoints.

6. Conclusion

In the simulated scenario, a law firm and a healthcare provider need to know whether adapting Nemotron 3.5 Super VL to their documents helped before paying for it. Of the three measurements tested on BF16-09212026, held-out accuracy on the customer's own domain places all three adapters above the base, by 5.42 to 9.41 percentage points of token accuracy, with 95% intervals excluding zero. The held-out rows come from each adapter's training distribution, and this measurement scores no generated answer.

The base answers 91.75% of the post-2018 legal benchmark built for this study, so that benchmark does not separate models on this checkpoint. In the paired CaseHOLD comparisons, neither legal adapter scores significantly above the base; the legal adapter's 0.97-point lead over all items comes from answering items on which the base runs out of its 16,384-token budget, and on the items both answer it is 1.41 points lower. The judged win rate is withdrawn because the judge's verdict moves with the wording of its instructions, by up to 35.1 points between two samples of the base.

On these measurements, Thoughtworks recommends that a real customer hold out part of its own documents before training and compare the adapted and original model on them question by question, as this study did with held-out rows of public datasets. A judged win rate needs a base-against-base control before it can decide between two models.