aboutteamprojectsblogschat
aboutteamprojectsblogschat
© 2026 ottomind.ai
← Back to blogs

How Can a Language Model Be Made Better in a Specific Domain?

18 min read · Mar 12, 2026

General-purpose language models look impressive at first glance. They produce answers on many topics, write code, explain things, and sometimes give genuinely strong results. So from the outside, it is easy to think like this: if this model already writes code, then it should work comfortably in any software domain we want.

But in practice, it does not work that way.

Because “knowing code” and actually being useful inside a specific development environment are not the same thing. A model that produces code in general does not keep the same level of performance once the task moves into a narrower domain. Especially if that domain has its own libraries, its own class structures, its own usage patterns, and its own repeated development habits, the model falls apart more easily there. That was exactly the point we were looking at: how does a code model get better in a more specific area like Unity? [1][4][5]

What we did in this research was roughly this: we took Code Llama, extracted data from Unity documentation, turned that data into question-code pairs, fine-tuned the model for this domain, and then compared the base model with its fine-tuned versions side by side. We also did not leave it at a single dataset; we tried it with both a smaller and a larger dataset. That let us see not only whether fine-tuning works, but also what changes as the dataset grows. [1][4][5]

Why is a general-purpose model not always enough?

Language models are trained on very large corpora. That naturally gives them a wide range of motion. They can say something about many topics and produce work at a certain level across many tasks. But breadth does not always mean depth. The model knows many things a little, but it does not carry the internal logic of a specific domain, its recurring structures, and the production style people in that domain actually expect with the same clarity. [1][2]

This becomes even more visible on the code side. Because here, it is not enough to produce something that merely looks correct. What is written has to run. The classes being used have to fit the context. The code has to be internally consistent. And it has to do all that in the way the developer expects. In a place like Unity, that is exactly the issue. Even if the model has general C# knowledge, if it does not carry Unity’s component logic, flow structure, API usage patterns, and common practices well enough, what comes out may look decent on paper but still stay weak in real use. [4][5]

Why does a code model need extra steering in a more specific area like Unity?

Because Unity is not just “a place where things are written in C#.” It has its own concepts, its own flows, and its own recurring solution patterns. General code knowledge takes the model only so far there. If we want it to carry Unity-specific expectations more cleanly, we need to feed it examples from that domain as well. [4][5]

Just because the model writes code in general does not mean it will automatically produce the same quality in Unity too. Once it sees more Unity-style question formats, code patterns, class relationships, and commonly used structures, it becomes more reliable in that area. That is why the goal here was not to invent a new model, but to push a model that was already suitable for code generation toward a narrower and more concrete domain of use. [1][4][5]

What does it actually mean to push a model toward a specific domain?

When we say we want to move a language model closer to a specific domain, what we actually mean is not turning the model into something completely different from top to bottom. The goal here is not building a model from scratch either. What we wanted was to make a model that already had a certain capacity more accurate, more consistent, and more reliable inside a particular task domain. [2][6][8]

So “a model that can write code” and “a model that writes better code in Unity” are not the same thing. That difference usually does not come from replacing the model completely. It comes from pushing the model’s behavior toward a narrower context. [2][6]

What does fine-tuning change here?

The easiest way to think about fine-tuning is this: on top of the big, general world the model already knows, you layer the habits of a narrower domain more intensely.

In other words, the model does not learn from zero. You do not build a new intelligence from scratch. You make the target domain more visible on top of what is already there. The model then sees more of that domain’s question formats, code structures, and recurring patterns. Later, when it answers, it carries that domain better. [2][6][8]

Fine-tuning is not a magic wand. It does not teach the model everything perfectly at once. But when it is built with the right data, it really does change the model’s behavior in a specific domain. If the target domain is clear and the training data actually represents that domain, the model produces more organized, cleaner, and more accurate outputs in that context. [2][6][8]

What exactly were we trying to do in this research?

Our goal here was not to build a new model from scratch. We already had a model designed for code generation. What we wanted to see was this: can we make this model perform better in a specific domain? [1][3]

That domain was Unity. Because the point was not just “make it write code.” We wanted to see whether we could get more meaningful, more correct, and more useful outputs on the Unity side. In other words, we took a model that generally knew code and tried to make it behave better inside the Unity context. [1][4][5]

Why did we not build a new model from scratch?

Because that was not the need.

We already had a strong base model. In a case like that, instead of rebuilding everything from zero, it made much more sense to push that model toward the area we needed. Because the problem was not “there is no model.” The problem was that the model we had did not behave as well as we wanted in the exact place we wanted. [1][2][6]

So we did not ask “should we build a new model?” We asked “how can we make the existing model better for this area?” That is why we moved toward fine-tuning.

How did we give the model domain knowledge?

If you want to move a model closer to a specific domain, you first need to build a structure that can give that domain’s knowledge to the model. Saying “make it better at Unity” does not solve anything by itself. You need to turn that knowledge into a form the model can read, build relationships with during training, and see over and over again. [2][6]

We started with Unity documentation. Because the cleanest, most direct, and most useful knowledge about Unity was already there. Which class does what, how a structure is used, which code is written in which context — all of that was already in the documentation. If your target is a model that writes better code for Unity, it is very natural that the first place you look is there. [4][5]

The important point for us was this: we did not want to give the model random C# code. Because the target was not general C#; it was directly Unity. So the data source also had to come directly from the Unity world. The first thing we did was identify which documentation pages would actually be useful for us. Then we turned that content into something the model could learn from.

The format we built here followed a question-answer logic. More precisely, we built a “question → code answer” structure. Because that was what we expected from the model: it should see a question and produce code in response. Then the dataset also had to look like that.

So what we did was this: we took code examples from Unity documentation, generated questions that could correspond to them, and in that way created question-code pairs the model could read during training. We turned domain knowledge directly into training examples.

How did we collect the content?

We built a Python flow here. First we mapped the site structure, then separated the pages that would actually be useful for the dataset. After that we pulled the content from those pages and saved it. In the first stage, we kept them as JSON.

This step mattered for us because we first had to collect the raw material. A dataset does not appear out of nowhere. You first find the source where you will get the domain knowledge, then pull the useful parts out of that source, and then organize them. That was exactly what we did on the scraping side.

How did we solve question generation for code blocks?

This is where the work became more interesting. Because we had code, but we did not want to train the model only on code fragments. What we expected from the model was to generate code when it saw a question. So the dataset also had to resemble that.

To solve that, we used the Gemini API. We processed the content we had put into JSON one by one and generated suitable questions for those code fragments. In other words, we did not leave the code examples alone; we placed question sentences next to them that would naturally ask for that code. Then we turned those question and code pairs into the dataset.

Why did we enlarge the dataset?

After building the first dataset, the question that came to mind was very simple: if the dataset grows, does the model actually get better?

To see that, we prepared a second, larger dataset as well. We took the examples in the first dataset as a base, generated similar new questions, produced Unity-side code answers for those questions, then checked and validated the incoming answers and built the second, larger dataset.

How did we fine-tune Code Llama for Unity?

Once the dataset was ready, the next part of the work was training. We now had Unity-side question-code pairs. So we knew what kinds of questions we wanted the model to see, and we knew what kinds of code outputs we expected in response. From there, what we had to do was take that data and really push Code Llama toward this domain.

First we set up the environment where we could run the model. Then we prepared the dataset in a form suitable for the model. After that we arranged the training flow and fine-tuned the model.

How did we build the training flow?

First we brought the model and the dataset into the training environment, then converted the data into a form the model could understand, then built the prompt structure and started training.

The important thing here was not just storing the examples in the dataset. The model had to see them in a certain format. So we embedded the question and the answer directly into the training prompt. In other words, we showed the model the same behavior again and again: a question will come, and you will give a Unity-focused code answer to it.

That is why we arranged the data in a way that resembled the model’s behavior at real usage time.

Why did LoRA and PEFT become critical here?

Because we did not want to retrain the whole model. We wanted to build a lighter and more manageable fine-tuning flow. We did not completely dismantle and rebuild the base model; we adjusted it so it would behave closer to the Unity side. [6][7][8]

If you think about it in everyday terms, the issue is this: you are not creating a new character from zero, you are changing some habits of an existing one. The model is still the same model, but its reflexes in a particular domain become a little different. That was the comfort LoRA and PEFT gave us. [6][7][8]

Why did we run separate trials with two different dataset sizes?

Because we did not want to look only at whether fine-tuning works. We also wanted to see what really changes as the dataset grows.

That is why we did not stop at a single dataset. We trained with a smaller set and with a larger set. That let us see not only the difference between the base model and the fine-tuned model, but also whether there was a difference between the model steered with the smaller dataset and the one steered with the larger dataset.

So, did it really make a difference?

To answer that, we placed three separate setups side by side: on one side was the base model, on another side was the model fine-tuned with the small dataset, and on the other side was the model fine-tuned with the large dataset.

So we did not look at it only as “before and after.” We also wanted to see whether there was a difference between the version steered with less data and the one steered with more data.

Test setup: base model, small-dataset fine-tune, large-dataset fine-tune

We gave the same kinds of questions to all three models and looked at the outputs side by side. That let us evaluate not only whether the fine-tuned model was better, but also how much dataset size mattered in this process.

We ran the tests on two separate sets: we prepared 20-question and 50-question tests. Then we ran all three models on those sets.

In the 20-question test, the results were:
base model 43.3,
fine-tuned model with the small dataset 78,
fine-tuned model with the large dataset 85.3.

In the 50-question test, the table was:
base model 65.9,
fine-tuned model with the small dataset 83.7,
fine-tuned model with the large dataset 84.1.

Even those numbers alone said a lot. After fine-tuning, the model’s behavior on the Unity side clearly became more coherent. And even the improvement that came with the smaller dataset was already quite strong.

How did we prepare the questions and evaluate the outputs?

We prepared separate question sets for testing. We did not want to feed back the examples used in training. Because in that case, it becomes harder to see whether the model is really generalizing. So we used questions that followed a similar logic, but were prepared separately for the test.

Then we gave those questions to the models and evaluated the answers we got back. What we looked at was not only “did it write something?” We looked at whether what it wrote was useful, whether it was logical, whether it was readable, and whether it actually answered the question.

What did fine-tuning really change?

The fine-tuned models gave better results than the base model. In other words, the model’s behavior on the Unity side really changed. A clear difference emerged in giving answers that fit the question better, producing more useful code, and behaving more consistently.

What kind of difference appeared between the base model and the fine-tuned model?

The biggest difference showed up here.

The base model was not empty. We were already working with a model that could write code. On some questions it produced useful things too. But as the task moved closer to Unity, that general knowledge was no longer enough. Some answers looked decent, but did not fully fit the context. In some places there was structure, but in practice it was not as clean as a developer would expect. In some places the model signaled what it was trying to do, but its Unity-specific reflexes were not strong enough.

After fine-tuning, the model started to carry Unity-style question formats, usage patterns, and code structures more effectively. What changed was not only that it “wrote more code.” It started giving more fitting, more accurate, and more domain-native answers.

In short, the model had general code knowledge, but after fine-tuning, that knowledge became more usable on the Unity side.

How wide was the gap between the small dataset and the large dataset?

Even the small dataset made a serious difference.

The first major change came between the base model and the fine-tuned model. The moment we pulled the model toward the Unity side, we already got a clear improvement. Once the larger dataset entered the picture, that improvement increased a bit more, but the biggest jump came in the first stage.

In the 20-question test, the score rose to 78 with the small dataset, and to 85.3 with the large dataset.
In the 50-question test, the small dataset gave 83.7, while the large dataset gave 84.1.

So the larger dataset helped, but the gap did not become a cliff in every case. In some places it gave a strong increase, in others it stayed more limited.

Why does enlarging the dataset not always bring the same level of extra gain?

Because data quantity by itself is not magical.

Feeding a model more data from the same domain usually helps, but that gain does not always grow linearly. At first, when you make the missing domain visible to the model, you get a serious jump. Then as the data grows, the gains continue, but not at the same rate. At some point, the model has already seen the main patterns of that domain; extra data refines it more, stabilizes it more, but does not create as large a jump as that first break. [2][6][8]

Something like that happened on our side too. The first steering step created the big change. The later increase mostly strengthened that difference.

What do these results tell us?

What came out of this for us was quite clear: the difference does not come only from model selection.

Of course, choosing a good model at the start matters. Starting with a model already suitable for code generation made the work easier. But it is incomplete to tie the whole difference only to that. Because what really created the change was feeding the model the right domain data.

So “a good model” alone was not enough. We had to push that good model into the domain we actually needed. The dataset we built with Unity-specific examples, question formats, and code structures became decisive for exactly that reason.

What really makes a model more usable?

There is no one-word answer here. A few things stack on top of each other.

The right base model matters.
The right domain data matters.
How you give the data to the model matters.
How you build the fine-tuning setup matters.
Even how you test it matters.

So what makes a model more usable is not only “a strong model.” It is the meeting point of a strong model with the right domain, the right data format, and the right training flow.

Conclusion

What makes a model better in a specific domain is not only the model itself, but the right data, the right setup, and the right steering.

That is exactly where fine-tuning becomes useful. If you can show the right domain to the model in the right way, you can make a general model far more fitting and far more useful.

References

[1] B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. Canton Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve, Code Llama: Open Foundation Models for Code, 2023.
https://arxiv.org/abs/2308.12950

[2] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom, Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023.
https://arxiv.org/abs/2307.09288

[3] Meta Llama, CodeLlama-7b-hf Model Card, Hugging Face, accessed: 2026.
https://huggingface.co/meta-llama/CodeLlama-7b-hf

[4] Unity Technologies, Unity Documentation, accessed: 2026.
https://docs.unity.com/

[5] Unity Technologies, Unity Scripting API, accessed: 2026.
https://docs.unity3d.com/6000.0/Documentation/ScriptReference/index.html

[6] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, LoRA: Low-Rank Adaptation of Large Language Models, 2021.
https://arxiv.org/abs/2106.09685

[7] Hugging Face, LoRA, PEFT Documentation, accessed: 2026.
https://huggingface.co/docs/peft/developer_guides/lora

[8] Hugging Face, PEFT Documentation, accessed: 2026.
https://huggingface.co/docs/peft/index