aboutteamprojectsblogschat
aboutteamprojectsblogschat
© 2026 ottomind.ai
← Back to blogs

How to Choose the Right Examples in Few-Shot?

10 min read · Jul 17, 2025

In the previous piece, we looked at this: the result in language models cannot be explained only by the knowledge inside the model. The same model, the same task, and the same question can still behave differently under different contexts. Once we moved to the few-shot side, that became even more visible. Because the moment we give examples to the model, we do not only add information; we also affect how the task will be read. To read first part: https://ottomind.ai/blogs/what-changes-between-zero-shot-and-few-shot

And that naturally brought us to another question:
if not every few-shot example creates the same effect, then is it possible to choose those examples more deliberately?

That is exactly what we look at in this piece.

Why is example selection a separate problem?

Few-shot is often described as if it were a single, fixed technique. As if giving the model a few examples is enough to solve the problem. But the more we looked at the experimental side, the more clearly we saw that it does not work that way.

Because for the model, those examples are not neutral pieces sitting inside the prompt. They directly build context. In other words, the model does not only see the question; it also infers how it should approach that question from the examples in front of it. And naturally, that means not every example has the same value. [2][3]

Some examples carry the task more cleanly. Some stay noisier. Some build the right frame for the model. Others do not create the same effect. So the issue is not only “did we use few-shot?” The issue is what we put inside few-shot.

First question: is random example selection enough?

This was the first thing we tried. Because in the first stage, we had already seen that when we changed the few-shot examples themselves, their order, or their correctness, the result changed too. That led us directly to this thought: if examples create that much difference, then choosing them randomly may not be enough.

Put more plainly, the issue became this: giving the model a few examples is one thing; giving the model the right examples is another.

That is why the next step was to make example selection more deliberate.

First approach: selecting better examples with a fixed logic

In our first deliberate selection approach, this is what we did: instead of leaving the examples that would go into few-shot to chance, we selected them beforehand according to a certain criterion and kept that selection fixed. In other words, we did not redefine the examples for each new question; we first analyzed the example pool we had, then selected the question-answer pairs we thought were stronger, and built the few-shot structure on top of them.

The starting point here was simple. If not every few-shot example creates the same effect, then we need to choose the examples the model carries better. To understand that, we first used the model’s previous outputs. We looked at which questions it answered more closely to correct, which questions it handled more consistently, and which examples gave a cleaner direction as context. Then we selected the examples we would use in few-shot based on those observations.

In the first step of this approach, we tried some selection attempts starting from the examples the model had answered incorrectly. But that side did not provide as strong a direction as we expected. So we shifted the focus toward the examples the model had answered correctly, or more closely to correct. That is how we started to build the fixed few-shot pool out of the examples the model read relatively better. In particular, criteria such as closeness to the correct answer and lower deviation became more decisive here.

That is why we call this approach “fixed.” Because the example selection here was deliberate, but still one-time. Once we selected a set of strong examples, those examples continued to be used later as few-shot context for different questions. The aim was not to find new examples for every question, but to build a stronger few-shot set from the start.

What did fixed selection show us?

The first thing this approach showed was that example selection was important enough that it could not be left to chance. Which examples we put into few-shot directly changed the context the model built. That is why a fixed few-shot structure built from examples the model handled better, answered more closely to correct, or behaved more consistently started to provide a more meaningful direction than randomly selected examples.

This was the first point where we saw clearly that the power of few-shot comes not just from giving a few examples, but from the quality of those few examples.

But at the same time, the limits of this approach also started to become visible. Yes, building a fixed structure out of stronger examples worked. But in the end, we were still working with a single example set. A set of examples may be well selected in general, but there is no guarantee that it will build the best context for every new question. Because what we had already seen in the previous piece was this: model behavior is sensitive not only to the presence of few-shot, but also to the relation those examples establish with the question.

So the fixed selection logic showed us the right direction, but it did not take us to the final point.

Do we need different examples for each question?

This was the second big question for us. Because if each question carries a slightly different relation, a different emphasis, and a different closeness structure, then a single fixed few-shot pool may not always be enough.

The dynamic approach emerged exactly at that point.

The idea here was not to choose the few-shot examples once and give the same set to everyone. It was to reselect the closest, most relevant, and most meaningful examples each time a new question arrived. Once you look at it that way, the issue stops being only “giving examples”; it becomes matching the right examples to the right question.

How did we build the dynamic approach?

In this approach, we reselected the examples to be given as few-shot for each question. To do that, we tried to find the question-answer pairs that were semantically closest to the question. So instead of placing just any few examples in front of the model, we placed examples that were closer in content to the question we wanted it to answer at that moment.

To do this, we updated the test infrastructure and used BERT-based similarity selection to measure semantic closeness. That moved the experimental setup beyond a simple zero-shot / few-shot comparison and turned it into a more layered structure that also tested the quality of example selection. [4]

The logic here was simple, but strong: the model should not only see examples; it should also enter a context that is closer to the question it actually has to solve.

Why did the dynamic approach work better?

The reason this approach produced stronger results is actually quite intuitive. If not every question is the same for the model, then the context that works for every question will not be the same either. In the fixed approach, we had a set of examples that was generally well selected. But in the dynamic approach, the context shown to the model was built directly around that specific question. That made the examples not only “good,” but also “relevant.”

The difference here is subtle, but important. The fixed approach relied on selecting high-quality examples. The dynamic approach, on top of that, selected examples that were close to the question. That turned the context from something general into something task-specific. From the model’s perspective, that means reading the task more accurately, seeing similar patterns more easily, and interpreting the new question within a more suitable frame.

And the results showed exactly that. Dynamic selection created a stronger effect than static selection. When we took the semantically closest examples for each question into the few-shot context, we saw the clearest improvement there. The gap widened even more compared with the more random, shuffled, or wrong-answer example structures we had tried at the beginning.

In other words, what became much clearer there was that the real difference comes not from using few-shot, but from making few-shot meaningful on a per-question basis.

What did we learn from this study?

By the end of this study, the clearest thing for us was this: in language models, the result is not determined only by the model itself. Of course, the model’s capacity, the data it was trained on, and its general ability all matter. But the kind of context we put in front of the model becomes just as decisive. This effect becomes much more visible especially on the few-shot side. Because there, the model does not only look at the question; it also reads the examples it saw before the question and builds its new answer through them. [1][2][3]

What we saw here was not only a general conclusion like “few-shot works.” Throughout the study, we saw that:

  • when the few-shot examples themselves changed, the result changed
  • when the same examples were given in a different order, the result changed again
  • even when the correctness of the examples was broken, the effect did not move in a straight line
  • selecting better examples deliberately worked better than random selection
  • reselecting the most relevant examples for each question gave the strongest result

In other words, few-shot is not just a matter of adding a few more lines to the prompt. It is genuinely a context layer that affects how the model interprets the task.

Conclusion

Today, when language models are discussed, the focus often shifts to model size, number of parameters, or the kind of data the model was trained on. Those things matter, of course. But this study showed us something else more clearly: in real use, the result is not determined only by the model. What we ask from the model, how we ask for it, and how we present the task to it become just as important as the model itself. [1][2][3]

That is why few-shot should not be thought of as a shallow operation of just adding a few examples into the prompt. Few-shot actually gives the model a task logic, a solution frame, and a way of transferring context. The more deliberately that context is built, the more the model’s answer changes.

That was what became clear for us by the end of this study: sometimes the thing that makes the real difference is not the model itself, but how we speak to the model.

References

[1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, Language Models are Few-Shot Learners, in Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020, pp. 1877–1901.
https://arxiv.org/abs/2005.14165

[2] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?, in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), 2022.
https://aclanthology.org/2022.emnlp-main.759/

[3] T. Gao, A. Fisch, and D. Chen, Making Pre-trained Language Models Better Few-shot Learners, in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), 2021, pp. 3816–3830.
https://aclanthology.org/2021.acl-long.295/

[4] L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, C. O’Brien, B. Schoelkopf, E. Skowron, and T. Sutawika, A framework for few-shot language model evaluation, version v0.0.1, Zenodo, 2021.
https://zenodo.org/records/5371629