📝 Scaling FineWeb to 1000+ languages: Step 1: finding signal in 100s of evaluation tasks

Following the strong community reception of our FineWeb English dataset, we have been hard at work on a multilingual version, which will cover 1000+ languages (that we hope to release soon!).

However, we quickly encountered a significant challenge: how can one effectively evaluate models across different languages during training?

For English, it's straightforward: we can utilize well-established benchmarks like MMLU or HellaSwag, widely used by most labs and implemented in all the major evaluation frameworks. Unfortunately, non-English tasks are often scarce and lack broader community validation and, when available, are frequently of questionable quality: many are machine-translated and may even include English words in their formulations. Additionally, they are often unsuitable for early pre-training evaluation due to suboptimal task formulations and/or too high difficulty resulting in random scores.

To address these challenges, we developed a scalable and data-driven framework for evaluation task selection, which allows anyone to choose strong model evaluations for their language from existing tasks! We then applied this framework to a set of 9 diverse languages, resulting in the creation of FineTasks - a comprehensive and diverse multilingual evaluation suite.

In this blog post, we discuss:

1. Our data-driven process to create a multilingual evaluation suite: FineTasks
2. Results of evaluating 35 major open and closed-source models on FineTasks
3. A guide for extending FineTasks to your target language

What Makes a Task "Fine"?

Covering all 7000+ languages spoken over the world would be monumental endeavor, so we settled on using 9 languages that offered diversity in script, language family and resource availability: Chinese, French, Arabic, Russian, Thai, Hindi, Turkish, Swahili, and Telugu.

For these languages, we collected all available tasks that we could find, implementing a total of 185 tasks across languages in LightEval, HuggingFace's model evaluation library.

Then, we began task selection with two primary goals: ensuring evaluation diversity, and making sure each task provided a reliable signal during pre-training.

For evaluation diversity, we aimed to assess a broad range of model capabilities, including:

• Reading comprehension (RC): Understanding provided context and answering questions based on it.
• General knowledge (GK): Answering questions about facts from various fields without added context.
• Natural Language Understanding (NLU): Comprehending the semantics of provided input.
• Common-sense reasoning (RES): Demonstrating the ability to perform simple reasoning requiring embodied knowledge.
• Generative tasks: Ability to generate text in the target language without the "help" of multiple choice options.

We consider that tasks provide a reliable signal if they provide a dependable score. This means the score should be above the random baseline, increase as training progresses, show low variability across different seeds, and provide consistent model ranking at each training stepFor similar sized models trained with the same hyperparameters on the same amount of data..

Finding how much signal our tasks give during pre-training

To thoroughly examine the signal our tasks provide, we trained many 1.5B parameter models for each language, using 30B tokens from subsets of the supported languages of the five largest openly available multilingual web datasets. These models were trained with the same hyperparameters and tokenizer. We then evaluated them at regular checkpoint intervals on the collected tasks (with no instruction and no system prompt in a 0-shot setting).

This process required multiple evaluation runs for each task due to iterations on its implementation, resulting in a total of 73 000 GPU hours consumed 🔥!

With 49 models trained we could finally define what a reliable signal means to us!

Monotonicity

One of our core requirements for a task is that it can be learned from training data and this learning can be gradually observed as the training progresses. Without this improvement through time, it's uncertain whether there will ever be an improvement in the future.

To measure this, we used the Spearman rank correlation to quantify the correlation between steps and score. Spearman rank correlation can capture monotonicity even when scores don't evolve linearly with the number of steps. We required each task to have at least an average correlation of 0.5 over all model training runs.

Low noise

When comparing model performance on tasks, we need to consider whether differences are due to evaluation noise or genuine performance variations.

Noise can arise from the stochastic processes involved in model training, such as random token sampling, data shuffling, or model initialization. To measure how sensitive each task is to this noise, we trained four additional models on our own monolingual corpora (unfiltered CommonCrawl data in each language) using different seeds.

For each task, we computed:

1. First, a standard deviation of model scores for every step (approximately every 1B tokens), which we call the per-step-std.
2. Then, to obtain a global variability measurement, we averaged all the per-step-std values to get the avg-std over the full training. We assume this value is an upper-bound across model architectures and training datasets (as it was approximated by models trained on a "dirtier" dataset, therefore with higher variability).
3. Finally, we computed the signal-to-noise ratio (SNR) as the main metric for task variability. We calculate SNR as the mean score at 30B tokens of all runs divided by the avg-std. This metric measures how significant the overall score is relative to the score variations (noise).

We aimed for each task to have an SNR > 20. The only exception to this rule are generative tasks, which typically have relatively low SNR, but are still worth including as they provide insights into how the model behaves when prompted to generate unconstrained (without answer options). In a multilingual setting, this is particularly relevant as some models trained on multiple languages can exhibit high task scores but then suddenly reply in the wrong language for generative tasks!

Assuming model performance is normally distributed across different seeds, we want the benchmark-run performance to be at least 3 final-stds above the benchmark random baseline. This would mean that 99.85% of seed scores are above the random baseline (formally, benchmark-run performance - benchmark random baseline > 3 * final-std).

Non-Random Performance

Many model capabilities are acquired later in training, thus many tasks (especially harder ones, such as math-related ones) show baseline-level performance for an extended period. While these tasks are useful, they're not ideal for early pre-training evaluation, and we did not want to keep them for this setting.

We first computed the baseline random performance of the task (as the sum of 1/n_choices for all samples for multiple choice questions, and as zero for generative evaluations). Then we calculated the task's distance from the baseline as the maximum score across all models minus the baseline.

Model Ordering Consistency

Let's not forget that the main goal of these evaluations is to compare models and datasets!

In the future, we want to use these evaluations to select the best datasets for full model pretraining. This means our tasks should rank datasets trained using very few tokens (we typically run data ablations on 30B tokens), in the same order as they would when trained for longer, after significantly more steps.

In other words, we would like tasks to have predictive capability regarding future performance during pre-training: if pre-training dataset A outperforms pre-training dataset B at 30 billion tokens, we would like this trend to continue at 300 billion tokens.

Proving this is inherently impossible, but there is a necessary preliminary condition that we can test for: for the results to be consistent at large scales, they must also first show consistency at smaller scales!

To measure this consistency in task ordering, we computed the average Kendall's Tau of models ranking between every two consecutive steps. We only considered steps starting after 15B tokens of pre-training, as we found orderings before the range incredibly noisy. A high value of this metric indicates that the ordering remains consistent as training progresses.

We had no strict minimum value requirement for this property, instead using it to establish comparisons between tasks.

Important properties of evaluation impacting stability

Now that we covered what we were looking for in our tasks, let's examine two important aspects that can affect the above properties: task formulations and metric choice.

Both of these aspects are thoroughly described and studied in the brilliant OLMES paper, which greatly inspired our work.

Task Formulations

The way tasks are presented to the model is crucial, particularly for multiple-choice (MC) tasks. In these scenarios, we must carefully determine how the choices are displayed and what the model is expected to predict.

There are two common approaches: Cloze Formulation (CF) and Multi-Choice Formulation (MCF). In CF, choices are not provided in context, allowing the model to predict each option directly. In contrast, MCF presents the choices in the prompt, using A/B/C/D prefixes, with the targets being those letter prefixes.

It's important to know that:

• The choice of formulation significantly impacts task scores .
• Both formulations behave very differently during training. As noted by both OLMES and DataComp-LM, when employing MCF, task scores initially show random performance over extended training periods before experiencing a sudden increase. Conversely, with CF, task scores improve right from the beginning but tend to plateau relatively early.

Therefore, we decided to utilize CF for task selection and MCF for later evaluation of major open source models, as they have generally undergone enough training for these evaluations to have a signal.

Metrics

As the targets in CF of multiple choice tasks are choices themselves, each target can have a different number of tokens, characters, and unconditional probability (probability of generating the choice without a context prefix).

Measuring accuracy without normalization would have the models prefer answers with fewer tokens, for example.

To account for this, we consider the following accuracy variations:

• Accuracy :
  acc = \underset{i}{\arg\max}(ln(P (a_i|q)))
• Accuracy normalized over character length :
  acc_char = \underset{i}{\arg\max}\frac{ln(P (a_i|q))}{num\_characters(a_i)}
• Accuracy normalized over token length :
  acc_token = \underset{i}{\arg\max}\frac{ln(P (a_i|q))}{num\_tokens(a_i)}
• PMI Accuracy :
  acc_pmi = \underset{i}{\arg\max}ln\frac{P (a_i|q)}{P (a_i|u)}, where u =''Answer:''

Where a_i is the answer choice i, q is a question prompt and P (a_i|q) is the probability of having a_i follow q. For more details see and .

acc_pmi metric measures how much more likely a model is to predict A_i if provided with question context compared to if there was no context at all. This can be useful if the correct choice contains generally unlikely tokens, making the model less likely to choose such an answer.

For our generative tasks on the other hand, we used the following metrics:

• prefix_match: Exact match where only the prefix of the answer must match
• f1: F1 score computed over predicted/gold words extracted using a word tokenizer

For both generative metrics, minor preprocessing is applied to remove articles and punctuation, and lowercase the text.

The Fine selection

With our goals and evaluation setup properly defined, we proceeded with task selection!

We reviewed tasks one by one, choosing based on the quantified properties. For each language, we aimed to have at least one task for each of the four categories outlined above. Additionally we wanted to have at least 1 generative task for each language.

In cases where multiple versions of a task existed (e.g., MMLU with different translation methods or native versions), we prioritized native versions as long as their metrics were reasonable, followed by human translations of English tasks. If no such version was available, we made our selection entirely based on metrics.

Thus, after removing about half of the tasks, we arrived at 96 final ones, forming "FineTasks."

Explore tasks

Use the dropdowns below to navigate the list of tasks and how different metrics affect them.

All tasks from the selection comply with the criteria outlined in previous sections, with the only exception being indicqa_tel, which we chose to include to ensure we had at least one generative task for Telugu. Overall we managed to cover all task categories for each language (the only exception being Thai Reasoning, where all tasks were unfortunately too noisy with low monotonicity to consider them).

One of the biggest surprises was that some tasks, even when translated using the same method, were reliable in one language but not in others. This was evident with xWinograd, which worked quite well for Russian but did not meet our conditions for French. An even more extreme example was XNLI, which performed well for 6 out of 7 languages, failing to satisfy the reliability properties for Chinese. We had to test four different implementations before finding a reliable version, which, interestingly, was the only one that was created by native speakers and not machine translated.

Feel free to use the dropdowns below to explore the evolution of scores over training for all tested tasks and metrics.

Metrics recommendation

Selecting the best evaluation metrics proved to be a challenging task. Not only is there no single metric that consistently outperforms the rest, but we often encountered situations where one metric had better monotonicity while another had a higher signal-to-noise ratio. In such cases, we typically made our decision based on the selected metric for tasks' implementation in a different language. We are aware that such hand-picking is often not possible and thus offer the following recommendations:

Multichoice Tasks

• We found base accuracy to perform well for tasks with answer options varying subtly (e.g. Yes/No/Also), particularly NLI tasks. In such cases, where the answer options are often each a single token, the base accuracy is advisable to use.
• While OLMES recommends using PMI for tasks with unusual words, we found PMI to be highly effective for "difficult" reasoning and knowledge tasks like AGIEVAL or MMLU. In these cases, PMI provided the best results and was often the only metric delivering performance above random. That said, PMI was, on average, the weakest metric across all other tasks, while also being two times more expensive to compute. We therefore only recommend its use for complex reasoning and knowledge tasks.
• The metrics we found to be most reliable overall were length normalization metrics (token or character-based). However, the best choice was dependent on language, rather than being consistent for a given task. Due to that, we recommend using the maximum of acc_char and acc_token for the most reliable results.Note that acc_token is heavily tokenizer dependent. On our ablations all models were trained using the same tokenizer.

Generative Tasks

For generative metrics, the choice is clearer: we suggest using the F1 score unless exact matching is required, as in math-related tasks. F1 is generally less noisy and more resilient to small changes in the generations.

Open/Closed Source models tackle FineTasks

Since we spent a lot of time and compute on task selection, we were interested in how well major open-source models would do on FineTasks. Given that our evaluation suite primarily targets pretrained models, we focused on these, with a few exceptions for models that don't offer a base (pretrained) version. These exceptions were included mainly out of curiosity, and their results should be interpreted with caution. Such models may significantly outperform other models due to the inclusion of supervised fine-tuning (SFT) data.

To assess the multilingual performance disparity between open-source and closed-source models, we expanded our selection by adding a closed source model: gpt-4o-mini.

As outlined in the task formulations, we are using MCF for this evaluation and employing a 5-shot approach, as recommended by OLMES (and made possible by the large context size of the models).

Computing a global "multilingual" score

In the previous sections, we treated each task independently. However, to determine an overall "multilingual" score of a model, we need to aggregate the results from these tasks. We begin by rescaling the individual task scores in line with the OpenLLM leaderboard . Then, we average the scores across task types (GK, RES, etc) for each language separately. To compute the score for each language, we take the average of the task type scores.We first average by task type to properly measure all model capabilities without letting a single category dominate.

For the final global "multilingual" score we followed a different approach. Instead of averaging the language scores directly, we ranked the model's performance across languages in comparison to other models and then averaged those rank scores. This method ensures that the result reflects the overall model's performance across all languages, preventing an exceptionally high score in one language from skewing the final outcome.

FineTasks Results

After spending even more compute 🔥 on evaluating the selected models, we gathered the results in the following table. Here are our insights:

Qwen family of models takes both top spots!

The Qwen models perform exceptionally well, taking both first and second place with their 72B and 32B versions. Their key strength appears to be in handling high- and mid-resource languages (particularly Chinese), where they consistently ranked first. However, they struggled with lower-resource languages, especially Swahili and Telugu, where their performance lagged.

General Knowledge: The curse of monolingual models

The most surprising finding from our evaluation is how models explicitly trained to specialize in a narrow set of languages — like Sarvam-2B-v0.5 for Telugu, or Typhoon-v1.5-8B for Thai — tend to perform exceptionally well on generative tasks, while falling short when it comes to reasoning and general knowledge (GK) tasks, oftentimes getting close to random performance. We hypothesize two explanations: The models haven't undergone extensive enough training to be able to understand the MCF format or the higher exposure to various languages and especially English allows the non-specialized models to perform better at such GK/RES tasks. We note that good generative task performance reveals a good understanding of the target language.

The only exceptions to this rule are typhoon-v1.5-72b and Yi-1.5-34B, both tackling the RES/GK tasks well and managing to rank in the top 4 for their respective languages. We note that typhoon-v1.5-72b is based on Qwen models, and that Yi also included English in its training data.

A lower resource winner: Gemma-2

Although it didn't take first place, Gemma2 performed really well in the multilingual domain, especially considering its size. It showed consistent results across all the languages we tested, excelling in low-resource languages like Telugu and Swahili. For anyone working with low-resource languages, we highly recommend Gemma-2 as a strong option.

Is there even a gap between open and closed source models?

As mentioned in the beginning, comparing closed-source models requires extra caution. These models often undergo extensive supervised fine-tuning (SFT), employ highly optimized prompting techniques, and may even generate multiple responses and select the best one. Despite these advantages, the o4-mini ranks only just above the medium-sized 27B Gemma-2. Based on this evidence, we believe that the gap between open-source and closed-source models is very narrow, if not entirely negligible.

Evaluating on FineTasks

If you would like to evaluate your models on FineTasks and expand the above table we made it easy for you. Simply run the following command with your model of choice:

lighteval accelerate\
    --model_args vllm,pretrained=model_name,pairwise_tokenization=True \
    --custom_task lighteval.tasks.multilingual.tasks \
    --tasks 'examples/tasks/finetasks/{cf,mcf}/{ara,fra,rus,tur,swa,hin,tel,tha,zho}' \
    --max_samples '1000'

Can we cover all the languages of the world together?

FineTasks is just the beginning of our multilingual journey. As a first step in the creation of the future FineWeb multilingual release, we are using this evaluation setup to curate a high quality pretraining dataset covering a large number of languages. You can expect more news from us soon! We plan to also continue working to make evaluation in non-English domains as seamless as it is in English—and we need your help to achieve that!

LightEval now supports over 550 tasks across various non-English languages, making it the evaluation framework with the best multilingual coverage available. However, there's still much more to do. For many languages, no tasks exist yet, despite our ongoing work. This is where we believe the strong Hugging Face community can make a difference.

We've made it incredibly easy to contribute new tasks, by developing a templating system which supports most of the popular task types, while maintaining authenticity of native language use, right down to correct punctuation. Even if you aren't able to contribute full evaluation tasks, you can still help. Many languages currently lack translations for anchor words used in evaluation, leaving hundreds of tasks unusable. You can help fill this gap by adding them following our mini guide.

We're looking forward to revisiting this analysis in the future, not with just 9 languages, but at least 50—thanks to community contributions! Let's level the playing field between English and other languages together! 🤗