Stratified LLM Subsets: Balanced Training Data at 100K-1M Scale
text
Stratified LLM Subsets: Balanced Training Data at 100K-1M Scale
new_line
new_line
Released three training datasets using embedding-based k-means clustering to create balanced subsets from large-scale corpora:
text
Released three training datasets using embedding-based k-means clustering to create balanced subsets from large-scale corpora:
new_line
new_line
Interactive cluster visualization:
text
Interactive cluster visualization:
new_line
https://amanpriyanshu.github.io/Stratified-LLM-Subsets-100K-1M-Scale/
https://amanpriyanshu.github.io/Stratified-LLM-Subsets-100K-1M-Scale/
link
new_line
new_line
Pre-Training (FineWeb-Edu + Proof-Pile-2)
text
Pre-Training (FineWeb-Edu + Proof-Pile-2)
new_line
https://huggingface.co/datasets/AmanPriyanshu/stratified-kmeans-diverse-pretraining-100K-1M
AmanPriyanshu/stratified-kmeans-diverse-pretraining-100K-1M
dataset
resource
https://huggingface.co/datasets/AmanPriyanshu/stratified-kmeans-diverse-pretraining-100K-1M
new_line
new_line
Instruction-Following (Tulu-3 + Orca AgentInstruct)
text
Instruction-Following (Tulu-3 + Orca AgentInstruct)
new_line
https://huggingface.co/datasets/AmanPriyanshu/stratified-kmeans-diverse-instruction-following-100K-1M
AmanPriyanshu/stratified-kmeans-diverse-instruction-following-100K-1M
dataset
resource
https://huggingface.co/datasets/AmanPriyanshu/stratified-kmeans-diverse-instruction-following-100K-1M
new_line
new_line
Reasoning (Llama-Nemotron with sqrt balancing)
text
Reasoning (Llama-Nemotron with sqrt balancing)
new_line
https://huggingface.co/datasets/AmanPriyanshu/stratified-kmeans-diverse-reasoning-100K-1M
AmanPriyanshu/stratified-kmeans-diverse-reasoning-100K-1M
dataset
resource
https://huggingface.co/datasets/AmanPriyanshu/stratified-kmeans-diverse-reasoning-100K-1M
new_line
new_line
Methodology: k-means clustering on Snowflake Arctic-embed-xs embeddings (100 iterations), selecting cluster centroids as representatives. Balancing applied to imbalanced datasets to reduce category dominance.
text
Methodology: k-means clustering on Snowflake Arctic-embed-xs embeddings (100 iterations), selecting cluster centroids as representatives. Balancing applied to imbalanced datasets to reduce category dominance.
new_line
new_line
Available at 50k, 100k, 250k, 500k, and 1M scales.
text
Available at 50k, 100k, 250k, 500k, and 1M scales.
Stratified LLM Subsets: Balanced Training Data at 100K-1M Scale

Released three training datasets using embedding-based k-means clustering to create balanced subsets from large-scale corpora:

Interactive cluster visualization:
https://amanpriyanshu.github.io/Stratified-LLM-Subsets-100K-1M-Scale/

Pre-Training (FineWeb-Edu + Proof-Pile-2)
https://huggingface.co/datasets/AmanPriyanshu/stratified-kmeans-diverse-pretraining-100K-1M

Instruction-Following (Tulu-3 + Orca AgentInstruct)
https://huggingface.co/datasets/AmanPriyanshu/stratified-kmeans-diverse-instruction-following-100K-1M

Reasoning (Llama-Nemotron with sqrt balancing)
https://huggingface.co/datasets/AmanPriyanshu/stratified-kmeans-diverse-reasoning-100K-1M

Methodology: k-means clustering on Snowflake Arctic-embed-xs embeddings (100 iterations), selecting cluster centroids as representatives. Balancing applied to imbalanced datasets to reduce category dominance.

Available at 50k, 100k, 250k, 500k, and 1M scales.
👍
John6666