flaubert_base_cased-ft-AS13_stgdir-100

This model is a fine-tuned version of flaubert/flaubert_base_cased on the dataset described below.

It is the best model in our French stage direction classification collection.

It achieves the following results on the evaluation set:

• Loss: 0.5948
• Accuracy: 0.8764

Model description

Fine-tuned for stage direction classification in French, using the dataset at https://nakala.fr/10.34847/nkl.fde37ug3.

The categorization scheme and rationale are described in the following publication:

Schneider, Alexia., & Ruiz Fabo, Pablo. (2024). Stage direction classification in French theater: Transfer learning experiments. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024) (pp. 278–286). Association for Computational Linguistics. https://aclanthology.org/2024.latechclfl-1.28/

Intended uses & limitations

Stage direction classification in French.

Training and evaluation data

Stage direction dataset annotated with 13 categories by Alexia Schneider & Pablo Ruiz.

The categories were derived from those available at FreDraCor (and originally in the Théâtre Classique platform).

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 40

Training results

Results on held-out data:

Label	Precision	Recall	F1-score	Support
action	0.8910	0.8580	0.8742	486
aggression	0.7439	0.8133	0.7771	75
aparte	1.0000	0.3571	0.5263	14
delivery	0.8994	0.7559	0.8214	213
entrance	0.8824	0.7031	0.7826	128
exit	0.7978	0.9132	0.8516	242
interaction	0.8077	0.8235	0.8155	102
movement	0.6000	0.6807	0.6378	119
music	0.9588	0.9671	0.9629	577
narration	0.6712	0.8167	0.7368	120
object	0.8233	0.8510	0.8369	208
setting	0.9091	0.8421	0.8743	190
toward	0.9646	0.9710	0.9678	449
Accuracy	—	—	0.8720	2923
Macro avg	0.8422	0.7964	0.8050	2923
Weighted avg	0.8775	0.8720	0.8722	2923

Training details:

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.1258	1.0	585	0.5231	0.8464
0.477	2.0	1170	0.4582	0.8777
0.3661	3.0	1755	0.5354	0.8717
0.2986	4.0	2340	0.5211	0.8807
0.2472	5.0	2925	0.5948	0.8764

Framework versions

• Transformers 4.57.2
• Pytorch 2.9.0+cu126
• Datasets 4.0.0
• Tokenizers 0.22.1