| --- |
| datasets: |
| - tner/tweetner7 |
| metrics: |
| - f1 |
| - precision |
| - recall |
| model-index: |
| - name: tner/roberta-large-tweetner7-selflabel2021 |
| results: |
| - task: |
| name: Token Classification |
| type: token-classification |
| dataset: |
| name: tner/tweetner7 |
| type: tner/tweetner7 |
| args: tner/tweetner7 |
| metrics: |
| - name: F1 (test_2021) |
| type: f1 |
| value: 0.6460286973223365 |
| - name: Precision (test_2021) |
| type: precision |
| value: 0.6315440689198144 |
| - name: Recall (test_2021) |
| type: recall |
| value: 0.6611933395004626 |
| - name: Macro F1 (test_2021) |
| type: f1_macro |
| value: 0.5944660768713126 |
| - name: Macro Precision (test_2021) |
| type: precision_macro |
| value: 0.5801646971717881 |
| - name: Macro Recall (test_2021) |
| type: recall_macro |
| value: 0.6174983598336771 |
| - name: Entity Span F1 (test_2021) |
| type: f1_entity_span |
| value: 0.7857183209988137 |
| - name: Entity Span Precision (test_2020) |
| type: precision_entity_span |
| value: 0.7680583167660703 |
| - name: Entity Span Recall (test_2021) |
| type: recall_entity_span |
| value: 0.8042095524459351 |
| - name: F1 (test_2020) |
| type: f1 |
| value: 0.6475365457498646 |
| - name: Precision (test_2020) |
| type: precision |
| value: 0.6768534238822863 |
| - name: Recall (test_2020) |
| type: recall |
| value: 0.6206538661131292 |
| - name: Macro F1 (test_2020) |
| type: f1_macro |
| value: 0.6064934754479069 |
| - name: Macro Precision (test_2020) |
| type: precision_macro |
| value: 0.63365172906493 |
| - name: Macro Recall (test_2020) |
| type: recall_macro |
| value: 0.5889063993107413 |
| - name: Entity Span F1 (test_2020) |
| type: f1_entity_span |
| value: 0.7663146493365827 |
| - name: Entity Span Precision (test_2020) |
| type: precision_entity_span |
| value: 0.8012457531143827 |
| - name: Entity Span Recall (test_2020) |
| type: recall_entity_span |
| value: 0.7343020238713025 |
|
|
| pipeline_tag: token-classification |
| widget: |
| - text: "Get the all-analog Classic Vinyl Edition of `Takin' Off` Album from {@herbiehancock@} via {@bluenoterecords@} link below: {{URL}}" |
| example_title: "NER Example 1" |
| --- |
| # tner/roberta-large-tweetner7-selflabel2021 |
|
|
| This model is a fine-tuned version of [roberta-large](https://huggingface.co/roberta-large) on the |
| [tner/tweetner7](https://huggingface.co/datasets/tner/tweetner7) dataset (`train` split). This model is fine-tuned on self-labeled dataset which is the `extra_2021` split of the [tner/tweetner7](https://huggingface.co/datasets/tner/tweetner7) annotated by [tner/roberta-large](https://huggingface.co/tner/roberta-large-tweetner7-2020)). Please check [https://github.com/asahi417/tner/tree/master/examples/tweetner7_paper#model-fine-tuning-self-labeling](https://github.com/asahi417/tner/tree/master/examples/tweetner7_paper#model-fine-tuning-self-labeling) for more detail of reproducing the model. |
| Model fine-tuning is done via [T-NER](https://github.com/asahi417/tner)'s hyper-parameter search (see the repository |
| for more detail). It achieves the following results on the test set of 2021: |
| - F1 (micro): 0.6460286973223365 |
| - Precision (micro): 0.6315440689198144 |
| - Recall (micro): 0.6611933395004626 |
| - F1 (macro): 0.5944660768713126 |
| - Precision (macro): 0.5801646971717881 |
| - Recall (macro): 0.6174983598336771 |
|
|
|
|
|
|
| The per-entity breakdown of the F1 score on the test set are below: |
| - corporation: 0.5021008403361344 |
| - creative_work: 0.4589000591366056 |
| - event: 0.45184799583550234 |
| - group: 0.602966540186271 |
| - location: 0.667091836734694 |
| - person: 0.8345784418356457 |
| - product: 0.6437768240343348 |
| |
| For F1 scores, the confidence interval is obtained by bootstrap as below: |
| - F1 (micro): |
| - 90%: [0.63733724830433, 0.6556095472315113] |
| - 95%: [0.6353273787551952, 0.6574352280031737] |
| - F1 (macro): |
| - 90%: [0.63733724830433, 0.6556095472315113] |
| - 95%: [0.6353273787551952, 0.6574352280031737] |
| |
| Full evaluation can be found at [metric file of NER](https://huggingface.co/tner/roberta-large-tweetner7-selflabel2021/raw/main/eval/metric.json) |
| and [metric file of entity span](https://huggingface.co/tner/roberta-large-tweetner7-selflabel2021/raw/main/eval/metric_span.json). |
| |
| ### Usage |
| This model can be used through the [tner library](https://github.com/asahi417/tner). Install the library via pip. |
| ```shell |
| pip install tner |
| ``` |
| [TweetNER7](https://huggingface.co/datasets/tner/tweetner7) pre-processed tweets where the account name and URLs are |
| converted into special formats (see the dataset page for more detail), so we process tweets accordingly and then run the model prediction as below. |
| |
| ```python |
| import re |
| from urlextract import URLExtract |
| from tner import TransformersNER |
| |
| extractor = URLExtract() |
| |
| def format_tweet(tweet): |
| # mask web urls |
| urls = extractor.find_urls(tweet) |
| for url in urls: |
| tweet = tweet.replace(url, "{{URL}}") |
| # format twitter account |
| tweet = re.sub(r"\b(\s*)(@[\S]+)\b", r'\1{\2@}', tweet) |
| return tweet |
| |
|
|
| text = "Get the all-analog Classic Vinyl Edition of `Takin' Off` Album from @herbiehancock via @bluenoterecords link below: http://bluenote.lnk.to/AlbumOfTheWeek" |
| text_format = format_tweet(text) |
| model = TransformersNER("tner/roberta-large-tweetner7-selflabel2021") |
| model.predict([text_format]) |
| ``` |
| It can be used via transformers library but it is not recommended as CRF layer is not supported at the moment. |
| |
| ### Training hyperparameters |
| |
| The following hyperparameters were used during training: |
| - dataset: ['tner/tweetner7'] |
| - dataset_split: train |
| - dataset_name: None |
| - local_dataset: {'train': 'tweet_ner/2021.extra.tner/roberta-large-2020.txt', 'validation': 'tweet_ner/2020.dev.txt'} |
| - model: roberta-large |
| - crf: True |
| - max_length: 128 |
| - epoch: 30 |
| - batch_size: 32 |
| - lr: 1e-05 |
| - random_seed: 0 |
| - gradient_accumulation_steps: 1 |
| - weight_decay: 1e-07 |
| - lr_warmup_step_ratio: 0.15 |
| - max_grad_norm: 1 |
| |
| The full configuration can be found at [fine-tuning parameter file](https://huggingface.co/tner/roberta-large-tweetner7-selflabel2021/raw/main/trainer_config.json). |
| |
| ### Reference |
| If you use the model, please cite T-NER paper and TweetNER7 paper. |
| - T-NER |
| ``` |
| |
| @inproceedings{ushio-camacho-collados-2021-ner, |
| title = "{T}-{NER}: An All-Round Python Library for Transformer-based Named Entity Recognition", |
| author = "Ushio, Asahi and |
| Camacho-Collados, Jose", |
| booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations", |
| month = apr, |
| year = "2021", |
| address = "Online", |
| publisher = "Association for Computational Linguistics", |
| url = "https://aclanthology.org/2021.eacl-demos.7", |
| doi = "10.18653/v1/2021.eacl-demos.7", |
| pages = "53--62", |
| abstract = "Language model (LM) pretraining has led to consistent improvements in many NLP downstream tasks, including named entity recognition (NER). In this paper, we present T-NER (Transformer-based Named Entity Recognition), a Python library for NER LM finetuning. In addition to its practical utility, T-NER facilitates the study and investigation of the cross-domain and cross-lingual generalization ability of LMs finetuned on NER. Our library also provides a web app where users can get model predictions interactively for arbitrary text, which facilitates qualitative model evaluation for non-expert programmers. We show the potential of the library by compiling nine public NER datasets into a unified format and evaluating the cross-domain and cross- lingual performance across the datasets. The results from our initial experiments show that in-domain performance is generally competitive across datasets. However, cross-domain generalization is challenging even with a large pretrained LM, which has nevertheless capacity to learn domain-specific features if fine- tuned on a combined dataset. To facilitate future research, we also release all our LM checkpoints via the Hugging Face model hub.", |
| } |
| |
| ``` |
| - TweetNER7 |
| ``` |
| |
| @inproceedings{ushio-etal-2022-tweet, |
| title = "{N}amed {E}ntity {R}ecognition in {T}witter: {A} {D}ataset and {A}nalysis on {S}hort-{T}erm {T}emporal {S}hifts", |
| author = "Ushio, Asahi and |
| Neves, Leonardo and |
| Silva, Vitor and |
| Barbieri, Francesco. and |
| Camacho-Collados, Jose", |
| booktitle = "The 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing", |
| month = nov, |
| year = "2022", |
| address = "Online", |
| publisher = "Association for Computational Linguistics", |
| } |
| |
| ``` |
| |
| |
| |
