CS4248 Project Group 12 - Modification 2 (POS tags injection)

Instructions on running the model

Note: This is a framework adapted from the AllenNLP package and original credits goes to: https://github.com/grammarly/gector

To run the code first navigate into the gector-master directory

Install all the required packages by:

pip install -r requirements.txt

Note: You might need updated version of VisualStudio C++

The model takes in a set of processed dataset. For a given set of m2 files, you should expand the files to include a .src file and .tgt files using the m2 command (This has already been done for you). What is required next is to use the expanded files to generate an output file using the command:

python utils/preprocess_data.py -s SOURCE -t TARGET -o OUTPUT_FILE

These will be the files that your model will handle. We'll need a train set, a dev set and a test set for our experiments Note: For the purposes of this project, the bea19 dataset has been prepared for you to do the training.

To train the model with non-limited word tokens (No restriction baseline model), run the following command:

python train.py --train_set TRAIN_SET --dev_set DEV_SET --model_dir MODEL_DIR

The model iterations would be dumped in the stated MODEL_DIR directory. (In this case it would be the dirmodels folder)

To run the forward inference process of the model, you would need to call the predict.py script using the command.

python predict.py --model_path MODEL_PATH [MODEL_PATH ...] --vocab_path VOCAB_PATH --input_file INPUT_FILE --output_file OUTPUT_FILE

This would generate an output file which is the corrected text.

Note that the current model is trained using FCE, Wi-Locness, NUCLE dataset. You can find the trained model here.

---

This repository provides code for training and testing state-of-the-art models for grammatical error correction with the official PyTorch implementation of the following paper:

GECToR – Grammatical Error Correction: Tag, Not Rewrite
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, Oleksandr Skurzhanskyi
Grammarly
15th Workshop on Innovative Use of NLP for Building Educational Applications (co-located with ACL 2020)

It is mainly based on AllenNLP and transformers.

Installation

The following command installs all necessary packages:

pip install -r requirements.txt

The project was tested using Python 3.7.

Datasets

All the public GEC datasets used in the paper can be downloaded from here.
Synthetically created datasets can be generated/downloaded here.
To train the model data has to be preprocessed and converted to special format with the command:

python utils/preprocess_data.py -s SOURCE -t TARGET -o OUTPUT_FILE

Pretrained models

Pretrained encoder	Confidence bias	Min error prob	CoNNL-2014 (test)	BEA-2019 (test)
BERT [link]	0.1	0.41	61.0	68.0
RoBERTa [link]	0.2	0.5	64.0	71.8
XLNet [link]	0.2	0.5	63.2	71.2

Note: The scores in the table are different from the paper's ones, as the later version of transformers is used. To reproduce the results reported in the paper, use this version of the repository.

Train model

To train the model, simply run:

python train.py --train_set TRAIN_SET --dev_set DEV_SET \
                --model_dir MODEL_DIR

There are a lot of parameters to specify among them:

• cold_steps_count the number of epochs where we train only last linear layer
• transformer_model {bert,distilbert,gpt2,roberta,transformerxl,xlnet,albert} model encoder
• tn_prob probability of getting sentences with no errors; helps to balance precision/recall
• pieces_per_token maximum number of subwords per token; helps not to get CUDA out of memory

In our experiments we had 98/2 train/dev split.

Training parameters

We described all parameters that we use for training and evaluating here.

Model inference

To run your model on the input file use the following command:

python predict.py --model_path MODEL_PATH [MODEL_PATH ...] \
                  --vocab_path VOCAB_PATH --input_file INPUT_FILE \
                  --output_file OUTPUT_FILE

Among parameters:

• min_error_probability - minimum error probability (as in the paper)
• additional_confidence - confidence bias (as in the paper)
• special_tokens_fix to reproduce some reported results of pretrained models

For evaluation use M^2Scorer and ERRANT.

Text Simplification

This repository also implements the code of the following paper:

Text Simplification by Tagging
Kostiantyn Omelianchuk, Vipul Raheja, Oleksandr Skurzhanskyi
Grammarly
16th Workshop on Innovative Use of NLP for Building Educational Applications (co-located w EACL 2021)

For data preprocessing, training and testing the same interface as for GEC could be used. For both training and evaluation stages utils/filter_brackets.py is used to remove noise. During inference, we use --normalize flag.

	SARI		FKGL
Model	TurkCorpus	ASSET
TST-FINAL [link]	39.9	40.3	7.65
TST-FINAL + tweaks	41.0	42.7	7.61

Inference tweaks parameters:

iteration_count = 2
additional_keep_confidence = -0.68
additional_del_confidence = -0.84
min_error_probability = 0.04

For evaluation use EASSE package.

Note: The scores in the table are very close to those in the paper, but not fully match them due to the 2 reasons:

• in the paper, we reported average scores of 4 models trained with different seeds;
• we merged codebases for GEC and Text Simplification tasks and updated them to the newer version of transformers lib.

Noticeable works based on GECToR

• Improving Sequence Tagging approach for Grammatical Error Correction task [paper][code]
• LM-Critic: Language Models for Unsupervised Grammatical Error Correction [paper][code]

Citation

If you find this work is useful for your research, please cite our papers:

GECToR – Grammatical Error Correction: Tag, Not Rewrite

@inproceedings{omelianchuk-etal-2020-gector,
    title = "{GECT}o{R} {--} Grammatical Error Correction: Tag, Not Rewrite",
    author = "Omelianchuk, Kostiantyn  and
      Atrasevych, Vitaliy  and
      Chernodub, Artem  and
      Skurzhanskyi, Oleksandr",
    booktitle = "Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications",
    month = jul,
    year = "2020",
    address = "Seattle, WA, USA → Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.bea-1.16",
    pages = "163--170",
    abstract = "In this paper, we present a simple and efficient GEC sequence tagger using a Transformer encoder. Our system is pre-trained on synthetic data and then fine-tuned in two stages: first on errorful corpora, and second on a combination of errorful and error-free parallel corpora. We design custom token-level transformations to map input tokens to target corrections. Our best single-model/ensemble GEC tagger achieves an F{\_}0.5 of 65.3/66.5 on CONLL-2014 (test) and F{\_}0.5 of 72.4/73.6 on BEA-2019 (test). Its inference speed is up to 10 times as fast as a Transformer-based seq2seq GEC system.",
}

Text Simplification by Tagging

@inproceedings{omelianchuk-etal-2021-text,
    title = "{T}ext {S}implification by {T}agging",
    author = "Omelianchuk, Kostiantyn  and
      Raheja, Vipul  and
      Skurzhanskyi, Oleksandr",
    booktitle = "Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications",
    month = apr,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.bea-1.2",
    pages = "11--25",
    abstract = "Edit-based approaches have recently shown promising results on multiple monolingual sequence transduction tasks. In contrast to conventional sequence-to-sequence (Seq2Seq) models, which learn to generate text from scratch as they are trained on parallel corpora, these methods have proven to be much more effective since they are able to learn to make fast and accurate transformations while leveraging powerful pre-trained language models. Inspired by these ideas, we present TST, a simple and efficient Text Simplification system based on sequence Tagging, leveraging pre-trained Transformer-based encoders. Our system makes simplistic data augmentations and tweaks in training and inference on a pre-existing system, which makes it less reliant on large amounts of parallel training data, provides more control over the outputs and enables faster inference speeds. Our best model achieves near state-of-the-art performance on benchmark test datasets for the task. Since it is fully non-autoregressive, it achieves faster inference speeds by over 11 times than the current state-of-the-art text simplification system.",
}