Preprocessing

[sangHa0411]

전처리 Pull Request 합니다.

1. 적용 과정
   1. train.py

    dataset_name = "metamong1/summarization"
    datasets = load_dataset(dataset_name + "_part" if data_args.is_part else dataset_name,    use_auth_token=USE_AUTH_TOKEN)
    data_preprocessor = Preprocessor()
    data_filter = Filter(min_size=5, max_size=100)

    ## data preprocessing
    datasets = datasets.map(data_preprocessor.for_train)
    datasets = datasets.filter(data_filter)

    train_dataset = datasets['train']
    valid_dataset = datasets['validation']

2. predict.py

    dataset_name = "metamong1/summarization"
    datasets = load_dataset(dataset_name + "_part" if data_args.is_part else dataset_name, use_auth_token=USE_AUTH_TOKEN)
    data_preprocessor = Preprocessor()
    datasets = datasets.map(data_preprocessor.for_test)
    valid_dataset = datasets['validation']