the msra2src.py may have some problems

[cooper12121]

i think your py file have problem to process the raw dataset,such as :
当 O
希 O
望 O
工 O
程 O
救 O
助 O
的 O
百 O
万 O
儿 O
童 O
成 O
长 O
起 O
来 O
， O
科 O
教 O
兴 O
国 O
蔚 O
然 O
成 O
风 O
时 O
， O
今 O
天 O
有 O
收 O
藏 O
价 O
值 O
的 O
书 O
你 O
没 O
买 O
， O
明 O
日 O
就 O
叫 O
你 O
悔 O
不 O
当 O
初 O
！ O

藏 O
书 O
本 O
来 O
就 O
是 O
所 O
有 O
传 O
统 O
收 O
藏 O
门 O
类 O
中 O
的 O
第 O
一 O
大 O
户 O
， O
只 O
是 O
我 O
们 O
结 O
束 O
温 O
饱 O
的 O
时 O
间 O
太 O
短 O
而 O
已 O
。 O
so it doesn't work for raw dataset you gived ,and it also doesn't equal to the mrc fomat dataset you gived

[dsuanya]

hello！ I also met some problems when running the msra2mrc.py?do you know how to deal with the problem of invilid inputs?thank you so much if you could give me a reply!!!

[cooper12121]

hello！ I also met some problems when running the msra2mrc.py?do you know how to deal with the problem of invilid inputs?thank you so much if you could give me a reply!!!

there are some different formats of this input content,but they all didn't match the paper's data process program.After some trying I gave up,so sorry can't help you

[dsuanya]

hello！ I also met some problems when running the msra2mrc.py?do you know how to deal with the problem of invilid inputs?thank you so much if you could give me a reply!!!

there are some different formats of this input content,but they all didn't match the paper's data process program.After some trying I gave up,so sorry can't help you

Anyway,Thank you for your reply!I

[algorithm007]

def convert_file(input_file, output_file, tag2query_file):
    """
    Convert MSRA raw data to MRC format
    """
    origin_count = 0
    new_count = 1
    tag2query = json.load(open(tag2query_file))
    mrc_samples = []
    contexts, labels = [], []
    with open(input_file) as fin:
        for line in fin:
            line = line.strip()
            if line:
                context, label = line.split(" ")
                contexts.append(context)
                labels.append(label)
            else:

                tags = bmes_decode(char_label_list=[(char, label) for char, label in zip(contexts, labels)])

    #         tags = bmes_decode(char_label_list=[(char, label) for char, label in zip(src.split(), labels.split())])
                for label, query in tag2query.items():
                    start_position = [tag.begin for tag in tags if tag.tag == label]
                    end_position = [tag.end-1 for tag in tags if tag.tag == label]
                    mrc_samples.append(
                        {
                            "qas_id": "{}.{}".format(origin_count, new_count),
                            "context": " ".join(contexts),
                            "start_position": start_position,
                            "end_position": end_position,
                            "query": query,
                            "impossible": False if start_position and end_position else True,
                            "entity_label": label,
                            "span_position": [f"{start};{end}" for start, end in zip(start_position, end_position)]
                        }
                    )
                    new_count += 1

                contexts, labels = [], []
                origin_count += 1
                new_count = 1

    json.dump(mrc_samples, open(output_file, "w"), ensure_ascii=False, sort_keys=True, indent=2)