Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

train.json cant't find #38

Open
Lirsakura opened this issue Dec 20, 2020 · 1 comment
Open

train.json cant't find #38

Lirsakura opened this issue Dec 20, 2020 · 1 comment

Comments

@Lirsakura
Copy link

can you tell me Where is train.json

FileNotFoundError: [Errno 2] No such file or directory: '/data/bq/train.json'

@okcd00
Copy link

okcd00 commented Apr 22, 2021

Data files for bq all have the postfix .tsv (not .json)
I modify the function in class BQProcessor as below, and it successfully loads bq corpus,
you can view the changes in my fork:

# Updated
class BQProcessor(DataProcessor):
    """Processor for the dbqa data set """

    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(
            self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(
            self._read_tsv(os.path.join(data_dir, "valid.tsv")),
            "dev_matched")

    def get_test_examples(self, data_dir):
        """See base class."""
        return self._create_examples(
            self._read_tsv(os.path.join(data_dir, "test.tsv")),
            "dev_matched")

    def get_labels(self):
        """See base class."""
        return ["0", "1"]

    # Original
    def _create_examples_old(self, lines, set_type):
        """Creates examples for the training and dev sets."""
        examples = []
        for (i, line) in enumerate(lines):
            line = json.loads(line[0])
            if i < 2:
               print("-"*10) 
               print("check  loading example") 
               print(line)
               print(type(line)) 
            guid = "%s-%s" % (set_type, i)
            text_a = line[1] # .replace(" ", "")
            text_b = line[2] # .replace(" ", "")
            label = line
            examples.append(
                InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
        return examples

    def _create_examples(self, lines, set_type):
        """Creates examples for the training and dev sets."""
        examples = []
        for (i, line) in enumerate(lines):
            if i < 2:
               print("-"*10) 
               print("check  loading example") 
               print(line)
               print(type(line)) 
            guid = "%s-%s" % (set_type, i)
            text_a = line[1] # .replace(" ", "")
            text_b = line[2] # .replace(" ", "")
            label = line[0]
            examples.append(
                InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
        return examples

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants