Maximum bufferSize

[MahdiSaberi]

I encountered an issue while processing a CSV file of approximately 1 GB with 10 million records using the following approach:

CsvReader<CsvRecord> reader = CsvReader
                                .builder()
                                .fieldSeparator(datasourceFile.getDelimiter())
                                .build(new CsvRecordHandler(customModifier()), new InputStreamReader(inputStream, StandardCharsets.UTF_8));

private FieldModifier customModifier() {
        return new FieldModifier() {
            @Override
            public String modify(long startingLineNumber, int fieldIdx, boolean quoted, String field) {
                field = field.replaceAll("[\\n\\r\\t]", " ");
                if (field.length() > 256) field = field.substring(0, 256);
                return field;
            }
        };
    }

However, I encountered the following exception:

de.siegmar.fastcsv.reader.CsvParseException: The maximum buffer size of 16777216 is insufficient to read the data of a single field. This issue typically arises when a quotation begins but does not conclude within the confines of this buffer's maximum limit.

Unfortunately, I couldn't find a method to increase the maximum buffer size to handle this case. Could you please guide me on how to resolve this issue?