Merge pull request #40 from o19s/esci-data

Adding ESCI for UBI

README.md

@@ -2,6 +2,8 @@
 
 This repository contains the search quality evaluation framework as described in the [RFC](https://github.com/opensearch-project/OpenSearch/issues/15354).
 
+Note: Some of the data files in this repository are tracked by `git lfs`.
+
 ## Repository Contents
 
 * `data` - The data directory contains scripts for creating random UBI queries and events for purposes of development and testing.

data/esci/.gitattributes

@@ -0,0 +1 @@
+ubi_queries_events_1000.ndjson.bz2 filter=lfs diff=lfs merge=lfs -text

data/esci/README.md

@@ -0,0 +1,15 @@
+# ESCI Data in UBI Format
+
+This directory contains ESCI data in the UBI format. Created using https://github.com/opensearch-project/user-behavior-insights/tree/main/ubi-data-generator.
+
+https://github.com/amazon-science/esci-data
+
+```
+@article{reddy2022shopping,
+title={Shopping Queries Dataset: A Large-Scale {ESCI} Benchmark for Improving Product Search},
+author={Chandan K. Reddy and Lluís Màrquez and Fran Valero and Nikhil Rao and Hugo Zaragoza and Sambaran Bandyopadhyay and Arnab Biswas and Anlu Xing and Karthik Subbian},
+year={2022},
+eprint={2206.06588},
+archivePrefix={arXiv}
+}
+```

data/esci/index.sh

@@ -0,0 +1,3 @@
+#!/bin/bash -e
+
+curl -X POST "http://localhost:9200/_bulk?pretty" -H "Content-Type: application/x-ndjson" --data-binary @ubi_queries_events_1000.ndjson

opensearch-search-quality-evaluation-plugin/docker-compose.yaml

@@ -9,6 +9,8 @@ services:
       plugins.security.disabled: "true"
       logger.level: info
       OPENSEARCH_INITIAL_ADMIN_PASSWORD: SuperSecretPassword_123
+      http.max_content_length: 500mb
+      OPENSEARCH_JAVA_OPTS: "-Xms8192m -Xmx8192m"
     ulimits:
       memlock:
         soft: -1