Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove spacy dependancy in pyproject.toml #920

Merged
merged 3 commits into from
Dec 13, 2024
Merged

Remove spacy dependancy in pyproject.toml #920

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fbcd665
remove spacy dependancy in pyproject.toml
madanucd Nov 26, 2024
7e5119c
Update poetry.lock for fbcd6650
Nov 27, 2024
5193daf
linear error handling for codecov coverage
madanucd Dec 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 1 addition & 18 deletions backend/poetry.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 0 additions & 3 deletions backend/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,6 @@ typer = "^0.12.0"
beautifulsoup4 = "^4.12.3"
spacy = "^3.7.6"

[tool.poetry.dependencies.en_core_sci_sm]
url = "https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_sm-0.5.4.tar.gz"

[tool.poetry.group.dev.dependencies]
pytest = "^8.2.0"
mkdocs = ">=1.6.0"
Expand Down
30 changes: 29 additions & 1 deletion backend/src/monarch_py/implementations/spacy/spacy_implementation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
from monarch_py.interfaces.text_annotation_interface import TextAnnotatorInterface
from monarch_py.datamodels.model import TextAnnotationResult, SearchResult

import pystow
import tarfile

@dataclass
class SpacyImplementation(TextAnnotatorInterface):
Expand All @@ -16,10 +18,36 @@ class SpacyImplementation(TextAnnotatorInterface):
grounding_implementation = None

def init_spacy(self, grounding_implementation: GroundingInterface):
self.nlp = spacy.load("en_core_sci_sm")
# Define the URL for the Spacy model
model_url = "https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.4/en_core_sci_sm-0.5.4.tar.gz"

# Use pystow.ensure to download and cache the model archive
model_archive = pystow.ensure("spacy", "models", url=model_url)
# Define the expected unpacked directory
model_dir = model_archive.parent / "en_core_sci"

# Unpack the model if it's not already unpacked
if not model_dir.exists():
print("Unpacking Spacy model...")
with tarfile.open(model_archive, "r:gz") as tar:
tar.extractall(path=model_dir.parent)

model_subdir = next((d for d in model_archive.parent.iterdir() if d.is_dir() and d.name.startswith("en_core_sci")), None)

if model_subdir:
inner_model_dir = next((d for d in model_subdir.iterdir() if d.is_dir() and d.name.startswith("en_core_sci") and "egg-info" not in d.name), None)
if inner_model_dir:
# Load the model
self.nlp = spacy.load(str(str(model_archive.parent / model_subdir.name / inner_model_dir.name / model_subdir.name)))

# Assign the grounding implementation
self.grounding_implementation = grounding_implementation

# Test the model with a sample sentence
self.nlp("Nystagmus, strabismus, fundus, ocular albinism, lewis.")



def get_annotated_entities(self, text) -> List[TextAnnotationResult]:
"""Annotate text using SPACY"""
results: List[TextAnnotationResult] = []
Expand Down
Loading