Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added control string for sentence breaks #676

Draft
wants to merge 23 commits into
base: master
Choose a base branch
from
+325 −287
Draft

Added control string for sentence breaks #676

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
126c8d1
added vs code files to ignore
Dec 14, 2021
35d5608
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Dec 15, 2021
f5e5f07
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Dec 27, 2021
afb6b8d
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Dec 27, 2021
2f8f2f0
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Jan 3, 2022
39460a4
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Jan 3, 2022
6487405
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Jan 30, 2022
efe157b
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Feb 13, 2022
1aadc76
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Mar 22, 2022
09fd9b7
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Apr 7, 2022
6cd8b39
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Apr 15, 2022
027a08a
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Apr 28, 2022
6debd6f
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu May 16, 2022
76fed8c
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu May 31, 2022
3cfb3d5
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Jun 13, 2022
9c03764
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Jun 14, 2022
6db7073
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Jul 6, 2022
c841522
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Sep 28, 2022
a88e7d2
Merge branch 'master' of https://github.com/clulab/processors
MihaiSurdeanu Nov 17, 2022
def90ec
Added control string for sentence breaks
MihaiSurdeanu Nov 18, 2022
b047378
Make use of control strings optional
kwalcock Nov 18, 2022
cef093a
Test better
kwalcock Nov 18, 2022
3091960
Merge pull request #678 from clulab/kwalcock/sentencebreak
kwalcock Nov 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions main/src/main/java/org/clulab/processors/clu/tokenizer/OpenDomainLexer.g
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ FRACTION: ONE_TO_FOUR_DIGITS ('/' | '\u2044') ONE_TO_FOUR_DIGITS;
// abbreviations are handled in the org.clulab.processors.clulab.tokenizer.Tokenizer class.
WORD: ALPHANUM+ (('.'|'-'|'_'|'\'') ALPHANUM+ )* ;

// special control string that indicates forced sentence break
SENTENCEBREAK: '[SB]' ;

// Twitter user names and hashtags
TWITTER_NAME: '@' (LOWER_CASE_LETTER|UPPER_CASE_LETTER|'_') (LOWER_CASE_LETTER|UPPER_CASE_LETTER|'_'|DIGIT)*;
TWITTER_HASHTAG: '#' LETTER+;
Expand Down
Loading