Package org.languagetool
Class AnalyzedTokenReadings
java.lang.Object
org.languagetool.AnalyzedTokenReadings
- All Implemented Interfaces:
Iterable<AnalyzedToken>
An array of
AnalyzedToken
s used to store multiple POS tags and lemmas
for a given single token.-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate AnalyzedToken[]
private boolean
private String
private boolean
private boolean
private final boolean
private boolean
private boolean
private boolean
private final boolean
private final boolean
private boolean
private static final Pattern
private int
private String
private String
-
Constructor Summary
ConstructorsConstructorDescriptionAnalyzedTokenReadings
(List<AnalyzedToken> tokens, int startPos) AnalyzedTokenReadings
(AnalyzedToken[] tokens, int startPos) AnalyzedTokenReadings
(AnalyzedToken token, int startPos) AnalyzedTokenReadings
(AnalyzedTokenReadings oldAtr, List<AnalyzedToken> newReadings, String ruleApplied) -
Method Summary
Modifier and TypeMethodDescriptionprivate void
addHistoricalAnnotations
(String oldValue, String ruleApplied) void
addReading
(AnalyzedToken token, String ruleApplied) Add a new reading.private boolean
Used to configure the internal variable for lemma equality.boolean
getAnalyzedToken
(int idx) Get a token reading.int
Used to track disambiguator actions.int
Number of readings.int
getToken()
boolean
hasAnyLemma
(String... lemmas) Checks if one of the token's readings has one of the given lemmasboolean
hasAnyPartialPosTag
(String... posTags) Checks if the token has any of the given particular POS tags (only a part of the given POS tag needs to match)int
hashCode()
boolean
Checks if one of the token's readings has a particular lemma.boolean
hasPartialPosTag
(String posTag) Checks if the token has a particular POS tag, where only a part of the given POS tag needs to match.boolean
Checks if the token has a particular POS tag.boolean
hasPosTagAndLemma
(String posTag, String lemma) Checks if the token has a particular POS tag and lemma.boolean
hasPosTagStartingWith
(String posTag) Checks if the token has a POS tag starting with the given string.boolean
Checks if there is at least one POS tagboolean
Used to optimize pattern matching.void
Make the token ignored by all spelling rules.void
immunize()
boolean
boolean
Test if the token can be ignored by spelling rules.boolean
boolean
Returns true if the token equals\n
,\r
,\n\r
, or\r\n
.boolean
boolean
boolean
Test if the token's POStag equals null.boolean
boolean
boolean
isTagged()
boolean
boolean
iterator()
void
leaveReading
(AnalyzedToken token) Removes all readings but the one that matches the token given.boolean
matchesPosTagRegex
(String posTagRegex) Checks if at least one of the readings matches a given POS tag regex.void
removeReading
(AnalyzedToken token, String ruleApplied) Removes a reading from the list of readings.void
setChunkTags
(List<ChunkTag> chunkTags) private void
setHistoricalAnnotations
(String historicalAnnotations) Used to track disambiguator actions.private void
Sets the flag on AnalyzedTokens to make matching onUNKNOWN
POS tag correct in the Element class.void
Add a reading with a paragraph end token unless this is already a paragraph end.void
Add a SENT_END tag.void
setStartPos
(int position) void
setWhitespaceBefore
(String prevToken) toString()
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
Methods inherited from interface java.lang.Iterable
forEach, spliterator
-
Field Details
-
NON_WORD_REGEX
-
isWhitespace
private final boolean isWhitespace -
isLinebreak
private final boolean isLinebreak -
isSentStart
private final boolean isSentStart -
anTokReadings
-
startPos
private int startPos -
token
-
chunkTags
-
isSentEnd
private boolean isSentEnd -
isParaEnd
private boolean isParaEnd -
isWhitespaceBefore
private boolean isWhitespaceBefore -
isPosTagUnknown
private boolean isPosTagUnknown -
whitespaceBeforeChar
-
isImmunized
private boolean isImmunized -
isIgnoredBySpeller
private boolean isIgnoredBySpeller -
historicalAnnotations
-
hasSameLemmas
private boolean hasSameLemmas
-
-
Constructor Details
-
AnalyzedTokenReadings
-
AnalyzedTokenReadings
-
AnalyzedTokenReadings
-
AnalyzedTokenReadings
public AnalyzedTokenReadings(AnalyzedTokenReadings oldAtr, List<AnalyzedToken> newReadings, String ruleApplied) -
AnalyzedTokenReadings
AnalyzedTokenReadings(AnalyzedToken token)
-
-
Method Details
-
getReadings
-
getAnalyzedToken
Get a token reading. -
hasPosTag
Checks if the token has a particular POS tag.- Parameters:
posTag
- POS tag to look for
-
hasPosTagAndLemma
Checks if the token has a particular POS tag and lemma.- Parameters:
posTag
- POS tag and lemma to look for
-
hasReading
public boolean hasReading()Checks if there is at least one POS tag- Since:
- 4.7
-
hasLemma
Checks if one of the token's readings has a particular lemma.- Parameters:
lemma
- lemma POS tag to look for
-
hasAnyLemma
Checks if one of the token's readings has one of the given lemmas- Parameters:
lemmas
- to look for
-
hasPartialPosTag
Checks if the token has a particular POS tag, where only a part of the given POS tag needs to match.- Parameters:
posTag
- POS tag substring to look for- Since:
- 1.8
-
hasAnyPartialPosTag
Checks if the token has any of the given particular POS tags (only a part of the given POS tag needs to match)- Parameters:
posTags
- POS tag substring to look for- Since:
- 4.0
-
hasPosTagStartingWith
Checks if the token has a POS tag starting with the given string.- Parameters:
posTag
- POS tag substring to look for- Since:
- 4.0
-
matchesPosTagRegex
Checks if at least one of the readings matches a given POS tag regex.- Parameters:
posTagRegex
- POS tag regular expression to look for- Since:
- 2.9
-
addReading
Add a new reading.- Parameters:
token
- new reading, given asAnalyzedToken
-
removeReading
Removes a reading from the list of readings. Note: if the token has only one reading, then a new reading with an empty POS tag and an empty lemma is created.- Parameters:
token
- reading to be removed
-
leaveReading
Removes all readings but the one that matches the token given.- Parameters:
token
- Token to be matched- Since:
- 1.5
-
getReadingsLength
public int getReadingsLength()Number of readings. -
isWhitespace
public boolean isWhitespace() -
isLinebreak
public boolean isLinebreak()Returns true if the token equals\n
,\r
,\n\r
, or\r\n
. -
isSentenceStart
public boolean isSentenceStart()- Since:
- 2.3
-
isParagraphEnd
public boolean isParagraphEnd()- Returns:
- true when the token is a last token in a paragraph.
- Since:
- 2.3
-
setParagraphEnd
public void setParagraphEnd()Add a reading with a paragraph end token unless this is already a paragraph end.- Since:
- 2.3
-
isSentenceEnd
public boolean isSentenceEnd()- Returns:
- true when the token is a last token in a sentence.
- Since:
- 2.3
-
isFieldCode
public boolean isFieldCode()- Returns:
- true if the token is LibreOffice/OpenOffice field code.
- Since:
- 0.9.9
-
setSentEnd
public void setSentEnd()Add a SENT_END tag. -
getStartPos
public int getStartPos() -
getEndPos
public int getEndPos()- Since:
- 2.9
-
setStartPos
public void setStartPos(int position) -
getToken
-
setWhitespaceBefore
-
getWhitespaceBefore
-
isWhitespaceBefore
public boolean isWhitespaceBefore() -
immunize
public void immunize() -
isImmunized
public boolean isImmunized() -
ignoreSpelling
public void ignoreSpelling()Make the token ignored by all spelling rules.- Since:
- 2.5
-
isIgnoredBySpeller
public boolean isIgnoredBySpeller()Test if the token can be ignored by spelling rules.- Returns:
- true if the token should be ignored.
- Since:
- 2.5
-
isPosTagUnknown
public boolean isPosTagUnknown()Test if the token's POStag equals null.- Returns:
- true if the token does not have a POStag
- Since:
- 3.9
-
setNoRealPOStag
private void setNoRealPOStag()Sets the flag on AnalyzedTokens to make matching onUNKNOWN
POS tag correct in the Element class. -
getHistoricalAnnotations
Used to track disambiguator actions.- Returns:
- the historicalAnnotations
-
setHistoricalAnnotations
Used to track disambiguator actions.- Parameters:
historicalAnnotations
- the historicalAnnotations to set
-
addHistoricalAnnotations
-
setChunkTags
- Since:
- 2.3
-
getChunkTags
- Since:
- 2.3
-
toString
-
isTagged
public boolean isTagged()- Returns:
- true if AnalyzedTokenReadings has some real POS tag (= not null or a special tag)
- Since:
- 2.3
-
areLemmasSame
private boolean areLemmasSame()Used to configure the internal variable for lemma equality.- Returns:
- true if all
AnalyzedToken
lemmas are the same. - Since:
- 2.5
-
hasSameLemmas
public boolean hasSameLemmas()Used to optimize pattern matching.- Returns:
- true if all
AnalyzedToken
lemmas are the same.
-
isNonWord
public boolean isNonWord()- Returns:
- true if AnalyzedTokenReadings is a punctuation mark, bracket, etc
- Since:
- 4.4
-
hashCode
public int hashCode() -
equals
-
iterator
- Specified by:
iterator
in interfaceIterable<AnalyzedToken>
- Since:
- 2.3
-