R1:7ea6de6e9c9d
R1:7ea6de6e9c9d
Split Ferret engine strings for tokenization on any sequence of whitespace
Summary:
Ref T12819. Currently, strings are split only on spaces, but newlines (and, if they exist, tabs) should also split strings.
Without this, we can fail to get the proper term boundary tokens for words which begin at the start of a line or end at the end of a line.
Test Plan: Reindexed a document with "xyz\nabc", saw `"yz "` and `" ab"` term boundary tokens generate properly.
Reviewers: chad
Reviewed By: chad
Maniphest…
Summary:
Ref T12819. Currently, strings are split only on spaces, but newlines (and, if they exist, tabs) should also split strings.
Without this, we can fail to get the proper term boundary tokens for words which begin at the start of a line or end at the end of a line.
Test Plan: Reindexed a document with "xyz\nabc", saw `"yz "` and `" ab"` term boundary tokens generate properly.
Reviewers: chad
Reviewed By: chad
Maniphest…
Repository: R1 hydra
Commit Date: Sep 8 2017