[SPARK-16045][ML][Doc] Spark 2.0 ML.feature: doc update for stopwords and binarizer #13375
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #59557 has finished for PR 13375 at commit
|
docs/ml-features.md
Outdated
| `StopWordsRemover` instance. A boolean parameter `caseSensitive` indicates | ||
| if the matches should be case sensitive (false by default). | ||
| the `stopWords` parameter. Default stop words for some languages are provided | ||
| ("danish", "dutch", "english", "finnish", "french", "german", "hungarian", "italian", |
Member
There was a problem hiding this comment.
Languages should be like Danish, Dutch, ...
Contributor
Author
There was a problem hiding this comment.
I'm afraid it will confuse users as currently it does not support StopWordsRemover.loadDefaultStopWords("English") (with Capital E). Maybe we should use language.toLower in loadDefaultStopWords.
Member
There was a problem hiding this comment.
OK, in that case just clarify that these strings are arguments to the method.
Contributor
Author
|
Thanks for the review @srowen |
|
Test build #59592 has finished for PR 13375 at commit
|
Member
|
I think that's fine. |
Member
|
@hhbyyh Could you please create a new JIRA for this PR to make it clear in JIRA what it is addressing? Please link it to SPARK-15100. Thanks! |
hhbyyh
changed the title
Contributor
Author
|
created sub-task: https://issues.apache.org/jira/browse/SPARK-16045 and updated the title. |
Contributor
|
LGTM. Merged into master and branch-2.0. Thanks! |
asfgit
pushed a commit
that referenced
this pull request
Jun 21, 2016
… and binarizer ## What changes were proposed in this pull request? jira: https://issues.apache.org/jira/browse/SPARK-16045 2.0 Audit: Update document for StopWordsRemover and Binarizer. ## How was this patch tested? manual review for doc Author: Yuhao Yang <[email protected]> Author: Yuhao Yang <[email protected]> Closes #13375 from hhbyyh/stopdoc. (cherry picked from commit a58f402) Signed-off-by: Xiangrui Meng <[email protected]>
dongjoon-hyun
pushed a commit
that referenced
this pull request
Jun 28, 2023
### What changes were proposed in this pull request? This pr aims to upgrade netty from 4.1.92 to 4.1.93. ### Why are the changes needed? 1.v4.1.92 VS v4.1.93 netty/netty@netty-4.1.92.Final...netty-4.1.93.Final 2.The new version brings some bug fix, eg: - Reset byte buffer in loop for AbstractDiskHttpData.setContent ([#13320](netty/netty#13320)) - OpenSSL MAX_CERTIFICATE_LIST_BYTES option supported ([#13365](netty/netty#13365)) - Adapt to DirectByteBuffer constructor in Java 21 ([#13366](netty/netty#13366)) - HTTP/2 encoder: allow HEADER_TABLE_SIZE greater than Integer.MAX_VALUE ([#13368](netty/netty#13368)) - Upgrade to latest netty-tcnative to fix memory leak ([#13375](netty/netty#13375)) - H2/H2C server stream channels deactivated while write still in progress ([#13388](netty/netty#13388)) - Channel#bytesBefore(un)writable off by 1 ([#13389](netty/netty#13389)) - HTTP/2 should forward shutdown user events to active streams ([#13394](netty/netty#13394)) - Respect the number of bytes read per datagram when using recvmmsg ([#13399](netty/netty#13399)) 3.The release notes as follows: - https://netty.io/news/2023/05/25/4-1-93-Final.html 4.Why not upgrade to `4-1-94-Final` version? Because the return value of the 'threadCache()' (from `PoolThreadCache` to `PoolArenasCache`) method of the netty Inner class used in the 'arrow memory netty' version '12.0.1' has changed and belongs to break change, let's wait for the upgrade of the 'arrow memory netty' before upgrading to the '4-1-94-Final' version. The reference is as follows: https://github.com/apache/arrow/blob/6af660f48472b8b45a5e01b7136b9b040b185eb1/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java#L164 https://github.com/netty/netty/blob/da1a448d5bc4f36cc1744db93fcaf64e198db2bd/buffer/src/main/java/io/netty/buffer/PooledByteBufAllocator.java#L732-L736 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GA. Closes #41681 from panbingkun/upgrade_netty. Authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
jira: https://issues.apache.org/jira/browse/SPARK-16045
2.0 Audit: Update document for StopWordsRemover and Binarizer.
How was this patch tested?
manual review for doc