Adding content-focused tests to Travis CI

[jdkato]

Hi,

I'm the author of Vale, a natural language linter that has been used to find a number of minor issues (e.g., #921 and #1418) in your documentation.

I'd like to discuss taking this a bit further by incorporating some content-oriented tests into your CI suite. I know #1418 mentions that you never formally adopted Vale due to false positives, but I still believe that—with the right tuning—automated testing can be a powerful tool.

Overview

I've run an automated analysis on your docs directory that looks for common typos/mistakes. A summary of this analysis is given below (a more detailed report is available in the accompanying Gist):

Mistakes found in 945 Markdown files.

Spelling	Grammar	Style
160	11	91

I think many of these issues (in particular, spelling and style) could be caught prior to doing a manual review of the content.

Implementation

By default, Vale runs a lot of tests that are somewhat context- and opinion-dependent. These are mostly intended to be examples of what it's capable of doing—not rules that I think are generally applicable to writing. And, since Linode seems to have a good editing process already in place, I think we should keep the configuration simple: spelling and capitalization.

Handling non-prose sections

Vale takes a multistage approach to this problem:

1. In Markdown, code blocks (both fenced and indented), code spans, and front matter are ignored by default.
2. You may specify regex-based IgnorePatterns, which represent non-standard sections to be ignored. This was designed with template engines in mind, but it's also a good fit for shortcodes. From what I can tell, {{< file >}}, {{< file-excerpt >}}, {{< output >}}, and {{< highlight ... >}} should be ignored.
3. You may specify a list of HTML tags to ignore. You typically don't need to do this since (1) handles most cases, but it seems like Linode uses <strong> similarly to <code> in many cases.

Spelling

It's surprisingly difficult to check spelling without having to sift through hundreds of false positives, even when using standard spelling dictionaries (such as Hunspell, which Vale supports). The most common solution to this problem is to create a "personal" vocabulary of terms (e.g., the one the Rust team made for their book).

For starters, I've generated a vocabulary for Linode with 1,459 initial terms. After a misspelling is reported, the workflow would be to either add the term to the known list or fix the mistake.

Capitalization

This is pretty straightforward: as you encounter incorrectly capitalized terms, you'd add them to a rule extending the Substitution check. I've started by making a rule for the issues outlined in the linked Gist, but there are likely some that were not reported since I just looked for a few of the most common ones.

Performance

The performance hit on your builds should be negligible. It should take Vale less than 10 seconds to spell check and lint the entirety of your docs directory.

---

If this seems like something you'd be interested in exploring, I'll put together a PR implementing the ideas above.

Thoughts?

[GuessWhoSamFoo]

@jdkato Thanks for the write-up. You definitely captured a lot of areas where testing can improve.

1. Front matter - there definitely are cases where front matter should not be ignored for spelling (e.g Minor typo in the Wercker guide Title #1181), possibly keywords as well. Probably could handle this separately.

2. This may be tricky with handling malformed shortcodes.

3. There really shouldn't be many HTML tags inside markdown files with the exception of gifs/videos from Wistia.

The idea of keeping a dictionary has been floating around for quite some time now. A fork of Pyenchant was floating around but never came to fruition. PRs are definitely welcome - I don't think there is any strong opposition to adding Vale into the CI pipeline.

[GuessWhoSamFoo]

On https://github.com/linode/docs/blame/master/docs/databases/elasticsearch/monitor-nginx-web-server-logs-using-filebeat-elastic-stack-centos-7.md#L166 from the report that unnecessary is spelled incorrectly twice on the same line but only one is reported. Not sure if that's reporting or Vale related.

[jdkato]

Great -- I'll put together a PR.

Regarding front matter and HTML tags, the first thing Vale does is convert the Markdown to HTML. This allows it to both handle many types of markup (e.g., reStructuredText, AsciiDoc, etc.) with the same logic and avoid doing any real parsing itself (which I've found is best left to dedicated libraries). So, <strong> and <code> would be ** and ` respectively.

However, things like links and YAML syntax are difficult to handle well in front matter because most markup -> HTML parsers don't support it (Vale uses Blackfriday for Markdown). A library like python-frontmatter could be used to feed Vale just the text part (but, as you said, this should probably be considered separately).

[GuessWhoSamFoo]

Closed via #1474