Applies a workaround for parser bug dealing with inline comments.

[dyoo]

This is some continuation of work for #63 for implementing special behavior for HTML comments. This PR splits off inline comments from content that immediately follows them, as a workaround for pulldown-cmark/pulldown-cmark#712.

[mgeisler]

Hi @dyoo, thanks for the PR! We could patch this on our side, though I wonder if you've looked into fixing it upstream? I don't really know the internals of pulldown_cmark very well, but it would be super if you could spend an afternoon looking into it and perhaps improve this for everybody 🙂

I think the corner cases affected by this problem (in the skipping code) might be small enough that we don't have to work around them. I just tested with variations of the unit test:

- foo
<!-- mdbook-xgettext: skip -->
- bar
- baz

works as expected and skips only - bar. I would be okay to say that

- foo
- <!-- mdbook-xgettext: skip --> bar
- baz

just isn't supported right now. Especially because the consequences are small and easy to see: translators will see when they suddenly lose access to the rest of the list because both bar and baz are no longer extracted.

[dyoo]

I did a little more analysis of the situation, and what is happening with the parser is following the CommonMark spec, so there is technically not a bug on their end.

The problematic situation is that, by starting with an HTML comment, we're creating an HTML block. According to the specification, that block lasts till the end of the line. The content of the HTML block is passed, unescaped, into the rendered content.

So the usage of block-level HTML comments is a little imperfect, and we have to be careful to be aware of this end-of-line behavior. That does mean that if we're going to use <!-- i18n ... -> as our escape hatch, we do need to say something in usage notes to the effect that it must be the only thing on a line: otherwise, unintuitive things can happen.

I think we can close this bug. I'll extract out some of the code refactorings I did to make the code a little nicer to read and do that as a separate PR.

[mgeisler]

So the usage of block-level HTML comments is a little imperfect, and we have to be careful to be aware of this end-of-line behavior. That does mean that if we're going to use <!-- i18n ... -> as our escape hatch, we do need to say something in usage notes to the effect that it must be the only thing on a line: otherwise, unintuitive things can happen.

Thanks for investigating this!

As you say, we should explain this eventually if/when people run into problems by trying to skip things in complex situations.