Handle tokens in .glyphs features

[rsheeter]

Continuing from #91, implement some or all of https://github.com/googlefonts/glyphsLib/blob/24b4d340e4c82948ba121dcfe563c1450a8e69c9/Lib/glyphsLib/builder/tokens.py#L12 and fail on whatever parts are not implemented. Used by https://github.com/googlefonts/glyphsLib/blob/24b4d340e4c82948ba121dcfe563c1450a8e69c9/Lib/glyphsLib/builder/features.py#L85. Consider looking at our actual sources to determine what is needed.

[rsheeter]

Files that might contain tokens from sources obtained using https://github.com/rsheeter/google_fonts_sources:

$ find sources -name '*.glyphs' -exec grep -l "[$]" {} \;
sources/ofl/leaguespartan/sources/LeagueSpartan.glyphs
sources/ofl/zenloop/sources/ZenLoop.glyphs
sources/ofl/notosansgurmukhi/sources/NotoSansGurmukhi.glyphs
sources/ofl/changa/sources/Changa.glyphs
sources/ofl/atkinsonhyperlegible/sources/AtkinsonHyperlegible-Italic.glyphs
sources/ofl/notonastaliqurdu/sources/NotoNastaliqUrdu.glyphs
sources/ofl/loversquarrel/sources/LoversQuarrel.glyphs
sources/ofl/notoseriftamil/sources/NotoSansTamil.glyphs
sources/ofl/redrose/glyphs/RedRose.glyphs
sources/ofl/karantina/Sources/karantina.glyphs
sources/ofl/ole/sources/Ole.glyphs
sources/ofl/rugeboogie/sources/RugeBoogiePro.glyphs
sources/ofl/anton/sources/Anton.glyphs
sources/ofl/ephesis/sources/Ephesis.glyphs
sources/ofl/baskervville/documents/workshop/drawing/Glyphs-171020/Baskerwille-171020 AF.glyphs
sources/ofl/festive/sources/Festive-Pro.glyphs
sources/ofl/signika/sources/Signika-Italic.glyphs
sources/ofl/signikanegative/sources/Signika-Italic.glyphs
sources/ofl/genos/sources/Genos-Italic.glyphs
sources/ofl/genos/sources/Genos.glyphs
sources/ofl/oi/sources/Oi.glyphs
sources/ofl/oi/sources/Old Glyphs file/Oi-2.glyphs
sources/ofl/oi/sources/Old Glyphs file/Oi-1.glyphs
sources/ofl/qwigley/sources/Qwigley.glyphs
sources/ofl/jetbrainsmono/sources/JetBrainsMono.glyphs
sources/ofl/jetbrainsmono/sources/JetBrainsMono-Italic.glyphs
sources/ofl/notosanstamil/sources/NotoSansTamil.glyphs
sources/ofl/yaldevi/documentation/old-sources/glyphs/10-09-2015/AyannaNarrow-04.glyphs
sources/ofl/karla/sources/Karla-Roman.glyphs
sources/ofl/notoserifgurmukhi/sources/NotoSansGurmukhi.glyphs
sources/ofl/brygada1918/sources/Temp Archive/Brygada_ITALIC_MASTER.glyphs

Eliminating a few (sources/ofl/oi/sources/Old Glyphs file/Oi-2.glyphs, sources/ofl/oi/sources/Old Glyphs file/Oi-1.glyphs, sources/ofl/yaldevi/documentation/old-sources/glyphs/10-09-2015/AyannaNarrow-04.glyphs, sources/ofl/baskervville/documents/workshop/drawing/Glyphs-171020/Baskerwille-171020 AF.glyphs, sources/ofl/brygada1918/sources/Temp Archive/Brygada_ITALIC_MASTER.glyphs) for looking suspicious we can grab things that look a bit like symbols:

NotoSansGurmukhi.glyphs feature bare_num $aumatraaddak_tta_fix
NotoSansGurmukhi.glyphs feature bare_num $ba_aumatraaddak_fix
NotoSansGurmukhi.glyphs prefix bare_num $addak_rightfix
NotoSansGurmukhi.glyphs prefix bare_num $aumatra_rightfix
NotoSansGurmukhi.glyphs prefix bare_num $aumatrabindi_rightfix
NotoNastaliqUrdu.glyphs prefix bare_num $spacer
NotoNastaliqUrdu.glyphs prefix number ${spacer*10}
NotoNastaliqUrdu.glyphs prefix number ${spacer*11}
NotoNastaliqUrdu.glyphs prefix number ${spacer*12}
NotoNastaliqUrdu.glyphs prefix number ${spacer*13}
NotoNastaliqUrdu.glyphs prefix number ${spacer*14}
NotoNastaliqUrdu.glyphs prefix number ${spacer*15}
NotoNastaliqUrdu.glyphs prefix number ${spacer*1}
NotoNastaliqUrdu.glyphs prefix number ${spacer*2}
NotoNastaliqUrdu.glyphs prefix number ${spacer*3}
NotoNastaliqUrdu.glyphs prefix number ${spacer*4}
NotoNastaliqUrdu.glyphs prefix number ${spacer*5}
NotoNastaliqUrdu.glyphs prefix number ${spacer*6}
NotoNastaliqUrdu.glyphs prefix number ${spacer*7}
NotoNastaliqUrdu.glyphs prefix number ${spacer*8}
NotoNastaliqUrdu.glyphs prefix number ${spacer*9}
NotoSansTamil.glyphs feature bare_num $anusvara_shift
NotoSansTamil.glyphs feature bare_num $anusvara_shift
NotoSansGurmukhi.glyphs feature bare_num $aumatraaddak_tta_fix
NotoSansGurmukhi.glyphs feature bare_num $ba_aumatraaddak_fix
NotoSansGurmukhi.glyphs prefix bare_num $addak_rightfix
NotoSansGurmukhi.glyphs prefix bare_num $aumatra_rightfix
NotoSansGurmukhi.glyphs prefix bare_num $aumatrabindi_rightfix

[rsheeter]

In https://raw.githubusercontent.com/notofonts/gurmukhi/main/sources/NotoSansGurmukhi.glyphs search for aumatra_rightfix. It appears to be a number and IIUC resolves to a fontMaster specific value in
https://github.com/googlefonts/glyphsLib/blob/24b4d340e4c82948ba121dcfe563c1450a8e69c9/Lib/glyphsLib/builder/tokens.py#L56. That would appear to leave us with variable features so ... how's that work?

[rsheeter]

@anthrotype pointed me at https://glyphsapp.com/learn/tokens

[rsheeter]

If we had adobe-type-tools/afdko#1350 perhaps we could convert simple cases to fea cleanly.

[simoncozens]

The cases above are not "tokens" but "number values" which are defined on each master. Tokens are a bit different - they are predicates like so:

@TAHim = [$[name contains "tah-ar.init"] $[name contains "tah-ar.medi"]];

(This expands to a class of all glyph names containing tah-ar.init or tah-ar.medi.) See https://github.com/googlefonts/glyphsLib/blob/main/Lib/glyphsLib/builder/tokens.py for a Python feature code preprocessor which expands tokens.

[cmyr]

okay, getting to this now.

naively, my plan is:

• adding parsing support for the glyphs extension syntax
• add a step (after parsing but before validation? maybe?) where we rewrite the AST, replacing this syntax with standard FEA
• and then we're compiling a bunch more fonts.