migrate gsheet api to public-google-sheets-parser to fix pfasLayer and spreadsheet utility #555

daemon1024 · 2021-08-16T17:46:23Z

migrate gsheet api to v4 public-google-sheets-parser
~~need of API_TOKEN * data reference not based of XML now direct json arrays~~

Fixes #553

gitpod-io · 2021-08-16T17:46:26Z

daemon1024 · 2021-08-16T17:47:08Z

working screenshot

daemon1024 · 2021-08-16T17:51:28Z

Some things I wanted to note.

I am using my personal API_TOKEN for now.
I made a spreadsheet copy from the public csv based on the older sheet, since the sheet itself was not available but the csv was at https://docs.google.com/spreadsheets/u/0/d/1cjQ3H_DX-0dhVL5kMEesFEKaoJKLfC2wWAhokMnJxV4/pub?output=csv

daemon1024 · 2021-08-16T17:53:41Z

Oof, will need to debug the tests too 👀

daemon1024 · 2021-08-17T14:53:31Z

Fixed the spreadsheet utility as well

* migrate gsheet api to v4 * need of API_TOKEN * data reference not based of XML now direct json arrays

for pfas layer

daemon1024 · 2021-08-20T01:40:54Z

No need of API Token anymore :D and it works too.

We still need the sheet to be public though.

daemon1024 · 2021-08-20T01:41:32Z

Will need to debug tests 😓

daemon1024 · 2021-08-20T10:20:14Z

Just wondering if it's just not transpiled the right way! the new npm module I added.

daemon1024 · 2021-08-20T10:58:01Z

Yes, the error is due to public-google-sheets-parser not being transpiled right. I will have to look into how to configure that in the Gruntfile.

After transpile

ReferenceError: Can't find variable: regeneratorRuntime at
dist/LeafletEnvironmentalLayers_babel.js:5432

this is part of public-google-sheets-parser code.

daemon1024 · 2021-08-20T11:56:21Z

@jywarren any hints/suggestions for how to proceed with fixing these tests?

jywarren · 2021-08-31T17:55:56Z

Hmm, looking... could it be related to:

fureweb-com/public-google-sheets-parser#3

jywarren · 2021-08-31T17:57:03Z

src/pfasLayer.js

@@ -1,156 +1,207 @@
+const PublicGoogleSheetsParser = require("public-google-sheets-parser");


Could we require this instead, as it'll have been babelified and transpiled already?

https://github.com/fureweb-com/public-google-sheets-parser/blob/main/dist/index.js

So... maybe:

Suggested change

const PublicGoogleSheetsParser = require("public-google-sheets-parser");

const PublicGoogleSheetsParser = require("node_modules/public-google-sheets-parser/dist/index.js");

It's worth a try!

OK, i tried this but there was an issue which I filed a fix for in the upstream library: fureweb-com/public-google-sheets-parser#11

If they don't merge it, we could change our package.json to point at: https://github.com/jywarren/public-google-sheets-parser/tree/patch-1

jywarren · 2021-08-31T17:58:50Z

The transpile theory makes sense... Jasmine has trouble with some untranspiled code, i think i remember.

Cypress shouldn't, but the errors there are different anyways:

  1) Loads demo page
       has layers menu control:

      AssertionError: Timed out retrying: Too many elements found. Found '22', expected '21'.

jywarren · 2021-08-31T18:55:59Z

Yay, that worked! Pointed at my own patch branch. We can repoint if they merge my PR.

jywarren · 2021-08-31T19:06:01Z

Ugh - looks like I solved one issue but the original regeneratorRuntime is still there:


   - leaflet is created......✓
   - baselayer is added by default......X
     TypeError: undefined is not a function (evaluating 'L.LayerGroup.EnvironmentalLayers({
    })') in file:///home/runner/work/leaflet-environmental-layers/leaflet-environmental-layers/spec/javascripts/basicImplementation.spec.js (line 26) (1)
   - include specific layers......X
     TypeError: undefined is not a function (evaluating 'L.LayerGroup.EnvironmentalLayers({
      include: ['eonetFiresLayer', 'Unearthing'],
      display: ['eonetFiresLayer']
    })') in file:///home/runner/work/leaflet-environmental-layers/leaflet-environmental-layers/spec/javascripts/basicImplementation.spec.js (line 33) (1)
   - exclude specific layers......X
     TypeError: undefined is not a function (evaluating 'L.LayerGroup.EnvironmentalLayers({
      exclude: ['eonetFiresLayer']
    })') in file:///home/runner/work/leaflet-environmental-layers/leaflet-environmental-layers/spec/javascripts/basicImplementation.spec.js (line 43) (1)
   - display shows layer......X
     TypeError: undefined is not a function (evaluating 'L.LayerGroup.EnvironmentalLayers({
      display: ['eonetFiresLayer'],
    })') in file:///home/runner/work/leaflet-environmental-layers/leaflet-environmental-layers/spec/javascripts/basicImplementation.spec.js (line 52) (1)
   - addLayersToMap shows all included layers......X
     TypeError: undefined is not a function (evaluating 'L.LayerGroup.EnvironmentalLayers({
      include: ['eonetFiresLayer', 'Unearthing'],
      addLayersToMap: true,
    })') in file:///home/runner/work/leaflet-environmental-layers/leaflet-environmental-layers/spec/javascripts/basicImplementation.spec.js (line 61) (1)
   - simpleLayerControl shows simple menu......X
     TypeError: undefined is not a function (evaluating 'L.LayerGroup.EnvironmentalLayers({
      simpleLayerControl: true,
    })') in file:///home/runner/work/leaflet-environmental-layers/leaflet-environmental-layers/spec/javascripts/basicImplementation.spec.js (line 72) (1)
   - embed shows embed control......X
     TypeError: undefined is not a function (evaluating 'L.LayerGroup.EnvironmentalLayers({
      embed: true,
    })') in file:///home/runner/work/leaflet-environmental-layers/leaflet-environmental-layers/spec/javascripts/basicImplementation.spec.js (line 80) (1)
 embed control properties
   default properties
     - encountered a declaration exception......X
       TypeError: undefined is not a constructor (evaluating 'new L.control.embed()') in file:///home/runner/work/leaflet-environmental-layers/leaflet-environmental-layers/spec/javascripts/embedControl.spec.js (line 8) (1)
   passed in properties

jywarren · 2021-08-31T19:06:56Z

Does that library by chance use async/await? https://stackoverflow.com/questions/33527653/babel-6-regeneratorruntime-is-not-defined says we need the babel-polyfill library.

daemon1024 · 2021-08-31T19:42:24Z

Oh... Hi 👋🏽 I will go through the updates here.

daemon1024 · 2021-08-31T21:10:05Z

The failures right now are due to using a combination of Babel 6 ( the new deps ) and Babel 7 ( what we have already )

daemon1024 · 2021-08-31T21:46:44Z

The failing cypress tests seems to be due to the new added test spreadsheet layer? Updating the count will do ig 🤔 If that is the case any new test will fail the test ig 👀

jywarren · 2021-09-05T20:08:22Z

Updated the counts to make some progress. There are still errors related to undefineds, though -- is this ES6 incompatibility with Jasmine again?

Then, i see a bunch related to missing images in the pfas layer test. Running it manually what do you think might be happening here?

  5 failing

  1) Pfas layer
       adds markers on click:
     AssertionError: Timed out retrying: Expected to find element: `.leaflet-marker-pane img`, but never found it.
      at Context.eval (http://localhost:8080/__cypress/tests?p=cypress/integration/pfasLayer.spec.js:17:42)

  2) Pfas layer
       has default markers in default mode:
     AssertionError: Timed out retrying: Expected to find element: `.leaflet-marker-pane img`, but never found it.
      at Context.eval (http://localhost:8080/__cypress/tests?p=cypress/integration/pfasLayer.spec.js:24:8)

  3) Pfas layer
       has circle markers in minimal mode:
     AssertionError: Timed out retrying: Expected to find element: `.leaflet-overlay-pane svg g`, but never found it.
      at Context.eval (http://localhost:8080/__cypress/tests?p=cypress/integration/pfasLayer.spec.js:33:10)

  4) Pfas layer
       shows popup:
     AssertionError: Timed out retrying: Expected to find element: `.leaflet-overlay-pane svg g`, but never found it.
      at Context.eval (http://localhost:8080/__cypress/tests?p=cypress/integration/pfasLayer.spec.js:41:8)

  5) Pfas layer
       removes markers from the map and the layer name from the hash when clicked again:
     AssertionError: Timed out retrying: Expected to find element: `.leaflet-overlay-pane svg g`, but never found it.
      at Context.eval (http://localhost:8080/__cypress/tests?p=cypress/integration/pfasLayer.spec.js:53:8)

jywarren · 2022-06-07T06:33:48Z

Retriggering tests so we can see the latest logs

daemon1024 requested a review from jywarren August 16, 2021 17:51

daemon1024 changed the title ~~fix pfasLayer~~ migrate gsheet api to v4 to fix pfasLayer and spreadsheet utility Aug 17, 2021

daemon1024 mentioned this pull request Aug 18, 2021

[Planning] Geographic Features Refinement publiclab/plots2#9698

Open

daemon1024 added 3 commits August 20, 2021 06:45

fix pfasLayer

359dcc3

* migrate gsheet api to v4 * need of API_TOKEN * data reference not based of XML now direct json arrays

Fix spreadsheet utility

7ed1706

use gsheet-parser from npm instead of gsheet api

a91125c

for pfas layer

daemon1024 force-pushed the migrate-gsheet-api branch from a198ea7 to a91125c Compare August 20, 2021 01:20

daemon1024 changed the title ~~migrate gsheet api to v4 to fix pfasLayer and spreadsheet utility~~ migrate gsheet api to public-google-sheets-parser to fix pfasLayer and spreadsheet utility Aug 20, 2021

use new parsing in spreadsheet utility

0448637

jywarren reviewed Aug 31, 2021

View reviewed changes

jywarren added 2 commits August 31, 2021 18:50

fixed with relative link to dist version of lib

94eba21

fixed spreadsheet dep

80df53b

jywarren added 2 commits August 31, 2021 19:10

babel-polyfill

98364eb

extra babel-polyfill deps

669b03c

update deprecated babel dep

dde98ee

jywarren added 3 commits September 5, 2021 16:02

update layer groups count

66c22a0

updated counts for new layer

180b750

update counts for new layer, to fix test

9ccfbe0

jywarren mentioned this pull request Sep 5, 2021

Refactor Cypress tests to not depend on static layer/group counts #580

Open

daemon1024 added 4 commits February 22, 2022 15:09

use upstream sheets api package since patch was merged

6f95f85

update node version in github actions

da86902

update node version in cypress github actions

d4c99a6

fix pfasLayer

57145bf

jywarren mentioned this pull request Jun 7, 2022

Tracking general cypress test failures #668

Open

jywarren closed this Jun 7, 2022

jywarren reopened this Jun 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migrate gsheet api to public-google-sheets-parser to fix pfasLayer and spreadsheet utility #555

migrate gsheet api to public-google-sheets-parser to fix pfasLayer and spreadsheet utility #555

daemon1024 commented Aug 16, 2021 •

edited

Loading

gitpod-io bot commented Aug 16, 2021

daemon1024 commented Aug 16, 2021

daemon1024 commented Aug 16, 2021

daemon1024 commented Aug 16, 2021

daemon1024 commented Aug 17, 2021

daemon1024 commented Aug 20, 2021 •

edited

Loading

daemon1024 commented Aug 20, 2021 •

edited

Loading

daemon1024 commented Aug 20, 2021

daemon1024 commented Aug 20, 2021 •

edited

Loading

daemon1024 commented Aug 20, 2021

jywarren commented Aug 31, 2021

jywarren Aug 31, 2021

jywarren Aug 31, 2021

jywarren Aug 31, 2021

jywarren commented Aug 31, 2021

jywarren commented Aug 31, 2021

jywarren commented Aug 31, 2021

jywarren commented Aug 31, 2021

daemon1024 commented Aug 31, 2021

daemon1024 commented Aug 31, 2021

daemon1024 commented Aug 31, 2021 •

edited

Loading

jywarren commented Sep 5, 2021

jywarren commented Jun 7, 2022

		@@ -1,156 +1,207 @@
		const PublicGoogleSheetsParser = require("public-google-sheets-parser");

migrate gsheet api to public-google-sheets-parser to fix pfasLayer and spreadsheet utility #555

Are you sure you want to change the base?

migrate gsheet api to public-google-sheets-parser to fix pfasLayer and spreadsheet utility #555

Conversation

daemon1024 commented Aug 16, 2021 • edited Loading

gitpod-io bot commented Aug 16, 2021

daemon1024 commented Aug 16, 2021

daemon1024 commented Aug 16, 2021

daemon1024 commented Aug 16, 2021

daemon1024 commented Aug 17, 2021

daemon1024 commented Aug 20, 2021 • edited Loading

daemon1024 commented Aug 20, 2021 • edited Loading

daemon1024 commented Aug 20, 2021

daemon1024 commented Aug 20, 2021 • edited Loading

daemon1024 commented Aug 20, 2021

jywarren commented Aug 31, 2021

jywarren Aug 31, 2021

Choose a reason for hiding this comment

jywarren Aug 31, 2021

Choose a reason for hiding this comment

jywarren Aug 31, 2021

Choose a reason for hiding this comment

jywarren commented Aug 31, 2021

jywarren commented Aug 31, 2021

jywarren commented Aug 31, 2021

jywarren commented Aug 31, 2021

daemon1024 commented Aug 31, 2021

daemon1024 commented Aug 31, 2021

daemon1024 commented Aug 31, 2021 • edited Loading

jywarren commented Sep 5, 2021

jywarren commented Jun 7, 2022

daemon1024 commented Aug 16, 2021 •

edited

Loading

daemon1024 commented Aug 20, 2021 •

edited

Loading

daemon1024 commented Aug 20, 2021 •

edited

Loading

daemon1024 commented Aug 20, 2021 •

edited

Loading

daemon1024 commented Aug 31, 2021 •

edited

Loading