First variable of dataset not defined in form: SurveyCTO repeat group .csv files that start with *_count field

[matthew-white]

The odkmeta do-file assumes that the first field of all repeat group .csv files is described in the XLSForm and is given char metadata by the do-file. However, SurveyCTO repeat group .csv files may start with a *_count field, to which the do-file does not attach metadata. This may result in the following uninformative error message:

. rename PARENT_KEY KEY

. 
. local pos : list posof "some_repeat_group_name" in repeats
. local child : word `pos' of `childfiles'
invalid syntax
r(198);

To remedy this, replace the following lines:

foreach dta of local dtas {
    use "`dta'", clear

    unab all : _all
    gettoken first : all
    local repeat : char `first'[Odk_repeat]

    ...

with:

foreach dta of local dtas {
    use "`dta'", clear

    ds, has(char Odk_type)
    local first : word 1 of `r(varlist)'
    local repeat : char `first'[Odk_repeat]

    ...

[hofmanpaul]

Hi Matthew, I'm getting this same error when using formhub data. However, if I use the solution you provide, the following bit of code gives me an error:

    * Attach field labels as variable labels and notes.
    ds, has(char Odk_long_name)
    foreach var in `r(varlist)' {
        * Variable label
        local label : char `var'[Odk_label]
        mata: st_varlabel("`var'", st_local("label"))

        * Notes
        if `:length local label' {
            char `var'[note0] 1
            mata: st_global("`var'[note1]", "Question text: " + ///
                st_global("`var'[Odk_label]"))
            mata: st_local("temp", ///
                " " * (strlen(st_global("`var'[note1]")) + 1))
            #delimit ;
            local fromto
                {           "`temp'"
                }           "{c )-}"
                "`temp'"    "{c -(}"
                '           "{c 39}"
                "`"         "{c 'g}"
                "$"         "{c S|}"
            ;
            #delimit cr
            while `:list sizeof fromto' {
                gettoken from fromto : fromto
                gettoken to   fromto : fromto
                mata: st_global("`var'[note1]", ///
                    subinstr(st_global("`var'[note1]"), "`from'", "`to'", .))
            }
        }
    }

    compress

    local repeats `"`repeats' "`repeat'""'
    tempfile child
    local childfiles : list childfiles | child

    local badnames
    ds, has(char Odk_bad_name)
    foreach var in `r(varlist)' {
        if `:char `var'[Odk_bad_name]' & ///
            ("`:char `var'[Odk_type]'" != "begin repeat" | ///
            ("`repeat'" != "" & ///
            "`:char `var'[Odk_name]'" == "SET-OF-`repeat'")) {
            local badnames : list badnames | var
        }
    }
    local allbadnames `"`allbadnames' "`badnames'""'

    ds, not(char Odk_name)
    local datanotform `r(varlist)'
    local exclude SubmissionDate _index _parent_index metainstanceID
    local datanotform : list datanotform - exclude
    local alldatanotform `"`alldatanotform' "`datanotform'""'

    save, replace
}

The error is:

[ invalid name
r(198);

after it has successfully saved the file. Any idea on how I could overcome this?

[matthew-white]

Curious. I don't see any use of `first' in your code, so it could be a different issue. I don't see right away what could be causing it: it looks like a characteristic is being referenced without a variable name, but unless `var' isn't expanding, I don't see how that could be. Is there any chance you could e-mail me the do-file along with example .csv files?

[matthew-white]

E-mailed @hofmanpaul:

[One of your .csv files] contains just a single select_multiple field, which odkmeta isn't able to attach metadata to for the same reason as above: formhub splits the field in the .csv, so odkmeta can't find it.

My understanding is that you can have formhub not split these fields, which would help. It's something like: click "advanced export" and select "DON'T split select multiple choice answers into separate columns". Another solution is to define `repeat' based on `dta' as follows:

local repeat : subinstr local dta "Data/1_raw/odkmeta/Data/hh_survey" "", count(local count)
assert `count' == 1
local repeat : subinstr local repeat "-" "", count(local count)
if "`repeat'" != "" ///
    assert `count' == 1

It's nice that formhub offers its own data processing, for instance, splitting select_multiple fields, but we'd see fewer issues with importing formhub data if there were an option to output the raw ODK data.

[amandarudin]

Hi Matthew,

I'm getting a similar error when running odkmeta, but I can't find the text you said should be replaced in the odkmeta dofile. Is there another way to correct this problem?

See error message below. Thank you!
Amanda

* Duplicate variable name with s01.a.-s01.b.-s01.b.b.-relation_decision_noint
. local pos : list posof "s01.a.-s01.b.-s01.b.b.-relation_decision_noint_other" in fields

. local var : word `pos' of `all'
invalid syntax
r(198);

[matthew-white]

@amandarudin, it's a similar error in that :list posof did not find a value it was expected to, but it's a different part of the do-file and likely a different cause. I can't tell from here what that is, but try contacting [email protected]. @boyercb @internetlindsey

[amandarudin]

Thanks so much Matthew, I have emailed IPA research support.

Have a great weekend,

Amanda

On Sat, Dec 19, 2015 at 12:08 PM, Matthew White [email protected]
wrote:

@amandarudin https://github.com/amandarudin, it's a similar error in
that :list posof did not find a value it was expected to, but it's a
different part of the do-file and likely a different cause. I can't tell
from here what that is, but try contacting
[email protected]. @boyercb https://github.com/boyercb
@internetlindsey https://github.com/internetlindsey

—
Reply to this email directly or view it on GitHub
#17 (comment)
.

Amanda Dahlstrand Rudin
Research Fellow
EPoD, Harvard Kennedy School

[liz-thomas]

Hi Matthew,

I'm getting a similar error when running odkmeta, but I can't find the text you said should be replaced in the odkmeta dofile. Is there another way to correct this problem?

See error message below. Thank you!
Amanda

* Duplicate variable name with s01.a.-s01.b.-s01.b.b.-relation_decision_noint
. local pos : list posof "s01.a.-s01.b.-s01.b.b.-relation_decision_noint_other" in fields

. local var : word `pos' of `all'
invalid syntax
r(198);

Curious if a resolution to this issue has been identified. TIA.

[matthew-white]

Hi @liz-thomas! This GitHub issue has to do with the "invalid syntax" error message that the following command issues in specific situations:

local child : word `pos' of `childfiles'

However, another command earlier in the do-file can issue the same error message:

local var : word `pos' of `all'

While the first case is something we still want to fix, generally the second case means that the data does not conform to the ODK Briefcase export CSV format, which odkmeta requires.

We've started directing odkmeta support requests to the ODK forum. Try checking out this topic about this error message there. If that doesn't help, feel free to create a new topic, and either I or another member of the community will try to help!