Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Term 2023 issues #18

Open
mmaelicke opened this issue Jan 11, 2023 · 18 comments
Open

Term 2023 issues #18

mmaelicke opened this issue Jan 11, 2023 · 18 comments

Comments

@mmaelicke
Copy link
Member

@maxschmi for reference

This is a report on errors, warnings and issues logged by the server while uploading this years data.

Referencing @data-hydenv/students for your information. If you recognize some of the information below, you are welcome to comment, otherwise just ignore the mentioning.

metadata

I guess this is a student who dropped out.

Some entries are missing metadata:
----------------------------------
            name  hobo_id github_name data_available region  latitude  longitude exposition altitude influence description
14  ***, ***      NaN         NaN            NaN    NaN       NaN        NaN        NaN      NaN       NaN         NaN

Additionally, there was an exception, which means there might be missing metadata

[ERROR]: (psycopg2.errors.UndefinedFunction) operator does not exist: character varying = timestamp without time zone
LINE 3: ...RE details.key = 'altitude' AND details.str_value = '1900-01...
                                                             ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.

[SQL: SELECT details.id AS details_id, details.key AS details_key, details.str_value AS details_str_value, details.int_value AS details_int_value, details.float_value AS details_float_value, details.bool_value AS details_bool_value
FROM details
WHERE details.key = %(key_1)s AND details.str_value = %(str_value_1)s
 LIMIT %(param_1)s]
[parameters: {'key_1': 'altitude', 'str_value_1': datetime.datetime(1900, 1, 2, 12, 0), 'param_1': 1}]
(Background on this error at: https://sqlalche.me/e/14/f405)

This basically means one student had a date in the altitude column, which was parsed as the 2.1.1900 12:00. Maybe putted 2.1 into the cell and Excel formatted as a date?

Raw data

These are either extra HOBO time series and can be ignored, or the Metadata file references the wrong HOBO (typo?)

File ./data-master/hobo/2023/raw/10350089.csv references HOBO ID=10350089, which is not found.
File ./data-master/hobo/2023/raw/10347324.csv references HOBO ID=10347324, which is not found.
File ./data-master/hobo/2023/raw/10347327.csv references HOBO ID=10347327, which is not found.
File ./data-master/hobo/2023/raw/10347333.csv references HOBO ID=10347333, which is not found.
File ./data-master/hobo/2023/raw/10350107.csv references HOBO ID=10350107, which is not found.
File ./data-master/hobo/2023/raw/10347366.csv references HOBO ID=10347366, which is not found.
@MarilenaAlmeida
Copy link
Contributor

The value in the altitude column was supposed to be 2. Unfortunately I can no longer access the file to correct this.

@mmaelicke
Copy link
Member Author

The value in the altitude column was supposed to be 2. Unfortunately I can no longer access the file to correct this.

Thanks for reporting. I corrected this.

@maxschmi
Copy link
Contributor

Hey Mirko,
Yes their are additional HObos that are not in the Meta table. Those were my spare Hobos I did install as a backup.

@mmaelicke
Copy link
Member Author

@data-hydenv/students to let you all know, I have a minor issue uploading the quality-checked data. I will solve that during the day and then the quality checked data will be available in the database, as well.

@mmaelicke
Copy link
Member Author

@data-hydenv/students the 2023 quality checked data is uploaded.

There were some warnings and errors, read through them and reply if you feel like you know something about it.

File ./data-master/hobo/2023/hourly/10347324_Th.csv references HOBO ID=10347324, which is not found.

Parsing file './data-master/hobo/2023/hourly/10347355_Th.csv' was not successfull.
Do not edit the files by hand!
Error: 'DataFrame' object has no attribute 'tstamp' 

File ./data-master/hobo/2023/hourly/10350089_Th.csv references HOBO ID=10350089, which is not found.

Parsing file './data-master/hobo/2023/hourly/10349996_Th.csv' was not successfull.
Do not edit the files by hand!
Error: time data '2022.12.01. 00:00:00' does not match format '%Y.%m.%d %H:%M:%S' 

File ./data-master/hobo/2023/hourly/10347333_Th.csv references HOBO ID=10347333, which is not found.

Parsing file './data-master/hobo/2023/hourly/10350049_Th.csv' was not successfull.
Do not edit the files by hand!
Error: 'DataFrame' object has no attribute 'tstamp' 

File ./data-master/hobo/2023/hourly/10350107_Th.csv references HOBO ID=10350107, which is not found.

The error Error: 'DataFrame' object has no attribute 'tstamp' indicates, that reading the ddtm column failed. If it was your file, please check it carefully.

Respond to this thread, if you changed something about your data

@arrpit19
Copy link
Contributor

arrpit19 commented Jan 20, 2023 via email

@kg200
Copy link
Contributor

kg200 commented Jan 20, 2023 via email

@nico51two
Copy link
Contributor

nico51two commented Jan 20, 2023 via email

@adia-saurey
Copy link
Contributor

Hello,
The ref_id and ref_r2 columns for hobo id 10350095 have been added.
Thank you!

@MarilenaAlmeida
Copy link
Contributor

Hi,
I've corrected and re-uploaded the data from id 10347333.
I hope now it works!

@arrpit19
Copy link
Contributor

arrpit19 commented Jan 20, 2023 via email

@mmaelicke
Copy link
Member Author

@arrpit19, @MarilenaAlmeida, @nico51two, @kg200 thanks for your quick replies!

We produced a new error :)

[ERROR]: File: './data-master/hobo/2023/hourly/10350049_Th.csv' errored. Message: (psycopg2.errors.DatatypeMismatch) column "value" is of type real but expression is of type text
LINE 1: ...ity_flag_id) SELECT meta_id, variable_id, tstamp, value, qua...
                                                             ^
HINT:  You will need to rewrite or cast the expression.

[SQL: INSERT INTO data (meta_id, variable_id, tstamp, value, quality_flag_id) SELECT meta_id, variable_id, tstamp, value, quality_flag_id FROM hnnwjjpjmcizpgpz ON CONFLICT (meta_id, variable_id, tstamp) DO NOTHING]
(Background on this error at: https://sqlalche.me/e/14/f405)

it is the file: './data-master/hobo/2023/hourly/10350049_Th.csv', please check the temperature column. I guess there is some kind of character there, as it presents itself as a character datatype.

@nico51two
Copy link
Contributor

yeah sorry I only now managed to work my backlog from being sick in the first week so I guess my file still had NAs in there... i'm finishing up as we speak and will upload in ~5 mins so check again in 15 :D ...sorry for the inconvenience

@nico51two
Copy link
Contributor

Aaaaaaand it's up...thanks for your patience

@xamfru
Copy link
Contributor

xamfru commented Jan 24, 2023

@mmaelicke

My uploaded file was read in correctly, when i was trying to aggregate for calculating indices of all HOBOS in 2023 i saw that there are only 38 unique timestamps in my file. So i uploaded it with corrected timestamps. Is it possible to update the db data table once again?

Sorry for the inconvience

@mmaelicke
Copy link
Member Author

@mmaelicke

My uploaded file was read in correctly, when i was trying to aggregate for calculating indices of all HOBOS in 2023 i saw that there are only 38 unique timestamps in my file. So i uploaded it with corrected timestamps. Is it possible to update the db data table once again?

Sorry for the inconvience

Thanks for correcting the file. I will try to update everything this evening, but everyone's really busy with the database right now, and I don't want to interrupt. If your data in the database is a bit messed up, don't worry and continue with the version online, that's alright.

@kg200
Copy link
Contributor

kg200 commented Jan 25, 2023

@mmaelicke
I still can't find my HOBO in the metadata. I changed the ID about a week ago, because I had to use one of Max's HOBOs.
Did you also update the metadata table? If so I'll probably need to do some changes again.

@kg200
Copy link
Contributor

kg200 commented Jan 27, 2023

Hey group!
Did anyone succed to have different icons for the different terms and a color coding for the mean temperatures at the same time in plot_ly?
Would be happy if you someone coud help me there!!
Thank you! And have a nice weekend to all that are ready with their report :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@mmaelicke @maxschmi @nico51two @xamfru @arrpit19 @MarilenaAlmeida @kg200 @adia-saurey