Add fetch_logbook.py

[erikwallin86]

I figured out how to download the logbook and added this as 'fetch_logbook.py'. There were things in requestProblemsJson in fetch_problems which I did not understand. I made the new script as simple as possible, based on fetch_problems.py. The script simpy fetches the logbook, and saves it in logbook.json as well as in logbook_{username}.json.

(Thanks for making it possible to download moonboard data! I'm also working on a repo for visualizing the data, https://github.com/erikwallin86/moonboard-plots)

[musoke]

@erikwallin86, I'm curious about which parts of fetch_problems.py don't make sense - it's possible I could have been more concise in my additions there.

It does make sense to have this in a separate script though.

[spookykat]

Both look good to me. Will check and merge!

[erikwallin86]

@musoke First I must mention that I am not very familiar with neither the request package nor restapi :)
I didn't understand the 'n' parameter, which is hard coded as 0, but yet there is some if-else for when it's not. I didn't understand the second if-else either or what's special about the length 5000. Since it worked fine without this, I removed it for the fetch_logbook script.

Thanks for sorting out the fetching of problems for all holdsets! :)

[musoke]

I'm not sure exactly how @spookykat worked out the original implementation, but as far as I know the API is not documented publicly. I had to do some packet sniffing and guesswork to extend it to other holdsets. The implementation could very easily break if the official Moonboard app changes.

As far as n: The API returns only 5000 problems at a time. n tells it which problems you already have so it can return the next 5000 problems. I guess it's possible that this is required for the logbook too, but you'd need a large logbook to check. What you've done should be good enough for most people.

[spookykat]

I put a proxy between the android app and the server, the app doesn’t have ssl pinning. That made it quite easy to find the necessary API calls!

[erikwallin86]

I tried to setup a proxy server and test this myself but I think I have only managed to catch the http traffic. I'm curious what calls are made when searching and entering a user profile to see the summary and logbook (last session), and if this actually gives access to the entire logbook. Have you tried this out @spookykat ?

(I have no goto proxy-server but I tried with 'Fiddler' and 'postman'.)

[erikwallin86]

OK, to answer my own question: Finally managed to get the proxy working, using mitmproxy in 'wireguard' mode. Turns out the user information is the same as in the app: the summary graph and last session. If one would like to retrieve that information it's an url like https://restapimoonboard.ems-x.com/v1/_moonapi/users/userprofile/<uuid-version4-of-length-32(hex)>/17/1?v=8.3.4 where 17 and 1 in the end are the holdset/angle mappings. If one wants to do a user search, this is done as e.g.
URL = "https://restapimoonboard.ems-x.com/v1/_moonapi/Users/Search"
query = {"Query": "the-search-string"}
json = requests.post(URL, headers=headers, json=query).json()
This returns a json list with a dict for each user and includes the uuid required to retrieve the userprofile.

[smchase]

Any idea how to query the API for data on the breakdowns of user attempts, user ratings, and user stars, as well as comments? I tried fiddling around with a proxy but didn't get anything usable, so any tips would be appreciated :)

[spookykat]

@smchase I used burp as a proxy on my pc then I connected my android phone to it. Don’t know about the app now but when I did it they didn’t use SSL pinning so you could see all https traffic.

[erikwallin86]

OK, since this was still open I pushed another two commits. The idea is to enable overriding the username and password in secret.py using keyword arguments, but does not change the default behavior. My motivation is to be able to import fetch_logbook.request_data into other scripts and use to download logbooks.

I added the possibility to send username and password to getRefreshToken as keyword arguments. request_data in fetch_logbook.py is given a 'access_kwargs' parameter which is passed to getAccessToken. By default it's empty so nothing is passed and the defaults from secret.py are used.

I also added a if __name__ == '__main__': to simplify imports from fetch_logbook.py

[erikwallin86]

Any idea how to query the API for data on the breakdowns of user attempts, user ratings, and user stars, as well as comments? I tried fiddling around with a proxy but didn't get anything usable, so any tips would be appreciated :)

From what I have seen, this data is taken from moonboard.com, and not the restapi. It seem to be possible to access some problem page like https://www.moonboard.com/Problems/View/192428/ironarm, where 192428 is the 'problemId' ironarm is the name in lowercase letters. However, this does not show comments or any breakdowns. From what I've seen it is possible to get this information as https://www.moonboard.com/Problems/Details/20153/<128-bit-random-hex-string-assosiated-with-problem>?ui=<128-bit-hex-string-for-some-user>. The data you get is html-code, exactly as it's shown in the app. The problem is that I have no idea how to get the '128-bit-random-hex-string-assosiated-with-problem' except that I got it for one problem with the proxy server...

[erikwallin86]

Hi! In case anyone would find it interesting. I've gotten my moonboard-plots repo into decent shape and created a webpage where one can get plots from logbook data. The site is https://www.moonplt.com/ and it also contains some examples. Sorry for the spamming in the pull request!

[smchase]

Just thought I'd mention that based on the stuff in this thread, I was able to get a guidebook site going with some extra info compared to the app like a sandbag score based on user ratings: https://moonboard.herokuapp.com. Thanks!