-
Notifications
You must be signed in to change notification settings - Fork 1
Statistics
Ideas for Angualr-Fronted Definitions should go to https://github.com/shownotes/snotes20-angular-webapp/wiki/Statistics
All definite definitions should go to https://github.com/shownotes/snotes20-restapi/wiki/API#apistatistics
Perform analyses of all publications on shownot.es and generate some cool statistics with that data. Therefore we need to generate word frequency tables and tf-idf tables on each episode of each podcast. We also need to build an overall corpus and generate the same tables for this. We can do this analyses on all text and on all URLs separately. First of all there must be a definition of the tables in the database. By getting a clear structure the statistic feature must be implemented as a separate django application namely statistic.
Later on, the data from the extracted features must be reachable via REST. By achieving this, we need to build a proper API to generate and deliver the data for each graph to the angular frontend. The graphs will be implemented with the library d3.js. Therefore it's necessary to discover which JSON data is needed for the graphs.
We need:
- Word frequency and tf-idf analysis on
- all episodes
- all episodes form a podcast (combined result)
- over the whole corpora
The application must not effect the running web application. Assuming that the most episodes are quit short we could calculate the word frequencies as thread after a publication was generated or updated. Therefore the last state_id must be used. By doing this we were able to use this data on the searching feature (wordcloud) immediately. The calculating of the whole corpus and the overall Podcast could be done by a cronjob once per day.
- Should be implemented in
- All episodes have a created_date but not a date
- Date is the date where the episode was live (discovered by hoersuppe API)
- Maybe we should use episodes numbers not dates as X on the graph
- Should be implemented in
- Should be implemented in