Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataset Collector #1, Chudinova Alla - 19FPL2 #37

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Dataset Collector #1, Chudinova Alla - 19FPL2 #37

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
8e7ef83
Minor changes for PR
x-ae-a-12 Mar 6, 2021
9b85d53
[DEV] Updated due to main
dmitry-uraev Mar 6, 2021
3093ddf
Scrapper commitment
x-ae-a-12 Mar 9, 2021
d457510
Updated my scrapper.py
x-ae-a-12 Mar 10, 2021
8b29ba8
Article.py update
x-ae-a-12 Mar 10, 2021
e180f47
I SHOULD PASS
x-ae-a-12 Mar 10, 2021
660637c
Merge remote-tracking branch 'upstream/main' into HEAD
dmitry-uraev Mar 12, 2021
13a0982
Updated raw_metadata_test
x-ae-a-12 Mar 12, 2021
de6c71f
Merge remote-tracking branch 'upstream/main' into HEAD
dmitry-uraev Mar 26, 2021
bd71d25
added a line to .pylintrc
x-ae-a-12 Apr 3, 2021
d9ee31e
added pipeline realisation
x-ae-a-12 Apr 3, 2021
5ced262
updated due to lint
x-ae-a-12 Apr 3, 2021
d831701
fix analysis
x-ae-a-12 Apr 3, 2021
bbe41b0
added INTJ tag
x-ae-a-12 Apr 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions scrapper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ class Crawler:
Crawler implementation
"""
def __init__(self, seed_urls: list, max_articles: int):
pass
self.seed_urls = []
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

initialize other params in the constructor


@staticmethod
def _extract_url(article_bs):
Expand Down Expand Up @@ -94,4 +94,8 @@ def validate_config(crawler_path):

if __name__ == '__main__':
# YOUR CODE HERE
pass
response = requests.get('')
print('')
sleep (5) #import time first
response = request.get
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's start to implement code. I think it's a good time.

#ILY
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

2 changes: 1 addition & 1 deletion target_score.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Target score for scrapper.py:
6
8
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that s good switch.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that s good switch.

No, I do not think so. It is eminent I'd better switch to 4 or 6 later.


# Target score for pipeline.py:
0