Dataset Collector #1, Kazyulina Marina - 19FPL1 #42
Conversation
config/constants.py
Outdated
| Useful constant variables | ||
| """ | ||
|
|
||
| import os |
There was a problem hiding this comment.
all constants should be left in constants.py in the root of the project
| @@ -7,3 +7,5 @@ | |||
| PROJECT_ROOT = os.path.dirname(os.path.realpath(__file__)) | |||
| ASSETS_PATH = os.path.join(PROJECT_ROOT, 'tmp', 'articles') | |||
| CRAWLER_CONFIG_PATH = os.path.join(PROJECT_ROOT, 'crawler_config.json') | |||
| LINKS_STORAGE = os.path.join(PROJECT_ROOT, 'links') | |||
| URL_START = 'https://burunen.ru' | |||
There was a problem hiding this comment.
if it is seed url or any other configuration of the crawler, then use configuration file
There was a problem hiding this comment.
nope, it is the beginning of every url in my sourse
you see, as in many webpages, the href tags in my source are filled with the continuation of the url, emitting the 'https://burunen.ru', so each and every time i try to request the link i found automatically i have to concatenate it with this beginning first.
i thought about making an additional attribute to the Crawler class (something like self.url_start = "), but then i decided that if it is constant, it should be placed among other constants... am i wrong?
demid5111
added
the
Changes required
dmitry-uraev
added
the
Review Required
|
|
||
| def _scan_dataset(self): | ||
| """ | ||
| Register each dataset entry | ||
| """ | ||
| pass | ||
| path = Path(ASSETS_PATH) | ||
| for file in path.iterdir(): |
There was a problem hiding this comment.
can you find a way to directly get files by the given name template?
demid5111
added
Changes required
pipeline.py
Outdated
| raise NotADirectoryError | ||
| if not list(path.iterdir()): | ||
| raise EmptyDirectoryError | ||
| files = [str(file.relative_to(path)) for file in path.iterdir()] |
pipeline.py
Outdated
|
|
||
| def get_articles(self): | ||
| """ | ||
| Returns storage params | ||
| """ | ||
| pass | ||
| self._scan_dataset() |
There was a problem hiding this comment.
it cannot be in the plain getter - read the requirements
demid5111
added
🏆 Pipeline accepted
🥇
and removed
Changes required
No description provided.