Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add issues and issue_comments streams #9

Merged
merged 4 commits into from
Sep 11, 2021
+128 −6
Merged

Add issues and issue_comments streams #9

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
59bdbb9
Update schema for issues stream
laurentS Sep 10, 2021
71b07b7
Update issue comments stream to be a child of repositories and update…
laurentS Sep 10, 2021
da12574
Add issue number to issue_comments stream
laurentS Sep 10, 2021
c9fcc33
Fix black style
laurentS Sep 11, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update issue comments stream to be a child of repositories and update… 
… schema
@laurentS
laurentS committed Sep 10, 2021
commit 71b07b7ba4cdfc13f7a2c651252d163206e5c56f
26 changes: 22 additions & 4 deletions tap_github/streams.py
View file
Original file line number Diff line number Diff line change
@@ -244,13 +244,17 @@ def http_headers(self) -> dict:


class IssueCommentsStream(GitHubStream):
"""Defines 'Issues' stream."""
"""
Defines 'Issues' stream.
Issue comments are fetched from the repository level (as opposed to per issue)
to optimize for API quota usage.
"""

name = "issue_comments"
path = "/repos/{org}/{repo}/issues/{issue_number}/comments"
path = "/repos/{org}/{repo}/issues/comments"
primary_keys = ["id"]
replication_key = "updated_at"
parent_stream_type = IssuesStream
parent_stream_type = RepositoryStream
state_partitioning_keys = ["repo", "org"]
ignore_parent_replication_key = False

@@ -278,11 +282,25 @@ def get_url_params(

schema = th.PropertiesList(
th.Property("id", th.IntegerType),
th.Property("node_id", th.StringType),
th.Property("repo", th.StringType),
th.Property("org", th.StringType),
th.Property("issue_number", th.IntegerType),
th.Property("issue_url", th.IntegerType),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the issue_number property was previously coming from parent context. What do you think of adding back via post_process() method?

Something like "/".split(record["issue_url"])[-1] might do the trick.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that! I'll add it in. It does seem handy to have the issue number available.

th.Property("updated_at", th.DateTimeType),
th.Property("created_at", th.DateTimeType),
th.Property("author_association", th.StringType),
th.Property("body", th.StringType),
th.Property(
"user",
th.ObjectType(
th.Property("login", th.StringType),
th.Property("id", th.IntegerType),
th.Property("node_id", th.StringType),
th.Property("avatar_url", th.StringType),
th.Property("gravatar_id", th.StringType),
th.Property("html_url", th.StringType),
th.Property("type", th.StringType),
th.Property("site_admin", th.BooleanType),
),
),
).to_dict()