Add issues and issue_comments streams #9
Conversation
| th.Property("repo", th.StringType), | ||
| th.Property("org", th.StringType), | ||
| th.Property("issue_number", th.IntegerType), | ||
| th.Property("issue_url", th.IntegerType), |
There was a problem hiding this comment.
I think the issue_number property was previously coming from parent context. What do you think of adding back via post_process() method?
Something like "/".split(record["issue_url"])[-1] might do the trick.
There was a problem hiding this comment.
I like that! I'll add it in. It does seem handy to have the issue number available.
| th.Property("updated_at", th.DateTimeType), | ||
| th.Property("created_at", th.DateTimeType), | ||
| # th.Property("closed_at", th.DateTimeType), # Nulls causing parse error | ||
| th.Property("closed_at", th.DateTimeType), |
There was a problem hiding this comment.
Thanks for restoring this. I think the null behavior problem came up very early in testing and I forgot to retest and restore the field.
There was a problem hiding this comment.
@laurentS - I think we can merge this quickly. Just one requested change regarding issue number on the comments stream. Let me know what you think. Thanks!
|
I've update the code as per your comments. I've been bothered by #11 but it's not critical for merging this, it just lets too much data through I think. |
This PR does a couple of things:
issuesstream schema to match the docs (and actual API output) a bit more closely. I've not included all fields, namely excluding the "obvious" ones that can be computed trivially by appending a common postfix to the base url.issue_commentsstream to be a child ofrepositoriesas discussed in Using different API endpoints for a "same" stream #8 and update the schema like aboveDepending on how quickly this gets merged, I might add
issue_eventshere, or in a separate PR, following the same logic.