scm collector not collecting pull request and issues data Also It appends bucket.org in the url when passed like ( https://[email protected]/team/repo.git)

[Sjnahak]

Hi ,

I am configuring collector for bitbucket cloud, I am able to get commits for branch but not the pull request and issues.

Here is my appllication.properties

dbname=dashboarddb
dbhost=localhost
dbport=27017
dbusername=dashboarduser
dbpassword=dbpassword
logging.file=./logs/bitbucket.log
git.cron=0 0/5 * * * *
git.host=api.bitbucket.org/
git.api=/api/2.0/repositories/
[email protected](mail_id)
git.password=password (base 64)
git.commitThresholdDays=15
git.pageSize=25
git.product=cloud
bitbucket.key=generatedkey

Also in api.properties i have

key=generatedkey

In The widget i have passed

Repo type : bitbucket
Repo URL: https://bitbucket.org/comapneyname/reponame
username: user id
password : password

Error logs :

2020-09-13 04:45:00,000 [taskScheduler-1] INFO  c.c.d.collector.CollectorTask - Running Collector: Bitbucket
2020-09-13 04:45:00,015 [taskScheduler-1] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:559, serverValue:2759}] to localhost:27017
2020-09-13 04:45:00,017 [taskScheduler-1] INFO  c.c.d.collector.CollectorTask - -----------------------------------
2020-09-13 04:45:00,017 [taskScheduler-1] INFO  c.c.d.collector.CollectorTask - Starting...
2020-09-13 04:45:00,017 [taskScheduler-1] INFO  c.c.d.collector.CollectorTask - -----------------------------------
2020-09-13 04:45:01,465 [taskScheduler-1] INFO  c.c.d.collector.GitCollectorTask - {password=enxrypted, personalAccessToken=, branch=dev, userID=emaild, url=https://bitbucket.org/companyname/reponame, lastUpdateCommit=002111624e9d4223632c17da494674a4145d3181, lastUpdate=1599906901415}::dev::get pulls
2020-09-13 04:45:01,465 [taskScheduler-1] INFO  c.c.d.collector.PullRequestCollector - Executing [https://bitbucket.org/api/2.0/repositories/team/reponame/pullrequests?at=refs%2Fheads%2Fdev&state=all&limit=0
2020-09-13 04:45:02,303 [taskScheduler-1] ERROR c.c.d.collector.PullRequestCollector - Exception block: null
2020-09-13 04:45:02,304 [taskScheduler-1] INFO  c.c.d.collector.CollectorTask - Repo Count              1        2s
2020-09-13 04:45:02,304 [taskScheduler-1] INFO  c.c.d.collector.CollectorTask - New Commits             0        2s
2020-09-13 04:45:02,304 [taskScheduler-1] INFO  c.c.d.collector.CollectorTask - Finished                         2s
2020-09-13 04:46:06,670 [pool-3-thread-1] INFO  org.mongodb.driver.connection - Closed connection [connectionId{localValue:559, serverValue:2759}] to localhost:27017 because it is past its maximum allowed idle time.
^C

UI output :

Let me know what am i missing to get the full data.

[Sjnahak]

Also i tried to generate encrypted key using https://hygieia.github.io/Hygieia/collectors.html#encryption-for-private-repos

In hygieia core dir i get below error:

/opt/hygieia-core/target# java -jar core-3.10.0.jar  com.capitalone.dashboard.util.Encryption
no main manifest attribute, in core-3.10.0.jar

[Sjnahak]

Hi This issue helped me clear my doubt on what to password field in widget repo hygieia/hygieia#2219
Also generating the API Key.

But Still I am not getting the pull request details:
I get the below error when we pass url in the widget : https://[email protected]/team/repo.git

bucket.org/ is getting appended

2020-09-14 03:45:00,160 [taskScheduler-1] ERROR c.c.d.c.DefaultBitbucketCloudClient - 404 Not **Found:https://bitbucket.org/api/2.0/repositories/bucket.org/team/reponame/commits?sha=dev&since=2020-08-31T03:45Z   (problem line)**
2020-09-14 03:45:00,160 [taskScheduler-1] INFO  c.c.d.collector.GitCollectorTask - {password=encrypt, personalAccessToken=, branch=dev, userID=userid, url=https://[email protected]/team/reponame}::dev::get pulls
2020-09-14 03:45:00,160 [taskScheduler-1] INFO  c.c.d.collector.PullRequestCollector - Executing [https://bitbucket.org/api/2.0/repositories/team/reponame/pullrequests?at=refs%2Fheads%2Fdev&state=all&limit=25
2020-09-14 03:45:01,116 [taskScheduler-1] ERROR c.c.d.collector.PullRequestCollector - Exception block: null

Any help on it?

[Sjnahak]

@rvema @chtompki @Sbrenthughes any thoughts and inputs here , still struggling.

[shishirchaturvedi]

@Sjnahak We are using an on-prem instance of Bitbucket Server. So I have never tested cloud-related code in the collector. As I can't test this code so I can't help much here than taking a look at that code. I will take a look at the DefaultBitbucketCloudClient and will get back to you.
//SC

[shishirchaturvedi]

I see that @raghulraj has the most recently enhanced BitbucketApiUrlBuilder to support pull request endpoint for the cloud Bitbucket. @raghulraj Can comment on this issue.
@Sjnahak Can you access REST API directly through any REST client? This will nullify any problem with the REST Endpoint configuration. Please try to access the PR endpoint through the REST client/browser - https://bitbucket.org/api/2.0/repositories/team/reponame/pullrequests?at=refs%2Fheads%2Fdev&state=all&limit=0, you might need to supply user credentials in the Authorization Header.
//SC

[Sjnahak]

I get below out while trying the above url

Access denied. You must have write or admin access.

[shishirchaturvedi]

@Sjnahak Are you providing your username/password as well while accessing REST API?

[Sjnahak]

@shishirchaturvedi I am getting output now tried using curl .

[shishirchaturvedi]

@Sjnahak This is great. Now you can match that URL and input from the collector log to identify potential problems.

[Sjnahak]

@shishirchaturvedi That is the same url executed when collector runs correct?

[shishirchaturvedi]

@Sjnahak It should be, if not then you can find the problem by comparing both urls.

[Sjnahak]

Both url's are same. Can't find any difference also does repo widget support multiple branch

[shishirchaturvedi]

@Sjnahak What http response code are you getting in your collector log? is it 400(bad request) or 404(not found) or 401/403?

[Sjnahak]

1. in repo widget when i passed url like https://bitbucket.org/comapneyname/reponame
   I get commits , but pull request part null block

2020-09-13 04:45:01,465 [taskScheduler-1] INFO  c.c.d.collector.GitCollectorTask - {password=enxrypted, personalAccessToken=, branch=dev, userID=emaild, url=https://bitbucket.org/companyname/reponame, lastUpdateCommit=002111624e9d4223632c17da494674a4145d3181, lastUpdate=1599906901415}::dev::get pulls
2020-09-13 04:45:01,465 [taskScheduler-1] INFO  c.c.d.collector.PullRequestCollector - Executing [https://bitbucket.org/api/2.0/repositories/team/reponame/pullrequests?at=refs%2Fheads%2Fdev&state=all&limit=0
2020-09-13 04:45:02,303 [taskScheduler-1] ERROR c.c.d.collector.PullRequestCollector - Exception block: null

2. When i pass url like https://[email protected]/team/repo.git
   I get null block for pull request and 404 commits as bucket.org gets append during collector run where i receive 404 error

2020-09-14 03:45:00,160 [taskScheduler-1] ERROR c.c.d.c.DefaultBitbucketCloudClient - 404 Not **Found:https://bitbucket.org/api/2.0/repositories/bucket.org/team/reponame/commits?sha=dev&since=2020-08-31T03:45Z   
**

2020-09-14 03:45:00,160 [taskScheduler-1] INFO  c.c.d.collector.GitCollectorTask - {password=encrypt, personalAccessToken=, branch=dev, userID=userid, url=https://[email protected]/team/reponame}::dev::get pulls
2020-09-14 03:45:00,160 [taskScheduler-1] INFO  c.c.d.collector.PullRequestCollector - Executing [https://bitbucket.org/api/2.0/repositories/team/reponame/pullrequests?at=refs%2Fheads%2Fdev&state=all&limit=25
2020-09-14 03:45:01,116 [taskScheduler-1] ERROR c.c.d.collector.PullRequestCollector - Exception block: null

[shishirchaturvedi]

@Sjnahak What happens when you curl this URL from option 1 you mentioned - "https://bitbucket.org/api/2.0/repositories/team/reponame/pullrequests?at=refs%2Fheads%2Fdev&state=all&limit=0" ?

[Sjnahak]

@shishirchaturvedi when i fire a curl command for option 1
i can see output something like below but why in collector run it fails

{"pagelen": 10, "values": [{"description": "### JIRA tickets\r\n\r\n---\r\n\r\n[jira id](https://atlassian.net/browse/jira id)\r\n\r\n### What's the change ?\r\n\r\n---\r\n\r\n* commentsfrom <20% to 89.9%\r\n* comments\\( `Op` etc\\) couldn’t be covered.  \r\n    \r\n\r\n### Any questions ?\r\n\r\n---\r\n\r\nNone\r\n\r\n### How it has been tested ?\r\n\r\n---\r\n\r\nUnit tests\r\n\r\n### Prerequisite tasks\r\n\r\n---\r\n\r\nNone\r\n\r\n### Additional Details\r\n\r\n---\r\n\r\nNone\r\n\r\n### Checklist\r\n\r\n---\r\n\r\n* \\[x\\] Check that PR title is clear and has JIRA ticket ID.\r\n* \\[x\\] Verify Source and Base branches.\r\n* \\[x\\] Ensure  \\(both positive and negative\\).\r\n* \\[x\\] .\r\n* \\[ \\] Ensure that package version \\(\\\\) is updated, and  file is updated with the details of this PR.\r\n* \\[x\\] Add reviewers and assign to respective person \\(yourself\\).\r\n\r\n‌", "links": {"decline": {"href": "https://bitbucket.org/!api/2.0/repositories/team/reponame/pullrequests/69/decline"}, "diffstat": {"href": "https://bitbucket.org/!api/2.0/repositories/team/reponame/diffstat/team/reponame:a90b9e16d6d8%0Dcf6d30450c51?from_pullrequest_id=69"}, "commits": {"href": "https://bitbucket.org/!api/2.0/repositories/team/reponame/pullrequests/69/commits"}, "self": {"href": "https://bitbucket.org/!api/2.0/repositories/team/reponame/pullrequests/69"}, "comments": {"href": "https://bitbucket.org/!api/2.0/repositories/team/reponame/pullrequests/69/comments"}, "merge": {"href": "https://bitbucket.org/!api/2.0/repositories/team/reponame/pullrequests/69/merge"}, "html": {"href": "https://bitbucket.org/team/reponame/pull-requests/69"}, "activity": {"href": "https://bitbucket.org/!api/2.0/repositories/team/reponame/pullrequests/69/activity"}, "diff": {"href": "https://bitbucket.org/!api/2.0/repositories/team/reponame/diff/team/reponame:a90b9e16d6d8%0Dcf6d30450c51?from_pullrequest_id=69"},

[Sjnahak]

@shishirchaturvedi any input ?

[shishirchaturvedi]

@Sjnahak The collector log is not showing the right reason for the Exception. Could you please increase the log level of Sprintframewok packages to DEBUG so that we can get more information?
You can set log levels to DEBUG in the application properties file.

logging.level.com.capitalone.dashboard=DEBUG
logging.level.org.springframework=DEBUG

[Sjnahak]

@shishirchaturvedi After checking the debug logs i was getting 403 error for my userid , used different user id which resolved the 403 error and we can see pull request data in the collector logs but in UI Pull request data seems to be missing.

Also Same issue we are facing with JIRA, where jira collector collects the data successfully but not visible in the UI.

[Sjnahak]

@shishirchaturvedi I guess repo widget in the UI Also not refreshing now

[shishirchaturvedi]

@Sjnahak Can you please enable DEBUG logs for API module as well to get more insight into what's going on there? Sounds trivial but please make sure that data is getting populated in MongoDB collections of hygieia db instance and collector/api is using a common database instance.
//SC

[Sjnahak]

@shishirchaturvedi i can see data in hygieia db but looks like lot of data is missing in mongodb.
Also gitrequest collections is empty but commits collection has data.

I will enable debug logs for api what information you want me capture from api logs?

[shishirchaturvedi]

@Sjnahak Gitrequest collection should have some data. This explains why you are not seeing any PR data on the widget. Do you see this log message in your SCM collector logs - pull Request count when that collector finishes. For API logs I wanted you to see log details to narrow the problem area. You should try to troubleshoot SCM collectors to make sure that appropriate collection objects get populated.
//SC

[Sjnahak]

@shishirchaturvedi i don't see this message "collector logs - pull Request count when that collector finishes"
but i can see lot of PR details in the logs not in database.
When collector finishes i can see:

2020-09-22 13:30:08,867 [taskScheduler-1] INFO  c.c.d.collector.CollectorTask - Repo Count              3        8s
2020-09-22 13:30:08,867 [taskScheduler-1] INFO  c.c.d.collector.CollectorTask - New Commits             0        8s
2020-09-22 13:30:08,867 [taskScheduler-1] INFO  c.c.d.collector.CollectorTask - Finished                         8s

When collector is running i can see something like for commits collections, same way it should happen for Gitrequest collection which is not happening.

2020-09-22 13:15:06,662 [taskScheduler-1] DEBUG o.s.data.mongodb.core.MongoTemplate - findOne using query: { "collectorItemId" : { "$oid" : "5f683eecc9f7411c8909f83d"}
, "scmRevisionNumber" : "d4415bf5d0a54c9efb9693c56dfae14cd5c6ec79"} in db.collection: dashboarddb.commits

I can see below in the logs as well

2020-09-22 13:30:08,497 [taskScheduler-1] DEBUG o.s.web.client.RestTemplate - GET request for "https://bitbucket.org/api/2.0/repositories/team/reponame/pullrequests?at=refs%2Fheads%2Fdev&state=all&limit=25" resulted in 200 (OK)
2020-09-22 13:30:08,498 [taskScheduler-1] DEBUG o.s.web.client.RestTemplate - Reading [java.lang.String] as "application/json;charset=utf-8" using [org.springframework.http.converter.StringHttpMessageConverter@5dab73f]
2020-09-22 13:30:08,865 [taskScheduler-1] ERROR c.c.d.collector.PullRequestCollector - Exception block: null
2020-09-22 13:30:08,865 [taskScheduler-1] DEBUG o.s.b.f.s.DefaultListableBeanFactory - Returning cached instance of singleton bean 'enableEncryptablePropertySourcesPostProcessor'

[shishirchaturvedi]

@Sjnahak This is exception - ERROR c.c.d.collector.PullRequestCollector - Exception block: null. Could you please attach debugger to the collector and troubleshoot. I suspect some data issues in PR collector. I don't have access to your data so can't comment much. Please DM me on Gitter if you need to discuss about it.
//SC

[Sjnahak]

@shishirchaturvedi Pinged you in gitter

[secheverri]

Hi @Sjnahak, I'm getting the same issue "ERROR c.c.d.collector.PullRequestCollector - Exception block: null", could you please share the possible solution here when you solve it.
Thanks.

[Sjnahak]

@secheverri, @shishirchaturvedi has suggested some changes will do once the testing is successful