feat: 0.2.0 update

CHANGELOG.md

@@ -4,14 +4,33 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
+## [0.2.0](https://github.com/WoozyMasta/guassp/releases/tag/0.1.1) - 2022-05-22
+
+### Added
+
+* `/task_manual/<int:prj_id>` endpoint;
+* Workaround for searching for a project by the full name of the project in
+  SonarQube if the search for ALM integration was unsuccessful.  
+  A fully qualified name equal to `$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME` or
+  `project.path_with_namespace` is recommended when registering a project.
+* An item with [requirements](README.md#requirements) for the normal operation
+  of the solution has been added to the project documentation
+
+### Changed
+
+* Only the project ID is published to the queue, the full status of the task was
+  previously published, which could lead to data disclosure.
+* Update project preparation script example `extra/sq-integration-taks.sh`,
+  updated logic for searching and creating a project
+
 ## [0.1.1](https://github.com/WoozyMasta/guassp/releases/tag/0.1.1) - 2022-05-06
 
 ### Added
 
-* `/health` endpoint
-* docker-compose example
-* nginx config example
-* improve documentation
+* `/health` endpoint;
+* docker-compose example;
+* nginx config example;
+* improve documentation.
 
 ## [0.1.0](https://github.com/WoozyMasta/guassp/releases/tag/0.1.0) - 2022-05-06
 

README.md

@@ -8,6 +8,7 @@ in the same way as project permissions in GitLab.
 > This document is available in languages: [eng 🇬🇧][], [ua 🇺🇦][], [rus 🇷🇺][]
 
 * [Implementation](#implementation)
+* [Requirements](#requirements)
 * [Components](#components)
 * [Container image](#container-image)
 * [Quick start](#quick-start)
@@ -36,6 +37,46 @@ in accordance with GitLab.
 ![permissions][]  
 ![role-mapping][]
 
+## Requirements
+
+### Necessarily
+
+* SonarQube must have configured [ALM][] integration with your GitLab, and you
+  must specify the name of the integration key in `SONARQUBE_ALM_KEY`;
+* Authentication in SonarQube must occur only through [ALM][] GitLab, otherwise
+  the search for users will be violated, the rights are synchronized only for
+  the explicitly corresponding account that came from GitLab;
+* You must be able to provide an access token with administrative privileges to
+  SonarQube in `SONARQUBE_TOKEN` for the worker to work correctly;
+* You must create a service user in GitLab, which must have at least
+  **developer** access in all projects that you want to analyze in SonarQube,
+  with this user you must log in to SonarQube and give him global rights to
+  perform analysis and create projects. Then you can use that user's token:
+  * User token received in SonarQube as `SONARQUBE_TOKEN` used in CI pipeline to
+    perform analysis and project management.
+  * User token received in GitLab as `GITLAB_TOKEN` for worker, needed to get a
+    list of all members of a group or project, including inherited and invited
+    members
+* The name of the project registered in SonarQube must be equal to
+  `$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME` and the project key is recommended to
+  be set to `gitlab:$CI_PROJECT_ID`. Only with these settings, synchronization
+  will be successful, other states are possible but have not been tested.
+
+### Recommended
+
+* The best experience will be gained by using the
+  [sonarqube-community-branch-plugin][] plugin which adds branch analysis and PR
+  support to your projects. Also here you will need a previously created system
+  user in GitLab with **developer** rights, which will allow you to decorate MR
+  in your projects.
+* For transparent integration, it is best to use the example script
+  [sq-integration-taks] for the GitLab CI pipeline, otherwise other project
+  registration options may not work correctly or even break the work. The
+  example is also tailored for the presence of the
+  [sonarqube-community-branch-plugin][] plugin and the execution of OWASP
+  DependencyCheck
+* Use Prometheus and Grafana metrics to analyze synchronizer performance.
+
 ## Components
 
 The project is implemented on [flask][], for WSGI it is used [bjoern][],
@@ -148,24 +189,33 @@ curl -sL http://127.0.0.1:5000/tasks | jq -er '.tasks | keys'
 
 ### Task Status
 
-> GET **`/task/<job_uuid>`**
+> GET **`/task/<uuid:job_uuid>`**
 
 ```bash
 curl -sL http://127.0.0.1:5000/task/8b155172-cfcf-4777-b9f4-bfce53b6eb0e | jq
 ```
 
 ### Removing a task from the queue
 
-> DELETE **`/task/<job_uuid>`**
+> DELETE **`/task/<uuid:job_uuid>`**
 
 ```bash
 curl -sL http://127.0.0.1:5000/task/8b155172-cfcf-4777-b9f4-bfce53b6eb0e \
   -X DELETE | jq
 ```
 
+### Manual task registration by GitLab project ID
+
+> POST **`/task_manual/<int:prj_id>`**
+
+```bash
+curl -sL http://127.0.0.1:5000/task_manual/111 \
+  -X POST | jq
+```
+
 ### Health
 
-> GET **`/task/health`**
+> GET **`/health`**
 
 API health, application availability and job queues.
 
@@ -266,6 +316,7 @@ And for simplicity, run through a script `guassp`
 [rq]: https://github.com/rq/rq
 [rq-exporter]: https://github.com/mdawar/rq-exporter
 [docker-compose]: https://docs.docker.com/compose/
+[sonarqube-community-branch-plugin]: https://github.com/mc1arke/sonarqube-community-branch-plugin
 
 <!-- Containers -->
 [quay]: https://quay.io/repository/woozymasta/guassp

app.py

@@ -141,56 +141,90 @@ def sonarqube_connect():
         os.sys.exit(1)
 
 
-def sync_permissions(job_state: dict, alm_key: str = None):
-    """Sync permissions from GitLab to SonarQube
+def sq_search_key_by_alm(sq: object, project: object, alm_key: str = None):
+    """Serch project by ALM key
 
     Args:
-        job_state (dict): job data returned from GitLab by CI_JOB_TOKEN
+        sq (object): SonarQube connection object
+        project (object): GitLab project object generator
         alm_key (str, optional): ALM integration key name. Defaults to None.
     """
-
-    # Get GL project ID
-    gl_project_id = job_state['pipeline']['project_id']
-
     # Set ALM key
     alm_key = alm_key or app.config['SONARQUBE_ALM_KEY']
 
-    # Connect to SQ and GL
-    sq = sonarqube_connect()
-    gl = gitlab_connect()
-
-    # Get GL project by ID
-    try:
-        gl_project = gl.projects.get(gl_project_id)
-    except exceptions.GitlabGetError as e:
-        log.error('Projects id %d: %s', gl_project_id, e)
-        return
-
     # Search project by name in GL ALM projects
     # pylint: disable=unexpected-keyword-arg,no-value-for-parameter
     sq_gl_repos = sq.alm_integrations.search_gitlab_repos(
         almSetting=alm_key,
-        projectName=gl_project.name).get('repositories')
+        projectName=project.name).get('repositories')
+    log.debug('Load %d projects from SonarQube by %s ALM and %s name',
+        len(sq_gl_repos), alm_key, project.name)
 
     # Get SQ project key by GL ID
     sq_project_keys = [
-        i.get('sqProjectKey') for i in sq_gl_repos if i['id'] == gl_project_id
+        i.get('sqProjectKey') for i in sq_gl_repos if i['id'] == project.id
     ]
+    log.debug('Available projects keys: %s', ', '.join(sq_project_keys))
 
     # Check SQ project key
     if len(sq_project_keys) != 1 or sq_project_keys[0] is None:
-        log.error('Projects id %d not found SonarQube', gl_project_id)
+        log.error('Projects id %d not found in SonarQube', project.id)
         return
     else:
-        sq_project_key = sq_project_keys[0]
+        return sq_project_keys[0]
+
 
-    # Get SQ project data
-    sq_projects = list(sq.projects.search_projects(projects=sq_project_key))
-    if len(sq_projects) != 1:
-        log.error('Projects key %s not matched', sq_project_key)
+def sync_permissions(gl_project_id: int, alm_key: str = None):
+    """Sync permissions from GitLab to SonarQube
+
+    Args:
+        gl_project_id (int): GitLab project ID
+        alm_key (str, optional): ALM integration key name. Defaults to None.
+    """
+
+    # Connect to SQ and GL
+    sq = sonarqube_connect()
+    gl = gitlab_connect()
+
+    # Get GL project by ID
+    try:
+        gl_project = gl.projects.get(gl_project_id)
+        log.debug('Success get project %d data from GitLab', gl_project_id)
+    except exceptions.GitlabGetError as e:
+        log.error('Projects id %d: %s', gl_project_id, e)
         return
+
+    # Try get SQ project key by ALM
+    if sq_project_key := sq_search_key_by_alm(sq, gl_project, alm_key):
+        log.debug('Use ALM project with key %s', sq_project_key)
+
+        # Get SQ project data
+        sq_projects = list(sq.projects.search_projects(projects=sq_project_key))
+        if len(sq_projects) != 1:
+            log.error('Projects key %s not matched', sq_project_key)
+            return
+        else:
+            sq_project = sq_projects[0]
+            log.info('Use key %s founded by ALM: %s', sq_project_key, alm_key)
+
+    # Search project in SQ by project name
+    # $CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME
     else:
+        log.debug('Cant find project with ALM %s and try search it', alm_key)
+        query = gl_project.path_with_namespace
+        sq_projects = list(sq.projects.search_projects(q=query))
+
+        if not sq_projects:
+            log.error('Cant find any project by query: %s', query)
+            return
+        if len(sq_projects) > 1:
+            log.error('Found more than one project by query: %s', query)
+            return
+
         sq_project = sq_projects[0]
+        sq_project_key = sq_project['key']
+        log.info('Use key %s founded by query: %s', sq_project_key, query)
+
 
     # Update SQ project visibility as GL project visibility
     if gl_project.visibility != sq_project.get('visibility', 'private'):
@@ -457,7 +491,7 @@ def register_task():
     # Add task to queue
     task = queue.enqueue_call(
         func=sync_permissions,
-        args=(gl_job_state,),
+        args=(prj_id,),
         result_ttl=app.config['QUEUE_RESULT_TTL']
     )
 
@@ -467,6 +501,35 @@ def register_task():
     ), 202, {'Location': url_for('get_task', task_uuid=task.id)}
 
 
+@app.route('/task_manual/<int:prj_id>', methods=['POST'])
+def register_task_manual(prj_id: int):
+    """Register task manual in queue by project ID
+
+    Returns:
+        json: responce
+    """
+    # Check prroject ID
+    if not prj_id or not isinstance(prj_id, int):
+        abort(400, 'Token is invalid or corrupted')
+
+    # Deduplicate tasks
+    for q in queue.jobs:
+        if prj_id == q.args[0]['pipeline']['project_id']:
+            abort(409, f'Task for project {prj_id} already queued {q.id}')
+
+    # Add task to queue
+    task = queue.enqueue_call(
+        func=sync_permissions,
+        args=(prj_id,),
+        result_ttl=app.config['QUEUE_RESULT_TTL']
+    )
+
+    return jsonify(
+        message=f'Manual task added to the queue for project {prj_id}',
+        id=task.id
+    ), 202, {'Location': url_for('get_task', task_uuid=task.id)}
+
+
 if __name__ == '__main__':
     app.run(
         host=app.config['LISTEN_ADDRESS'],