Add configure SS and pipeline #162

mamedin · 2018-01-17T13:52:14Z

This PR will use the features added by the following PRs:

-artefactual/archivematica#681
-artefactual/archivematica-storage-service#213

Specifically, this PR allows to:

a) Create a SS superuser
b) Create a dashboard admin user and register the pipeline on the SS.

To configure SS and pipeline on a deployment, the following boolean
extra-vars have to be enabled:

archivematica_src_configure_dashboard : To create the dashboard admin user and
to register pipeline on SS
archivematica_src_configure_ss : To create SS superuser

They are disabled by default.

connects to #161

mamedin · 2018-01-17T13:52:57Z

This fixes #161

mamedin · 2018-01-17T14:04:08Z

You can use the following variables on vars.yml file for testing:

archivematica_src_configure_ss_user: "ss_user"
archivematica_src_configure_ss_password: "password_ss"
archivematica_src_configure_ss_email: "[email protected]"
#archivematica_src_configure_ss_api_key: "9fd9a15258c244b2ca4bbee0a2393809"
archivematica_src_configure_ss_url: "http://SS_IP_ADDRESS:8000"
archivematica_src_configure_am_user: "dashboard_user"
archivematica_src_configure_am_password: "password_dashboard"
archivematica_src_configure_am_email: "[email protected]"
#archivematica_src_configure_am_api_key: "058d6ca988fecd5a6bf4ab24492e7ee8c954f485"
archivematica_src_configure_am_org_name: "org_name"
archivematica_src_configure_am_org_id: "12345"
#archivematica_src_configure_am_whitelist: '"1.2.3.3 127.0.0.1"'

To run the playbook with the configure feature, use the extravars:

"archivematica_src_configure_ss_dashboard=yes": To configure SS and dashboard
"archivematica_src_configure_ss=yes": To create SS superadmin only
"archivematica_src_configure_dashboard=yes": To create the dashboard andmin and register the pipeline.

mamedin · 2018-01-17T17:08:08Z

I forgot to mention the more important thing. There's a breaking change in the PR artefactual/archivematica#681 about the dashboard API whitelist: an empty whitelist now means: "allow from all".

Do you think that a better default whitelist could be "localhost" or another value?

scollazo

Some feedback provided

scollazo · 2018-01-19T11:56:56Z

defaults/main.yml

+archivematica_src_configure_am_password: "test"              # Dashboard admin password
+archivematica_src_configure_am_email: "[email protected]"     # Dashboard admin password
+archivematica_src_configure_am_org_name: "test"              # Dashboard admin Org name
+archivematica_src_configure_am_org_id: "test"                # Dashboard admin Org id


Can you add also the "First name" and "last name" fields? This way, we have all the configuration options exposed here.

scollazo · 2018-01-19T11:57:44Z

tasks/pipeline-configure.yml

+
+- name: "Create Dashboard user and register pipeline on SS"
+  django_manage:
+    command: "install --username={{ archivematica_src_configure_am_user }} --password={{ archivematica_src_configure_am_password }} --email={{ archivematica_src_configure_am_email }} --org-name={{ archivematica_src_configure_am_org_name }} --org-id={{ archivematica_src_configure_am_org_id }} --api-key={{ archivematica_src_configure_am_api_key }} --ss-url={{ archivematica_src_configure_ss_url }} --ss-user={{ archivematica_src_configure_ss_user }} --ss-api-key={{ archivematica_src_configure_ss_api_key }} --whitelist={{archivematica_src_configure_am_whitelist}}"


This should be one line for each switch, it would improve readability

scollazo · 2018-01-19T11:59:32Z

tasks/ss-configure.yml

+- set_fact: archivematica_src_configure_ss_api_key={{ 999999999999999999999 | random | to_uuid | hash('md5') }}
+  when: archivematica_src_configure_ss_api_key is undefined
+
+- name: "Create SS superuser"


What do you think about deleting the default test user when this is set?

scollazo · 2018-01-19T12:01:01Z

tasks/pipeline-configure.yml

+- set_fact: archivematica_src_configure_am_api_key={{ 999999999999999999998 | random | to_uuid | hash('md5') }}
+  when: archivematica_src_configure_am_api_key is undefined
+
+- name: "Create Dashboard user and register pipeline on SS"


You have archivematica_src_configure_dashboard and archivematica_src_configure_ss_dashboard: "no", but seems like this configures both things always.

scollazo · 2018-01-19T12:02:31Z

defaults/main.yml

+# Configure AM
+# archivematica_src_configure_am_api_key & archivematica_src_configure_ss_api_key can be defined as vars. If not defined they will be autogenerated by the playbook
+archivematica_src_configure_ss_user: "test"                  # SS superuser
+archivematica_src_configure_ss_password: "test"              # SS superuser password


I think passwords should be something line "PleaseChangeMe!", so users don't forget to set them

scollazo · 2018-01-19T12:04:08Z

tasks/main.yml

+# Configure SS
+#
+
+- include: "ss-configure.yml"


As the files have only two tasks each, what about consolidating them into a single one?

mamedin · 2018-01-29T18:03:54Z

Thanks Santi, I have considered all your suggestions except for entering the variables "First name" and "Last name"

The variables "First name" and "Last name" were not contemplated in the artefactual/archivematica#681 PR, so it requires an AM additional change.

I have simplified the variables and tags since it could lead to confusion and it made the configuration more complicated. Now there is only the amsrc-configure tag and the variables ``archivematica_src_configure_ss and archivematica_src_configure_dashboard.

The default SS test user is deleted when adding a custom SS superuser.

I have consolidated all task into a single file, called configure.yml.

The default password was set to "PleaseChangeMe!"

The format is now cleaner introducing some line breaks to facilitate the reading of the code.

This PR is ready for CR again.

scollazo

Some more comments

scollazo · 2018-01-30T16:32:40Z

tasks/configure.yml

+  become: "yes"
+  shell: |
+    cd {{ archviematica_src_ss_app }}
+    export SS_DB_NAME={{ archivematica_src_ss_environment['SS_DB_NAME'] }}


I think you can set the env in a line, as we do here:

ansible-archivematica-src/tasks/ss-db.yml

Line 17 in a38b2d1

environment: "{{ archivematica_src_ss_environment }}"

scollazo · 2018-01-30T16:32:56Z

tasks/configure.yml

+    executable: /bin/bash
+  when: "archivematica_src_configure_ss|bool"
+
+#source {{ archivematica_src_ss_virtualenv }}/bin/activate;


Is this dead code?

scollazo · 2018-01-30T16:40:10Z

I'm thinking about adding a "security" measure, to avoid breaking currently configured systems:

Check if there is an "admin" user created
If so, trigger an ansible prompt and ask if we want to continue.

sevein · 2018-03-25T15:33:25Z

@mamedin 🏓

sevein

Let's try to get this merged? It's a nice feature.

mamedin · 2018-03-26T07:50:21Z

Hi @sevein to merge:

-I need to change the PR to take account the last @scollazo's suggestions:
* Remove dead code
* Set one-line environment

Import these Jisc commits to Archivematica and Storage Service (to make idempotent):
* dashboard: make create_super_user idempotent in installer steps JiscSD/archivematica#46
* dashboard: make setup_pipeline funcs idempotent JiscSD/archivematica#55
* https://github.com/JiscRDSS/archivematica-storage-service/pull/19/files

sevein · 2018-03-26T14:47:27Z

@mamedin ops ok I see. Any of that is going to happen for AM17 so I guess we can ignore this PR for now. Thanks!

scollazo · 2018-05-03T23:08:59Z

Although not idempotent, the changes in this PR work on am 1.7. @mamedin can you address my comments about setting environment vars so we can merge this?

We should also create a bug in archivematica in order to get the changes needed for idempotency into 1.7.1, or at least in qa/1.x

mamedin · 2018-05-21T09:01:28Z

@scollazo It is ready for QA, I have included your suggestions about stting environment.

scollazo

Feedback added. What happens if you run this playbook twice? Does the password get reset?

scollazo · 2018-05-22T18:13:05Z

defaults/main.yml

+archivematica_src_configure_dashboard: "no"
+archivematica_src_configure_ss: "no"
+
+#Components to configure


Duplicated?

scollazo · 2018-05-22T18:13:43Z

defaults/main.yml

+archivematica_src_configure_am_user: "test"                  # Dashboard admin
+archivematica_src_configure_am_password: "PleaseChangeMe!"   # Dashboard admin password
+archivematica_src_configure_am_email: "[email protected]"     # Dashboard admin password
+archivematica_src_configure_am_org_name: "test"              # Dashboard admin Org name


Use "test org" for org, and "test id" for id?

scollazo · 2018-05-22T18:17:19Z

tasks/configure.yml

+
+# Create api-key if not defined
+- set_fact: archivematica_src_configure_am_api_key={{ 999999999999999999998 | random | to_uuid | hash('md5') }}
+  when: archivematica_src_configure_am_api_key is undefined


For the ss api key, the condition is when: "archivematica_src_configure_ss_api_key is undefined and archivematica_src_configure_ss|bool", but here you are not checking for archivematica_src_configure_dashboard

Yes, it was done intentionally in case just SS is installed.

I'm not getting your point, I think that the other case (deploying the dashboard, with the ss in other server), will also fail.

Thinking out loud , does it make sense to place this steps in pipeline-dbconf.yml for the dashboard, and ss-db.yml for the storage service? This way you can rely on archivematica_src_install_am and archivematica_src_install_ss to avoid this issue.

scollazo · 2018-05-22T19:31:35Z

tasks/configure.yml

+- name: "Delete default SS superuser"
+  become: "yes"
+  shell: |
+    cd {{ archviematica_src_ss_app }}


I think that using cd, and chdir later, is not needed

Yes, this "cd" is not needed. The typo is prior to this change, and this variable is used several times in the role. I'll include the typo fix as a separate commit.

scollazo · 2018-05-22T19:34:13Z

tasks/configure.yml

+
+# Create api-key if not defined
+- set_fact: archivematica_src_configure_am_api_key={{ 999999999999999999998 | random | to_uuid | hash('md5') }}
+  when: archivematica_src_configure_am_api_key is undefined


I'm not getting your point, I think that the other case (deploying the dashboard, with the ss in other server), will also fail.

Thinking out loud , does it make sense to place this steps in pipeline-dbconf.yml for the dashboard, and ss-db.yml for the storage service? This way you can rely on archivematica_src_install_am and archivematica_src_install_ss to avoid this issue.

mamedin · 2018-06-13T15:49:08Z

Pending to the artefactual/archivematica-storage-service#364 (comment)

I found some problems that I want to fix:

Deleting the SS test user could be a problem when running this role several times
It fails using the amsrc-configure tag when the archivematica_src_mcpclient_clamav_tcp_port variable is not defined

sevein · 2018-06-13T20:00:01Z

Related: artefactual/archivematica#1131.

sevein · 2018-06-13T20:34:13Z

Related: artefactual/archivematica#1134.

mamedin · 2018-06-26T08:41:58Z

I have updated again this PR. I added 2 basic test looking at the dashboard database to check whether the pipeline is already registered or the dashboard user already exists. With these 2 basic tests we make sure that this role only runs in new deployments, which is where it makes sense.

I have rebased again to check the last commits and now there are no problems when using the amsrc-configure tag and some variables are not defined.

It is ready for CR again.

There are new changes that needs to be cr'ed again

scollazo · 2018-06-27T13:58:58Z

Instead of aborting if the user is already created, why not check for it first, and only create it when needed?

mamedin · 2018-06-27T18:00:37Z

Hi @scollazo , the artefactual/archivematica#1165 PR will make the dashboard installer idempotent, so the last commit could be removed because it is not needed.

jhsimpson · 2018-07-24T05:50:44Z

I made a copy of this branch and rebased my copy on top of the recent changes in the master branch of this repo: https://github.com/jhsimpson/ansible-archivematica-src/tree/dev/configure-ss-dashboard

I tested this deploying to an Ubunt 16.04 host, using this dev branch of deploy-pub:

https://github.com/artefactual/deploy-pub/tree/dev/deploy-am-post17

This worked great for me, on a fresh install. I haven't tested upgrades or other os'es. I did have to play around a bit to figure out what ansible variables I actually needed. You can see the variables I used here:https://github.com/artefactual/deploy-pub/blob/dev/deploy-am-post17/playbooks/archivematica/vars-singlenode-qa.yml

@mamedin I think that last commit could stay, it is not bad to check if the users exist, even if the dashboard installer itself is idempotent, it is possible for users to be created by other methods (in db directly, by django shell commands).

mamedin · 2018-08-02T16:26:20Z

In this last commit I have changed a bit the order of the tasks and checks. Now instead of abort when it detects the pipeline was configured or the pipeline user exists, it does nothing and sends a message.

This PR will use the features added by the following PRs: -artefactual/archivematica#681 -artefactual/archivematica-storage-service#213 Specifically, this PR allows to: a) Create a SS superuser b) Create a dashboard admin user and register the pipeline on the SS. To configure SS and pipeline on a deployment, the following boolean extra-vars have to be enabled: - archivematica_src_configure_dashboard: To create the dashboard admin user and to register pipeline on SS - archivematica_src_configure_ss: To create SS superuser They are disabled by default.

This commit adds two test to check whether the dashboard user `archivematica_src_configure_am_user` exists and the pipeline is already registered before running the configure tasks. These new tasks will abort the playbook when any of these basic test fails.

scollazo · 2018-08-08T09:34:16Z

It has been a long way, but imho, this is ready for merge, kudos @mamedin !

We should add those new values to the docs in a separate commit.

scollazo

LGTM!

mamedin self-assigned this Jan 17, 2018

mamedin requested review from sevein, scollazo and hakamine January 17, 2018 13:52

mamedin force-pushed the dev/configure-ss-dashboard branch from 69530b3 to 8a0541e Compare January 17, 2018 13:54

scollazo requested changes Jan 19, 2018

View reviewed changes

mamedin force-pushed the dev/configure-ss-dashboard branch 3 times, most recently from 3a76a3c to 5b64f22 Compare January 30, 2018 13:11

scollazo requested changes Jan 30, 2018

View reviewed changes

sevein reviewed Mar 25, 2018

View reviewed changes

sevein added the Status: in progress Issue that is currently being worked on. Waffle label. label Apr 3, 2018

mamedin changed the base branch from qa/1.7.x to stable/1.7.x May 9, 2018 08:03

mamedin force-pushed the dev/configure-ss-dashboard branch 2 times, most recently from 4f6ed81 to 5928a31 Compare May 11, 2018 16:25

scollazo requested changes May 22, 2018

View reviewed changes

mamedin force-pushed the dev/configure-ss-dashboard branch 2 times, most recently from 9753a88 to 37d5e5c Compare May 23, 2018 10:13

scollazo previously approved these changes May 30, 2018

View reviewed changes

sevein mentioned this pull request Jun 8, 2018

Delete create_testuser signal handler artefactual/archivematica-storage-service#364

Closed

sevein mentioned this pull request Jun 13, 2018

Problem: install's command fails when the superuser exists artefactual/archivematica#1131

Closed

mamedin force-pushed the dev/configure-ss-dashboard branch 2 times, most recently from 74458ee to c46e0e8 Compare June 26, 2018 09:02

jhsimpson added this to the 1.8.0 milestone Jul 24, 2018

jhsimpson added the IISH label Jul 24, 2018

mamedin force-pushed the dev/configure-ss-dashboard branch 2 times, most recently from aae7f21 to 8b6323a Compare August 2, 2018 20:46

mamedin added 4 commits August 8, 2018 10:40

configure: remove abort on fails

005575f

configure: add environment to register pipeline

82d2e7f

mamedin force-pushed the dev/configure-ss-dashboard branch from 8b6323a to 82d2e7f Compare August 8, 2018 08:40

scollazo approved these changes Aug 8, 2018

View reviewed changes

mamedin merged commit 5698070 into stable/1.7.x Aug 8, 2018

helenst removed the Status: in progress Issue that is currently being worked on. Waffle label. label Aug 8, 2018

mamedin deleted the dev/configure-ss-dashboard branch August 8, 2018 09:41

mamedin mentioned this pull request Aug 8, 2018

Problem: Cannot configure SS and dashboard #161

Closed

mamedin mentioned this pull request Aug 24, 2018

playbooks: add am-configure to AM deploys artefactual/deploy-pub#88

Merged

Add configure SS and pipeline #162

Add configure SS and pipeline #162

Conversation

mamedin commented Jan 17, 2018 • edited by jhsimpson Loading

mamedin commented Jan 17, 2018

mamedin commented Jan 17, 2018

mamedin commented Jan 17, 2018 • edited Loading

scollazo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mamedin commented Jan 29, 2018

scollazo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scollazo commented Jan 30, 2018

sevein commented Mar 25, 2018

sevein left a comment

Choose a reason for hiding this comment

mamedin commented Mar 26, 2018 • edited Loading

sevein commented Mar 26, 2018

scollazo commented May 3, 2018

mamedin commented May 21, 2018

scollazo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mamedin commented Jun 13, 2018

sevein commented Jun 13, 2018

sevein commented Jun 13, 2018

mamedin commented Jun 26, 2018

scollazo commented Jun 27, 2018

mamedin commented Jun 27, 2018

jhsimpson commented Jul 24, 2018 • edited Loading

mamedin commented Aug 2, 2018 • edited Loading

scollazo commented Aug 8, 2018

scollazo left a comment

Choose a reason for hiding this comment

mamedin commented Jan 17, 2018 •

edited by jhsimpson

Loading

mamedin commented Jan 17, 2018 •

edited

Loading

mamedin commented Mar 26, 2018 •

edited

Loading

jhsimpson commented Jul 24, 2018 •

edited

Loading

mamedin commented Aug 2, 2018 •

edited

Loading