Add better logging

[cormier]

Implement better logging

Introduction

• Following on the discussion in ticket #3177 and @spiette's logging recommendations, this ticket aims at implementing an intelligent logging system in SFLvault that will eventually serve as a basis for auditing.

In general

• This pull request proposes adding a log_command function in vault.py. This log_command function may take any argument and will output it to JSON.
• The output is JSON, meaning that it can be easily picked up by a log indexing solution such as logstash.

What exactly is logged?

All commands that have an effect on the database are logged. More precisely, here are the logged commands with a sample log message:

Command	Message
customer-add	{"username": "admin", "command": "customer_add", "user_id": 1, "arguments": {"customer_id": 8, "customer_name": "new customer"}, "success": true}
customer-del	{"username": "admin", "command": "customer_del", "user_id": 1, "arguments": {"customer_id": 8}, "success": true}
customer-edit	{"username": "admin", "command": "customer_put", "user_id": 1, "arguments": {"customer_id": 7, "data": {"name": "Test name"}}, "success": true}
group-add	{"username": "admin", "user_id": 1, "success": true, "command": "customer_put", "arguments": {"customer_id": 1, "data": {"name": "sdfdsfsd"}}}
group-add-service	{"username": "admin", "command": "group_add_service", "user_id": 1, "arguments": {"service_id": 1, "group_id": 2}, "success": true}
group-del-service	{"username": "admin", "command": "group_del_service", "user_id": 1, "arguments": {"service_id": 1, "group_id": 2}, "success": true}
group-add-user	{"username": "admin", "command": "group_add_user", "user_id": 1, "arguments": {"group_id": 2, "user_id": 2}, "success": true}
group-del-user	{"username": "admin", "command": "group_del_user", "user_id": 1, "arguments": {"group_id": 2, "user_id": 2}, "success": true}
group-edit	{"username": "admin", "command": "group_put", "user_id": 1, "arguments": {"group_id": 2, "data": {"name": "New group name"}}, "success": true}
machine-add	{"username": "admin", "command": "machine_add", "user_id": 1, "arguments": {"machine_id": 21}, "success": true}
machine-del	{"username": "admin", "command": "machine_del", "user_id": 1, "arguments": {"machine_id": 3}, "success": true}
machine-edit	{"username": "admin", "command": "machine_put", "user_id": 1, "arguments": {"machine_id": 21, "data": {"name": "test name", "ip": "test ip", "notes": "test notes", "fqdn": "test fqdn", "location": "test location", "customer_id": "2"}}, "success": true}
service-add	{"username": "admin", "command": "service_add", "user_id": 1, "arguments": {"service_id": 2}, "success": true}
service-del	{"username": "admin", "command": "service_del", "user_id": 1, "arguments": {"service_id": 2}, "success": true}
service-edit	{"username": "admin", "command": "service_put", "user_id": 1, "arguments": {"service_id": 2, "data": {"machine_id": "2", "parent_service_id": "1", "metadata": {"a": "2", "b": "3"}}}, "success": true}
service-passwd	{"username": "admin", "command": "service_passwd", "user_id": 1, "arguments": {"service_id": 2}, "success": true}
user-add	{"username": "admin", "command": "user_add", "user_id": 1, "arguments": {"username": "david", "new_user_id": 4, "is_admin": false}, "success": true}
user-del	{"username": "admin", "command": "user_del", "user_id": 1, "arguments": {"username": "david"}, "success": true}
user-setup	{"username": "admin", "command": "user_setup", "user_id": 1, "arguments": {"username": "david"}, "success": true}

Please see [1]
The following immutable commands are of crucial importance for anyone willing to use the logs to perform an audit and are also logged:

Command	Message
show and connect	{"username": "admin", "command": "show", "user_id": 1, "arguments": {"service_ids": [1, 2]}, "success": true}

You will notice that show now returns two lists of service_ids. Here is their descriptions:

Option	Description
all_service_ids	All the service ids returned by show, regardless of the ability of the possibility of the users to see them.
accessible_service_ids	The services that the user should be able to see given the group he's in.

The reason we make this distintion is because SFLvault sends the secret to the user regardless of the groups he's in. Maybe this shouldn't be happening, and the server should only send secrets to "legitimate" users. I think we should eventually fix this, but this should be part of another PR.

Notes

1. The log message for user-setup should read { "username": "None", "user_id":"None" }. This is a separate issue. (See The myself_id and myself_username variables should not be set directly on the vault #14)

[ghost]

Normally, a change from "human-redable" logging to JSON logging would have many implications, but because we never logged anything that mattered, the implications of this PR are small. So yay or logging.

I would have put the with_groups commit in a separate PR though.