diff --git a/README.md b/README.md
index 887fd06..8449b0c 100644
--- a/README.md
+++ b/README.md
@@ -44,13 +44,16 @@ cvmfs::domain{'example.net'
To use puppet's mount type rather that autofs
a typical configuration might be the following. This
-examples configures a cvmfs domain, a configuration
+example configures a cvmfs domain for common settings, a configuration
repository and finally a particular repository for
-mount.
+mount. In this example the `cvmfs-config.example.org` has
+been marked as the per client configuration repository and will always
+be mounted first.
```puppet
class{'cvmfs':
mount_method => 'mount',
+ config_repo => 'cvmfs-config.example.org',
}
cvmfs::domain{'example.org':
cvmfs_server_url => 'http://web.example.org/cvmfs/@fqrn@'
diff --git a/REFERENCE.md b/REFERENCE.md
index 4ae4339..f7cfcb3 100644
--- a/REFERENCE.md
+++ b/REFERENCE.md
@@ -8,12 +8,12 @@
#### Public Classes
-* [`cvmfs`](#cvmfs): Installs and Configures CvmFS
+* [`cvmfs`](#cvmfs): Installs and Configures CVMFS
#### Private Classes
* `cvmfs::apt`: Configure cvmfs apt repositories
-* `cvmfs::config`: Central configuration of CvmFS
+* `cvmfs::config`: Central configuration of CVMFS
* `cvmfs::fsck`: enable check_fsck as a cron or systemd timer
* `cvmfs::install`: Install cvmfs from a yum repository.
* `cvmfs::service`: Manages the cvmfs services. Opionally this also manages the autofs services
@@ -89,7 +89,7 @@ class{'cvmfs':
}
```
-##### New parameters with CvmFS 2.11.0
+##### New parameters with CVMFS 2.11.0
```puppet
class{'cvmfs':
@@ -109,6 +109,7 @@ class{'cvmfs':
The following parameters are available in the `cvmfs` class:
* [`mount_method`](#-cvmfs--mount_method)
+* [`config_repo`](#-cvmfs--config_repo)
* [`manage_autofs_service`](#-cvmfs--manage_autofs_service)
* [`cvmfs_quota_limit`](#-cvmfs--cvmfs_quota_limit)
* [`cvmfs_quota_ratio`](#-cvmfs--cvmfs_quota_ratio)
@@ -187,6 +188,17 @@ skips all mounting. Note that migrating between for instance *autofs* and then
Default value: `'autofs'`
+##### `config_repo`
+
+Data type: `Optional[Stdlib::Fqdn]`
+
+When using the `mount_method` as `mount` it may be nescessary to specify a CVMFS located configuration_repository.
+This is a repository containing extra cvmfs configuration required to be mounted before any other
+repositories. There is at most one config_repo client. In addition the config_repo must actually be mounted
+explicitly with a `cvmfs::mount{$config_repo:}`, this is **not** automatic.
+
+Default value: `undef`
+
##### `manage_autofs_service`
Data type: `Boolean`
@@ -566,7 +578,7 @@ Default value: `false`
Data type: `Optional[Boolean]`
Install or disable fuse3 variant of cvmfs, if left `undef` no change will be made. Note that changing
-this value when CvmFS mounts are active may well destroy those mounts.
+this value when CVMFS mounts are active may well destroy those mounts.
Not availabe on Ubuntu 18.04.
Default value: `undef`
@@ -766,7 +778,7 @@ The following parameters are available in the `cvmfs::domain` defined type:
Data type: `Stdlib::Fqdn`
-The domain of CvmFS repositories to mount
+The domain of CVMFS repositories to mount
Default value: `$name`
@@ -961,6 +973,27 @@ cvmfs::mount{'foobar.example.org':
}
```
+##### Mount a repository with mount (not automount)
+
+```puppet
+class{ 'cvmfs':
+ mount_method => 'mount',
+}
+cvmfs::mount{'quark.example.org':}
+```
+
+##### Mount a repository with mount and a config_repo as well.
+
+```puppet
+
+class{ 'cvmfs':
+ mount_method => 'mount',
+ config_mount => 'cvmfs-config.example.org',
+}
+cvmfs::mount{'cvmfs-config.example.org':}
+cvmfs::mount{'down.example.org':}
+```
+
#### Parameters
The following parameters are available in the `cvmfs::mount` defined type:
@@ -1091,11 +1124,11 @@ Default value: `undef`
##### `mount_method`
-Data type: `Enum['autofs','mount','none']`
+Data type: `Optional[String[1]]`
-Should the mount attempt be made with autofs or tranditional fstab mount. Do no use this.
+Deprecated, do not set this, set mount_method for the whole client only on the main class.
-Default value: `$cvmfs::mount_method`
+Default value: `undef`
##### `cvmfs_repo_list`
@@ -1203,9 +1236,9 @@ Default value: `undef`
##### `mount_options`
-Data type: `String[1]`
+Data type: `Variant[String[1],Array[String[1]]]`
-Mount options to use for fstab style mounting.
+Mount options to use for fstab style mounting. mount_method==mount only
-Default value: `'defaults,_netdev,nodev'`
+Default value: `['defaults','_netdev','nodev']`
diff --git a/manifests/config.pp b/manifests/config.pp
index 70497d9..3d20d70 100644
--- a/manifests/config.pp
+++ b/manifests/config.pp
@@ -1,4 +1,4 @@
-# @summary Central configuration of CvmFS
+# @summary Central configuration of CVMFS
# @api private
#
class cvmfs::config (
diff --git a/manifests/domain.pp b/manifests/domain.pp
index 59d9dab..492f9d0 100644
--- a/manifests/domain.pp
+++ b/manifests/domain.pp
@@ -4,7 +4,7 @@
# cvmfs_server_url = 'http://example.org/cvmfs/@fqrn@',
# }
#
-# @param domain The domain of CvmFS repositories to mount
+# @param domain The domain of CVMFS repositories to mount
# @param cvmfs_quota_limit Per mount quota limit, not relavent to shared cache. Sets cvmfs_quota_limit
# @param cvmfs_server_url Stratum 1 list, typically `;` delimited. Sets CVMFS_SERVER_URL parameter.
# @param cvmfs_http_proxy List of http proxies to use. Sets CVMFS_PROXY_LIST parameter.
diff --git a/manifests/init.pp b/manifests/init.pp
index bb05d86..627269d 100644
--- a/manifests/init.pp
+++ b/manifests/init.pp
@@ -1,4 +1,4 @@
-# @summary Installs and Configures CvmFS
+# @summary Installs and Configures CVMFS
#
# @see https://cvmfs.readthedocs.io/en/stable/apx-parameters.html CVMFS configuration parameters
#
@@ -38,7 +38,7 @@
# }
# }
#
-# @example New parameters with CvmFS 2.11.0
+# @example New parameters with CVMFS 2.11.0
# class{'cvmfs':
# cvmfs_cache_symlinks => 'yes',
# cvmfs_streaming_cache => 'no',
@@ -54,6 +54,11 @@
# The `autofs` option will configure cvmfs to be mounted with autofs. The `mount` option will
# use puppet's mount type, currently adding a line to /etc/fstab. The *none* option
# skips all mounting. Note that migrating between for instance *autofs* and then *mount* is not supported.
+# @param config_repo
+# When using the `mount_method` as `mount` it may be nescessary to specify a CVMFS located configuration_repository.
+# This is a repository containing extra cvmfs configuration required to be mounted before any other
+# repositories. There is at most one config_repo client. In addition the config_repo must actually be mounted
+# explicitly with a `cvmfs::mount{$config_repo:}`, this is **not** automatic.
# @param manage_autofs_service should the autofs service be maintained.
# @param cvmfs_quota_limit The cvmfs quota size in megabytes.
# @param cvmfs_quota_ratio
@@ -113,7 +118,7 @@
# @param cvmfs_fsck_onreboot Should fsck be run after every reboot
# @param fuse3
# Install or disable fuse3 variant of cvmfs, if left `undef` no change will be made. Note that changing
-# this value when CvmFS mounts are active may well destroy those mounts.
+# this value when CVMFS mounts are active may well destroy those mounts.
# Not availabe on Ubuntu 18.04.
# @param cvmfs_cache_symlinks If set to yes, enables symlink caching in the kernel.
# @param cvmfs_streaming_cache If set to yes, use a download manager to download regular files on read.
@@ -143,6 +148,7 @@
Variant[Undef,String] $cvmfs_http_proxy,
Optional[Variant[Enum['absent'], Array[String[1]]]] $repo_includepkgs,
Enum['autofs','mount','none'] $mount_method = 'autofs',
+ Optional[Stdlib::Fqdn] $config_repo = undef,
Boolean $manage_autofs_service = true,
Integer $default_cvmfs_partsize = 10000,
Variant[Enum['auto'],Integer] $cvmfs_quota_limit = 1000,
diff --git a/manifests/mount.pp b/manifests/mount.pp
index 45eeb01..9a55a95 100644
--- a/manifests/mount.pp
+++ b/manifests/mount.pp
@@ -17,6 +17,21 @@
# }
# }
#
+# @example Mount a repository with mount (not automount)
+# class{ 'cvmfs':
+# mount_method => 'mount',
+# }
+# cvmfs::mount{'quark.example.org':}
+#
+# @example Mount a repository with mount and a config_repo as well.
+#
+# class{ 'cvmfs':
+# mount_method => 'mount',
+# config_mount => 'cvmfs-config.example.org',
+# }
+# cvmfs::mount{'cvmfs-config.example.org':}
+# cvmfs::mount{'down.example.org':}
+#
# @param repo The fully qualified repository name to mount
# @param cvmfs_quota_limit Per mount quota limit, not relavent to shared cache. Sets cvmfs_quota_limit
# @param cvmfs_server_url Stratum 1 list, typically `;` delimited. Sets CVMFS_SERVER_URL parameter.
@@ -29,7 +44,7 @@
# @param cvmfs_max_ttl Maximum effective TTL in seconds for DNS queries of proxy server names. Sets CVMFS_MAX_TTL
# @param cvmfs_env_variables Sets per repo environments variables for magic links.
# @param cvmfs_use_geoapi Set CVMFS_MAX_GEOAPI
-# @param mount_method Should the mount attempt be made with autofs or tranditional fstab mount. Do no use this.
+# @param mount_method Deprecated, do not set this, set mount_method for the whole client only on the main class.
# @param cvmfs_repo_list If true the repository will added to the list of repositories maintained in `/etc/cvmfs/default.local`
# @param cvmfs_mount_rw sets CVMFS_MOUNT_RW
# @param cvmfs_memcache_size Sets CVMFS_MEMCACHE_SIZE in Megabytes.
@@ -43,7 +58,7 @@
# @param cvmfs_external_timeout_direct Sets CVMFS_EXTERNAL_TIMEOUT_DIRECT
# @param cvmfs_external_url Sets CVMFS_EXTERNAL_URL
# @param cvmfs_repository_tag Sets CVMFS_REPOSITORY_TAG
-# @param mount_options Mount options to use for fstab style mounting.
+# @param mount_options Mount options to use for fstab style mounting. mount_method==mount only
#
define cvmfs::mount (
Stdlib::Fqdn $repo = $name,
@@ -65,8 +80,8 @@
Optional[Hash[Variant[Integer,String],Integer, 1]] $cvmfs_uid_map = undef,
Optional[Hash[Variant[Integer,String],Integer, 1]] $cvmfs_gid_map = undef,
Optional[Stdlib::Yes_no] $cvmfs_follow_redirects = undef,
- String[1] $mount_options = 'defaults,_netdev,nodev',
- Enum['autofs','mount','none'] $mount_method = $cvmfs::mount_method,
+ Variant[String[1],Array[String[1]]] $mount_options = ['defaults','_netdev','nodev'],
+ Optional[String[1]] $mount_method = undef,
Optional[String] $cvmfs_external_fallback_proxy = undef,
Optional[String] $cvmfs_external_http_proxy = undef,
Optional[Integer] $cvmfs_external_timeout = undef,
@@ -76,6 +91,19 @@
) {
include cvmfs
+ #
+ # deprecations
+ #
+ if $mount_method {
+ deprecation("mount_method on cvmfs::mount{${repo}:}", 'Never set mount method on a cvmfs::mount. It should only ever be set on the main class for the whole client')
+ }
+ if $mount_options =~ String {
+ deprecation("mount_options on cvmfs::mount{${repo}:}", 'Setting mount_options as a string is deprecated, set as an array of options instead')
+ $_mount_options = $mount_options.split(',')
+ } else {
+ $_mount_options = $mount_options
+ }
+
$_cvmfs_id_map_file_prefix = "/etc/cvmfs/config.d/${repo}"
if $cvmfs_uid_map {
cvmfs::id_map { "${_cvmfs_id_map_file_prefix}.uid_map":
@@ -130,18 +158,29 @@
content => "${repo},",
}
}
- if $mount_method == 'mount' {
+ if $cvmfs::mount_method == 'mount' {
file { "/cvmfs/${repo}":
ensure => directory,
owner => 'cvmfs',
group => 'cvmfs',
require => Package['cvmfs'],
}
+
+ #
+ # Require the config repo for all repos except the config_repo
+ #
+ if $cvmfs::config_repo and $cvmfs::config_repo != $repo {
+ $_my_mount_options = $_mount_options + ["x-systemd.requires-mounts-for=/cvmfs/${cvmfs::config_repo}"]
+ Mount["/cvmfs/${cvmfs::config_repo}"] -> Mount["/cvmfs/${repo}"]
+ } else {
+ $_my_mount_options = $_mount_options
+ }
+
mount { "/cvmfs/${repo}":
ensure => mounted,
device => $repo,
fstype => 'cvmfs',
- options => $mount_options,
+ options => $_my_mount_options.unique.join(','),
atboot => true,
require => [File["/cvmfs/${repo}"],File["/etc/cvmfs/config.d/${repo}.local"],Concat['/etc/cvmfs/default.local'],File['/etc/fuse.conf']],
}
diff --git a/spec/defines/mount_spec.rb b/spec/defines/mount_spec.rb
index 59ef257..065e0eb 100644
--- a/spec/defines/mount_spec.rb
+++ b/spec/defines/mount_spec.rb
@@ -3,13 +3,14 @@
require 'spec_helper'
describe 'cvmfs::mount' do
- let(:pre_condition) do
- 'class{"cvmfs": cvmfs_http_proxy => undef}'
- end
let(:title) { 'files.example.org' }
on_supported_os.each do |os, facts|
context "on #{os}" do
+ let(:pre_condition) do
+ ['class{"cvmfs": cvmfs_http_proxy => undef}']
+ end
+
let(:facts) do
facts
end
@@ -22,19 +23,25 @@
it { is_expected.to compile.with_all_deps }
it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local') }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with_content("# cvmfs files.example.org.local file installed with puppet.\n# this files overrides and extends the values contained\n# within the files.example.org.conf file.\n\n") }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').without_content(%r{.*CVMFS_MEMCACHE_SIZE.*$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').without_content(%r{.*CVMFS_USE_GEOAPI.*$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').without_content(%r{.*CVMFS_FOLLOW_REDIRECTS.*$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').without_content(%r{.*CVMFS_CLAIM_OWNERSHIP.*$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').without_content(%r{.*CVMFS_REPOSITORY_TAG.*$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').without('content' => %r{^CVMFS_HTTP_PROXY.*$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').without('content' => %r{^CVMFS_QUOTA_LIMIT.*$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').without('content' => %r{^CVMFS_EXTERNAL_FALLBACK_PROXY=.*$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').without('content' => %r{^CVMFS_EXTERNAL_HTTP_PROXY=.*$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').without('content' => %r{^CVMFS_EXTERNAL_TIMEOUT=.*$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').without('content' => %r{^CVMFS_EXTERNAL_TIMEOUT_DIRECT=.*$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').without('content' => %r{^CVMFS_EXTERNAL_URL=.*$}) }
+
+ it {
+ is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').
+ without_content(%r{.*CVMFS_MEMCACHE_SIZE.*$}).
+ without_content(%r{.*CVMFS_USE_GEOAPI.*$}).
+ without_content(%r{.*CVMFS_FOLLOW_REDIRECTS.*$}).
+ without_content(%r{.*CVMFS_CLAIM_OWNERSHIP.*$}).
+ without_content(%r{.*CVMFS_REPOSITORY_TAG.*$}).
+ without_content(%r{^CVMFS_HTTP_PROXY.*$}).
+ without_content(%r{^CVMFS_QUOTA_LIMIT.*$}).
+ without_content(%r{^CVMFS_EXTERNAL_FALLBACK_PROXY=.*$}).
+ without_content(%r{^CVMFS_EXTERNAL_HTTP_PROXY=.*$}).
+ without_content(%r{^CVMFS_EXTERNAL_TIMEOUT=.*$}).
+ without_content(%r{^CVMFS_EXTERNAL_TIMEOUT_DIRECT=.*$}).
+ without_content(%r{^CVMFS_EXTERNAL_URL=.*$}).
+ with_content("# cvmfs files.example.org.local file installed with puppet.\n# this files overrides and extends the values contained\n# within the files.example.org.conf file.\n\n")
+ }
+
+ it { is_expected.not_to contain_mount('/cvmfs/files.example.org') }
context 'with lots of parameters set' do
let(:params) do
@@ -56,24 +63,77 @@
}
end
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_MEMCACHE_SIZE=2000$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_USE_GEOAPI='yes'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_FOLLOW_REDIRECTS='yes'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_CLAIM_OWNERSHIP='yes'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_REPOSITORY_TAG='testing'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_UID_MAP='/etc/cvmfs/config.d/files.example.org.uid_map'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_GID_MAP='/etc/cvmfs/config.d/files.example.org.gid_map'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_QUOTA_LIMIT='54321'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_KEYS_DIR='/etc/cvmfs/keys/example.org'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_EXTERNAL_FALLBACK_PROXY='http://external-fallback.example.org:3128'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_EXTERNAL_HTTP_PROXY='http://http-proxy.example.org:2138'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_EXTERNAL_TIMEOUT='100'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_EXTERNAL_TIMEOUT_DIRECT='450'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with('content' => %r{^CVMFS_EXTERNAL_URL='http://external-url.example.org:80'$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.uid_map').with('content' => %r{^123 12$}) }
- it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.gid_map').with('content' => %r{^137 42$}) }
+ it {
+ is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.local').with_content(%r{^CVMFS_MEMCACHE_SIZE=2000$}).
+ with_content(%r{^CVMFS_USE_GEOAPI='yes'$}).
+ with_content(%r{^CVMFS_FOLLOW_REDIRECTS='yes'$}).
+ with_content(%r{^CVMFS_CLAIM_OWNERSHIP='yes'$}).
+ with_content(%r{^CVMFS_REPOSITORY_TAG='testing'$}).
+ with_content(%r{^CVMFS_UID_MAP='/etc/cvmfs/config.d/files.example.org.uid_map'$}).
+ with_content(%r{^CVMFS_GID_MAP='/etc/cvmfs/config.d/files.example.org.gid_map'$}).
+ with_content(%r{^CVMFS_QUOTA_LIMIT='54321'$}).
+ with_content(%r{^CVMFS_KEYS_DIR='/etc/cvmfs/keys/example.org'$}).
+ with_content(%r{^CVMFS_EXTERNAL_FALLBACK_PROXY='http://external-fallback.example.org:3128'$}).
+ with_content(%r{^CVMFS_EXTERNAL_HTTP_PROXY='http://http-proxy.example.org:2138'$}).
+ with_content(%r{^CVMFS_EXTERNAL_TIMEOUT='100'$}).
+ with_content(%r{^CVMFS_EXTERNAL_TIMEOUT_DIRECT='450'$}).
+ with_content(%r{^CVMFS_EXTERNAL_URL='http://external-url.example.org:80'$})
+ }
+
+ it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.uid_map').with_content(%r{^123 12$}) }
+ it { is_expected.to contain_file('/etc/cvmfs/config.d/files.example.org.gid_map').with_content(%r{^137 42$}) }
+ end
+ end
+
+ context 'with mount_method mount set on main class' do
+ let(:pre_condition) do
+ 'class{"cvmfs": cvmfs_http_proxy => undef, mount_method => "mount"}'
+ end
+
+ it { is_expected.to compile.with_all_deps }
+
+ it {
+ is_expected.to contain_mount('/cvmfs/files.example.org').with(
+ options: 'defaults,_netdev,nodev',
+ device: 'files.example.org'
+ )
+ }
+
+ context 'with mount_options set to an array' do
+ let(:params) do
+ {
+ mount_options: %w[one two three],
+ }
+ end
+
+ it { is_expected.to contain_mount('/cvmfs/files.example.org').with_options('one,two,three') }
end
end
+
+ context 'with mount_method mount and a config_repo set on main class' do
+ let(:pre_condition) do
+ [
+ 'class{"cvmfs": cvmfs_http_proxy => undef, mount_method => "mount", config_repo => "cvmfs-config.example.org"}',
+ 'cvmfs::mount{"cvmfs-config.example.org":}',
+ ]
+ end
+
+ it { is_expected.to compile.with_all_deps }
+
+ it {
+ is_expected.to contain_mount('/cvmfs/files.example.org').with(
+ options: 'defaults,_netdev,nodev,x-systemd.requires-mounts-for=/cvmfs/cvmfs-config.example.org',
+ device: 'files.example.org'
+ )
+ }
+
+ it {
+ is_expected.to contain_mount('/cvmfs/cvmfs-config.example.org').with(
+ options: 'defaults,_netdev,nodev',
+ device: 'cvmfs-config.example.org'
+ )
+ }
+ end
end
end
end