Add a "retries" attribute to retry a failed perlbrew_perl provider (if that's what is in fact failing)

[mrballcb]

I'm requesting adding to the perlbrew_perl provider:

retries node['perlbrew']['install_retries']

The default value for this attribute can be zero so that it operates exactly as it does now, however it would be advantageous to be allowed to retry this function upon failure. Here's why

I built an AWS stack using a CloudFormation template with 10 hosts that were all building perlbrew 5.16.3 on the same CentOS 6.7 ami that we built from a public CentOS 6.7 ami. Chef finished to completion on 6 of the hosts with no issues. On the other 4, perlbrew stopped in exactly the same way. Here is the chef log:

[2016-02-19T18:38:55+00:00] INFO: Processing yum_package[perl] action install (perlbrew::default line 23)
[2016-02-19T18:38:56+00:00] INFO: Processing yum_package[curl] action install (perlbrew::default line 23)
[2016-02-19T18:38:56+00:00] INFO: Processing directory[/opt/perlbrew] action create (perlbrew::default line 31)
[2016-02-19T18:38:56+00:00] INFO: directory[/opt/perlbrew] created directory /opt/perlbrew
[2016-02-19T18:38:56+00:00] INFO: Processing bash[perlbrew self-upgrade] action run (perlbrew::default line 35)
[2016-02-19T18:38:56+00:00] INFO: Processing bash[perlbrew-install] action run (perlbrew::default line 46)
[2016-02-19T18:38:58+00:00] INFO: bash[perlbrew-install] ran successfully
[2016-02-19T18:38:58+00:00] INFO: Processing perlbrew_perl[perl-5.16.3] action install (perlbrew::default line 60)
[2016-02-19T18:38:58+00:00] INFO: Processing execute[Install perlbrew perl perl-5.16.3] action run (/var/chef/cache/cookbooks/perlbrew/prov
iders/perl.rb line 29)
================================================================================
Error executing action `run` on resource 'execute[Install perlbrew perl perl-5.16.3]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '25'
---- Begin output of /opt/perlbrew/bin/perlbrew install perl-5.16.3 --as perl-5.16.3  ----
STDOUT: Fetching perl 5.16.3 as /opt/perlbrew/dists/perl-5.16.3.tar.bz2
Download http://www.cpan.org/src/5.0/perl-5.16.3.tar.bz2 to /opt/perlbrew/dists/perl-5.16.3.tar.bz2
Installing /opt/perlbrew/build/perl-5.16.3 into /opt/perlbrew/perls/perl-5.16.3

This could take a while. You can run the following command on another shell to track the status:

  tail -f /opt/perlbrew/build.perl-5.16.3.log
STDERR: Use of uninitialized value $_[0] in join or string at /opt/perlbrew/bin/perlbrew line 115.
Installation process failed. To spot any issues, check

  /opt/perlbrew/build.perl-5.16.3.log

If some perl tests failed and you still want to install this distribution anyway,
do:

  (cd /opt/perlbrew/build/perl-5.16.3; make install)

You might also want to try upgrading patchperl before trying again:

  perlbrew install-patchperl

Generally, if you need to install a perl distribution known to have minor test
failures, do one of these command to avoid seeing this message

  perlbrew --notest install perl-5.16.3
  perlbrew --force install perl-5.16.3
---- End output of /opt/perlbrew/bin/perlbrew install perl-5.16.3 --as perl-5.16.3  ----
Ran /opt/perlbrew/bin/perlbrew install perl-5.16.3 --as perl-5.16.3  returned 25

Looking in the build log, it complains about a failure in the test harness:

#   Failed test '$err != 0 or @res == 0 for host=TbK4jM2M0OS.lm57DWIyu4i/service=ftp/socktype=SOCK_STREAM'
#   at t/getaddrinfo.t line 92.
# family=2 socktype=1 protocol=6 addr=[16 bytes]
#   addr=0x02, 0x00, 0x00, 0x15, 0x36, 0x99, 0x4a, 0x19, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
# family=2 socktype=1 protocol=6 addr=[16 bytes]
#   addr=0x02, 0x00, 0x00, 0x15, 0x36, 0x99, 0x19, 0xb4, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00
# Looks like you failed 1 test of 30.
../cpan/Socket/t/getaddrinfo.t ....................................
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/30 subtests

<snip>

Test Summary Report
-------------------
../cpan/CGI/t/tmpdir.t                                          (Wstat: 0 Tests: 9 Failed: 0)
  TODO passed:   3, 6, 8
../cpan/Socket/t/getaddrinfo.t                                  (Wstat: 256 Tests: 30 Failed: 1)
  Failed test:  24
  Non-zero exit status: 1
Files=2317, Tests=522057, 991 wallclock secs (78.55 usr  8.99 sys + 587.74 cusr 54.68 csys = 729.96 CPU)
Result: FAIL
make: *** [test_harness] Error 1
##### Brew Failed #####

I reran chef-client on these nodes, they succeeded, and things seem to be normal.

At this point, I want to ask if it's possible that the perlbrew install actually failed? If yes, that would possibly explain the perlbrew_perl failure.

My configuration: All 10 of these hosts use the same cookbook recipe to install perlbrew. In my cookbook "myapp", there are some global attributes configured:

attributes/default.rb
23:default['perlbrew']['default_version'] = "perl-5.16.3"
24:default['perlbrew']['configure_users'] = {
30:# These control the perlbrew cookbook recipes
31:default['perlbrew']['perlbrew_root'] = "/opt/perlbrew"
32:default['perlbrew']['perls'] = [ node['perlbrew']['default_version'] ]
33:default['perlbrew']['self_upgrade'] = false

The "myapp::common" recipe is loaded in recipes used on all of the hosts:

include_recipe 'myapp::common'

and in myapp/recipes/common.rb is a boilerplate perlbrew include:

# Using perlbrew cookbook to build a more recent, non-system perl
# and manage all cpan modules using these recipes.
include_recipe "perlbrew::default"

I am attaching a sanitized build.perl-5.16.3.log.txt (hostname foo-ized but otherwise undisturbed) build log in the hope it will be useful. I see nothing that stands out.

The nodes (both failed and successful) were spread amongst 2 availability zones in the same region, so it's not related to location. The root volume space is exactly the same on these nodes, and is only about 30% full, so it's not related to disk space. All of these nodes do a git checkout from bitbucket, so they all have the same internet connectivity.

Is adding a retry the correct fix? Or is there an underlying issue that needs to be addressed first (the install is actually failing and it marks as success)?

[xdg]

I've stopped using Chef, so I have no plans to do more with this repo. You're welcome to adopt if, if you'd like.