Should not depend on php & python

[swrobel]

I realize these are necessary for their respective monitors, but they should be recommends rather than depends as their dependencies shouldn't be installed if we're just using the ruby agent.

[chr4]

I super-agree with this. Unfortunately (to my knowledge) recommends and suggests do not work.
In my experience, when using one of the commands instead of depends, the recipe fails as soon as it is trying to access something from those cookbooks.
Did you test whether that works? If so, feel free to file a pull-request. I have similar issues with several other cookbooks, and it annoys me too, that there are so so many cookbooks which would work fine for me without dependencies and sub-depdendencies.

[jeffbyrnes]

@swrobel I've had the same experience; depends is the only one that seems to work properly.

As a workaround, it would be possible to remove the dependencies completely from metadata.rb, adjust the Berksfile to load the cookbooks similarly to Apache in the integration group, so testing works properly.

The caveat to this is that for the end-user of this cookbook, they will need to ensure that whatever wrapper cookbook they use depends on the PHP and Python cookbooks if they plan on using the related providers/recipes in this cookbook.

[swrobel]

Hm, I'm a chef newb, so I'll definitely trust your experiences over mine. Is this what I'm doomed to - installing tons of packages that I don't need just to satisfy the broadest possible cookbook dependencies? But I digress...

What about splitting the cookbook out into others for each specific agent?

[jeffbyrnes]

@swrobel with such a variety of technologies, having multiple cookbooks, one for each language/agent, may be the best path forward. This cookbook originally started life (correct me if I'm wrong @chr4) as a way to install the repository, server monitor, and Plugin Agent. It then grew to accommodate the generic Ruby agent, and most recently, the PHP agent (courtesy of me).

So it might make sense to have a base cookbook for the repository & server monitor, then separate agent cookbooks:

• newrelic-ng-plugin-agent-cookbook
• newrelic-ng-generic-agent-cookbook
• newrelic-ng-php-agent-cookbook

This would also play nicely with Berkshelf, and allow the testing to be simplified all-around.

[swrobel]

👍

[chr4]

Here are the cookbooks that we're depending on:

• python
• build-essential
• php
• xml
• mysql
• openssl

Mmh, I suppose that Opscode has to take some action here, to make recommends and suggests work in a way that they make sense. Splitting up the cookbook would be a solution, I cannot see this being common practise so far in the Chef world though (which doesn't mean it doesn't suck), and also adds up a lot of maintenance. Plus, they will probably depend on one another anyways, unless we duplicate some code (which is also not nice).

[jeffbyrnes]

True, the keywords in metadata should work properly, but we don't have control over that.

As for splitting the cookbook up, I'd argue that's relatively common, especially in a Berkshelf-driven workflow. For me, there's no issue of a cookbook depending on another; that's kind of the point, in fact. We've got a number of "wrapper" cookbooks that are just a way to customize how a more generic cookbook functions. If you haven't checked it, Jamie Winsor's The Berkshelf Way presentation does a good job explaining this way of thinking with regards to wrapper cookbooks.

@chr4, you're more familiar than I am, but I don't see a ton of overlap in the recipes/providers/resources.

[chr4]

Thanks @jeffbyrnes for the link to the talk.

I do not recall whether there's overlapping resources in this cookbook, my statement was though in a more general way. I don't want seperate cookbooks for $database-client and $database-server, as several things like versions, repositories, etc make sense to be shared.

Maybe the best approach would be to open a ticket (unless it exists already?) at Opscode and ask what their plans for suggests and recommends is? Maybe we can even contribute a fix?

There are cookbooks (even from Opscode) with ridiculous amounts of dependencies and sub-dependencies, I think the community would love us for such a feature :)

[jeffbyrnes]

Let's definitely raise an issue with Opscode, for sure!

That being said, I'm not sure I follow with regards to separate cookbooks for $database-client and $database-server. Would you mind elaborating?

[chr4]

I was reffering to opscode-cookbooks/postgresql as an example where shared attributes makes sense. If you look at the default attributes you can see that e.g. the version is specified centrally for client as well as server cookbooks (which makes sense).

If those cookbooks where seperated in postgresql-client and postgresql-server, you'd have to make sure both version attributes are aligned when using them in the same environment.

[jeffbyrnes]

@chr4 ah, I see what you mean. In the case of PostgreSQL, yeah, makes perfect sense to keep them together.

Here, however, we have separate agents, which, with the exception of the repository, don't have any dependence on each other. I can envision a scenario where each agent cookbook depends on the repostiory/server monitor cookbook (which do, in my mind, belong as one & the same cookbook), and take advantage of license key, user/group, etc., but otherwise don't share attributes.

Food for thought; I'm great with the cookbook as-is, but I understand the desire to reduce the "dependencies" if you're not using PHP/Python.

[jeffbyrnes]

That all being said; just because those cookbooks are depended upon, doesn't mean that those cookbooks will actually have their recipes ever run. So, while the php and python cookbooks will be uploaded to your Chef Server, nothing more should come of it than that.

[chr4]

The php dependencies are getting worse. In recent versions, they added yum-epel, windows, iss, plus their sub-dependencies.
Did someone create a ticket about making the suggest attribute work in metadata.rb?

[chr4]

It seems Opscode is aware of this: https://tickets.opscode.com/browse/CHEF-3870
Feel free to vote up this issue.

[jeffbyrnes]

@chr4 I'm curious; ignoring that suggest should work properly, what's the harm in depending on a cookbook if you never include its recipes in a run list or a recipe? Perhaps I'm not as aware of how Chef works as I'd like, but my understanding is that, if none of a cookbook’s recipes are included somewhere, then nothing happens with regards to that cookbook.

Is the issue that all of those additional cookbooks are installed to your Chef Server? I can understand a desire for not cluttering things up.

[chr4]

How did you get suggest to work?
This change

# metadata.rb
-depends 'php'
+suggests 'php'

leads to this error (after doing a knife cookbook delete php)

Missing Cookbooks:
------------------
The following cookbooks are required by the client but don't exist on the server:
* php

Am I missing something?

It's indeed all the not required cookbooks being installed on the Chef server. While it "works", it's still annoying, plus, I'm usually using repo to manage my (bigger) Chef kitchens, as (provided you maintain a lot of cookbooks instead of just using them) Berkshelf/Librarian cannot cope with branches, rebasing, updating git repositories that well. Therefore I need to add each cookbook manually.

[jeffbyrnes]

We didn't get suggest to work; we just use depends everywhere for any potential dependencies, and let Berkshelf handle the rest.

I'm not familiar with repo, is that from a Knife plugin, or an aspect of Berkshelf/Librarian?

We have to maintain a number of our own cookbooks (most are private), but we just use Berkshelf, which deals with ensuring all the dependencies are uploaded if they don't exist on the Chef Server.

Berkshelf can definitely deal with branches, commits, and tags though! If you're testing a specific version, you'd want to update the Berksfile to point to the repository & a specific branch/tag/ref, then use Test Kitchen, or just plain Vagrant, to see if your changes converge properly.

If there's some interdependency of cookbooks, we usually specify in the Berksfile to look for the other cookbook(s) locally during testing, like so:

site :opscode

metadata

cookbook 'et_users', path: '../et_users'

All of our cookbook repositories are in the same folder, but you could set that path to anything, frankly.

Once things converge well, we discard the changes to the Berksfile in the cookbook in question, merge down to master (usually via a Pull Request for some review), and craft a release by updating the metadata.rb & CHANGELOG.

Does that help explain why I don't have much trouble with depends?

[chr4]

repo is a tool to manage multiple git repositories, invented and used by Google for Android.
It's not Chef related, but a pure git tool.

You are right, checking out different tags and branches is certianly possible with Berkshelf!
I meant something different though:

repo can

• checkout (not clone) different branches for several (or all) cookbooks
• automatically rebase local changes upon upstream changes (on sync)
• keep environemnts, nodes, data_bags under version control
• keep all repositories synced for all devops
• provides the ability to review code before merging them into master (also for e.g. a certian branch)

The downside is, though, that it doesn't manage dependencies automatically :)

Basically,imo it has advantages when you maintain multiple cookbooks instead of just using them.
To achieve something similar with Berkshelf I think you need to keep another "developing" copy of your cookbook somewhere, which then need to be syncronised invidvidually. Which probably makes sense if you just "use" cookbooks, instead of maintaining them.

[jeffbyrnes]

Interesting. I'd be curious to check out this repo tool; where might I find it?

As for keeping the environment, data_bags, etc. under version control, here's our structure:

.
├── .chef
│   └── contains knife config, etc.
├── cookbooks
│   ├── each cookbook is its own repo
│   └── this dir & its contents are untracked & ignored by "chef" repo
├── data_bags
│   └── tracked by "chef" repo
├── environments
│   └── tracked by "chef" repo
└── roles
    └── tracked by "chef" repo

Everything's under one "roof", but we avoid submodules etc.

[chr4]

Yes, but you cannot sync/branch/checkout all your branches at once. :)

repo can be found here: https://code.google.com/p/git-repo/ (it's also included in homebrew (for MacOS))
Also checkout the documentation: http://source.android.com/source/using-repo.html

To get started, you can use a default.xml in your manifest repository like this:

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <remote name="github" fetch="git://github.com" />
  <remote name="chr4" fetch="ssh://[email protected]/chr4-cookbooks" />
  <remote name="opscode" fetch="git://github.com/opscode-cookbooks" />
  <default revision="master" remote="opscode" />

  <project name="iptables-ng" path="cookbooks/iptables-ng" remote="chr4" />
  <project name="newrelic-ng" path="cookbooks/newrelic-ng" remote="chr4" />
  <project name="resolvconf" path="cookbooks/resolvconf" remote="chr4" />

  <!-- opscode cookbooks -->
  <project name="ssh_known_hosts" path="cookbooks/ssh_known_hosts" />
  <project name="partial_search" path="cookbooks/partial_search" />

  <!-- other cookbooks -->
  <project name="customink-webops/hostsfile" path="cookbooks/hostsfile" remote="github" />
</manifest>

[andytson]

Berkshelf supports:

cookbook 'mycookbook', :path => 'site-cookbooks/mycookbook'

This means you can manage your development cookbooks without berkshelf trashing them. It doesn't, however, manage the dependencies, but then ruby is ruby and Berksfile can contain ruby to pull down git repositories with a single definition in one place