facts_hash_hiera_merge removed in 4.0.0?

[Sparc0]

Below hiera config in 4.0.0 does not exist and had to revert to 3.5.0 to not overwrite my facter file.
facter::facts_hash_hiera_merge: true

Any reason why?

[wolfaba]

You can use following to merge facter::facts_hash. Maybe that's the reason, why it's not required anymore

lookup_options:
  facter::facts_hash:
    merge:
      strategy: deep
      merge_hash_arrays: true

[Sparc0]

That lookup_options config did not work with 4.0.0 it still overwrites manual changes i done in my facter file.
I tried multiple lookup_options even with regex to match anything after facter:: but nothing seems to fix it as it was in 3.5.0.

To not have it overwrite my facter file i had to add what you suggested and this to all my yaml profiles files

---
facter::facts_hash:
  profile:
    value: 'http'

[wolfaba]

We set following in data/common.yaml (as described in https://www.puppet.com/docs/puppet/8/hiera_merging.html#overriding-merge-behavior) and it works

lookup_options:
  facter::facts_hash:
    merge:
      strategy: deep
      merge_hash_arrays: true

Could you maybe paste general parts of your yaml configuration? Maybe the lookup_options for the facter are not loaded in hiera? And could you paste the puppet agent -t --noop output, what change does puppet want to make.

I tried multiple lookup_options even with regex to match anything after facter::

You don't need to match any thing. The class is facter and it has parameter facts_hash therefore facter::facts_hash is correct.

[Sparc0]

my hiera.yaml

---
version: 5
defaults:
  datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"
  data_hash: yaml_data

hierarchy:
  - name: "Secret data: common"
    lookup_key: eyaml_lookup_key
    path: "common.eyaml"
    options:
      pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
      pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem

  - name: "Node"
    path: "node/%{::fqdn}.yaml"

  - name: "Profile"
    path: "profile/%{::profile}.yaml"

  - name: "Role"
    path: "role/%{::role}.yaml"

  - name: "Groups"
    path: "groups/%{::groups}.yaml"

  - name: "Domain"
    path: "domain/%{::domain}.yaml"

  - name: "OS"
    paths:
      - "os/%{os.family}.yaml"
      - "os/%{os.name}.yaml"

  - name: "Common"
    path: "common.yaml"

my hieradata/common.yaml i have this

---
classes:
  - facter

# facter
facter::facts_file: 'facts.txt'

lookup_options:
  facter::facts_hash:
    merge:
      strategy: deep
      merge_hash_arrays: true

This is my puppet agent -t --noop output

Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Notice: Requesting catalog from redacted:8140 (192.168.1.15)
Notice: Catalog compiled by redactedmaster
Info: Applying configuration version '1738168939'
Notice: /Stage[main]/Adcli::Join/Exec[adcli_join]/returns: current_value 'notrun', should be ['0'] (noop) (corrective)
Notice: Class[Adcli::Join]: Would have triggered 'refresh' from 1 event
Notice: /Stage[main]/Types/Types::Type[service]/Service[tftp]/ensure: current_value 'stopped', should be 'running' (noop)
Info: /Stage[main]/Types/Types::Type[service]/Service[tftp]: Unscheduling refresh on Service[tftp]
Notice: Types::Type[service]: Would have triggered 'refresh' from 1 event
Notice: Class[Types]: Would have triggered 'refresh' from 1 event
Notice: /Stage[main]/Facter/Concat[facts_file]/File[/etc/facter/facts.d/facts.txt]/content:
--- /etc/facter/facts.d/facts.txt    2025-01-29 16:50:12.219477771 +0100
+++ /tmp/puppet-file20250129-107397-1uamu7k     2025-01-29 17:42:23.849473135 +0100
@@ -2,4 +2,3 @@
 #DO NOT EDIT
 site_code=ams
-profile=http

Notice: /Stage[main]/Facter/Concat[facts_file]/File[/etc/facter/facts.d/facts.txt]/content: current_value '{sha256}bdc3674efdb22a7794e5eb072dddbc3c8ad8cb8ff3896e9be851ef46cfa29c7b', should be '{sha256}9cedb8a305f91140332434c792b17b60c0945e97c124a32fb085ab99d3b457d8' (noop)
Notice: facts_file: Would have triggered 'refresh' from 1 event
Notice: Concat[facts_file]: Would have triggered 'refresh' from 2 events
Notice: /Stage[main]/Sssd::Service/Service[sssd.service]/ensure: current_value 'stopped', should be 'running' (noop)
Info: /Stage[main]/Sssd::Service/Service[sssd.service]: Unscheduling refresh on Service[sssd.service]
Notice: Class[Sssd::Service]: Would have triggered 'refresh' from 1 event
Notice: Class[Sssd]: Would have triggered 'refresh' from 1 event
Notice: Class[Facter]: Would have triggered 'refresh' from 1 event
Notice: Stage[main]: Would have triggered 'refresh' from 5 events
Notice: Applied catalog in 1.63 seconds

The value site_code=ams comes from a file under domain/%{::domain}.yaml

[wolfaba]

And in what file do you set the profile? Where is defined the

---
facter::facts_hash:
  profile:
    value: 'http'

[Sparc0]

I put that as a workaround in profile/http.yaml
But before i did not have to do that and i could do manual changes in my facter.txt like adding profile=http without it being overwritten.

[wolfaba]

I am no puppet proffesional, but if I get together these important parts:

hiera.yaml

hierarchy:
  - name: "Secret data: common"
    ...
  - name: "Profile"
    path: "profile/%{::profile}.yaml"
    ...

profile/http.yaml

---
facter::facts_hash:
  profile:
    value: 'http'

Then I see the problem, that it should set the fact in the profile/http.yaml but the profile/http.yaml is loaded dynamically from hiera.yaml only if profile fact is set to any value.

As flow described:

The puppet starts processing hiera.yaml. The profile fact is empty, not yet set anywhere. Puppet tries to load profile/.yaml (the fact ::profile is empty). The fact profile has not been set, therefore it will be removed from facts file.

This behaviour is probably because of improving facts file content. There was a code in v3.5.0 in file manifests/fact.pp using file_line:

  file_line { "fact_line_${name}":
    path  => "${facts_dir}/${file}",
    line  => "${name}=${value}",
    match => $match,
  }

but this code keep non-existing facts, only add new one. But then would be kept facts in the file, which have not defined. Then there was a change 4786443 to generate the whole file new only with defined facts.

In fact, your hiera has no logic. You want to load some yaml file of the profile, which you define in the specific profile. Correctly, you should define fact profile in some other file (which is processed by default), in what you want to activate the other profile/PROFILE.yaml options.

We define fact role in the host yaml file and in the hier we load ROLE.yaml file, which then define options for the ROLE on the host.

The value site_code=ams comes from a file under domain/%{::domain}.yaml

Yes, because fact domain is set somewhere else (could you find, where?).

I hope this will help you.

[Sparc0]

The value site_code=ams comes from a file under domain/%{::domain}.yaml

Yes, because fact domain is set somewhere else (could you find, where?).

domain is part of facters core fact so it determines that based on the fqdn of the server. https://www.puppet.com/docs/puppet/8/core_facts.html#networking

Sure i could load the role from a static puppet file but dont really want to. We will probably look at some ENC in the future.
But the old WoW we had was to manually change the local facts.txt file to set certain facts like roles or profiles that we wanted to load on the server.

---

When i was using facter v3.5.0 i could manually edit my facts.txt file like below
to give my server a profile value.

root@redact:~# cat /etc/facter/facts.d/facts.txt
# File managed by Puppet
#DO NOT EDIT
site_code=ams
profile=http

If i then run puppet agent it would load that profile on to the server.
It still does as you can see below it changes my motd
It then configure and starts my tftp and httpd services.
But then it also removes my line in /etc/facter/facts.d/facts.txt
that i had manually added.

root@redact:~# puppet agent -t --noop
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Notice: Requesting catalog from redact:8140 (10.5)
Notice: Catalog compiled by redac
Info: Applying configuration version '1738331169'
Notice: /Stage[main]/Motd/File[/etc/motd]/content:
--- /etc/motd   2024-12-04 09:03:08.291727081 +0100
+++ /tmp/puppet-file20250131-674625-1w0cmz2     2025-01-31 14:46:14.917465923 +0100
@@ -3,7 +3,7 @@

 Located in:                   ams
 OS profile:                   Ubuntu 24.04.1 LTS
-Puppet profile:
+Puppet profile:               http
 Puppet role:



Notice: /Stage[main]/Motd/File[/etc/motd]/content: current_value '{sha256}054cd10eae6a89ad79a083fa35eb13afb37b3961390a8ae21504c2ea46434937', should be '{sha256}c4e11bf21694bbd9515bbb49c38211bc6a1bad70b514a9c671cb1daaa9d8160a' (noop)
Notice: Class[Motd]: Would have triggered 'refresh' from 1 event
Notice: /Stage[main]/Types/Types::Type[package]/Package[tftp-server]/ensure: current_value 'purged', should be 'present' (noop)
Notice: /Stage[main]/Types/Types::Type[package]/Package[httpd]/ensure: current_value 'purged', should be 'present' (noop)
Notice: Types::Type[package]: Would have triggered 'refresh' from 2 events
Notice: /Stage[main]/Types/Types::Type[service]/Service[tftp]/ensure: current_value 'stopped', should be 'running' (noop)
Info: /Stage[main]/Types/Types::Type[service]/Service[tftp]: Unscheduling refresh on Service[tftp]
Notice: /Stage[main]/Types/Types::Type[service]/Service[httpd]/ensure: current_value 'stopped', should be 'running' (noop)
Info: /Stage[main]/Types/Types::Type[service]/Service[httpd]: Unscheduling refresh on Service[httpd]
Notice: Types::Type[service]: Would have triggered 'refresh' from 2 events
Notice: Class[Types]: Would have triggered 'refresh' from 2 events
Notice: /Stage[main]/Facter/Concat[facts_file]/File[/etc/facter/facts.d/facts.txt]/content:
--- /etc/facter/facts.d/facts.txt    2025-01-31 14:46:02.069613293 +0100
+++ /tmp/puppet-file20250131-674625-1sozu4o     2025-01-31 14:46:15.343461037 +0100
@@ -2,4 +2,3 @@
 #DO NOT EDIT
 site_code=ams
-profile=http

Notice: /Stage[main]/Facter/Concat[facts_file]/File[/etc/facter/facts.d/facts.txt]/content:
--- /etc/facter/facts.d/facts.txt    2025-01-31 14:46:02.069613293 +0100
+++ /tmp/puppet-file20250131-674625-9v6dqo      2025-01-31 14:46:15.354460910 +0100
@@ -2,4 +2,3 @@
 #DO NOT EDIT
 site_code=ams
-profile=http

Notice: /Stage[main]/Facter/Concat[facts_file]/File[/etc/facter/facts.d/facts.txt]/content: current_value '{sha256}bdc3674efdb22a7794e5eb072dddbc3c8ad8cb8ff3896e9be851ef46cfa29c7b', should be '{sha256}9cedb8a305f91140332434c792b17b60c0945e97c124a32fb085ab99d3b457d8' (noop) (corrective)
Notice: facts_file: Would have triggered 'refresh' from 1 event
Notice: Concat[facts_file]: Would have triggered 'refresh' from 2 events
Notice: Class[Facter]: Would have triggered 'refresh' from 1 event
Notice: Stage[main]: Would have triggered 'refresh' from 3 events
Notice: Applied catalog in 3.42 seconds

And next puppet run it will remove the profile from my motd

root@redact:~# puppet agent -t --noop
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Notice: Requesting catalog from redact:8140 (10.5)
Notice: Catalog compiled by redact
Info: Applying configuration version '1738331692'
Notice: /Stage[main]/Motd/File[/etc/motd]/content:
--- /etc/motd   2025-01-31 14:54:32.949753338 +0100
+++ /tmp/puppet-file20250131-675645-1e7klql     2025-01-31 14:54:57.817468083 +0100
@@ -3,7 +3,7 @@

 Located in:                   ams
 OS profile:                   Ubuntu 24.04.1 LTS
-Puppet profile:               http
+Puppet profile:
 Puppet role:


Notice: /Stage[main]/Motd/File[/etc/motd]/content: current_value '{sha256}c4e11bf21694bbd9515bbb49c38211bc6a1bad70b514a9c671cb1daaa9d8160a', should be '{sha256}054cd10eae6a89ad79a083fa35eb13afb37b3961390a8ae21504c2ea46434937' (noop)
Notice: Class[Motd]: Would have triggered 'refresh' from 1 event
Notice: Stage[main]: Would have triggered 'refresh' from 1 event
Notice: Applied catalog in 3.29 seconds

So really what i am asking is why my facter::facts_hash is not merging with local changes not defined anywhere in my puppet code.
Maybe this WoW is not possible anymore with 4.0.0 then i have to accept it and redo all my profiles.

[wolfaba]

domain is part of facters core fact

Exactly

Sure i could load the role from a static puppet file but dont really want to ... But the old WoW we had was to manually change the local facts.txt file to set certain facts like roles or profiles that we wanted to load on the server. ... When i was using facter v3.5.0 i could manually edit my facts.txt

Please, don't take it offensive, but IMHO your WoW is incorrect. You shouldn't do anything manually on the server. You should configure everything in puppet.

What is for you the difference to edit plain text file facts.txt on the server and to configure profile in the host's yaml file? You have to enter the profile somewhere, in one file. And the main configuration is the puppet (either puppet code .pp or hiera .yaml). The content of the server (incl. facts.txt file) should be only run time copy of the puppet configuration.

When i was using facter v3.5.0 i could manually edit my facts.txt ...

Yes, you could. And IMHO this behaviour was incorrect (bug). If you had some fact already in the file, then there was no possibility to delete it from the file. In this case, either you had to delete it manually from the file or there was a risk, that the server will be configured incorrectly because of defined incorrect fact. Now, it works correctly. The facts.txt file should be only reproduction (copy) of the puppet configuration. You should be able to configure the whole server only from puppet without requiring to do anything on the server manually.

Imagine you have to reinstall 50 servers. After basic installation you have to go to all these 50 servers, find out, which profile do they use, edit some file and then run puppet to make correct installation. If you have everything in puppet, then you run a few times puppet after base host installation and you have the same configuration as before.

We use puppet different way than you. We rebuild server using openstack server rebuild, which creates new clean VM with default debian image, is starts cloudinit to make first basic configuration and puppet agent installation. Then puppet gets the complete host configuration from the puppet server. The host's yaml file contains role, rolefamily, accounts, backup and other included or direct configuration (like profile in your env) and gets the complete configuration from these files. The cloudinit runs the puppet agent multiple times and in about 20 minutes we have running and completely configured system as before rebuild.

Consider to change your WoW and configuration to define profile in host's file to load profile/PROFILE.yaml configration.

[Sparc0]

Okey i see it was a bug. Yes i know our WoW is incorrect, it was done that way because of legacy reasons.
Thanks a lot for your time very appreciated.