Using hiera data model in Puppet5 modules

[Aethylred]

Puppet5 is doing away with params.pp. Yay!

This has been replaced with a hiera.yaml file in the module's root directory and a data directory containing the Hiera datastore.

How do you tell rspec-puppet to use this Hiera datastore?

I've created a new Puppet5 module here where I've tried to:

• set heira config in spec_helper.rb (error log)

c.hiera_config = "hiera.yaml"

• set up hiera in a spec test (error log):

require 'spec_helper'

describe 'puppet5' do
  on_supported_os.each do |os, facts|
    context "on #{os}" do
      let(:facts) do
        facts
      end
      # Initialise hiera
      let(:hiera_config) { 'hiera.yaml' }
      hiera = Hiera.new(:config => 'hiera.yaml')

      it { is_expected.to compile.with_all_deps }
      it { should contain_class('puppet5::oscheck') }

      context "with no paramters" do
        it { should contain_package('puppet-agent').with(
          'ensure' => hiera.lookup('ensure_package', nil, nil),
          'name'   => hirea.lookup('package', nil, nil)
        ) }
      end

    end
  end
end

I suspect if I got the path correct it ought to just work, in which case we just need the documentation updated for Puppet5.

[rodjek]

Thanks for the report! I haven't looked into the details of how the hiera data in modules is implemented in Puppet 5, so I'm not sure if I need to make any changes yet to support it.

Much ❤️ for providing an example module, I'll do a bit of experimentation tonight and let you know :)

[rodjek]

Well, that's fun. So, rspec-puppet is using the in module hiera data just fine. But accessing that data from in your spec is not so easy. The Hiera library can not handle the new config file (bails out if the version >= 4), so it's not as easy as using Hiera.lookup in rubyland anymore and involves poking deep into Puppet::Pops::Lookup which requires access to the compiler scope which I'm not currently exposing to the example group... anyway, it's possible but it's not a quick fix I'm afraid and will probably need to wait until the next minor release given the scope of the changes.

In the meantime, you should be able to get going by specifying the expected values

diff --git a/spec/classes/puppet_spec.rb b/spec/classes/puppet_spec.rb
index c5b9322..0dee158 100644
--- a/spec/classes/puppet_spec.rb
+++ b/spec/classes/puppet_spec.rb
@@ -1,25 +1,36 @@
 require 'spec_helper'
 
 describe 'puppet5' do
+  package_details = {
+    'redhat-6-x86_64' => {
+      :ensure => '5.0.0-1.el6',
+    },
+    'redhat-7-x86_64' => {
+      :ensure => '5.0.0-1.el7',
+    },
+    'centos-6-x86_64' => {
+      :ensure => '5.0.0-1.el6',
+    },
+    'centos-7-x86_64' => {
+      :ensure => '5.0.0-1.el7',
+    },
+  }
+
   on_supported_os.each do |os, facts|
     context "on #{os}" do
       let(:facts) do
         facts
       end
-      # Initialise hiera
-      let(:hiera_config) { 'hiera.yaml' }
-      hiera = Hiera.new(:config => 'hiera.yaml')
 
       it { is_expected.to compile.with_all_deps }
       it { should contain_class('puppet5::oscheck') }
 
       context "with no paramters" do
         it { should contain_package('puppet-agent').with(
-          'ensure' => hiera.lookup('ensure_package', nil, nil),
-          'name'   => hirea.lookup('package', nil, nil)
+          'ensure' => package_details[os][:ensure],
         ) }
       end
 
     end
   end
-end
\ No newline at end of file
+end

[Aethylred]

Oh, I think I have a workaround. If I create 'spec/hiera.yaml' and specify version 4 and call it a filthy kludge...

...instead of the proper file it might stop the Hiera library from dying.

[Aethylred]

I'll use your recommended solution as my tests with older hiera.yaml configs didn't work.

I've branched my module for testing hiera lookups with rspec_puppet

Using a Hiera 3.3. config file in spec/hiera.yaml seems to load up hiera but gives a lot of errors

1) puppet5 on centos-6-x86_64 with no paramters 
     Failure/Error: 'ensure' => hiera.lookup('ensure_package', nil, nil),
     
     NoMethodError:
       undefined method `lookup' for #<Hiera:0x0000000315dff8>

[cdenneen]

Similar hiera5 error when loaded in spec tests:

Failure/Error: hiera = Hiera.new(config: 'spec/fixtures/hiera.yaml')

RuntimeError:
  v4 hiera.yaml is only to be used inside an environment or a module and cannot be given to the global hiera

Odd it says v4 even though it is v5.
(hiera 3.4.1 gem loaded)

[LongLiveCHIEF]

I've found that due to the nature of hiera layer 3 data sources being the end-all-be-all of data sources, that any hiera_config location values you pass to rspec wind up getting overridden by the modules own hiera.yaml.

Puppet 5 modules actually don't require a hiera fixture at all, which is quite nice, but this has caused problems when the actual data sources are exectable in a test environment (for example, eyaml data sources voxpupuli/hiera-eyaml#253)

[nward]

I am having trouble with this also, so posting here partly to get notifications, and partly to raise another issue.

When doing away with params.pp, we are encouraged not to include the top level class of each module. The expectation (as I understand it) is that we pull in data from a top level module class in to the more specific classes by using global scoped variables, i.e.:

class module (
  String $data,
) {
  contains module::specific
}

class module::specific {
  notify { $::module::data: }
}

When writing a spec for module::specific, I have seem to specify module::data in node_params, which is quite tedious, as I have to do this for every class other than the top level class.

Have I done something wrong here, or, is this intended, or in need of a fix of some sort?

[dandunckelman]

I'd love to avoid moving all my hiera data into my specs. Is that still the only workaround?

[LongLiveCHIEF]

You shouldn't have to move all the data into specs, it works just fine if you have the hiera.yaml file defined in the module root.

You just have to test the module in isolation, instead if treating it as if you are providing it data from layer 1 or layer 2.

(All data for the module must be in the module).

This should not be confused with facts. In puppet 5 facts and hiera data are not the same thing.

[dandunckelman]

@LongLiveCHIEF I should have specified that in order for me to verify certain values, I must move them from hiera to the spec (akin to what was specified here: #551 (comment)).

Otherwise, when I put this in my spec:

# this loads the hiera.yaml from the module's root
hiera = Hiera.new( :config => 'hiera.yaml' )

# normally, I'd then be able to fetch values from hiera like this:
api_version = hiera.lookup("api_version", nil, facts)

I get this error:

v5 hiera.yaml is only to be used inside an environment or a module and cannot be given to the global hiera

I've been wondering if testing specific values isn't effective (code smell? really...what's the point?), so it's not really a loss.

• Unless the values may be different based on certain conditions (different os/release/etc)
• The important thing to test is presence of resources especially when certain conditions affect which resources are present.

Thoughts?

[LongLiveCHIEF]

What's going on here, (what puppet tries to tell you but does a bad job of), is that you're calling a lookup for a piece of data outside the scope of the module, instead of a piece of data within the module.

You're also using a lookup function that is deprecated.

Hiera 5 is part of Puppet 5. They're no longer separate things, so you don't need to do a hiera lookup for the data in the module, nor do you need to define and load hiera in your specs. It's automatically loaded by Puppet.

So:

1. You don't need Hiera.new() to load the hiera file/data. This is now a part of Puppet APL and comes in as module scoped data/variables
2. hiera.lookup should just be lookup()
3. Looking up data needs to refer to the correct scope chain for Puppet/Hiera 5, so api_version is bad, instead use modulename::api_version.
4. There are 2 ways to include this data in the module:

• make a class parameter for api_version, then in your modules data sources provide a resolution for mymodule::api_version. From there on out, everywhere in your module (even subclasses), all you need to do to get the value of the data for this is refer to $mymodule::api_version. No lookup or hiera functions required. Hiera data is within scope
• if you don't make a class parameter for api_version, then in your manifest use $myvar = lookup('mymodule::api_version'). The hiera.yaml is located at the root of your module, and is loaded automatically in Puppet 5 and merges into APL

Some examples of modules I maintain that show you how this works:

• https://github.com/voxpupuli/puppet-gitlab
• https://github.com/voxpupuli/puppet-prometheus

[LongLiveCHIEF]

If you want more help, join the Slack Puppet Community and check out the #testing channel.

[nward]

Hi @LongLiveCHIEF,

Ironically I was googling this again having forgotten I'd been in this discussion, and found my comment - and though "That's exactly my problem" and realised then that it was my own words.

Anyway, I've had a look at the examples you gave, and I see where I'm going wrong. I'm writing a spec file for each class (which is how PDK generates them, unless I'm using that incorrectly), rather than for the top level class which includes and/or contains the various subclasses. In your case, you only write specs for the top level class.

That helps a lot, and I've got a bunch of internal refactoring to do.. but, for much the better I think.

How would you test a case where there are 2 classes which can be included, one is compulsary, one optional, i.e.:

• someservice (main class)
• someservice::monitoring

Where the some service::monitoring class requires someservice to exist and pulls parameters from it, rather than being passed them directly? Is that a reasonable case to use something like:

let(:node_params) do
  {
   'someservice::myparam' => 1234,
  }
end

I could of course have a param on someservice which causes the monitoring subclass to be included, however, I'm concerned about a huge number of parameters that I'm adding to my top level class.

[nward]

Ack, just saw your next message.. joining that now.

[m0dular]

For anyone hitting this in 2024, following this article worked for me. I didn't have to change the spec helper, the only steps were to:

• Add the new hierarchy in hiera-rspec.yaml
• Add the desired data in the new data/rspec.yaml
• Load this hiera config in the desired tests with let(:hiera_config) { 'hiera-rspec.yaml' }

[davidsandilands]

@m0dular just to clarify Puppet forked this repo and it is maintained here, im not sure if the approach to hiera documented has everything you need https://github.com/puppetlabs/rspec-puppet/?tab=readme-ov-file#hiera-integration