Class: Rudder::DSL::Pipeline

Inherits:
Object
  • Object
show all
Includes:
Util
Defined in:
lib/rudder/dsl/pipeline.rb

Overview

Concourse Pipeline. Main entry of the DSL. Evaluates user defined pipelines.

DSL Usage:

Pipeline's are composed of various components:

  • Resource: basic inputs and output of jobs.

  • Job: basic computation unit of a pipeline

  • ResourceType: custom resource definitions

  • Group: logical grouping of jobs in the UI. Either every job is in a Group or no job is (hard Concourse requirement)

Adding Components

Components are added to the Pipeline by component type, followed by name, optional arguments, then typically a block.

Loading Other Pipelines

Pipeline's can load other pipeline definitions using #load. This is a useful mechanism for abstracting out common subsections of pipelines, then merging them into larger pipelines.

Loading Individual Components

Individual pipeline components can also be defined on a per-file basis and then loaded into a Pipeline using #load_component. This is useful for factoring out common resources for multiple pipeline's to use.

Examples:

Adding Components to Pipelines

#
# my_pipeline_definition.rb
#
resource :my_git_repo, :git do
  source[:uri]    = 'https://github.com/my/repo.git'
  source[:branch] = :master
end

resource :daily, :time do
  source[:interval] = '24h'

job :build_project do
  plan << [in_parallel: [{ get: :my_git_repo }, { get: :daily, trigger: true}]]
  build = { task: 'build my project', config: {
    platform: :linux,
    image_resource: { type: 'docker-image', source: { repository: 'busybox' } },
    run: { path: 'my_git_repo/build.sh' }
  }}
  plan << build
end

Loading / Importing Pipelines

#
# load_neighbor.rb
#
neighbor = load 'neighbor_pipeline.rb'

# merge all the neighboring resources and jobs into this pipeline
resources.merge! neighbor.resources
jobs.merge! neighbor.jobs

resource_type :slack_notification, 'docker-image' do
  source[:repository] = 'some/slack-docker-repo'
end

resource :our_slack_channel, :slack_notification do
  source[:url] = '((slack-team-webhook))'
end

# Add a slack notification task to the end
# of every job
jobs.values.each do |job|
  job.plan << {
    put: :our_slack_channel,
    params: { text: "Job #{job.name} complete!" }
  }
end

Loading / Importing Individual Components

#
# operations_scripts_resource.rb
#
type :git
source[:uri]    = 'https://github.com/<our org>/operations_scripts.git'
source[:branch] = 'master'

#
# some_operations_pipeline.rb
#

# load the resource into the pipeline. Automatically includes
# the resource into the resources list with the name :scripts
load_component 'operations_scripts_resource.rb', :resource, :scripts

job :audit do |pipeline|
  plan << {
    task: 'run the audit script', config: {
      platform: :linux,
      image_resource: {
        type: 'docker-image',
        source: { repository: 'alpine/git' }
      },
      run: {
        path: pipeline.scripts.sub_path('audit.rb')
      }
    }
  }
end

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Util

#_convert_h_val, #_deep_to_h

Constructor Details

#initialize(file_path = nil, resources: {}, jobs: {}, groups: {}, resource_types: {}) ⇒ Pipeline

All pipelines require:

  • Jobs

  • Resources

Concourse Pipelines may optionally provide:

  • Resource Types

  • Groups

Rudder Pipelines may optionally include a file_path. This is required when loading resources from neighboring files.

All pipeline requirements are only needed at the Pipeline render time (after evaluation), and need not be specified for initialization.

Parameters:

  • file_path (String) (defaults to: nil) —

    path to this Rudder::DSL::Pipeline definition.

  • resources (Hash<(String, Symbol), Rudder::DSL::Resource] map of Resource names to their definitions.) —

    esources [Hash<(String, Symbol), Rudder::DSL::Resource] map of Resource names to their definitions.

  • jobs (Hash<(String, Symbol), Rudder::DSL::Job] map of Job names to their definitions.) —

    obs [Hash<(String, Symbol), Rudder::DSL::Job] map of Job names to their definitions.

  • groups (Hash<(String, Symbol), Rudder::DSL::Group] map of Group names to their definitions.) —

    roups [Hash<(String, Symbol), Rudder::DSL::Group] map of Group names to their definitions.

  • resources_types (Hash<(String, Symbol), Rudder::DSL::ResourceType] map of Resource Type names to their definitions.) —

    esources_types [Hash<(String, Symbol), Rudder::DSL::ResourceType] map of Resource Type names to their definitions.



166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
 
# File 'lib/rudder/dsl/pipeline.rb', line 166

def initialize(file_path = nil, resources: {}, jobs: {},
               groups: {}, resource_types: {})
  @resources      = resources
  @jobs           = jobs
  @groups         = groups
  @resource_types = resource_types
  # rubocop:disable Layout/AlignHash, Layout/SpaceBeforeComma
  @known_classes  = {
    resource:      { clazz: Resource    , pipeline_group: @resources      },
    job:           { clazz: Job         , pipeline_group: @jobs           },
    group:         { clazz: Group       , pipeline_group: @groups         },
    resource_type: { clazz: ResourceType, pipeline_group: @resource_types }
  }
  # rubocop:enable Layout/AlignHash, Layout/SpaceBeforeComma
  @pipelines = {}
  @file_path = file_path
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(method, *args, &component_block) ⇒ Rudder::DSL::Component?

Populates this Rudder::DSL::Pipeline with components and optionally fetches defined components.

Fetching

When method is called with no arguments it is treated as a Rudder::DSL::Pipeline getter method. method is translated to the name of a Component and the Component is returned if defined, otherwise nil is returned.

Setting

When method is passed any arguments (positional, placement, or block) then method is treated as a setter.

When setting, method must be the name of a known Component. The first argument is a required name for the component. All arguments and keyword arguments are then delegated to the Component's specific initializer.

Finally, when a block is provided it is evaluated within the context of the newly constructed Component with full priveleges to operate on it.

Returns:

Raises:

  • (RuntimeError) —

    when attempting to define an unknown Component



237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
 
# File 'lib/rudder/dsl/pipeline.rb', line 237

def method_missing(method, *args, &component_block)
  local_component = _get_local_component(method)
  if !@known_classes.include?(method) && !local_component
    return super.send(method, args, component_block)
  end

  # Look up a previously defined component from the pipeline
  return local_component if local_component && args.empty? && !block_given?

  component_group = @known_classes[method][:pipeline_group]
  name = args[0]
  raise "Overlapping component name: #{method}" if component_group.include? name

  component = @known_classes[method][:clazz].new(*args)

  component.instance_exec self, &component_block if block_given?
  component_group[name] = component
end

Instance Attribute Details

#groups ⇒ Hash<(String, Symbol), Rudder::DSL::Group>

Hash of names to Group

Returns:



138
139
140
 
# File 'lib/rudder/dsl/pipeline.rb', line 138

def groups
  @groups
end

#jobs ⇒ Hash<(String, Symbol), Rudder::DSL::Job>

Hash of names to Job

Returns:



132
133
134
 
# File 'lib/rudder/dsl/pipeline.rb', line 132

def jobs
  @jobs
end

#resource_types ⇒ Hash<(String, Symbol), Rudder::DSL::ResourceType>

Hash of names to ResourceType

Returns:



135
136
137
 
# File 'lib/rudder/dsl/pipeline.rb', line 135

def resource_types
  @resource_types
end

#resources ⇒ Hash<(String, Symbol), Rudder::DSL::Resource>

Hash of names to Resource

Returns:



129
130
131
 
# File 'lib/rudder/dsl/pipeline.rb', line 129

def resources
  @resources
end

Instance Method Details

#_get_local_component(component) ⇒ Object

Yikes! Seems like a bad idea - if someone uses the same name twice (say, 1 resource and 1 job), then this will return one pretty much at random. TODO: Make this not bad Oh well.. TODO: This may be returning a non-nil/non-falsey type that causes some issues



363
364
365
366
367
368
369
370
371
 
# File 'lib/rudder/dsl/pipeline.rb', line 363

def _get_local_component(component)
  component = component.to_sym
  locals = @known_classes.values.map do |m|
    m[:pipeline_group][component]
  end.compact
  # TODO: Add a logger here..
  puts "Found multiple bindings for: #{p}. Getting first found" unless locals.size <= 1
  locals[0]
end

#eval(file_path = nil) ⇒ Rudder::DSL::Pipeline

Evaluates the given file path. If file_path nil, defaults to the one provided at construction time If both are nil, raises an exception

Parameters:

  • file_path (String, nil) (defaults to: nil) —

    path to Rudder::DSL::Pipeline definition to evaluate. Uses the current file_path if nil

Returns:

Raises:

  • (RuntimeError) —

    if file_path and #file_path are both nil



279
280
281
282
283
284
285
286
287
288
289
 
# File 'lib/rudder/dsl/pipeline.rb', line 279

def eval(file_path = nil)
  @file_path = file_path || @file_path
  if @file_path.nil?
    raise 'File path must be provided at Pipeline initialization or eval call'
  end

  File.open(@file_path) do |f|
    instance_eval f.read, @file_path
  end
  self
end

#load(other_pipeline_path, resources: {}, resource_types: {}, jobs: {}, groups: {}) ⇒ Object

Given a path relative to this pipeline, loads another pipeline and returns it

Note that this includes nothing from the relative pipeline in this one, instead just returning the pipeline to be manipulated

May also optionally provides hashes for

  • resources

  • resource_types

  • jobs

  • groups

Parameters:



314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
 
# File 'lib/rudder/dsl/pipeline.rb', line 314

def load(other_pipeline_path, resources: {}, resource_types: {},
         jobs: {}, groups: {})
  if @pipelines.key? other_pipeline_path
    @pipelines[other_pipeline_path]
  else
    dir = File.dirname(@file_path)
    full_path = File.join(dir, other_pipeline_path)
    pipeline = Rudder::DSL::Pipeline.new(
      full_path, resources: resources, resource_types: resource_types,
                 jobs: jobs, groups: groups
    ).eval
    @pipelines[other_pipeline_path] = pipeline
    pipeline
  end
end

#load_component(component_path, class_sym, name, *args) ⇒ Object

Given a path to a component, its class, and any args required to construct it, creates a new component

Note that this automatically includes the component into this pipeline

Parameters:

  • component_path (String) —

    path, relative to this pipeline, containing a Component to load

  • class_sym (Symbol) —

    symbol of a Component. May be one of: (:job, :resource, :resource_type, :group)

  • name (String, Symbol) —

    name to use for the loaded Component. Must not be nil.

  • *args —

    any additional arguments to pass to the Component constructor.

Raises:

  • RuntimeError if name is nil or an uknown class_sym is provided.



347
348
349
350
351
352
353
354
355
356
 
# File 'lib/rudder/dsl/pipeline.rb', line 347

def load_component(component_path, class_sym, name, *args)
  raise "Unable to load #{class_sym}" unless @known_classes.keys.include? class_sym
  raise 'Name must not be nil' if name.nil?

  full_path = File.join(File.dirname(@file_path), component_path)
  component = @known_classes[class_sym][:clazz].new(name, *args)
  component.instance_eval File.read(full_path), full_path
  @known_classes[class_sym][:pipeline_group][name] = component
  component
end

#respond_to?(name, _include_all = true) ⇒ Boolean

Returns:

  • (Boolean) 


265
266
267
 
# File 'lib/rudder/dsl/pipeline.rb', line 265

def respond_to?(name, _include_all = true)
  @known_classes.key? name
end

#respond_to_missing?(*_) ⇒ Boolean

Rudder::DSL::Pipeline's respond to missing

Returns:

  • (Boolean) —

    true



261
262
263
 
# File 'lib/rudder/dsl/pipeline.rb', line 261

def respond_to_missing?(*_)
  true
end

#to_h ⇒ Hash

Renders all of this pipeline's components to their Hash representations.

Returns:

  • (Hash) —

    YAML friendly Hash representation of this Pipeline if either groups or resource_types is empty they will not be included in the rendering at all.



192
193
194
195
196
197
198
199
200
 
# File 'lib/rudder/dsl/pipeline.rb', line 192

def to_h
  h = {
    'resources' => _convert_h_val(@resources.values),
    'jobs' => _convert_h_val(@jobs.values)
  }
  h['groups'] = _convert_h_val(@groups.values) unless @groups.empty?
  h['resource_types'] = _convert_h_val(@resource_types.values) unless @resource_types.empty?
  h
end