Class: Rudder::DSL::Pipeline

Inherits:
Object
  • Object
show all
Includes:
Util
Defined in:
lib/rudder/dsl/pipeline.rb

Overview

Concourse Pipeline. Main entry of the DSL. Evaluates user defined pipelines.

DSL Usage:

Pipeline's are composed of various components:

  • Resource: basic inputs and output of jobs.

  • Job: basic computation unit of a pipeline

  • ResourceType: custom resource definitions

  • Group: logical grouping of jobs in the UI. Either every job is in a Group or no job is (hard Concourse requirement)

Adding Components

Components are added to the Pipeline by component type, followed by name, optional arguments, then typically a block.

Loading Other Pipelines

Pipeline's can load other pipeline definitions using #load. This is a useful mechanism for abstracting out common subsections of pipelines, then merging them into larger pipelines.

Merging Pipeline Components

Pipeline's can merge loaded or defined Rudder::DSL:Pipelines's, Hash's of components, or Array's of components into the current definition using #merge_components.

Including Other Pipelines

Pipeline's can include other pipeline definitions using #include_pipeline. This is similar to Rudder::DSL:Pipeline#load except that all the components are automatically included into this Pipeline

Including Individual Components

Individual pipeline components can also be defined on a per-file basis and then loaded into a Pipeline using #include_component. This is useful for factoring out common resources for multiple pipeline's to use.

Examples:

Adding Components to Pipelines

#
# my_pipeline_definition.rb
#
resource :my_git_repo, :git do
  source[:uri]    = 'https://github.com/my/repo.git'
  source[:branch] = :master
end

resource :daily, :time do
  source[:interval] = '24h'

job :build_project do
  plan << [in_parallel: [{ get: :my_git_repo }, { get: :daily, trigger: true}]]
  build = { task: 'build my project', config: {
    platform: :linux,
    image_resource: { type: 'docker-image', source: { repository: 'busybox' } },
    run: { path: 'my_git_repo/build.sh' }
  }}
  plan << build
end

Loading / Importing Pipelines

#
# load_neighbor.rb
#
neighbor = load 'neighbor_pipeline.rb'

# merge all the neighboring resources and jobs into this pipeline
resources.merge! neighbor.resources
jobs.merge! neighbor.jobs

resource_type :slack_notification, 'docker-image' do
  source[:repository] = 'some/slack-docker-repo'
end

resource :our_slack_channel, :slack_notification do
  source[:url] = '((slack-team-webhook))'
end

# Add a slack notification task to the end
# of every job
jobs.values.each do |job|
  job.plan << {
    put: :our_slack_channel,
    params: { text: "Job #{job.name} complete!" }
  }
end

Loading / Importing Individual Components

#
# operations_scripts_resource.rb
#
type :git
source[:uri]    = 'https://github.com/<our org>/operations_scripts.git'
source[:branch] = 'master'

#
# some_operations_pipeline.rb
#

# load the resource into the pipeline. Automatically includes
# the resource into the resources list with the name :scripts
include_component 'operations_scripts_resource.rb', :resource, :scripts

job :audit do |pipeline|
  plan << {
    task: 'run the audit script', config: {
      platform: :linux,
      image_resource: {
        type: 'docker-image',
        source: { repository: 'alpine/git' }
      },
      run: {
        path: pipeline.scripts.sub_path('audit.rb')
      }
    }
  }
end

See Also:

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods included from Util

#_convert_h_val, #_deep_to_h

Constructor Details

#initialize(file_path = nil, resources: {}, jobs: {}, groups: {}, resource_types: {}) ⇒ Pipeline

All pipelines require:

  • Jobs

  • Resources

Concourse Pipelines may optionally provide:

  • Resource Types

  • Groups

Rudder Pipelines may optionally include a file_path. This is required when loading resources from neighboring files.

All pipeline requirements are only needed at the Pipeline render time (after evaluation), and need not be specified for initialization.

Parameters:

  • file_path (String) (defaults to: nil) —

    path to this Rudder::DSL::Pipeline definition.

  • resources (Hash<(String, Symbol), Rudder::DSL::Resource] map of Resource names to their definitions.) —

    esources [Hash<(String, Symbol), Rudder::DSL::Resource] map of Resource names to their definitions.

  • jobs (Hash<(String, Symbol), Rudder::DSL::Job] map of Job names to their definitions.) —

    obs [Hash<(String, Symbol), Rudder::DSL::Job] map of Job names to their definitions.

  • groups (Hash<(String, Symbol), Rudder::DSL::Group] map of Group names to their definitions.) —

    roups [Hash<(String, Symbol), Rudder::DSL::Group] map of Group names to their definitions.

  • resources_types (Hash<(String, Symbol), Rudder::DSL::ResourceType] map of Resource Type names to their definitions.) —

    esources_types [Hash<(String, Symbol), Rudder::DSL::ResourceType] map of Resource Type names to their definitions.



183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
 
# File 'lib/rudder/dsl/pipeline.rb', line 183

def initialize(file_path = nil, resources: {}, jobs: {},
               groups: {}, resource_types: {})
  @resources      = resources
  @jobs           = jobs
  @groups         = groups
  @resource_types = resource_types
  # rubocop:disable Layout/AlignHash, Layout/SpaceBeforeComma
  @known_classes  = {
    resource:      { clazz: Resource    , pipeline_group: @resources      },
    job:           { clazz: Job         , pipeline_group: @jobs           },
    group:         { clazz: Group       , pipeline_group: @groups         },
    resource_type: { clazz: ResourceType, pipeline_group: @resource_types }
  }
  # rubocop:enable Layout/AlignHash, Layout/SpaceBeforeComma
  @pipelines = {}
  @file_path = file_path
end

Dynamic Method Handling

This class handles dynamic methods through the method_missing method

#method_missing(method, *args, &component_block) ⇒ Rudder::DSL::Component?

Populates this Rudder::DSL::Pipeline with components and optionally fetches defined components.

Fetching

When method is called with no arguments it is treated as a Rudder::DSL::Pipeline getter method. method is translated to the name of a Component and the Component is returned if defined, otherwise nil is returned.

Setting

When method is passed any arguments (positional, placement, or block) then method is treated as a setter.

When setting, method must be the name of a known Component. The first argument is a required name for the component. All arguments and keyword arguments are then delegated to the Component's specific initializer.

Finally, when a block is provided it is evaluated within the context of the newly constructed Component with full priveleges to operate on it.

Returns:

Raises:

  • (RuntimeError) —

    when attempting to define an unknown Component



261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
 
# File 'lib/rudder/dsl/pipeline.rb', line 261

def method_missing(method, *args, &component_block)
  local_component = _get_local_component(method)
  if !@known_classes.include?(method) && !local_component
    return super.send(method, args, component_block)
  end

  # Look up a previously defined component from the pipeline
  return local_component if local_component && args.empty? && !block_given?

  component_group = @known_classes[method][:pipeline_group]
  name = args[0]
  raise "Overlapping component name: #{method}" if component_group.include? name

  component = @known_classes[method][:clazz].new(*args)

  component.instance_exec self, &component_block if block_given?
  component_group[name] = component
end

Instance Attribute Details

#groups ⇒ Hash<(String, Symbol), Rudder::DSL::Group>

Hash of names to Group

Returns:



155
156
157
 
# File 'lib/rudder/dsl/pipeline.rb', line 155

def groups
  @groups
end

#jobs ⇒ Hash<(String, Symbol), Rudder::DSL::Job>

Hash of names to Job

Returns:



149
150
151
 
# File 'lib/rudder/dsl/pipeline.rb', line 149

def jobs
  @jobs
end

#resource_types ⇒ Hash<(String, Symbol), Rudder::DSL::ResourceType>

Hash of names to ResourceType

Returns:



152
153
154
 
# File 'lib/rudder/dsl/pipeline.rb', line 152

def resource_types
  @resource_types
end

#resources ⇒ Hash<(String, Symbol), Rudder::DSL::Resource>

Hash of names to Resource

Returns:



146
147
148
 
# File 'lib/rudder/dsl/pipeline.rb', line 146

def resources
  @resources
end

Instance Method Details

#eval(file_path = nil) ⇒ Rudder::DSL::Pipeline

Evaluates the given file path. If file_path nil, defaults to the one provided at construction time If both are nil, raises an exception

Parameters:

  • file_path (String, nil) (defaults to: nil) —

    path to Rudder::DSL::Pipeline definition to evaluate. Uses the current file_path if nil

Returns:

Raises:

  • (RuntimeError) —

    if file_path and #file_path are both nil



303
304
305
306
307
308
309
310
311
312
313
 
# File 'lib/rudder/dsl/pipeline.rb', line 303

def eval(file_path = nil)
  @file_path = file_path || @file_path
  if @file_path.nil?
    raise 'File path must be provided at Pipeline initialization or eval call'
  end

  File.open(@file_path) do |f|
    instance_eval f.read, @file_path
  end
  self
end

#include_component(component_path, class_sym, name, *args) ⇒ Object

Note:

This automatically includes the component into this pipeline

Given a path to a component, its class, and any args required to construct it, creates a new component

Parameters:

  • component_path (String) —

    path, relative to this pipeline, containing a Component to load

  • class_sym (Symbol) —

    symbol of a Component. May be one of: (:job, :resource, :resource_type, :group)

  • name (String, Symbol) —

    name to use for the loaded Component. Must not be nil.

  • *args —

    any additional arguments to pass to the Component constructor.

Raises:

  • RuntimeError if name is nil or an uknown class_sym is provided.



371
372
373
374
375
376
377
378
379
380
 
# File 'lib/rudder/dsl/pipeline.rb', line 371

def include_component(component_path, class_sym, name, *args)
  raise "Unable to load #{class_sym}" unless @known_classes.keys.include? class_sym
  raise 'Name must not be nil' if name.nil?

  full_path = File.join(File.dirname(@file_path), component_path)
  component = @known_classes[class_sym][:clazz].new(name, *args)
  component.instance_eval File.read(full_path), full_path
  @known_classes[class_sym][:pipeline_group][name] = component
  component
end

#include_pipeline(other_pipeline_path) ⇒ nil

Note:

Any component provided that has an overlapping name with a previously defined component in this pipeline will override the previous definition

Includes another Rudder::DSL::Pipeline from a file into this Rudder::DSL::Pipeline.

Parameters:

Returns:

  • (nil) 


393
394
395
396
 
# File 'lib/rudder/dsl/pipeline.rb', line 393

def include_pipeline(other_pipeline_path)
  pipeline = load other_pipeline_path
  merge_components(pipeline)
end

#load(other_pipeline_path, resources: {}, resource_types: {}, jobs: {}, groups: {}) ⇒ Object

Given a path relative to this pipeline, loads another pipeline and returns it

Note that this includes nothing from the relative pipeline in this one, instead just returning the pipeline to be manipulated

May also optionally provides hashes for

  • resources

  • resource_types

  • jobs

  • groups

Parameters:



338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
 
# File 'lib/rudder/dsl/pipeline.rb', line 338

def load(other_pipeline_path, resources: {}, resource_types: {},
         jobs: {}, groups: {})
  if @pipelines.key? other_pipeline_path
    @pipelines[other_pipeline_path]
  else
    dir = File.dirname(@file_path)
    full_path = File.join(dir, other_pipeline_path)
    pipeline = Rudder::DSL::Pipeline.new(
      full_path, resources: resources, resource_types: resource_types,
                 jobs: jobs, groups: groups
    ).eval
    @pipelines[other_pipeline_path] = pipeline
    pipeline
  end
end

#merge_components(components) ⇒ nil

Note:

Any component provided that has an overlapping name with a previously defined component in this pipeline will override the previous definition

Merges the provided Rudder::DSL::Pipeline components into this Rudder::DSL::Pipeline.

Modes of Operation:

When provided a Pipeline, merges all like components into this Pipeline

  • Hash<String, Component>:

When provided a Hash, merges all the components into the like Hash in this Pipeline

  • Array<Component>:

When provided an Array, merges all Components into this Pipeline. Inserts components into their respective Pipeline group by class.

Parameters:

Returns:

  • (nil)

Raises:

  • (RuntimeError) —

    when type provided is unsupported



425
426
427
428
429
430
431
432
 
# File 'lib/rudder/dsl/pipeline.rb', line 425

def merge_components(components)
  case components
  when Pipeline then _merge_pipeline(components)
  when Hash then _merge_hash(components)
  when Array then _merge_array(components)
  else raise 'Unable to merge #{components}: unsupported type'
  end
end

#p_convert_h_val(hash) ⇒ Object

Wraps Util#_convert_h_val since it will always set use_name to false



222
223
224
 
# File 'lib/rudder/dsl/pipeline.rb', line 222

def p_convert_h_val(hash)
  _convert_h_val(hash, false)
end

#respond_to?(name, _include_all = true) ⇒ Boolean

Returns:

  • (Boolean) 


289
290
291
 
# File 'lib/rudder/dsl/pipeline.rb', line 289

def respond_to?(name, _include_all = true)
  @known_classes.key? name
end

#respond_to_missing?(*_) ⇒ Boolean

Rudder::DSL::Pipeline's respond to missing

Returns:

  • (Boolean) —

    true



285
286
287
 
# File 'lib/rudder/dsl/pipeline.rb', line 285

def respond_to_missing?(*_)
  true
end

#to_h ⇒ Hash

Renders all of this pipeline's components to their Hash representations.

Returns:

  • (Hash) —

    YAML friendly Hash representation of this Pipeline if either groups or resource_types is empty they will not be included in the rendering at all.



209
210
211
212
213
214
215
216
217
 
# File 'lib/rudder/dsl/pipeline.rb', line 209

def to_h
  h = {
    'resources' => p_convert_h_val(@resources.values),
    'jobs' => p_convert_h_val(@jobs.values)
  }
  h['groups'] = p_convert_h_val(@groups.values) unless @groups.empty?
  h['resource_types'] = p_convert_h_val(@resource_types.values) unless @resource_types.empty?
  h
end