README.md in xmorph-0.1.14 vs README.md in xmorph-0.1.16

- old
+ new

@@ -1,127 +1,114 @@ + + + # XMorph Morphs things from one kind to another. Transcodes, in local speak, for example.. -# Features! +# Features - - XMorph can get validate an asset against given set of validations (set of video and audio parameters). - - Can choose a pre-defined proifile based on asset's mediainfo, and transcode using the profiles corresponding command. + - XMorph can validate asset before transcoding against given set of validations (set of video and audio parameters). + - Perform pre transcode processing on an asset, for example meta data extraction on mxf file + - Can choose a pre-defined profile based on asset's mediainfo, and transcode using the profiles corresponding command. + - Perform validations after transcoding + - Perform post transcode processing - if you want to extract something for a ts file. # Installation: ```sh $ gem install xmorph ``` -> For a specific version: + For a specific version: ```sh -$ gem install xmorph -v 0.1.5 +$ gem install xmorph -v 0.1.15 ``` -# Usage - -Once you have written XMorph profile for a customer, test if it's working fine. -Get base class object for a given customer -```ruby -transcoder = XMorph::Base.get_transcode_template(HOST, ACCOUNT_DOMAIN, ACCOUNT_NAME, ASSET_PATH) - -transcoder = XMorph::Base.get_transcode_template("cinedigm", "Cinedigm", "amagi", "/home/tejaswini/volt/cinedigm/PRAYERLINK112718CC.mxf") -``` -transcoder is object for a customer class, in case XMorph is able to find corresponding customers transcoding setup, else it'll raise TranscoderError exception. -You can also pass extra param, which will return the filepath it loaded inorder to pick customer specific transcode setup. -```ruby -transcoder, loaded_filepath = XMorph::Base.get_transcode_template("cinedigm", "Cinedigm", "amagi", "/home/tejaswini/volt/cinedigm/PRAYERLINK112718CC.mxf", true) -``` -Here loaded_filepath will be -```ruby -/home/tejaswini/src/xmorph/lib/xmorph/customers/cinedigm/Cinedigm/transcode.rb -``` -Get mediainfo for the asset with: -```ruby -transcoder.get_asset_details -``` -If XMorph is not able to get mediainfo of the asset, it will raise exception. If it was success you can access mediainfo with, which will be a hash. -```ruby -transcoder.mediainfo_output -``` -Perform default validations on the asset once you get mediainfo. -```ruby -transcoder.validate_asset -``` -This function will validate the asset against the requirements mentioned in video_checks and audio_checks methods for a given customer. It will either return true or raise exception incase of failure. - -Once the asset is validated, try to get suitable profile for the same. -```ruby ffmpeg_command = transcoder.get_profile``` or ```profile_name, ffmpeg_cmd = transcoder.get_profile``` to get profile name as well. - -The ffmpeg command come with %{IN} and %{OUT} replace them with input filepath and output filepath and pass as parameter for, ```transcoder.transcode(updated_ffmpeg_command)``` - -# Execution -```sh -$ cd xmorph -$ ./bin/console -$ transcoder = XMorph::Base.get_transcode_template("cinedigm", "Cinedigm", "amagi", "/home/tejaswini/volt/cbn/PRAYERLINK112718CC.mxf") -$ transcoder, loaded_filepath = XMorph::Base.get_transcode_template("cinedigm", "Cinedigm", "amagi", "/home/tejaswini/volt/cbn/PRAYERLINK112718CC.mxf", true) -$ transcoder.get_asset_details -$ transcoder.validate_asset -$ ffmpeg_cmd = transcoder.get_profile -$ profile_name, ffmpeg_cmd = transcoder.get_profile(true) -$ transcoder.transcode(ffmpeg_cmd % {IN: asset_path, OUT: tmp_file.path}) -``` - # Adding new XMorph profile for a customer ```sh $ cd xmorph/lib/xmorph/customers -$ mkdir -p HOST/ACCOUNT_DOMAIN/ -$ cd HOST/ACCOUNT_DOMAIN/ -$ vi transcode.rb +$ mkdir -p HOST/ACCOUNT_DOMAIN/Ingest +$ cd HOST/ACCOUNT_DOMAIN/Ingest +$ vi validate.rb #mandatory +$ vi transcode.rb #optional +$ vi pre_processor.rb #optional +$ vi post_processor.rb #optional ``` -> Here, HOST is customer in customer.amagi.tv and ACCOUNT_DOMAIN is account.domain -> EX: for cinedigm.amagi.tv HOST is cinedigm and account's domain is Cinedigm -> Interfaces to be implemented in transcodee.rb, -```ruby -class Transcode < XMorph::Base - #define constants for set of profiles which says what the corresponding command does, or what the input asset is - SCALE_TO_720x480_4_3 = "720x480_4_3" - SCALE_TO_720x480_16_9 = "720x480_16_9" - SCALE_TO_1920x1080_16_9 = "1920x1080_16_9" - SCALE_TO_720x480_3_2 = "720x480_3_2" - #set the profiles using the constants defined earlier - def set_profiles - self.profiles = { - SCALE_TO_720x480_4_3 => "ffmpeg -y -i %{IN} $options to perform transcoding/encoding$ %{OUT} 2>&1", - SCALE_TO_720x480_16_9 => "ffmpeg -y -i %{IN} $options to perform transcoding/encoding$ %{OUT} 2>&1", - SCALE_TO_1920x1080_16_9 => "ffmpeg -y -i %{IN} $options to perform transcoding/encoding$ %{OUT} 2>&1", - SCALE_TO_720x480_3_2 => "ffmpeg -y -i %{IN} $options to perform transcoding/encoding$ %{OUT} 2>&1", - } - end - +how to get HOST, ACCOUNT DOMAIN ? http://**gusto**[*host*].amagi.tv/**gusto**[*account_domain*]/**GUSTO**[*feed_code*]/medias +## File Contents +```ruby +#validate.rb +class Validate < XMorph::BaseValidator #values for the sollowing validations can be Range, Array or IGNORE(upcase) #Range - (n1..n2) - applicable if the values are numbers, validates if value x is between n1 and n2, it also includes fractions. ex 5: in (1..10) -> true and 29.97 in (25..30) -> true #Array - [n1,n2] - validates if value x is in the given array. ex: 5 in [1,10] -> false and 1 in [1,10] -> true #IGNORE will skip validation for the corresponding parameter. it will not even check if media has that parameter. i.e if mediainfo is not able to read parameter x for the asset, its validation will still return true. #These methods are to be implemented in the sub class, there's no defaut method defined in the base class, exception is raised if these methods are not found. - - def video_checks + #Checks before transcoding are mandatory + #ValidatorError will be raised if video_checks_before_transcoding and audio_checks_before_transcoding functions are not defined + def video_checks_before_transcoding { ALLOWED_ASPECT_RATIO => ["4:3", "16:9"] || IGNORE, ALLOWED_HEIGHT => (400..1080) || [720, 1080, 546] || IGNORE, ALLOWED_WIDTH => (640..720) || [720, 1920] || IGNORE, ALLOWED_FRAME_RATE => (25..30) || [25, 29.970, 24] || IGNORE, ALLOWED_VIDEO_BIT_RATE => (8..32) || [8, 32] || IGNORE, ALLOWED_SCAN_TYPE => ['progressive', 'interlaced'] || IGNORE, } - end - - def audio_checks + end + + def audio_checks_before_transcoding { PRESENCE_OF_AUDIO_TRACK => IGNORE || VALIDATE, ALLOWED_NUMBER_OF_AUDIO_TRACKS => (1..16)|| [2, 4, 6, 8] || IGNORE, ALLOWED_AUDIO_CODECS => ['aac', 'ac-3', 'mp2', 'mp4', 'dolby e', 'mpeg audio'] || IGNORE, ALLOWED_AUDIO_BIT_RATE => (120..317) || [192, 317] || IGNORE, ALLOWED_NUMBER_OF_AUDIO_CHANNELS => (1..16) || [1, 2] || IGNORE, + ALLOWED_SAMPLING_RATE => (41..48) || [41, 48] || IGNORE, } - end - + end + + #The below functions are optional + #We are ignoring Frame rate check after transcoding, since mediainfo is not able to read frame rate of the transcoded asset. + def video_checks_after_transcoding + { + ALLOWED_ASPECT_RATIO => ["4:3", "16:9"] || IGNORE, + ALLOWED_HEIGHT => (400..1080) || [720, 1080, 546] || IGNORE, + ALLOWED_WIDTH => (640..720) || [720, 1920] || IGNORE, + ALLOWED_VIDEO_BIT_RATE => (8..32) || [8, 32] || IGNORE, + ALLOWED_SCAN_TYPE => ['progressive', 'interlaced'] || IGNORE, + } + end + + def audio_checks_after_transcoding + { + PRESENCE_OF_AUDIO_TRACK => IGNORE || VALIDATE, + ALLOWED_NUMBER_OF_AUDIO_TRACKS => (1..16)|| [2, 4, 6, 8] || IGNORE, + ALLOWED_AUDIO_CODECS => ['aac', 'ac-3', 'mp2', 'mp4', 'dolby e', 'mpeg audio'] || IGNORE, + ALLOWED_AUDIO_BIT_RATE => (120..317) || [192, 317] || IGNORE, + ALLOWED_NUMBER_OF_AUDIO_CHANNELS => (1..16) || [1, 2] || IGNORE, + ALLOWED_SAMPLING_RATE => (41..48) || [41, 48] || IGNORE, + } + end + +end +``` +```ruby +#transcode.rb +class Transcode < XMorph::BaseTranscoder + + PRO_1080_16_TRACKS = "1080_16" + PRO_720_STEREO = "720_stereo" + + def set_profiles + self.profiles = { + PRO_1080_16_TRACKS => "ffmpeg -y -i %{IN} -vf \"fps=25.000000,scale=1920x1080\" -filter_complex \"[0:a:8][0:a:9]amerge=inputs=2[aout0]\" -pix_fmt yuv420p -vcodec h264 -g 13 -bf 2 -x264opts nal-hrd=cbr -profile:v high -flags +ilme+ildct -top 1 -vb 10000000 -minrate:v 10000000 -maxrate:v 10000000 -bufsize:v 20000000 -acodec libfdk_aac -profile:a aac_low -ab 192k -map 0:v -map \"[aout0]\" -muxrate 11211200 -streamid 0:2064 -streamid 1:2068 %{OUT} 2>&1", + #If you have sequence of commands to be executed + #Make sure you folow file naming convention + PRO_720_STEREO => ["ffmpeg -y -i %{IN} -s 1920x1080 -vcodec h264 -profile:v high -r 25 -g 13 -pix_fmt yuv420p -bf 2 -x264opts nal-hrd=cbr -vb 15000000 -map 0:v -streamid 0:2064 -bufsize:v 24000000 -acodec libfdk_aac -ac 2 -ar 48000 -profile:a aac_low -ab 192k -map 0:a:0 -muxrate 13411200 -streamid 1:2068 -vsync 1 -async 1 -v verbose %{TEMPFILE_1.ts}", "../transcoder -if %{TEMPFILE_1.ts} -of %{TEMPFILE_2.ts} -vp -acm \"1,2;1,2;1,2\" -ac \"aac;ac3;pass\" -lt -24 -ltp -2 -ac3_dialnorm -24", "../transcoder -if %{TEMPFILE_2.ts} -of %{OUT} -vp -acm \"1,2;3,4;5,6;5,6\" -ac \"pass;pass;pass;aac\" -lt -12 -ltp -2"], + } + end + #Write summary as what are the combinations of profiles we get for this customer, and based on which parameters we choose the profile. # have a set of conditional statements to choose a profile defined above. def set_profile_name self.profile_name = nil @@ -136,5 +123,146 @@ #Ensure there's only if..elseif statements rather than if..else XMorph::Base.logger.debug("XMorph#set_profile_name#Cinedigm: using profile #{self.profile_name}") unless self.profile_name.nil? return true end ``` +```ruby +#pre_processor.rb or post_processor.rb +class PreProcessor < XMorph::BaseProcessor + EXTRACT_META_DATA = "extract_meta_data" + + def set_volt_commands + self.volt_commands = { + EXTRACT_META_DATA => "docker run --rm -v %{IP_MOUNT_PATH}:%{IP_MOUNT_PATH} %{DOCKER_IMAGE} ./volt/extract_mxf_meta.sh %{IN_FILE}" + } + end + + def process_asset + #To convert the above mentioned commands to executable ones + self.transform_volt_commands + status, response = XMorph::Util.run_cmd_with_response(self.volt_commands[EXTRACT_META_DATA]) + raise ProcessorError.new(response) unless status + #Write code to convert response to hash which can be consumed by blip/end user + end +end +``` +# Usage + +Once you have written XMorph profile for a customer, test if it's working fine. +```ruby +#Install latest xmorph gem +require 'xmorph' + +#Initialise XMorph as +XMorph::Base.initialize(VOLT_TAG, HOST, DOMAIN, ACCOUNT_NAME) +XMorph::Base.initialize("amagidevops/volt:mr_2.1.14", "gusto", "gusto", "amagi") + +#For validations before transcoding +#validator can be nil, if validations are not defined +#This function will validate the asset against the requirements mentioned in video_checks_before_transcoding and audio_checks_before_transcoding methods for a given customer. It will either return true or raise exception incase of failure. + +validator = XMorph::BaseValidator.get_validator(ASSET_PATH) +#this function gives ValidatorError if any validation fails. +validator.validate_asset_before_transcoding + +#pre transcode processing +#metadata should be a hash consisting of data we expect to be extracted out of the asset +status, metadata = XMorph::BaseProcessor.process_before_transcoding(ASSET_PATH) + +#Transcoding +#This will raise TranscoderError if template is not found, if it's not able to get transcoding profile or if transcoding fails. +transcoder = XMorph::BaseTranscoder.get_transcode_template(ASSET_PATH) +transcoder.transcode(ASSET_PATH) + +#post transcode validations +#performs validations if defined, raises ValidatorError, if validations are not met +validator.validate_asset_after_transcoding + +#post transcode processing +status, metadata = XMorph::BaseProcessor.process_after_transcoding(ASSET_PATH) +``` + +# Trying to keep things simple +Dump mediainfo of an asset to a file and execute a command to check what profile it chooses to transcode. + +Write mediainfo to a file: +``` sh +$ mediainfo --output=XML #{ASSET_PATH} > ./xmorph/spec/xmorph/customers/{HOST}/{ACCOUNT_DOMAIN}/#{ASSET_NAME}.xml +``` + +Change the below template to be used for a new customer, change contents within %{ } and file_names and profiles within data +**To verify validations:** +```ruby +#In ./xmorph/spec/xmorph/customers/%{gusto}/%{gusto}/transcode_spec.rb +require "spec_helper" +describe 'Transcode' do + + before (:each) do + load "./lib/xmorph/customers/%{gusto}/%{gusto}/ingest/validate.rb" + @validator = Validate.new('/home/www/EXP/Media/asd.mp4') + end + + #have mediainfo xml files in same dir as _spec files within a customer specific directory. + #./xmorph/spec/xmorph/customers/%{gusto}/%{gusto}/#{ASSET_NAME}.xml + it "should perform validations for a given mediainfo" do + data = [ + {"mediainfo_filepath" => "sample.xml", "transcoded" => false, "valid" => true, "error" => nil}, + ] + validate(Validate.new(""), data) #,true) + end + +end +``` +And RUN: +```sh +$ bundle exec rspec spec/xmorph/customers/%{gusto}/%{gusto}/transcode_spec.rb +``` + +**To verify transcoding profile:** + +```ruby +#In ./xmorph/spec/xmorph/customers/%{gusto}/%{gusto}/transcode_spec.rb +require "spec_helper" +describe 'Transcode' do + + before (:each) do + load "./lib/xmorph/customers/%{gusto}/%{gusto}/ingest/transcode.rb" #give correct load path + @transcoder = Transcode.new('/home/www/EXP/Media/asd.mp4') #Asset path can be anything + end + + #have mediainfo xml files in same dir as _spec files within a customer specific directory. + #./xmorph/spec/xmorph/customers/gusto/gusto/#{ASSET_NAME}.xml + it "should choose corresponding profiles for a given mediainfo" do + data = [ + {"mediainfo_filepath" => "Ep1004_1_TomatoOnionRaita_Clean.xml", "profile" => Transcode::PRO_720_STEREO}, + ] + #pass true at the end for dry run. i.e. it will not run rspec instead it will print matched profile/error for a given mediainfo + get_profiles(Transcode.new('sample_asset'), data) #,true) + end +end +``` +And RUN: +```sh +$ bundle exec rspec spec/xmorph/customers/%{gusto}/%{gusto}/transcode_spec.rb +``` +This will tell, if the profile you mentioned was matched or no, and if it wasn't matched it prints error message telling why it did not match. + +# Deployment +Gem: +```sh +$ cd ./xmorph +$ gem build xmorph.gemspec +$ gem push xmorph-<version>.gem #This will be availble in rubygems.org from where you can install henceforth +``` + +Deploy GIT TAG: +```sh +$ cd ./xmorph +$ cap production deploy +Please enter server (newproduction.amagi.tv): <test.amagi.tv> +Please enter ssh_port (22): <PORT> +Please enter ssh_password: <PASSWORD> +Please enter branch (): <GIT TAG +``` + +WRT Blip, if we use cap deploy, changes required in blip are made to use deployed xmorph on host and auto-scaled instances. +This will be available on 7.x from next release \ No newline at end of file