require 'date'
require 'drb'
require 'csv'
require 'fileutils'
require 'yaml'

#
# :main:KirbyBase
# :title:KirbyBase Class Documentation
# KirbyBase is a class that allows you to create and manipulate simple,
# plain-text databases.  You can use it in either a single-user or
# client-server mode.  You can select records for retrieval/updating using
# code blocks.
#
# Author::    Jamey Cribbs (mailto:jcribbs@twmi.rr.com)
# Homepage::  http://www.netpromi.com/kirbybase.html
# Copyright:: Copyright (c) 2005 NetPro Technologies, LLC
# License::   Distributes under the same terms as Ruby
# History:
# 2005-03-28:: Version 2.0
# * This is a completely new version.  The interface has changed
#   dramatically.
# 2005-04-11:: Version 2.1
# * Changed the interface to KirbyBase#new and KirbyBase#create_table.  You
#   now specify arguments via a code block or as part of the argument list.
# * Added the ability to specify a class at table creation time.
#   Thereafter, whenever you do a #select, the result set will be an array
#   of instances of that class, instead of instances of Struct.  You can
#   also use instances of this class as the argument to KBTable#insert,
#   KBTable#update, and KBTable#set.
# * Added the ability to encrypt a table so that it is no longer stored as
#   a plain-text file.
# * Added the ability to explicity specify that you want a result set to be
#   sorted in ascending order.
# * Added the ability to import a csv file into an existing table.
# * Added the ability to select a record as if the table were a Hash with
#   it's key being the recno field.
# * Added the ability to update a record as if the table were a Hash with
#   it's key being the recno field.
# 2005-05-02:: Version 2.2
# * By far the biggest change in this version is that I have completely
#   redesigned the internal structure of the database code.  Because the
#   KirbyBase and KBTable classes were too tightly coupled, I have created
#   a KBEngine class and moved all low-level I/O logic and locking logic
#   to this class.  This allowed me to restructure the KirbyBase class to
#   remove all of the methods that should have been private, but couldn't be
#   because of the coupling to KBTable.  In addition, it has allowed me to
#   take all of the low-level code that should not have been in the KBTable
#   class and put it where it belongs, as part of the underlying engine.  I
#   feel that the design of KirbyBase is much cleaner now.  No changes were
#   made to the class interfaces, so you should not have to change any of
#   your code.
# * Changed str_to_date and str_to_datetime to use Date#parse method.
# * Changed #pack method so that it no longer reads the whole file into
#   memory while packing it.
# * Changed code so that special character sequences like &linefeed; can be
#   part of input data and KirbyBase will not interpret it as special
#   characters.
# 2005-08-09:: Version 2.2.1
# * Fixed a bug in with_write_lock.
# * Fixed a bug that occurred if @record_class was a nested class.
# 2005-09-08:: Version 2.3 Beta 1
# * Added ability to specify one-to-one links between tables.
# * Added ability to specify one-to-many links between tables.
# * Added ability to specify calculated fields in tables.
# * Added Memo and Blob field types.
# * Added indexing to speed up queries.
# 2005-10-03:: Version 2.3 Beta 2
# * New column type:  :YAML.  Many thanks to Logan Capaldo for this idea!
# * Two new methods:  #add_table_column and #drop_table_column.
# * I have refined the select code so that, when you are doing a one-to-one
#   or one-to-many select, if an appropriate index exists for the child
#   table, KirbyBase automatically uses it.
# * I have changed the designation for a one-to-one link from Link-> to
#   Lookup-> after googling helped me see that this is a more correct term
#   for what I am trying to convey with this link type.
# 2005-10-10:: Version 2.3 Production
# * Added the ability to designate a table field as the "key" field, for
#   Lookup purposes.  This simply makes it easier to define Lookup fields.
# * This led me to finally give in and add "the Hal Fulton Feature" as I am
#   forever going to designate it.  You can now specify a Lookup field
#   simply by specifying it's field type as a table, for example:
#   :manager, :person (where :manager is the field name, and :person is the
#   name of a table).  See the docs for the specifics or ask Hal. :)
#
#---------------------------------------------------------------------------
# KirbyBase
#---------------------------------------------------------------------------
class KirbyBase
    include DRb::DRbUndumped

    attr_reader :engine

    attr_accessor(:connect_type, :host, :port, :path, :ext)

    #-----------------------------------------------------------------------
    # initialize
    #-----------------------------------------------------------------------
    #++
    # Create a new database instance.
    #
    # *connect_type*:: Symbol (:local, :client, :server) specifying role to
    #                  play.
    # *host*:: String containing IP address or DNS name of server hosting
    #          database. (Only valid if connect_type is :client.)
    # *port*:: Integer specifying port database server is listening on.
    #          (Only valid if connect_type is :client.)
    # *path*:: String specifying path to location of database tables.
    # *ext*:: String specifying extension of table files.
    #
    def initialize(connect_type=:local, host=nil, port=nil, path='./',
     ext='.tbl')
        @connect_type = connect_type
        @host = host
        @port = port
        @path = path
        @ext = ext

        # See if user specified any method arguments via a code block.
        yield self if block_given?

        # After the yield, make sure the user doesn't change any of these
        # instance variables.
        class << self
            private(:connect_type=, :host=, :path=, :ext=)
        end

        # Did user supply full and correct arguments to method?
        raise ArgumentError, 'Invalid connection type specified' unless (
         [:local, :client, :server].include?(@connect_type))
        raise "Must specify hostname or IP address!" if \
         @connect_type == :client and @host.nil?
        raise "Must specify port number!" if @connect_type == :client and \
         @port.nil?
        raise "Invalid path!" if @path.nil?
        raise "Invalid extension!" if @ext.nil?

        @table_hash = {}

        # If running as a client, start druby and connect to server.
        if client?
            DRb.start_service()
            @server = DRbObject.new(nil, 'druby://%s:%d' % [@host, @port])
            @engine = @server.engine
            @path = @server.path
            @ext = @server.ext
        else
            @engine = KBEngine.create_called_from_database_instance(self)
        end

        # The reason why I create all the table instances here is two
        # reasons:  (1) I want all of the tables ready to go when a user
        # does a #get_table, so they don't have to wait for the instance
        # to be created, and (2) I want all of the table indexes to get
        # created at the beginning during database initialization so that
        # they are ready for the user to use.  Since index creation
        # happens when the table instance is first created, I go ahead and
        # create table instances right off the bat.
        #
        # Also, I use to only execute the code below if this was either a
        # single-user instance of KirbyBase or if client-server, I would
        # only let the client-side KirbyBase instance create the table
        # instances, since there was no need for the server-side KirbyBase
        # instance to create table instances.  But, since I want indexes
        # created at db initialization and the server's db instance might
        # be initialized long before any client's db is initialized, I now
        # let the server create table instances also.  This is strictly to
        # get the indexes created, there is no other use for the table
        # instances on the server side as they will never be used.
        # Everything should and does go through the table instances created
        # on the client-side.
        @engine.tables.each do |tbl|
            @table_hash[tbl] = KBTable.create_called_from_database_instance(
             self, tbl, File.join(@path, tbl.to_s + @ext))
        end
    end

    #-----------------------------------------------------------------------
    # server?
    #-----------------------------------------------------------------------
    #++
    # Is this running as a server?
    #
    def server?
        @connect_type == :server
    end

    #-----------------------------------------------------------------------
    # client?
    #-----------------------------------------------------------------------
    #++
    # Is this running as a client?
    #
    def client?
        @connect_type == :client
    end

    #-----------------------------------------------------------------------
    # local?
    #-----------------------------------------------------------------------
    #++
    # Is this running in single-user, embedded mode?
    #
    def local?
        @connect_type == :local
    end

    #-----------------------------------------------------------------------
    # tables
    #-----------------------------------------------------------------------
    #++
    # Return an array containing the names of all tables in this database.
    #
    def tables
        return @engine.tables
    end

    #-----------------------------------------------------------------------
    # get_table
    #-----------------------------------------------------------------------
    #++
    # Return a reference to the requested table.
    # *name*:: Symbol of table name.
    #
    def get_table(name)
        raise('Do not call this method from a server instance!') if server?
        raise('Table not found!') unless table_exists?(name)

        if @table_hash.has_key?(name)
            return @table_hash[name]
        else
            @table_hash[name] = \
             KBTable.create_called_from_database_instance(self,
             name, File.join(@path, name.to_s + @ext))
            return @table_hash[name]
        end
    end

    #-----------------------------------------------------------------------
    # create_table
    #-----------------------------------------------------------------------
    #++
    # Create new table and return a reference to the new table.
    # *name*:: Symbol of table name.
    # *field_defs*:: List of field names (Symbols), field types (Symbols),
    #                field indexes, and field extras (Indexes, Lookups,
    #                Link_manys, Calculateds, etc.)
    # *Block*:: Optional code block allowing you to set the following:
    # *encrypt*:: true/false specifying whether table should be encrypted.
    # *record_class*:: Class or String specifying the user create class that
    #                  will be associated with table records.
    #
    def create_table(name=nil, *field_defs)
        raise "Can't call #create_table from server!" if server?

        t_struct = Struct.new(:name, :field_defs, :encrypt, :record_class)
        t = t_struct.new
        t.name = name
        t.field_defs = field_defs
        t.encrypt = false
        t.record_class = 'Struct'

        yield t if block_given?

        raise "Name must be a symbol!" unless t.name.is_a?(Symbol)
        raise "No table name specified!" if t.name.nil?
        raise "No table field definitions specified!" if t.field_defs.nil?

        @engine.new_table(t.name, t.field_defs, t.encrypt,
         t.record_class.to_s)

        return get_table(t.name)
    end

    #-----------------------------------------------------------------------
    # drop_table
    #-----------------------------------------------------------------------
    #++
    # Delete a table.
    #
    # *tablename*:: Symbol of table name.
    #
    def drop_table(tablename)
        raise "Table does not exist!" unless table_exists?(tablename)
        @table_hash.delete(tablename)
        return @engine.delete_table(tablename)
    end

    #-----------------------------------------------------------------------
    # table_exists?
    #-----------------------------------------------------------------------
    #++
    # Return true if table exists.
    #
    # *tablename*:: Symbol of table name.
    #
    def table_exists?(tablename)
        return @engine.table_exists?(tablename)
    end

    #-----------------------------------------------------------------------
    # add_table_column
    #-----------------------------------------------------------------------
    #++
    # Add a column to a table.
    #
    # Make sure you are executing this method while in single-user mode
    # (i.e. not running in client/server mode).  After you run it, it is
    # probably a good idea to release your handle on the db and
    # re-initialize KirbyBase, as this method changes the table structure.
    #
    # *tablename*:: Symbol of table name.
    # *col_name*:: Symbol of column name to add.
    # *col_type*:: Symbol (or Hash if includes field extras) of column type
    #              to add.
    # *after*:: Symbol of column name that you want to add this column
    #           after.
    #
    def add_table_column(tablename, col_name, col_type, after=nil)
        raise "Do not execute this method from the server!!!" if server?

        raise "Invalid table name!" unless table_exists?(tablename)

        raise "Invalid field name in 'after': #{after}" unless after.nil? \
         or @table_hash[tablename].field_names.include?(after)

        # Does this new column have field extras (i.e. Index, Lookup, etc.)
        if col_type.is_a?(Hash)
            temp_type = col_type[:DataType]
        else
            temp_type = col_type
        end

        raise 'Invalid field type: %s' % temp_type unless \
         KBTable.valid_field_type?(temp_type)

        @engine.add_column(@table_hash[tablename], col_name, col_type,
         after)

        # Need to reinitialize the table instance and associated indexes.
        @engine.remove_recno_index(tablename)
        @engine.remove_indexes(tablename)
        @table_hash.delete(tablename)
        @table_hash[tablename] = \
         KBTable.create_called_from_database_instance(self, tablename,
          File.join(@path, tablename.to_s + @ext))
    end

    #-----------------------------------------------------------------------
    # drop_table_column
    #-----------------------------------------------------------------------
    #++
    # Drop a column from a table.
    #
    # Make sure you are executing this method while in single-user mode
    # (i.e. not running in client/server mode).  After you run it, it is
    # probably a good idea to release your handle on the db and
    # re-initialize KirbyBase, as this method changes the table structure.
    #
    # *tablename*:: Symbol of table name.
    # *col_name*:: Symbol of column name to add.
    #
    def drop_table_column(tablename, col_name)
        raise "Do not execute this method from the server!!!" if server?

        raise "Invalid table name!" unless table_exists?(tablename)

        raise 'Invalid column name: ' % col_name unless \
         @table_hash[tablename].field_names.include?(col_name)

        raise "Cannot drop :recno column!" if col_name == :recno

        @engine.drop_column(@table_hash[tablename], col_name)

        # Need to reinitialize the table instance and associated indexes.
        @engine.remove_recno_index(tablename)
        @engine.remove_indexes(tablename)
        @table_hash.delete(tablename)
        @table_hash[tablename] = \
         KBTable.create_called_from_database_instance(self, tablename,
          File.join(@path, tablename.to_s + @ext))
    end
end

#---------------------------------------------------------------------------
# KBEngine
#---------------------------------------------------------------------------
class KBEngine
    include DRb::DRbUndumped

    EN_STR = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' + \
     '0123456789.+-,$:|&;_ '
    EN_STR_LEN = EN_STR.length
    EN_KEY1 = ")2VER8GE\"87-E\n"           #*** DO NOT CHANGE ***
    EN_KEY = EN_KEY1.unpack("u")[0]
    EN_KEY_LEN = EN_KEY.length

    # Make constructor private.
    private_class_method :new

    #-----------------------------------------------------------------------
    # KBEngine.create_called_from_database_instance
    #-----------------------------------------------------------------------
    def KBEngine.create_called_from_database_instance(db)
        return new(db)
    end

    #-----------------------------------------------------------------------
    # initialize
    #-----------------------------------------------------------------------
    def initialize(db)
        @db = db
        @recno_indexes = {}
        @indexes = {}

        # This hash will hold the table locks if in client/server mode.
        @mutex_hash = {} if @db.server?
    end

    #-----------------------------------------------------------------------
    # init_recno_index
    #-----------------------------------------------------------------------
    def init_recno_index(table)
        return if recno_index_exists?(table)

        with_write_locked_table(table) do |fptr|
            @recno_indexes[table.name] = KBRecnoIndex.new(table)
            @recno_indexes[table.name].rebuild(fptr)
        end
    end

    #-----------------------------------------------------------------------
    # rebuild_recno_index
    #-----------------------------------------------------------------------
    def rebuild_recno_index(table)
        with_write_locked_table(table) do |fptr|
            @recno_indexes[table.name].rebuild(fptr)
        end
    end

    #-----------------------------------------------------------------------
    # remove_recno_index
    #-----------------------------------------------------------------------
    def remove_recno_index(tablename)
        @recno_indexes.delete(tablename)
    end

    #-----------------------------------------------------------------------
    # recno_index_exists?
    #-----------------------------------------------------------------------
    def recno_index_exists?(table)
        @recno_indexes.include?(table.name)
    end

    #-----------------------------------------------------------------------
    # init_index
    #-----------------------------------------------------------------------
    def init_index(table, index_fields)
        return if index_exists?(table, index_fields)

        with_write_locked_table(table) do |fptr|
            @indexes["#{table.name}_#{index_fields.join('_')}".to_sym] = \
             KBIndex.new(table, index_fields)
            @indexes["#{table.name}_#{index_fields.join('_')}".to_sym
             ].rebuild(fptr)
        end
    end

    #-----------------------------------------------------------------------
    # rebuild_index
    #-----------------------------------------------------------------------
    def rebuild_index(table, index_fields)
        with_write_locked_table(table) do |fptr|
            @indexes["#{table.name}_#{index_fields.join('_')}".to_sym
             ].rebuild(fptr)
        end
    end

    #-----------------------------------------------------------------------
    # remove_indexes
    #-----------------------------------------------------------------------
    def remove_indexes(tablename)
        re_table_name = Regexp.new(tablename.to_s)
        @indexes.delete_if { |k,v| k.to_s =~ re_table_name }
    end

    #-----------------------------------------------------------------------
    # add_to_indexes
    #-----------------------------------------------------------------------
    def add_to_indexes(table, rec, fpos)
        @recno_indexes[table.name].add_index_rec(rec.first, fpos)

        re_table_name = Regexp.new(table.name.to_s)
        @indexes.each_pair do |key, index|
            index.add_index_rec(rec) if key.to_s =~ re_table_name
        end
    end

    #-----------------------------------------------------------------------
    # delete_from_indexes
    #-----------------------------------------------------------------------
    def delete_from_indexes(table, rec, fpos)
        @recno_indexes[table.name].delete_index_rec(rec.recno)

        re_table_name = Regexp.new(table.name.to_s)
        @indexes.each_pair do |key, index|
            index.delete_index_rec(rec.recno) if key.to_s =~ re_table_name
        end
    end

    #-----------------------------------------------------------------------
    # index_exists?
    #-----------------------------------------------------------------------
    def index_exists?(table, index_fields)
        @indexes.include?("#{table.name}_#{index_fields.join('_')}".to_sym)
    end

    #-----------------------------------------------------------------------
    # update_recno_index
    #-----------------------------------------------------------------------
    def update_recno_index(table, recno, fpos)
        @recno_indexes[table.name].update_index_rec(recno, fpos)
    end

    #-----------------------------------------------------------------------
    # update_to_indexes
    #-----------------------------------------------------------------------
    def update_to_indexes(table, rec)
        re_table_name = Regexp.new(table.name.to_s)
        @indexes.each_pair do |key, index|
            index.update_index_rec(rec) if key.to_s =~ re_table_name
        end
    end

    #-----------------------------------------------------------------------
    # get_index
    #-----------------------------------------------------------------------
    def get_index(table, index_name)
        return @indexes["#{table.name}_#{index_name}".to_sym].get_idx
    end

    #-----------------------------------------------------------------------
    # get_recno_index
    #-----------------------------------------------------------------------
    def get_recno_index(table)
        return @recno_indexes[table.name].get_idx
    end

    #-----------------------------------------------------------------------
    # table_exists?
    #-----------------------------------------------------------------------
    def table_exists?(tablename)
        return File.exists?(File.join(@db.path, tablename.to_s + @db.ext))
    end

    #-----------------------------------------------------------------------
    # tables
    #-----------------------------------------------------------------------
    def tables
        list = []
        Dir.foreach(@db.path) { |filename|
            list << File.basename(filename, '.*').to_sym if \
             File.extname(filename) == @db.ext
        }
        return list
    end

    #-----------------------------------------------------------------------
    # build_header_field_string
    #-----------------------------------------------------------------------
    def build_header_field_string(field_name_def, field_type_def)
        # Put field name at start of string definition.
        temp_field_def = field_name_def.to_s + ':'

        # if field type is a hash, that means that it is not just a
        # simple field.  Either is is being used in an index, it is a
        # Lookup field, it is a Link_many field, or it is a Calculated
        # field.  This next bit of code is to piece together a proper
        # string so that it can be written out to the header rec.
        if field_type_def.is_a?(Hash)
            raise 'Missing :DataType key in field type hash!' unless \
             field_type_def.has_key?(:DataType)

            temp_type = field_type_def[:DataType]

            raise 'Invalid field type: %s' % temp_type unless \
             KBTable.valid_field_type?(temp_type)

            temp_field_def += field_type_def[:DataType].to_s

            if field_type_def.has_key?(:Key)
                temp_field_def += ':Key->true'
            end
            if field_type_def.has_key?(:Index)
                raise 'Invalid field type for index: %s' % temp_type \
                 unless KBTable.valid_index_type?(temp_type)

                temp_field_def += ':Index->' + field_type_def[:Index].to_s
            end
            if field_type_def.has_key?(:Lookup)
                if field_type_def[:Lookup].is_a?(Array)
                    temp_field_def += \
                     ':Lookup->%s.%s' % field_type_def[:Lookup]
                else
                    tbl = @db.get_table(field_type_def[:Lookup])
                    temp_field_def += \
                     ':Lookup->%s.%s' % [field_type_def[:Lookup],
                     tbl.lookup_key]
                end
            elsif field_type_def.has_key?(:Link_many)
                raise 'Field type for Link_many field must be :ResultSet' \
                 unless temp_type == :ResultSet
                temp_field_def += \
                 ':Link_many->%s=%s.%s' % field_type_def[:Link_many]
            elsif field_type_def.has_key?(:Calculated)
                temp_field_def += \
                 ':Calculated->%s' % field_type_def[:Calculated]
            end
        else
            if KBTable.valid_field_type?(field_type_def)
                temp_field_def += field_type_def.to_s
            elsif @db.table_exists?(field_type_def)
                tbl = @db.get_table(field_type_def)
                temp_field_def += \
                 '%s:Lookup->%s.%s' % [tbl.field_types[
                 tbl.field_names.index(tbl.lookup_key)], field_type_def,
                 tbl.lookup_key]
            else
                raise 'Invalid field type: %s' % field_type_def
            end
        end
        return temp_field_def
    end

    #-----------------------------------------------------------------------
    # new_table
    #-----------------------------------------------------------------------
    #++
    # Create physical file holding table. This table should not be directly
    # called in your application, but only called by #create_table.
    #
    def new_table(name, field_defs, encrypt, record_class)
        # Can't create a table that already exists!
        raise "Table already exists!" if table_exists?(name)

        raise 'Must have a field type for each field name' \
         unless field_defs.size.remainder(2) == 0
        temp_field_defs = []
        (0...field_defs.size).step(2) do |x|
            temp_field_defs << build_header_field_string(field_defs[x],
             field_defs[x+1])
        end

        # Header rec consists of last record no. used, delete count, and
        # all field names/types.  Here, I am inserting the 'recno' field
        # at the beginning of the fields.
        header_rec = ['000000', '000000', record_class, 'recno:Integer',
         temp_field_defs].join('|')

        header_rec = 'Z' + encrypt_str(header_rec) if encrypt

        begin
            fptr = open(File.join(@db.path, name.to_s + @db.ext), 'w')
            fptr.write(header_rec + "\n")
        ensure
            fptr.close
        end
    end

    #-----------------------------------------------------------------------
    # delete_table
    #-----------------------------------------------------------------------
    def delete_table(tablename)
        with_write_lock(tablename) do
            remove_indexes(tablename)
            remove_recno_index(tablename)
            File.delete(File.join(@db.path, tablename.to_s + @db.ext))
            return true
        end
    end

    #----------------------------------------------------------------------
    # get_total_recs
    #----------------------------------------------------------------------
    def get_total_recs(table)
        return get_recs(table).size
    end

    #-----------------------------------------------------------------------
    # get_header_vars
    #-----------------------------------------------------------------------
    def get_header_vars(table)
        with_table(table) do |fptr|
            line = get_header_record(table, fptr)

            last_rec_no, del_ctr, record_class, *flds = line.split('|')
            field_names = flds.collect { |x| x.split(':')[0].to_sym }
            field_types = flds.collect { |x| x.split(':')[1].to_sym }
            field_indexes = [nil] * field_names.size
            field_extras = [nil] * field_names.size

            flds.each_with_index do |x,i|
                field_extras[i] = {}
                if x.split(':').size > 2
                    x.split(':')[2..-1].each do |y|
                        if y =~ /Index/
                            field_indexes[i] = y
                        else
                            field_extras[i][y.split('->')[0]] = \
                             y.split('->')[1]
                        end
                    end
                end
            end
            return [table.encrypted?, last_rec_no.to_i, del_ctr.to_i,
             record_class, field_names, field_types, field_indexes,
             field_extras]
        end
    end

    #-----------------------------------------------------------------------
    # get_recs
    #-----------------------------------------------------------------------
    def get_recs(table)
        encrypted = table.encrypted?
        recs = []

        with_table(table) do |fptr|
            begin
                # Skip header rec.
                fptr.readline

                # Loop through table.
                while true
                    # Record current position in table.  Then read first
                    # detail record.
                    fpos = fptr.tell
                    line = fptr.readline
                    line.chomp!
                    line_length = line.length

                    line = unencrypt_str(line) if encrypted
                    line.strip!

                    # If blank line (i.e. 'deleted'), skip it.
                    next if line == ''

                    # Split the line up into fields.
                    rec = line.split('|', -1)
                    rec << fpos << line_length
                    recs << rec
                end
            # Here's how we break out of the loop...
            rescue EOFError
            end
            return recs
        end
    end

    #-----------------------------------------------------------------------
    # get_recs_by_recno
    #-----------------------------------------------------------------------
    def get_recs_by_recno(table, recnos)
        encrypted = table.encrypted?
        recs = []
        recno_idx = get_recno_index(table)

        with_table(table) do |fptr|
            # Skip header rec.
            fptr.readline

            recnos.collect { |r| [recno_idx[r], r]
             }.sort.each do |r|
                fptr.seek(r[0])
                line = fptr.readline
                line.chomp!
                line_length = line.length

                line = unencrypt_str(line) if encrypted
                line.strip!

                # If blank line (i.e. 'deleted'), skip it.
                next if line == ''

                # Split the line up into fields.
                rec = line.split('|', -1)
                raise "Index Corrupt!" unless rec[0].to_i == r[1]
                rec << r[0] << line_length
                recs << rec
            end
            return recs
        end
    end

    #-----------------------------------------------------------------------
    # get_rec_by_recno
    #-----------------------------------------------------------------------
    def get_rec_by_recno(table, recno)
        encrypted = table.encrypted?
        recno_idx = get_recno_index(table)

        return nil unless recno_idx.has_key?(recno)

        with_table(table) do |fptr|
            fptr.seek(recno_idx[recno])
            line = fptr.readline
            line.chomp!
            line_length = line.length

            line = unencrypt_str(line) if encrypted
            line.strip!

            return nil if line == ''

            # Split the line up into fields.
            rec = line.split('|', -1)

            raise "Index Corrupt!" unless rec[0].to_i == recno
            rec << recno_idx[recno] << line_length
            return rec
        end
    end

    #-----------------------------------------------------------------------
    # insert_record
    #-----------------------------------------------------------------------
    def insert_record(table, rec)
        with_write_locked_table(table) do |fptr|
            # Auto-increment the record number field.
            rec_no = incr_rec_no_ctr(table, fptr)

            # Insert the newly created record number value at the beginning
            # of the field values.
            rec[0] = rec_no

            fptr.seek(0, IO::SEEK_END)
            fpos = fptr.tell

            write_record(table, fptr, 'end', rec.join('|'))

            add_to_indexes(table, rec, fpos)

            # Return the record number of the newly created record.
            return rec_no
        end
    end

    #-----------------------------------------------------------------------
    # update_records
    #-----------------------------------------------------------------------
    def update_records(table, recs)
        with_write_locked_table(table) do |fptr|
            recs.each do |rec|
                line = rec[:rec].join('|')

                # This doesn't actually 'delete' the line, it just
                # makes it all spaces.  That way, if the updated
                # record is the same or less length than the old
                # record, we can write the record back into the
                # same spot.  If the updated record is greater than
                # the old record, we will leave the now spaced-out
                # line and write the updated record at the end of
                # the file.
                write_record(table, fptr, rec[:fpos],
                 ' ' * rec[:line_length])
                if line.length > rec[:line_length]
                    fptr.seek(0, IO::SEEK_END)
                    new_fpos = fptr.tell
                    write_record(table, fptr, 'end', line)
                    incr_del_ctr(table, fptr)

                    update_recno_index(table, rec[:rec].first, new_fpos)
                else
                    write_record(table, fptr, rec[:fpos], line)
                end
                update_to_indexes(table, rec[:rec])
            end
            # Return the number of records updated.
            return recs.size
        end
    end

    #-----------------------------------------------------------------------
    # delete_records
    #-----------------------------------------------------------------------
    def delete_records(table, recs)
        with_write_locked_table(table) do |fptr|
            recs.each do |rec|
                # Go to offset within the file where the record is and
                # replace it with all spaces.
                write_record(table, fptr, rec.fpos, ' ' * rec.line_length)
                incr_del_ctr(table, fptr)

                delete_from_indexes(table, rec, rec.fpos)
            end

            # Return the number of records deleted.
            return recs.size
        end
    end

    #-----------------------------------------------------------------------
    # add_column
    #-----------------------------------------------------------------------
    def add_column(table, col_name, col_type, after)
        temp_field_def = build_header_field_string(col_name, col_type)

        if after.nil?
            insert_after = -1
        else
            if table.field_names.last == after
                insert_after = -1
            else
                insert_after = table.field_names.index(after)+1
            end
        end

        with_write_lock(table.name) do
            fptr = open(table.filename, 'r')
            new_fptr = open(table.filename+'temp', 'w')

            line = fptr.readline.chomp

            if line[0..0] == 'Z'
                header_rec = unencrypt_str(line[1..-1]).split('|')
                if insert_after == -1
                    header_rec.insert(insert_after, temp_field_def)
                else
                    header_rec.insert(insert_after+3, temp_field_def)
                end
                new_fptr.write('Z' + encrypt_str(header_rec.join('|')) +
                 "\n")
            else
                header_rec = line.split('|')
                if insert_after == -1
                    header_rec.insert(insert_after, temp_field_def)
                else
                    header_rec.insert(insert_after+3, temp_field_def)
                end
                new_fptr.write(header_rec.join('|') + "\n")
            end

            begin
                while true
                    line = fptr.readline.chomp

                    if table.encrypted?
                        temp_line = unencrypt_str(line)
                    else
                        temp_line = line
                    end

                    rec = temp_line.split('|')
                    rec.insert(insert_after, '')

                    if table.encrypted?
                        new_fptr.write(encrypt_str(rec.join('|')) + "\n")
                    else
                        new_fptr.write(rec.join('|') + "\n")
                    end
                end
            # Here's how we break out of the loop...
            rescue EOFError
            end

            # Close the table and release the write lock.
            fptr.close
            new_fptr.close
            File.delete(table.filename)
            FileUtils.mv(table.filename+'temp', table.filename)
        end
    end

    #-----------------------------------------------------------------------
    # drop_column
    #-----------------------------------------------------------------------
    def drop_column(table, col_name)
        col_index = table.field_names.index(col_name)
        with_write_lock(table.name) do
            fptr = open(table.filename, 'r')
            new_fptr = open(table.filename+'temp', 'w')

            line = fptr.readline.chomp

            if line[0..0] == 'Z'
                header_rec = unencrypt_str(line[1..-1]).split('|')
                header_rec.delete_at(col_index+3)
                new_fptr.write('Z' + encrypt_str(header_rec.join('|')) +
                 "\n")
            else
                header_rec = line.split('|')
                header_rec.delete_at(col_index+3)
                new_fptr.write(header_rec.join('|') + "\n")
            end

            begin
                while true
                    line = fptr.readline.chomp

                    if table.encrypted?
                        temp_line = unencrypt_str(line)
                    else
                        temp_line = line
                    end

                    rec = temp_line.split('|')
                    rec.delete_at(col_index)

                    if table.encrypted?
                        new_fptr.write(encrypt_str(rec.join('|')) + "\n")
                    else
                        new_fptr.write(rec.join('|') + "\n")
                    end
                end
            # Here's how we break out of the loop...
            rescue EOFError
            end

            # Close the table and release the write lock.
            fptr.close
            new_fptr.close
            File.delete(table.filename)
            FileUtils.mv(table.filename+'temp', table.filename)
        end
    end

    #-----------------------------------------------------------------------
    # pack_table
    #-----------------------------------------------------------------------
    def pack_table(table)
        with_write_lock(table.name) do
            fptr = open(table.filename, 'r')
            new_fptr = open(table.filename+'temp', 'w')

            line = fptr.readline.chomp
            # Reset the delete counter in the header rec to 0.
            if line[0..0] == 'Z'
                header_rec = unencrypt_str(line[1..-1]).split('|')
                header_rec[1] = '000000'
                new_fptr.write('Z' + encrypt_str(header_rec.join('|')) +
                 "\n")
            else
                header_rec = line.split('|')
                header_rec[1] = '000000'
                new_fptr.write(header_rec.join('|') + "\n")
            end

            lines_deleted = 0

            begin
                while true
                    line = fptr.readline

                    if table.encrypted?
                        temp_line = unencrypt_str(line)
                    else
                        temp_line = line
                    end

                    if temp_line.strip == ''
                        lines_deleted += 1
                    else
                        new_fptr.write(line)
                    end
                end
            # Here's how we break out of the loop...
            rescue EOFError
            end

            # Close the table and release the write lock.
            fptr.close
            new_fptr.close
            File.delete(table.filename)
            FileUtils.mv(table.filename+'temp', table.filename)

            # Return the number of deleted records that were removed.
            return lines_deleted
        end
    end

    #-----------------------------------------------------------------------
    # get_memo
    #-----------------------------------------------------------------------
    def get_memo(filepath)
        begin
            f = File.new(filepath)
            return f.readlines
        ensure
            f.close
        end
    end

    #-----------------------------------------------------------------------
    # get_blob
    #-----------------------------------------------------------------------
    def get_blob(filepath)
        begin
            f = File.new(filepath, 'rb')
            return f.read
        ensure
            f.close
        end
    end


    #-----------------------------------------------------------------------
    # PRIVATE METHODS
    #-----------------------------------------------------------------------
    private

    #-----------------------------------------------------------------------
    # with_table
    #-----------------------------------------------------------------------
    def with_table(table, access='r')
        begin
            yield fptr = open(table.filename, access)
        ensure
            fptr.close
        end
    end

    #-----------------------------------------------------------------------
    # with_write_lock
    #-----------------------------------------------------------------------
    def with_write_lock(tablename)
        begin
            write_lock(tablename) if @db.server?
            yield
        ensure
            write_unlock(tablename) if @db.server?
        end
    end

    #-----------------------------------------------------------------------
    # with_write_locked_table
    #-----------------------------------------------------------------------
    def with_write_locked_table(table, access='r+')
        begin
            write_lock(table.name) if @db.server?
            yield fptr = open(table.filename, access)
        ensure
            fptr.close
            write_unlock(table.name) if @db.server?
        end
    end

    #-----------------------------------------------------------------------
    # write_lock
    #-----------------------------------------------------------------------
    def write_lock(tablename)
        # Unless an key already exists in the hash holding mutex records
        # for this table, create a write key for this table in the mutex
        # hash.  Then, place a lock on that mutex.
        @mutex_hash[tablename] = Mutex.new unless (
         @mutex_hash.has_key?(tablename))
        @mutex_hash[tablename].lock

        return true
    end

    #----------------------------------------------------------------------
    # write_unlock
    #----------------------------------------------------------------------
    def write_unlock(tablename)
        # Unlock the write mutex for this table.
        @mutex_hash[tablename].unlock

        return true
    end

    #----------------------------------------------------------------------
    # write_record
    #----------------------------------------------------------------------
    def write_record(table, fptr, pos, record)
        if table.encrypted?
            temp_rec = encrypt_str(record)
        else
            temp_rec = record
        end

        # If record is to be appended, go to end of table and write
        # record, adding newline character.
        if pos == 'end'
            fptr.seek(0, IO::SEEK_END)
            fptr.write(temp_rec + "\n")
        else
            # Otherwise, overwrite another record (that's why we don't
            # add the newline character).
            fptr.seek(pos)
            fptr.write(temp_rec)
        end
    end

    #----------------------------------------------------------------------
    # write_header_record
    #----------------------------------------------------------------------
    def write_header_record(table, fptr, record)
        fptr.seek(0)

        if table.encrypted?
            fptr.write('Z' + encrypt_str(record) + "\n")
        else
            fptr.write(record + "\n")
        end
    end

    #----------------------------------------------------------------------
    # get_header_record
    #----------------------------------------------------------------------
    def get_header_record(table, fptr)
        fptr.seek(0)

        if table.encrypted?
            return unencrypt_str(fptr.readline[1..-1].chomp)
        else
            return fptr.readline.chomp
        end
    end

    #-----------------------------------------------------------------------
    # reset_rec_no_ctr
    #-----------------------------------------------------------------------
    def reset_rec_no_ctr(table, fptr)
        last_rec_no, rest_of_line = get_header_record(table, fptr).split(
         '|', 2)
        write_header_record(table, fptr, ['%06d' % 0, rest_of_line].join(
         '|'))
        return true
    end

    #-----------------------------------------------------------------------
    # incr_rec_no_ctr
    #-----------------------------------------------------------------------
    def incr_rec_no_ctr(table, fptr)
        last_rec_no, rest_of_line = get_header_record(table, fptr).split(
         '|', 2)
        last_rec_no = last_rec_no.to_i + 1

        write_header_record(table, fptr, ['%06d' % last_rec_no,
         rest_of_line].join('|'))

        # Return the new recno.
        return last_rec_no
    end

    #-----------------------------------------------------------------------
    # incr_del_ctr
    #-----------------------------------------------------------------------
    def incr_del_ctr(table, fptr)
        last_rec_no, del_ctr, rest_of_line = get_header_record(table,
         fptr).split('|', 3)
        del_ctr = del_ctr.to_i + 1

        write_header_record(table, fptr, [last_rec_no, '%06d' % del_ctr,
         rest_of_line].join('|'))

        return true
    end

    #-----------------------------------------------------------------------
    # encrypt_str
    #-----------------------------------------------------------------------
    def encrypt_str(s)
        # Returns an encrypted string, using the Vignere Cipher.

        new_str = ''
        i_key = -1
        s.each_byte do |c|
            if i_key < EN_KEY_LEN - 1
                i_key += 1
            else
                i_key = 0
            end

            if EN_STR.index(c.chr).nil?
                new_str << c.chr
                next
            end

            i_from_str = EN_STR.index(EN_KEY[i_key]) + EN_STR.index(c.chr)
            i_from_str = i_from_str - EN_STR_LEN if i_from_str >= EN_STR_LEN
            new_str << EN_STR[i_from_str]
        end
        return new_str
    end

    #-----------------------------------------------------------------------
    # unencrypt_str
    #-----------------------------------------------------------------------
    def unencrypt_str(s)
        # Returns an unencrypted string, using the Vignere Cipher.

        new_str = ''
        i_key = -1
        s.each_byte do |c|
            if i_key < EN_KEY_LEN - 1
                i_key += 1
            else
                i_key = 0
            end

            if EN_STR.index(c.chr).nil?
                new_str << c.chr
                next
            end

            i_from_str = EN_STR.index(c.chr) - EN_STR.index(EN_KEY[i_key])
            i_from_str = i_from_str + EN_STR_LEN if i_from_str < 0
            new_str << EN_STR[i_from_str]
        end
        return new_str
    end
end


#---------------------------------------------------------------------------
# KBTypeConversionsMixin
#---------------------------------------------------------------------------
module KBTypeConversionsMixin
    UNENCODE_RE = /&(?:amp|linefeed|carriage_return|substitute|pipe);/

    #-----------------------------------------------------------------------
    # convert_to
    #-----------------------------------------------------------------------
    def convert_to(data_type, s)
        return nil if s.empty? or s.nil?

        case data_type
        when :String
            if s =~ UNENCODE_RE
                return s.gsub('&linefeed;', "\n").gsub('&carriage_return;',
                 "\r").gsub('&substitute;', "\032").gsub('&pipe;', "|"
                 ).gsub('&amp;', "&")
            else
                return s
            end
        when :Integer
            return s.to_i
        when :Float
            return s.to_f
        when :Boolean
            if ['false', 'False', nil, false].include?(s)
                return false
            else
                return true
            end
        when :Date
            return Date.parse(s)
        when :Time
            return Time.parse(s)
        when :DateTime
            return DateTime.parse(s)
        when :YAML
            # This code is here in case the YAML field is the last
            # field in the record.  Because YAML normall defines a
            # nil value as "--- ", but KirbyBase strips trailing
            # spaces off the end of the record, so if this is the
            # last field in the record, KirbyBase will strip the
            # trailing space off and make it "---".  When KirbyBase
            # attempts to convert this value back using to_yaml,
            # you get an exception.
            if s == "---"
                return nil
            elsif s =~ UNENCODE_RE
                y = s.gsub('&linefeed;', "\n").gsub('&carriage_return;',
                 "\r").gsub('&substitute;', "\032").gsub('&pipe;', "|"
                 ).gsub('&amp;', "&")
                return YAML.load(y)
            else
                return YAML.load(s)
            end
        when :Memo
            return KBMemo.new(@tbl.db, s)
        when :Blob
            return KBBlob.new(@tbl.db, s)
        else
            raise "Invalid field type: %s" % data_type
        end
    end
end


#---------------------------------------------------------------------------
# KBTable
#---------------------------------------------------------------------------
class KBTable
    include DRb::DRbUndumped

    # Make constructor private.  KBTable instances should only be created
    # from KirbyBase#get_table.
    private_class_method :new

    VALID_FIELD_TYPES = [:String, :Integer, :Float, :Boolean, :Date, :Time,
                         :DateTime, :Memo, :ResultSet, :YAML]

    VALID_INDEX_TYPES = [:String, :Integer, :Float, :Boolean, :Date, :Time,
                         :DateTime]

    # Regular expression used to determine if field needs to be
    # encoded.
    ENCODE_RE = /&|\n|\r|\032|\|/

    attr_reader :filename, :name, :table_class, :db, :lookup_key

    #-----------------------------------------------------------------------
    # KBTable.valid_field_type
    #-----------------------------------------------------------------------
    #++
    # Return true if valid field type.
    #
    # *field_type*:: Symbol specifying field type.
    #
    def KBTable.valid_field_type?(field_type)
        VALID_FIELD_TYPES.include?(field_type)
    end

    #-----------------------------------------------------------------------
    # KBTable.valid_index_type
    #-----------------------------------------------------------------------
    #++
    # Return true if valid index type.
    #
    # *field_type*:: Symbol specifying field type.
    #
    def KBTable.valid_index_type?(field_type)
        VALID_INDEX_TYPES.include?(field_type)
    end

    #-----------------------------------------------------------------------
    # create_called_from_database_instance
    #-----------------------------------------------------------------------
    #++
    # Return a new instance of KBTable.  Should never be called directly by
    # your application.  Should only be called from KirbyBase#get_table.
    #
    def KBTable.create_called_from_database_instance(db, name, filename)
        return new(db, name, filename)
    end

    #-----------------------------------------------------------------------
    # initialize
    #-----------------------------------------------------------------------
    #++
    # This has been declared private so user's cannot create new instances
    # of KBTable from their application.  A user gets a handle to a KBTable
    # instance by calling KirbyBase#get_table for an existing table or
    # KirbyBase.create_table for a new table.
    #
    def initialize(db, name, filename)
        @db = db
        @name = name
        @filename = filename
        @encrypted = false
        @lookup_key = :recno

        # Alias delete_all to clear method.
        alias delete_all clear

        update_header_vars
        create_indexes
        create_table_class unless @db.server?
    end

    #-----------------------------------------------------------------------
    # encrypted?
    #-----------------------------------------------------------------------
    #++
    # Returns true if table is encrypted.
    #
    def encrypted?
        if @encrypted
            return true
        else
            return false
        end
    end

    #-----------------------------------------------------------------------
    # field_names
    #-----------------------------------------------------------------------
    #++
    # Return array containing table field names.
    #
    def field_names
        return @field_names
    end

    #-----------------------------------------------------------------------
    # field_types
    #-----------------------------------------------------------------------
    #++
    # Return array containing table field types.
    #
    def field_types
        return @field_types
    end

    #-----------------------------------------------------------------------
    # field_extras
    #-----------------------------------------------------------------------
    #++
    # Return array containing table field extras.
    #
    def field_extras
        return @field_extras
    end

    #-----------------------------------------------------------------------
    # field_indexes
    #-----------------------------------------------------------------------
    #++
    # Return array containing table field indexes.
    #
    def field_indexes
        return @field_indexes
    end

    #-----------------------------------------------------------------------
    # insert
    #-----------------------------------------------------------------------
    #++
    # Insert a new record into a table, return unique record number.
    #
    # *data*:: Array, Hash, Struct instance containing field values of
    #          new record.
    # *insert_proc*:: Proc instance containing insert code. This and the
    #                 data parameter are mutually exclusive.
    #
    def insert(*data, &insert_proc)
        raise 'Cannot specify both a hash/array/struct and a ' + \
         'proc for method #insert!' unless data.empty? or insert_proc.nil?

        raise 'Must specify either hash/array/struct or insert ' + \
         'proc for method #insert!' if data.empty? and insert_proc.nil?

        # Update the header variables.
        update_header_vars

        # Convert input, which could be an array, a hash, or a Struct
        # into a common format (i.e. hash).
        if data.empty?
            input_rec = convert_input_data(insert_proc)
        else
            input_rec = convert_input_data(data)
        end

        # Check the field values to make sure they are proper types.
        validate_input(input_rec)

        return @db.engine.insert_record(self, @field_names.zip(@field_types
         ).collect do |fn, ft|
            convert_to_string(ft, input_rec.fetch(fn, ''))
        end)
    end

    #-----------------------------------------------------------------------
    # update_all
    #-----------------------------------------------------------------------
    #++
    # Return array of records (Structs) to be updated, in this case all
    # records.
    #
    # *updates*:: Hash or Struct containing updates.
    #
    def update_all(*updates)
        update(*updates) { true }
    end

    #-----------------------------------------------------------------------
    # update
    #-----------------------------------------------------------------------
    #++
    # Return array of records (Structs) to be updated based on select cond.
    #
    # *updates*:: Hash or Struct containing updates.
    # *select_cond*:: Proc containing code to select records to update.
    #
    def update(*updates, &select_cond)
        raise ArgumentError, "Must specify select condition code " + \
         "block.  To update all records, use #update_all instead." if \
         select_cond.nil?

        # Update the header variables.
        update_header_vars

        # Get all records that match the selection criteria and
        # return them in an array.
        result_set = get_matches(:update, @field_names, select_cond)

        return result_set if updates.empty?

        set(result_set, updates)
    end

    #-----------------------------------------------------------------------
    # []=
    #-----------------------------------------------------------------------
    #++
    # Update record whose recno field equals index.
    #
    # *index*:: Integer specifying recno you wish to select.
    # *updates*:: Hash, Struct, or Array containing updates.
    #
    def []=(index, updates)
        return update(updates) { |r| r.recno == index }
    end

    #-----------------------------------------------------------------------
    # set
    #-----------------------------------------------------------------------
    #++
    # Set fields of records to updated values.  Returns number of records
    # updated.
    #
    # *recs*:: Array of records (Structs) that will be updated.
    # *data*:: Hash, Struct, Proc containing updates.
    #
    def set(recs, data)
        # Convert updates, which could be an array, a hash, or a Struct
        # into a common format (i.e. hash).
        update_rec = convert_input_data(data)

        # Make sure all of the fields of the update rec are of the proper
        # type.
        validate_input(update_rec)

        updated_recs = []

        # For each one of the recs that matched the update query, apply the
        # updates to it and write it back to the database table.
        recs.each do |rec|
            updated_rec = {}
            updated_rec[:rec] = \
             @field_names.zip(@field_types).collect do |fn, ft|
                convert_to_string(ft, update_rec.fetch(fn, rec.send(fn)))
            end
            updated_rec[:fpos] = rec.fpos
            updated_rec[:line_length] = rec.line_length
            updated_recs << updated_rec
        end
        @db.engine.update_records(self, updated_recs)

        # Return the number of records updated.
        return recs.size
    end

    #-----------------------------------------------------------------------
    # delete
    #-----------------------------------------------------------------------
    #++
    # Delete records from table and return # deleted.
    #
    # *select_cond*:: Proc containing code to select records.
    #
    def delete(&select_cond)
        raise ArgumentError, 'Must specify select condition code ' + \
         'block.  To delete all records, use #clear instead.' if \
         select_cond.nil?

        # Get all records that match the selection criteria and
        # return them in an array.
        result_set = get_matches(:delete, [:recno], select_cond)

        @db.engine.delete_records(self, result_set)

        # Return the number of records deleted.
        return result_set.size
    end

    #-----------------------------------------------------------------------
    # clear
    #-----------------------------------------------------------------------
    #++
    # Delete all records from table. You can also use #delete_all.
    #
    # *reset_recno_ctr*:: true/false specifying whether recno counter should
    #                     be reset to 0.
    #
    def clear(reset_recno_ctr=true)
        delete { true }
        pack

        @db.engine.reset_recno_ctr if reset_recno_ctr
    end

    #-----------------------------------------------------------------------
    # []
    #-----------------------------------------------------------------------
    #++
    # Return the record(s) whose recno field is included in index.
    #
    # *index*:: Array of Integer(s) specifying recno(s) you wish to select.
    #
    def [](*index)
        return nil if index[0].nil?

        return get_match_by_recno(:select, @field_names, index[0]) if \
         index.size == 1

        recs = select_by_recno_index(*@field_names) { |r|
            index.includes?(r.recno)
        }

        return recs
    end

    #-----------------------------------------------------------------------
    # select
    #-----------------------------------------------------------------------
    #++
    # Return array of records (Structs) matching select conditions.
    #
    # *filter*:: List of field names (Symbols) to include in result set.
    # *select_cond*:: Proc containing select code.
    #
    def select(*filter, &select_cond)
        # Declare these variables before the code block so they don't go
        # after the code block is done.
        result_set = []

        # Validate that all names in filter are valid field names.
        validate_filter(filter)

        filter = @field_names if filter.empty?

        # Get all records that match the selection criteria and
        # return them in an array of Struct instances.
        return get_matches(:select, filter, select_cond)
    end

    #-----------------------------------------------------------------------
    # select_by_recno_index
    #-----------------------------------------------------------------------
    #++
    # Return array of records (Structs) matching select conditions.  Select
    # condition block should not contain references to any table column
    # except :recno.  If you need to select by other table columns than just
    # :recno, use #select instead.
    #
    # *filter*:: List of field names (Symbols) to include in result set.
    # *select_cond*:: Proc containing select code.
    #
    def select_by_recno_index(*filter, &select_cond)
        # Declare these variables before the code block so they don't go
        # after the code block is done.
        result_set = []

        # Validate that all names in filter are valid field names.
        validate_filter(filter)

        filter = @field_names if filter.empty?

        # Get all records that match the selection criteria and
        # return them in an array of Struct instances.
        return get_matches_by_recno_index(:select, filter, select_cond)
    end

    #-----------------------------------------------------------------------
    # pack
    #-----------------------------------------------------------------------
    #++
    # Remove blank records from table, return total removed.
    #
    def pack
        lines_deleted = @db.engine.pack_table(self)

        update_header_vars

        @db.engine.remove_recno_index(@name)
        @db.engine.remove_indexes(@name)

        create_indexes
        create_table_class unless @db.server?

        return lines_deleted
    end

    #-----------------------------------------------------------------------
    # total_recs
    #-----------------------------------------------------------------------
    #++
    # Return total number of undeleted (blank) records in table.
    #
    def total_recs
        return @db.engine.get_total_recs(self)
    end

    #-----------------------------------------------------------------------
    # import_csv
    #-----------------------------------------------------------------------
    #++
    # Import csv file into table.
    #
    # *csv_filename*:: filename of csv file to import.
    #
    def import_csv(csv_filename)
        tbl_rec = @table_class.new(self)

        CSV.open(csv_filename, 'r') do |row|
            tbl_rec.populate([nil] + row)
            insert(tbl_rec)
        end
    end

    #-----------------------------------------------------------------------
    # PRIVATE METHODS
    #-----------------------------------------------------------------------
    private

    #-----------------------------------------------------------------------
    # create_indexes
    #-----------------------------------------------------------------------
    def create_indexes
        # Create the recno index.  A recno index always gets created even if
        # there are no user-defined indexes for the table.
        @db.engine.init_recno_index(self)

        # There can be up to 5 different indexes on a table.  Any of these
        # indexes can be single or compound.
        ['Index->1', 'Index->2', 'Index->3', 'Index->4',
         'Index->5'].each do |idx|
            index_col_names = []
            @field_indexes.each_with_index do |fi,i|
                next if fi.nil?
                index_col_names << @field_names[i] if fi.include?(idx)
            end

            # If no fields were indexed on this number (1..5), go to the
            # next index number.
            next if index_col_names.empty?

            # Create this index on the engine.
            @db.engine.init_index(self, index_col_names)

            # For each index found, add an instance method for it so that
            # it can be used for #selects.
            select_meth_str = <<-END_OF_STRING
            def select_by_#{index_col_names.join('_')}_index(*filter,
             &select_cond)
                result_set = []
                validate_filter(filter)
                filter = @field_names if filter.empty?
                return get_matches_by_index(:select,
                 [:#{index_col_names.join(',:')}], filter, select_cond)
            end
            END_OF_STRING
            self.class.class_eval(select_meth_str)
        end
    end

    #-----------------------------------------------------------------------
    # create_table_class
    #-----------------------------------------------------------------------
    def create_table_class
        #This is the class that will be used in #select condition blocks.
        @table_class = Class.new(KBTableRec)

        get_meth_str = ''
        get_meth_upd_res_str = ''
        set_meth_str = ''

        @field_names.zip(@field_types, @field_extras) do |x|
            field_name, field_type, field_extra = x

            @lookup_key = field_name if field_extra.has_key?('Key')

            # These are the default get/set methods for the table column.
            get_meth_str = <<-END_OF_STRING
            def #{field_name}
                return @#{field_name}
            end
            END_OF_STRING
            get_meth_upd_res_str = <<-END_OF_STRING
            def #{field_name}_upd_res
                return @#{field_name}
            end
            END_OF_STRING
            set_meth_str = <<-END_OF_STRING
            def #{field_name}=(s)
                @#{field_name} = convert_to(:#{field_type}, s)
            end
            END_OF_STRING

            # If this is a Lookup field, modify the get_method.
            if field_extra.has_key?('Lookup')
                lookup_table, key_field = field_extra['Lookup'].split('.')
                if key_field == 'recno'
                    get_meth_str = <<-END_OF_STRING
                    def #{field_name}
                        table = @tbl.db.get_table(:#{lookup_table})
                        return table[@#{field_name}]
                    end
                    END_OF_STRING
                else
                    begin
                        @db.get_table(lookup_table)
                    rescue RuntimeError
                        raise "Must create child table first when using " +
                         "'Lookup'"
                    end

                    if @db.get_table(lookup_table).respond_to?(
                     'select_by_%s_index' % key_field)
                        get_meth_str = <<-END_OF_STRING
                        def #{field_name}
                            table = @tbl.db.get_table(:#{lookup_table})
                            return table.select_by_#{key_field}_index { |r|
                             r.#{key_field} == @#{field_name} }.first
                        end
                        END_OF_STRING
                    else
                        get_meth_str = <<-END_OF_STRING
                        def #{field_name}
                            table = @tbl.db.get_table(:#{lookup_table})
                            return table.select { |r|
                             r.#{key_field} == @#{field_name} }.first
                        end
                        END_OF_STRING
                    end
                end
            end

            # If this is a Link_many field, modify the get/set methods.
            if field_extra.has_key?('Link_many')
                lookup_field, rest = field_extra['Link_many'].split('=')
                link_table, link_field = rest.split('.')

                begin
                    @db.get_table(link_table)
                rescue RuntimeError
                    raise "Must create child table first when using " +
                     "'Link_many'"
                end

                if @db.get_table(link_table).respond_to?(
                 'select_by_%s_index' % link_field)
                    get_meth_str = <<-END_OF_STRING
                    def #{field_name}
                        table = @tbl.db.get_table(:#{link_table})
                        return table.select_by_#{link_field}_index { |r|
                         r.send(:#{link_field}) == @#{lookup_field} }
                    end
                    END_OF_STRING
                else
                    get_meth_str = <<-END_OF_STRING
                    def #{field_name}
                        table = @tbl.db.get_table(:#{link_table})
                        return table.select { |r|
                         r.send(:#{link_field}) == @#{lookup_field} }
                    end
                    END_OF_STRING
                end

                get_meth_upd_res_str = <<-END_OF_STRING
                def #{field_name}_upd_res
                    return nil
                end
                END_OF_STRING
                set_meth_str = <<-END_OF_STRING
                def #{field_name}=(s)
                    @#{field_name} = nil
                end
                END_OF_STRING
            end

            # If this is a Calculated field, modify the get/set methods.
            if field_extra.has_key?('Calculated')
                calculation = field_extra['Calculated']

                get_meth_str = <<-END_OF_STRING
                def #{field_name}()
                    return #{calculation}
                end
                END_OF_STRING
                get_meth_upd_res_str = <<-END_OF_STRING
                def #{field_name}_upd_res()
                    return nil
                end
                END_OF_STRING
                set_meth_str = <<-END_OF_STRING
                def #{field_name}=(s)
                    @#{field_name} = nil
                end
                END_OF_STRING
            end

            @table_class.class_eval(get_meth_str)
            @table_class.class_eval(get_meth_upd_res_str)
            @table_class.class_eval(set_meth_str)
        end
    end

    #-----------------------------------------------------------------------
    # convert_to_string
    #-----------------------------------------------------------------------
    def convert_to_string(data_type, x)
        case data_type
        when :YAML
            y = x.to_yaml
            if y =~ ENCODE_RE
                return y.gsub("&", '&amp;').gsub("\n", '&linefeed;').gsub(
                 "\r", '&carriage_return;').gsub("\032", '&substitute;'
                 ).gsub("|", '&pipe;')
            else
                return y
            end
        when :String
            if x =~ ENCODE_RE
                return x.gsub("&", '&amp;').gsub("\n", '&linefeed;').gsub(
                 "\r", '&carriage_return;').gsub("\032", '&substitute;'
                 ).gsub("|", '&pipe;')
            else
                return x
            end
        else
            return x.to_s
        end
    end

    #-----------------------------------------------------------------------
    # validate_filter
    #-----------------------------------------------------------------------
    #++
    # Check that filter contains valid field names.
    #
    def validate_filter(filter)
        # Each field in the filter array must be a valid fieldname in the
        # table.
        filter.each { |f|
            raise 'Invalid field name: %s in filter!' % f unless \
             @field_names.include?(f)
        }
    end

    #-----------------------------------------------------------------------
    # convert_input_data
    #-----------------------------------------------------------------------
    #++
    # Convert data passed to #input, #update, or #set to a common format.
    #
    def convert_input_data(values)
        if values.class == Proc
            tbl_struct = Struct.new(*@field_names[1..-1])
            tbl_rec = tbl_struct.new
            begin
                values.call(tbl_rec)
            rescue NoMethodError
                raise 'Invalid field name in code block: %s' % $!
            end
            temp_hash = {}
            @field_names[1..-1].collect { |f|
                temp_hash[f] = tbl_rec[f] unless tbl_rec[f].nil?
            }
            return temp_hash
        elsif values[0].class.to_s == @record_class or \
         values[0].class == @table_class
            temp_hash = {}
            @field_names[1..-1].collect { |f|
                temp_hash[f] = values[0].send(f) if values[0].respond_to?(f)
            }
            return temp_hash
        elsif values[0].class == Hash
            return values[0].dup
        elsif values[0].kind_of?(Struct)
            temp_hash = {}
            @field_names[1..-1].collect { |f|
                temp_hash[f] = values[0][f] if values[0].members.include?(
                 f.to_s)
            }
            return temp_hash
        elsif values[0].class == Array
            raise ArgumentError, 'Must specify all fields in input array!' \
             unless values[0].size == @field_names[1..-1].size
            temp_hash = {}
            @field_names[1..-1].collect { |f|
                temp_hash[f] = values[0][@field_names.index(f)-1]
            }
            return temp_hash
        elsif values.class == Array
            raise ArgumentError, 'Must specify all fields in input array!' \
             unless values.size == @field_names[1..-1].size
            temp_hash = {}
            @field_names[1..-1].collect { |f|
                temp_hash[f] = values[@field_names.index(f)-1]
            }
            return temp_hash
        else
            raise(ArgumentError, 'Invalid type for values container!')
        end
    end

    #-----------------------------------------------------------------------
    # validate_input
    #-----------------------------------------------------------------------
    #++
    # Check input data to ensure proper data types.
    #
    def validate_input(data)
        raise 'Cannot insert/update recno field!' if data.has_key?(:recno)

        @field_names[1..-1].each do |f|
            next unless data.has_key?(f)

            next if data[f].nil?
            case @field_types[@field_names.index(f)]
            when /:String|:Memo|:Blob/
                raise 'Invalid String value for: %s' % f unless \
                 data[f].respond_to?(:to_str)
            when :Boolean
                raise 'Invalid Boolean value for: %s' % f unless \
                 data[f].is_a?(TrueClass) or data[f].kind_of?(FalseClass)
            when :Integer
                raise 'Invalid Integer value for: %s' % f unless \
                 data[f].respond_to?(:to_int)
            when :Float
                raise 'Invalid Float value for: %s' % f unless \
                 data[f].respond_to?(:to_f)
            when :Date
                raise 'Invalid Date value for: %s' % f unless \
                 data[f].is_a?(Date)
            when :Time
                raise 'Invalid Time value for: %s' % f unless \
                 data[f].is_a?(Time)
            when :DateTime
                raise 'Invalid DateTime value for: %s' % f unless \
                 data[f].is_a?(DateTime)
            when :YAML
                raise 'Invalid YAML value for: %s' % f unless \
                 data[f].respond_to?(:to_yaml)
            end
        end
    end

    #-----------------------------------------------------------------------
    # update_header_vars
    #-----------------------------------------------------------------------
    #++
    # Read header record and update instance variables.
    #
    def update_header_vars
        @encrypted, @last_rec_no, @del_ctr, @record_class, @field_names, \
         @field_types, @field_indexes, @field_extras = \
         @db.engine.get_header_vars(self)
    end

    #-----------------------------------------------------------------------
    # get_result_struct
    #-----------------------------------------------------------------------
    def get_result_struct(query_type, filter)
        case query_type
        when :select
            return Struct.new(*filter) if @record_class == 'Struct'
        when :update
            return Struct.new(*(filter + [:fpos, :line_length]))
        when :delete
            return Struct.new(:recno, :fpos, :line_length)
        end
        return nil
    end

    #-----------------------------------------------------------------------
    # create_result_rec
    #-----------------------------------------------------------------------
    def create_result_rec(query_type, filter, result_struct, tbl_rec, rec)
        # If this isn't a select query or if it is a select query, but
        # the table record class is simply a Struct, then we will use
        # a Struct for the result record type.
        if query_type != :select
            result_rec = result_struct.new(*filter.collect { |f|
             tbl_rec.send("#{f}_upd_res".to_sym) })
        elsif @record_class == 'Struct'
            result_rec = result_struct.new(*filter.collect { |f|
             tbl_rec.send(f) })
        else
            if Object.full_const_get(@record_class).respond_to?(:kb_create)
                result_rec = Object.full_const_get(@record_class
                 ).kb_create(*@field_names.collect { |f|
                     # Just a warning here:  If you specify a filter on
                     # a select, you are only going to get those fields
                     # you specified in the result set, EVEN IF
                     # record_class is a custom class instead of Struct.
                     if filter.include?(f)
                         tbl_rec.send(f)
                     else
                         nil
                     end
                 })
            elsif Object.full_const_get(@record_class).respond_to?(
             :kb_defaults)
                result_rec = Object.full_const_get(@record_class).new(
                 *@field_names.collect { |f|
                     tbl_rec.send(f) || Object.full_const_get(
                      @record_class).kb_defaults[@field_names.index(f)]
                 }
                )
            end
        end

        unless query_type == :select
            result_rec.fpos = rec[-2]
            result_rec.line_length = rec[-1]
        end
        return result_rec
    end

    #-----------------------------------------------------------------------
    # get_matches
    #-----------------------------------------------------------------------
    #++
    # Return records from table that match select condition.
    #
    def get_matches(query_type, filter, select_cond)
        result_struct = get_result_struct(query_type, filter)
        match_array = KBResultSet.new(self, filter, filter.collect { |f|
         @field_types[@field_names.index(f)] })

        tbl_rec = @table_class.new(self)

        # Loop through table.
        @db.engine.get_recs(self).each do |rec|
            tbl_rec.populate(rec)
            next unless select_cond.call(tbl_rec) unless select_cond.nil?

            match_array << create_result_rec(query_type, filter,
             result_struct, tbl_rec, rec)

        end
        return match_array
    end

    #-----------------------------------------------------------------------
    # get_matches_by_index
    #-----------------------------------------------------------------------
    #++
    # Return records from table that match select condition using one of
    # the table's indexes instead of searching the whole file.
    #
    def get_matches_by_index(query_type, index_fields, filter, select_cond)
        good_matches = []

        idx_struct = Struct.new(*(index_fields + [:recno]))

        begin
            @db.engine.get_index(self, index_fields.join('_')).each do |rec|
                good_matches << rec[-1] if select_cond.call(
                 idx_struct.new(*rec))
            end
        rescue NoMethodError
            raise 'Field name in select block not part of index!'
        end

        return get_matches_by_recno(query_type, filter, good_matches)
    end

    #-----------------------------------------------------------------------
    # get_matches_by_recno_index
    #-----------------------------------------------------------------------
    #++
    # Return records from table that match select condition using the
    # table's recno index instead of searching the whole file.
    #
    def get_matches_by_recno_index(query_type, filter, select_cond)
        good_matches = []

        idx_struct = Struct.new(:recno)

        begin
            @db.engine.get_recno_index(self).each_key do |key|
                good_matches << key if select_cond.call(
                 idx_struct.new(key))
            end
        rescue NoMethodError
            raise "Field name in select block not part of index!"
        end

        return nil if good_matches.empty?
        return get_matches_by_recno(query_type, filter, good_matches)
    end

    #-----------------------------------------------------------------------
    # get_match_by_recno
    #-----------------------------------------------------------------------
    #++
    # Return record from table that matches supplied recno.
    #
    def get_match_by_recno(query_type, filter, recno)
        result_struct = get_result_struct(query_type, filter)
        match_array = KBResultSet.new(self, filter, filter.collect { |f|
         @field_types[@field_names.index(f)] })

        tbl_rec = @table_class.new(self)

        rec = @db.engine.get_rec_by_recno(self, recno)
        return nil if rec.nil?
        tbl_rec.populate(rec)

        return create_result_rec(query_type, filter, result_struct,
         tbl_rec, rec)
    end

    #-----------------------------------------------------------------------
    # get_matches_by_recno
    #-----------------------------------------------------------------------
    #++
    # Return records from table that match select condition.
    #
    def get_matches_by_recno(query_type, filter, recnos)
        result_struct = get_result_struct(query_type, filter)
        match_array = KBResultSet.new(self, filter, filter.collect { |f|
         @field_types[@field_names.index(f)] })

        tbl_rec = @table_class.new(self)

        @db.engine.get_recs_by_recno(self, recnos).each do |rec|
            next if rec.nil?
            tbl_rec.populate(rec)

            match_array << create_result_rec(query_type, filter,
             result_struct, tbl_rec, rec)
        end
        return match_array
    end
end


#---------------------------------------------------------------------------
# KBMemo
#---------------------------------------------------------------------------
class KBMemo
    attr_reader :filepath, :memo

    #-----------------------------------------------------------------------
    # initialize
    #-----------------------------------------------------------------------
    def initialize(db, filepath)
        @filepath = filepath
        @memo = db.engine.get_memo(@filepath)
    end
end

#---------------------------------------------------------------------------
# KBBlob
#---------------------------------------------------------------------------
class KBBlob
    attr_reader :filepath, :blob

    #-----------------------------------------------------------------------
    # initialize
    #-----------------------------------------------------------------------
    def initialize(db, filepath)
        @filepath = filepath
        @blob = db.engine.get_blob(@filepath)
    end
end


#---------------------------------------------------------------------------
# KBIndex
#---------------------------------------------------------------------------
class KBIndex
    include KBTypeConversionsMixin

    UNENCODE_RE = /&(?:amp|linefeed|carriage_return|substitute|pipe);/

    #-----------------------------------------------------------------------
    # initialize
    #-----------------------------------------------------------------------
    def initialize(table, index_fields)
        @idx_arr = []
        @table = table
        @index_fields = index_fields
        @col_poss = index_fields.collect {|i| table.field_names.index(i) }
        @col_names = index_fields
        @col_types = index_fields.collect {|i|
         table.field_types[table.field_names.index(i)]}
    end

    #-----------------------------------------------------------------------
    # get_idx
    #-----------------------------------------------------------------------
    def get_idx
        return @idx_arr
    end

    #-----------------------------------------------------------------------
    # rebuild
    #-----------------------------------------------------------------------
    def rebuild(fptr)
        @idx_arr.clear

        encrypted = @table.encrypted?

        # Skip header rec.
        fptr.readline

        begin
            # Loop through table.
            while true
                line = fptr.readline

                line = unencrypt_str(line) if encrypted
                line.strip!

                # If blank line (i.e. 'deleted'), skip it.
                next if line == ''

                # Split the line up into fields.
                rec = line.split('|', @col_poss.max+2)

                # Create the index record by pulling out the record fields
                # that make up this index and converting them to their
                # native types.
                idx_rec = []
                @col_poss.zip(@col_types).each do |col_pos, col_type|
                    idx_rec << convert_to(col_type, rec[col_pos])
                end

                # Were all the index fields for this record equal to NULL?
                # Then don't add this index record to index array; skip to
                # next record.
                next if idx_rec.compact.empty?

                # Add recno to the end of this index record.
                idx_rec << rec.first.to_i

                # Add index record to index array.
                @idx_arr << idx_rec
            end
        # Here's how we break out of the loop...
        rescue EOFError
        end
    end

    #-----------------------------------------------------------------------
    # add_index_rec
    #-----------------------------------------------------------------------
    def add_index_rec(rec)
        @idx_arr << @col_poss.zip(@col_types).collect do |col_pos, col_type|
            convert_to(col_type, rec[col_pos])
         end + [rec.first.to_i]
    end

    #-----------------------------------------------------------------------
    # delete_index_rec
    #-----------------------------------------------------------------------
    def delete_index_rec(recno)
        i = @idx_arr.rassoc(recno.to_i)
        @idx_arr.delete_at(@idx_arr.index(i)) unless i.nil?
    end

    #-----------------------------------------------------------------------
    # update_index_rec
    #-----------------------------------------------------------------------
    def update_index_rec(rec)
        delete_index_rec(rec.first.to_i)
        add_index_rec(rec)
    end
end


#---------------------------------------------------------------------------
# KBRecnoIndex
#---------------------------------------------------------------------------
class KBRecnoIndex
    #-----------------------------------------------------------------------
    # initialize
    #-----------------------------------------------------------------------
    def initialize(table)
        @idx_hash = {}
        @table = table
    end

    #-----------------------------------------------------------------------
    # get_idx
    #-----------------------------------------------------------------------
    def get_idx
        return @idx_hash
    end

    #-----------------------------------------------------------------------
    # rebuild
    #-----------------------------------------------------------------------
    def rebuild(fptr)
        @idx_hash.clear

        encrypted = @table.encrypted?

        begin
            # Skip header rec.
            fptr.readline

            # Loop through table.
            while true
                # Record current position in table.  Then read first
                # detail record.
                fpos = fptr.tell
                line = fptr.readline

                line = unencrypt_str(line) if encrypted
                line.strip!

                # If blank line (i.e. 'deleted'), skip it.
                next if line == ''

                # Split the line up into fields.
                rec = line.split('|', 2)

                @idx_hash[rec.first.to_i] = fpos
            end
        # Here's how we break out of the loop...
        rescue EOFError
        end
    end

    #-----------------------------------------------------------------------
    # add_index_rec
    #-----------------------------------------------------------------------
    def add_index_rec(recno, fpos)
        raise 'Table already has index record for recno: %s' % recno if \
         @idx_hash.has_key?(recno.to_i)
        @idx_hash[recno.to_i] = fpos
    end

    #-----------------------------------------------------------------------
    # update_index_rec
    #-----------------------------------------------------------------------
    def update_index_rec(recno, fpos)
        raise 'Table has no index record for recno: %s' % recno unless \
         @idx_hash.has_key?(recno.to_i)
        @idx_hash[recno.to_i] = fpos
    end

    #-----------------------------------------------------------------------
    # delete_index_rec
    #-----------------------------------------------------------------------
    def delete_index_rec(recno)
        raise 'Table has no index record for recno: %s' % recno unless \
         @idx_hash.has_key?(recno.to_i)
        @idx_hash.delete(recno.to_i)
    end
end


#---------------------------------------------------------------------------
# KBTableRec
#---------------------------------------------------------------------------
class KBTableRec
    include KBTypeConversionsMixin

    def initialize(tbl)
        @tbl = tbl
    end

    def populate(rec)
        @tbl.field_names.zip(rec).each do |fn, val|
            send("#{fn}=", val)
        end
    end

    def clear
        @tbl.field_names.each do |fn|
            send("#{fn}=", nil)
        end
    end
end


#---------------------------------------------------------
# KBResultSet
#---------------------------------------------------------------------------
class KBResultSet < Array
    #-----------------------------------------------------------------------
    # KBResultSet.reverse
    #-----------------------------------------------------------------------
    def KBResultSet.reverse(sort_field)
        return [sort_field, :desc]
    end

    #-----------------------------------------------------------------------
    # initialize
    #-----------------------------------------------------------------------
    def initialize(table, filter, filter_types, *args)
        @table = table
        @filter = filter
        @filter_types = filter_types
        super(*args)

        @filter.each do |f|
            get_meth_str = <<-END_OF_STRING
            def #{f}()
                if defined?(@#{f}) then
                    return @#{f}
                else
                    @#{f} = self.collect { |x| x.#{f} }
                    return @#{f}
                end
            end
            END_OF_STRING
            self.class.class_eval(get_meth_str)
        end
    end

    #-----------------------------------------------------------------------
    # to_ary
    #-----------------------------------------------------------------------
    def to_ary
        to_a
    end

    #-----------------------------------------------------------------------
    # set
    #-----------------------------------------------------------------------
    #++
    # Update record(s) in table, return number of records updated.
    #
    def set(*updates, &update_cond)
        raise 'Cannot specify both a hash and a proc for method #set!' \
         unless updates.empty? or update_cond.nil?

        raise 'Must specify update proc or hash for method #set!' if \
         updates.empty? and update_cond.nil?

        if updates.empty?
            @table.set(self, update_cond)
        else
            @table.set(self, updates)
        end
    end

    #-----------------------------------------------------------------------
    # sort
    #-----------------------------------------------------------------------
    def sort(*sort_fields)
        sort_fields_arrs = []
        sort_fields.each do |f|
            if f.to_s[0..0] == '-'
                sort_fields_arrs << [f.to_s[1..-1].to_sym, :desc]
            elsif f.to_s[0..0] == '+'
                sort_fields_arrs << [f.to_s[1..-1].to_sym, :asc]
            else
                sort_fields_arrs << [f, :asc]
            end
        end

        sort_fields_arrs.each do |f|
            raise "Invalid sort field" unless @filter.include?(f[0])
        end

        super() { |a,b|
            x = []
            y = []
            sort_fields_arrs.each do |s|
                if [:Integer, :Float].include?(
                 @filter_types[@filter.index(s[0])])
                    a_value = a.send(s[0]) || 0
                    b_value = b.send(s[0]) || 0
                else
                    a_value = a.send(s[0])
                    b_value = b.send(s[0])
                end
                if s[1] == :desc
                    x << b_value
                    y << a_value
                else
                    x << a_value
                    y << b_value
                end
            end
            x <=> y
        }
    end

    #-----------------------------------------------------------------------
    # to_report
    #-----------------------------------------------------------------------
    def to_report(recs_per_page=0, print_rec_sep=false)
        result = collect { |r| @filter.collect {|f| r.send(f)} }

        # How many records before a formfeed.
        delim = ' | '

        # columns of physical rows
        columns = [@filter].concat(result).transpose

        max_widths = columns.collect { |c|
            c.max { |a,b| a.to_s.length <=> b.to_s.length }.to_s.length
        }

        row_dashes = '-' * (max_widths.inject {|sum, n| sum + n} +
         delim.length * (max_widths.size - 1))

        justify_hash = { :String => :ljust, :Integer => :rjust,
         :Float => :rjust, :Boolean => :ljust, :Date => :ljust,
         :Time => :ljust, :DateTime => :ljust }

        header_line = @filter.zip(max_widths, @filter.collect { |f|
            @filter_types[@filter.index(f)] }).collect { |x,y,z|
                 x.to_s.send(justify_hash[z], y) }.join(delim)

        output = ''
        recs_on_page_cnt = 0

        result.each do |row|
            if recs_on_page_cnt == 0
                output << header_line + "\n" << row_dashes + "\n"
            end

            output << row.zip(max_widths, @filter.collect { |f|
                @filter_types[@filter.index(f)] }).collect { |x,y,z|
                    x.to_s.send(justify_hash[z], y) }.join(delim) + "\n"

            output << row_dashes + '\n' if print_rec_sep
            recs_on_page_cnt += 1

            if recs_per_page > 0 and (recs_on_page_cnt ==
             num_recs_per_page)
                output << '\f'
                recs_on_page_count = 0
            end
        end
        return output
    end
end


#---------------------------------------------------------------------------
# Object
#---------------------------------------------------------------------------
class Object
    def full_const_get(name)
        list = name.split("::")
        obj = Object
        list.each {|x| obj = obj.const_get(x) }
        obj
    end
end


#---------------------------------------------------------------------------
# NilClass
#---------------------------------------------------------------------------
class NilClass
    #-----------------------------------------------------------------------
    # method_missing
    #-----------------------------------------------------------------------
    #
    # This code is necessary because if, inside a select condition code
    # block, there is a case where you are trying to do an expression
    # against a table field that is equal to nil, I don't want a method
    # missing exception to occur.  I just want the expression to be nil.  I
    # initially had this method returning false, but then I had an issue
    # where I had a YAML field that was supposed to hold an Array.  If the
    # field was empty (i.e. nil) it was actually returning false when it
    # should be returning nil.  Since nil evaluates to false, it works if I
    # return nil.
    # Here's an example:
    # #select { |r| r.speed > 300 }
    # What happens if speed is nil (basically NULL in DBMS terms)?  Without
    # this code, an exception is going to be raised, which is not what we
    # really want.  We really want this expression to return nil.
    def method_missing(method_id, *stuff)
        return nil
    end
end


#---------------------------------------------------------------------------
# Symbol
#---------------------------------------------------------------------------
class Symbol
    #-----------------------------------------------------------------------
    # -@
    #-----------------------------------------------------------------------
    #
    # This allows you to put a minus sign in front of a field name in order
    # to specify descending sort order.
    def -@
        ("-"+self.to_s).to_sym
    end

    #-----------------------------------------------------------------------
    # +@
    #-----------------------------------------------------------------------
    #
    # This allows you to put a plus sign in front of a field name in order
    # to specify ascending sort order.
    def +@
        ("+"+self.to_s).to_sym
    end
end