Contents

# frozen_string_literal: true

module Klue
  module Langcraft
    # Tokenizer class
    class Tokenizer
      attr_reader :tokens

      def initialize(code)
        @code = code
        @tokens = []
      end

      def tokenize
        # Implement logic to convert code into tokens
        # Handle strings, symbols, keywords, and delimiters
      end
    end
  end
end

Version data entries

5 entries across 5 versions & 1 rubygems

Version	Path
klue-langcraft-0.4.0	lib/klue/langcraft/tokenizer-old-needs-revisit/tokenizer.rb
klue-langcraft-0.3.0	lib/klue/langcraft/tokenizer-old-needs-revisit/tokenizer.rb
klue-langcraft-0.2.0	lib/klue/langcraft/tokenizer-old-needs-revisit/tokenizer.rb
klue-langcraft-0.1.1	lib/klue/langcraft/tokenizer.rb
klue-langcraft-0.1.0	lib/klue/langcraft/tokenizer.rb