Ruby 1.9 style hashes broken in > 0.9.5

[sethvargo]

In my testing, the Ruby 1.9 hash style syntax was supported in 0.9.5, but has not worked since then. Is there any way to bring back that syntax in 1.x?

[sethvargo]

Here's a diff of 0.9.5 and 0.9.6. It worked in 0.9.5, and was broken in 0.9.6 and up.

diff --git acoderay-0.9.5/lib/README bcoderay-0.9.6/lib/README
index 38f89bb..e08ca59 100644
--- acoderay-0.9.5/lib/README
+++ bcoderay-0.9.6/lib/README
@@ -18,7 +18,7 @@ And with line numbers.
 * is what everybody should have on their website
 * solves all your problems and makes the girls run after you

-Version: 0.9.5
+Version: 0.9.6
 Author:: murphy (Kornelius Kalnbach)
 Contact:: murphy rubychan de
 Website:: coderay.rubychan.de[http://coderay.rubychan.de]
diff --git acoderay-0.9.5/lib/coderay/.DS_Store bcoderay-0.9.5/lib/coderay/.DS_Store
deleted file mode 100644
index fb583d8..0000000
Binary files acoderay-0.9.5/lib/coderay/.DS_Store and /dev/null differ
diff --git acoderay-0.9.5/lib/coderay/scanner.rb bcoderay-0.9.6/lib/coderay/scanner.rb
index 74079a8..b12c865 100644
--- acoderay-0.9.5/lib/coderay/scanner.rb
+++ bcoderay-0.9.6/lib/coderay/scanner.rb
@@ -69,6 +69,7 @@ module CodeRay
         def normify code
           code = code.to_s
           if code.respond_to?(:encoding) && (code.encoding.name != 'UTF-8' || !code.valid_encoding?)
+            code = code.dup
             original_encoding = code.encoding
             code.force_encoding 'Windows-1252'
             unless code.valid_encoding?
diff --git acoderay-0.9.5/lib/coderay/scanners/ruby/patterns.rb bcoderay-0.9.6/lib/coderay/scanners/ruby/patterns.rb
index 69ae012..ba3ac8a 100644
--- acoderay-0.9.5/lib/coderay/scanners/ruby/patterns.rb
+++ bcoderay-0.9.6/lib/coderay/scanners/ruby/patterns.rb
@@ -16,7 +16,7 @@ module Scanners
     DEF_KEYWORDS = %w[ def ]
     UNDEF_KEYWORDS = %w[ undef ]
     ALIAS_KEYWORDS = %w[ alias ]
-    MODULE_KEYWORDS = %w[class module]
+    MODULE_KEYWORDS = %w[ class module ]
     DEF_NEW_STATE = WordList.new(:initial).
       add(DEF_KEYWORDS, :def_expected).
       add(UNDEF_KEYWORDS, :undef_expected).
@@ -25,7 +25,8 @@ module Scanners

     PREDEFINED_CONSTANTS = %w[
       nil true false self
-      DATA ARGV ARGF __FILE__ __LINE__
+      DATA ARGV ARGF
+      __FILE__ __LINE__ __ENCODING__
     ]

     IDENT_KIND = WordList.new(:ident).
diff --git acoderay-0.9.5/lib/coderay/scanners/ruby.rb bcoderay-0.9.6/lib/coderay/scanners/ruby.rb
index cef8d83..3cadc64 100644
--- acoderay-0.9.5/lib/coderay/scanners/ruby.rb
+++ bcoderay-0.9.6/lib/coderay/scanners/ruby.rb
@@ -181,17 +181,20 @@ module Scanners
               if last_token_dot
                 kind = if match[/^[A-Z]/] and not match?(/\(/) then :constant else :ident end
               else
-                kind = patterns::IDENT_KIND[match]
-                if kind == :ident
-                  if match[/^[A-Z]/] and not match[/[!?]$/] and not match?(/\(/)
-                    kind = :constant
-                  elsif scan(/:(?= )/)
-                    match << ':'
-                    kind = :symbol
+                if value_expected != :expect_colon && scan(/:(?= )/)
+                  tokens << [match, :key]
+                  match = ':'
+                  kind = :operator
+                else
+                  kind = patterns::IDENT_KIND[match]
+                  if kind == :ident
+                    if match[/\A[A-Z]/] and not match[/[!?]$/] and not match?(/\(/)
+                      kind = :constant
+                    end
+                  elsif kind == :reserved
+                    state = patterns::DEF_NEW_STATE[match]
+                    value_expected = :set if patterns::KEYWORDS_EXPECTING_VALUE[match]
                   end
-                elsif kind == :reserved
-                  state = patterns::DEF_NEW_STATE[match]
-                  value_expected = :set if patterns::KEYWORDS_EXPECTING_VALUE[match]
                 end
               end
               value_expected = :set if check(/#{patterns::VALUE_FOLLOWS}/o)
@@ -401,7 +404,9 @@ module Scanners
 # }}}

           unless kind == :error
-            value_expected = value_expected == :set
+            if value_expected = value_expected == :set
+              value_expected = :expect_colon if match == '?' || match == 'when'
+            end
             last_token_dot = last_token_dot == :set
           end

diff --git acoderay-0.9.5/lib/coderay.rb bcoderay-0.9.6/lib/coderay.rb
index f54e463..b439bf9 100644
--- acoderay-0.9.5/lib/coderay.rb
+++ bcoderay-0.9.6/lib/coderay.rb
@@ -134,7 +134,7 @@ module CodeRay
   # Minor: feature milestone
   # Teeny: development state, 0 for pre-release
   # Revision: Subversion Revision number (generated on rake gem:make)
-  VERSION = '0.9.5'
+  VERSION = '0.9.6'

   require 'coderay/tokens'
   require 'coderay/token_classes'
diff --git acoderay-0.9.5/test/functional/basic.rb bcoderay-0.9.6/test/functional/basic.rb
index 150089e..8ecd3d3 100755
--- acoderay-0.9.5/test/functional/basic.rb
+++ bcoderay-0.9.6/test/functional/basic.rb
@@ -115,4 +115,8 @@ more code  # and another comment, in-line.
     assert CodeRay::Scanners.list.include?('plaintext')
   end

+  def test_scan_a_frozen_string
+    CodeRay.scan RUBY_VERSION, :ruby
+  end
+  
 end

[korny]

Wow, this is unfortunate. I thought it was working, strange that it didn't come up until now. Thank you for reporting!

I guess the feature was lost in the transition to 1.0 somehow. It relies on an older variant of the value_expected flag, so it's not easy to put back in to the new Ruby scanner. I will think of something.

In the meantime, please take a look at this test case I prepared:

new = { hash: :syntax }
call(with, hash: :syntax)
this_crazy(hash: :syntax)
test \
  hash: :syntax,
  multi: :line

Can you think of more examples that should be supported?

[sethvargo]

It broke in 0.9.6 actually.

And I would test the case without a space in between (foo:"bar" and foo:bar)

Thanks for looking into this.

Seth

[korny]

It broke in 0.9.6 actually.

Actually, it only changed: 0.9.6 introduced the :key token type and used it for the hash keys, so it should be a
different color:

Maybe your stylesheet doesn't include a color for this class, but it's there in 0.9.6+.

1.0.0 fails to highlight it as anything meaningful and just falls back to :ident.

[korny]

Just for reference, the updated test:

new = { hash: :syntax }
call(with, hash: :syntax)
this_crazy(hash: :syntax)
test \
  hash: :syntax,
  multi: :line

more hash::tests
more hash:"tests"
more hash:42
more hash:??
more hash:{}
more hash:nil
more hash:foo
more hash://
_ _:_

[sethvargo]

Anything I can do to help?