few more ideas

[gurgeous]

Thanks for merging the PR! A few more questions for you:

1. I think I can speedup decode_entities with a few changes. Should be easy now that we have some testing. Interested?
2. What does smart_format_tags do? LMK and I can write a test for it if you want.
3. I noticed that some of the Sterile methods modify strings in place. Is this be design? Should we consider changing? I think the typical Ruby convention would be:

# return new string, don't modify str
new_str = str.decode_entities
new_str = Sterile.decode_entities(str)

# modify str in place, return str if modified (nil if unchanged)
str.decode_entities!

[pbhogan]

1. Sure—have at it.
2. This is actually one of the primary reasons I wrote this gem: it applies smart_format on text inside HTML. Our use-case is for running article content through smart_format_tags to make things consistent. The counterpart is plain_format_tags. This is what nokogiri is used for. It could probably be named/aliased something more intuitive like smart_format_html or smart_format_xml 🤷‍♂️
3. That's unintentional and should be considered a bug. The intention is to conform to the Ruby convention. It would be nice to test this. It should be as simple as running each method on a frozen string and expecting no exception.

Along the lines of 3., there are almost certainly optimization opportunities in writing the modify-in-place versions separately. I took the shortcut of making the modify-in-place just use replace with the return-modified version.

[gurgeous]

OK, let's start with (1), speeding up decode_entites. Here's my test script (see below) and the output. The single pass version is quite a bit faster. It has some other minor advantages too, avoiding accidental double decodes. I'll put together a PR and add some more tests.

$ ruby benchmark.rb
the quick brown fox ° ° ° jumped over the lazy dog
the quick brown fox ° ° ° jumped over the lazy dog
the quick brown fox ° ° ° jumped over the lazy dog
       user     system      total        real
orig  0.959834   0.014008   0.973842 (  0.999022)
chr   0.853535   0.010538   0.864073 (  0.884131)
re    0.526134   0.006874   0.533008 (  0.543680)

$LOAD_PATH.unshift("#{__dir__}/lib")
require 'benchmark'
require "sterile"

module Sterile
  class << self
    # use chr instead of pack
    def decode_entities_chr(string)
      string.gsub!(/&#(\d{1,7});/) { $1.to_i.chr(Encoding::UTF_8) }
      string.gsub!(/&#x([a-fA-F0-9]{1,7});/) { $1.to_i(16).chr(Encoding::UTF_8) }
      string.gsub(/&([a-zA-Z0-9]+);/) do
        codepoint = html_entities_data[$1]
        codepoint ? codepoint.chr(Encoding::UTF_8) : $&
      end
    end

    DECODE_ENTITIES_RE = /
      &(?:
        \#(\d{1,7})|           # base 10
        \#x([a-fA-F0-9]{1,7})| # base 16
        ([a-zA-Z0-9]+)         # text
      );/x

    # single pass with regexp
    def decode_entities_re(string)
      return string if !string.include?("&")

      string.gsub(DECODE_ENTITIES_RE) do
        if $1
          $1.to_i.chr(Encoding::UTF_8)
        elsif $2
          $2.to_i(16).chr(Encoding::UTF_8)
        elsif $3
          codepoint = html_entities_data[$3]
          codepoint ? codepoint.chr(Encoding::UTF_8) : $&
        end
      end
    end
  end
end

s = "the quick brown fox &#176; &deg; &#xb0; jumped over the lazy dog"
puts Sterile.decode_entities(s.dup)
puts Sterile.decode_entities_chr(s.dup)
puts Sterile.decode_entities_re(s.dup)

n = 100000
Benchmark.bm do
  _1.report("orig") { n.times { Sterile.decode_entities(s.dup) } }
  _1.report("chr ") { n.times { Sterile.decode_entities_chr(s.dup) } }
  _1.report("re  ") { n.times { Sterile.decode_entities_re(s.dup) } }
end