Add new ZipWithoutBlock cop

[corsonknowles]

Performance Cop for the more efficient way to generate an Array of Arrays.

• Performs 40-90% faster than.map to iteratively wrap array contents.
• Performs 5 - 55% faster on converting a range to an array of arrays with .map, depending on size.
• Performs faster than mutating an array into an array of arrays in place with .map!
• Performs faster than using each_with_object

This optimization is particularly helpful in performance sensitive paths or in large scale applications that need to generate arrays of arrays at scale. For example, leveraging the bulk enqueuing feature in Sidekiq requires an array of arrays, and is by definition used at scale.

A performance gain is always present for all sizes of arrays and ranges. The gain is smallest with small ranges, but still significant for small arrays.

This is not a style cop, but it is poetic that the more performant approach also has simpler and shorter syntax.

.zip has been intentionally optimized in Ruby. This has been discussed publicly since at least 2012:

• https://stackoverflow.com/a/5254764

Official .zip documentation:

• https://ruby-doc.org/3.3.5/Array.html#method-i-zip
• https://ruby-doc.org/3.3.5/Enumerable.html#method-i-zip

Source code for .zip:

• https://github.com/ruby/ruby/blob/v3_3_0/array.c#L4418
• https://github.com/ruby/ruby/blob/v3_3_0/enum.c#L3448

This performance cop isn't just an announcement to use .zip in the spirit of appreciating Ruby's great features, it is also a useful and necessary tool to leverage Rubocop to clean up and add rigor in large Ruby code bases. Rubocop is a much better approach than pattern matching for clean up at scale here, and it comes with the added benefit of proactive user feedback as additions are made to the code base going forward.

For Arrays with 1000 entries, a common size for bulk operations, this performs 70% faster. Here it is at 70% faster, using benchmark-ips:

gem install benchmark-ips

require 'benchmark'
require 'benchmark/ips'

array = (1..1000).to_a; Benchmark.ips do |x|
  x.report(".zip:") do
    array.zip
  end
  
  x.report(".map { |id| [id] }:") do
    array.map { |id| [id] }
  end
end

ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23]
Warming up --------------------------------------
               .zip:     4.019k i/100ms
 .map { |id| [id] }:     2.404k i/100ms
Calculating -------------------------------------
               .zip:     41.193k (± 1.4%) i/s -    208.988k in   5.074349s
 .map { |id| [id] }:     24.277k (± 2.8%) i/s -    122.604k in   5.054623s

---

Before submitting the PR make sure the following are checked:

• The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
• Wrote good commit messages.
• Commit message starts with [Fix #issue-number] (if the related issue exists).
• Feature branch is up-to-date with master (if not - rebase it).
• Squashed related commits together.
• Added tests.
• Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
• Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

[corsonknowles]

Some of the gain here comes from simply not allocating the block. But I think most of it comes from the lower level conversion of the array into an array of arrays.

[corsonknowles]

Dear @koic and @Earlopain,
Are there any questions I can answer for you or modifications I can provide?

Thank you so much for your consideration!

[Earlopain]

This largely makes sense to me. Never really thought about using zip without arguments, that's a nice idea.

I think this pattern is relatively common in rails where they need to handle composite primary keys. I'll edit my findings in later if I decide to check. Edit: Nope, I was thinking about wrapping a single value in an array like [id]

Tip: use benchmark-ips for future benchmarks. The output is much more legible.

[corsonknowles]

I really appreciate the thoughtful comments @Earlopain

Fully revised and ready for review.

I had run the original version on 50,000 files. I added 2 more tests and diversified the input slightly to get confident in the new on_send approach. I predict this should be safe to merge now.

[corsonknowles]

Thanks for the benchmark-ips tip. I added some data using it for large and small arrays.

[Earlopain]

Almost done from my side, just a few remaining things.

Can you also add a test for map and map {}? These are common places for errors in the past. I think it works fine in your case right now

[technicalpickles]

First off, this will be a nice improvement for our code base, as we often use and recommend the map {|id| [id]} for Sidekiq.perform_bulk. Disclaimer: work with @corsonknowles 😁

I ran some of my own benchmarks based on @corsonknowles's in the body against small (4), medium (1000) and big (50k) arrays using benchmark-ips, and also added benchmark-memory for good measure. High level, we have:

• small: zip is 1.45x faster
• medium: zip is 1.74x faster
• big: zip is 1.93x faster

I'm kinda surprised, but memory usage ends up the same which I'm not quite sure how to explain yet 😅

Here is the benchmark:

require 'benchmark/ips'
require 'benchmark/memory'

arrays = {
  small: (1..4).to_a,
  medium: (1..1000).to_a,
  big: (1..50000).to_a,
}

arrays.each do |size, array|
  puts "=== #{size} ==="
  Benchmark.ips do |x|
    x.report(".map { |id| [id] }:") do
      array.map { |id| [id] }
    end

    x.report(".zip:") do
      array.zip
    end

    x.compare! order: :baseline
  end

  Benchmark.memory do |x|
    x.report(".map { |id| [id] }:") do
      array.map { |id| [id] }
    end

    x.report(".zip:") do
      array.zip
    end
  end
end

And the results:

=== small ===
ruby 3.3.2 (2024-05-30 revision e5a195edf6) [arm64-darwin23]
Warming up --------------------------------------
 .map { |id| [id] }:   405.112k i/100ms
               .zip:   590.068k i/100ms
Calculating -------------------------------------
 .map { |id| [id] }:      4.029M (± 1.1%) i/s  (248.17 ns/i) -     20.256M in   5.027463s
               .zip:      5.846M (± 6.7%) i/s  (171.06 ns/i) -     29.503M in   5.085950s

Comparison:
 .map { |id| [id] }::  4029499.2 i/s
               .zip::  5845845.8 i/s - 1.45x  faster

Calculating -------------------------------------
 .map { |id| [id] }:   240.000  memsize (     0.000  retained)
                         5.000  objects (     0.000  retained)
                         0.000  strings (     0.000  retained)
               .zip:   240.000  memsize (     0.000  retained)
                         5.000  objects (     0.000  retained)
                         0.000  strings (     0.000  retained)
=== medium ===
ruby 3.3.2 (2024-05-30 revision e5a195edf6) [arm64-darwin23]
Warming up --------------------------------------
 .map { |id| [id] }:     2.042k i/100ms
               .zip:     3.533k i/100ms
Calculating -------------------------------------
 .map { |id| [id] }:     20.169k (± 1.7%) i/s   (49.58 μs/i) -    102.100k in   5.063731s
               .zip:     35.106k (± 1.6%) i/s   (28.48 μs/i) -    176.650k in   5.033169s

Comparison:
 .map { |id| [id] }::    20169.1 i/s
               .zip::    35106.3 i/s - 1.74x  faster

Calculating -------------------------------------
 .map { |id| [id] }:    48.040k memsize (     0.000  retained)
                         1.001k objects (     0.000  retained)
                         0.000  strings (     0.000  retained)
               .zip:    48.040k memsize (     0.000  retained)
                         1.001k objects (     0.000  retained)
                         0.000  strings (     0.000  retained)
=== big ===
ruby 3.3.2 (2024-05-30 revision e5a195edf6) [arm64-darwin23]
Warming up --------------------------------------
 .map { |id| [id] }:    44.000 i/100ms
               .zip:    64.000 i/100ms
Calculating -------------------------------------
 .map { |id| [id] }:    425.048 (± 0.7%) i/s    (2.35 ms/i) -      2.156k in   5.072641s
               .zip:    818.393 (± 2.3%) i/s    (1.22 ms/i) -      4.096k in   5.007817s

Comparison:
 .map { |id| [id] }::      425.0 i/s
               .zip::      818.4 i/s - 1.93x  faster

Calculating -------------------------------------
 .map { |id| [id] }:     2.400M memsize (     0.000  retained)
                        50.001k objects (     0.000  retained)
                         0.000  strings (     0.000  retained)
               .zip:     2.400M memsize (     0.000  retained)
                        50.001k objects (     0.000  retained)
                         0.000  strings (     0.000  retained)

[corsonknowles]

Thanks @Earlopain !

• Additional specs added, including the derivative cases that you noted, also showing that it plays nice in a method chain.
• Both block types/patterns are folded into the on_send now
• Added RangeHelp to get the corrector range to match both the method call and the block, but not map's receiver.
• Alphabetized the RESTRICT_ON_SEND placement, which seems to be the norm here for constant declaration order.

[corsonknowles]

For the very curious, .zip is also faster than .map! {[_1]}.

.each_with_object([]) {|id, object| object << [id]} is among the slowest of the reasonable strategies.

[corsonknowles]

Furthermore there's some CI failures, so please take a look at them (do you need to rebase from master too?)

I ran git rebase upstream and bundle update and I now do see some CI failures:

Is this what you are seeing?

Failed examples:

rspec './spec/rubocop/cop/performance/start_with_spec.rb[1:3:123]' # RuboCop::Cop::Performance::StartWith when `SafeMultiline: false` registers an offense and corrects str =~ /\A\^/
rspec './spec/rubocop/cop/performance/start_with_spec.rb[1:3:33]' # RuboCop::Cop::Performance::StartWith when `SafeMultiline: false` registers an offense and corrects str.match? /\A\^/
rspec './spec/rubocop/cop/performance/start_with_spec.rb[1:3:124]' # RuboCop::Cop::Performance::StartWith when `SafeMultiline: false` registers an offense and corrects /\A\^/ =~ str
rspec './spec/rubocop/cop/performance/start_with_spec.rb[1:3:213]' # RuboCop::Cop::Performance::StartWith when `SafeMultiline: false` registers an offense and corrects str.match /\A\^/
rspec './spec/rubocop/cop/performance/start_with_spec.rb[1:3:34]' # RuboCop::Cop::Performance::StartWith when `SafeMultiline: false` registers an offense and corrects /\A\^/.match? str
rspec './spec/rubocop/cop/performance/start_with_spec.rb[1:3:214]' # RuboCop::Cop::Performance::StartWith when `SafeMultiline: false` registers an offense and corrects /\A\^/.match str

I was able to reproduce this on the primary branch upstream as well.

I was able to demonstrate that these specs will pass again when Rubocop is locked to 1.69.2:

• Demo clean CI run before upstream introduces failures #485

[dvandersluis]

@corsonknowles ahh it's from a change I made upstream in rubocop/rubocop@f450f62. I'll fix it, thank you.

[dvandersluis]

CI is fixed not but "check the oldest supported rubocop version" is failing. This is because in RuboCop 1.48.1, the default target version for tests is 2.6, and numblocks were added in 2.7. Can you please tag your numblock specs with :ruby27?

[corsonknowles]

CI is fixed now but "check the oldest supported rubocop version" is failing. This is because in RuboCop 1.48.1, the default target version for tests is 2.6, and numblocks were added in 2.7. Can you please tag your numblock specs with :ruby27?

Great catch! Added these tags. I originally added numblock handling because of the InternalAffairs rule. Ruby 2.7 went end of life in Spring '23, so if Rubocop intends to continue to test for and support even earlier versions much longer, we might want to push some guidance up into the InternalAffairs rule to include this knowledge.

[dvandersluis]

How about ZipForNestedArrays, ZipForArraysOfArrays, or simply NestedArrays? Maybe we still need a verb, like GeneratingNestedArrays ?

@koic do one of these names work for you, or do you have a different suggestion?

[koic]

Typically, cop names don't include For. In this case, a name without For or Of, and probably including Zip, would be better. I’ll think about it as well. Naming is often the hardest part of implementing a cop.

[corsonknowles]

Typically, cop names don't include For. In this case, a name without For or Of, and probably including Zip, would be better. I’ll think about it as well. Naming is often the hardest part of implementing a cop.

I think you may have just named it perfectly in the revised Description & Message. If we focus on what the fix is, like we would suggest in code review, the performant choice is to use "Zip Without a Block" -- I renamed it, let me know what you think.

[koic]

Performance/ZipWithoutBlock cop is a better name than Performance/ZipForArrayWrapping cop. I also considered the name Performance/MapAsArrayWrapper cop, but I’m not confident about it. If no better name comes to mind for a while, Performance/ZipWithoutBlock cop will likely be the candidate.