I have postet here that I think I can solve it in 2 hours, and got some downvotes for that; so I have decided to really give it a try. Long story short, it took me about 2 hours and 35 minutes.

Before I started developing, I made some quick assumptions to simplify things:

• I want to code it in Ruby, this is the language where I am most productive.
• I can assume the size of the stripes is well known, and I have hardcoded this size.
• The image can be converted to RAW with an external tool, and written into RAW.

# 0:08 - how to read binary files...
# 0:14 - index RGB values
# 0:21 - difference calculation seems to privide a reasonable number
# 0:25 - create all pairings of each row (too slow: need to use logic for all slices)
# 0:30 - create all pairings
# 0:31 - sorted pairings
# 0:35 - copy and number image to mspaint to verify
# 0:36 - bugfix, wrong slice row calculation
# 0:48 - getting stuck with combining pairs
# 1:03 - recombining
# 1:14 - thinking about how to ensure not to combine invalid groups (duplicates!)
# 1:33 - got first reasonable order. Hopefully the correct order!!
# 1:47 - wrote first image, wrong order
# 1:53 - got an almost correct image! one slice was aligned wrong.
# 1:58 - no luck with grayscale
# 2:28 - damn black-white skyscraper!
# 2:29 - SUCCESSSS!!!!!!
#
# time not measured: I thought about the problem about 5 minutes before starting hacking.
# So my time is probably about 2:34.
class Img
	attr_reader :w, :h, :slices
	def initialize
		# unpack as 8bit unsigned chars
		@d = IO.binread("TokyoPanoramaShredded.raw").unpack("C*")
		@w = 640
		@h = 359
		@slices = 20
		@sw = @w / @slices
	end

	def write(sorted)
		data = @d.dup
		source_slice = 0
		sorted.each do |target_slice|
			# copy one slice
			@sw.times do |w|
				@h.times do |h|
					target_idx = idx(w + @sw * source_slice, h)
					source_idx = idx(w + @sw * target_slice, h)
					data[target_idx] = @d[source_idx]
					data[target_idx+1] = @d[source_idx+1]
					data[target_idx+2] = @d[source_idx+2]
				end
			end
			source_slice += 1
		end

		File.open("out.raw", "wb") do |f|
			f.write data.pack("C*")
		end
	end

	def idx(x, y)
		(y * @w + x) * 3
	end

	# top left is 0, 0
	def rgb(x, y)
		i = idx(x, y)
		[@d[i], @d[i+1], @d[i+2]]
	end	

	# calculate sum of difference between r, g, b of two columns
	def difference(x1, x2)
		diff = 0
		@h.times do |y|
			# find best in range. This is required because otherwise the black syscraper fucks things up.
			lower = y - 15
			lower = 0 if (lower < 0)

			upper = y + 15
			upper = @h-1 if upper >= @h

			best = 1e100
			lower.upto(upper) do |r|
				v = 0
				rgb1 = rgb(x1, r)
				rgb2 = rgb(x2, r)

				d = rgb1[0] - rgb2[0]
				v += d*d
				d = rgb1[1] - rgb2[1]
				v += d*d
				d = rgb1[2] - rgb2[2]
				v += d*d

				best = v if (v < best)
			end
			diff += best
		end
		diff
	end

	def slice_start_idx(s)
		@sw * s
	end

	def slice_end_idx(s)
		@sw * s + @sw - 1
	end
end

# calculate
img = Img.new

# create ALL pairings.
slices = 20
pairs = []
slices.times do |s1|
	slices.times do |s2|
		next if s1 == s2

		pairs.push [img.difference(img.slice_end_idx(s1), img.slice_start_idx(s2)), [s1, s2] ]
	end
end

def add_to_group(combined, new_group)
	# create array of fixed numbers
	taken_numbers = {}
	taken_left = {}
	taken_right = {}
	combined.each do |group|
		group[1..-2].each do |x|
			taken_numbers[x] = true
		end
		taken_left[group.last] = true
		taken_right[group.first] = true
	end

	was_found = false
	combined.each do |group|
		next if was_found
		if (group.last == new_group.first)
			# insert at back
			was_found = true
			new_group.delete_at(0)
			if taken_numbers[new_group.last] || taken_left[new_group.last] || new_group.last == group.first
				return
			end
			new_group.each do |x|
				group.push x
			end
		elsif (group.first == new_group.last)
			# insert at front
			was_found = true
			new_group.pop
			if taken_numbers[new_group.first] || taken_right[new_group.first] || new_group.first == group.last
				return
			end
			new_group.reverse.each do |x|
				group.insert(0, x)
			end
		end
	end
	if !was_found
		cf = combined.flatten
		new_group.each do |x|
			was_found = was_found || cf.index(x)
		end
		if !was_found
			combined.push new_group
		end
	end
end

# sort, lowest first
pairs.sort!

# create combinations.
# combined has an array of all correspondences.
combined = []
pairs.each do |diff, new_group|

	# insert new group
	add_to_group(combined, new_group)

	# try to recombine everything, until nothing changes any more
	was_recombined = true
	while was_recombined
		new_combined = []
		combined.each do |group|
			add_to_group(new_combined, group)
		end
		was_recombined = combined.size != new_combined.size
		combined = new_combined
	end

	if (combined.size == 1 && combined[0].size == img.slices)
		# we got everything! Write output image.
		img.write(combined[0])
		exit
	end
end

Result

Here is the result I got with this code:

Algorithm

In hindsight, the algorithm combines two ideas: Hierarchical Agglomerative Clustering, and a quick and dirty novel distance-metric that defines how “good” a pairing is.

The hierarchical agglomerative clustering is a bit difficult to code, because there are a lot of corner cases when you are allowed to combine pairings and when not. I am not sure if my code is correctly, this should be recoded without time pressure.

The first version of the distance metric was very simple: calculate the sum of the quadratic differences between the two neighboring pixels when two slices are put together. Unfortunately this does not work for the slices with skyscraper that has black-white stripes: here lots of pixels are white on the left side, but black on the right side since the building is just slightly tilt. After realizing this problem, the solution is simple: for each pixel, find the best matching pixel of the other slice within a certain range. Through trial and error I have chosen +-15 pixels, and finally got the correct image.

Happy hacking,
Martin

Fools ignore complexity; pragmatists suffer it; experts avoid it; geniuses remove it.
– Alan Perlis

Which reminds me of a little code piece I have written recently. I’ve recently tried to implement a small, little parser for a very simple custom data format, in C++. To do this, I have tried several approaches:

1. Boost.Spirit

Since we use Boost in our projects, I have started reading about Boost.Spirit, and took some time to decipher the tutorials which contains code like this:

bool r = phrase_parse(first, last,
  //  Begin grammar
  (
      '(' >> double_[ref(rN) = _1]
          >> -(',' >> double_[ref(iN) = _1]) >> ')'
  |   double_[ref(rN) = _1]
  ),
  //  End grammar
 space);

After half an hour I got annoyed because it simply is too much effort. I don’t care how well thought out the library is and how powerful it is, it is simply unusable. Maybe I am too stupid, but I am sure that even when I manage to understand it enough to write a decent parser, half a year later I can never debug my code again: it’s simply too clever.

2. Coco

I’ve ditched Boost.Spirit, and tried to use Coco. I am unfamiliar with this but have seen a colleague use it, so I gave it a try. I was reading the documentation, which looks nice but has 42 pages and since I am a lazy bastard I stopped right there because I just want to get something working, and quickly.

3. Hand Written Large Switch

I have ditched Cocomo, and started to write my own, very simple code that basically looked like this:

while (instream >> sym) {
  switch (symbol_map[sym]) {
  case START:
    // do this
    break;
  case WHATEVER:
    // do that
    break;
  }
}

After just 10 minutes I got a minimal parser that worked good enough and was extremly readable and understandable code. Everybody with basic C++ understanding can skim over this code and get it. The number of WTF’s per minute when reading this code is close to zero. I am very happy with this approach, and it really rocks because it is so dead simple.

You can rightfully say that a simple switch is very inflexible, and not extensible. You are right, but who cares? Almost all code that I have seen that was planed ahead for flexibility that you might need, gets too complicated because what you planed ahead for might never be needed; even worse: most of the time you need flexibility that you cannot know in advance, it only becomes apparent when you have something and running and use it for a while.

Final Words

Back to the original quote, based on my experience I would extend it a bit:

Ignorants add complexity; fools ignore complexity; pragmatists suffer it; experts avoid it; geniuses remove it.
– Martin Ankerl

If you find this interesting you might also like consider reading the Three Laws of Software Development.

What do you think?

This is an attempt to format the subversion log of a one-month period in the following way:

• Beautiful HTML output.
• Compact representation of lots of information
• Usable with a not-so color rich beamer.
• Fully automatic.

Usage

1. Install Ruby (both 1.8 or 1.9 should work).
2. Download svn-shortlog.rb.
3. Open svn-shortlog.rb with your favourite text editor, and configure the config section according to your needs.
4. Doubleclick svn-shortlog.rb
5. Open the generated changelog_....html file with your favourite browser.

Sample Output

Here is a sample output of one month of boost commits into trunk, taken from the public repository. The output is quite information dense, a quick description is in the screenshot:

Issues

Ideas, suggestions, problems? Please post them as a comment here, at the bug tracker.

Credits

This tool is based on the idea from my colleague Christoph Heindl and inspired by Linus’ Kernel shortlog and Gmail.

• Text over the colored background should be easily readable
• Colors should be very distinct
• The number of required colors is not initially known

Naïve Approach

The first and simplest approach is to create random colors by simply using a random number between [0, 256[ for the R, G, B values. I have created a little Ruby script to generate sample HTML code:

# generates HTML code for 26 background colors given R, G, B values.
def gen_html
  ('A'..'Z').each do |c|
    r, g, b = yield
    printf "<span style=\"background-color:#%02x%02x%02x; padding:5px; -moz-border-radius:3px; -webkit-border-radius:3px;\">#{c}</span> ", r, g, b
  end
end

# naive approach: generate purely random colors
gen_html { [rand(256), rand(256), rand(256)] }

The generated output looks like this:

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

As you can see this is quite suboptimal. Some letters are hard to read because the background is too dark (B, Q, S), other colors look very similar (F, R).

Using HSV Color Space

Let's fix the too dark / too bright problem first. A convenient way to do this is to not use the RGB color space, but HSV (Hue, Saturation, Value). Here you get equally bright and colorful colors by using a fixed value for saturation and value, and just modifying the hue.

Based on the description provided by the wikipedia article on conversion from HSV to RGB I have implemented a converter:

# HSV values in [0..1[
# returns [r, g, b] values from 0 to 255
def hsv_to_rgb(h, s, v)
  h_i = (h*6).to_i
  f = h*6 - h_i
  p = v * (1 - s)
  q = v * (1 - f*s)
  t = v * (1 - (1 - f) * s)
  r, g, b = v, t, p if h_i==0
  r, g, b = q, v, p if h_i==1
  r, g, b = p, v, t if h_i==2
  r, g, b = p, q, v if h_i==3
  r, g, b = t, p, v if h_i==4
  r, g, b = v, p, q if h_i==5
  [(r*256).to_i, (g*256).to_i, (b*256).to_i]
end

Using the generator and fixed values for saturation and value:

# using HSV with variable hue
gen_html { hsv_to_rgb(rand, 0.5, 0.95) }

returns something like this:

Golden Ratio

Using just rand() to choose different values for hue does not lead to a good use of the whole color spectrum, it simply is too random.

Here I have generated 2, 4, 8, 16, and 32 random values and printed them all on a scale. Its easy to see that some values are very tightly packed together, which we do not want.

Lo and behold, some mathematician has discovered the Golden Ratio more than 2400 years ago. It has lots of interesting properties, but for us only one is interesting:

[...] Furthermore, it is a property of the golden ratio, Φ, that each subsequent hash value divides the interval into which it falls according to the golden ratio!
-- Bruno R. Preiss, P.Eng.

Using the golden ratio as the spacing, the generated values look like this:

Much better! The values are very evenly distributed, regardless how many values are used. Also, the algorithm for this is extremly simple. Just add 1/Φ and modulo 1 for each subsequent color.

# use golden ratio
golden_ratio_conjugate = 0.618033988749895
h = rand # use random start value
gen_html {
  h += golden_ratio_conjugate
  h %= 1
  hsv_to_rgb(h, 0.5, 0.95)
}

The final result:

You can see that the first few values are very different, and the difference decreases as more colors are added (Z and E are already quite similar). Anyways, this is good enough for me.

And because it is so beautiful, here are some more colors
s=0.99, v=0.99, s=0.25, h=0.8, and s=0.3, v=0.99

Have fun!
Martin

Write a program that generates all two-word anagrams of the string “documenting”. Here’s a word list you might want to use: wordlist.zip.

When you’re done, send the results to selfdocumenting@hotmail.com.

Good luck!

So this caught my interest and i wrote a little entry in Ruby that is 23 lines long with whitespace and very nice to read. But I won’t show you this code until the contest is over, and this is not the reason for this post. The reason is, that the nice version takes about 2 seconds, and somebody else has coded a Python solution that takes only 1 second (I have no idea what his code looks like). This post is about a fast anagram finding algorithm, and how I developed this algorithm. The final result takes about 0.11 seconds.

Algorithm

The most basic algorithm has two phases:

1. Read in the file
2. Build all combinations of two words and compare the letter count with the query.

Building the combinations is usually done with two nested loops and takes O(n^2) runtime. This is slow, so I have added another step in between:

Idea #1: Filter out Candidate Words

The second step is really slow, but it would be a lot faster if it has to handle less words. So I wrote a little filtering step that lets only words through which are made out of the same letters as the query word.

For example when the query is documenting, the word men or go and even too are extracted, even if the number of letters might not match. But that’s not important, what is important is that the number of possible words are reduced a lot, and so the next phase is faster.

Idea #2: Use a Commutative Hashing Function

String comparisons are slow. To common way to find out if the strings coming with tuned is an anagram of the word documenting is to sort the letters and make a comparison, like this:

irb(main):003:0> "documenting".unpack("c*").sort.pack("c*")
=> "cdegimnnotu"
irb(main):004:0> ("coming" + "tuned").unpack("c*").sort.pack("c*")
=> "cdegimnnotu"

The strings are equal, so we have a match. But this comparison is terribly slow! What’s worse, the computations have to be redone for each match. It would be much better to just compare hash values, and find a hash function to quickly check if we might have a match, and only do the string comparison when the hash check matches. The hash has to be good enough that we don’t have too much false positives (hashes are equal but the real comparisons not) to get a speed advantage. So why not just sum up all the letters bytes?

irb(main):005:0> "documenting".sum
=> 1181
irb(main):006:0> "coming".sum + "tuned".sum
=> 1181

Ruby’s String#sum does exactly this. we can now precalculate the sum for each word, and to find a match we just add the two hashes and compare the result to the query’s hash:

irb(main):007:0> query="documenting"; first="coming"; second="tuned"
=> "tuned"
irb(main):008:0> first.sum + second.sum == query.sum
=> true

When this very quick check returns true, we have to do the string comparison to be absolutely sure it is a match. This considerably speeds up the whole program, but it is still O(n^2).

Idea #3: Reformulate Problem

Now here comes the trickiest and coolest part. Since Idea #2 the slowest part is matching the numbers, with still quadratic complexity. But the hard task is not anagram finding any more, we have reduced it to finding two hashes that combined have the same hash as the query. We can reformulate this problem into something completely detached from the anagram problem:

Given a list of numbers, find all combination of two numbers that add up to a given number

When we concentrate on just this problem and ignore the rest, we might come up with a better way of doing things.

I came up with a fast solution, described below. Somebody posted a better solution that is both faster and simpler, if you want just this final solution skip ahead to Idea #4 as the following description is outdated.

It clearly looks stupid to just try all combinations to add the numbers.
So lets sort them first. Quicksort is fast, especially with numbers, so no worries here. Now consider a list of numbers like this example:

1   3   7   10   10   12   17   20   22   23   24   24   25   26   30

Find all the combinations of two number that add up to 27. They are

• 1 + 26 = 27
• 3 + 24 = 27
• 7 + 20 = 27
• 10 + 17 = 27
• 10 + 17 = 27 (a second time)

You can detect a pattern here: the first number always increases, the second number always decreases! We can now formulate an algorithm for this:

We can have two pointers to the array, one starting from the left side, the other starting from the right side. When the numbers behind the pointers add up to a bigger result than the query (e.g. 1 + 30 = 31), we decrease the right pointer to find a smaller combination (1 + 26 = 27). When the sums are too small (1 + 25 = 26), we move the left pointer to the right (3 + 25 = 28).

This way we walk through the whole array in O(n) time and the sum of the pointers is always kept as close the the desired result as possible. When the pointers meet each other, we can stop the whole process or otherwise we would just reverse the words.

This algorithm gets a bit more complicated when you consider that we might have lots of numbers in it that are equal, whenever this happens you have to fall back into an O(n^2) matching algorithm for just this section.

Idea #4: Use Hash directly

UPDATE Scrap the implementation in idea #3. A blog post here from a reader of this article posted a way to do this really in O(n), without any sorting which is O(n*log(n)). The idea is to use a hashmap that maps from the hash key of the word to its matches:

M = {}
S = the target sum
for each element e in the list
      if M[S-e] exists? (e,S-e) is a pair
      add e to the M

Just use a Hashmap that maps from the cummulative hash of a word to a list of words that have the same hash. Whenever a new word is added, get the list of words that is stored under query.sum - current_word.sum. When the hashes are the same we just have to create a list of all the matches under this key, and check each of the matches sequentially for equality. This is just normal hash collision handling through a linked list. That’s very simple and works like a charm.

I have revised the code, it got both simpler and faster. That’s a win-win situation, wohoo!

The Sourcecode

I hope the code is understandable now with the above explanation. If you have any questions or ideas, please share them here!

#!/usr/bin/ruby

# created by Martin Ankerl http://martin.ankerl.com/

class String
	# creates an array of characters
	def letters
		unpack("c*")
	end
end

class Array
	# converts an array of letters back into a String
	def word
		pack("c*")
	end
end

query = "documenting"
query_letters_sorted = query.letters.sort
txt = File.read('wordlist.txt').downcase

# to quickly check if a letter is part of the query word
used_letters = Array.new(256, nil)
query_letters_sorted.each do |letter|
	used_letters[letter] = true
end

# Maps from cummulative hash of a word to a list of words that have this hash code.
hashToWords = Hash.new do |hash, key|
	hash[key] = Array.new
end

query_hash = query.sum

prev = 0
txt_size = txt.size
separator = 10
idx = txt.index(separator, prev)
while prev < txt_size

	letter_idx = prev

	# no need to check end of word because it is \n
	# which is not part of the word anyways
	while used_letters[txt[letter_idx]]
		letter_idx += 1
	end

	# ignore word if the above quick check fails
	if letter_idx == idx
		word = txt[prev, idx-prev]

		# check all key matches
		key = word.sum
		hashToWords[query_hash - key].each do |other_word|
			if (word.letters + other_word.letters).sort == query_letters_sorted
				puts "#{word} #{other_word}"
				puts "#{other_word} #{word}"
			end
		end

		# insert word
		hashToWords[key] << word
	end

	prev = idx + 1

	# no need to check end of file because we have to end with new line
	idx = txt.index(separator, prev)
end

When you rewrite the algorithm in C++ or Java or Python I am sure it will be faster than this one. But this is not the point of this post. The point is, “The Best Optimizer is between Your Ears” (Michael Abrash, Graphics Programming Black Book).

Have fun!

Screenshot

Unfortunately I don’t really have the time nor the interest to continue development for XDCC-Fetch. Please contact me if you are interested to continue development.

What?

Dice-RPG is a free dice throwing program that can be used for role playing games. Although I have never played a RPG in my entire life, my brother forced me to write this tool for the good of mankind (actually, I wrote it because I did not have anything better to do, but don’t tell him).

Here is a screenshot:

How?

To use Dice-RPG,

1. Install the Ruby one-click installer from here if you don’t have it already. This is a runtime, like Java or C#.
2. Download Dice-RPG.rbw (right click and save the link) and doubleclick it.

Have fun!

The motivation behind this app was that someone thought this had to be in C++ to be fast, so I proved him wrong: search-on-typing with more than 500,000 indexed filenames is easily possible

I have tried to make this little tool very configurable so that is can be useful to everyone. Some of the features are:

• Hierarchical presented search results
• Search-on-typing
• Define rules to execute on mouseclick

Download:

• rfind-1.0.zip

This tool is written in Ruby and requires fxruby 1.0, which is a bit out of date. I will try to update it to a recent version of fxruby soon.

e *.rar

and some files are not extractable, e continuous to extract the other files and prints an error message with all the failed files when it has finished. More info and download of e is here.