Write a program that generates all two-word anagrams of the string “documenting”. Here’s a word list you might want to use: wordlist.zip.

When you’re done, send the results to selfdocumenting@hotmail.com.

Good luck!

So this caught my interest and i wrote a little entry in Ruby that is 23 lines long with whitespace and very nice to read. But I won’t show you this code until the contest is over, and this is not the reason for this post. The reason is, that the nice version takes about 2 seconds, and somebody else has coded a Python solution that takes only 1 second (I have no idea what his code looks like). This post is about a fast anagram finding algorithm, and how I developed this algorithm. The final result takes about 0.11 seconds.

Algorithm

The most basic algorithm has two phases:

1. Read in the file
2. Build all combinations of two words and compare the letter count with the query.

Building the combinations is usually done with two nested loops and takes O(n^2) runtime. This is slow, so I have added another step in between:

Idea #1: Filter out Candidate Words

The second step is really slow, but it would be a lot faster if it has to handle less words. So I wrote a little filtering step that lets only words through which are made out of the same letters as the query word.

For example when the query is documenting, the word men or go and even too are extracted, even if the number of letters might not match. But that’s not important, what is important is that the number of possible words are reduced a lot, and so the next phase is faster.

Idea #2: Use a Commutative Hashing Function

String comparisons are slow. To common way to find out if the strings coming with tuned is an anagram of the word documenting is to sort the letters and make a comparison, like this:

irb(main):003:0> "documenting".unpack("c*").sort.pack("c*")
=> "cdegimnnotu"
irb(main):004:0> ("coming" + "tuned").unpack("c*").sort.pack("c*")
=> "cdegimnnotu"

The strings are equal, so we have a match. But this comparison is terribly slow! What’s worse, the computations have to be redone for each match. It would be much better to just compare hash values, and find a hash function to quickly check if we might have a match, and only do the string comparison when the hash check matches. The hash has to be good enough that we don’t have too much false positives (hashes are equal but the real comparisons not) to get a speed advantage. So why not just sum up all the letters bytes?

irb(main):005:0> "documenting".sum
=> 1181
irb(main):006:0> "coming".sum + "tuned".sum
=> 1181

Ruby’s String#sum does exactly this. we can now precalculate the sum for each word, and to find a match we just add the two hashes and compare the result to the query’s hash:

irb(main):007:0> query="documenting"; first="coming"; second="tuned"
=> "tuned"
irb(main):008:0> first.sum + second.sum == query.sum
=> true

When this very quick check returns true, we have to do the string comparison to be absolutely sure it is a match. This considerably speeds up the whole program, but it is still O(n^2).

Idea #3: Reformulate Problem

Now here comes the trickiest and coolest part. Since Idea #2 the slowest part is matching the numbers, with still quadratic complexity. But the hard task is not anagram finding any more, we have reduced it to finding two hashes that combined have the same hash as the query. We can reformulate this problem into something completely detached from the anagram problem:

Given a list of numbers, find all combination of two numbers that add up to a given number

When we concentrate on just this problem and ignore the rest, we might come up with a better way of doing things.

I came up with a fast solution, described below. Somebody posted a better solution that is both faster and simpler, if you want just this final solution skip ahead to Idea #4 as the following description is outdated.

It clearly looks stupid to just try all combinations to add the numbers.
So lets sort them first. Quicksort is fast, especially with numbers, so no worries here. Now consider a list of numbers like this example:

1   3   7   10   10   12   17   20   22   23   24   24   25   26   30

Find all the combinations of two number that add up to 27. They are

• 1 + 26 = 27
• 3 + 24 = 27
• 7 + 20 = 27
• 10 + 17 = 27
• 10 + 17 = 27 (a second time)

You can detect a pattern here: the first number always increases, the second number always decreases! We can now formulate an algorithm for this:

We can have two pointers to the array, one starting from the left side, the other starting from the right side. When the numbers behind the pointers add up to a bigger result than the query (e.g. 1 + 30 = 31), we decrease the right pointer to find a smaller combination (1 + 26 = 27). When the sums are too small (1 + 25 = 26), we move the left pointer to the right (3 + 25 = 28).

This way we walk through the whole array in O(n) time and the sum of the pointers is always kept as close the the desired result as possible. When the pointers meet each other, we can stop the whole process or otherwise we would just reverse the words.

This algorithm gets a bit more complicated when you consider that we might have lots of numbers in it that are equal, whenever this happens you have to fall back into an O(n^2) matching algorithm for just this section.

Idea #4: Use Hash directly

UPDATE Scrap the implementation in idea #3. A blog post here from a reader of this article posted a way to do this really in O(n), without any sorting which is O(n*log(n)). The idea is to use a hashmap that maps from the hash key of the word to its matches:

M = {}
S = the target sum
for each element e in the list
      if M[S-e] exists? (e,S-e) is a pair
      add e to the M

Just use a Hashmap that maps from the cummulative hash of a word to a list of words that have the same hash. Whenever a new word is added, get the list of words that is stored under query.sum - current_word.sum. When the hashes are the same we just have to create a list of all the matches under this key, and check each of the matches sequentially for equality. This is just normal hash collision handling through a linked list. That’s very simple and works like a charm.

I have revised the code, it got both simpler and faster. That’s a win-win situation, wohoo!

The Sourcecode

I hope the code is understandable now with the above explanation. If you have any questions or ideas, please share them here!

#!/usr/bin/ruby

# created by Martin Ankerl http://martin.ankerl.com/

class String
	# creates an array of characters
	def letters
		unpack("c*")
	end
end

class Array
	# converts an array of letters back into a String
	def word
		pack("c*")
	end
end

query = "documenting"
query_letters_sorted = query.letters.sort
txt = File.read('wordlist.txt').downcase

# to quickly check if a letter is part of the query word
used_letters = Array.new(256, nil)
query_letters_sorted.each do |letter|
	used_letters[letter] = true
end

# Maps from cummulative hash of a word to a list of words that have this hash code.
hashToWords = Hash.new do |hash, key|
	hash[key] = Array.new
end

query_hash = query.sum

prev = 0
txt_size = txt.size
separator = 10
idx = txt.index(separator, prev)
while prev < txt_size

	letter_idx = prev

	# no need to check end of word because it is \n
	# which is not part of the word anyways
	while used_letters[txt[letter_idx]]
		letter_idx += 1
	end

	# ignore word if the above quick check fails
	if letter_idx == idx
		word = txt[prev, idx-prev]

		# check all key matches
		key = word.sum
		hashToWords[query_hash - key].each do |other_word|
			if (word.letters + other_word.letters).sort == query_letters_sorted
				puts "#{word} #{other_word}"
				puts "#{other_word} #{word}"
			end
		end

		# insert word
		hashToWords[key] << word
	end

	prev = idx + 1

	# no need to check end of file because we have to end with new line
	idx = txt.index(separator, prev)
end

When you rewrite the algorithm in C++ or Java or Python I am sure it will be faster than this one. But this is not the point of this post. The point is, “The Best Optimizer is between Your Ears” (Michael Abrash, Graphics Programming Black Book).

Have fun!

Approximation of pow() in Java

public static double pow(final double a, final double b) {
    final int x = (int) (Double.doubleToLongBits(a) >> 32);
    final int y = (int) (b * (x - 1072632447) + 1072632447);
    return Double.longBitsToDouble(((long) y) << 32);
}

This is really very compact. The calculation only requires 2 shifts, 1 mul, 2 add, and 2 register operations. That’s it! In my tests it usually within an error margin of 5% to 12%, in extreme cases sometimes up to 25%. A careful analysis is left as an exercise for the reader. This is very usable for in e.g. metaheuristics or neural nets.

I use Linux, Java 1.6.0-b105 with the server VM, and execute the benchmark with this command:

sudo nice -n -20 java -cp . -server PowTest

The approximation is 27 times faster than Math.pow() on my Pentium-M. On a Pentium 4 it is 41 times faster. Unfortunately, microbenchmarks are difficult to do in Java, so your mileage may vary. You can download the benchmark PowTest.java and have a look, I have tried to prevent overoptimization while still having a low overhead.

Approximation of pow() in C and C++

double pow(double a, double b) {
    int tmp = (*(1 + (int *)&a));
    int tmp2 = (int)(b * (tmp - 1072632447) + 1072632447);
    double p = 0.0;
    *(1 + (int * )&p) = tmp2;
    return p;
}

Compiled on my Pentium-M with gcc 4.1.2:

gcc -O3 -march=pentium-m -fomit-frame-pointer -fno-strict-aliasing

This version is 7.8 times faster than pow() from the standard library.

WARNING! you HAVE to use the -fno-strict-aliasing option, or this does not work!

Approximation of pow() in C#

Jason Jung has posted a port of the this code to C#:

public static double PowerA(double a, double b) {
  int tmp = (int)(BitConverter.DoubleToInt64Bits(a) >> 32);
  int tmp2 = (int)(b * (tmp - 1072632447) + 1072632447);
  return BitConverter.Int64BitsToDouble(((long)tmp2) << 32);
}

How the Approximation was Developed

It is quite impossible to understand what is going on in this function, it just magically works. To shine a bit more light on it, here is a detailed description how I have developed this.

Approximation of e^x

As described here, the paper “A Fast, Compact Approximation of the Exponential Function” develops a C macro that does a good job at exploiting the IEEE 754 floating-point representation to calculate e^x. This macro can be transformed into Java code straightforward, which looks like this:

public static double exp(double val) {
    final long tmp = (long) (1512775 * val + (1072693248 - 60801));
    return Double.longBitsToDouble(tmp << 32);
}

Use Exponential Functions for a^b

Thanks to the power of math, we know that a^b can be transformed like this:

1. Take exponential

   a^b = e^(ln(a^b))

2. Extract b

   a^b = e^(ln(a)*b)

Now we have expressed the pow calculation with e^x and ln(x). We already have the e^x approximation, but no good ln(x). The old approximation is very bad, so we need a better one. So what now?

Approximation of ln(x)

Here comes the big trick: Rember that we have the nice e^x approximation? Well, ln(x) is exactly the inverse function! That means we just need to transform the above approximation so that the output of e^x is transformed back into the original input.

That’s not too difficult. Have a look at the above code, we now take the output and move backwards to undo the calculation. First reverse the shift:

final double tmp = (Double.doubleToLongBits(val) >> 32);

Now solve the equation

tmp = (1512775 * val + (1072693248 - 60801))

for val:

1. The original formula

   tmp = (1512775 * val + (1072693248 - 60801))

2. Perform subtraction

   tmp = 1512775 * val + 1072632447

3. Bring value to other side

   tmp - 1072632447 = 1512775 * val

4. Divide by factor

   (tmp - 1072632447) / 1512775 = val

5. Finally, val on the left side

   val = (tmp - 1072632447) / 1512775

Voíla, now we have a nice approximation of ln(x):

public double ln(double val) {
    final double x = (Double.doubleToLongBits(val) >> 32);
    return (x - 1072632447) / 1512775;
}

Combine Both Approximations

Finally we can combine the two approximations into e^(ln(a) * b):

public static double pow1(final double a, final double b) {
    // calculate ln(a)
    final double x = (Double.doubleToLongBits(a) >> 32);
    final double ln_a = (x - 1072632447) / 1512775;

    // ln(a) * b
    final double tmp1 = ln_a * b;

    // e^(ln(a) * b)
    final long tmp2 = (long) (1512775 * tmp1 + (1072693248 - 60801));
    return Double.longBitsToDouble(tmp2 << 32);
}

Between the two shifts, we can simply insert the tmp1 calculation into the tmp2 calculation to get

public static double pow2(final double a, final double b) {
    final double x = (Double.doubleToLongBits(a) >> 32);
    final long tmp2 = (long) (1512775 * (x - 1072632447) / 1512775 * b + (1072693248 - 60801));
    return Double.longBitsToDouble(tmp2 << 32);
}

Now simplify tmp2 calculation:

1. The original formula

   tmp2 = (1512775 * (x - 1072632447) / 1512775 * b + (1072693248 - 60801))

2. We can drop the factor 1512775

   tmp2 = (x - 1072632447) * b + (1072693248 - 60801)

3. And finally, calculate the substraction

   tmp2 = b * (x - 1072632447) + 1072632447

The Result

That’s it! Add some casts, and the complete function is the same as above.

public static double pow(final double a, final double b) {
    final int tmp = (int) (Double.doubleToLongBits(a) >> 32);
    final int tmp2 = (int) (b * (tmp - 1072632447) + 1072632447);
    return Double.longBitsToDouble(((long) tmp2) << 32);
}

This concludes my little tutorial on microoptimization of the pow() function. If you have come this far, I congratulate your presistence

UPDATE Recently there several other approximative pow calculation methods have been developed, here are some others that I have found through reddit:

• Fast pow() With Adjustable Accuracy — This looks quite a bit more sophisticated and precise than my approximation. Written in C and for float values. A Java port should not be too difficult.
• Fast SSE2 pow: tables or polynomials? — Uses SSE operation and seems to be a bit faster than the table approach from the link above with the potential to scale better when due to less cache usage.

Please post what you think about this!

Markus Holzmann, an intern at Profactor of my fellow colleague Philipp Hartl, had the opportunity to experiment with Ajax during his job. He wrote a tutorial about how to push events from the server to the client. For example, display popup messages on all browsers at the same time (see screencast in full resolution here):

Cometd Hello World

I’ve read Chris Bucchere’s Say Hello World to Comet and built an application based on this using a more current version of Jetty (version 6.1.5) which I embedded into a Tomcat v5.5 Server. For the developing I used Eclipse 3.2.

Start your Engines

At first you have to get the server running. As I mentioned I embedded Jetty into a Tomcat server. Therefore you have configure the libraries:

1. Add the packages org.mortbay.cometd and dojox.cometd to your source folder and delete the client package in the org.mortbay.cometd package.
2. Add jetty-util-6.1.5.jar, jetty-6.1.5.jar and servlet-api-2.5-6.1.5.jar to your build path.
3. Copy the jetty-util-6.1.5.jar file into the /lib folder in the WEB-INF directory.

Replace the existing servlets in your web.xml – file in the WEB-INF – folder with the following servlets:

<servlet>
  <servlet-name>cometd</servlet-name>
  <servlet-class>org.mortbay.cometd.continuation.ContinuationCometdServlet</servlet-class>
  <load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
  <servlet-name>cometd</servlet-name>
  <url-pattern>/cometd/*</url-pattern>
 </servlet-mapping>

For the project I used the dojo toolkit (version 0.4.3) which has an integrated COMETd class that makes it easy to build comet projects. Download it and add it to your WebContent folder.

When you’ve done all this, the hardest piece of work for this program is already done.

Hack the Code

Now you can implement the code for the client side: You need a HTML file with a button on it. The code for this looks like this (download):

<html>
  <head>
    <script type="text/javascript" src="../dojo.js"></script>
    <script type="text/javascript">
      dojo.require("dojo.io.cometd");

      cometd.init({}, "cometd");

      cometd.subscribe("/hello/world", false, "publishHandler");

      publishHandler = function(msg) {
        alert(msg.data.test);
      }
    </script>
  </head>
  <body>
    <input type="button"
       onclick="cometd.publish('/hello/world', { test: 'hello world' } )"
       value="Click Me!">
  </body>
</html>

Line by line, the above bold code works like this:

1. In the line

   <script type="text/javascript" src="../dojo.js"></script>

   you integrate the dojo toolkit into the project.

2. To activate the cometd class of dojo:

   dojo.require("dojo.io.cometd");

3. Connect the server with the client:

   cometd.init({}, "cometd");

4. Here we say what to do when there is a subscribe event:

   cometd.subscribe("/hello/world", false, "publishHandler");

5. Last but not least, the publishHandler function serves as the callback function, which uses alert to show a simple message box:

   publishHandler = function(msg) {
     alert(msg.data.test);
   }

Give it a Try

When you load the HTML file now, you can click on the button and an alert box saying hello world will appear:

The reason for this is that when you click the code

cometd.publish('/hello/world', { test: 'hello world' } )

is executed which publishes a text on the channel with the id /hello/world.

The funny thing is that this is able to run on any number of browsers. Everytime when a client clicks the button, on all browsers that view this page the alert box is shown. (See screencast above).

Pushing Data from Server to Client

You can also add serverside code to trigger an event. I wrote a JSP file with the following code:

<%@page import="java.util.*"%>
<%@page import="dojox.cometd.*" %>
<%
Bayeux b = (Bayeux)getServletContext().getAttribute(Bayeux.DOJOX_COMETD_BAYEUX);
Channel c = b.getChannel("/hello/world",false);

Map<String,Object> message = new HashMap<String,Object>();
message.put("test", "jsp: hello world");

c.publish(b.newClient("server_user",null),message, "new server message");
%>

When this page is loaded, an alert popup appears at the page saying jsp: hello world.

That’s it. Happy hacking!

The highlighting looks like this:

Here is how to get this to work:

Install Erlang Language File

1. Download the file erlang.lang.
2. Copy the file to /usr/share/gtksourceview-1.0/language-specs/erlang.lang.
3. Start gedit, open an Erlang file, and choose View > Highlight Mode > Sources > Erlang.

Automatically Recognize *.erl

If you want gedit to automatically recognize that all .erl files should be correctly highlighted, you have to define the mime type (more info is here):

1. Create directory to override mime types, in the command line type

   mkdirhier ~/.local/share/mime/packages

2. Download the custom mime file Override.xml into this directory (if the file already exists, you have to copy the relevant lines by hand):

   cd ~/.local/share/mime/packages
   wget http://martin.ankerl.com/files/Override.xml

3. Update the mime database by merging the file:

   update-mime-database ~/.local/share/mime

4. Restart nautilus (or logout & login again):

   killall nautilus

There you go, Erlang code in all its glory. Happy hacking!

This program is only a little prototype, that shows that this technique seems to work. It’s public domain, feel free to do whatever you like with it:

Download

textanalyze.rb, Licence: Public Domain.

Example

1. Build an index with a reasonably large amount of data, it should be much larger than the text you want to analyze. For example, I have indexed 76 of Grimm’s fairy tales with this command:

   cat *.txt | ruby ../textanalzye.rb c

   This creates the file wordcount.dat that contains the word count of each word.

2. To find out which words are characteristic for a specific text, the previously generated reference data is used. To continue the example:

   cat LittleRedRidingHood.txt |ruby ../textanalzye.rb a

   This produces the output

   hood, grandma, riding, hunter, red

   So the above words seem to be very relevant to LittleRedRidingHood.txt when compared to all of Grimm’s tales.

Other Uses

The previous example seems a bit useless, but there certainly are a lot of useful applications. Here are some ideas:

• Quickly find out what an unknown text is about
• Automatically extract important words from blog entries
• Find out what a text is about by reading just 5 words
• Automatically create very short descriptions for a large number of documents

The currently implemented algorithm even works well with HTML files (To my own surprise. Actually, I am surprised that it works at all…)

Algorithm

The main idea is quite simple: the algorithm assumes, that important words are :

1. Often used in the to-be-analyzed text
2. Seldom used in other texts

For example, the second condition ensures that words like “the”, “and” etc. are not considered important.

The full algorithm to calculate the score of a word (higher==more important) is done with this formula:

tanh(curVal/curWords*200) - 5*tanh((allVal-curVal)/(allWords-curWords)*200)

The variables:

• curVal: How often the word to score is present in the to-be-analyzed text.
• curWords: Total number of words in the to-be-analyzed text.
• allVal: How often the word to score is present in the indexed dataset.
• allWords: Total number of words of the indexed dataset.

Please don’t ask me how or why this works. I have no idea. I have invented this formula in one of the rare moments when I was enlighted for approximately 10 seconds, quickly wrote it down, and immediately forgot how it worked because my mind was overwhelmed by its beauty and simplicity… Or something like that

1. A developer must write code that creates value.
2. A developer must make their code easy to maintain, except where such expenditure will conflict with the first law.
3. A developer must reduce their code to the smallest size possible, as long as such reduction does not conflict with the first two laws.

That’s about all there is to software development. Print this out, make copies for all your coworkers, and engrave it in stone plates so that our childrens children will still be able to live by these timeless wisdoms!

Lately I was busy adding tests that are able to verify if some code (e.g. an optimizer that uses random, like genetic algorithm, simulated annealing, …) produces the desired in e.g. 95% of the cases (Wikipedia has a nice practical example for confidence intervals).

Example Usage

Here is an example that tests the nonsense code Math.random() * 2.

ensure(new Experiment() {
    public double measure() {
        return Math.random() * 2;
    }
}).between(0.9, 1.1, 0.95).sample(10, 100);

The code most likely does not make much sense out of context like this, so here is an explanation of what it does:

Explanation

In that example we want to verify that the code returns values whose 95% confidence interval is between 0.9 and 1.1. At first take 10 samples to verify this. If the mean is not as expected between the interval, there is still the chance that this was just bad luck. We can take further samples (up to 1000) to rule bad luck out. When we have to take more than 1000 samples and the mean is still not in the specified still not met, there is a very high probability that the code does not produce the desired result, so we have to fail.

I hope the above description is somewhat understandable. It’s no piece of cake, and there is no way to cheat around the complexity involved with that kind of statistical tests. I have tried to make the interface as clean and simple as possible, suggestions are always welcome.

Open Source?

I am developing ensure4j partly at work, and partly in my free time. I would like to make it open source, but I need to get the OK from my boss first.

ensure4j is the the only implementation I know of that uses statistical tests that can be used with JUnit. If you know something similar than this, please post!