Arpith Siromoney 💬

Adventures with mmap

This week I started at the Recurse Center, a self directed program where everyone is working at becoming a better programmer. If you’ve been considering it, you should definitely do it! It’s even more awesome than you’ve heard!

The first project I’m working on is a distributed in-memory datastore. But it’s primarily an excuse to play around with stuff I’ve been reading about and haven’t gotten around to! This is the story of my adventure with mmap.

The mmap function in Go’s syscall package returns a byte slice with the contents of the file you’re interested in. I passed it a file descriptor, the size of the mapping I wanted (this turned out to be the capacity of the returned slice), the memory protection needed and the type of mapping.

It looks like the memory protection needs to match the flags passed when you open the file (to obtain the file descriptor).

func (db *db) mmap(size int) {
    fmt.Println("mmapping: ", size)
    data, err := syscall.Mmap(db.fd, 0, size, syscall.PROT_WRITE|syscall.PROT_READ, syscall.MAP_SHARED)
    if err != nil {
        fmt.Println("Error mmapping: ", err)
    }
    db.data = data
}

MAP_SHARED is what makes this fun — changes you make to the byte array are carried through to the file. My understanding is that this happens “eventually”, which is why msync exists. From this code it looks like it calls fput to do the writing.

The code I’m using to resize my database file looks like this:

func (db *db) resize(size int) {
    fmt.Println("Resizing: ", size)
    err := syscall.Ftruncate(db.fd, int64(size))
    if err != nil {
        fmt.Println("Error resizing: ", err)
    }
}

func (db *db) open() {
    fmt.Println("Getting file descriptor")
    f, err := os.OpenFile(db.filename, os.O_CREATE|os.O_RDWR, 0)
    if err != nil {
        fmt.Println("Could not open file: ", err)
    }
    db.fd = int(f.Fd())
    db.file = f
}

func (db *db) extend(size int) {
    db.file.Close()
    db.open()
    db.resize(size)
    db.mmap(size)
}

I’m using ftruncate to extend the file before mmaping it again. Doing a bunch of ftruncates resulted in “invalid argument” errors after the first hundred and fifty or so (I hadn’t been doubling the size).

On OS X, this is either because the file descriptor references a socket (and not a file) or the file descriptor is not open for writing. So I’m currently opening the file each time I need to extend it, to get a fresh file descriptor. I’m also closing using the old descriptor because I wound up having too many files open.

Things I want to know:

  1. Is there a better way to do this!

  2. When does the kernel write the changes to disk?

Things I want to do:

  1. Reduce the number of times I resize the database file by doubling the size.

  2. Use msync in case the server dies before the kernel gets around to writing the changes to disk.

Some bonus fun stuff:

  1. glibc’s malloc uses mmap!

  2. LevelDB used to use mmap, until this broke bitcoin on OS X.

Special thanks to James Porter, Lisa Neigut and Irina Gossman!