Ost and the future of the web qua web

Ost and the future of the web qua web – Anil Dash’s critique of the contemporary web is apt.

He argues that web services exert excessive control over their users, limiting the public’s creativity.

More so than at any point in the past, the web is a set of vertically-integrated data silos.

Nevertheless our most meaningful content is stored across these silos: thoughts, images, and records of friendships or interests.

Anil sees the short arc of the web’s history inevitably bending back towards freedom.

We’d like to build software that helps get it there.

While shepherding Ost through its private beta, we’ve continuously asked the question: what is the future of the web qua web?

There are some existing visions out there for web services.

Tent.io focuses on decentralization of social data, while App.net focuses on users as customers, not as products for advertisers.

We’re excited by these projects (Ost connects to App.net as a data provider) and we are working on a complementary vision.

Rather than focus on building the one web service to rule them all (i.e., hold all the content), we’d like to build the web service that brings your content together and puts it under your control regardless of where it resides.

While we’re at it, we’d like to make it easy for you to curate and share your content using the existing web infrastructure.

Collections.me is a project philosophically aligned with Ost, but with the key distinction that it brings your content back to the desktop.

Clearly there are benefits to this approach, such as a more responsive native UI, but it feels like a retreat from the web.

With Ost you can send a friend a link to any space you create, with any collection of services (though we remove some data automatically to protect private accounts), and they can view it in their browser.

The early days of the web were exciting because its basic functions were open to anyone.

Creating a blog and connecting it with other blogs was easy.

Now web services provide more than simple links, and we think our tools should acknowledge this.

We don’t interact with static pages, but with streams of data from many sources.

We don’t just link to content, instead we comment on it, favorite it, and reblog it.

The diverse landscape of the web is not going away.

Web services big and small have their own cultures and uses, whether by fiat or convention, and it would be unproductive to combine these communities into to one uber-service.

For us the future of the web is a collection of services tied together through something resembling a free trade agreement.

Ost is a proof of concept for such a future.

So keep your content in the services you love and use Ost to do new and interesting things with it.

We’re pleased to announce that Ost is out of private beta and open to the public.

You can create a free account right here, and if you like Ost you can pay $16 a year for multiple account support and unlimited workspaces.

We have some ambitious goals and we’re excited to move towards them in public view.

Per William Gibson: The future is already here — it’s just not very evenly distributed.

If you’d like a peek, use Ost.…

Build blocks

Build blocks – I remember becoming engrossed in building things with blocks as a child.

I could spend entire days on the floor exploring the possible configurations of a particular tower or imaginary city.

I don’t think (remembering the feeling as best I can) there was an idea of perfection.

Each project had its own way of being that would emerge after a bit of work.

To be building something felt like an act of discovery, as if the thing I was making wanted to be a certain way and I had to figure out what it was.

I remember feeling like the process was impossible to describe but could be refined by patience and attention.

The only way to become better at it was to do it.

These early experiences of making things were of course constrained by the shape of the blocks themselves — well defined objects that are immutable and known.

Why is it that I have these memories of exploring something unknown?

This doesn’t make sense, since the rules of whatever system I happened to be inventing while building were completely voluntary.

I imposed those rules on the blocks, I made them up.

There must be something about the blocks themselves that enabled my close attention.

What caused such intense focus?

I love seeing other people’s homes, and especially the space that they work in.

As adults we operate in a world with a different scale of blocks, but I think the same behavior is still present.

With the appropriately scaled module people can become lovingly engrossed in arranging and preening their environment.

Some people find themselves at Home Depot, others go to Ikea.

I see this behavior persist in the digital tools we use.

Run a perfectly customized Ubuntu or keep all your files on the desktop in OS X.

Perhaps I’m projecting my experience on to the world, but I see a similar experience to the one I enjoyed as a child when I watch people work to arrange their environment.

It’s a somewhat circular thought, but I like to keep this in mind when making things: people make things with the things I make.

Everything I put in to the world becomes a part of someone’s set of blocks.

Finding the appropriate scale of abstraction, the appropriate block size, is a design problem.

We tend to think about design as something similar to how an individual block looks and feels, but equally important is considering how each block participates in a system.

Some individual companies, like Ikea, are really successful at creating systems that allow people to engage at their preferred level of abstraction.

Customize a shelving system or just buy one pre-configured.

Think about individual pieces of furniture or use an existing room design.

Perhaps it’s because of my fond memory of building things as a child, but I like to think that good design is not too prescriptive.

Great systems encourage the people who use them to participate in them and build at the level of abstraction they’re comfortable with.

I see building as a universal and enlivening fascination.

To support that I try not to think of making the world a certain way.

Instead, I keep my attention on building systems for people to build with.…

Rethinking the value of Scrabble tiles

Rethinking the value of Scrabble tiles – When Alfred Butts invented Scrabble in 1938, he based the values and distribution of letters on the frequency of their appearance on the front page of the New York Times.

Today, Butts’ distribution is still the standard for English play.

What has changed in the intervening years is the set of acceptable words, the corpus, for competitive play.

As an enthusiastic amateur player I’ve annoyed several relatives with words like QI and ZA, and I think the annoyance is justified: the values for Scrabble tiles were set when such words weren’t acceptable, and they make challenging letters much easier to play.

So what would a modern distribution look like?

To find out, I’ve developed an open source package called Valett for determining letter valuations in word games based on statistical analyses of corpora.

In addition to calculating the frequency of each letter in a corpus, Valett calculates the frequency by word length and the incoming and outgoing entropy for each letter’s transition probabilities.

One can then weight these properties of the corpus based on the structure of the game and arrive at a suggested value for each letter.

For Scrabble, Valett provides three advantages over Butts’ original methodology.

First, it bases letter frequency on the exact frequency in the corpus, rather than on an estimate.

Second, it allows one to selectively weight frequency based on word length.

This is desirable because in a game like Scrabble, the presence of a letter in two- or three-letter words is valuable for playability (one can more easily play alongside tiles on the board), and the presence of a letter in seven- or eight-letter words is valuable for bingos.

Finally, by calculating the transition probabilities into and out of letters it quantifies the likelihood of a letter fitting well with other tiles in a rack.

So, for example, the probability distribution out of Q is steeply peaked at U, and thus the entropy of Q’s outgoing distribution is quite low.

Intuitively, I’ve long felt that letters like Z and X were overvalued in Scrabble, especially X since it is prevalent in the two-letter word list: XI XU AX EX OX.

In contrast, V and C seem undervalued, with no two-letter words.

Using Valett with an even weighting of letter frequency, frequency by length, and transition entropy I’ve generated a new value distribution that roughly matches my intuition:

A: 1 B: 3 C: 2 D: 2 E: 1 F: 3 G: 3 H: 2 I: 1 J: 6 K: 4 L: 2 M: 2
N: 1 O: 1 P: 2 Q: 10 R: 1 S: 1 T: 1 U: 2 V: 5 W: 4 X: 5 Y: 3 Z: 6
Note: This distribution is calculated with TWL06. G drops from 3 to 2 using SOWPODS.

For the weighting of frequency by length, I most heavily favor two-letter words, three- and seven-letter words, and eight-letter words, in that order.

Incoming and outgoing entropy are weighted evenly.

There are several things I like about this new distribution.

Looking at the statistics, Q is clearly an outlier both in frequency and entropy, and in this distribution it is also an outlier in value.

V bumps up to five points to match X, and close to J and Z, which have dropped to six.

U, as the most challenging vowel, jumps up to two points.

Overall there is downward pressure on the valuations to keep the justified separation from Q at ten points.

Most mysteriously to me, C drops to two points despite its absence on the two-letter word list.

G jumping to three points is also surprising, though it stays at two using the SOWPODS corpus instead of TWL06.

(As a side note, it’s nice that the distribution changes are minor from TWL06 to SOWPODS, as they should be for word lists based on the same language.)

While this distribution is interesting, I’m not suggesting that it’s the best one.

I’m an amateur player and my perspective on the relative importance of frequency vs. transition entropy and frequency at various word lengths is informed by my imperfect knowledge of the game.

By publishing the code, which easily allows one to set all the weights, I hope to enable a data-driven discussion around letter valuation in Scrabble.

More broadly, I think Valett can provide the foundation for answering other interesting questions in word games, such as how to quantify the difficulty of Boggle boards (perhaps useful in a tournament setting as a means of normalization).

To that end I would welcome any pull requests on GitHub that add to the statistics generated from corpora, or add game-specific analyses like the included Scrabble analysis.

If you’d rather not write code but have ideas regarding Valett, just drop me a line!…

A response to Chew’s “Catastrophic Outrage”

A response to Chew’s “Catastrophic Outrage” – John Chew, copresident of the North American Scrabble Players Association, recently replied to my post on Valett, which suggested altering the point values of some Scrabble tiles based on a statistical analysis of tournament-legal Scrabble words.

Chew’s post highlights how an intelligent and skilled Scrabble player might misunderstand what Valett does and its intent.

I’d like to clarify these issues, if possible, and show how Valett’s results suggest changes to Scrabble that are very much in line with its rich tradition and dynamic play.

Fundamentally there are two related but separate aspects of Scrabble: the structure of the game (its rules, board, tile distribution, word list, etc.) and the play of the game (rack composition, board position, remaining tiles in the bag, strategy, etc.).

Because essentially all of the formal analysis of Scrabble (mainly via the excellent software package Quackle) is on the play of the game,

John mistakenly conflates the goals of Valett and Quackle, and tile value and equity value, when they in fact deal with two separate domains of analysis: structure vs play.

Valett puts two specific aspects of the structure of the game in harmony: the word list and tile values.

One could also revisit tile distribution, as John suggests, but that relationship is a bit less formal since you need to artificially limit the number of S tiles.

We could run Quackle on a version of Scrabble where its structure (tile point values) have been changed as per Valett’s suggestions, and you’d end up with different equity values.

So the concept of equity value and tile value are closely related (tile value partially determines equity value), but they’re absolutely not interchangeable.

Tile value is a basic component of the structure of the game, and equity value is a calculated value based on extensive simulation of game play taking into account rack composition, board position, and so forth.

John ignores the distinction between these values in his reply, “Given that the ‘I’ currently has a face value of 1, if you wanted to create a ‘fair’ SCRABBLE game that didn’t penalize players for drawing an ‘I’, you’d want to increase the face value by 2 points to 3.”

Changing some tile values in Scrabble wouldn’t alter equity values in the direct manner Chew suggests.

Later on, Chew mischaracterizes Valett’s tile frequency by word length analysis, “Valett’s requirement that you specify the rates at which words of each length are played is also problematic…”

When weighting frequency by length, Valett is not making any assumptions about how frequently words of those lengths are played, but rather how important words of different lengths are given the rules of Scrabble.

Two- and three-letter words are important in Scrabble because it is a crossword game, and the presence of a letter in a two- or three-letter word makes that letter easier to hook off of existing letters on the board.

Similarly, because there is a substantial bonus for playing all the tiles in one’s rack (which has seven tiles), seven- and eight-letter words are particularly powerful.

Without saying anything about how they might be played, Valett allows one to weight the frequency of letters in these word lengths more highly than at less notable lengths like five or twelve.

Chew is also a bit disingenuous here, “If you did this, you’d reduce a little bit of the luck of the draw, but at the same time you’d be reducing the skill involved in recognizing which tiles are good or bad and playing accordingly.

You’d end up with a game that was a little closer to just rolling a die to determine the winner.”

In fact, reducing the luck of the draw by aligning tile values more closely with the word list (and causing the related changes in equity value and game strategy) would make the game less about luck and more about skill, and therefore further from just rolling a die to determine the winner.

Tournament players benefit from a system with a little less luck because it makes tournaments more accurate.

While something like Elo is reliable in a game with a lot of luck when it draws from a very large sample of games, in tournaments only a certain number of games can be played due to time and stamina constraints.

So the more luck in the game, the less accurate tournaments are in determining who’s the best.

Now, one might object that if I want to reduce the luck in the game, why aren’t I suggesting to remove the blanks, or have a computer distribute tiles to player’s racks fairly.

Well, I like Scrabble, and a game without blanks isn’t Scrabble.

Alfred Butts clearly went to a lot of trouble to get the structure of the game to reflect English use in the 30s, but he wanted the game to have entertaining aspects like blank tiles and the luck of the draw.

Slightly modifying tile values to track changes in the set of legal Scrabble plays respects this tradition and maintains the lucky elements of the game, such as drawing a blank, and the unlucky, such as drawing all vowels.

Valett is an attempt to keep the intentional luck in the game, and remove the unintentional luck that has crept in over time as the use of English has changed.

I hope Chew and I might see eye to eye on that goal, and I look forward to further spirited discussion.…

draft.js: Map changes in the filesystem to events on the client

draft.js: Map changes in the filesystem to events on the client – I studied architecture at a school that really emphasized hand drawing.

I spent a lot of time at a drafting table with a set of old Dietzgen drafting tools.

One thing I came to really like about drafting tools is that they’re atomic.

Finding your preferred workflow with drawing tools takes time, and everyone has their own way of working.

Developing a good relationship with your tools both by using them, and by finding the ones you prefer noticeably improves your work.

It seems like Bret Victor understands this. Light Table looks like a good start.

What I want are discrete tools for making the process of building software faster and more fluid, and I want them to work with the stack I’m already chest-deep in: Node on the server, Backbone on the client.

So I built the simplest shim I could think of that cuts down time between writing code and seeing it work. It’s called Draft, and you can find it on GitHub here.

Draft maps changes in the file system on the server to an array of methods on the client.

See a change in your compiled CoffeeScript? Reload the page. Change in the CSS?

Add a method to refresh it. See a new image file appear? Recreate that element.

Draft comes with the reload method built in, and can be extended however you like.

I wish coding was more like drawing, and I wish drawing was more like coding.

Coding separates thinking about how something will work and seeing it work completely.

This means that I have to wait some amount of time between making a change and seeing it in action.

Drawing is the opposite. I see the changes as I make them, and they become a part of my thought process as I’m drawing.

Drawing is kind of extension of thought. It’s natural and fluid — completely non-segmented, completely smooth.

We use drawing to refine our thinking about something we’re going to make.

Aside from fine art, drawings are rarely the end result.

They sit next to the final product in deep stacks of sketches, schematics, plans, and so on.

This is one of the greatest things about writing code.

The code you write is exactly what performs out in the world.

A sketch in code can become a complete piece of software and still retain the ten lines written as a proof of concept.

I find this really satisfying.

Draft is a tiny start at eroding the gap between drawing and coding, and I find it a lot cleaner than Cmd + Tab, Cmd + R.

It’s also atomic and extensible.

I doubt any piece of software will dominate our entire experience of writing code, just as no piece of machinery dominated the experience of drafting plans.…

collate.js: Concatenate and minify .js/.coffee and .css/.less

collate.js: Concatenate and minify .js/.coffee and .css/.less – collate.js is an open source Node.js module and command line utility for concatenating and minifying .js/.coffee and .css/.less.

At Ost we use collate on our dev machines to watch our client-side scripts and styles and reconcatenate them into a single client.js and styles.css whenever we make a change.

On the server we recommend configuring collate to run via git hook on pushes.

We use browserify for Node-style require in our client code, and collate works just fine on top of it.

We evaluated two other packages before building collate: connect-assetmanager and Buildr.

connect-assetmanager works well but slows down server boot times.

Buildr has a lot of great features, but it takes a more monolithic approach (only one .js/.css output combo at a time) and its CLI isn’t functional at the moment.

Your needs may differ from ours and both packages are worth evaluating alongside collate.

Check out our source on GitHub for installation and usage.…

The Dreaded Self-Narrative, Destroyer of Ambition

The Dreaded Self-Narrative, Destroyer of Ambition – “So what are you up to these days?”

“Um. Well I finished grad school last year and I’m working half time as a postdoc.

And then with the other half of my time I’m working with a friend of mine from high school on a startup.”

“Did you hear that Instagram got bought for a billion dollars?”

“Shut. Up.”

The self-narrative, that peculiar form of performance art with a well-meaning audience of parents’ friends, acquaintances from the past, and distant relatives, is fraught.

You need a good story. It’s hard to make a major life decision without thinking how it will sound as part of the narrative.

Unfortunately the narratives created through big ambitions often have a lot in common with those created through no ambitions.

They’re circuitous, peppered with uncertainty, and a little incoherent.

Even though you’re not making much sense, the audience’s expectations are probably skewed.

Most startups fail. Not so for startups that people have heard of.

That’s also true for actors and musicians, but the difficulty of succeeding in those professions is notorious.

“I’m an engineer at Google,” is simple, easy to admire, and almost mystical in its meaning to many audiences.

Yet that narrative represents a life choice that limits both reward and risk.

My narrative is becoming more complex rather than less.

But I don’t want to choose a less ambitious path just because I can easily summarize it.

I should say, “Just pretend I’m starting a band.”…

Be a Free User, Be a Paid User

Be a Free User, Be a Paid User – Objects tell people what to do with them through their design.

When something fits nicely in to your hand it’s telling you to hold it.

When something is expensive it’s telling you to be careful with it.

Software doesn’t appeal to our intuition the way physical objects do.

Software as a service has the potential to change continuously, in many cases taking the things you made with it along for the ride.

Digital tools don’t keep a separation between the stuff that we make and the tools that we use, which can often be frustrating.

Buying a better hammer is a lot simpler than switching from Facebook to Path.

At its best software gives us new ways of organizing our thinking, and it’s exciting to try out new services for this reason.

Part of what a good service should offer is a sense of how it will evolve either alongside your data or as a steward of it.

Whether you’re a free user or a paying user is only part of the picture: what matters is being able to have a clear sense of the institution you’re participating in.

A project you paid for might disappear as its employees are acqui-hired by another company, and a free project could sell all of your data to someone else.

Companies often make an effort to sell early adoption as a new and exciting experience, and it can be, but you’re also doing some really important work on the behalf of the company.

Your participation helps steer the product.

Early users have the potential to help shape the direction of a company by giving feedback, discussing features and talking with the founders about their usage.

This dialogue (which can oftentimes occur only in the form of usage statistics) actually helps make decisions about what gets built.

This can be a lot of fun for both sides, and there’s no easy formula for deciding what projects are worth your effort to get involved in and which aren’t.

It is likely, however, that the updates made to a hammer or a large software service are only nominally connected to your participation.

We think you should be evaluating us as much as the things we make.

We hope to be as open as possible about our thinking so you can assess both what we make, and how we think about making it.…

Why Grad School is Hard

Why Grad School is Hard – My cohort of PhD students at the UC San Diego Cognitive Science department started with nine members and only two of us finished.

This isn’t an indictment of the department or those who left or moved.

The department is great and so was my cohort. It’s because grad school is hard.

Most students figure it’s hard, but going in it’s not obvious why.

If you’re going to grad school you’re a great student.

Great students impress a small group of people (professors and TAs) through competition with a medium group of people (other students) while doing mostly derivative work.

To succeed in grad school you need to impress a large group of people (those in your field, at least) through competition with that same large group while doing novel work.

This is a sharp distinction and the skills that led you to grad school won’t ensure your success there.

So most graduate students experience a crisis of confidence. Can I do novel work that impresses my peers?

This crisis can then be compounded by pragmatic challenges:

  • You might want to make more money. If you’re in a reputable grad program you should be getting paid, but usually not much. Some of your peers that didn’t go to grad school are making a lot of money.
  • You might not get along with your advisor. It’s a relationship that needs to work well for five years, normally entered into after one meeting in person. Also, your best interests (e.g., finishing) may not match your advisor’s (e.g., having skilled grad students in the lab).
  • Your research might not pan out. An experiment fails, you lose some data, or you get scooped.
  • Even if you finish, getting a tenure-track faculty job directly is rare. Usually you’ll spend several years as a postdoc, which is also no guarantee of a good job.

In combination with any of these pragmatic factors, a crisis of confidence can cause you to leave.

You don’t want to spend five years of your life chasing uncertain prospects with a real chance of failure — especially when you’re used to success.…

What we built with Twitter that now won’t fly

What we built with Twitter that now won’t fly – TL;DR – Check out the demo. Faster, better-looking data from Twitter integrated with other services.

Early this year we set out to build for cloud data what Photoshop is for images: a general purpose tool for manipulation, filtering and publishing called Ost.

We decided to build our MVP on a small set of services that we loved and felt ideologically connected to: Twitter, Instagram, Dropbox, and Instapaper.

At the time these four services felt more like protocols than platforms, a nice set of APIs for creating, consuming, and persisting text and images.

Ost rests on top of these services and provides affordances for working with data both within and between them.

We are particularly proud of what we do with Twitter data and unfortunately we now run rampantly afoul of Twitter’s impending display guidelines.

Check out News, a page we’ve published using Ost that tracks our favorite news sources.

You’ll notice two things. First, our design is dense yet readable.

We put more tweets on a page than twitter.com and TweetDeck, while letting the text breathe and removing visual distractions.

Second, try clicking on a @username. The transition time between feeds (in place, or in a separate panel through the right-click menu) is much faster than Twitter’s native web clients.

Public pages have a lot of tools removed, and with an account you can send a tweet and its associated image to Dropbox with one click,

create and share pages like News, and filter the feeds in various ways. You can also create feeds showing Dropbox and Instagram data.

We think what we’re building is unique, and could help people engage with Twitter data in new and interesting ways.

We think Twitter becomes even more useful to people alongside other services.

As a startup we enjoy the privilege of being small. We can take risks that big companies can’t take.

We can explore design choices that our larger counterparts might not be comfortable even trying out.

We can invent features and experiment with new technology quickly, exploring what might be possible in the immediate future.

We see this work as good for the long-term health of the services we build on: we discover possibilities for innovation and we create goodwill on their behalf.

It’s an inherent risk in building a product like Ost that the services we use change their terms.

We may have to pull some or all of our Twitter integration, or modify it in a way that compromises our goals of readability and speed.

That’s OK. Our vision isn’t about one service, but about the useful interplay between many.

So come fly with Ost before Twitter grounds us.

P.S. We grabbed an App.net dev account. Drop us a line if you’d like to see App.net in Ost.…