Auto Scroll
Select text to annotate, Click play in YouTube to begin
so since is splash this is a programming conference I want to start by reassuring you that this talk is in fact about trading programs okay I want to start
with that because it might not seem like that's what I'm talking about I'm not going to be talking about writing code I'm not gonna be talking about building software and we normally think about it but this is very deeply about creating
programs and kind of in the same way that we create sounds when we talk we create letters no you're right this is about creating programs as a means of or person-to-person communication as a
means of representing thought this is this is kind of a personal talk in certain ways and so I wanted to start out just with a little bit of personal background food so you can kind of see
where I'm coming from I got my start like you know many people you know making games and apps and whatnot I think I kind of found my stride designing creative tools made a number of musical instruments visual design
tools that sort of thing that start gave me interested in UI design and after studying that I went to Apple inventing and prototyping UI concepts for experimental projects which ended up influencing these things and perhaps
other things yet to come and it was also very interested in information design I helped Al Gore make a book app about climate change designed a series of interactive information graphics to
enable readers to understand things in ways that be difficult to understand and text and then I got on a train and started living on the train it's a
wandering research hobo among other things designing new kinds of reading material where it's not the author lecturing at the reader but the author is guiding the reader through exploring dynamic models simulations that the
authors provided the read archive has the agency to try different things out to say what if to critique the model be more active people are participating reader so making these things and then
also prototyping these different creative tool environments for for dynamic things so tools for working with mathematical systems for making software making electronics animation tools for making
things that have behavior that do stuff and my focus always seems to be on the representation of that behavior how could author x' see what the thing they were making is actually doing and what
are powerful ways of seeing so they can deeply understand what the thing is doing and understand it in many different ways so this notion of
representation representation of dynamic behavior seemed to be very central and so I made all these things and made all these prototypes and this kind of hobo period was just looked good you know a
couple years or so and then over the last year I've just kind of been reflecting on all this this raw material and trying to trying to hear what the
prototypes are trying to say because all those things that that I showed you know each one of them was kind of individually it's sort of useful in its own little domain but that wasn't really the point of any of this all those were kind of a vantage point to get a glimpse
of something much bigger kind this new way of working and thinking in the dynamic medium and that that bigger picture is what I want to try to talk
about today so this talk has two parts the first half is about the past and the second half is about the future in the first half I'm going to try to make the argument that representations by which I
mean the the ways we externalized thought have been responsible over the last two thousand years in large part for the intellectual progress of humanity enabling us to think thoughts that we couldn't think before but while these media that we've invented have
empowered us in certain ways they've also crippled us in other ways and we now have the opportunity to design a new medium of thought that undoes some of that damage that's both humane and empowering so that's the first half and
the second half I'm going to try to sketch out what that medium might look like so what it might be like to communicate to work to think in a humane medium so we'll start with
representations what I mean by representation is the physical form of a thought it's something out there in the world that you can sense with your senses and manipulate with your body and
as an example I want to tell you about when my favorite forms of representation of all time which was invented by William Playfair in 1786 William Playfair was writing a book about England's trade with other nations and
he had a lot of data and back then data was always expressed it was always represented in tables of numbers if you're doing economics or business or science you had these tables and numbers you had to learn how to read and write
them and data and tables of data they were the same concept you couldn't tease them apart the data and the way they were representing the data people couldn't think of them separately and playfair had a very interesting thought
he realized that there's a certain form of understanding that we bring to the table the numbers but there's a very different way of understanding that we bring to things like maps there's kind
of a different set of perceptual and cognitive capabilities that we bring into play when we're looking at a map and playfair had the unprecedented thought of making a very peculiar kind
of map where instead of left-right being west east it would be earlier later instead of up-down being north/south it would be more money less money and so he drew this funny kind of map which was
the first data graphic the first time anybody plotted data and people immediately caught on to like wow this is a really powerful way of understanding data as these things happened it didn't people got it but it
didn't actually catch on as a thing for another hundred years or so but eventually it did and today of course the data plot underlies almost all of modern science and engineering most sprints as a science and engineering
will be almost unthinkable without the plot it's just totally fundamental to everything we do to the extent that it almost seems obvious right and it's hard to realize that we've only invented a couple of years ago so
the big point here and this is kind of what I want you to hold through this entire talk what play fair did was he realized that human beings have a certain capability for understanding maps for understanding this kind of
cartographic abstraction and this capability was not being used when they're trying to understand data and so we invented a representation which represented the data in such a way that this latent capability could be brought
into service right and that's what representations do that's how they work their magic is they draw on certain latent innate capabilities that we have and your capabilities that originally we
in evolved for like Yoona in a hunting gathering context and bring these latent capabilities into service repurposing them for a more abstract purpose than
they're originally intended for and you can see the sort of pattern in play in many of the great representations throughout history in mathematics of course one of the first amazing ones was
the invention of Arabic police value numerals you can't do or thematic in Roman numerals you can't think arithmetic in Roman numerals you have to like go over to your abacus police value
Arabic numerals were the first user interface for numbers that allowed calculation to be done on paper this was totally transformative same sort of story with the invention of algebraic notation as opposed to like writing out
imperative knowledge and big blocks of text you could write things declarative and small number of symbols absolutely transformative leibnitz's notation for the calculus Leibnitz was the UI
designer of the 17th century he was obsessed with notation always trying out different notations always like talking with his friends about notation because he realized that a lot of the power and
idea lies in the form in which it's expressed because that's what allows people to think it and in almost any given field you can go in there and find a particular form of representation
which transformed or ignited the field so the periodic table for example Mendeleev correct table is basically the beginning of a theory industry that's where it all started Bohr's model of the atom which of course
was originally inspired by Copernicus's model of the solar system Faraday's representation of magnetism as lines of force which influenced Maxwell to make his theory
although Maxwell did not write Maxwell's equations Maxwell wrote 20 equations and 20 unknowns kind of scattered throughout his paper it was absolutely a brilliant theory but there's something much deeper
there that nobody could see and it was Oliver Heaviside who invented the language of vector calculus specifically for the sole purpose of being able to write Maxwell's equations in that kind
of compact for lime symmetric form so of course that representation of Maxwell's equations has been a very influential but more so heavy sides representation
of the vector basically powered 100 years of physics so when you're thinking of the great ideas of history often what you really want to be thinking about is the great representations that enable people to think those ideas so ideas
live in representations and the representations in turn have to live in a medium so there's different me of thought there's some ideas that you can talk about in speech there's some ideas that you are carried well in song or
theater but the biggie of course was print the invention of the printing press 500 years ago which basically made all that stuff on the left possible all that stuff on the left was basically designed for the medium of print with
the possible exception of numerals but even those they've been around for a few hundred years but they're basically kind of a niche thing until print came along and that's when they started to come widespread so a powerful medium is what
enables powerful representations to be used and seen and spread so you can think about the invention of powerful representations and the invention of powerful media to host powerful
representations as being one of the big drivers of last 2,000 years of the intellectual progress of humanity because each representation allows us to think thoughts that we couldn't think before we kind of
continuously expand our think about territory so you can think of this as tied to you know the grand meta-narrative of the scent of humanity moving away from myth and superstition
and ignorance and towards a deeper understanding of ourselves in the world around us I bring this up explicitly because I think it's good for people to acknowledge the motivation for their
work and this is this story of the intellectual progress of humanity is something that I find very motivating inspiring and is something that I feel like I want to contribute to but I think
that if this if you take this as your motivation you kind of have to be honest with yourself that that there definitely has been ascent we have improved in many
ways but there are also other ways in which our history has not been ascent so we invent technology we media technology
to kind of help us make this this climb but every technology is a double-edged sword every technology enables us has a potential de navels in certain ways while debilitating us in other ways and
that's especially true for representations because the way the reputations work is they draw on certain capabilities that we have so if we go all in in a particular medium like we
did with print so the capabilities that are not well supported in that medium they get neglected in they atrophy and we atrophy I wish I knew who drew the picture
because it's it's a wonderful depiction of what I'm trying to express here and even a little misleading because the person the last stage they're kind of hunched over is tiny rectangle we reach
that stage accomplish that stage with the printing press and cheap paper book based knowledge the invention of paper-based bureaucracy paper-based
working we invented this lifestyle this way of working where to do knowledge work meant to sit at a desk and stare at your little tiny rectangle make a little motions of your hand you know started
out as sitting at a desk staring at papers or books and making little motions with a pen and now it's sitting at a desk staring at a computer screen making a little motions with your on a keyboard but it's basically the same
thing we've this is what it means to do knowledge work nowadays this is what it means to be a thinker it means to be sitting and working with symbols on a little tiny rectangle to the extent that
again it almost seems inseparable you can't separate the representation for what it actually is and and this is basically just an accident of history this is just the way that our media
technology happen to evolve and then we kind of designed a way of knowledge work for that media that we happen to have and I think so I'm going to make the claim that this style of knowledge work
this lifestyle is inhumane and I mean something very precise by that so I'm going to try to explain with an analogy imagined imagine you adopt a puppy so
your Duff is puppy and your name um puddles you take puddles home and you give them a little snack and you stick puddles in the cage and you lock the door forever and never open it again and
puddle slips out the rest of his poor life trapped inside this little cage so that's wrong I think most of us would agree that's sadistic that obviously inhumane so well let's ask what exactly
is wrong about that you know what what is inhuman about sticking puddles in the cage well you know we kind of have a notion of what it means to live a full doggy life right dogs have to run around dogs run
around and they sniff other dogs and they pee on things and that's kind of what it means to be a dog they've inherited this the set of capabilities capabilities and traits from their wolf ancestors and we recognize that a dog
has to be allowed the full free expression of its entire range of capabilities by sticking the cage or constraining his range of experience you're not letting him do all the things that dogs can do and this is exactly
what we've done to ourselves we've invented media that severely constrained our range of intellectual experience that of all the many capabilities that
we have all the many ways of thinking that we have we have constrained ourselves to a tiny subset we're not allowed to use our full intellect and to see that I think we need to take a look
at you know what is that full range of capability is what are all the different ways we can think about things and it turns out it's a very vast and amorphous space our modes of understanding so
there's been number of different attempts to add structure to that space to carve it up one way of looking at it is by looking at the different sensory channels that we use to make sense of
things so an example if I say music you'll probably think oh that's something that we understand arleigh right but there's another representation of music which is visual which is sheet music which allows us to understand
music and a very different and very powerful way if you're making music then you're often thinking about it tactile II it's a tactile understanding it's an interaction between your hands and the instrument you could define
dance as an understanding of music which lives in the body which lives in the movements of the body and if you've ever played in an orchestra or band you can appreciate a kind of spatial understanding of music where you know
the guitarist over there the drummer's over there you're not thinking about music in the spatially distributed way if music is just an example there's any number of things which can be represented at any or all
of these these forms these channels and it's a more than the sum of their parts thing that we have multiple representations they compound and reinforce each other incredibly powerful ways and allow to allow for
understandings that would not be possible in any single channel so that's one I guess one attempt to draw basis vectors into this kind of space of understandings another useful one was
presented by Jerome Bruner have adapted from PJ's theories so he talked about the action based image based and language based understandings writing bike is something that you understand by doing it you can't talk about how you
ride a bike it's the understanding lives in the Performing of it there's an image based understanding you can look at the drivetrain of bike and see how the sprocket is pulling the chain or derailleur and you have understand this complex mechanical system in this image
based wordless way and then of course there's language based understanding understand you can articulate you can talk about parts of like you can talk about gear ratios and calculate them I think most of us in this crowd tend to
gravitate towards the symbolic and so I think it's important you know maybe to to remember that speech is purely symbolic and the most the most important
invention in the history of humanity was the invention of writing where he took speech and turned it into an image based iconic second channel form right so that's what happens when you get multiple channels involved as you lead
to leads to understanding that would not be possible with any any single one I'm gonna put up with a couple others just to kind of give you a sense of the breadth of this space so there's Howard Gardner's multiple intelligences some
people abuse this to say things like Johnny is visual and Sally is verbal or whatever you know don't do that but the way to use this is to realize that every human being has all these capabilities
and we can design reputation across them especially like gardeners because he explicitly calls out empathetic forms of understanding the way that we under and other people understand ourselves understand nature we can use all of
these capabilities my current favorite right now is cured Ekans framework he's got inspired by Vygotsky and it's a little too subtle to talk about here but if you're interested in this sort of thing I recommend you read Ian's book
the educated mind because it's fantastic so the point of all of this is really just that the space of the ways we understand things the space of our
cognitive capabilities is so vast and so diverse that even people who have devoted their entire careers to trying to reduce it down into a neat little Theory have come up with very different answers because there's there's so much
there it's such a rich space let's just focus on these two for now so basically every every circle up there is a superpower right every circle up there
is a capability that we've been honing for hundreds of thousands of years that we use in you know innumerable ways and can combine with the other ones in really powerful ways and what happened
with the invention of the printing press in the invention of tiny rectangle based knowledge work is this so all that stuff drops out and we're working with visual
symbols right we're reading visual symbols or manipulating visual symbols that's what it means to do intellectual work nowadays and you might think oh okay maybe that was true with like paper
and books but you know with computers it's getting better right and no with computers it is getting worse so with a book you know a book is at least a physical object which exists in the world if some amount of tactile response
you can hold it and move it around with a book you can make a shelf which is a spatial representation of knowledge it's not a very good one but it's at least something that you can understand spatially when you're writing with ink
on paper you can move freely between drawing imagery and writing in language because the paper doesn't really care what kind of marks you make on it so you have the freedom at least to move between those two modes but then we
invent these things and they have these flat glassy screens that have no tactile response and that drops out and their expensive little screens they take up a little tiny
portion of your field of view and the spatial stuff drops out and they have these keyboards basically the only convenient thing to do is punch in symbols especially for us and anything from writing an email to write a
computer program anything other than just typing in letters is incredibly cumbersome regardless what you're trying to express you express it in symbols because that's what the interface encourages you to do so that drops out
and so this is the cage that we have trapped ourselves in this is the way in which we have constrained our range of experience which we have created a tiny
subset of our intellectual capabilities and restrict ourselves to this tiny subset and have forbidden ourselves to use our full intellect so there's two
things wrong with this one is that it's inhumane so in the same way you know keeping puddles in the cages inhumane because he can't do all those doggy things using media that restrict our thinking to this tiny subset is inhumane
because we can't do all our thinking things we can't think in all the ways that human beings are able to think so that's there's kind of a moral argument there if you don't buy that there's also a practical argument which is that is
just wasteful one way to think about it is you know as programmers matching you have like an equal processor and you're writing some code it's kind of a sequential algorithm can't parallelize it so you're running this code it's
maxing out one of the cores and the other seven pores are just idling right and that that hurts right like as a program you kind of get this feeling in your stomach like you
know god this is so inefficient like if only I could paralyze algorithm OOP like there's just like so much latent power I can draw on and that's exactly the emotion that I get when I'm looking at this tiny rectangle based knowledge work
that we do is that the there's all these cores have all these capabilities and they're just sitting there idle and if we could only paralyze our representations across all these capabilities who knows what
would be capable of thinking so the good news is that we now have an opportunity to do something about this so we're now at a I believe unique moment in history where we're inventing the next medium of
thought after the printing press we're inventing the dynamic medium so in the same way there's certain thoughts that can be conveyed pretty well in speech then thoughts can fade in theater or in
print there's I believe a very wide range of thoughts that can be conveyed in programs once we understand understand how to do that so this medium that we're inventing the dynamic medium
has three very interesting properties one is that its computational which means that it's capable of simulation can simulate processes out in the world it's responsive so unlike a book where you poke at the book and book it doesn't
do anything dynamic things can respond to stimuli so when they're simulating stuff in the world they can respond and show you the results in many different ways and it's connected so unlike a book here in a book there they can't you know
they're isolated objects dynamic material can exchange information and talk to each other and so what this means if you're trying to represent something like a system today in language if you're trying to represent a
system you describe it you talk about it it's a very poor way of representing it in a computational medium you can model it you can actively model and explore that system and then see what's doing in
many different ways and you can start to fill in this we've got a responsive medium and you know today we mostly think of computational responsiveness as a visual thing you know you've got a screen and it does
stuff and some amount of aural responsiveness and I think things are only going to get really interesting once we have tactile responsiveness I'm not really talking about haptics here I'm talking basically about taking
computation out of the screen and into the physical world so infusing physical matter with computation and letting kind of dynamic responsiveness be something that exists
in the world and something you can touch and feel and interact with with your hands the way we've been working with tools for millions of years so once we have kind of a full physical responsiveness we can start to fill that
in and as a connected medium that means that you have a space filled with dynamic material it's not all isolated they can talk to each other they can work together to make large-scale
representations so we no longer have to work with like little things that we hold in our hands but all the material can host things that our body scale or
room scale start to fill that in and we can start to have the potential at least to design a medium that is humane and empowering and these are two sides of
the same coin humane meaning allowing us to draw on our full range of capabilities getting us out of the cage allowing us to express our full intellect and empowering meaning taking advantage of that drawing on our full
intellect to enable people to think thoughts that they couldn't think before so in the second half of the talk I want to take you through a little sketch of what it might be like working in a fully
dynamic medium how thoughts would be represented in these different modes person-to-person one-to-one conversations or one-to-many hesitations thing two-person things like what does it mean to read wired knowledge from a
knowledge object or to browse many of them person to thing what does it mean to author the sort of material or to use it as the way of thinking so this is 40
years out as we may think or Dynabook kind of thing but it's not really intended as a prediction so much as just a guide post to kind of steer the ship towards I found it very useful as
guidance for my own work and so I hope you'll find it useful as well I am only going to be talking about the human experience of working and thinking in this medium I'm not really going to address the
implementation of this stuff the the technology that underlies it isn't really the point here but I've been reminded that there are people at this conference who are very interested in technology and I would be remiss to
leave it unaddressed so here's the part of the talk about technology I'm taking the liberty of inviting in legendary and mighty professor Gerry Sussman author of scheme and SiC tea and Sikkim and all
these wonderful things and Sussman is going to tell us about the technology of the future but here in the future it's going to be the case that that computers are so cheap and so easy to make that
you can put in the size of a grain of sand complete with a megabyte of RAM you're going to buy them by the but by the bushel I suppose you can pour them into your concrete and you buy your
concrete by a mega flop and you have a wall that's smart okay so long as you can skip the power to them and they could do something you know that's going to happen right it's sort of the way I said remember your cells are about are
pretty smart and they but they seem to talk to each other and do useful things so we have to begin to worry about that kind of a world that'll happen it's going to happen there's no other way to make the future happen
so if Sussman says it's going to happen that's good enough for me and as he says we have to begin to worry about that kind of a world what he's worrying about in this case is the engineering problem of how do we build
working reliable systems out of this kind of distributed computational material what I'm worrying about and I'm hoping to impress onto you as well as the humanist problem of what should we
build and why are we building it and what will we do with it and what's what is it going to do to us right because this technology they talking about you know this kind of you know 300 computers per square inch or
whatever infused into the into physical matter that that sounds like really far off right that sounds like you know we've got time but you know technology does the exponential thing and we're always blindsided by Exponential's so
this technology that is talking about it's going to be here before we know it and so it's kind of up to us to prepare for that you know we have the choice to to say okay let's start envisioning and
sketching and pro prototyping designing designing something that we you know understand to be humane so then when the technology is ready we're like okay we know we know what we want to build with this we know it's going to work we know
it's going to be good so you can do that or you can just kind of do the incremental thing and kind of you know jump on whatever technology bandwagon comes on and let technology follow its own course and I can guarantee you that
that is not going to be humane that result is just going to lead you into a tighter and tighter cage because when technology takes its own path it finds certain veins of the capable
incapabilities and just kind of like drills further in those and leaves the rest of the capability landscape untouched so even though it doesn't seem feasible right now I think we need to
start thinking about how to design the humane medium with the technology that we want to have so let's start off with the most fundamental active communication which is a real-time
conversation between two people so this is something that we've been doing for hundreds of thousands of years and we do it nowadays basically in the same way that we have been doing
at 400,000 years is by spoken words going back and forth that's how we communicate that's how we converse is like spoken words going back and forth with our hands around maybe maybe we'll like draw some sketches on the
whiteboard or a cave wall and there's some things that you can represent well in in that medium and there's many things especially kind of modern things that we need to talk about nowadays
which are not well represented in spoken language and one of those is systems so for so we live in an era of systems there's natural systems like the
environment eco systems biological systems pathological systems there's the stuff that we make political economic infrastructure Allah make out of
concrete and metal electronics and the wrong way to understand a system is to talk about it to describe it the right way to understand it is to kind of get in there and model it and explore it and
you can't do that in words and so what we have is people are using these very old tools people are explaining and convincing through reasoning and
rhetoric instead of the newer tools of evidence and explorable models we want a medium that supports that so we can start to think about a medium a medium for conversation that is naturally show
until we're depicting and describing or on equal footing it's just as easy to create an image as it is to create a description and what you are depicting
are not static images as we do today but actual working models you're creating simulations you're creating programs and so for example if you want to talk about how in this case how a
airplane wing generates lift if you want to talk about the effect of some policy change all these things can be modeled we want to have a medium where you can model them and then explore them and have be the content of the conversation and
you want these models that you're building to be evidence backed you want them to draw upon the facts and knowledge that we know about the world incorporate them into the model and have everybody be able to see the the facts
that you're bringing in the provenance of the facts everybody see everyone to see how the models working and so this kind of leads to the first first of several big juicy research questions
which is how how do we do this kind of modeling because kind of the way that we create dynamic models today is with this activity that we call programming which is kind of going off and staring at a
screen and bashing on a keyboard for a few hours or weeks and I'm talking about getting this down to seconds improvising sketching dynamic models in real time as part of the real-time give-and-take of
conversation we don't know how to do that we don't know how to create working dynamic material in seconds and you might think that's impossible like how can you program in seconds but that I
would agree this probably isn't possible if you're thinking about staring at a screen and bashing on the keyboard I'm talking about something that works using all the capabilities of the human body
and I've you know in this we're pretty fluent in talking in words and somehow we got very fluent in writing and drawing like can you just kind of do that without thinking about that playing a musical instrument we can't do that
without thinking about it if you've been trained and I think it will be the same thing for dynamic modeling that if we find the right medium then we can do that in real time as part of the
give-and-take of a conversation there again this is kind of a big research question not a lot of people working on it if you've seen like some of Ken Perlin latest work with chalk talk he's starting to make the initial baby steps
on creating a dynamic language that can be used in real time conversation this who I'm talking about here this is not computer-supported collaborative work this is not
Google Docs those things are about making the old representations more efficient those things are about sticking with words and static pictures and moving those around faster I'm talking about pushing words to the
sidelines and having dynamic modeling creating programs in real time be the content of the conversation you're modeling exploring together with the other person so now we can think about
one too many kinds of communication now obviously let's get away from this clicking through a PowerPoint deck something come closer to a blackboard where we're kind of kind of sketching
material from scratch where you can go in any direction and again it should be dynamic material so what you're where you're sketching as per your presentation is working models and you're kind of exploring them there with
the audience and there are evidence-based models meaning that conversations today tend to become a string of anecdotes like oh I heard that and then you say well I will trust him because he's an authority figure or he
looks trustworthy or someone else said he was just worthy and there's this complex notion of trust which is hopefully going to be obsolete if I am up here building my models and you can
see the model I'm building you don't have to take my word for it you can see the model you can see the facts in knowledge I'm bringing it to support the not model you can see where those are coming from you can kind of see the provenance of everything I'm using to
support my argument and you teak the model you can like check the facts yourself and you make all that visible it leads to a very different notion of trust and integrity and this is I think
a really important part of the the empowering aspect is that trusting authority is disempowering giving people the ability to be independent is empowering
so presentations in the form of improvising dynamic models that are explored together with the audience but I think the most interesting possibility has to do with using the space so right
now I'm on a stage and the stage is completely empty right there's the podium and there's like a guy and you know big pictures over there and you know it would be nice if I had like you
know props that I could be showing you things but I don't know I would have had to kill I carry that in my suitcase I don't have a way of downloading things to the stage but if we have an environment which is kind of you know
full of dynamic material then we can start to think about actually using human scale representations using body scale and room scale representations having a presentation where the
presenters have moving around interacting with real things in the environment so we can start to think about things like mapping concepts based a physical space like the different
topics of the presenter is presenting actually are are there you can see them they don't go away the presenter presents by moving their body from one topic to another as the argument moves
from one topic to another having a very strong spatial grounding start to think about things like having the the outline of the talk evident in the space that what I talked about five minutes ago was there where I'm going to talk about in
five minutes is there it's not just like one thing after another after another as it is today so again using the medium to kind of invoke these kinesthetic and spatial forms of understanding that are
really important so for this real-time form of conversation I'm really talking about improvising dynamic models as the content of the conversation of the presentation and exploring them with the
other person now we can think about non real-time forms of communication what is the equivalent of a book or website books are websites nowadays are basically big piles of words right and
so they inherit the same problems as speech is in that the authors are explaining and convincing through reasoning and rhetoric not through evidence explorable models and they have
an additional problem which is that there are one-size-fits-all there mass-manufactured and so I read a book you read a book we're both we both read exactly the same words even though we have were coming to a different levels of knowledge and
we're we want to get different things out of it right and you know in a dynamic medium that makes no sense so we can start to think about first of all a context-sensitive dynamic material what
I read and what you read are not necessarily the same thing the material takes into account who we are and what we want to get out of this this is again have a big juicy research question how do you create contact sensitive material
many years back I wrote a paper called magic ink about on the topic of contact sensitive reading material but there's there's so much more there that needs to be done and again the big thing is
explorable models so what the the author is sending out to the reader is not a big pile of words but it's a working thing it's a program that the reader then kind of actively explores and what
this can lead to one possibility is the reader getting more out of the material than the author put into it so in the same way that game designers are always surprised by what players end up doing in their game and authors of
creative tools are always surprised by what people make in their tools I'd like to see authors being surprised by what readers end up learning from their material because the authors not just sending out something static they're
sending out a program which is capable of merchants behavior and so the reader will be able to try out different things and discover things that the author hadn't intended and when you have this kind of active based reading it kind of
brings back in some of the things that we lost when we went from learning through active dialogues with another person to just kind of reading a single argument this allows you to kind of have more interrogated mode we can have a
kind of dialogue with the material but kind of like what I said with presentations I think the the most exciting possibility here has to do with the form factor so what I've depicted up
here is kind of you know the legacy of the printing press it's kind of flat thing you hold in your hands I don't really think that's what a book should look like in the dynamic medium I think a book wants to be a space that you walk
around in so something that feels a little bit more like a music am gallery than a book today so you read this book by walking around in it engaging again visually spatially
tangibly using all those capabilities that we've evolved for understanding spaces and environments so you want to learn linear algebra for example you download the linear algebra textbook which is this entire space maybe each
floor there is a particularly for the space interacting with the things the concepts are represented in tangible physical form you can use or spatial forms of perception to understand the
gist of the entire material I want to clarify that what I'm talking about here is not vr this is not vr it's not AR it's just our dr if you like dynamic
reality it's a real physical matter that you can you can touch but you know real physical matter that offers all the thousands of perceptual cues and physical forces that we've been honing
over the last hundred thousand years but it's dynamic in the same way a picture on a computer screen is a dynamic picture does this have dynamic physicality so if that's what a book looks like you start to think about what
a library looks like and no surprise I think that a library should also be a physical space they walk around in libraries today are physical spaces but walk around and you just see a bunch of spines which aren't very useful and so
you'd start to think about a spatial representation of the entire breadth of human knowledge having different areas of space represent different areas of knowledge you can walk into a space kind of just glance around and kind of get
the gist of some particular branch of knowledge if you want to go deeper on something you just kind of move towards it gets more detailed you move towards that it gets more detailed so you have this kind of continuous transition
between browsing and skimming and engaging deeply that you can do again with your body so so far I've been talking about knowledge in this in this
medium being represented not as big the words but as essential use programs as dynamic models that the reader is kind of actively engaging with and so we can ask the question how do you create that material how do you create those
programs what does authorship look like in this medium and the closest thing that we have right now is this activity that we call programming and I think that
programming as we understand it today conflates two very different activities two very different things that we've kind of both lumped in with programming which I'm going to call engineering and
authoring so engineering is thing to thing and authoring is person to person engineering is building a working reliable system that meets a spec meeting metal spec if you're designing a
op amp or a hydroelectric dam you have this material that you're working with is just really a matter of what is the configuration of my material that you know meets the spec it's kind of between you and the material and the forces of nature
so that's thing to thing and then there's offering which is person-to-person authoring if you think about writing a newspaper article or creating a picture or something the entire point is to create an impression
in another person's mind the thing that you're making is just kind of intermediary what you really care about is getting your message from the person to the person so these are I believe very different activities and we've got
both of those things into programming and we do them both in the same way with the same tools both of them just kind of writing code I think that moving forward we need to disentangle those so moving
forward engineering might continue using code but authoring I don't think code is an appropriate way of doing that because it's indirect manipulation so if you
think about a journalist quitting a newspaper article they're directly manipulating the words and the article if you think about an artist creating a picture the directly manipulating the ink on the paper there's something very
important about being able to directly and manipulate the experience that the recipient is going to get right because the author kind of has to empathizing with the recipient and has
to be able to kind of see exactly what they're going to experience and so if you're putting a newspaper article you directly manipulating the words if you're creating dynamic behavior and dynamic medium then I think you need to
be directly manipulating that dynamic behavior not going through the intermediary of code but kind of directly manipulating the end result so this notion of direct manipulation of time paper is something that I and others have been working on for some
time and you know there's certain threads in programming language research which kind of go back several decades but still kind of a niche obscure thing and I think that's also a big important research question for this crowd
especially it's good to work on developing methods for that engineering stuff but then we also need to think about how people are going to be creating dynamic messages for other people and I think it needs to be more direct manipulation and needs to be kind
of a more sketchy improvisational mode so getting away from kind of the engineers obsession with precision and more towards kind of free expression so
lastly we can think about using dynamic material to think with a mathematician you know eventually prepares their proof for publication but before they get there they're doing a whole bunch of stuff on scratch paper so we kind of
think about that what is using dynamic material just to support your own thought process you know a lot of what it said already applies because the the representations we use to communicate end up being the same representations that we use in our head a lot we kind of think in the same language that we speak
in but we can start to think here about the form factors of dynamic material like what what is the stuff that we'll use to think with and I think that there's there's kind of a duality
there's certain representations that you want to hold in your hand and kind of inspect from the outside there's other representations that you want to kind of be embedded in and kind of explore from the inside so there's this duality
between objects environments and so almost all intellectual work that we do nowadays is flat intangible right it's ink on paper or it's pixels on a screen
and physical objects to think with you know we we used to have we used to have things like slide rules architectural models molecular models and these things are kind of going extinct because of
virtualization that a molecular model on a computer screen is weaker in some respects because you can't actually hold in your hands and kind of think about it at your hands but it's stronger because
it's dynamic and dynamic trumps everything and so we've been led to more and more virtualization not because we actually want virtualization what we want is a dynamic Mis we want dynamic
behavior and putting it virtually on a screen is currently the only way we know how to get dynamic behavior and so that's why it's so important to get things out of the screen infuse computation into the physical world
trade dynamic physical matter and that we can start to be able to think with our hands again I think the most intriguing possibility here is the possibility of abstract tangible
representations so the literal representations are pretty easy right like our textual model is just a house scale down molecular models just the molecule scaled up you know that's that's pretty easy but how do you
represent for example an algebraic equation in a form that you feel what what is the representation of y equals x to the N that that concept has always lived in flatland and it's almost
unthinkable like you know how could it be anything other than the flat symbols that we've been using so long to represent that again I think that this is entirely an accident of history that
we've designed a mathematics for flat and we can redesign mathematics that draws on all of our your physical tactile capabilities it's kind of a
weird thought what would algebraic equation look like in a physical object but you have to think about the Playfair thing what does reading maps have to do with understanding data they're totally different things but then play
in vendor representation the day graphic which allowed people to use their map reading ability to understand data and is totally transformative and I think it's going to be similarly transformative to draw on our
credibility profound capabilities that allow us to tie a shoelace or play a musical instrument or do all the things we do their hands and bring those capabilities to bear on more abstract
than game so lastly the the dual two objects are environments today we everybody works in some sort of environment we work in some sort of room that is in a building that is in a
neighborhood and all that is static and because it's static it can't participate in work that happens at the speed of thought so today all the work that happens at the speed of thought has to
basically happen within the reach of your arms because your hands are your only dynamic instrument that you have and so that's what led to the desk and you know working with papers on a desk
or objects on a desk is everything had to be within arm's reach but when you have a environment that's full of dynamic material that can respond at the speed of thought then you get away from that and you can start to think about
representations that can take up a large amount of space short to think about a form of intellectual work that involves walking around moving around looking around interacting with human scale spatial representations you know taking
advantage of peripheral vision and spatial scanning and sense of scale and all these things that we've you know we've been honing for millions of years
I mean it's kind of obviously debilitating to sit at a desk all day right and we've had to invent this very peculiar concept of artificial exercise to keep our bodies from atrophy and the
solution to that is not fitbit's its inventing a new form of knowledge of work which naturally incorporates the body that draws on the strength of the body that uses the body in the way that the body is always meant to be used
so this is my vision of a humane medium or thought this is my vision of a medium of thought that treats the human being as sacred that treats these capabilities
that we've been honing for hundreds of thousands of years as sacred treats a physical world and our interaction with the physical world as sacred and builds
on top of those things to continue this ascent of intellectual progress and enable people to think ever greater thoughts you might have a different
conception of what a humane medium would look like and that's that's totally great the point that I want to get across here the point that I want to leave you with is that humane won't just happen right this is not like Sussman's
technology which is going to happen because there's already really powerful forces at play humane is never a default and humane only comes out of very deliberate and conscientious design work
if you do the incremental thing and just you know ride the current wave of technology and let technology lead you over it leads you it's going to lead you into a tighter and tighter cage it's going to lead to more virtualization
more disembodiment more dehumanization and so if you believe in the possibility of a humane medium then you know it's up
to us to kind of fight that trend and it's up to us to make it happen thank you you
End of transcript