Select text to annotate, Click play in YouTube to begin
00:00:38
so since is splash this is a programming conference I want to start by reassuring you that this talk is in fact about trading programs okay I want to start
00:00:53
with that because it might not seem like that's what I'm talking about I'm not going to be talking about writing code I'm not gonna be talking about building software and we normally think about it but this is very deeply about creating
00:01:04
programs and kind of in the same way that we create sounds when we talk we create letters no you're right this is about creating programs as a means of or person-to-person communication as a
00:01:16
means of representing thought this is this is kind of a personal talk in certain ways and so I wanted to start out just with a little bit of personal background food so you can kind of see
00:01:29
where I'm coming from I got my start like you know many people you know making games and apps and whatnot I think I kind of found my stride designing creative tools made a number of musical instruments visual design
00:01:41
tools that sort of thing that start gave me interested in UI design and after studying that I went to Apple inventing and prototyping UI concepts for experimental projects which ended up influencing these things and perhaps
00:01:55
other things yet to come and it was also very interested in information design I helped Al Gore make a book app about climate change designed a series of interactive information graphics to
00:02:07
enable readers to understand things in ways that be difficult to understand and text and then I got on a train and started living on the train it's a
00:02:19
wandering research hobo among other things designing new kinds of reading material where it's not the author lecturing at the reader but the author is guiding the reader through exploring dynamic models simulations that the
00:02:31
authors provided the read archive has the agency to try different things out to say what if to critique the model be more active people are participating reader so making these things and then
00:02:43
also prototyping these different creative tool environments for for dynamic things so tools for working with mathematical systems for making software making electronics animation tools for making
00:02:58
things that have behavior that do stuff and my focus always seems to be on the representation of that behavior how could author x' see what the thing they were making is actually doing and what
00:03:10
are powerful ways of seeing so they can deeply understand what the thing is doing and understand it in many different ways so this notion of
00:03:24
representation representation of dynamic behavior seemed to be very central and so I made all these things and made all these prototypes and this kind of hobo period was just looked good you know a
00:03:36
couple years or so and then over the last year I've just kind of been reflecting on all this this raw material and trying to trying to hear what the
00:03:47
prototypes are trying to say because all those things that that I showed you know each one of them was kind of individually it's sort of useful in its own little domain but that wasn't really the point of any of this all those were kind of a vantage point to get a glimpse
00:04:02
of something much bigger kind this new way of working and thinking in the dynamic medium and that that bigger picture is what I want to try to talk
00:04:13
about today so this talk has two parts the first half is about the past and the second half is about the future in the first half I'm going to try to make the argument that representations by which I
00:04:28
mean the the ways we externalized thought have been responsible over the last two thousand years in large part for the intellectual progress of humanity enabling us to think thoughts that we couldn't think before but while these media that we've invented have
00:04:40
empowered us in certain ways they've also crippled us in other ways and we now have the opportunity to design a new medium of thought that undoes some of that damage that's both humane and empowering so that's the first half and
00:04:54
the second half I'm going to try to sketch out what that medium might look like so what it might be like to communicate to work to think in a humane medium so we'll start with
00:05:05
representations what I mean by representation is the physical form of a thought it's something out there in the world that you can sense with your senses and manipulate with your body and
00:05:17
as an example I want to tell you about when my favorite forms of representation of all time which was invented by William Playfair in 1786 William Playfair was writing a book about England's trade with other nations and
00:05:31
he had a lot of data and back then data was always expressed it was always represented in tables of numbers if you're doing economics or business or science you had these tables and numbers you had to learn how to read and write
00:05:43
them and data and tables of data they were the same concept you couldn't tease them apart the data and the way they were representing the data people couldn't think of them separately and playfair had a very interesting thought
00:05:56
he realized that there's a certain form of understanding that we bring to the table the numbers but there's a very different way of understanding that we bring to things like maps there's kind
00:06:09
of a different set of perceptual and cognitive capabilities that we bring into play when we're looking at a map and playfair had the unprecedented thought of making a very peculiar kind
00:06:21
of map where instead of left-right being west east it would be earlier later instead of up-down being north/south it would be more money less money and so he drew this funny kind of map which was
00:06:34
the first data graphic the first time anybody plotted data and people immediately caught on to like wow this is a really powerful way of understanding data as these things happened it didn't people got it but it
00:06:47
didn't actually catch on as a thing for another hundred years or so but eventually it did and today of course the data plot underlies almost all of modern science and engineering most sprints as a science and engineering
00:07:00
will be almost unthinkable without the plot it's just totally fundamental to everything we do to the extent that it almost seems obvious right and it's hard to realize that we've only invented a couple of years ago so
00:07:13
the big point here and this is kind of what I want you to hold through this entire talk what play fair did was he realized that human beings have a certain capability for understanding maps for understanding this kind of
00:07:25
cartographic abstraction and this capability was not being used when they're trying to understand data and so we invented a representation which represented the data in such a way that this latent capability could be brought
00:07:38
into service right and that's what representations do that's how they work their magic is they draw on certain latent innate capabilities that we have and your capabilities that originally we
00:07:54
in evolved for like Yoona in a hunting gathering context and bring these latent capabilities into service repurposing them for a more abstract purpose than
00:08:05
they're originally intended for and you can see the sort of pattern in play in many of the great representations throughout history in mathematics of course one of the first amazing ones was
00:08:19
the invention of Arabic police value numerals you can't do or thematic in Roman numerals you can't think arithmetic in Roman numerals you have to like go over to your abacus police value
00:08:31
Arabic numerals were the first user interface for numbers that allowed calculation to be done on paper this was totally transformative same sort of story with the invention of algebraic notation as opposed to like writing out
00:08:42
imperative knowledge and big blocks of text you could write things declarative and small number of symbols absolutely transformative leibnitz's notation for the calculus Leibnitz was the UI
00:08:54
designer of the 17th century he was obsessed with notation always trying out different notations always like talking with his friends about notation because he realized that a lot of the power and
00:09:06
idea lies in the form in which it's expressed because that's what allows people to think it and in almost any given field you can go in there and find a particular form of representation
00:09:18
which transformed or ignited the field so the periodic table for example Mendeleev correct table is basically the beginning of a theory industry that's where it all started Bohr's model of the atom which of course
00:09:30
was originally inspired by Copernicus's model of the solar system Faraday's representation of magnetism as lines of force which influenced Maxwell to make his theory
00:09:43
although Maxwell did not write Maxwell's equations Maxwell wrote 20 equations and 20 unknowns kind of scattered throughout his paper it was absolutely a brilliant theory but there's something much deeper
00:09:56
there that nobody could see and it was Oliver Heaviside who invented the language of vector calculus specifically for the sole purpose of being able to write Maxwell's equations in that kind
00:10:08
of compact for lime symmetric form so of course that representation of Maxwell's equations has been a very influential but more so heavy sides representation
00:10:21
of the vector basically powered 100 years of physics so when you're thinking of the great ideas of history often what you really want to be thinking about is the great representations that enable people to think those ideas so ideas
00:10:36
live in representations and the representations in turn have to live in a medium so there's different me of thought there's some ideas that you can talk about in speech there's some ideas that you are carried well in song or
00:10:48
theater but the biggie of course was print the invention of the printing press 500 years ago which basically made all that stuff on the left possible all that stuff on the left was basically designed for the medium of print with
00:11:01
the possible exception of numerals but even those they've been around for a few hundred years but they're basically kind of a niche thing until print came along and that's when they started to come widespread so a powerful medium is what
00:11:13
enables powerful representations to be used and seen and spread so you can think about the invention of powerful representations and the invention of powerful media to host powerful
00:11:27
representations as being one of the big drivers of last 2,000 years of the intellectual progress of humanity because each representation allows us to think thoughts that we couldn't think before we kind of
00:11:39
continuously expand our think about territory so you can think of this as tied to you know the grand meta-narrative of the scent of humanity moving away from myth and superstition
00:11:51
and ignorance and towards a deeper understanding of ourselves in the world around us I bring this up explicitly because I think it's good for people to acknowledge the motivation for their
00:12:02
work and this is this story of the intellectual progress of humanity is something that I find very motivating inspiring and is something that I feel like I want to contribute to but I think
00:12:16
that if this if you take this as your motivation you kind of have to be honest with yourself that that there definitely has been ascent we have improved in many
00:12:27
ways but there are also other ways in which our history has not been ascent so we invent technology we media technology
00:12:39
to kind of help us make this this climb but every technology is a double-edged sword every technology enables us has a potential de navels in certain ways while debilitating us in other ways and
00:12:51
that's especially true for representations because the way the reputations work is they draw on certain capabilities that we have so if we go all in in a particular medium like we
00:13:03
did with print so the capabilities that are not well supported in that medium they get neglected in they atrophy and we atrophy I wish I knew who drew the picture
00:13:20
because it's it's a wonderful depiction of what I'm trying to express here and even a little misleading because the person the last stage they're kind of hunched over is tiny rectangle we reach
00:13:31
that stage accomplish that stage with the printing press and cheap paper book based knowledge the invention of paper-based bureaucracy paper-based
00:13:44
working we invented this lifestyle this way of working where to do knowledge work meant to sit at a desk and stare at your little tiny rectangle make a little motions of your hand you know started
00:13:56
out as sitting at a desk staring at papers or books and making little motions with a pen and now it's sitting at a desk staring at a computer screen making a little motions with your on a keyboard but it's basically the same
00:14:08
thing we've this is what it means to do knowledge work nowadays this is what it means to be a thinker it means to be sitting and working with symbols on a little tiny rectangle to the extent that
00:14:20
again it almost seems inseparable you can't separate the representation for what it actually is and and this is basically just an accident of history this is just the way that our media
00:14:32
technology happen to evolve and then we kind of designed a way of knowledge work for that media that we happen to have and I think so I'm going to make the claim that this style of knowledge work
00:14:47
this lifestyle is inhumane and I mean something very precise by that so I'm going to try to explain with an analogy imagined imagine you adopt a puppy so
00:15:01
your Duff is puppy and your name um puddles you take puddles home and you give them a little snack and you stick puddles in the cage and you lock the door forever and never open it again and
00:15:14
puddle slips out the rest of his poor life trapped inside this little cage so that's wrong I think most of us would agree that's sadistic that obviously inhumane so well let's ask what exactly
00:15:27
is wrong about that you know what what is inhuman about sticking puddles in the cage well you know we kind of have a notion of what it means to live a full doggy life right dogs have to run around dogs run
00:15:40
around and they sniff other dogs and they pee on things and that's kind of what it means to be a dog they've inherited this the set of capabilities capabilities and traits from their wolf ancestors and we recognize that a dog
00:15:53
has to be allowed the full free expression of its entire range of capabilities by sticking the cage or constraining his range of experience you're not letting him do all the things that dogs can do and this is exactly
00:16:07
what we've done to ourselves we've invented media that severely constrained our range of intellectual experience that of all the many capabilities that
00:16:19
we have all the many ways of thinking that we have we have constrained ourselves to a tiny subset we're not allowed to use our full intellect and to see that I think we need to take a look
00:16:31
at you know what is that full range of capability is what are all the different ways we can think about things and it turns out it's a very vast and amorphous space our modes of understanding so
00:16:44
there's been number of different attempts to add structure to that space to carve it up one way of looking at it is by looking at the different sensory channels that we use to make sense of
00:16:55
things so an example if I say music you'll probably think oh that's something that we understand arleigh right but there's another representation of music which is visual which is sheet music which allows us to understand
00:17:08
music and a very different and very powerful way if you're making music then you're often thinking about it tactile II it's a tactile understanding it's an interaction between your hands and the instrument you could define
00:17:22
dance as an understanding of music which lives in the body which lives in the movements of the body and if you've ever played in an orchestra or band you can appreciate a kind of spatial understanding of music where you know
00:17:34
the guitarist over there the drummer's over there you're not thinking about music in the spatially distributed way if music is just an example there's any number of things which can be represented at any or all
00:17:44
of these these forms these channels and it's a more than the sum of their parts thing that we have multiple representations they compound and reinforce each other incredibly powerful ways and allow to allow for
00:17:58
understandings that would not be possible in any single channel so that's one I guess one attempt to draw basis vectors into this kind of space of understandings another useful one was
00:18:11
presented by Jerome Bruner have adapted from PJ's theories so he talked about the action based image based and language based understandings writing bike is something that you understand by doing it you can't talk about how you
00:18:24
ride a bike it's the understanding lives in the Performing of it there's an image based understanding you can look at the drivetrain of bike and see how the sprocket is pulling the chain or derailleur and you have understand this complex mechanical system in this image
00:18:37
based wordless way and then of course there's language based understanding understand you can articulate you can talk about parts of like you can talk about gear ratios and calculate them I think most of us in this crowd tend to
00:18:52
gravitate towards the symbolic and so I think it's important you know maybe to to remember that speech is purely symbolic and the most the most important
00:19:07
invention in the history of humanity was the invention of writing where he took speech and turned it into an image based iconic second channel form right so that's what happens when you get multiple channels involved as you lead
00:19:20
to leads to understanding that would not be possible with any any single one I'm gonna put up with a couple others just to kind of give you a sense of the breadth of this space so there's Howard Gardner's multiple intelligences some
00:19:34
people abuse this to say things like Johnny is visual and Sally is verbal or whatever you know don't do that but the way to use this is to realize that every human being has all these capabilities
00:19:47
and we can design reputation across them especially like gardeners because he explicitly calls out empathetic forms of understanding the way that we under and other people understand ourselves understand nature we can use all of
00:20:01
these capabilities my current favorite right now is cured Ekans framework he's got inspired by Vygotsky and it's a little too subtle to talk about here but if you're interested in this sort of thing I recommend you read Ian's book
00:20:13
the educated mind because it's fantastic so the point of all of this is really just that the space of the ways we understand things the space of our
00:20:25
cognitive capabilities is so vast and so diverse that even people who have devoted their entire careers to trying to reduce it down into a neat little Theory have come up with very different answers because there's there's so much
00:20:39
there it's such a rich space let's just focus on these two for now so basically every every circle up there is a superpower right every circle up there
00:20:52
is a capability that we've been honing for hundreds of thousands of years that we use in you know innumerable ways and can combine with the other ones in really powerful ways and what happened
00:21:05
with the invention of the printing press in the invention of tiny rectangle based knowledge work is this so all that stuff drops out and we're working with visual
00:21:17
symbols right we're reading visual symbols or manipulating visual symbols that's what it means to do intellectual work nowadays and you might think oh okay maybe that was true with like paper
00:21:30
and books but you know with computers it's getting better right and no with computers it is getting worse so with a book you know a book is at least a physical object which exists in the world if some amount of tactile response
00:21:43
you can hold it and move it around with a book you can make a shelf which is a spatial representation of knowledge it's not a very good one but it's at least something that you can understand spatially when you're writing with ink
00:21:56
on paper you can move freely between drawing imagery and writing in language because the paper doesn't really care what kind of marks you make on it so you have the freedom at least to move between those two modes but then we
00:22:08
invent these things and they have these flat glassy screens that have no tactile response and that drops out and their expensive little screens they take up a little tiny
00:22:20
portion of your field of view and the spatial stuff drops out and they have these keyboards basically the only convenient thing to do is punch in symbols especially for us and anything from writing an email to write a
00:22:33
computer program anything other than just typing in letters is incredibly cumbersome regardless what you're trying to express you express it in symbols because that's what the interface encourages you to do so that drops out
00:22:48
and so this is the cage that we have trapped ourselves in this is the way in which we have constrained our range of experience which we have created a tiny
00:23:03
subset of our intellectual capabilities and restrict ourselves to this tiny subset and have forbidden ourselves to use our full intellect so there's two
00:23:16
things wrong with this one is that it's inhumane so in the same way you know keeping puddles in the cages inhumane because he can't do all those doggy things using media that restrict our thinking to this tiny subset is inhumane
00:23:30
because we can't do all our thinking things we can't think in all the ways that human beings are able to think so that's there's kind of a moral argument there if you don't buy that there's also a practical argument which is that is
00:23:41
just wasteful one way to think about it is you know as programmers matching you have like an equal processor and you're writing some code it's kind of a sequential algorithm can't parallelize it so you're running this code it's
00:23:54
maxing out one of the cores and the other seven pores are just idling right and that that hurts right like as a program you kind of get this feeling in your stomach like you
00:24:05
know god this is so inefficient like if only I could paralyze algorithm OOP like there's just like so much latent power I can draw on and that's exactly the emotion that I get when I'm looking at this tiny rectangle based knowledge work
00:24:19
that we do is that the there's all these cores have all these capabilities and they're just sitting there idle and if we could only paralyze our representations across all these capabilities who knows what
00:24:32
would be capable of thinking so the good news is that we now have an opportunity to do something about this so we're now at a I believe unique moment in history where we're inventing the next medium of
00:24:47
thought after the printing press we're inventing the dynamic medium so in the same way there's certain thoughts that can be conveyed pretty well in speech then thoughts can fade in theater or in
00:24:58
print there's I believe a very wide range of thoughts that can be conveyed in programs once we understand understand how to do that so this medium that we're inventing the dynamic medium
00:25:10
has three very interesting properties one is that its computational which means that it's capable of simulation can simulate processes out in the world it's responsive so unlike a book where you poke at the book and book it doesn't
00:25:23
do anything dynamic things can respond to stimuli so when they're simulating stuff in the world they can respond and show you the results in many different ways and it's connected so unlike a book here in a book there they can't you know
00:25:37
they're isolated objects dynamic material can exchange information and talk to each other and so what this means if you're trying to represent something like a system today in language if you're trying to represent a
00:25:50
system you describe it you talk about it it's a very poor way of representing it in a computational medium you can model it you can actively model and explore that system and then see what's doing in
00:26:04
many different ways and you can start to fill in this we've got a responsive medium and you know today we mostly think of computational responsiveness as a visual thing you know you've got a screen and it does
00:26:16
stuff and some amount of aural responsiveness and I think things are only going to get really interesting once we have tactile responsiveness I'm not really talking about haptics here I'm talking basically about taking
00:26:28
computation out of the screen and into the physical world so infusing physical matter with computation and letting kind of dynamic responsiveness be something that exists
00:26:41
in the world and something you can touch and feel and interact with with your hands the way we've been working with tools for millions of years so once we have kind of a full physical responsiveness we can start to fill that
00:26:54
in and as a connected medium that means that you have a space filled with dynamic material it's not all isolated they can talk to each other they can work together to make large-scale
00:27:08
representations so we no longer have to work with like little things that we hold in our hands but all the material can host things that our body scale or
00:27:18
room scale start to fill that in and we can start to have the potential at least to design a medium that is humane and empowering and these are two sides of
00:27:34
the same coin humane meaning allowing us to draw on our full range of capabilities getting us out of the cage allowing us to express our full intellect and empowering meaning taking advantage of that drawing on our full
00:27:48
intellect to enable people to think thoughts that they couldn't think before so in the second half of the talk I want to take you through a little sketch of what it might be like working in a fully
00:28:02
dynamic medium how thoughts would be represented in these different modes person-to-person one-to-one conversations or one-to-many hesitations thing two-person things like what does it mean to read wired knowledge from a
00:28:14
knowledge object or to browse many of them person to thing what does it mean to author the sort of material or to use it as the way of thinking so this is 40
00:28:26
years out as we may think or Dynabook kind of thing but it's not really intended as a prediction so much as just a guide post to kind of steer the ship towards I found it very useful as
00:28:39
guidance for my own work and so I hope you'll find it useful as well I am only going to be talking about the human experience of working and thinking in this medium I'm not really going to address the
00:28:52
implementation of this stuff the the technology that underlies it isn't really the point here but I've been reminded that there are people at this conference who are very interested in technology and I would be remiss to
00:29:04
leave it unaddressed so here's the part of the talk about technology I'm taking the liberty of inviting in legendary and mighty professor Gerry Sussman author of scheme and SiC tea and Sikkim and all
00:29:17
these wonderful things and Sussman is going to tell us about the technology of the future but here in the future it's going to be the case that that computers are so cheap and so easy to make that
00:29:30
you can put in the size of a grain of sand complete with a megabyte of RAM you're going to buy them by the but by the bushel I suppose you can pour them into your concrete and you buy your
00:29:43
concrete by a mega flop and you have a wall that's smart okay so long as you can skip the power to them and they could do something you know that's going to happen right it's sort of the way I said remember your cells are about are
00:29:55
pretty smart and they but they seem to talk to each other and do useful things so we have to begin to worry about that kind of a world that'll happen it's going to happen there's no other way to make the future happen
00:30:08
so if Sussman says it's going to happen that's good enough for me and as he says we have to begin to worry about that kind of a world what he's worrying about in this case is the engineering problem of how do we build
00:30:23
working reliable systems out of this kind of distributed computational material what I'm worrying about and I'm hoping to impress onto you as well as the humanist problem of what should we
00:30:36
build and why are we building it and what will we do with it and what's what is it going to do to us right because this technology they talking about you know this kind of you know 300 computers per square inch or
00:30:47
whatever infused into the into physical matter that that sounds like really far off right that sounds like you know we've got time but you know technology does the exponential thing and we're always blindsided by Exponential's so
00:31:00
this technology that is talking about it's going to be here before we know it and so it's kind of up to us to prepare for that you know we have the choice to to say okay let's start envisioning and
00:31:13
sketching and pro prototyping designing designing something that we you know understand to be humane so then when the technology is ready we're like okay we know we know what we want to build with this we know it's going to work we know
00:31:25
it's going to be good so you can do that or you can just kind of do the incremental thing and kind of you know jump on whatever technology bandwagon comes on and let technology follow its own course and I can guarantee you that
00:31:38
that is not going to be humane that result is just going to lead you into a tighter and tighter cage because when technology takes its own path it finds certain veins of the capable
00:31:50
incapabilities and just kind of like drills further in those and leaves the rest of the capability landscape untouched so even though it doesn't seem feasible right now I think we need to
00:32:03
start thinking about how to design the humane medium with the technology that we want to have so let's start off with the most fundamental active communication which is a real-time
00:32:16
conversation between two people so this is something that we've been doing for hundreds of thousands of years and we do it nowadays basically in the same way that we have been doing
00:32:28
at 400,000 years is by spoken words going back and forth that's how we communicate that's how we converse is like spoken words going back and forth with our hands around maybe maybe we'll like draw some sketches on the
00:32:41
whiteboard or a cave wall and there's some things that you can represent well in in that medium and there's many things especially kind of modern things that we need to talk about nowadays
00:32:56
which are not well represented in spoken language and one of those is systems so for so we live in an era of systems there's natural systems like the
00:33:09
environment eco systems biological systems pathological systems there's the stuff that we make political economic infrastructure Allah make out of
00:33:21
concrete and metal electronics and the wrong way to understand a system is to talk about it to describe it the right way to understand it is to kind of get in there and model it and explore it and
00:33:36
you can't do that in words and so what we have is people are using these very old tools people are explaining and convincing through reasoning and
00:33:50
rhetoric instead of the newer tools of evidence and explorable models we want a medium that supports that so we can start to think about a medium a medium for conversation that is naturally show
00:34:04
until we're depicting and describing or on equal footing it's just as easy to create an image as it is to create a description and what you are depicting
00:34:17
are not static images as we do today but actual working models you're creating simulations you're creating programs and so for example if you want to talk about how in this case how a
00:34:30
airplane wing generates lift if you want to talk about the effect of some policy change all these things can be modeled we want to have a medium where you can model them and then explore them and have be the content of the conversation and
00:34:44
you want these models that you're building to be evidence backed you want them to draw upon the facts and knowledge that we know about the world incorporate them into the model and have everybody be able to see the the facts
00:34:57
that you're bringing in the provenance of the facts everybody see everyone to see how the models working and so this kind of leads to the first first of several big juicy research questions
00:35:10
which is how how do we do this kind of modeling because kind of the way that we create dynamic models today is with this activity that we call programming which is kind of going off and staring at a
00:35:23
screen and bashing on a keyboard for a few hours or weeks and I'm talking about getting this down to seconds improvising sketching dynamic models in real time as part of the real-time give-and-take of
00:35:36
conversation we don't know how to do that we don't know how to create working dynamic material in seconds and you might think that's impossible like how can you program in seconds but that I
00:35:49
would agree this probably isn't possible if you're thinking about staring at a screen and bashing on the keyboard I'm talking about something that works using all the capabilities of the human body
00:36:00
and I've you know in this we're pretty fluent in talking in words and somehow we got very fluent in writing and drawing like can you just kind of do that without thinking about that playing a musical instrument we can't do that
00:36:14
without thinking about it if you've been trained and I think it will be the same thing for dynamic modeling that if we find the right medium then we can do that in real time as part of the
00:36:27
give-and-take of a conversation there again this is kind of a big research question not a lot of people working on it if you've seen like some of Ken Perlin latest work with chalk talk he's starting to make the initial baby steps
00:36:42
on creating a dynamic language that can be used in real time conversation this who I'm talking about here this is not computer-supported collaborative work this is not
00:36:55
Google Docs those things are about making the old representations more efficient those things are about sticking with words and static pictures and moving those around faster I'm talking about pushing words to the
00:37:08
sidelines and having dynamic modeling creating programs in real time be the content of the conversation you're modeling exploring together with the other person so now we can think about
00:37:23
one too many kinds of communication now obviously let's get away from this clicking through a PowerPoint deck something come closer to a blackboard where we're kind of kind of sketching
00:37:36
material from scratch where you can go in any direction and again it should be dynamic material so what you're where you're sketching as per your presentation is working models and you're kind of exploring them there with
00:37:49
the audience and there are evidence-based models meaning that conversations today tend to become a string of anecdotes like oh I heard that and then you say well I will trust him because he's an authority figure or he
00:38:02
looks trustworthy or someone else said he was just worthy and there's this complex notion of trust which is hopefully going to be obsolete if I am up here building my models and you can
00:38:14
see the model I'm building you don't have to take my word for it you can see the model you can see the facts in knowledge I'm bringing it to support the not model you can see where those are coming from you can kind of see the provenance of everything I'm using to
00:38:25
support my argument and you teak the model you can like check the facts yourself and you make all that visible it leads to a very different notion of trust and integrity and this is I think
00:38:38
a really important part of the the empowering aspect is that trusting authority is disempowering giving people the ability to be independent is empowering
00:38:50
so presentations in the form of improvising dynamic models that are explored together with the audience but I think the most interesting possibility has to do with using the space so right
00:39:02
now I'm on a stage and the stage is completely empty right there's the podium and there's like a guy and you know big pictures over there and you know it would be nice if I had like you
00:39:14
know props that I could be showing you things but I don't know I would have had to kill I carry that in my suitcase I don't have a way of downloading things to the stage but if we have an environment which is kind of you know
00:39:27
full of dynamic material then we can start to think about actually using human scale representations using body scale and room scale representations having a presentation where the
00:39:40
presenters have moving around interacting with real things in the environment so we can start to think about things like mapping concepts based a physical space like the different
00:39:53
topics of the presenter is presenting actually are are there you can see them they don't go away the presenter presents by moving their body from one topic to another as the argument moves
00:40:06
from one topic to another having a very strong spatial grounding start to think about things like having the the outline of the talk evident in the space that what I talked about five minutes ago was there where I'm going to talk about in
00:40:18
five minutes is there it's not just like one thing after another after another as it is today so again using the medium to kind of invoke these kinesthetic and spatial forms of understanding that are
00:40:30
really important so for this real-time form of conversation I'm really talking about improvising dynamic models as the content of the conversation of the presentation and exploring them with the
00:40:43
other person now we can think about non real-time forms of communication what is the equivalent of a book or website books are websites nowadays are basically big piles of words right and
00:40:56
so they inherit the same problems as speech is in that the authors are explaining and convincing through reasoning and rhetoric not through evidence explorable models and they have
00:41:09
an additional problem which is that there are one-size-fits-all there mass-manufactured and so I read a book you read a book we're both we both read exactly the same words even though we have were coming to a different levels of knowledge and
00:41:21
we're we want to get different things out of it right and you know in a dynamic medium that makes no sense so we can start to think about first of all a context-sensitive dynamic material what
00:41:34
I read and what you read are not necessarily the same thing the material takes into account who we are and what we want to get out of this this is again have a big juicy research question how do you create contact sensitive material
00:41:47
many years back I wrote a paper called magic ink about on the topic of contact sensitive reading material but there's there's so much more there that needs to be done and again the big thing is
00:42:00
explorable models so what the the author is sending out to the reader is not a big pile of words but it's a working thing it's a program that the reader then kind of actively explores and what
00:42:13
this can lead to one possibility is the reader getting more out of the material than the author put into it so in the same way that game designers are always surprised by what players end up doing in their game and authors of
00:42:25
creative tools are always surprised by what people make in their tools I'd like to see authors being surprised by what readers end up learning from their material because the authors not just sending out something static they're
00:42:38
sending out a program which is capable of merchants behavior and so the reader will be able to try out different things and discover things that the author hadn't intended and when you have this kind of active based reading it kind of
00:42:51
brings back in some of the things that we lost when we went from learning through active dialogues with another person to just kind of reading a single argument this allows you to kind of have more interrogated mode we can have a
00:43:04
kind of dialogue with the material but kind of like what I said with presentations I think the the most exciting possibility here has to do with the form factor so what I've depicted up
00:43:19
here is kind of you know the legacy of the printing press it's kind of flat thing you hold in your hands I don't really think that's what a book should look like in the dynamic medium I think a book wants to be a space that you walk
00:43:31
around in so something that feels a little bit more like a music am gallery than a book today so you read this book by walking around in it engaging again visually spatially
00:43:44
tangibly using all those capabilities that we've evolved for understanding spaces and environments so you want to learn linear algebra for example you download the linear algebra textbook which is this entire space maybe each
00:43:59
floor there is a particularly for the space interacting with the things the concepts are represented in tangible physical form you can use or spatial forms of perception to understand the
00:44:11
gist of the entire material I want to clarify that what I'm talking about here is not vr this is not vr it's not AR it's just our dr if you like dynamic
00:44:27
reality it's a real physical matter that you can you can touch but you know real physical matter that offers all the thousands of perceptual cues and physical forces that we've been honing
00:44:39
over the last hundred thousand years but it's dynamic in the same way a picture on a computer screen is a dynamic picture does this have dynamic physicality so if that's what a book looks like you start to think about what
00:44:53
a library looks like and no surprise I think that a library should also be a physical space they walk around in libraries today are physical spaces but walk around and you just see a bunch of spines which aren't very useful and so
00:45:07
you'd start to think about a spatial representation of the entire breadth of human knowledge having different areas of space represent different areas of knowledge you can walk into a space kind of just glance around and kind of get
00:45:21
the gist of some particular branch of knowledge if you want to go deeper on something you just kind of move towards it gets more detailed you move towards that it gets more detailed so you have this kind of continuous transition
00:45:33
between browsing and skimming and engaging deeply that you can do again with your body so so far I've been talking about knowledge in this in this
00:45:47
medium being represented not as big the words but as essential use programs as dynamic models that the reader is kind of actively engaging with and so we can ask the question how do you create that material how do you create those
00:45:59
programs what does authorship look like in this medium and the closest thing that we have right now is this activity that we call programming and I think that
00:46:11
programming as we understand it today conflates two very different activities two very different things that we've kind of both lumped in with programming which I'm going to call engineering and
00:46:23
authoring so engineering is thing to thing and authoring is person to person engineering is building a working reliable system that meets a spec meeting metal spec if you're designing a
00:46:37
op amp or a hydroelectric dam you have this material that you're working with is just really a matter of what is the configuration of my material that you know meets the spec it's kind of between you and the material and the forces of nature
00:46:51
so that's thing to thing and then there's offering which is person-to-person authoring if you think about writing a newspaper article or creating a picture or something the entire point is to create an impression
00:47:04
in another person's mind the thing that you're making is just kind of intermediary what you really care about is getting your message from the person to the person so these are I believe very different activities and we've got
00:47:17
both of those things into programming and we do them both in the same way with the same tools both of them just kind of writing code I think that moving forward we need to disentangle those so moving
00:47:28
forward engineering might continue using code but authoring I don't think code is an appropriate way of doing that because it's indirect manipulation so if you
00:47:41
think about a journalist quitting a newspaper article they're directly manipulating the words and the article if you think about an artist creating a picture the directly manipulating the ink on the paper there's something very
00:47:54
important about being able to directly and manipulate the experience that the recipient is going to get right because the author kind of has to empathizing with the recipient and has
00:48:07
to be able to kind of see exactly what they're going to experience and so if you're putting a newspaper article you directly manipulating the words if you're creating dynamic behavior and dynamic medium then I think you need to
00:48:21
be directly manipulating that dynamic behavior not going through the intermediary of code but kind of directly manipulating the end result so this notion of direct manipulation of time paper is something that I and others have been working on for some
00:48:34
time and you know there's certain threads in programming language research which kind of go back several decades but still kind of a niche obscure thing and I think that's also a big important research question for this crowd
00:48:46
especially it's good to work on developing methods for that engineering stuff but then we also need to think about how people are going to be creating dynamic messages for other people and I think it needs to be more direct manipulation and needs to be kind
00:48:59
of a more sketchy improvisational mode so getting away from kind of the engineers obsession with precision and more towards kind of free expression so
00:49:13
lastly we can think about using dynamic material to think with a mathematician you know eventually prepares their proof for publication but before they get there they're doing a whole bunch of stuff on scratch paper so we kind of
00:49:25
think about that what is using dynamic material just to support your own thought process you know a lot of what it said already applies because the the representations we use to communicate end up being the same representations that we use in our head a lot we kind of think in the same language that we speak
00:49:38
in but we can start to think here about the form factors of dynamic material like what what is the stuff that we'll use to think with and I think that there's there's kind of a duality
00:49:51
there's certain representations that you want to hold in your hand and kind of inspect from the outside there's other representations that you want to kind of be embedded in and kind of explore from the inside so there's this duality
00:50:04
between objects environments and so almost all intellectual work that we do nowadays is flat intangible right it's ink on paper or it's pixels on a screen
00:50:17
and physical objects to think with you know we we used to have we used to have things like slide rules architectural models molecular models and these things are kind of going extinct because of
00:50:31
virtualization that a molecular model on a computer screen is weaker in some respects because you can't actually hold in your hands and kind of think about it at your hands but it's stronger because
00:50:43
it's dynamic and dynamic trumps everything and so we've been led to more and more virtualization not because we actually want virtualization what we want is a dynamic Mis we want dynamic
00:50:57
behavior and putting it virtually on a screen is currently the only way we know how to get dynamic behavior and so that's why it's so important to get things out of the screen infuse computation into the physical world
00:51:09
trade dynamic physical matter and that we can start to be able to think with our hands again I think the most intriguing possibility here is the possibility of abstract tangible
00:51:21
representations so the literal representations are pretty easy right like our textual model is just a house scale down molecular models just the molecule scaled up you know that's that's pretty easy but how do you
00:51:33
represent for example an algebraic equation in a form that you feel what what is the representation of y equals x to the N that that concept has always lived in flatland and it's almost
00:51:47
unthinkable like you know how could it be anything other than the flat symbols that we've been using so long to represent that again I think that this is entirely an accident of history that
00:51:58
we've designed a mathematics for flat and we can redesign mathematics that draws on all of our your physical tactile capabilities it's kind of a
00:52:11
weird thought what would algebraic equation look like in a physical object but you have to think about the Playfair thing what does reading maps have to do with understanding data they're totally different things but then play
00:52:23
in vendor representation the day graphic which allowed people to use their map reading ability to understand data and is totally transformative and I think it's going to be similarly transformative to draw on our
00:52:36
credibility profound capabilities that allow us to tie a shoelace or play a musical instrument or do all the things we do their hands and bring those capabilities to bear on more abstract
00:52:47
than game so lastly the the dual two objects are environments today we everybody works in some sort of environment we work in some sort of room that is in a building that is in a
00:53:02
neighborhood and all that is static and because it's static it can't participate in work that happens at the speed of thought so today all the work that happens at the speed of thought has to
00:53:16
basically happen within the reach of your arms because your hands are your only dynamic instrument that you have and so that's what led to the desk and you know working with papers on a desk
00:53:28
or objects on a desk is everything had to be within arm's reach but when you have a environment that's full of dynamic material that can respond at the speed of thought then you get away from that and you can start to think about
00:53:41
representations that can take up a large amount of space short to think about a form of intellectual work that involves walking around moving around looking around interacting with human scale spatial representations you know taking
00:53:54
advantage of peripheral vision and spatial scanning and sense of scale and all these things that we've you know we've been honing for millions of years
00:54:05
I mean it's kind of obviously debilitating to sit at a desk all day right and we've had to invent this very peculiar concept of artificial exercise to keep our bodies from atrophy and the
00:54:19
solution to that is not fitbit's its inventing a new form of knowledge of work which naturally incorporates the body that draws on the strength of the body that uses the body in the way that the body is always meant to be used
00:54:37
so this is my vision of a humane medium or thought this is my vision of a medium of thought that treats the human being as sacred that treats these capabilities
00:54:52
that we've been honing for hundreds of thousands of years as sacred treats a physical world and our interaction with the physical world as sacred and builds
00:55:03
on top of those things to continue this ascent of intellectual progress and enable people to think ever greater thoughts you might have a different
00:55:16
conception of what a humane medium would look like and that's that's totally great the point that I want to get across here the point that I want to leave you with is that humane won't just happen right this is not like Sussman's
00:55:29
technology which is going to happen because there's already really powerful forces at play humane is never a default and humane only comes out of very deliberate and conscientious design work
00:55:40
if you do the incremental thing and just you know ride the current wave of technology and let technology lead you over it leads you it's going to lead you into a tighter and tighter cage it's going to lead to more virtualization
00:55:55
more disembodiment more dehumanization and so if you believe in the possibility of a humane medium then you know it's up
00:56:09
to us to kind of fight that trend and it's up to us to make it happen thank you you
End of transcript