Select text to annotate, Click play in YouTube to begin
00:00:08
Good night, everyone. A world without apps. Now that sounds like a silly proposition because without apps your phones would look like that. It would be the desert. In fact, you would need to have a phone because you wouldn't be able to do anything with them. You wouldn't need to have any computers since without apps we cannot do anything. But of course, see it already doesn't
00:00:33
work, of course we love apps. There are app stores was literally millions of apps out there. I'm sure in this room there are several people who have more than 100 maybe 200 apps on their phones. You can count. On my phone, I don't like apps, but on my phone I have six apps to look at the public transportation in Paris.
00:00:56
Everybody who has come to our lab knows that coming from Paris to our lab is kind of a journey. And all these apps have different features but I cannot put them together into my own set of best features. And when I'm talking about apps I'm not talking just about smartphone apps. I'm talking about apps on your tablets, on your laptops, web apps, apps that are in the cloud.
00:01:23
In fact, we are so used today to juggle these things and especially this crowd here you've become expert at juggling all these devices and all these apps but think of all the people who are suffering out there trying to do the same thing. So in a sense you know Mark Weisbrot must be very happy wherever he is because ubiquitous computing has finally arrived. We have this rich ecosystem of devices, of applications,
00:01:48
of services, connectivity almost everywhere and things should be smooth. But I think ubicomp has changed meaning over the years and ubicomp is not really about ubiquitous computing anymore. It's more about ubiquitous complexity. Because as you know when you want to transfer a file from one device to another, when you want to work with someone are you
00:02:15
are you going to write his paper in LaTex or in Word. Are you on Dropbox or on Google Drive? And again these are really difficult issues for people who are not used to computers who don't understand the underlying technology. So, I think if we were to think of transferring what we're doing in the digital world in the physical world say
00:02:42
about washing clothes well I think our world would look like this. Because it would mean like for any brand of clothes you have in your closet you would need a different washing machine. And so our laundry rooms would look like this, would look like a laundromat. I think our smartphones our laundromats. We have collections of apps that do almost the same thing
00:03:05
but then don't talk to each other. And so, I think, what we want to do is to go beyond that idea that things have to be structured in this world and I see two main problems. The first one is what we call the world of information silos. Each app itself tends to be a silo that only knows
00:03:28
about a specific file format and traps you into this file format so that if you want to move your data into another application it doesn't necessarily work well. And the information silos are also the silos that we have these days in the cloud where you're forced to put your data in some kind of container in order to use a particular service. It doesn't have to be the same.
00:03:51
It doesn't have to be this way. I've worked in CSCW for a long time. We have algorithms there to manage distributed data. Things don't need to be centralized. The second problem that goes with information silos is the problem of the world gardens as some people call it. The idea is that one app is its own walled garden so once you are in the garden things are cool.
00:04:16
Everything has been designed nicely. As long as you don't know step on the flowers and stuff you're safe. But this walled garden is different from and works differently from the one next door and from one next door. And so we learn all the ideosyncrasies of each app we work with and we constantly juggle between these things. So we cannot capitalize on our knowledge of one garden to go
00:04:42
into the other. And the gates to these gardens are not as nice as this one because as you all know getting stuff in and out these walled gardens is not a nice experience. In fact, I think it looks more like that. It's like getting stuff in and out airport security. As you know some things are allowed some things are not. It depends on the airport and the country. Stuff gets lost and stolen and we get used to it,
00:05:06
but it sucks. So does it have to be that way? Well, it doesn't have to be that way. In fact, it didn't used to be that way. The very first graphical user interface, the Xerox Star GUI, commercial one commercial interface looked like that and it didn't have apps. It was designed for secretaries and secretaries don't need to know about apps. What they want to know is about document because their job is to create documents.
00:05:35
And so the Xerox Star lets you create documents with text, with images, with tables, with a histogram synchronize with a table without having to go into other apps and copying and pasting and doing fancy footwork to get all the pieces together. But it was not just about the GUI that had this idea of interfaces that are adapted to the users
00:05:58
rather than to the needs of the programmers. Remember email, still use email here? Yeah. Well email you don't need to be on the same email provider than the person you want to send email to. You know I can be on Gmail and you are an Outlook and it's fine. I can be using Apple Mail and you use Thunderbird and it's fine. Because email is defined by a protocol
00:06:22
it's not defined by a proprietary app. And these days when we look at Facebook and Twitter and Skype and all these systems out there for communication well we fall into the silo. You have to have an account on there and you can only talk to the people who have account in this service. These systems could be defined by protocols as well.
00:06:46
They don't have to be defined by provider formats. Same thing with files. We used to have text files and you had the Emacs guys and the Vi guys and you know they argued at the coffee machine but they could work on the same files. These days there are proprietary formats and again, as I said before there is the cloud. And each of these cloud services they have their own interface
00:07:11
to manage your files. They are all sort of similar and sort of different and of course they don't interoperate. And Unix was also this toolkit literally where you could put together your own tools by assembling these little bits and pieces. It didn't have a fancy interface but it did work and it allowed people to really create and customize the environments.
00:07:34
But with apps today you are given a full complex software where you don't have a choice. You have to have all of it or none of it and you have to learn it and you have to relearn it at each update and you have to learn a different version when you shift from one vendor to another. OK, so how do we fix that?
00:07:58
Well, I think we need to go back to basics and to go back to understanding how we operate in the physical world to see if we can take advantage of that to operate in the digital world. In the physical world we are really good at understanding objects, materials, matter, physical matter. We know an object can sustain our strength
00:08:24
and we can sit on it, or can contain something, or can use to be drawn or written upon. We understand this from very early on. So what I want to do is to create the same thing for digital matter. But for this to work, we need to have the equivalent of the laws of physics. We need the laws of information. We need to be able to rely on things that when I see text
00:08:49
it should be editable. When I see an image it should be interpretable as pixels or as with [INAUDIBLE] graphics. Now, the reason this is important because in the physical world our understanding of objects operates at multiple levels. The physical world is in fact very flexible.
00:09:13
Here you see two pictures we immediately recognize a pencil and a mug. And the pencil is designed so we can draw on paper and the mug is designed so you can bring your coffee. But without even thinking in a different context you're going to use the pencil to draw a line and you can use the mug as a paperweight or maybe to draw a circle. Now the way this works is because we are appropriating
00:09:37
properties of these objects that are maybe not really relevant to what they were designed for but are nevertheless present. But with digital-- with digital material this is not the case. If you use a word processor all it can ever do is edit words and text. And yes, you can paste images and yes there is a little bit of image editing capabilities
00:10:02
but if there is something missing, then you have to again shift outside and into another app. With a physical paper and the typewriter and the pencil you can appropriate paper to write this kind of thing or turn it into a musical score just by drawing lines and then drawing the notes on them. You can even do that. I can guarantee that no future version of Word
00:10:26
will ever let you do that. That's a challenge from Microsoft. Now of course there are systems that are appropriated by users. And one of the best examples is spreadsheets. Spreadsheets were designed to do boring things like this adding columns of numbers and then turning them into a cheesy 3D pie charts. But a lot of people have used spreadsheets
00:10:51
for many different things and here are three examples. People have done the animated Ninja Turtle in Excel or strategy game or this animated visualization of a New York energy consumption I believe. So people are-- we are able to do digital tools that people can appropriate. Now, to get out of this world of apps
00:11:14
and of predefined functionality and rigid interfaces I have two propositions for you. The first one is tools. We are very good at using tools. We create tools for everything. This is a small examples of woodworking tools and there are thousands and thousands of tools. In fact, the reason we're good at using tools
00:11:37
is because we've been doing this for over 3 million years. 3 million years ago, not only people we're making tools, but they were also teaching their kids how to make tools. So tool making is really a defining characteristic of humans. And what's impressive with a tool is that once you hold a tool in your hand it changes the world around you.
00:12:01
It changes the things you can do to the world. So our brain is wired for this kind of behavior. And so, what I think we should try to build is interfaces that I call instrumental interfaces. Interfaces based on tools. Now, not just tool palettes that are stuck inside applications but tools that can go from one applications to another, from one context to another.
00:12:26
If I want to design a color picking tool, while, I want to be able to select one that I like. Maybe go on the tool store and buy it for $0.99 and then put it in my environment and use it immediately. And again, tool based interfaces are already out there there's a few example there. But they think of tools only in their own context.
00:12:51
And I think the key idea here is to make tools more generic, general, and independent of the environment in which they are going to be used. Of course tools are going to be made of digital matter so tools will be also able to be operated upon with other tools and we hopefully can recreate in the digital world the same rich ecology of tools and tool
00:13:15
making tools that we have in the physical world. So that was idea one, make tools. The second thing is the one I want to talk a little bit more about is this notion of substrates that in our group we've been sort of talking about for a while now. Substrate is a word that is used in a number of scientific fields so in biology it's
00:13:40
a nourishing environment for a living organism. In material science it's a support to which you can build other things. So these are physical substrates but we also use to what I would call information substrates. When I see the thing on the left immediately I recognize it's a music score because it has the staffs and the clefs.
00:14:04
And I interpret these black dots as notes. On the right it looks very unstructured but it's a painter's palette and the painter decided to put his colors in this particular places because he was anticipating how he just going to mix them. So we constantly organize our information world on things like paper and other substrates with intent in our mind and with structures
00:14:28
that expresses what it can do and what we want to do with it. And there are substrates in the digital world already if you want to look at existing systems with this lens. So for example, on the left, the all the blue arrows are the dependencies between the cells in an Excel spreadsheet. And I can guarantee that the person who created the spreadsheet has a mental representation
00:14:52
of this network and that's the real structure of the spreadsheet. On the right we have the familiar organization of layers in tools like Photoshop and others to organize content and create a complex visual effects. So what I think is that we should think of the things we design as substrates. As digital material that can be interoperable
00:15:19
with different types of tools. Once we know we have a particular structure then we can imagine different tools to work with it but we don't have to predefine all the tools. We don't have to predefine all the properties of a substrate so that in the future new uses and new ideas and new tools can be used. So in our group we've been experimenting with this and we've been collaborating with our dear friends
00:15:44
from University of Aarhus in Denmark in particular Clements Klokmose who's here on Webstrates which was presented here in 2015. In 2017 was Codestrates which is now depicted here and yesterday Clements presented Videostrates. So we've been exploring with this idea of having a substrate where you don't need to predefine the tools.
00:16:09
You only define the type of information substrate and then you can bring and build your own tools. Now of course there are challenges if you want to push this idea because a webstrate is really great but it still works inside a web browser. It's still constrained by the limits of what the web lets you do. So, I see different levels of challenges.
00:16:33
One is that if we really want to take this idea further we need to rethink the whole stack. The whole software stack all the way to the operating system. I think operating systems today were designed into support heavyweight processes and heavyweight applications. In a world where tools are going to come and go and substrates are going to change you need something way more dynamic. We need to redefine all the libraries and layers there
00:17:01
to support this flexibility and this malleability of software. And we need to do that as well at the level of programming languages. I think programming languages these days are really inapt to controlling and defining interaction. We need languages that are reactive that are distributed. And what was the third one here, dynamic of course.
00:17:27
Because these languages were designed for an algorithmic world not an interactive world. And to finish what I want to say is to this crowd is there is a lot of opportunities in rethinking the way we create interfaces of the future with this perspective of not bundling them
00:17:50
inside of apps but in terms of interoperable substrates and tools. This applies to virtual reality, maybe to voice interfaces, certainly to tangible interaction, maybe to the crazy thing that June is going to talk about in a minute. I think if we think beyond this world of applications we can better design for the unexpected. And I think as a community this is what we should really
00:18:17
strive for. Not predefine the experiences of users of the future but rather enable the experiences that we have not even thought about. So my challenge to you is to look at Will Smith here and the Neuralyzer. Go ahead. And forget everything you know about interfaces. Imagine you wake up tomorrow your apps are gone from your laptops and smartphones
00:18:49
your data is still there but your apps are gone and because you are UIST you are tasked with redesigning the future interfaces. Do you think you will reinvent the apps if you had never seen an app before? Well, I hope not. And so I think a world without app is not the kind of desert picture I showed in the beginning but on the contrary it's this kind of luscious environments where rich substrates can create a rich ecology of tools
00:19:17
and content and digital matter. Thank you. [APPLAUSE] So we have lots of time for a Q&A. So we'll just pass the mic around we have a couple student volunteers, anybody? Thank you so much, it us really inspiring. I have one question I think. We very often don't talk about secure systems.
00:19:53
And I think also looking through the program and I think one of those things with the apps on the kernel stuff and all. I think a lot of those things make it really easy to take security into the system even if developers are not smart. A lot of the ideas you have now is sort of in a world where security doesn't matter. I completely buy in but I think nowadays
00:20:17
if I look at the challenges it may be not the interface it may be not the complexity it may be really that we are not able to create systems that are secure. Yeah very good question. That is often asked when I talk about this. So I have one answer to all future questions including this one. Which is a quote that used to be on my late stepfather's desk
00:20:46
and I often think about. Yes, security is an issue. I think as researchers we need to make choices. We cannot solve everything at once. And when I say we need to revisit the entire software's stack, what I mean is we need to work with people to build operating systems, with people who build programming
00:21:09
languages, with people who build security to understand the new trade offs we have to make, the new mechanism we have to put in place. I also strongly believe that users should have a bit more control over the security and the risks they are willing to take. We learn by doing, we learn by failing, we learn by doing mistakes, we learn by breaking things. And sometimes I want a secure environment
00:21:32
and sometimes I want to experiment and see what happens if I put my fingers in the plug. It's a way of learning. So I think, again, there is a spectrum here of things we could do. And I think it's an interesting challenge to security people to figure out how to embed security in this more fluid and flexible environments
00:21:55
but I don't have the answer today. Hi, I'm [INAUDIBLE]. I'm from Stanford University. Yes, your ideas really resonate with me like I have like a few years ago I used to jailbreak my iPhone and then-- You used, pardon. I used to jailbreak my iPhone. Oh, you jailbreak your iPhone. After you jailbreak it the first thing that you install is called Cydia Substrate and what's Cydia Substrate does is it's substrate that you can put into every IOS app.
00:22:25
You can change your functionality into IOS app and a lot of plugins in jailbreak stores it's just as you said it's like a tool that you can-- once you apply it will apply to almost every app you own. For example, one of my favorite plugin will convert the copy like if you press on [INAUDIBLE],, it will show like a copy paste and stuff. You replace this with icons and it works with every app. But it seems like nowadays like so like it seems
00:22:50
like both the platform on Google and then Apple are like fighting down does this kind of plugin stuff happening on their platform. So what do you think about this kind of movement. And how do you think those kind of plugin should do moving forward? Well, so, first we should acknowledge that the success of the iPhone that
00:23:14
created the smartphone industry is because it was safe. Again, I think the issue here is not to say that I don't think it is incompatible to have more flexible environments and security. What I'm advocating for is the ability for people to make choices. On macOS you have Gatekeeper and so you can install apps
00:23:44
and by default it won't run them unless you do the right maneuver and click the right button to say I understand what I'm doing. Or I don't understand but I'm willing to take the risk. And I think you know there is maybe something cultural here between Europe and North America. I think and as you know my wife is American and Canadian so we
00:24:10
talk about these things in North America parents tend to be very protective. In many different types of ways they drive their kids to school because they want them to walk or take the public transportation, they protect them from germs and et cetera. And I think in Europe people tend to say well,
00:24:34
you know the kid can drive, can drive, no. Well you know when I was a kid I was driving tractors when I was a well way under age like all the kids in my neighborhood. But can walk to school and it's a controlled risk and that's how you learn. So I think the issue of security is a very difficult one but the basic is we need to educate people first.
00:25:04
So then they can make their own decisions. And I don't think making all the decisions for the users is the right way to go. Hi, Daniel [INAUDIBLE] from U of T and Chatham labs. I'm thinking back to some of the early days when we were trying to get responsive interfaces into people's hands a lot more and the reaction-- like we think is as engineers, of course.
00:25:35
You just sort of specify the parameters. The interface will lay itself out properly and just relax. But I was managing design teams at Microsoft doing some of the early Surface work and the designers just flipped out if they were not allowed to control pixel level details of what the button would look like and exactly where it would be laid out in these pieces of things.
00:25:59
And I think one of the things that's really interesting to me as I think about what you've presented is what is the role of design in a world as you've described it? And especially because I think about some of the work that, well that Wendy has done, and talking about designing whole experiences and understanding the users and what is the whole task that you're trying to support?
00:26:24
And now I'm hearing this advocacy for building much smaller tools that are less responsive. Yeah very good question. If you look at your home, did you hire an interior designer to put everything in the right place in your house? Some people do but most people don't. And yet there is room for design, there is room for the design of objects,
00:26:48
there is a room for educating people on how to put the right curtains with the right sofa and stuff like this. So I think what I'm advocating is to have more again freedom to organize things the way you want. It does not preclude the fact that you could buy preset collections of tools assembled in something that would look like an app today.
00:27:13
It's like when you go to an art shop you know you can pick your individual brushes and colors and stuff yourself. Or you can buy a box where there's a starting kit for people who really don't know better. And so there is room for all of this. I think right now we are stuck in everything has to be in a box and so I just try to open that box. Michel, beautiful, thank you for this.
00:27:42
I want to pick up on two themes that at least inspired me that maybe you can unpack a little bit more. One of them is I felt one thing in the messages were missing in the work we do in the kind of products we have there's a lack of I mean I'll just say sort of like poetry that we're just sort of missing and it seemed like you were putting some guideposts on where we could kind of navigate through that. And the other one was you really showed
00:28:07
this kind of idea of inclusivity where you said there's these other tools, or other practices, or other disciplines that are not part of the conversation. And maybe there's other things that aren't part of the conversation that could really radically pivot us as a community. And so maybe give us a little more insight into the thinking of where you would like to see that go. Well, on the poetry side I'm not sure how to answer that.
00:28:34
I think we want people to be more creative. I think we want computers to empower people. It's fine that sometimes the computer does stuff for us and we delegate and there's lots of people working on very sophisticated tools for that. In our group we are more interested in how computers can empower people and make them more creative in general.
00:28:59
So I don't know if this is what you were alluding to. The other thing is, yes we need to engage more than just our community. I think it's sad that we think we don't talk enough to people we're doing the network protocols that the operating system, the programming languages and that's only within computer science. I think this conference is showing how investing the fields of digital fabrication and sensing
00:29:27
and all that opens up new worlds. Although I'm always a little bit worried about what people who are from mechanical engineering and the signal processing and all that would think of what we do and if we talk enough to them. And yes, unfortunately we have silos also in our disciplines and it would be good to break those as well.
00:29:55
Scott. I really liked this and I want to say that I think as a community this is-- there's a bunch of interesting, hard technical problems but we can do this. But it also pains me to suggest that it's at least one of the very hard problems is not technical.
00:30:18
Which is business models and interests are really in the way of doing this. How do you react to that? So you are talking about the business models? Yeah. Well, this is my answer. Now, this is a question I get a lot. I do not believe that this type of change
00:30:41
will come from the companies that dominate the computing at the moment in the same way that the GUIs come out of IBM at the time. So then the question is what is the strategy? How do we get this to work? My take and I'm not you know I'm the researcher because I would
00:31:04
be a terrible businessman. But my take is not to try to directly address the wider audience but to try with niche markets I would say. We're doing some work with other scientists because scientific scientists use a lot of computing
00:31:28
and a lot of pretty crappy apps in terms of user interfaces. And I think there is a space there a lot of the software is open source so maybe there is more wiggle room to do things. We also doing things with creative professionals and they are more willing to experiment with things and to change their practices. So I think if this happens, it will start in one of those few places and maybe
00:31:53
hopefully go viral from there. But I'm pretty sure it cannot come from big companies. Apple tried OpenDoc at some point and it was a disaster. It was too little too late and they were still building document based apps on top of app base. So they didn't they didn't get it. So, I don't know. I wish the PhD students and the future startups here
00:32:20
will be willing to explore these things. Well, thank you very much for this presentation. One of the sort of unfulfilled promises of Dynabook is end user programming. And we really made very little progress in terms of actually getting users to program their own devices. So and also I think it's a bit of a litmus test,
00:32:43
I'm not going to say Turing test but litmus test, of an interface like this if people can actually use it to program the tools. So how would you envision that and also how does that relate to what you said about we need different programming environments to create this kind of system? So what is programming? That's a good question. Because to me programming and interaction are two sides of the same coin.
00:33:06
You know with interaction you do things now and with programming you do things in the future in a sense. But there is kind of a continuum there. We're still stuck in this syntactic form of programs, whether it's visual like Max or Grasshopper or things like this. Or text, it has is very heavy structure
00:33:30
which of course matches was my idea of substrates here. Whereas interaction takes many different forms and the underlying structure in terms of programming actions for the future is not as visible. So I think-- I think we still haven't found the right bridge between these two worlds. I see the, I'm sorry I don't remember your name
00:33:54
and you gave a paper yesterday that was showing how you could manipulate the output to change the program. And so there's more things to do there. I am sure one thing is that yes we need many different programming languages and many different programming models to bridge this gap in different ways for different users for different, I'm not going to say applications I'm going to say substrates et cetera.
00:34:19
And that's a field that is still I think pretty wide open and I'm glad to see that UIST is still doing a bit of software so there is hope. Thank you. Brad Myers from CMU and I hate to be the old guy that says I did this 20 years ago, but we had a paper called the Open Data Model in 1998 that
00:34:51
called for the same thing and talked about an architecture that would allow that. And the web, the early web, had some standards like XML that would have allowed the interoperability and there were architectures for GUIs and so forth that would have allowed this. It didn't happen. I think now we have web service data, millions
00:35:19
of different APIs that you can get data from the web in a way that would interoperate. But it's still not happening so it's probably not a technical problem to make this possible. Maybe it's just too hard. Why do you think it's not happening when we've known how to do it for 20 years?
00:35:43
Well, first I agree that a number of these ideas have been surfacing in different times and in different ways and my talks with Engelbart. So, honestly I don't know but you know it took remember Larry Tesler got the CHI Lifetime Practice Award a few years ago. And he explained how it took him 15 years
00:36:08
to get copy and paste into the Xerox PARC interfaces. So I don't think I'm as good as Larry Tesler so I'm giving myself a bit more than 15 years. These things take time. The thing is I don't think this should mean we should not keep trying. The only reason for not keeping trying is if we can prove that it cannot work. Right?
00:36:34
So, unless we disprove the theory we should keep trying. Continue this over the banquet so let's give Michel a hand. [APPLAUSE]
End of transcript