Select text to annotate, Click play in YouTube to begin
00:00:08
it's with tremendous pleasure that i'd like to introduce our closing keynote for uh foreign 2021 nine years ago in 2012 uh
00:00:21
we wrote a blog post called soul searching and then it was one very specific observation um and i quote we suspect that if an open
00:00:34
annotation standard for annotation is successful long term that our front end will eventually be built into the browser where it belongs in addition to being duplicated enhanced or possibly reinvented by others leveraging a common
00:00:46
api that sounds like success to us it means our job on the front end right now is to create a phenomenal reference implementation for the user interface that for instance mozilla my built-in to firefox
00:00:59
i'll say it again now if annotation is ever to become mainstream and fulfill our larger dreams and ambitions then it must become native to browsers not just one browser but all browsers
00:01:11
as native to browsers as the link or the tab and that doesn't diminish the role of the indication community or particularly us at hypothesis in my opinion rather it complements it and extends it
00:01:25
in important ways and so for all of us who believe this deeply we have a new champion and that champion is david bocham david is a software developer on the web platform team at google who
00:01:37
works on chromium the engine under google chrome that is used by lots of browsers now from microsoft edge to brave opera vivaldi and of course and of course chrome itself
00:01:49
over the last several years david has helped think about and ultimately deliver a precursor to web annotation called scroll to text which is a capability now live in chrome that allows the browser to scroll to any
00:02:02
arbitrary text on any web page through a simple url enhancement and it's a feature inside of of chrome that if you go to google search results in a google search and click on on
00:02:15
a result you may see or remember seeing that when google takes you to that page it scrolls you down to the to the part of the page that referenced ultimately the search query that you made um
00:02:28
and that uses um scroll to text and in bringing that feature to chrome um david really worked um in the open um posting his code and prototype code on on github and recruiting a lot of us in
00:02:42
the annotation community to the discussion about how the feature should work and how it should relate to the w3c web webination standard so about when about a month ago um
00:02:55
he sent me an email and shared a document with me that that outlined a vision for bringing open internet in our interoperable federated annotation to chromium in a much bigger way
00:03:07
i was pretty excited and in short david's without stealing his thunder um david's concept is as close as i can imagine to come to checking all the boxes that i think would need to
00:03:21
be checked in in such an implementation so i won't get in his way any longer um let me uh allow him to explain this to you for himself david the floor is yours
00:03:32
thanks dan uh for the introduction and uh for inviting me here to speak uh i'm excited to share uh some of the explorations that we've been looking at um and and the vision that i think we're trying to build um
00:03:44
so i've got my email here uh on the screen uh feel free to reach out to me with any comments or questions uh after this talk uh and yeah if you want to i've got a couple of links in the slides
00:03:56
um so if you want to follow along i've got the link to these slides here as well but yeah i just want to start off and build a little bit of context so dan already mentioned some of this but about a year ago
00:04:10
we on the web platform team in chrome shipped an extension to url fragments that we call text fragments and a text fragment allows a url to point to a specific snippet of text on a destination page
00:04:23
so it makes links a lot more granular and precise and incidentally this is something uh that i think annotations also really want to do uh so like dan mentioned we got a lot of engagement from the annotation community in that work
00:04:35
um and that was kind of how i got introduced into like what annotations are and and what the community is trying to do so fast forward a little bit more recently our browser team
00:04:46
has actually been working on building on top of some of the work we did for text fragments and just building out like more useful sharing scenarios so actually just last week we launched
00:05:00
uh this new feature which is allows users to generate these kinds of links so if you go in the latest version of chrome you should be able to see a copy link to highlight menu item and on android you have a
00:05:13
similar option now in like the sharing hub and so now that we have all these sharing features built on this there's some really natural next steps that people are asking about for example like it would be really nice
00:05:26
if you could share a highlight with like a comment or a little bit of discussion um or maybe some reactions and like emojis and stickers are really popular in other apps what about making these highlights like persistent so when you come back to a
00:05:38
page and if you've highlighted things before maybe you want to see them again uh so when i heard a bunch of these um like something clicked in my head and i was just like oh these are annotations so when you're sharing these you can't
00:05:50
assume that the recipient is going to be using chrome or any specific uh client so this is something that would really have to work in like all browsers in the long term uh and one of the principles that we have
00:06:02
on on the web platform team in chrome is like neutrality so we don't want to build things that are specific to any one party so if like chrome or google can do something then others should also be able to have the same
00:06:14
capabilities so this brings me to what i want to talk about today which is how do we enable annotations that are built into the browser so this talk is based on an explainer
00:06:26
that we recently published so this is the document that dan was mentioning you can check it out at the link on the screen and by the way feel free to leave comments as annotations there so i've been using hypothesis and i had
00:06:40
a good discussion with dan about it but feel free to jump in and share your own thoughts so in this document we're trying to clarify a vision for how annotations might work in the browser and what we would need to add to
00:06:52
the web platform to make it possible i also discussed some of like the reasoning of why i think this is specifically a challenge for browsers since we already have annotation tools that work quite well
00:07:08
so um i just want to clarify a couple of things before we really jump in this is all really early stages i'm quite optimistic but i also don't want to over promise or give a false impression
00:07:21
this kind of a change does carry quite a bit of risk so it's very possible that like we'll do a bunch of explorations try some things here and it doesn't pan out that's just the nature of uh of the web and technology in general
00:07:35
doing this right does require a ton of work across many fronts so this is something that we want to socialize and get feedback early on so that's why i'm here we want to work out in the open and being very inclusive
00:07:48
and gathering contributions since this is a change to the web platform and it affects more than just chrome and its users and everything i'm discussing may sound somewhat vague at this point and that's because we haven't really made any decisions yet
00:08:02
we still have quite a bit of convincing to do even internally within the chrome team so please keep in mind that nothing i'm presenting today is a completely fully baked proposal or something that we've committed to these are more just ideas and then kind
00:08:14
of the directions that we're looking into but the flip side of this is is that if if you have opinions you can still this is the best time to to have influence i think uh and i'd like to focus here mostly on
00:08:26
the platform aspects uh and not so much on like the ui and product decisions um so i'll clarify a little bit more uh about what i mean here in a moment uh and the last thing is uh i'm somewhat new to annotation
00:08:39
certainly within this crowd i think and so my thinking isn't going to be super sophisticated here part of my goal is to see the discussion here and get more awareness for this work but really i'd also like to hear and learn
00:08:52
more from you all about some of these issues and how we might solve them and how best to think about them so i'm hoping that we'll have some time at the end and have a discussion but if we run at a time or if we don't
00:09:03
get to your question or just if you need some more time to digest this feel free to shoot me an email or tweet anytime or check out our github repo and leave comments there so i just want to start by explaining
00:09:18
what we're not trying to do when people hear the words browser and annotation um often immediately their mind just jumps to like here's a commenting section for the web and we really don't want to have this
00:09:30
one big commenting section there's a lot of examples i think of like large platforms with commenting and in the best cases it requires just an immense amount of work to even make it usable personally in my
00:09:44
own opinion i think like when a community is overly broad the commentary becomes less useful and less thoughtful so this is a case i think where having some decentralization
00:09:56
would be very helpful and it lets communities and groups self-organize and self-moderate at an appropriate scale for them and i think this is really helpful to mitigate some of the biggest challenges
00:10:08
and criticisms that we might face from bringing annotations to the web so i've mentioned that we want to focus on the platform aspects so i just want to clarify what i meant by that
00:10:22
very coarsely the platform is the bits that we would actually want to standardize somewhere in a specification so that clients and other players in an ecosystem can interoperate so i'm a biased browser engineer so when
00:10:35
i say web platform i'm thinking about browsers and web pages but the web is a lot bigger than just browsers there already exists the web annotation spec which is already standardized like
00:10:46
what an annotation on the web looks like but it doesn't mention web browsers uh at all i certainly missed it and as somebody who's like spent the better part of the last decade working on on the web in a browser
00:10:59
i actually only just learned about web annotation in the last year or two so this is a gap that we would actually need to bridge so the platform is things like formats and protocols
00:11:10
for annotation data which web annotation already provides uh but we also need to think about things like opt-in signals for pages uh and ways for pages to announce uh that an annotation source exists
00:11:24
uh for example like how does it how to style highlights the reason that i want to focus on this is that i think in order for the ecosystem to flourish more or less everybody kind of needs to
00:11:36
agree about these issues i don't want to talk too much about like the individual clients and how how they're implemented and how they present annotations because i think that those are things that can easily change and those are the
00:11:49
things that would likely be iterated on based on feedback and experience and there are also the things that chrome or another browser or another agent they can disagree on that and make different choices depending on their own product goals and
00:12:02
user populations and i think that this disagreement is good it allows differentiation and innovation so for example chrome serves like billions of users and so it's limited
00:12:14
in how complicated we can make our ui because it has to be usable by basically everybody on the other hand if you're an academic who spends every day annotating you might require something that's more purpose-built
00:12:27
so i think it's healthy to let clients be opinionated as long as we make sure that they're all speaking the same language
00:12:39
so i want to talk a little bit about the players in this ecosystem and how we've been thinking about them the w3c has this really great priority of constituencies
00:12:51
which puts basically the needs of users ahead of the needs of page authors ahead of the needs of implementers or browsers however now we also have
00:13:03
annotation services or annotators uh just to be clear when i say annotator i'm talking about like an annotation service uh which is not to be confused because you can also think of a user making annotations as an annotator
00:13:17
so yeah uh but if you look at this picture uh we already uh have some questions so annotators seem to be somewhere between like a user and a browser uh but are they
00:13:31
more or less important than page authors and even within users there's multiple class of users we have just everyday users browsing the web but somebody who creates an annotation on a
00:13:43
web page is also technically a user in a way there's some sort of similar to a page author in this case but not really so clearly we've got a little bit of ambiguity but i also don't want to rush to be
00:13:56
explicit about creating an ordering here because i think in a healthy ecosystem the interests of all these parties are not going to be zero sum and so we should look to find a healthy balance of interests rather than trying
00:14:08
to find like a stack ranking and elevating one over the other um so annotations aren't new um like hypothesis has existed for a long time as have other tools so i think it's worth discussing why i
00:14:23
think building this into a browser introduces some unique challenges the main one is that we have a lot of users um like a lot a lot uh and they come with a wide range of like tech literacy
00:14:37
and diversity and everything so really users need to be able to intuitively understand all of the different features in the browser and so this makes it quite challenging
00:14:49
the browser is already super complex and has like millions of features many of which you probably don't know about so even making small ui changes is really difficult and annotation i think is very ui heavy
00:15:03
unlike when a user seeks out and installs an annotation client our users might not actually know about annotations or want them at all so this is a very big difference like if you go and if you install hypothesis the
00:15:16
extension you're signaling that you actually want to see annotations this is not going to be true for for most browser users i think and since the they're not intentional users uh we have to make sure that they
00:15:29
can understand what's going on uh so it could be very bad for like security or privacy if users get confused right so if they're seeing annotations um if they might confuse them for uh
00:15:41
as coming from the page uh or coming from the browser so we really don't want this to become like a new vector for phishing or security issues um and this does affect authors i think at a different scale
00:15:53
so even with existing annotation tools where the user set was fairly limited some page authors were already fairly upset and they don't want their content annotated so there are legitimate reasons here
00:16:05
either for like branding reasons or just to avoid abuse this will definitely become a bigger issues if annotations are now visible to all the users and i think this really
00:16:17
boils down to the browser users and pages are not primarily annotation focused it's a challenge for us to figure out exactly how to mediate the different needs in the absence of having a good interest
00:16:30
signal that said i think this is a really worthy goal to pursue there's a lot of really exciting uh potential for like new ideas here um i have a couple that came to mind so
00:16:44
i've listed them here you know there's shared commentary on the web which i think is the most obvious and biggest use case and everybody here is probably familiar with it but there's also like this notion of like a more collaborative browsing
00:16:56
so i would like to see the web become less lonely being able to share commentary and reaction with friends maybe you can even publish a feed of annotations that your friends can follow and be able to tag friends on a page
00:17:09
this would be really nice you know perhaps you can have like wikipedia for example providing contextual annotations as you browse a web page or some kind of fact checking services another use case that has been pretty
00:17:21
common in the past is like allowing users to suggest corrections and provide a bit of a feedback loop with authors which would be helpful in cases where pages are abandoned
00:17:33
we also have like aggregator type sites so like if you think of like reddit or hacker news it's kind of a regular trope that people don't really read the article and they jump straight to the comments what if we could move the commentary to
00:17:45
happen in line with the article would that make the discussion better and i think providing a layer on top of the content can also enable some more automated use cases
00:17:58
so we can help users be more productive and and navigate better so for example like at work it might be helpful uh for your email client or your document client to provide links to like relevant emails and documents uh
00:18:11
depending on the page that you're looking at or having an annotation show you how long a page usually takes to read or a tag cloud of the content that it has what i'm really excited about here is that this really does open up something
00:18:24
of a new dimension and by building it as a platform i can see it being used like very creatively so that's the good um but like what makes this really hard
00:18:41
if you read our explainer there's a long list of challenges but i'm going to focus here on what i see as the the biggest ones some of them are more technical in nature i'll discuss those a little bit
00:18:55
later and i'm sure that there'll be a robust debate when the time comes to actually implement those but i think at least those are somewhat more quantifiable so i'm less worried about the technical
00:19:07
aspects it's the more subjective issues that i think are the most challenging and the ones that i want to highlight here so they have some smaller sub issues but i think they can be broadly bucketed
00:19:19
into two themes the first is moderation so how do we deal with content that's like abusive or dangerous um what about like security related uh challenges or or spam no one's gonna use
00:19:33
annotations if ninety percent of what they see are ads for pills and if there's one thing i've learned about the web is that if they can somebody always wants to sell me pills so one option is that we can pass this
00:19:45
challenge down to annotation services so if users can choose the services that they subscribe to those that do a good job of moderation will be preferred and so those like the incentive structure here would be good this seems like a really
00:19:59
attractive option but i don't think it can be the end of the story so users aren't always very intentional about what they install um so i'm sure that this image is maybe not totally in
00:20:11
uh it's a little bit of an exaggeration but i'm sure it brings back some memories for some people um so i mean if we do have this concept of a subscription um we would need to consider how we can
00:20:25
uh make it clear and safe uh and basically provide enough friction that it's difficult to do unintentionally and and easy to undo which really brings up the next question is like
00:20:37
can we make sure that annotations are always clearly attributable if users can know where an annotation is coming from it would become a lot easier for them to either block or remove bad actors
00:20:50
but this does rely on users having a good mental model of what's happening and so i don't think it's it's that simple at the very least this will need quite a bit of thought um there's also the question of like how
00:21:04
much can the browser do to detect spam or like male notes um if if there are things that we can do um there might be room for like the user agent to surface that to a user somehow
00:21:15
so for example if we can figure out that some text is is spam maybe those annotations get hidden or grayed out or at least like shifted to the bottom it does also seem like it would be
00:21:29
really useful to give services some signals and tools so that they can help do this job better so for example a lot of existing tools have a method for users to be able to
00:21:41
flag specific annotations should we standardize this kind of feedback maybe there's also other data that a browser could provide to a service that might be useful for them to do moderation
00:21:53
i don't necessarily know what all of this is yet but depending on how the ui looks like the browser might be able to tell how useful an annotation is to a user surfacing that to the service might be something that
00:22:05
would be useful so finally i think we can structure the incentives here um in a way that might discourage misuse um it wouldn't completely eliminate it but like for example if annotations are hidden behind a click
00:22:20
they become much less as a surface much less attractive as a surface for like spam so the second major challenge is uh that at least that i see is how much control do we give page
00:22:34
authors this is a complex and contentious issue and i think there's good arguments on both sides on the one hand like the universality of annotations is kind of the point
00:22:46
if annotations work only on some pages it'll be really confusing to users and they'll be much less useful and we also don't want like unscrupulous actors to be able to shut down legitimate debate and discussion
00:22:58
but on the other hand some authors don't want to deal with abuse and particularly those from vulnerable communities people also have built businesses on the web and we don't want to enable mobs to go brigading across their virtual
00:23:10
storefronts and destroying their livelihoods separate from that in some cases it may actually be better for the user experience for annotations not to be shown at least by default so for example there are some very
00:23:23
dynamic pages like basically video games for example probably wouldn't wouldn't benefit from this so i know this has already been a challenge for existing services and this is something that they've dealt
00:23:36
with in the past and i'm sure are still dealing with but bringing this to a broader audience is definitely going to amplify the stakes so personally i think that giving authors at least some way to signal their preference
00:23:49
would be a good step at the same time it seems like users should be able to override the page preference after all the browser is the user's agent but something else to consider is are
00:24:02
there any signals that the user agent can actually gather in aggregate here to help make better decisions so for example like if many users are overriding a specific page's preference
00:24:14
should the user agent start doing that for all users i'm not sure yet this isn't the question that's like necessarily answered but it does seem like that's something that different user agents could at least take different positions on
00:24:29
and personally i think that a lot of this will come down to what the default presentation looks like so there's a trade-off here to make on how how visible annotations are by default
00:24:40
we don't want them to be annoying but we do want them to be discoverable so on one hand like the more visible they are the more objections we'll get from authors
00:24:55
maybe on pages that wants to like opt out annotations and their highlights wouldn't initially be visible but maybe an icon in the ui somewhere might light up which would then show the annotations so
00:25:08
this is just to give you an idea but i did a quick little mockup here as a disclaimer this is just a mock it's not indicative of anything that we're doing yet but this does seem to me like it would
00:25:20
strike a fairly good balance but i am curious to hear what other people think about this so getting a little bit more technical here
00:25:35
everything i'm about to discuss is just ideas we haven't put down any code or even proposal text yet it's likely to change due to feedback or just meeting reality but if anybody does have strong opinions feel free to leave suggestions or
00:25:48
criticisms on our github repo so this diagram is kind of like the main one that i use to help explain how this might work in a browser the green and notched boxes at the top represent what i call an annotation
00:26:01
source a source is basically just an endpoint from which the browser can go and fetch annotation data so these would typically be an annotation service uh but for example the browser itself
00:26:14
may actually be a store of annotations as well um so you can see here there's a user box um these would be like private annotations that the user doesn't necessarily want to share so maybe annotations could also be
00:26:26
embedded directly in a url i'll discuss a little bit of these a couple of options here in a future slide the browser would fetch from all of the sources that it knows about
00:26:39
for any given page and then it would aggregate them and then display them in an annotation surface so in my view a surface would either be an installed extension or the browser's native ui
00:26:52
if an extension doesn't consume a given annotation then it can be passed over to browser ui for display so i can think of a few ways annotation sources might be added to a page and maybe there's others but this is
00:27:07
basically how i've been thinking about things you could configure it in a browser or an extension settings page so this is basically the model that we have today with existing tools and each page load would go and query
00:27:20
the source for any annotations it has this of course requires that users really trust the source um since naively implemented like their browsing history would basically be leaked to that source it'd be nice if we could make this more
00:27:33
privacy preserving for example maybe you can use a trusted proxy to fetch the annotations or you would fetch all of the annotations or enlarge buckets so that the source can't find out what you're browsing
00:27:47
another option is that urls themselves could specify a source um so this wouldn't require users to subscribe to uh to an annotator uh or do any kind of configuration so this would be really useful for like
00:28:00
aggregating websites like reddit um so this this could be a way for example i mentioned earlier of like bringing commentary to the destination page um you could imagine like linking to a news story and then saying like
00:28:12
go go to the news site and then also fetch annotations from this other separate source i think separate to that this would also be really useful for discovery since users probably aren't going to go
00:28:24
out of their way to go and install an annotation service but this is a way that they would kind of a stepping stone into getting into annotations um related to this a url itself
00:28:36
could be a source um so we could actually embed the data into the url this this kind of mirrors the text fragment approach we had this would be really nice for like very
00:28:48
ephemeral type annotations so say that you just want to share a quick comment with a friend you wouldn't even need a server in this case you would just encode all of the data that you need into the url and finally pages themselves might also
00:29:03
want to be able to specify annotations um so for example you could imagine that uh today pages already host like some commentary sections but it would be really nice if they can just dump a link rel
00:29:16
element on their page and say like annotations come from for example hypothesis and then all of the commentary on the page would actually appear as annotations so speaking about ui i think
00:29:33
the browser ui very likely would just provide like a lowest common denominator i think you can think of this as like basic text with some limited formatting there are questions of like how far should we go uh over like
00:29:47
how uh how much pages or the annotations can offer like styling maybe some pages can provide some hints as to how annotations should render just so that they fit well with the page and its theme
00:30:01
as a very basic example text fragments today already allow you to style the highlight using some css we'd probably want to allow that for annotations but um yeah there's a lot more freedom i think
00:30:14
uh in annotations i don't think that we want to allow just like arbitrary rendering certainly not like javascript but there's a lot of questions over like exactly how much control do we give
00:30:26
over presentation i do think there's still room here for extensions though so if the browser is like the lowest common denominator extensions would be able to supplement this and provide extra features that
00:30:39
aren't available in browsers so this would be really useful for just experimentation and trying out new things but also for building more specialist tools so ideally specialized features would provide
00:30:52
some kind of like a basic fallback that would at least be readable by users who don't have the extension so to give you an example like maybe the browser can't display images maybe initially
00:31:05
as a fallback we might just show like a link so that a user at least the user who doesn't have like an extension that does allow images would then be able to go and see that so to integrate well with the browser i think
00:31:20
extensions would need some new apis basically so that they could receive the annotation data from the browser it would also be helpful i think to expose some common annotation functionality so for example like text
00:31:34
matching apis so that they're consistent with the browser and we definitely wouldn't want to duplicate annotations both in the browser ui and extension ui so i have a basically an idea of how
00:31:48
that might look again this is all very speculative and then not necessarily fully baked but here you can see kind of an event-based model where for each annotation uh you can call prevent
00:32:00
default on it like you would in many other cases uh in the web platform uh and that would let an extension mark the annotation as being handled so it doesn't end up in the browser ui in that case
00:32:16
um so finally for consistency i think it would be great if we could allow extensions to directly render into the browser's ui um this would let all of like the ui and and controls be consistent but
00:32:29
let extensions add new functionality this i think would be best case scenario something like very very long term but if if this does turn out to be like a wildly successful idea i think this would be a great
00:32:42
direction to go in rather than having an extension build like its own side panel that looks very different let them kind of customize the inside of the annotation that the browser provides um so as we talked about author controls
00:32:59
might be necessary i think one important criteria here is that some authors don't necessarily control their server or domain and so we need a way for this to be applied
00:33:12
by specific page content so hosted blogs here are a common example for these kinds of authors they might not be able to set like http headers or per domain mechanism won't
00:33:25
necessarily work for them i'm not yet sure how complex or granular this needs to be it's possible authors might want to be able to block
00:33:36
by url of the annotation service this would be consistent with some other like security or like content security policy and other features on the web let you block by origin on the other
00:33:50
hand uh maybe this is a little bit overkill and it would be simpler just to have like an allow block for all annotations because again at least in my in my mind i don't think
00:34:01
that we would want to block annotations but more just like control how visible they are and how much action the user needs to take to see them so yeah that's pretty much all i've got i've got some parting thoughts
00:34:17
i think that this would be a pretty major change in how users interact with the web from today it's really exciting but it also comes with quite a bit of risk and so i think we want to be really
00:34:29
cautious and thoughtful about how we approach it we want to do this in a way that includes the whole web ecosystem um so we're trying to work in a way that's transparent and includes anybody who'd like to participate
00:34:41
and we can't do it alone for me success means that this is available in all browsers not just chrome and we have a healthy ecosystem of clients and services so i'll just reiterate again this is all
00:34:54
very early stages and it's premature to say that like chrome is committed to anything there is a chance this could all fizzle out but that said i think so far so good we've had some very supportive comments both internally and
00:35:07
externally and so i'm looking to kind of broaden the the community that's that's involved here and and get more feedback i think is is the most important thing
00:35:18
uh frankly i think like this community of and annotators and people in interested in annotations i think i kind of know i think what the opinion will be just because like people are excited here to see
00:35:30
annotations become more mainstream that's not going to be the case for all communities so i am trying to broaden those circles but to maximize the chance of success here um i'd like to work in like small
00:35:44
incremental uh stages rather than trying to like put together one big vision and then deliver it all in like a big bang um so the main benefits here are that like if we can uh this would let us deliver
00:35:57
some value much sooner and if the priorities do eventually change like that value would still remain i also think it's easier for us to get feedback in smaller chunks and we can learn from our mistakes and iterate on those
00:36:10
and this is maybe a bit of a guess but i also expect that it'll be easier to digest for the entire ecosystem if we take like small steps in incrementally um so just one thing i would note is like once we do eventually uh publish some
00:36:24
kind of proposals or build features don't be surprised if they don't completely like satisfy this vision that i've outlined we are going to try to do this in like small pieces so yeah that's basically it i would be
00:36:37
happy to answer any questions or just like hear comments and see some discussion awesome david thank you that was all really terrific um you uh you
00:36:51
definitely i think get a lot of questions and so forth that we're gonna in your paper um i will note by the way that the url that you uh pub pushed published in there which was
00:37:03
i think an alias for the github page uh isn't working for us so we we did publish the url to the to that um github page in the chat here and so that they can see but you might note
00:37:16
that um so i'll i'll just i'll uh start with with one or two um and and let uh people post more in the q a um and
00:37:28
um also at some point nate can come on and maybe pull some of these q a questions into the into the window here um but um right here at the end you talked
00:37:41
and i know you've kind of put together a proposal which is at your page and you know i really enjoyed uh annotating um that with you um but is there another
00:37:52
more large larger or more formal proposal that might come from google you know or kind of white paper or descriptive object before you got you know kind of involved
00:38:06
to work uh or got down to work uh sorry you can uh can you clarify is there a is there a more formal proposal that might come from google
00:38:18
about this work more formal than for instance your speculative paper yes um so definitely like once we actually uh have something more concrete we would be
00:38:32
putting forward like we would have a spec for this uh or like a a draft spec uh so the way we worked on this in like text fragments is we put out an explainer we had some ideas
00:38:43
uh and then we uh in chrome the way we do this is we will build the feature like behind a flag uh and we won't ship it until like we have uh approval from like our api owners and
00:38:56
the entire kind of like chrome shipping process uh but we would have like more formal uh certainly like a like i mentioned like a draft spec which would actually specify exactly what you need to do in other user agents to have
00:39:08
an interoperable implementation um i would i think like anything that we do i would also put out um another explainer i guess with more specifics um just because like specs are difficult
00:39:21
to read even for the experts um and so having something that's like human readable about like here's what we are proposing and then have some debate and iteration on that great um
00:39:34
uh let's see let me just start going uh here to the community um not in any particular order but just because i'm looking at it mark graham from the internet archive uh director of the wayback machine says
00:39:47
a couple things but um um can you speak to annotation interoperability um and maybe what that means fully expressed in the kind of implementation you're talking about
00:40:00
yeah so i think the the data and protocols um are something that we would want to be interoperable um and luckily we already have the the web annotation standard uh so i think that is something that we would want to leverage uh
00:40:13
and and basically build on that uh so in in a world like that uh an annotation created by chrome would then be viewable like hypothesis it would be viewable in safari or whoever else decides to
00:40:27
implement that of course web annotation is an incredibly detailed and large spec so for example chrome might not interpret every possible property that you can set on an annotation but at the very least they would
00:40:40
understand what those look like and yeah basically let you interoperate um brad brad myers from carnegie mellon
00:40:52
asks um uh could annotations also apply to web apps like gmail or google docs or maps which is an interesting one um so we could
00:41:03
uh says so that the interoperable nature of how we imagine annotation might apply to some of the things which implement versions of that i guess already yeah so the the distinction between like
00:41:18
a web app and a document is kind of blurry so naively yes unfortunately like dynamically generated content is something that's a challenge i think for like existing annotations already and it's something that we would
00:41:31
want to solve in the case of like maps for example i think that's one that would be like naively very challenging because you basically have like some kind of a canvas rendering and it's it's not really clear
00:41:44
like what you can attach an annotation to but we could certainly look at like providing uh api so that uh like to take maps as an example um maybe they want to actually be able to
00:41:56
specify things in their canvas that like this is an annotation like an anchor for an annotation and you can add things to it i think this would be a very like long-term uh play because generally i think like
00:42:08
until we have a lot of users and this is something people are asking for it's going to be a tough sell for for teams that already have like limited resources to go and add this but yeah it's possible it's just like
00:42:19
this wouldn't be our first priority um great matthias from norton asks how might we bridge annotation features already built within
00:42:32
a web app um like for instance uh the annotated features in norton's reader um with a broader platform so that annotations in an e-book can cross the boundary into the browser's
00:42:44
annotation stream yeah so this is interesting um if if i understand correctly maybe to put it in an example i'm more familiar with is like in google docs for example you can go and make comments
00:42:57
on a document and so that's like a form of annotation um and so how do we uh make those more consistent i guess that is something where i think like long term um
00:43:08
like docs shouldn't have to build that kind of feature themselves or at the very least should uh should be able to go and like augment the browser's ui with uh with like apis um but again this is
00:43:22
i think relates to the last answer where it's like this is like a long-term thing and if it's very popular it's some that's like a direction that we would want to go in um but it is uh it's it's possible
00:43:33
but uh yeah it requires lots of thought challenging indeed um [Music] uh chris chris aldridge uh asks how can we better
00:43:45
annotate uh embedded images uh audio video it would be nice to annotate an audio file directly that's about page rather than pages in which it's embedded that's an interesting one yeah to be honest i haven't really
00:44:00
thought about this very much um yeah embedding like rich media in particular things like videos um sorry annotating uh things like videos seems a little bit more difficult
00:44:13
um yeah this is this would be something i think we would consider later on um certainly the other direction where i think you might want to add things like videos and audio as an annotation
00:44:25
um seems like an easier problem to solve i think um but yeah in terms of like the generality of annotations it would be nice to be able to annotate videos but it would be a bit of a challenge
00:44:40
um let's see uh mark graham asks um how might you work um with mozilla um other other browsers um who are
00:44:55
not even built on chromium for instance um and he also asks how might others collaborate with you maybe two two hands of the same coin yeah so um i mean we have uh standard processes for how
00:45:08
browsers work with each other it's a bit of a challenge because different browsers have different priorities at different times and so getting all of the relevant people in the same room can sometimes be challenging certainly once i think we have something
00:45:21
a little bit more concrete i'll definitely reach out to like engineers from webkit and gecko and just make sure that at the very least they're aware of what we're doing and can provide any kind of feedback
00:45:33
if they want to if they want to be more involved and like kind of help push this along i would be all too happy to have this sorry can you just repeat the second part of the question well how might
00:45:45
others collaborate with you like uh if um you know somebody at the internet archive um had had uh ideas or capability you know notions in mind
00:45:57
yeah so the the github repo which uh i think you mentioned you dropped the link here somewhere um that would be a great place um to just go and file issues um if and when this does become a little bit more formal
00:46:09
we would want to move that github repo into the web incubating community group web platform incubation community group the ycg um so that's kind of like the the area where we go and incubate um
00:46:22
new uh projects on the web platform um to let others uh chime in and see progress and give feedback and help out if they want to um talk a little bit about timing um
00:46:37
big big thing um you know what kind you know how does this evolve given your experience and launching you know kind of other big features within chrome and what what might people expect to see
00:46:52
in in approximately time what kind of time frame yeah so this this is probably the toughest question um so just to give a little context like text fragments um from when i started working on it it
00:47:05
kind of predates me by a little bit but i think from from that until when we actually shipped was probably about two years and text fragments in comparison seems like an easy problem
00:47:16
um so uh yeah if if this does turn out to be successful i think this would be like a long-term work stream rather than just like one feature launch and then done in terms of like when you might see
00:47:29
things um that we're building i i don't actually know the answer to that question um certainly not within the next like three to four months i think at the very least this would be kind of like a half year to a year thing
00:47:42
where you might start to see like changes in chrome um i'm hopeful that we might have some kind of proposals out in before that so that at least like the the gears of it
00:47:54
uh and the process behind it might uh might be more visible but like in terms of actually getting features in the chrome that can take quite a bit of time um we got a question about accessibility have you thought about
00:48:08
um accessibility and maybe in the in the narrower sense around things like screen readers and so forth but or maybe in the broader sense just in terms of um you know general accessibility for for
00:48:22
everybody yeah i think this comes down to uh more on like the browser ui side um so for example like when when our teams would be building out the ui for this um this would be like front
00:48:34
and center of like how do you make it accessible um how do you make it uh usable by like diverse set of users so like different languages yeah i think building on like web
00:48:46
annotation i think is another thing that it's helpful here where uh w3c standards i think have to go through this already um so in terms of like uh the design of the feature itself
00:48:59
um i would expect a lot of that has already been thought through um again this is all very early so i i don't think that we've uh personally i haven't gone through and thought about this that much yet um but like by the time we get to
00:49:12
shipping this is why it actually takes so long to get features and uh into browsers is like we have to go through like all these different steps and accessibility would certainly be one of those yeah but if if there are things um that
00:49:24
we should be thinking about here this would be a great piece of feedback to get on on like our github repo um the the relationship that you've kind of laid out or the or
00:49:36
the door that you've opened in terms of suggesting that this is a general capability that services could plug into hypothesis or maybe like a reddit or
00:49:48
whatever um is is is i mean love to hear that obviously are there other examples that you can point to of
00:49:59
web features that have been built in to the browser that have this kind of service other than the browser itself which obviously the tabs are pulling content from services but other than that you know kind of
00:50:11
obvious example of a kind of a service provider browser feature relationship um that are maybe suggestive so the main analog
00:50:24
i could think of here was like the uh like the user's default search engine um so all browsers i think let you specify which search engine you want to use like when you type into the address bar um i think that it's somewhat similar it's a little
00:50:36
bit different in that you wouldn't want just like one annotation service i think you would want you might want to add like a list of them um and the other difference here is that i think like like you mentioned different pages might
00:50:49
be able to uh add services like through links or through other methods so yeah there are some analogs here but i think this is fairly uh it's a fairly new capability
00:51:04
um let's see if we have a question about vandalism of course content moderation um you know you spoke a little bit to that but um can you um
00:51:16
um you know i i guess can you do do you have any more further thoughts about um about that challenge and and in ways that you think um um you know make sense to you in
00:51:30
terms of how to approach the problem yeah i mean most of this will just be like my own opinion i think keeping things in smaller circles would definitely be helpful maybe if not even smaller just like user
00:51:43
where the user can choose what they see so for example uh we don't want the situation of like when you go to a web page and you see what the rest of the world has said about that webpage
00:51:56
maybe you just add like a circle of your close friends and whatever you go to a web page you see things that they have said about it so i think that would help quite a bit um i do also think like the default treatment that we give
00:52:07
annotations in ui can be helpful here um so for example when you visit a page uh maybe we don't show like all of the comments immediately on the page maybe we have like some kind of subtle hint that like hey there's something here
00:52:21
but you actually have to click through to go see it and so the vandalism might not be visible unless you actually go and seek it out um yeah this is like i said this is one of the biggest challenges and so any kind of ideas we have here on
00:52:34
how to better address it i'm all open ears um so this is this is a little bit of a provocative question so i'll apologize ahead of time
00:52:45
google um annotation in in you know is a lot of things it can be very public um it can be very private it can even just be personal notes um but when it is public it is
00:53:00
social um and google has experimented with some more social features before and and social networks and so forth do you how do you see this with respect to the
00:53:13
to the notion of social networks and you know obviously google i would imagine would want to be one of those service providers in a feature like this yeah i mean i think it's right that this is very social
00:53:26
um i'm not sure yet uh what like the non-chrome part of google uh would want to do certainly we haven't heard that yet um but uh yeah i think there's room for for different kinds of use cases here so
00:53:40
there are also non-social aspects to it um so like i mentioned some of like the more automated use cases um yeah i i don't think this will replace like social networks for example
00:53:52
um like there's still gonna be room for twitter and facebook and so on um but this this does feel like a bit of a social network at a smaller scale um so like the exciting thing to me would be like
00:54:04
if i can take my my small circle of like uh like a whatsapp group for example and be able to like go and read and annotate the web together that seems like the most exciting aspect of it to me
00:54:18
right um uh let's see um uh you also talked about um um incremental pieces um
00:54:31
you know some of which may even be um under the hood and not even exposed as uh as user-facing features but could be underlying uh you know kind of wiring
00:54:45
that um that the browser could um you know could support can you talk to what some of the first primitives of that kind of uh nature might be
00:54:59
um that that would obviously support a kind of a broader annotation capability yeah so uh i think like the for example having uh author uh um author controls
00:55:13
um so being able to uh specify like a preference for a page um to me that seems like a fairly easy one to kind of break out and something that would be useful um like even if we don't have annotations by the browser um like
00:55:25
that's something hypothesis and genius and others can uh can leverage uh and use so that's like one example of something that i think would could come early i think
00:55:37
we would want to look for things where like we can build a bit of ui and try this out without having to do everything so perhaps like i mentioned the idea of like you can embed
00:55:50
uh annotation content into the url um so that might be an easy thing to do just in terms of like we don't have to set up servers we don't have to set up like a lot of like the networking aspects of it but it would help us build like the ui and start to accustom users
00:56:04
and see how users react to actually seeing these um so that could be a first step um what about things like that are not traditionally thought of as like annotations like a
00:56:16
tweet or something like this is that are you imagining that the harness that you are creating might support um surfacing a tweet that tweeted about a
00:56:29
page on the web um as something that could be in line so i think this is what's really exciting and powerful about building this as like a platform feature
00:56:39
is that this is something that like we as as chrome or whoever wouldn't have to uh consider this uh if if like twitter for example wanted to surface tweets
00:56:52
uh in the form of annotations twitter could set up an annotation service and start uh serving those out so i think that's like really the uh the important part about like making things interoperable and using standards is
00:57:05
like different services could be built um and there's i'm sure there's like millions of ideas that i haven't thought of about like how this could be used and so we want to build something that can be extended and
00:57:18
new ideas built on top of it rather than just like we have an idea for how to do sharing on the web and then just like build the static version of that um what about personal note-taking so
00:57:30
one of the um things that's most uh useful about annotation is that you're uh you know you can sit there and take your own personal notes over any page on the web and
00:57:45
um in in fact annotation in certain constructs kind of in the from the web annotation point of view is identical to like a bookmark um or
00:57:57
something so tagging and bookmarking be you know um an annotation might become kind of a of a continuum rather than a set of discrete you know unique things can you
00:58:09
thought a little bit about the personal aspect of this yeah i think that is an important use case and maybe one that's actually easier than a lot of what we're talking about since it doesn't touch on those issues of like
00:58:22
uh page authors and uh content moderation um so it definitely seems more tractable uh we have like thought about like for example text fragments is like an easy way to go and do
00:58:34
highlights on the web um so like if you can make those persistent that's like a little bit of like persistent highlighting and then it's basically just a short jump to add comments to those and then you have like personal annotations
00:58:46
um so yeah we have been thinking about that um i'm not necessarily sure uh how and when that would materialize but uh that is something that that could happen like in that diagram i had us like
00:58:57
we had like a special bucket uh for the annotation sources inside of the browser um so this is something that like we could do and keep it all uh consistent with with sort of the rest of the vision
00:59:11
thank you that was amazing um really appreciate you coming um sharing your thoughts with us um this is super exciting and um i just also want to give a big shout out
00:59:24
to the way that you that you've gone about not only this but also the scroll to text functionality and kind of working in the open and bringing us uh all into the conversation
00:59:35
feels great and um and then just you know can't say thank you enough for that thank you i'm excited to see how this goes and uh yeah if i would love to see this continue so
00:59:49
yeah just uh just leave feedback and comments and ideas on the github repo terrific you
End of transcript