Auto Scroll
Select text to annotate, Click play in YouTube to begin
i am frannie from hypothesis i'm on the marketing team here and i am so happy to be here and love this conference and love open annotation and just the
amazing people who are taking part in it and i'm going to turn it over in a minute to one of our amazing people our moderator bodong um but i want to introduce our guests today
as well we have jin ranju and yanji jung and chris andrews and like i said i'm super excited about this
and um we are recording it so if anybody ends up coming in late they'll they can watch the beginning after the fact so with that i'm gonna be quiet and turn it over to bodom
thank you so much for the introduction franny hi everybody i hope you are doing well and i know this is a international event attracting colleagues and friends from
around the globe so i'm really happy to be here and this is my second um i annotate conference um my first attendance was in
three years ago actually in 2018 uh in san francisco so really happy to be back to this community and say to see the growth of of work related to hypothesis
and annotation um my name is bo dong chem i am an associate professor from the university of minnesota and i see myself as a learning scientist who
does who does research on learning processes and also design technologies and pedagogical practices for for learning so i'm happy to really see three interesting projects represented
in this panel and i will let our panelists introduce themselves of course and their project as well a bit later um thank you for putting up the slides um
so our panelists as friendly mentioned um yunji jin from nyu xin and drew from minnesota university of minnesota and chris andrews from university indiana and i see there's um
and by the way of them are phd students actively working on cutting-edge research in the learning sciences or and also in the uh field of learning analytics and i'm actually very excited to listen
to their talks and and hear from them about what they're doing and what are they designing and what are the research in um and so on so and how um next slide please here
and this is a really a tentative agenda for this panel and the structure of this panel will be for each panelist to spend 50 minutes to present their work and then have a really quick
5-minute q a after their talk and and and after every panel let's present their work we will have a panel discussion we will invite everybody to
post uh questions along the way so that you know those questions will be either addressed after their talk or add it to the final discussion um so i hope that's um
i believe we'll have plenty of time to really listen to each other and also to have a conversation during the panel so without further ado um our first presentation will be
um from here take this from here hello everyone um thank you vodon for the great introduction so my name is sin andrew
a phd student from university of minnesota and it's really a great honor to be here to talk about collaborative annotations um in general it's really a fun and great conference i heard
lots of inspiring voices from people from all different areas and so for my interest what do i do i see myself as a learning science
researcher and learning experience designer i'm interested in learning analytics and design based research to develop educational innovations and implement learning theories into the real world
so those also have been my guide to the study i like to share in a minute so about a study as we all know hypothesis can support note-taking in many different areas so today i'd like to share how it
has been used in educational settings so this study is based on our collaboration with college-level instructors who are highlighting the hypothesis tool
so we started this design-based research to support their integration and study how collaboration how collaborative annotations work in their classrooms so here's the study designing support
for productive social interaction and knowledge co-construction in collaborative annotation so i also would like to introduce our team here hong shui who is also a phd student
at university of minnesota she might be here today as well and budo he's the moderator here as you already know and they all contributed a lot of wonderful ideas and efforts in this
project and yeah and shanna and boron just mentioned shanna in the chat has been supporting our study through at the beginning
so let's get started of the study as you may see a key element of the study is the design so what is the design as a preview we designed two scaffolding strategies to support teachers teaching and
students learning the first strategy is called the dynamic grouping strategy so this first strategy the purpose is to create or design a learning community as
the first step for example when a class has really big size like 100 students the instructors may want to divide students into smaller groups like 10 or 15 students per group
this could avoid a student be overwhelmed by a hundred annotations per rating and another example could be the instructors may want to assign
different ratings to different groups of students so they can later share their understanding of different readings and achieve a higher understanding of the topics and in general this strategy will help
the instructors to create a learning community at the first place and that can fit their teaching goal the context and the community itself and the second strategy is called the
participation role strategy with which is also today's focus so for this strategy we want to provide an opportunity for students to take more responsibility of their own learning
and support the process of their learning and to improve social interaction and knowledge co-construction i'll dive into more details later about this strategy
so why did we design this study it was initially conducted at the university of minnesota in fall 2020 when the campus shut down due to kobe 19 and then many instructors piloted to
online instruction and were looking for solutions so based on our observation there were generally two directions the first um given the limited time some instructor may want to
by using some technologies or tools to replicate their face-to-face instruction um and then another direction would be they want to transform the student-teacher relationship
by taking this opportunity so this is where our collaboration started because we share the same understanding that the effective usage of technology requires the consideration
of both technology and pedagogy so we think there is a need to design meaningful scaffoldings when using technologies sometimes even redesign of the curriculum instead of just throwing the
tool directly to students and by meaning pedagogy here the pedagogy is drawn from both theories and practice such as the computer supported
collaborative learning theory and some other learning series and based on that we want to um through the design let students take more responsibilities in learning and we want to transform the dynamics between
students and teachers embrace nanotechnology and then we want to facilitate a natural space for social interaction and also engage knowledgeable construction in online learning
so about the study design we want to support collaborative web annotation in college classrooms by designing sophisticated participation roles methods we use is a co-design between
researchers and instructors to design scaffolding roles and support their implementation with course-specific class customization and we use hypothesis as the
social annotation tool and this hypothesis in the current study has been integrated into the lms system of umn canvas so the participants were from three
fully online undergraduate classes in liberal arts and the focus of the current study is dance history class i had 13 students and one teaching assistant
and one instructor so what is the design what is the participation roles strategy in general is a generic scaffolding framework
comprising three scripted participation roles based on the computer-supported collaborative learning literature which includes a facilitator a synthesizer and a summarizer the facilitator
is responsible for stimulating conversations by finding connections seeking clarifications and encouraging their peers to consistently tag their annotations for an entire week
and for synthesizer they usually synthesize the initial ideas highlights agreement or disagreement and suggest directions for further discussion in the middle of the week for example in the dance history class
in the middle of the week there is a synchronous class discussion via zoom so the synthesizer are required to submit a paragraph or two or some nice bully points as to
synthesis before the class discussion and during the class discussion students can have a further discussion based on the sentences and other peers annotations and then for
summarizer they usually summarize group conversations at the end of the week based on both the class discussion and all the annotations for the whole class and also at the end
of week for this class every students were required to do individual reflective writing and all those roles and time frame can be adjusted by the instructor accordingly based on their
own teaching goals and context in order to study if the design worked or not work we we had two research questions
the first one is how did the activity design facilitate social annotation the second is how did the design facilitate knowledge co-construction so first we conducted a social network
analysis to study the social interaction such as the participation patterns in the collaborative annotation activity then we conducted a content analysis to study the knowledge co-construction
levels so for the coding scheme we use is a revised interaction analysis model of collaborative annotation we developed this coding scheme based on
guna vaderna's iem model and arubia and angles model of collaborative knowledge construction and we identified four levels of knowledge construction the first one is
called initiation where students started to share initial understandings and ask questions and share resources without too much elaboration and then it's called level two is
exploration when uh in this phase students started to elaborate on the text or contact personal experiences with some critical reasoning and then level three is negotiation
where students started to ask a question through critical reasoning or negotiate disagreement or connect readings with critical reasoning and the highest level level four
is called co-construction in this level students started to reach a consensus on the previous questions and they apply the knowledge or way of thinking and also they can make metacognitive
statement illustrating their learning outcome so did the design work in general the answer is yes for social interaction as you can see from the table the integrate our degree
means how many uh replies or annotations they received and were sent out so the facilitators in general sent out more replies and they reached out to more peers
and they also received more replies and from the table the betweenness constrained and dominance they're all centrality measures so the higher scores in betweenness and dominance
and lower scores in constrained meaning those students are in the center of this community so the facilitators they were always in they were always influential in the collaborative annotation
activities which means they are always in the center position and also the social interaction pattern varied across the facilitators in different weeks and for synthesizer they participated more than more than
non-role takers in terms of the numbers of posts they sent out but not as much as facilitators did since they tended to focus more on synthesizing the readings and
annotations on their own and for summarizers they participated as same as non-role takers which is also expected since the responsibility for them was to write the weekly summary on their own
and for knowledge construction the facilitator they generally ask questions or provided answers with elaboration examples critical reasoning to start and push the discussion
for example here's an example thread facilitated by one facilitator as you can see the facilitator tried to connect other two students annotations
when replying to student a and also the facilitator proposed some follow-up questions to engage thinking and engage with deeper thinking and um for knowledge construction level
also varied across the facilitators in different weeks and the synthesizer their posts were also mostly classified into levels two and level three in terms of the knowledge code
construction well for summarizer they on average contributed much less annotations in all levels most of their posts were in level two this result over in line with the scripts role in
the scaffolding framework for example because they focus on the class discussion during zoom meetings and composed summary that connected zoom discussion with annotations
so in general the result indicated that to a great extent the designed activity was enacted by students properly and then the role assignment was associated with
student social interaction pattern to some extent and different role takers may have different strategies when they're playing the roles and most importantly in weeks when role takers posted more
higher level posts the knowledge construction level from non-role takers tended to be high too so um implication of the design first we proposed a scaffolding
framework for collaborative annotation which is applicable to many college-level classes including the both the dynamic grouping strategy and the participation role strategy
and then we developed a revised interaction analysis model for collaborative annotation that is more appropriate for analysis of students discussion anchored in web documents
this can also support teaching as a reference for evaluation of the annotations finally the result of data analysis has shown promise of the designed scaffolding framework for
facilitating productive collaborative annotation in a study context in particular the first the facilitators and the synthesizers played roles in deepening collaborative
annotation so as for some final words um just to illiterate our purpose of the design we believe effective usage of technology
includes the consideration of both technology and pedagogy so for pedagogy students are not always natural collaborators and they need we need to make intentional efforts to help them become
better collaborators and for instructors they need to provide careful scaffolding and detailed guidelines for students to take various roles and the technology also
needs to connect students and teachers needs to provide a natural and effective environment for collaboration and there's a plus sign here i just want to say that this placement
doesn't mean the relationship between technology and pedagogy is linear actually they impact each other at every moment so the pedagogy can impact the technology design and development
and in return the technology can impact the pedagogy in terms of such as curriculum design and the class evaluation so um so we need to go back to the fundamental
questions to rethink the relationship between pedagogy and technology for example we call hypothesis as a note-taking tool when we discuss how discuss how to make
better notes or how to support note-taking in a better way we also want to go back to the questions that why students are taking those and how are they going to use it in the future
so then we can think how we as researchers designers and teachers can do a better job to support the process and as the fighter knows i also would like to
invite everyone to rethink the relationship between technology and pedagogy and also between students and teachers what can be done as researchers designers and teachers
so that's all for my presentation thank you very much for listening please let me know if you have any questions or suggestions thank you
uh thank you so much for the presentation shiran i already see uh or two questions in the q a section um so the question is from michael welker
and michael was asking on the assigned roles were they rotated across multiple assignments also how much orientation was needed to define those roles
for the students yeah that's really a good question so first yes the role will rotate each student will have opportunity to try each different roles and each week there's
one facilitator for one for each reading one synthesizer for and one summarizer for each reading and for orientation at the beginning of the semester uh we actually the instructors spend a lot of
time to explain how the different roles are different from each other and we kind of spend two or three weeks to try and to adjust the roles to
kind of make a co-design not even between researchers and instructors and also between the students because during the class meetings the students also shared their understanding of the roles um and
then the students and the teachers adjust the role together i i think started from week four and uh all the roads have been settled down and the activity have been
uh smoothie and um yeah that did not answer your questions all right michael says awesome okay great thank you any other questions quick
questions for you i see another question coming from shanna would you recommend using the framework level one to five for rubric for grading and or
for scaffolding for students i think uh by framework machina means the coding framework you used to analyze the code construction yeah i'll go back to that slice so
um this this coding thing has four levels um i think it can be uh work as a reference for um grading or evaluation of students annotations but sometimes i'm hesitant
about using some framework or coding scheme to grade students annotations quality because it's really hard to to see if
to identify if an annotation is is a high quality or low high quality um even when some students are in level one or level two it doesn't mean they're not learning uh maybe they're
in some phases of learning or just their learning styles but in general i think it could be used as a helpful reference for teachers to see how students are learning you know
in a specific week and how other scaffolding or support need to need to be provided to the students great thank thanks for the questions and thanks for your response this year and
i think we will have more opportunities later after the presentations to engage in those questions again and at this point i want to thank you here and for your presentation and then invite chris
to share your screen and and give us your presentation all right let's see if i can get this to work here okay so i have a somewhat weak connection uh for those that are uh
listening in i'm gonna try and uh turn on my video for a minute i had to i had to drive down the road to find a better uh spot for my um for my phone to be able to hot spot in so we'll see
uh that's why i'm in a car right now it's also raining um but uh i'm happy to be here i'm excited to uh present with uh these others
um shinran that was that was awesome i loved the um what you've been doing i've already seen your your scaffolding framework from uh that you had posted up before so i think that's awesome
so what i want to present on today um is this this is kind of part of what i'm trying to do for my dissertation which is thinking about kind of instructors and how they use it um how they use social annotation uh
particularly within these undergraduate reading and composition courses which i'll talk about in just a second um uh yeah go ahead and go to the next slide um so some of these this is these are some things that we kind of already know
about uh social annotation um and shinran talked about this uh that kind of with the pandemic many people moved online needed something to um
you know kind of mimic uh what they were already doing in their classes or just something to help them with their their online courses that move to online and many turn to social annotation um
because it's this flexible tool that we can use even in a hybrid situation or even a face to face um and i love this quote from raymie kalier and antara garcia their
new book on annotation that they mentioned that social annotation is seldom and end in and of itself rather it most frequently complements a repertoire of other educational practices
so we need to focus on you know not just social annotation and what's happening in social annotation but also kind of how it interacts with other things particularly within a course um and so that's part of where i'm kind of thinking about this in terms of
instructors um and uh and then also what happens to these annotations once they're once they're created um so uh this also comes
again here i'm i'm quoting uh bodong and and shinran here uh on their most recent article um in the information learning sciences journal that we need more in-depth qualitative
inquiries into how instructors are using social annotation and these this interaction of social annotation activities with other course activities go ahead and go to the next slide so these are some of the questions that i'm
thinking about um as as i'm you know trying to understand analyze research um you know how do instructors implement social annotation in their online undergraduate reading and composition courses again we'll
get to the context here in just a second what are actual what are instructors actually doing with the annotations and then how do these social annotations impact or align with other course activities and and course goals
next slide so some of you may have may already be familiar with this um but i'm i'm at indiana university and hypothesis and uh the department of english at indiana university have partnered
to uh to do this this project that's really uh so what i'm doing is kind of a small part of this long-term project at iu particularly with
the english w131 classes that's their freshman composition courses reading writing and inquiry but uh part of the data that i'm looking at has to do with um a couple other courses as well
this introduction to fiction and introduction to poetry courses and i've got the nice little uh covid graphic there that basically what happened is again as we all know forced online um because of the pandemic
and uh so they started to implement they've been using hypothesis in some of their courses and when they started making this this course shell for um you know all of the instructors to use for english w131 the freshman
composition course they decided that they wanted to embed hypothesis as part of this and social annotation activities um as part of this so across all of these uh there's there's more than 50
sections of english w131 they're going to be using or that have used over this this last academic year um social annotations using hypothesis um and so there's a lot of data that we're going to be
sifting through and i'm just kind of piecing off a little bit of this no that was actually perfect you like read my mind um so spring 2021 semester this is uh where the data that i'm looking at in
particular there's uh several um data sources so uh the the semester started and you know we're collecting kind of course activities that are happening
including their social annotation uh assignments that are occurring throughout the semester as well as other assignments that they're working on we had two instructor questionnaires one at the beginning of the semester or near
the beginning of the semester and one at the end to kind of get an idea of what instructors are doing with the annotations and then i also facilitated what we called an instructor inquiry team
it was really kind of like a mix between a focus group and like a professional learning group professional learning community where basically we just talked once a month ish about uh the decisions that they were
making about their social annotation use in the classroom what kinds of uh intentional decisions were they making why were they making those choices um in terms of how they were using social annotation or
um how they were uh you know what they were doing with the annotations and that sort of thing um and so you kind of see that we've got kind of diff multiple time points all across the semester of kind of
trying to understand what's going on you can go to the next slide um so i kind of mentioned this already i guess that that we have the two instructor questionnaires um uh the the instructor focus groups there
was actually there were six instructors that participated that we split into two uh two groups where we were doing these deeper dives and then uh those are part of the course artifacts we were collecting these artifacts that
um that the instructors created but some of these artifacts were artifacts that students contributed to even though it was a instructor created and i'll mention that in in just a minute so we go to the next
slide so uh this is i don't know how many of you are you know college uh football or basketball fans well right as soon as the season ends um they come out with rankings for
the next year uh even though nobody knows anything that's gonna happen and they typically typically call those their uh way too early rankings so um these are my because i have not
dived into the analysis yet um this is uh just kind of my thoughts as i've uh as i participated in some of this data collection so i haven't really done analysis yet so this is my way too early
thoughts and analysis uh so we can uh keep going i think i think i have these down one by one so i'll just tell you to go to the next piece so you can go to the next one actually perfect so uh instructor's design of
annotation activities as and and what i mean by this is that instructors how they were thinking about using um annotations uh and i i have an ore between these but i don't
necessarily mean them to be um either or uh it might be somewhere along probably a continuum of some kind um but some of the instructors were thinking about
uh the annotation space as a student-only space not really a space that they really participate in they might provide some private feedback for example within canvas the the lms
because the hypothesis we were using the hypothesis within the learning management system and so some of the instructors were thinking about is a student-only space a space where the students didn't have to worry about the
instructor uh uh making sorts of uh you know judgments or comments on uh in the public space um but there are other peers so really kind of more of a space for
peer-to-peer engagement um or some of the instructors were thinking about it as this kind of joint space where they could together um make meaning and so again there's some
and i love how uh shinron put this idea of you know technology plus pedagogy and these are some intentional decisions that these instructors are making these pedagogical decisions as they're thinking about how
they want to build this space and why they want to build that the space that way um and so that was just one of the interesting things that came out of it and then there's this idea of individual engagement with the text or threaded conversations what is what is the goal
of these social annotation activities is it in in these reading and composition courses for example is our focus really on uh analysis of the text um or and
really kind of having that conversation with the text or are we trying to uh promote this this threaded conversation um you know focus more on the interpersonal
or the um community building ideas uh within that and again not that these are mutually exclusive you can do both but just as the teachers are thinking about uh how they want to approach the the
pedagogical uh process and pedagogical decisions they're thinking about some of these things and sometimes they're focused more on i need them to understand the text and sometimes they're focused more on i want them to build community
or uh share with each other understandings um and go ahead and go to the next one and then there's this idea of accountability and exposure to the text this is sort of similar to individual
engagement with the text but um some of the instructors are thinking about um this idea of accountability uh of you know is this just social annotation is maybe an easy way to find
out if students read the text or not and maybe that's all you need it to do obviously i think many of us here in this session and many of us that have used social annotation see
other rich and and wonderful uses for it but also some instructors all they really need is they want to make sure that the students have have read the text and had been exposed to the text and so some of them aren't thinking
about um this deeper power of social annotation um or another uh you know part of this and i see uh jeremy in the chat has mentioned kind of these tensions
um that these could be portrayed that way this idea of tensions that or are they can we use these social annotations as resources for future activities
can we um and this was in particular i'm thinking one of the instructors who was having students use tags pretty heavily as their reading particularly relating it to
um conventions of a particular genre of of writing and then they would go into a google doc a shared google doc where
students could go back to the annotations and they could search for these tags um and then they could kind of add some information okay this or you know about a character or about jean
a particular convention of a genre and and start to build out this um i don't want to say a catalog i'm trying to think i need a better word for this but um they were kind of building out this
this new resource so they were using social annotations as a resource to build a new resource for when they went to their essays where they could kind of organize they kind of self-organized all of these
um all of their annotations in in kind of new ways um so again just kind of some of these interesting tensions that some of these instructors were thinking about how do we use social annotation
um and and how do we use them in different ways than maybe we haven't thought about before uh and then going to instructors use of annotations um you can go to go to the next one i think most of us are totally
we get this idea of preparing for class um you know many of them were using the social annotations they'd go read their students annotations to uh prepare for uh their their next class they would you
know the they'd be due 24 hours beforehand or something like that they would read through them um maybe to see misconceptions or ideas from the text that students missed a lot of times they could tell by what
was what was annotated and what wasn't annotated that you know if they didn't annotate a whole section of the text that uh the instructor felt like was really important then they would you know bring that up in their in their
conversations or some of them one of the students um you know mentioned they used it as a warm calling on students in the sense that they knew a student had already made an annotation had already said
something about this and they would reference their annotation and maybe either ask them to expound on that or um or maybe there wasn't uh there weren't
replies to that particular annotation in the social uh space and so they would in their synchronous session they would uh you know warm call on that student and use that annotation as kind
of a jump start for uh further discussion um obviously there's part of this uh there's another tension here where we don't want to uh
rehash a conversation that has already been had in the social annotation space um you know we want to be careful about not having a redundancy of a conversation and so kind of trying to think about
what what that might mean um but uh that we can still use the annotations in this you know in this other synchronous space as well um and then this idea of public versus private feedback i kind of
mentioned that already in terms of this student-only space versus the joint student instructor space but thinking about okay when do i make
public comments and when do i make private comments to students and and just thinking about that uh that dynamic and and trying to identify when
at what point uh or you know what is happening in the annotations that i'm going to make a public comment versus a private comment where i'm going to go into the the annotation space where all the other students can see it or i'm just going to do it within the
the canvas lms and um you know give them some some private feedback on the assignment for example so again i reiterate that these are way too early thoughts and analysis
that uh i'm looking forward to diving into it more this summer um and uh being able to have some some more specific uh and uh much more backed up by data
and quotes and and other things that i can link to to to uh promote the trustworthiness of what i'm saying so this is this is just from the top of chris's head based on his participation in uh collecting the data um
but i think some of these things are are interesting and and hopefully will bear some some really uh some more really interesting data um and i think that's i think that was my last slide there so
i think i'm i think i'm done all right this is fascinating chris i already saw some really interesting conversation going on in the chat all right i see uh something popping up
um so yumji asked the question do you want to just directly ask to chris yeah sure chris thanks for a wonderful presentation and i really enjoyed your presentation but i
have a question or your personal thought about the potential impact of different areas of content like your classes were in english but
what if like we are designing such as like stem courses such as physics or statistics like their reading materials are really different from english or like social sciences do you
have any thoughts on that yeah let me think um i i know i i have a um i have a brother-in-law who teaches
physics and he's used another social annotation platform that i won't mention because it's not hypothesis and this is a hypothesis event but no i'm i'm just kidding i know hypothesis is all for uh any kind of social annotation but
um he uh he he talked about and and we know from some other some other places like um uh oh well now i'm forgetting there's the the the website where you can see
experts annotations on um uh some science content science in the classroom is that what it's called i think um so i i think there's some really uh there's there's some really great uh
ways that we can uh i i see that annotation can be different in terms of you know there's there's the disciplinary side of it right that um so for example i was talking about the tags you know genre conventions within
english you know as a specific thing but thinking about how what kinds of tags you might use in a you know a science or a stem situation to help students organize some of the
knowledge that they're you know co-creating or recognizing um the other thing i was thinking about that you know english is is primarily text based right but when we get into
stem um and maybe that's an over generalization of english or there's uh maybe uh jeremy who i know has an english background might might take some issue with that but um that i'm also thinking about like graphs
and um and uh you know other other kinds of figures that um annotation and on those kinds of um those kinds of representations um i
could see being you know quite a bit different uh than you know thinking about a paragraph um and trying to make sense of a paragraph and and connecting to a paragraph but the this idea of data analysis um you know
of these these figures or things like that those are just some of the things that come to my mind um i feel like maybe i didn't did i did i address your question i feel like we kind of tend to i tend to go off just a little bit there
yeah i feel like you address the potential like considerations that instructors might consider in terms like uh like the annotations and interaction can be mediated but not only by the english
text but also by like ref or like formulas or the others in stem courses yeah addressing yeah and and again i think that one of the key things that i think
is just so important and i think this gets a little bit to that that quote that i shared earlier um of this idea that social annotation you know is seldom an end in and of
itself that's just getting students to socially annotate is uh you know might be fun and interesting for students but unless it's tied to some sort of course goal or
um it's helping us achieve some sort of uh you know other objective um you know you mention this idea of mediation right that we want to think about not only how does social annotation
you know mediate our accomplishment of some of these goals but then how do those social annotations then become mediators for other activities in the course um and so that's part of what i'm thinking about like if
we need to make sure we we start with that or consider those goals so what what goals are the instructors trying to accomplish and then how does you know do these social annotations
um how can they impact that and then now we can start to think about okay how am i going to get students to create the kinds of social annotations they're going to be most useful for them to accomplish this you know it could
just be an essay but it also might be some other really interesting projects that they're that they're working on so those are just some of the things i'm thinking about thank you thank you both for for the
conversation i see another question from shanna but i think at this point uh i will save that question for the later panel discussion because i think that's relevant um to a lot of the project we already
see and now i want to invite umg to share your screen and share your work to us okay uh hi everyone could you see the screen
okay hi everyone i'm yunji and i'm a phd student at nyu and i think it's a nice trans transition from uh singlang and chris's presentations to my one because
uh my one is more about in what ways we could support uh quality and better social annotation activities by using another intervention like
social annotation itself can be on learning in intervention but i want to add one more layer to that so i can introduce that idea so today's presentation is mostly about
suggest this idea and conceptualization of what i mean by data informed action or like what i mean by data informative feedback which can be used to support
better participating in social annotation activities okay so i think everyone here is very interested in social education and
we are very understanding why social annotation engagement is very critical but uh by limiting this into a higher education context especially for the course
activities i wanna say uh quality learning engagement matters a lot for student success in higher education and so especially in nyu there are several
courses that apply social annotation tools to course activities to help students a better success in high in their courses and also better understand the course
concepts then what kind of effective learning would be occur with social annotation we can think about three different dimensions about it so the first thing is by using social annotation students can
interact with peers which can be an additional ways for their learning activities and also get exposed to diverse views and increase their perspective beyond their beyond their
previous ones and another way is by participating social annotations students can increase their additional participation in course activities as well as
additional social annotation activities and also put more effort to complete their learning tasks which we can say by behavioral engagement
and of course by participating in social annotation and reading the materials and reading other students ideas about the same text students can deepen their understanding about the course contents and also
formulate new inquiry or argumentations so students can increase their conceptual and cognitive engagement in the course but uh if there if uh
if here uh there are instructors for the courses who are using social annotation uh you could see there was a lot of low quality annotations and also students share the very
diverse level of engagement commonly reported also in the literature so then my question is how can we design learning support where students can effectively
engage with social annotation which have been expected to help students so one way i want to introduce about it is providing learning providing kind of the
learning analytics dashboard but i don't want to use the term dashboard because dashboard is more like very sophisticated term but the main idea about here is what if we provide uh
providing kind of the feedback or weekly summary report that summarizing students engagement progress in social annotations so because when when students
when students participate in hypothesis students produce a lot of different data sources such as annotation contents themselves and click stream data about when and how
when when and who and who put annotation and other metadata so if we make use of these different data sources and create a
important feedback that can help students to take actions to improve their social their engagement in social annotation activities or also
change their mindset to better engaged in social annotations it might be great so my general idea here is how could we make use of the social annotation data and how could we
create a great feedback design so the expected uh expected outcome is by using this data in front of feedback
not only students but also instructors can get a better sense of what students are doing and how students are engaging in social annotations and like compared to a
reading material b reading material got get most attention so this kind of quick diagnosis can be available and also based on that instructors can provide intervention
to particular students who seemed in need or problematic and also instructors can change the order of the reading materials or take away a specific material or they can change
this kind of modification in instructions but going back to students so students are actually not only the primary source of
this learning data but also they can be the main target for this feedback so the information generate generated by students themselves can be fed back into students themselves
to help them to take actions for better social education engagement or also can help them change their attitudinal or motivational changes in social annotation and
the main reason about we are focusing on this action taking and motivational change is that uh i have been uh participating in several graduate level courses in niu we which uses
a hypothesis and other social annotation tools such as perusal but students mainly sometimes dislike just the use of social annotation they sometimes
perceive these activities as additional things that burden their coursework so we want to see but we already but the instructors see see the high value
of using social annotations for students to better conceptually engage in course topics and also increase the interaction between students not only as in offline but also
in the online spaces so that what kind of action takings can be possible based on this kind of feedback so based on the literature so there can be three different kinds of
actions so awareness can be part of action as or as a prerequisite for action so by looking at what is showing about their progress in
in social annotations students can can be better aware of their status of learning and also they can reflect on what they are doing good or what they are having
problems in their previous weeks of social education activities or if we could provide the peer references or the classes average or classes
um data then students can also monitor there and their classes progress together and they can be better motivated to keep engaging in social annotation
and also by looking at the feedback students can also increase their intention attention to the course and materials and also they can be more responsible for social annotation activities
later and also they can actively eat listening to and speaking in annotations and they can also change their argument argue their their way of
participating in social annotations when they actually look at what they are doing in the social annotations such as they can develop an argument or they can change their positioning in the threads
or they can develop the new questions or change the questions and the most most cited actions were actually the help and resource seeking behaviors
so by looking at the feedback students can further go to instructors by asking what kind of additional resources i can see or what kind of or just
asking the help to better understand the reading materials that they were not good at in social annotation at that week or they can also start they can also manage or plan their
their way of participating social annotation in a more strategic way however we all know that just simple exposure to data is does not always lead to meaningful
awareness and actionable insights and going back to that students always when they got the data and from feedback they have they can be challenged in figuring out
what to do with this and also students are usually reluctant to make decisions or take actions to improve social engagement based on this because they don't know what to do
or they were just afraid of using that and by the literature this kind of misconnection it came from students mistrusting data even though the data itself shows their
progress some students say it's not my data it's out of think or it's not showing my true progress and also students are usually having less experiences in
looking at their own learning data so this kind of deficiency deficit experiences help them to reduce their confidence in making sense of what the data tells
about them or what to do with them so and also usually those feedback tools especially the dashboard usually focused on awareness or monitoring
rather than helping students to burden prompting students to do something after the after looking at the feedback so then my other uh
question is then okay so we all know this kind of uh potential exciting opportunities but also we know the challenges so how do we design actionable data and firm
feedback so my positioning about that is the most problems of challenges in making use of the previously developed feedback
is usually from either the researchers side or just the developer side or not including students or not including teachers for designing process
so i understand the co-design conceptualization for developing this kind of feedback tools and also involving teachers for that so what i mean by co-design with
students is to design and characterize the values or analytic metrics and tool features that can help students to take actions in a way to improve their
social annotation activities so we think this kind of way can be helpful because this can address several ethical per aspect of learning analytics what i
hear say la from students perspective by involving students in designing on this kind of co-design process students can develop their awareness of what kind of data from social notations can be collected
and they can also increase their agency in their privacy control and they can increase their trust in of data use and also we want to get some idea about how to
make a balance between um letting them know about we will gonna collect your data and then gaming behaviors happening and how could we make a balance about it
so we're going to have some broad idea about iterative co-design sessions so first we can start from needs and problem analysis that students have
especially about social annotation engagement so we need some the real experiences from students and by listening then we
we can make students to develop and formulate a set of values which they think very critical for effective engagement with social annotation so what i mean by values here
is for example oh i think uh providing um question is very important for social annotation activities then this kind of questioning can be one value
and like constant interaction with peers is very important then that can be another value so by setting up the values we can think about later than what kind of analytic metrics students
really want to see so for example it can be a like the graph showing uh the progress of their great quality of this
this questioning development or it can be just a it can be written by a text just one text of oh you are doing good in this week so we want to get some rare uh
perspective from students about it and then as a final stage we want to help help students to develop the and prototype the finer tool which we expected to
purchase as kind of the weekly report or as unlike html version that visualize a set of metrics and with explanations and in addition to that we also think it's very
very important to involve instructors indeed in designing the course activities of using this kind of data and firm feedback as part of the social annotation activities
um so we think this is another step for designing so the purpose of doing that is to we we get more we want to integrate students use of data informed
feedback activities into core course activities and therefore students can increase their awareness and action taking in a way to help them better participate
in social annotation so integrating strategies of feedback is very critical for them not only to revisit the tools but also make use of the feedback tool for their annotation
engagement so we kind of suggest four different principles from the literature so the first one is we can start from needs and problem analysis for
implementation and then we need to think about integration piece which is how to incorporate this feedback use into core course activities tied to course goals
how can we introduce this feedback into students and so and by and another piece is how to implement this feedback to increase student agency in their own
learning not only just introducing the tool and just letting them use and just observe what they are doing we really want to see how students make use of
this feedback into their own learning and um the next phase is about dialogue so based on the literature it has been very critically argued that it is very
important to being transparent especially in implementing this kind of analytic feedback or learning analytics dashboard to students so in what ways to inform students of the availability
of their data collection and how to get agreement about it and how to inform students of the existence of this feedback so being a transparency
way is really important and also in a way to facilitate dialogue for their social annotation engagement between students and instructors so what i expect is also to see
how instructors and students make a dialogue and conversation mediated by this feedback so then the next thing we can think about is
okay so all of the great imaginary ideas then how could we imagine how effective this design would be or how could we analyze its impact or how could they think about in what way students make
use of it so i have several questions here so the fundamental questions about daytime feedback is during social annotation activities do students really benefit
from looking at their own data and how do students feel about it so maybe as an audience you might think about oh okay this is a good idea but i'm not fully buying this idea because
it can be an additional burden for students so i think it's important to check and confirm that students can benefit from looking at their own data
and in what ways we could improve this tool in a way a better benefiting student and another point to think about is from learning awareness to actions for
improvement improvement in social annotation activities so how do they how do students make sense of what the feedback tells about their learning is a very critical question to see and
whether and how students translate what they find in the feedback into actions they want to take for further social annotation engagement
and then in what ways and of course as i introduced several sets of specific actions i also really want to see specific convergences about how does this experience help them
take actions for facilitating their engagement annotations like awareness um like adaptation or attention like what kind of actions do they take and how does this experience finally
impact their learning attitude in the course and also motivation in social annotation further so uh because uh the reason why i present uh these kind of uh conceptual
contents is i'm not conducting any study yet but we are working on uh implementing this interesting design to the classes in the upcoming bar we into the courses
in two courses which use hypothesis or other social annotation tools for their courses so we really want to implement this design and see what happens and why something did not happen will
not happen so our my general idea about it is first uh implementing social annotation activities to students like by
having them equally participating in the reading materials using hypothesis and then see in what ways they can get the data and firm feedback and also
how could they make use of it by interacting with them and then after that i wanna see what kind of impact happening during the term and also at the end of the term so yeah this is kind of the general
model that i have in mind about how to designing and supporting the actionable social annotation engagement in in courses so
yeah uh this is the end of the presentation so feel free to ask me questions or or so if you have any questions later or feel free to reach out to me using my
emails here thanks thank you so much yamg i see a round of virtual applause and heart for your for your talk um this is great and invite
um colleagues who are attending this to post questions in the q a or raise your hand if you want to ask ung a question directly i see another already a question coming from jeremy
which is uh is implementation whether it's a typo or not i don't know or information or data informed feedback personalized report of your study going to be produced by you or by social annotation tool so
yep i think that's a good question for clarification so yeah it's actually both so first what i'm trying to do is summarize a set of the data sources that
can be measured by or collected by the hypothesis tool or other annotation tools and provide students a set of these kind of potential data sources
and then make them produce the design or ideas for what would be interesting data sources they want to see to improve their social annotations and what kind of analytic metrics they want to see in the
feedback report and after doing that i want to make them develop some idea about what the report would look like and then as a final step
i want to uh the report with instructors and me together so definitely it's not just made by the social annotation tool
but uh i have some idea about it because the the other social annotation tool such as perusal has some automatically produced analytics but it's not for
students it's usually for the teachers and it's for grading so they allow automatic rating which uh which those instructors can use but students cannot see at all
so yeah do i answer your question okay i think so i think yeah that's a great response any other question quick clarification question for yonji
before we move on to the panel discussion and um i don't see more questions in the q and a or in chat um
at this point i think um i want to thank all three panelists for your wonderful presentations and especially um thanking you for sharing your ongoing dissertation
research i think that's a very very generous of you to share your ongoing thinking which is really inspirational to me and i see a lot of connections across
different projects and and shared investment in a few principles such as co-design or centering what we know about learning
and teaching and also the researcher instructor partnership i see that across all three projects as well and so i have some questions i want to invite you
to think with us and i also want to encourage the audience to post your question to the panel you could direct your question to the everybody on the panel or you could direct your question to a specific panelist
as we launch this panel discussion so my first question as i was really thinking about the design element that is cutting across different project the question i want to ask is because
you do work directly with instructors or with students i want to learn from you a little bit more in terms of how do you see the instructors contributing to the
design or the design product in your project which could be a framework or a student dashboard or tool and any tips you have that you can share
with the audience in terms of ways to really engage the instructors to be active co-designers or contributors to your design so feel free to jump in we have a we
have a thought to share with the audience i can go first if that's okay um so i would say i think instructors they're really a key element during our
collaboration because they're experts of teaching and the curriculum design so first as we collaborated i would say we all have shifted our identities a little bit
we're we're not just researchers and they're not just educators we all became the research informed practitioners as we're all trying to turn research into practice
by co-designing the social annotation so during our design we have co-design meetings and then they shared their course objectives their insights and their teaching
strategies then based on that we introduced some scaffolding activities and collaborative learning strategies from the research literature and i also really
appreciate the strong wrapper we have established during our collaboration i think that's really important so throughout this semester we kept our routine design meetings as a chance
to share both positive or negative updates and solve problems together so our co-design has always been an ongoing process it's not just we designed the activity at the beginning
and never talked again so there are always great ideas coming from the design meetings throughout the semester with the instructors and they have inspired us a lot as well
so i think the for the tip i think the first step is to build the partnership um and respect and listen to their needs insights and confusions as we are here to solve a problem
together yeah that that's my thoughts thanks shunyan anything to add um yum g and chris yep uh i cannot so uh zinglan shared a
great uh insight about how to like the importance of involving instructors in the whole process of designing and implementing
the specific intervention for social annotation activities and i want to add some another layer about the importance of involving instructors
or teachers in not only designing the intervention itself but also another work for designing the integration work to the course
so in my case i want to um not only not only make students participate in social annotation activities i also want to do another experiment of could
introducing the the data feedback to students so in that case usually instructors have been not involved in the implementation project but rather
rather instructors are usually involved in designing the tool of this kind of report or feedback but they were not leading the main role or reading role of
introducing this kind of data to the class or students so from my uh pilot study not involving social annotation but in overall discussion activities uh
students were really wanted to see see that instructors really emphasize the importance of why we are using uh social annotations or why we are using the discussion post
and how my activities really really impacts my grade or my future career etc so i think it's really important for
us to help instructors continuously participate and make interventions during the semester during the course of the semester by mentioning the
the purpose or importance of why using the using social social annotation tools or in my case also why using data feedback for social annotations yeah
yeah and i loved that yonji and shinron both you know emphasized in their presentations it's like this co-design idea with instructors because um i mean i i'm thinking about a couple of
things like one uh coming from you know what we know about learning and and uh you know learning sciences is that context matters and uh the instructor is
typically the one who understands the context the best and so when we're trying to design or understand what's going on i mean obviously students or whoever's
in that in that context is going to understand it better than the outside researcher typically and there are other approaches that we have such as research practice partnerships or um
implementation research or there's other kinds of approaches that we can take um that kind of involve their voice more in the research but um i i think it is so important to make sure that we're getting instructors voices
in uh in in this because uh some a lot of times they're left out of this you know we focus so much on students because uh we we want to make sure that students are learning but instructors can give us so much
insight on on what is going on in the classroom and little little things that we might not think about that are really important for that particular context i'm also probably quite biased i come from a practitioner background i taught high
school for seven years before starting my phd and i really love teaching and so there's there's a background part of that as well the other thing i want to mention with um with mine that i kind of i kind of
said this that the instructor inquiry team these focus groups that we had that um i tried to i intentionally designed it as a sort of a professional to mimic sort
of a professional learning community or professional learning group with the instructors that i didn't want it to be i'm coming in and i'm going to ask you some questions and you're just going to answer but we started each session with
what like what's going on with annotations in your class like what are some challenges that you're having um with annotations that you want to talk through with the group because we had other instructors who had been using
annotations and so i wanted that space to also be kind of for them to just uh yeah have this learning community to be able to say you know what um and you know one of there were two specific instances that made me feel
like okay i i don't know if i accomplished it 100 but at least uh some of the instructors saw this as a great opportunity because one of them said you know a lot of my students are just doing these one-off annotations there's not a
lot of threaded conversations happening what are other people doing to have threaded conversations now that was also you know one of you know a potential research question or something i wanted them to talk about but instead of me coming up with that
question they already had questions and i kind of let them lead that discussion and then another one where i said okay here's the general topic that we're going to talk about in this session and uh one of the instructors the first
thing he said uh in in the session was um i'm so glad that you posed this question because i've never thought about this before and i want to talk to everybody about it
um and so i i thought those were um kind of just a couple of instances that made me realize yeah like this is like for the research part that's great like i want to make sure
that i'm getting um you know good data for my dissertation but also i want to make sure that this is useful to the instructors and getting their perspective and having them uh you know talk through some of this can really bring some rich
data um that because they're invested in it now that they're uh you know they want to have some of these conversations and so because they're invested uh it makes for much more interesting
uh data and it also benefits them i wanted to make sure that you know part of this is it's benefiting these instructors thank you thank you that's a really good strategy to think about the community uh chris i really love the strategy and
we'll try to find a way to incorporate that in our future work as well and um i'm looking at the time and i want to really invite audience to if you have any questions for the panel
to uh ask the question um in the chat or directly raise your hand you'll see uh some um some really interesting conversation going on in the chat thread
which is great and and for me since there's no more question i'm really going to take that advantage of being as a speaker on the panel uh because i've been really thinking about this
question myself and want to really listen to the panelists as well is we use the term social annotation quite often in different contexts right and as a researcher myself from the learning
sciences we emphasize social cultural and political dimensions of learning besides cognitive and so on right there's multi-dimensional phenomena learning is a multi-dimensional phenomenon so i want to really engage
the panelists and ask the question what do you think about when you listen or use the word the term social annotation what types of um what do you mean by social
or how would you interpret the word social when you when you think about social notation because i'm really curious about your thoughts on this when i think about this term social and
when i use the term social annotation i was also thinking what social i mean i mean by because i think like this social annotation can be also widely used as like online collaborative
annotation or online um like collaborative discussion text or something like other there are so many different things so my answer is
i think it can be divided into three different factors so one is by the in what ways um the collaboration is and
is mediated by so the first thing is learner and learner like of course being social means making interaction between peers or between the learners but i think it's
like a social annotation is also individual activity at some level as the other audience said before so i think it's also more about learner to context interaction so like
learners can interact with the concepts and the materials themselves or text or topics in a more conceptual way so i think that's the second thing and the third thing is also
a learner to instructor learner to teachers so i think it depends on uh how much instructors want to involve in the annotation because usually some instructors do not want to
participate in annotation at that much but if yes then i think also getting scaffold from instructors in a better way to fully understand the concepts in the
materials or uh get exposed to new ideas from instructors is also another thing i can see as a social thing thanks a lot yunji xi you can go next
okay so yeah i i really like how umd um kind of emphasized the connections between the students and teachers and the context and i have a really similar ideas and first i think uh social of course does not mean
students are just talking to each other we're discussing things all the time but in the social annotations context or the learning context the social needs to be uh needs to support learning so it's not just two people
work socially but their mind or cognition associated to the learning context are communicating socially so sometimes i like to call it social cognitive learning
which means which means we expect there will be a cognitive achievement like understanding of the topics or application of the knowledge facilitate facilitated by the social
activity and also for my understanding of social learning or social annotation students don't just learn for themselves but also they take the responsibility
to contribute to the community by sharing their voices and helping others to understand the topics but as i said students are not natural collaborators um they're not natural social cognitive
beings as well so when we talk about social annotation we need to keep in mind that there are there needs to be a sophisticated design or support to
help them achieve the goal so in conclusion i think social is something like an ecosystem um in social annotations all students our teachers will take the their shared
responsibilities to contribute to the community's knowledge co-construction and that's my understanding of the social yeah and i don't i don't know that i
would add much more i thought those were both really lovely that i really like this idea of you know that we can't we don't want to lose the the learning aspect of it um in in the
work that we're doing in particular and so um you know i like that idea of uh you know thinking about that the co-construction of of knowledge in in this space and that comes from these
intertextual inter-subjective uh connections that we're making um as we're annotating uh so i i don't i don't think i have anything else to yeah i conquered those are really good
great points um i think really adding a lot of richness to what we think about social annotation and i think about the future work that's uh in front of us as a as a community if i could use this term
to describe our social annotation research community i think there's a lot work to be done and really appreciate of you sharing your work and i hope to learn as your projects move forward and
with that uh we are five minutes over and i want to thank all the panelists for your uh contribution to the panel and also thank you uh thanks to the attendees and for
participating in this panel for any uh anything from you you want to let us know or kind of housekeeping things to um to share with the audience
um hold on i'm gonna make it so that i'm on the screen there we go i have the power uh no i just that was great thank you so much um i just want to
encourage people you know to follow this incredible work that you all are doing and again thanks to everyone who's here and um we should probably wrap this up so that
people can get to some of the next sessions um check the schedule hang out in the lounge meet people talk to people and don't forget about annotate this party later today it's the penultimate one
uh it's going to be great i've got something kind of big plan for that so anyway that's another topic thank you again and uh yeah we'll wrap this up all right
sounds good take care everyone it's so great to see you
End of transcript