Auto Scroll
Select text to annotate, Click play in YouTube to begin
greetings greetings i annotate it's really lovely to have folks here um we are entering into the final the fifth educator office hours in what has been a week-long of i annotate
i anticipate that people will be trickling into our session over the next few minutes um i'll just briefly say uh as by way of introduction my name is raymie clear i'm an assistant professor of learning
design and technology at the university of colorado in denver it's been a pleasure over this past academic year to get to know everybody who's joined us today for this particular set of office hours
and so as we get started just a reminder for folks we have a chat that people can use as they you know want to say hi tell us where they're maybe joining from tell us where they may be teaching or
what they may be doing we can also use that chat for questions that we can then upvote and bring forward into the session and we'll see who who joins us for this informal conversation this afternoon
that's all i'll mention uh by way of at least introducing myself and and the platform i'm happy to answer again logistical questions as they might arise but let's let's just say hi to who we have here uh tell us a little bit about
who you are where you're joining us from what you're doing and maybe we'll begin with laura because it's always nice to take on laurel in these in these circumstances laura let's go oh thank you thank you for having us
we're excited to be here i'm speaking for all of us even though i can really only technically speak for myself but i'm laura rochet i'm a phd candidate at indiana university
and i got involved with this project with our um hypothesis work uh over a year ago now just by way of curriculum design and bringing it into
our college composition classroom um and i've just been learning about it about the platform and about social annotation ever since on to you mary helen
thanks laura so hi everyone i'm mary helen shuglia i'm also a phd candidate at indiana university but while laura and sarah are in the rhetoric and composition side i'm on the literature side and i got involved in the project after
using hypothesis in several of the literature classes that i've taught and we're really excited to be expanding it across our whole first year composition program in some of our lower level literature classes and we're excited to discuss annotation and
specifically the work that we've done with you all today and i'll pass things over to sarah hello i'm sarah as mary helen said i am in the rhetoric and composition part of iu's english department
um i got involved in this project a little under a year ago just as i was becoming more increasingly interested in like writing studies writing program administration and stuff like that so that leaves me to
pass it to justin hi everyone my name is justin hudson and i'm an associate professor in the department of english at indiana university where i specialize in digital rhetoric and technology and learning among
other sort of hats that i get to wear but one of the things that i that i sort of spearhead is our online 131 curriculum which is our freshman composition course and i think this project really began for us last
uh well i guess have been right before the pandemic hit we were asked to redesign our online course and make some really strategic changes and one of the cool cool things we decided to do was to include a hypothesis to improve our peer-to-peer engagement among the
students in our classes and at the time we were only looking at like eight sections of an online course that coming fall but then you know pandemic uh you know university political decisions and we decided to just go with
our online course design for all of our freshman composition uh program at least the regular 131 courses which end up being about 60 sections a semester um and so our hypothesis decision early on
for a small really boutique wonderful engaged peer-to-peer dynamic became sort of a staple now of our first year writing curriculum and so we decided that we have this giant thing now facing us at scale so let's collect
some data as we were going to anyways for programmatic review uh but let's turn into a research engagement and see what we can learn from uh what we're doing how our delivery works how our design works how the students are using the tool uh and and
so on and then from that we stumbled into uh raymie through uh hypothesis uh whom i had known jeremy and hypothesis for for quite some time and um once we got hooked up with raymond and started thinking about the strategic ways we could
operate at scale and do research at scale we've opened up a lot of really fun avenues for how we're thinking about hypothesis not just as writing teachers and literature teachers but really the sort of learning principles and educational
practices that go along with either the good decisions we've made or decisions we know we may want to change coming next year as we manipulate the tool a little bit and our practices a little bit so we're happy to be here and happy to answer questions and share with you what we know or don't
know at this particular juncture um and you know update you on on things as they as they unfold that's brilliant well i just again i want to welcome justin and laura and mary helen and sarah i'm just noticing that our session has
almost doubled in size just over the course of our introductions and so i'm just going to repeat for folks who've joined us in the last few moments a bit of context and another invitation for people to say hi in the chat tell us where you're joining
from maybe tell us why you're joining this particular educator office hours if you've joined this session this is our final again educator office hours of the i annotate conference these are intended to be very informal and laid
back conversations primarily pedagogically focused not diving too deep into the technical details of any particular tool or even hypothesis what's unique perhaps about this office
hours is that we have of course not only a whole mix of wonderful educators but also people who are involved in some research and so again as justin was just providing a bit of context and we can
also drop a link into the chat in a few moments we have been researching um this particular project at uh at iu over the past year and we'll continue to do so
so everybody in this session justin laura mary hell and sarah we're all wearing multiple hats hats as as educators hats as researchers um and just to kind of kick off the conversation and see where things go
i just be really curious to hear from from all of you in you know whatever order makes sense and whatever ideas make sense um you know we've been engaged in this project for you know a year now of designing this
research of facilitating these classes of collecting this initial data but you've all seen in a very kind of hands-on way in a very kind of close way
how social annotation has been supporting students and their learning in again the particular context of english and composition etc but also how you see educators
making use of social annotation and i'd just be curious like what are some of your like real strong takeaways right now or or insights or even questions that you really have at the front of your mind
given this past year of work that we've all done together you know we uh raymond we actually had this question in our on our wednesday meeting so our regular weekend which which i missed because if i hadn't missed if you had been there
we would have been able to inform you already it was a rehearsal for this yeah that's fine uh it turns out we don't know what we want to ask right like it's uh not and i say it in this way we when we began this project
and and designed it around including the tool around peer-to-peer engagement you know it was really a way to try and maximize the the experience of online learning like we're trying to improve that for students and then we went to scale we really knew
we needed to do things that were going to be different than the traditional every lectures in the zoombox approach but as um our course is designed in such a way that students have to read five the 131 course anyways students read
five course essays and they're asked to work primarily analytically with them and then apply them in their writing and their thinking throughout the course and you know one of the things that became at least from my perspective that became really
important was okay well if we are going to include something like hypothesis or social annotation activity which i think is a wonderful learning tool right is it going to make a measurable impact on their ability to use those texts and to
leverage those texts in their writing because again as a writing teacher it comes back to how can i help the students improve their writing and i always say there's no substitute for clarity of thought so the better they understand their thinking with the text the better they're going to write with the text
and so that's where my the sort of the position or the impetus began with thinking about this as a research project which is in what ways can we determine or can we even determine if social annotation improves students understanding the text and or if it
actually can be tracked to translate into their writing in the course and so i think that's one of our primary questions where we began and then yesterday we started getting or wednesday we starting into all kinds of now that we know so much more like oh
what would be nice to ask like can we get hot spots like a map of all the text and see where the clothes are showing up right those kinds of weird wonderful things so but i'll let the others because i know that we had some really great conversations on wednesday about um where we go next now that we have our
first sort of large chunk of data uh in place to use um sarah and i have talked a lot about our interest in the way students
collaborate um in the annotations right uh we've worked we've been toying around the language of collaborative annotations versus social annotations and we've been thinking a lot about how
students respond to each other and take those responses into their writing assignments so that the reading and writing processes aren't
so separate um it's not that they read this text and then they write about it three weeks later it's that they read this text they're thinking about this text they're talking about this text and it naturally progresses into
the written form um and so our a lot of the questions that we've been asking and we've been texting about during the meetings have to do
with the student-to-student interactions um but obviously there are so many different ways for us to engage with this data because we've got information
about instructors we've got information about instructor student relationships um but it like j like i think justin was alluding to there's a lot
we've got a lot of data we have a lot of students with a lot of annotations um so trying to decide where we want to focus our energy
has been an exciting challenge especially this week as we've been having these conversations more and more something else that laura brought up on wednesday in our meeting that was really
um interesting to think about and it was things that were sort of like in the back of my mind when we think of like research at least like in our experiences with like rhetoric and composition it's more smaller like focused like case studies here's what i
did in my classroom but here we're working with this data of so many different sections and so many different students that it almost feels like overwhelming with how to even know like what exactly do we want to ask because
there's just so much of it yeah another con thing that we've been kind of thinking about and wanting to work with in terms of the data is also the student's annotations and how that might show up later in their writing so we're looking not only at their direct
annotations on the text and the ways that their as laura and sarah were saying speaking with each other and thinking about interacting in a more kind of immediate context with the text but also the ways that reflects later in their writings in
the essays that they're asked to write across the course of the semester so annotations they might make especially towards the beginning of the semester can then become relevant as they start bringing together all of those
analytical ideas and move towards later text so we're really grateful to have this access in the research project to not only the annotations themselves but also to student perceptions of their
annotations to instructor perceptions we've been working with another colleague who's not joining us in this session today but who's in the department of education he's been actually interacting with instructors directly to kind of study the ways that they've
been building their courses around this and the ways they've been asking students questions and then we'll also have access as part of the study to students written products at the end that's one thing that i'm particularly interested in is seeing the ways that their annotations and the
ways they've been able to interact with each other showing up in those final analytical essays so we'd be happy to talk about that as well yeah that's so i really appreciate that i i just also appreciate that you're sharing such
you know varied perspectives on this really broad project and i think it shows just all the different points of entry into the work which is which is really exciting um i'm looking at the chat a little bit
and and i'm also asked there is one question that's come through but i did want to kind of scroll back in the chat um and i see that we were joined by uh by candace candace welcome um in your introduction
you mentioned that you're redesigning capstone courses at your particular institution and i think that at the end of the day this project and again all the expertise that we have here
is ultimately about kind of course design very intentional course design thinking about supporting faculty how to do professional learning for instructors how to get you know new kinds of
activities into the curricula and how to support students and engage in that over time and i'm wondering if again any of our of our guests today maybe have some perspective maybe for someone like candace who is in
the process of redesigning courses and what what you've all gleaned about course design and then support for instructors when something like
social annotation and specifically hypothesis in this case is a really pretty central component to those redesign activities like what does that look like again what insights might you be able to share with
us um yeah so first i'd let's make sure we don't forget to get back to the question in the chat or the q a at some point um but i think you know for us it begins with you know what are our
what are our course design goals so as a long time you know online educator and course designer and whatnot it's always a matter of for me i focus on what kind of learning experience can we create for students i think typically speaking we design courses and
humanities especially in english we design courses around a set of readings and then figure out what we want them to do with them which is sort of backwards in terms of um you know actually how a lot of course design works which is that we want to focus on what are the learning
experiences that we need to or that we can use to accomplish our goals or our objectives and outcomes and then in my case how can we create perhaps better more engaging more dynamic learning experiences whether it's through active learning strategies whether it's through learning
technologies whether it's through you know specific kinds of assignments that make them i do a lot of digital literacy stuff so having a student make a video instead of writing a paper is a fun way or a introducing a different mode can
ask them to create and engage with the content differently so those all kind of are like floating when we begin to think about these kind of course courses a capstone course typically though is a like a seminal measure of some sort of achievement or some sort of
uh depth of engagement and from my experience as a teacher and what i've seen at least in some of the early conversations with our 131 online design is that the role of hypothesis in this
dynamic particular reading text reading critically has been um perhaps second to none like it's amazing how much of a difference the conversation uh is or how it operates simply by
having students have read and annotated before coming to class even if they didn't read the whole text if they've taken them or if they didn't annotate all they just read the colleague their peers annotations like the the conversation starts about three moves further along
and then it would before it took me a while to adjust to that um and i think you know going forward well we need to an rn we're gonna i think make some changes to our design such that our instructors are better prepared for what that looks like
in the day-to-day activity of a class not just the pre-class stuff but um so i think for me it begins with that like can i use this in this case the hypothesis specific conversation can i use social annotation to improve the
quality depth uh or overall framework of engagement that students are going to have in my course and is that going to give me a better learning experience like can i craft a better learning experience using this tool and then how can i build out of that
so it's not like a simple like well this is an easy fix it's more like this is a tool but it does enable a whole a new set of pedagogical considerations and strategies uh in the same way that if i have students work and make a podcast that also enables a
different kind of set of strategies so i think to me that's a tension that exists no matter how we approach the the course design conversation just but it's a great place to start because you're like what what if i had to pick five experiences for students what would they be
or three and build out of those and sort of work backwards from them um that's been successful for us with our online model anyways yeah sort of
extending that um i've been thinking about how an instructor's use of or implementation of something like hypothesis represents their pedagogical values so if you just
assign it if it's just that you have to do this you need to complete five annotations come to class be ready to talk about it that's a very different use from the an instructor who says that you need
to respond deliberately to three of your peers you need to ask a question of the text you need to um ask a question of a peers annotation and through
that sort of guidance it seems as if instructors have an opportunity to articulate what sort of values they have for their classroom
it's an opportunity to foster not only relationships but sort of build empathy between students consider different teaching teach them how to um engage with different perspectives and there is of course this
performativity to the meditation the social annotations right students know that it's not that they just have to um cross off the annotations from their to-do list it's that
they have an audience beyond um their instructor when they're asking questions of the text when they're responding to a specific quote for example
and so i think when it comes to course design it's not enough to simply include a tool we have to think about how we use the tool um and how we want our students
to use it so just sort of extending justin's point there yeah and that was one thing that really became evident for me moving from doing this curriculum and previous iterations of curriculum especially
in person we'd always required annotation like this that had always been a part of the class but i think one of the great boons of social annotation particularly as laura and justin were talking about is that it engages students with each other and shows them that they really
are a part of the conversation which in either a composition class or an entry-level literature class is something that's very difficult to get across to students i think in in those early things especially if they're not traditional
humanities majors or if this is not a kind of reading technique that they're familiar with they may feel very alienated from the text and not and especially if they have questions or they don't feel like they understand a particular portion of it social annotation is a really nice way
to give them that access and to provide them with kind of a low barrier to get across to start asking questions about the text to see that other students are also maybe engaging with the same portions of the text and as laura was
mentioning this also is such a great thing for the instructor because then if the students are able to do this maybe before they come into class to read that text the instructor has access to those annotations and is able to dive into that discussion
with such panache because they have been able to see where the students are confused or where the students are really excited and kind of start there or maybe see on the opposite end sections of this text that have been left completely
unannotated and maybe start there to say hmm i wonder why the students are leaving this alone was it because they were confused was it because this is a less interesting section of the text maybe it needs to be explained in class and so as opposed to replacing class
discussion i think one of the really important things about social nation is that it comes in as a compliment to it and that's something that we've seen across both in my own personal teaching as we moved into this new version of the curriculum and i think in some of the data that we
started to see from the product yeah actually mary helen you started to touch on one of the things that jeremy asked earlier i think which is some of the differences between like the writing studies
perspective of freshman composition which includes reading but a very specific targeted kind of reading towards a writing target outcome versus the literature courses which you know you spend more time in the l two are l two three four and fives which are introduction to
what poetry prose and fiction right those are the three we we work through yeah poetry drama and fiction yeah what's your dominant fiction sorry oh and then we also have a prose course but it's not in that yeah and non-non-fiction prose which which right now we are studying but
maybe next semester maybe next time yeah let's see but it does i think it raises questions about how this something like this as a decision operates um across what is a clearly related field but also
not a related field right in terms of some of the pedagogical things that we do yeah yeah jeremy for folks who are following along here jeremy had asked a question quite a few recently but one is in the q a section of our chat
about any differences perhaps between again the use of social annotation as a teaching tool and rhetoric again compared to english and you know and literature and i don't know if folks have other thoughts about that
that we might chime in on before we move to any of these other questions i mean it's it's hard for us because we don't i think mary helen's the only one who legitimately is occupied both positions teaching the leaders of course and teaching a writing
course um so you know mary helens probably has the most insight and yeah well i think i think the main difference there i mean they're both as i was saying i think it's a valuable tool for either one
as justin mentioned i think the pedagogical concerns are a bit different so in the composition classes one of the things that we're trying to work them through at the heart of the class is really having them understand
what good analysis looks like how to explain their ideas how to integrate sources and through the ways that um they can translate that into lots of different contexts so that's one of the
goals of the class is not just that they're like well i took an english class and now i know how to write for english but rather that the kinds of heuristics that we do in the composition course are translatable across all sorts of different disciplines because they're
using techniques like using a source as a lens like thinking about comparing contrasting different viewpoints and then being able to explain what's going on between them so the difference between that and kind
of the things that we're focusing on composition course and then those entry-level literature courses is that we're really thinking about the ways that each one of those genres functions and so in each literature course
we're really asking students to kind of build on they'll they will have hopefully at least if they're going through the university program as they're supposed to have already taken that first year composition course prior to taking the literature courses and so it builds on those skills
as a way to really get into close reading techniques in each one of those genres whether it's poetry drama or fiction and then to be able to talk about those as ways of seeing the world as ways of understanding what kind of what each literary
genre can do and the processes that we move through so things like um in the drama section that i taught about a year ago now time just has flown by in the past year i'm sure as it has for everyone
uh one of the things that we were able to do is to do a comparison between uh shakespeare's original text of a midsummer night's stream and then a kind of modern updated translation so i asked them to do a hypothesis assignment
on each one and to look at kind of similarities and differences across what's going on there to think about the way that an early modern tech could be translated into
something so much more modern but it retains some of those same themes and still really looking for kind of some of those higher order concerns because hopefully they already have that foundation of the excellent work that we try to ask them to do in that
composition class so i'm sure i'm kind of just talking around the question now so i welcome you guys to jump in and i know we have a bunch of other questions in the chat that's awesome no that's fantastic i'm sure that's really
awesome like really good thank you i'm learning a lot during this real quick uh so i'm going to touch on chris's question real quick uh um in the q a and then because i wouldn't mate yeah but no but i want to say i'm
trying to um uh have laura talk about the social annotation and empathy what she meant by that because i think that's a really point because i can answer chris's question relatively quickly um in that we haven't at this point
spent a lot of time thinking about the rhetorical canon as a framework for how we've been approaching uh the social annotation activity it's one of the the you know sort of tensions of being a writing studies framework of a program
that's connected to rhetoric that is informed by rhetoricians but our research is really more about because of the course design about analytical writing but i think it's fascinating to think about how some of those canon elements absolutely uh are operating and you're
in my first response gonna be like i think the primary mode of this tool would be invincio which you then later point chris i think in the chat um and because it is by design you know interpretation is also closely related to an invention you know
heuristics versus heuretics and how those terms operate in our canon so we haven't done a lot with it in terms of memory and the way you sort of operate about exporting comments and and making them available in that regard um but it is an area i think that not
like just we really haven't spent any time thinking about it yet so which is odd given that you know in theory i am a i am a rhetorician right so uh as a scholar in this field you think i would have been better at maybe paying attention to those things um but yeah so i think uh within the
canon though uh inventio is obviously connected to it um i think memory could be depending on how you want to approach it the text is revisible so there's a kind of tension there and then we have issues with like delivery and arrangement and style but
the style one is also fascinating for me like i'm intrigued by how many times students include links or images in their comments rather than just text and so that's not only it's a content choice but it's a stylistic choice i mean how many
i want more tick tocks in the comments is that is that so much to ask i don't think that's a problem right so so that's my own take on that i'm happy to have an extensive conversation about where and what that would look like but i i'm i too am fascinated by this idea that laura's brought up a couple of times
in our meetings and in here which is the relationship of the opportunity to be empathetic uh to one another and the relationship between social annotation and empathy and so lara i want to know more what do you mean and how does this work
what i think people should know is that justin is my dissertation chair um so the way he's asking this question is a form of pushing me
to define my terms and um what i will say is i don't know that i'm prepared to define my terms yet but i do think that um my interest in social annotation in
encouraging students to sit with a text and see it as a living breathing thing that they engage with along with other people is really important because there's this caricature of the academic
as an isolated sort of like loner and when we can perpetuate that image when we say that you sit with and you read alone and you write alone
um and we only read and write alone um we i think we do our students a disservice and so when i think about social annotation as a tool for developing empathy i think about it
um in that way because i think empathy happens through exposure and as you're reading through students other students your peers for commentary when you're reading their questions
you're being sort of forced to consider the fact that other people are engaging with the text differently just by nature of being different from you and um so
i don't know if i'm really answering the question i don't know that i i necessarily would know how we identify when a student is being empathetic in their comment this is something you
know raymond we chat a little bit about yesterday in our data meeting which is you know involving sentiment analysis at scale to not only maybe to determine what's going on at least with some sort of sentimental kind of tech language but specifically to maybe mark
a subset of of annotations that we would then do anymore so we use data science basically to identify a subset to do a qualitative look at or rhetorical analysis of to see what is going on in those exchanges
um but i i don't know well i'll just mention a few quick things and this is to curtis's question earlier critics welcome it's really good to have you here i think that we can we can put to some examples let's say of
research where qualitative to your to your observation careers qualitative analyses of smaller sets of annotations do show respondents whether those are students or those
are educators in some cases doing things like showing affinity kind of sharing personal experience being responsive to vulnerability sharing kind of critical introspective commentary
we might think of those as maybe aspects of empathy and looking at how those annotation exchanges provide some evidence of kind of more empathetic interaction as mediated by social
annotation and there's a lot of research i shouldn't say a lot but there are quite a few studies that also show how students perceive their learning with social annotations that's a step removed from the annotation content
but again how students perceive their use of this type of again social and collaborative experience and they do perceive it as strengthening their ability to connect to peers or appreciating diverse perspectives or
again learning as was mentioned you know here just from the fact that they can have access to their peers thinking so again there may be aspects of this that we might broadly construe as kind of empathetic reading or to jeremy's point
here and i was just trying to find some related commentary on twitter but jeremy mentioned this idea of annotation as a kind of listening and one of our k-12 colleagues who was presenting earlier this week courtney cleffman she and i had an
exchange on twitter a little while ago about annotation as very much related to listening so i'll try and pull that up and throw that into the mix here but anyways i appreciate this line of of thinking it really resonates
so the one of the chat responses is uh from jeremy uh that if they're being empathetic that definitely ought to be looked at for or for in the response of those to whom they're being empathetic towards the question um
yeah how would we then also track like audience reception of empathy in a flattened you know it's at that point it's text it's so it's it's an artifact that is strangely removed from the person even though it is never removed from the person so
like there's um one of the the things we had brought up a couple times in the practices and things we thought we might be seeing is if you know if i'm a we have students have to make so many responses and are there certain types of
annotations that become viral in terms of like they've meet a threshold of like engagement from other students and that seems like uh in theory it seems like a great idea except that if it's a group of students who only ever respond to each other's posts
and then you know it's a sub community and so then it changes the dynamic in the same way that empathy may or may not be impacted by those friendships those you know those kinds of things so there's so many layers to how we would even begin to track
or identify this even within one course right least of all what across 1300 participants i think that's what what our our final numbers are at this point um and so yeah i don't know this is these
are fascinating questions that's that's just what i'm saying like i don't know what to do with them okay we'll get there right raymie we'll get there you know i would use this because you mentioned sentiment there has not been at least in my reading of the literature
many if a few empirical studies that apply sentiment analysis to a large corpus of annotation data and i'm going to kind of leave the jargon there you know we did one study in the marginal syllabus project that actually showed that
the deeper a thread grows right so again exchanges among participants over time the more neutral the sentiment actually became which is hard to interpret against sentiment analysis depending on its specificity
um does and does not highlight certain you know notable patterns so i'll just leave that there but i think there are a lot of ways to get into this question of how you would begin to define that term back to your you know back to your point laura and
then look for you know evidence of it and then try and make claims now to justin's point about what that might mean as students are making meaning of of their readings i think something that i've been
struggling with on this team is figuring out um where i want to um invest my time
like do i want to look solely at social annotation of the text in the text do i want to look at how social annotation
um then affects subsequent assignments do i want to look at how teachers are involving themselves in the social annotation process and i
think we alluded to it earlier but raymie keeps insisting that this is a very unique and exceptional opportunity to work with an enormous data set um and so
trying to like stay open to the possibilities that the um breadth can allow for while also saying i think i'm gonna dive into this one
thing is really really challenging um and i don't know if sarah mary helena felt that way justin loves to sit in the vastness and just say who cares let's take it all on who needs sleep
um but for those of us who um don't want to age more rapidly we set boundaries right um so i guess i'm interested in how sarah and mary helen you're
negotiating rhetorical listening yes subscribe to chat go ahead hello i just felt like we couldn't talk about
um like reading with empathy without bringing in ratcliffe's ideas about rhetorical listening um i don't know when i i read her work somewhat recently as i'm preparing for like comprehensive exams this fall um
one thing that really stuck out to me about her work was how she talked about how and when engaging in like this type of rhetorical listening which is different from her for like rhetorical reading um but similar at the same time um
is that there's like a necessary two-part component or like two components here like one is like understanding the other and also like better understanding oneself and that's one thing that i think a lot about like how does writing help us better understand ourselves
um so i wonder if like we see that playing out in these annotations as well but this is something i'm like just thinking of now this is not something necessarily that we've talked about in our meetings or anytime bring a rough draft thinking sarah bring it forward this is what it's all about
right how else are we supposed to get there right that's the the question but it does it does raise um i think laura to your point right i think it may be helpful for folks to understand like what what we're dealing
with what we're looking at it would um first off every course has a set of texts they choose from so we have like a library a defined library if you will of course is there are texts that are included in their course
uh to have students read and annotate in this particular so we have a uh where we can look across let's say these five courses have all used this this text and so we can look at that as a framework we've also gathered uh
all the student at their own writing for the semester what they've produced which if you take 1300 students and you gather you know uh what is it three course assignments that are like what thousand word each or more so they're
long form essays right and we have the social annotation data we have instructor feedback data we have you know uh course section breakdowns um we have grade distributions both on assignments and
in the course uh that are available to us so the to large point it is it can be overwhelming to figure out as a researcher what we want to do but as a as a pedagogue as a teacher as a designer like
the even the the without knowing what the data suggests yet we've already learned so much in terms of you know sort of what we don't call them best practices anymore we call them vet moves right laura that's what they are um sort of the the moves that the more experienced teachers have been doing
with them uh how they're investing themselves in the tool or what their experience has been like um because they we we've used this tool in our classes for two semesters now we're going to use it again
at scale uh next fall and next spring as well but the first the first semester last fall we didn't collect any data on it because we were still getting things in place and getting all the parts and and trying to figure out how we're even going to gather all the stuff right
um and so what that did it allowed us to have a semester of like a dry run with instructors whatever there's our first exposure to hypothesis and we got a lot of feedback some good some negative mostly about course design and mostly you know about how the online experience was and
wasn't working for them um and so we made some changes but but ultimately i think that stuff is going to resonate no matter whether we get lost in the nuances of a single course analysis or like me you're like let's find let's
just look at the large scale data and see what the implications are about you know just basic descriptives like raymond which we've been talking about how many students are annotating so we you know we know for a fact that in our w-131 courses
students should complete 25 annotations for the year five annotations per uh course essay three additives which means you know original compositional kind of things and two responses there they have questions they have themes they can do
whatever they want the instructor can control that but each student should complete five per essay for a total of 25. and so already we can tell you that not every student hit the 25 annotation mark um but then we had some
students who have well over 25 annotations and so like those kinds of things are already kind of indicative of of differences between courses differences between reading and most likely as i think we're finding at least
in my initial assumption at this point is differences between instructors and i think you know instructor buy-in has been maybe one of the single most important components because i think we've been seeing higher numbers
i think from instructors who also participated in our our focus group for you know advanced engagement so there's a there's a kind of you know just basic correlations that are already occurring despite the fact that we have this
immense amount of data to try and figure out are there patterns are there things are there correlations are they statistically significant or not and how we can go about that to report this back so we can all make better decisions about how we employ these tools in our
in our classes so that's that's sort of the framework uh whether we get there i don't know but it kind of leads into the question i've had in my back pocket for quite a few colleagues um during this this whole just during
the diana take cameras i think i asked this during some of our previous educator office hours earlier in the week um and then some other sessions here and there i'd love to hear from all of you about this again the session attendance keeps
growing so maybe this will be useful for folks who are just just joining us even in the last few moments you know we've been involved in this project now again for at least a year some of us longer again depending on connections in history with justin your
whole team you've all been again wearing multiple hats and you may also have different kinds of teaching responsibilities coming into this next school year i'd just be really curious like
given what we know now or what we think we are learning now how do you see yourself employing social annotation either in classes that you're teaching or how do you see yourself supporting instructors
who might be using social annotation what are you saying to them what are you planning to do i just emailed justin about this because i'm teaching
a uh 300 level writing intensive course in the fall and i really would like to use hypothesis but um we don't unless we get like a a grant for this
research project we don't necessarily have the money for the department to pay for hypothesis to be used in my classroom and so i'm seeing that candice asks the question of what are you noticing about instructor buy and
what are the obstacles for instructor and asking my students to pay for hypothesis for me is really difficult i it when i don't have to have my students pay for texts um i don't when i don't have to
have them pay for the technologies i don't um and so i'm really trying to weigh the pros and cons of in having students invest in hypothesis for our time together
in the semester um because i have been thoroughly convinced of the benefits of its um it as a tool in the classroom um and so i am really hopeful that i'm
going to get to use it with my 300 level students and um if i when i do let's say let's manifest it when i do
um i i am really hoping to encourage at the additive comments that justin has referenced i'm really hoping that the students will ask
questions of the text and respond to one another's questions um before coming into class because it's those before conversations it's being able to reference and point to something that mary helen said in the
course text two weeks ago that gets students excited that they were heard in some capacity before they even came and stepped foot into the classroom um so that's my plan i don't know
if well well let me add real quick because i i want to hear what sarah and marielle have to say but to lars to your point though one of the the purposes of of this kind of research is to help us make the case who the department and the
college and the university like this is not just another you know it's a tool we have a lot of tools we have a lot of enterprise level agreements that we pay for and students pay for things all the time so there is that's you know this it's no more expensive than another textbook if you think about in those
regards per students but our part of my goal is to demonstrate why this tool in fact is does have measurable value and measurable impact and why every college should invest in it right for every course who wants to take advantage of it um
in the same way that we you know we pay for things like the adobe licenses the microsoft licenses that are tools that they just think are commonly acceptable kind of things right so like that's that's an ongoing purpose behind what we're doing not just because
i think it's a great learning tool but because i think it's a learning tool that needs to be invested in at the university level such that it has a i think its impact could be more measured by the way but so and then we can work out the other stuff behind the scenes right but sarah and
mary mary what have you been telling people yeah i think one of the things to kind of address candace's question is one thing that we've been seeing is especially newer instructors have been a little like it on one side of the spectrum
together either really engaged with this tool and think that it's really fantastic or very wary of it and don't really know how to use it and just think that it's another i think one of the challenges that we've seen instructors come against is that they think
falsely we think that it's just another thing to grade like it's just another kind of evaluative tool to use and so one of the things that we're hoping the study will be able to show them is both through getting to use it in practice and also through
seeing what we find in the study that having this extra access into the text this additional access not and again i think that's one of the things we're trying to show them is it's not a replacement for students doing the reading on their own for students taking notes
for having class discussion like it's not meant to supersede any of those but rather to really be a part of the entire curriculum wherein the social annotation brings another level of engagement with the text
and so i think one of the challenges that instructors may have faced in these kind of early stages especially if they haven't used it before is that it felt like just another requirement hurdle for them to do like i like justin was saying each student is
asked to make five annotations on each of the five texts throughout the semester and so then instructor's like well now do i have to go in and you know count and see like ah yes bob made five annotations bob gets five points and
that's really not the not that we shouldn't you know also give them credit for doing work but that's not the main impetus behind using the tool the main impetus is to really get them to be able to interact with each other interact with the text
and so exactly as jeremy just mentioned the chat one of the things that we're hoping to show them is that this deeper engagement with the text and these multiple mobile conversations they're able to have in addition to class discussion with social annotation
make them kind of more careful readers and writers which will result in we hope and this is another thing that we're at the beginning stages of the study so one of the hypotheses ha that we are hoping to find is that
this really does improve kind of the quality of their writing because they're spending that time engaging with it yeah and i'm thinking about hypothesis much more from like a student
perspective than a teacher perspective um because i've never taught with hypothesis before um because i was teaching public speaking last year um and that doesn't really have text that you read um but thinking of
this from like more like if i already used hypothesis i was like a student um like even in a grad class or something it's much more inviting to respond to people's comments than it is on like a canvas discussion board or
like a blackboard discussion post where sometimes teachers will say our instructors will say okay post something and then respond to two other peers and then there's like jokes about how your posts are like great job this was so interesting and they put
like a word count on how many words you need to say but something about like annotations on hypothesis is much more specific and you can dive into specific parts of the text and it's it's less like you're sort of like floating and you're like i'm going to
talk about whatever i want it's pinned down to something much more specific so i think it's more inviting in that way sarah that really echoes a comment that was made by one of our k-12 colleagues in a panel earlier this week
morgan jackson she emphasized just time and again this kind of proximal relationship to the text and how we're just continually kind of inviting students to return to that close reading and then that kind of
close and things can then build out from there but to your points here they don't kind of start in the kind of stratosphere right they're like right there in that very proximal relationship to their reading
um anyways that's that's just very much on my mind and really echoes what what you're saying um yeah yeah no well you know we there's the two couple things that have come up for me that i i'm i'm i continue to
struggle with it right and one of which is that as we have currently designed it in the online w-131 course um the annotations themselves aren't meant to be graded in terms of quality they're just meant to be like
you've completed the activity and it's meant to be a quick rate there's been some some confusion on on that which is probably part of the attention there but it's been like you know you do all five and it's because it's through our canvas lms the speedgrader pulls it out very
quickly you can identify how many things so if it's accounting system it's pretty simple but it's not make a proxy for participation justin yes it really counts as part of our participation right so um and so there so there's that uh and
then we have this ongoing question about how involved should an instructor be in in the annotations right so like um we have a number of folks who think that that's a student's space and there's a really there's something something really nice
about staying out of it letting the students annotate and stay out of the conversation others like to sign post things uh and one of the things i'm and i've been using it in my spread of rhetoric and sports class um and i stayed out of it the first time
though the class like i didn't annotate anything i just gave them the guides and but i feel like one of the advantages and that's something i'm going to do in the future is to actively participate in the annotations after they've started
annotating right not before i don't want to i want to tell them where to read but to engage in conversation with various students in the annotations and actually slow down the process because what happens to me all the time and it's because again i
mean i spent my whole life reading and trained to read and know how to read and do it well i moved through text really quickly right and i moved to discussions really quickly and this is something i do as a teacher and some students get it some students don't we just move on and that's sort of like i've always been kind of willing to
accept that but something i've been doing the last couple years is really trying to do less do more with less and to slow things down and i think that there's a an opportunity with uh something like a social annotation
to more slowly and more thorough thoroughly engage a reading to position key concepts from the course position key frames or perspectives uh and then if you engage with students through that you not only create better feedback loops between you and them
but then it moves it very quickly from a completion activity or another task list inside of your lms to um well it's an experience at that point i still think it's already a good experience for the students in the way that we have
it set up at least as ensuring accountability getting to read through the text getting to think about the text i don't think it's problematic that way i just think that depending on how much affordance you have as an instructor or how much agency you might have over your course right there's those kind of
considerations i am tenured faculty so i occupy a little bit of a different perspective than a lot of folks but um but i think there's something to this and i don't know if it's better to annotate with them or not i don't know if it's better to turn into a series of
things they complete or not and i just don't know but i i now i want to flip and try the other side because i did this version and i'm like okay i like that but what if i spent time arguing with them in the in the text right or pointing out things like
you know like we read something like casey lemon and layman's piece about being in oxford mississippi and culture and race and football and and you know they they miss some of the the significant elements of his story as being an african-american male
uh in oxford mississippi in a football culture he came from he's a iu graduate um and so there's you know like and so i in class we talk about it but it seemed like there's an opportunity to extend that more meaningfully in the text
to pin some things down before that happens but i don't i don't know how it fits in my class like this is this is my challenge mike i don't know what to do with it yet so hey this is great i know justin in terms of you know you bringing to the four
aspects of participation you mentioned accountability we've touched resign assessment nate can we bring criticism question up uh from the q a and and maybe have folks just kind of riff on that a little bit if that's
relevant maybe is one of our last questions or getting towards the end of our arrow here so folks can can see this i'm curious if if our panel has thoughts on on a
question about combining social annotation with i guess labor-based grading if folks have a sense of what that means or how that is being approached or maybe other forms of assessment as well
real quick what do we specifically refer to in labor-based grading is that contract based grading is that what we're assuming we can bring them up on stage if you want yeah i mean i want to make sure i
understand yeah unless you want to raise your hand and i'll bring you up thanks for hanging hanging in there with us folks as we start juggling some tech here but this is this is kind of what it's all about curtis welcome
hello can you hear me yep yes yes i'll even now you can see me how about that i've loved this session so much i'm also an english faculty and so the the conversation and i am also as
i've said in chat engaged in a research project that i'm currently designing for the for next a year but um i've been really interested in uh the work assau inui has been doing in
labor-based contract particularly to kind of really address issues of equity and and and relationship to white language supremacy and so uh so it's a form of contract grading that
as we've been talking about social annotation is sort of evidence of participation um one thing asau does his he really has people keep a labor log so it's not about how many
posts in the annotation but it's how much time are you spending reading how much time are you spending responding how much time are you spending annotating or responding to annotations and documenting it and learning
the kind of thinker and reader and responder you are and documenting sort of in a metacognitive way how much time you're committing to the task and and how and and and what it's doing for your learning
and then combining that labor grace with also writing of essays so that we just extract grades from the entire operation yeah so that's why i thought you were going but i just wanted
to make sure this is a diverse audience uh so asal's work is a wonderful and in a lot of conversations and i do something similar and and it comes sort of but before his work came out which is this semester actually in my sports class i used uh
the learning record method which is pig cyruson and mary barr's work which is very similar to the labor-based uh approach that style is using which is students they they document their progress to the course they have regular reflections rather engagements
focusing on what they believe they're learning and they use those evidential things and the things they create to to make the case at the end of the semester like here's what i've learned and here's the great i think i deserve based on what i've learned not based on
a performance or based on a particular outcome so it's a little more like they have to argue for the grade rather than just keep track but but it's the same idea and that's exactly what happened in my sports class this semester was they were and and several of them pointed out like you
know on our annotation assignments you know here's the kind of work that i did in those annotation assignments or they were taking like one student said i only did two per post but they were taking me so long because i was really trying to think through my response which even if it's a lie i consider it i
bought it i'm like i'm in okay you're gonna you're gonna tell me that i'm in but uh so i think there's a lot of room for that um specifically you know if you get into practice and students get comfortable with it but my problem is it's the first time students have ever
experienced anything like this um and the first time i'd actually taught this version here at iu i did a version when i was at ut austin where jeremy dean was a grad student at the time and we both sort of you know worked with pig in a number of ways but
uh it is a i i found it to be a very useful vehicle to avoid this like i'm just doing annotations because it counts for a grade and it became a another piece of evidence in the demonstration they can make for
their engagement in my course and once they once we got over that hump after about week four or five when they started to realize what i was doing i wasn't grading anything i was just responding to their work right uh then i i think we got a lot of
buy-ins a couple folks dropped out but more folks really really bought into the process i think it's i think it's a rich opportunity for any of that framework um and but i'll let you know i'm gonna do it again next spring when that exact
class comes around and i'm gonna do a little more personal participation in there but uh i think the framework is really well suited for social annotation as a learning activity well especially if you're going to have
them not talk about labor practice but then like their identity in relation to labor their positionality in relation to what they're doing or what they're being asked to do how the language is imposing on them or not you know those wonderful sort of
distinctions that oftentimes gets swept under the rug in higher education despite our commitment to diversity uh and and things like that and underrepresented populations we still have structures that don't always make those things as visible and i think
having students do that which is you know really core to what asal's work is doing and and helping us think about so yeah i think it's great i think it's perfect for that yeah this is great yeah chris thanks so much for jumping in here a little bit of improv
towards the end of our hour but i really love it um we are coming up right on the hour um you know i'm always game to kind of hang out and chat a bit although i also want to be respectful of everybody's time and
i also just want to formally thank this entire team from iu for joining us today so although some folks may need to run and maybe some folks can also stay for a few more minutes um i just want to thank on behalf of
the conference organizers and hypothesis and the broader ianate community you know everyone for joining us today mary helen and sarah and laura and of course justin uh really it's been a pleasure personally to learn alongside you for the last
year i look forward of course to our future work together but thank you for sharing again your wisdom with with everybody who's able to join us today this session has been recorded um we'll be able to you know reference it
you know down the road and again folks are welcome to hang out for a few more minutes if they're available but again you know we're at the hour and so this is you know the end of uh actually the educator office hour is part of iit but
it's also really kind of the end of my annotate so i guess we've we've made it all the way right now you can put us together now there actually is there is one more session if you still have appetite for more
immediately following this where we'll be um reflecting on the week as a whole i know people are probably too tired to do that but unfortunately i have to go be the uh taxi for my kids okay uber dad but not in the way that
that sounds good right um but i really appreciate you all having us on and letting us share our story and talk with folks and um if people have questions about what we're doing or want to know what we've done and i mean the data will eventually be
parsed our research will publish but if you just have questions about programmatic design engagement of the way we teach with things i'm sure that i speak for all of us that feel free to reach out uh email me and or email any of us or i can connect you and we can have continue our
conversations after this
End of transcript