Select text to annotate, Click play in YouTube to begin
00:00:08
greetings greetings i annotate it's really lovely to have folks here um we are entering into the final the fifth educator office hours in what has been a week-long of i annotate
00:00:19
i anticipate that people will be trickling into our session over the next few minutes um i'll just briefly say uh as by way of introduction my name is raymie clear i'm an assistant professor of learning
00:00:32
design and technology at the university of colorado in denver it's been a pleasure over this past academic year to get to know everybody who's joined us today for this particular set of office hours
00:00:45
and so as we get started just a reminder for folks we have a chat that people can use as they you know want to say hi tell us where they're maybe joining from tell us where they may be teaching or
00:00:58
what they may be doing we can also use that chat for questions that we can then upvote and bring forward into the session and we'll see who who joins us for this informal conversation this afternoon
00:01:10
that's all i'll mention uh by way of at least introducing myself and and the platform i'm happy to answer again logistical questions as they might arise but let's let's just say hi to who we have here uh tell us a little bit about
00:01:23
who you are where you're joining us from what you're doing and maybe we'll begin with laura because it's always nice to take on laurel in these in these circumstances laura let's go oh thank you thank you for having us
00:01:36
we're excited to be here i'm speaking for all of us even though i can really only technically speak for myself but i'm laura rochet i'm a phd candidate at indiana university
00:01:46
and i got involved with this project with our um hypothesis work uh over a year ago now just by way of curriculum design and bringing it into
00:01:59
our college composition classroom um and i've just been learning about it about the platform and about social annotation ever since on to you mary helen
00:02:13
thanks laura so hi everyone i'm mary helen shuglia i'm also a phd candidate at indiana university but while laura and sarah are in the rhetoric and composition side i'm on the literature side and i got involved in the project after
00:02:26
using hypothesis in several of the literature classes that i've taught and we're really excited to be expanding it across our whole first year composition program in some of our lower level literature classes and we're excited to discuss annotation and
00:02:38
specifically the work that we've done with you all today and i'll pass things over to sarah hello i'm sarah as mary helen said i am in the rhetoric and composition part of iu's english department
00:02:52
um i got involved in this project a little under a year ago just as i was becoming more increasingly interested in like writing studies writing program administration and stuff like that so that leaves me to
00:03:06
pass it to justin hi everyone my name is justin hudson and i'm an associate professor in the department of english at indiana university where i specialize in digital rhetoric and technology and learning among
00:03:18
other sort of hats that i get to wear but one of the things that i that i sort of spearhead is our online 131 curriculum which is our freshman composition course and i think this project really began for us last
00:03:30
uh well i guess have been right before the pandemic hit we were asked to redesign our online course and make some really strategic changes and one of the cool cool things we decided to do was to include a hypothesis to improve our peer-to-peer engagement among the
00:03:42
students in our classes and at the time we were only looking at like eight sections of an online course that coming fall but then you know pandemic uh you know university political decisions and we decided to just go with
00:03:54
our online course design for all of our freshman composition uh program at least the regular 131 courses which end up being about 60 sections a semester um and so our hypothesis decision early on
00:04:07
for a small really boutique wonderful engaged peer-to-peer dynamic became sort of a staple now of our first year writing curriculum and so we decided that we have this giant thing now facing us at scale so let's collect
00:04:19
some data as we were going to anyways for programmatic review uh but let's turn into a research engagement and see what we can learn from uh what we're doing how our delivery works how our design works how the students are using the tool uh and and
00:04:32
so on and then from that we stumbled into uh raymie through uh hypothesis uh whom i had known jeremy and hypothesis for for quite some time and um once we got hooked up with raymond and started thinking about the strategic ways we could
00:04:44
operate at scale and do research at scale we've opened up a lot of really fun avenues for how we're thinking about hypothesis not just as writing teachers and literature teachers but really the sort of learning principles and educational
00:04:56
practices that go along with either the good decisions we've made or decisions we know we may want to change coming next year as we manipulate the tool a little bit and our practices a little bit so we're happy to be here and happy to answer questions and share with you what we know or don't
00:05:08
know at this particular juncture um and you know update you on on things as they as they unfold that's brilliant well i just again i want to welcome justin and laura and mary helen and sarah i'm just noticing that our session has
00:05:21
almost doubled in size just over the course of our introductions and so i'm just going to repeat for folks who've joined us in the last few moments a bit of context and another invitation for people to say hi in the chat tell us where you're joining
00:05:33
from maybe tell us why you're joining this particular educator office hours if you've joined this session this is our final again educator office hours of the i annotate conference these are intended to be very informal and laid
00:05:46
back conversations primarily pedagogically focused not diving too deep into the technical details of any particular tool or even hypothesis what's unique perhaps about this office
00:05:58
hours is that we have of course not only a whole mix of wonderful educators but also people who are involved in some research and so again as justin was just providing a bit of context and we can
00:06:10
also drop a link into the chat in a few moments we have been researching um this particular project at uh at iu over the past year and we'll continue to do so
00:06:22
so everybody in this session justin laura mary hell and sarah we're all wearing multiple hats hats as as educators hats as researchers um and just to kind of kick off the conversation and see where things go
00:06:34
i just be really curious to hear from from all of you in you know whatever order makes sense and whatever ideas make sense um you know we've been engaged in this project for you know a year now of designing this
00:06:45
research of facilitating these classes of collecting this initial data but you've all seen in a very kind of hands-on way in a very kind of close way
00:06:57
how social annotation has been supporting students and their learning in again the particular context of english and composition etc but also how you see educators
00:07:09
making use of social annotation and i'd just be curious like what are some of your like real strong takeaways right now or or insights or even questions that you really have at the front of your mind
00:07:22
given this past year of work that we've all done together you know we uh raymond we actually had this question in our on our wednesday meeting so our regular weekend which which i missed because if i hadn't missed if you had been there
00:07:37
we would have been able to inform you already it was a rehearsal for this yeah that's fine uh it turns out we don't know what we want to ask right like it's uh not and i say it in this way we when we began this project
00:07:49
and and designed it around including the tool around peer-to-peer engagement you know it was really a way to try and maximize the the experience of online learning like we're trying to improve that for students and then we went to scale we really knew
00:08:01
we needed to do things that were going to be different than the traditional every lectures in the zoombox approach but as um our course is designed in such a way that students have to read five the 131 course anyways students read
00:08:13
five course essays and they're asked to work primarily analytically with them and then apply them in their writing and their thinking throughout the course and you know one of the things that became at least from my perspective that became really
00:08:24
important was okay well if we are going to include something like hypothesis or social annotation activity which i think is a wonderful learning tool right is it going to make a measurable impact on their ability to use those texts and to
00:08:36
leverage those texts in their writing because again as a writing teacher it comes back to how can i help the students improve their writing and i always say there's no substitute for clarity of thought so the better they understand their thinking with the text the better they're going to write with the text
00:08:48
and so that's where my the sort of the position or the impetus began with thinking about this as a research project which is in what ways can we determine or can we even determine if social annotation improves students understanding the text and or if it
00:09:01
actually can be tracked to translate into their writing in the course and so i think that's one of our primary questions where we began and then yesterday we started getting or wednesday we starting into all kinds of now that we know so much more like oh
00:09:14
what would be nice to ask like can we get hot spots like a map of all the text and see where the clothes are showing up right those kinds of weird wonderful things so but i'll let the others because i know that we had some really great conversations on wednesday about um where we go next now that we have our
00:09:27
first sort of large chunk of data uh in place to use um sarah and i have talked a lot about our interest in the way students
00:09:41
collaborate um in the annotations right uh we've worked we've been toying around the language of collaborative annotations versus social annotations and we've been thinking a lot about how
00:09:55
students respond to each other and take those responses into their writing assignments so that the reading and writing processes aren't
00:10:08
so separate um it's not that they read this text and then they write about it three weeks later it's that they read this text they're thinking about this text they're talking about this text and it naturally progresses into
00:10:21
the written form um and so our a lot of the questions that we've been asking and we've been texting about during the meetings have to do
00:10:33
with the student-to-student interactions um but obviously there are so many different ways for us to engage with this data because we've got information
00:10:47
about instructors we've got information about instructor student relationships um but it like j like i think justin was alluding to there's a lot
00:10:58
we've got a lot of data we have a lot of students with a lot of annotations um so trying to decide where we want to focus our energy
00:11:12
has been an exciting challenge especially this week as we've been having these conversations more and more something else that laura brought up on wednesday in our meeting that was really
00:11:28
um interesting to think about and it was things that were sort of like in the back of my mind when we think of like research at least like in our experiences with like rhetoric and composition it's more smaller like focused like case studies here's what i
00:11:41
did in my classroom but here we're working with this data of so many different sections and so many different students that it almost feels like overwhelming with how to even know like what exactly do we want to ask because
00:11:54
there's just so much of it yeah another con thing that we've been kind of thinking about and wanting to work with in terms of the data is also the student's annotations and how that might show up later in their writing so we're looking not only at their direct
00:12:12
annotations on the text and the ways that their as laura and sarah were saying speaking with each other and thinking about interacting in a more kind of immediate context with the text but also the ways that reflects later in their writings in
00:12:24
the essays that they're asked to write across the course of the semester so annotations they might make especially towards the beginning of the semester can then become relevant as they start bringing together all of those
00:12:36
analytical ideas and move towards later text so we're really grateful to have this access in the research project to not only the annotations themselves but also to student perceptions of their
00:12:48
annotations to instructor perceptions we've been working with another colleague who's not joining us in this session today but who's in the department of education he's been actually interacting with instructors directly to kind of study the ways that they've
00:13:01
been building their courses around this and the ways they've been asking students questions and then we'll also have access as part of the study to students written products at the end that's one thing that i'm particularly interested in is seeing the ways that their annotations and the
00:13:13
ways they've been able to interact with each other showing up in those final analytical essays so we'd be happy to talk about that as well yeah that's so i really appreciate that i i just also appreciate that you're sharing such
00:13:25
you know varied perspectives on this really broad project and i think it shows just all the different points of entry into the work which is which is really exciting um i'm looking at the chat a little bit
00:13:37
and and i'm also asked there is one question that's come through but i did want to kind of scroll back in the chat um and i see that we were joined by uh by candace candace welcome um in your introduction
00:13:48
you mentioned that you're redesigning capstone courses at your particular institution and i think that at the end of the day this project and again all the expertise that we have here
00:14:00
is ultimately about kind of course design very intentional course design thinking about supporting faculty how to do professional learning for instructors how to get you know new kinds of
00:14:14
activities into the curricula and how to support students and engage in that over time and i'm wondering if again any of our of our guests today maybe have some perspective maybe for someone like candace who is in
00:14:26
the process of redesigning courses and what what you've all gleaned about course design and then support for instructors when something like
00:14:39
social annotation and specifically hypothesis in this case is a really pretty central component to those redesign activities like what does that look like again what insights might you be able to share with
00:14:51
us um yeah so first i'd let's make sure we don't forget to get back to the question in the chat or the q a at some point um but i think you know for us it begins with you know what are our
00:15:05
what are our course design goals so as a long time you know online educator and course designer and whatnot it's always a matter of for me i focus on what kind of learning experience can we create for students i think typically speaking we design courses and
00:15:19
humanities especially in english we design courses around a set of readings and then figure out what we want them to do with them which is sort of backwards in terms of um you know actually how a lot of course design works which is that we want to focus on what are the learning
00:15:31
experiences that we need to or that we can use to accomplish our goals or our objectives and outcomes and then in my case how can we create perhaps better more engaging more dynamic learning experiences whether it's through active learning strategies whether it's through learning
00:15:44
technologies whether it's through you know specific kinds of assignments that make them i do a lot of digital literacy stuff so having a student make a video instead of writing a paper is a fun way or a introducing a different mode can
00:15:56
ask them to create and engage with the content differently so those all kind of are like floating when we begin to think about these kind of course courses a capstone course typically though is a like a seminal measure of some sort of achievement or some sort of
00:16:09
uh depth of engagement and from my experience as a teacher and what i've seen at least in some of the early conversations with our 131 online design is that the role of hypothesis in this
00:16:21
dynamic particular reading text reading critically has been um perhaps second to none like it's amazing how much of a difference the conversation uh is or how it operates simply by
00:16:34
having students have read and annotated before coming to class even if they didn't read the whole text if they've taken them or if they didn't annotate all they just read the colleague their peers annotations like the the conversation starts about three moves further along
00:16:46
and then it would before it took me a while to adjust to that um and i think you know going forward well we need to an rn we're gonna i think make some changes to our design such that our instructors are better prepared for what that looks like
00:16:58
in the day-to-day activity of a class not just the pre-class stuff but um so i think for me it begins with that like can i use this in this case the hypothesis specific conversation can i use social annotation to improve the
00:17:11
quality depth uh or overall framework of engagement that students are going to have in my course and is that going to give me a better learning experience like can i craft a better learning experience using this tool and then how can i build out of that
00:17:23
so it's not like a simple like well this is an easy fix it's more like this is a tool but it does enable a whole a new set of pedagogical considerations and strategies uh in the same way that if i have students work and make a podcast that also enables a
00:17:36
different kind of set of strategies so i think to me that's a tension that exists no matter how we approach the the course design conversation just but it's a great place to start because you're like what what if i had to pick five experiences for students what would they be
00:17:48
or three and build out of those and sort of work backwards from them um that's been successful for us with our online model anyways yeah sort of
00:17:59
extending that um i've been thinking about how an instructor's use of or implementation of something like hypothesis represents their pedagogical values so if you just
00:18:12
assign it if it's just that you have to do this you need to complete five annotations come to class be ready to talk about it that's a very different use from the an instructor who says that you need
00:18:25
to respond deliberately to three of your peers you need to ask a question of the text you need to um ask a question of a peers annotation and through
00:18:39
that sort of guidance it seems as if instructors have an opportunity to articulate what sort of values they have for their classroom
00:18:51
it's an opportunity to foster not only relationships but sort of build empathy between students consider different teaching teach them how to um engage with different perspectives and there is of course this
00:19:04
performativity to the meditation the social annotations right students know that it's not that they just have to um cross off the annotations from their to-do list it's that
00:19:17
they have an audience beyond um their instructor when they're asking questions of the text when they're responding to a specific quote for example
00:19:30
and so i think when it comes to course design it's not enough to simply include a tool we have to think about how we use the tool um and how we want our students
00:19:42
to use it so just sort of extending justin's point there yeah and that was one thing that really became evident for me moving from doing this curriculum and previous iterations of curriculum especially
00:19:57
in person we'd always required annotation like this that had always been a part of the class but i think one of the great boons of social annotation particularly as laura and justin were talking about is that it engages students with each other and shows them that they really
00:20:10
are a part of the conversation which in either a composition class or an entry-level literature class is something that's very difficult to get across to students i think in in those early things especially if they're not traditional
00:20:22
humanities majors or if this is not a kind of reading technique that they're familiar with they may feel very alienated from the text and not and especially if they have questions or they don't feel like they understand a particular portion of it social annotation is a really nice way
00:20:35
to give them that access and to provide them with kind of a low barrier to get across to start asking questions about the text to see that other students are also maybe engaging with the same portions of the text and as laura was
00:20:47
mentioning this also is such a great thing for the instructor because then if the students are able to do this maybe before they come into class to read that text the instructor has access to those annotations and is able to dive into that discussion
00:21:00
with such panache because they have been able to see where the students are confused or where the students are really excited and kind of start there or maybe see on the opposite end sections of this text that have been left completely
00:21:11
unannotated and maybe start there to say hmm i wonder why the students are leaving this alone was it because they were confused was it because this is a less interesting section of the text maybe it needs to be explained in class and so as opposed to replacing class
00:21:24
discussion i think one of the really important things about social nation is that it comes in as a compliment to it and that's something that we've seen across both in my own personal teaching as we moved into this new version of the curriculum and i think in some of the data that we
00:21:36
started to see from the product yeah actually mary helen you started to touch on one of the things that jeremy asked earlier i think which is some of the differences between like the writing studies
00:21:49
perspective of freshman composition which includes reading but a very specific targeted kind of reading towards a writing target outcome versus the literature courses which you know you spend more time in the l two are l two three four and fives which are introduction to
00:22:02
what poetry prose and fiction right those are the three we we work through yeah poetry drama and fiction yeah what's your dominant fiction sorry oh and then we also have a prose course but it's not in that yeah and non-non-fiction prose which which right now we are studying but
00:22:15
maybe next semester maybe next time yeah let's see but it does i think it raises questions about how this something like this as a decision operates um across what is a clearly related field but also
00:22:28
not a related field right in terms of some of the pedagogical things that we do yeah yeah jeremy for folks who are following along here jeremy had asked a question quite a few recently but one is in the q a section of our chat
00:22:40
about any differences perhaps between again the use of social annotation as a teaching tool and rhetoric again compared to english and you know and literature and i don't know if folks have other thoughts about that
00:22:53
that we might chime in on before we move to any of these other questions i mean it's it's hard for us because we don't i think mary helen's the only one who legitimately is occupied both positions teaching the leaders of course and teaching a writing
00:23:08
course um so you know mary helens probably has the most insight and yeah well i think i think the main difference there i mean they're both as i was saying i think it's a valuable tool for either one
00:23:19
as justin mentioned i think the pedagogical concerns are a bit different so in the composition classes one of the things that we're trying to work them through at the heart of the class is really having them understand
00:23:31
what good analysis looks like how to explain their ideas how to integrate sources and through the ways that um they can translate that into lots of different contexts so that's one of the
00:23:44
goals of the class is not just that they're like well i took an english class and now i know how to write for english but rather that the kinds of heuristics that we do in the composition course are translatable across all sorts of different disciplines because they're
00:23:56
using techniques like using a source as a lens like thinking about comparing contrasting different viewpoints and then being able to explain what's going on between them so the difference between that and kind
00:24:08
of the things that we're focusing on composition course and then those entry-level literature courses is that we're really thinking about the ways that each one of those genres functions and so in each literature course
00:24:20
we're really asking students to kind of build on they'll they will have hopefully at least if they're going through the university program as they're supposed to have already taken that first year composition course prior to taking the literature courses and so it builds on those skills
00:24:33
as a way to really get into close reading techniques in each one of those genres whether it's poetry drama or fiction and then to be able to talk about those as ways of seeing the world as ways of understanding what kind of what each literary
00:24:45
genre can do and the processes that we move through so things like um in the drama section that i taught about a year ago now time just has flown by in the past year i'm sure as it has for everyone
00:24:58
uh one of the things that we were able to do is to do a comparison between uh shakespeare's original text of a midsummer night's stream and then a kind of modern updated translation so i asked them to do a hypothesis assignment
00:25:10
on each one and to look at kind of similarities and differences across what's going on there to think about the way that an early modern tech could be translated into
00:25:22
something so much more modern but it retains some of those same themes and still really looking for kind of some of those higher order concerns because hopefully they already have that foundation of the excellent work that we try to ask them to do in that
00:25:34
composition class so i'm sure i'm kind of just talking around the question now so i welcome you guys to jump in and i know we have a bunch of other questions in the chat that's awesome no that's fantastic i'm sure that's really
00:25:45
awesome like really good thank you i'm learning a lot during this real quick uh so i'm going to touch on chris's question real quick uh um in the q a and then because i wouldn't mate yeah but no but i want to say i'm
00:25:59
trying to um uh have laura talk about the social annotation and empathy what she meant by that because i think that's a really point because i can answer chris's question relatively quickly um in that we haven't at this point
00:26:12
spent a lot of time thinking about the rhetorical canon as a framework for how we've been approaching uh the social annotation activity it's one of the the you know sort of tensions of being a writing studies framework of a program
00:26:23
that's connected to rhetoric that is informed by rhetoricians but our research is really more about because of the course design about analytical writing but i think it's fascinating to think about how some of those canon elements absolutely uh are operating and you're
00:26:37
in my first response gonna be like i think the primary mode of this tool would be invincio which you then later point chris i think in the chat um and because it is by design you know interpretation is also closely related to an invention you know
00:26:49
heuristics versus heuretics and how those terms operate in our canon so we haven't done a lot with it in terms of memory and the way you sort of operate about exporting comments and and making them available in that regard um but it is an area i think that not
00:27:00
like just we really haven't spent any time thinking about it yet so which is odd given that you know in theory i am a i am a rhetorician right so uh as a scholar in this field you think i would have been better at maybe paying attention to those things um but yeah so i think uh within the
00:27:13
canon though uh inventio is obviously connected to it um i think memory could be depending on how you want to approach it the text is revisible so there's a kind of tension there and then we have issues with like delivery and arrangement and style but
00:27:25
the style one is also fascinating for me like i'm intrigued by how many times students include links or images in their comments rather than just text and so that's not only it's a content choice but it's a stylistic choice i mean how many
00:27:37
i want more tick tocks in the comments is that is that so much to ask i don't think that's a problem right so so that's my own take on that i'm happy to have an extensive conversation about where and what that would look like but i i'm i too am fascinated by this idea that laura's brought up a couple of times
00:27:49
in our meetings and in here which is the relationship of the opportunity to be empathetic uh to one another and the relationship between social annotation and empathy and so lara i want to know more what do you mean and how does this work
00:28:01
what i think people should know is that justin is my dissertation chair um so the way he's asking this question is a form of pushing me
00:28:12
to define my terms and um what i will say is i don't know that i'm prepared to define my terms yet but i do think that um my interest in social annotation in
00:28:27
encouraging students to sit with a text and see it as a living breathing thing that they engage with along with other people is really important because there's this caricature of the academic
00:28:41
as an isolated sort of like loner and when we can perpetuate that image when we say that you sit with and you read alone and you write alone
00:28:53
um and we only read and write alone um we i think we do our students a disservice and so when i think about social annotation as a tool for developing empathy i think about it
00:29:07
um in that way because i think empathy happens through exposure and as you're reading through students other students your peers for commentary when you're reading their questions
00:29:20
you're being sort of forced to consider the fact that other people are engaging with the text differently just by nature of being different from you and um so
00:29:32
i don't know if i'm really answering the question i don't know that i i necessarily would know how we identify when a student is being empathetic in their comment this is something you
00:29:44
know raymond we chat a little bit about yesterday in our data meeting which is you know involving sentiment analysis at scale to not only maybe to determine what's going on at least with some sort of sentimental kind of tech language but specifically to maybe mark
00:29:57
a subset of of annotations that we would then do anymore so we use data science basically to identify a subset to do a qualitative look at or rhetorical analysis of to see what is going on in those exchanges
00:30:09
um but i i don't know well i'll just mention a few quick things and this is to curtis's question earlier critics welcome it's really good to have you here i think that we can we can put to some examples let's say of
00:30:23
research where qualitative to your to your observation careers qualitative analyses of smaller sets of annotations do show respondents whether those are students or those
00:30:36
are educators in some cases doing things like showing affinity kind of sharing personal experience being responsive to vulnerability sharing kind of critical introspective commentary
00:30:49
we might think of those as maybe aspects of empathy and looking at how those annotation exchanges provide some evidence of kind of more empathetic interaction as mediated by social
00:31:03
annotation and there's a lot of research i shouldn't say a lot but there are quite a few studies that also show how students perceive their learning with social annotations that's a step removed from the annotation content
00:31:14
but again how students perceive their use of this type of again social and collaborative experience and they do perceive it as strengthening their ability to connect to peers or appreciating diverse perspectives or
00:31:28
again learning as was mentioned you know here just from the fact that they can have access to their peers thinking so again there may be aspects of this that we might broadly construe as kind of empathetic reading or to jeremy's point
00:31:41
here and i was just trying to find some related commentary on twitter but jeremy mentioned this idea of annotation as a kind of listening and one of our k-12 colleagues who was presenting earlier this week courtney cleffman she and i had an
00:31:54
exchange on twitter a little while ago about annotation as very much related to listening so i'll try and pull that up and throw that into the mix here but anyways i appreciate this line of of thinking it really resonates
00:32:11
so the one of the chat responses is uh from jeremy uh that if they're being empathetic that definitely ought to be looked at for or for in the response of those to whom they're being empathetic towards the question um
00:32:25
yeah how would we then also track like audience reception of empathy in a flattened you know it's at that point it's text it's so it's it's an artifact that is strangely removed from the person even though it is never removed from the person so
00:32:38
like there's um one of the the things we had brought up a couple times in the practices and things we thought we might be seeing is if you know if i'm a we have students have to make so many responses and are there certain types of
00:32:51
annotations that become viral in terms of like they've meet a threshold of like engagement from other students and that seems like uh in theory it seems like a great idea except that if it's a group of students who only ever respond to each other's posts
00:33:03
and then you know it's a sub community and so then it changes the dynamic in the same way that empathy may or may not be impacted by those friendships those you know those kinds of things so there's so many layers to how we would even begin to track
00:33:16
or identify this even within one course right least of all what across 1300 participants i think that's what what our our final numbers are at this point um and so yeah i don't know this is these
00:33:30
are fascinating questions that's that's just what i'm saying like i don't know what to do with them okay we'll get there right raymie we'll get there you know i would use this because you mentioned sentiment there has not been at least in my reading of the literature
00:33:43
many if a few empirical studies that apply sentiment analysis to a large corpus of annotation data and i'm going to kind of leave the jargon there you know we did one study in the marginal syllabus project that actually showed that
00:33:58
the deeper a thread grows right so again exchanges among participants over time the more neutral the sentiment actually became which is hard to interpret against sentiment analysis depending on its specificity
00:34:11
um does and does not highlight certain you know notable patterns so i'll just leave that there but i think there are a lot of ways to get into this question of how you would begin to define that term back to your you know back to your point laura and
00:34:24
then look for you know evidence of it and then try and make claims now to justin's point about what that might mean as students are making meaning of of their readings i think something that i've been
00:34:40
struggling with on this team is figuring out um where i want to um invest my time
00:34:53
like do i want to look solely at social annotation of the text in the text do i want to look at how social annotation
00:35:04
um then affects subsequent assignments do i want to look at how teachers are involving themselves in the social annotation process and i
00:35:16
think we alluded to it earlier but raymie keeps insisting that this is a very unique and exceptional opportunity to work with an enormous data set um and so
00:35:31
trying to like stay open to the possibilities that the um breadth can allow for while also saying i think i'm gonna dive into this one
00:35:44
thing is really really challenging um and i don't know if sarah mary helena felt that way justin loves to sit in the vastness and just say who cares let's take it all on who needs sleep
00:35:56
um but for those of us who um don't want to age more rapidly we set boundaries right um so i guess i'm interested in how sarah and mary helen you're
00:36:09
negotiating rhetorical listening yes subscribe to chat go ahead hello i just felt like we couldn't talk about
00:36:20
um like reading with empathy without bringing in ratcliffe's ideas about rhetorical listening um i don't know when i i read her work somewhat recently as i'm preparing for like comprehensive exams this fall um
00:36:33
one thing that really stuck out to me about her work was how she talked about how and when engaging in like this type of rhetorical listening which is different from her for like rhetorical reading um but similar at the same time um
00:36:46
is that there's like a necessary two-part component or like two components here like one is like understanding the other and also like better understanding oneself and that's one thing that i think a lot about like how does writing help us better understand ourselves
00:36:58
um so i wonder if like we see that playing out in these annotations as well but this is something i'm like just thinking of now this is not something necessarily that we've talked about in our meetings or anytime bring a rough draft thinking sarah bring it forward this is what it's all about
00:37:11
right how else are we supposed to get there right that's the the question but it does it does raise um i think laura to your point right i think it may be helpful for folks to understand like what what we're dealing
00:37:25
with what we're looking at it would um first off every course has a set of texts they choose from so we have like a library a defined library if you will of course is there are texts that are included in their course
00:37:38
uh to have students read and annotate in this particular so we have a uh where we can look across let's say these five courses have all used this this text and so we can look at that as a framework we've also gathered uh
00:37:50
all the student at their own writing for the semester what they've produced which if you take 1300 students and you gather you know uh what is it three course assignments that are like what thousand word each or more so they're
00:38:03
long form essays right and we have the social annotation data we have instructor feedback data we have you know uh course section breakdowns um we have grade distributions both on assignments and
00:38:14
in the course uh that are available to us so the to large point it is it can be overwhelming to figure out as a researcher what we want to do but as a as a pedagogue as a teacher as a designer like
00:38:26
the even the the without knowing what the data suggests yet we've already learned so much in terms of you know sort of what we don't call them best practices anymore we call them vet moves right laura that's what they are um sort of the the moves that the more experienced teachers have been doing
00:38:39
with them uh how they're investing themselves in the tool or what their experience has been like um because they we we've used this tool in our classes for two semesters now we're going to use it again
00:38:50
at scale uh next fall and next spring as well but the first the first semester last fall we didn't collect any data on it because we were still getting things in place and getting all the parts and and trying to figure out how we're even going to gather all the stuff right
00:39:03
um and so what that did it allowed us to have a semester of like a dry run with instructors whatever there's our first exposure to hypothesis and we got a lot of feedback some good some negative mostly about course design and mostly you know about how the online experience was and
00:39:16
wasn't working for them um and so we made some changes but but ultimately i think that stuff is going to resonate no matter whether we get lost in the nuances of a single course analysis or like me you're like let's find let's
00:39:28
just look at the large scale data and see what the implications are about you know just basic descriptives like raymond which we've been talking about how many students are annotating so we you know we know for a fact that in our w-131 courses
00:39:39
students should complete 25 annotations for the year five annotations per uh course essay three additives which means you know original compositional kind of things and two responses there they have questions they have themes they can do
00:39:52
whatever they want the instructor can control that but each student should complete five per essay for a total of 25. and so already we can tell you that not every student hit the 25 annotation mark um but then we had some
00:40:04
students who have well over 25 annotations and so like those kinds of things are already kind of indicative of of differences between courses differences between reading and most likely as i think we're finding at least
00:40:16
in my initial assumption at this point is differences between instructors and i think you know instructor buy-in has been maybe one of the single most important components because i think we've been seeing higher numbers
00:40:27
i think from instructors who also participated in our our focus group for you know advanced engagement so there's a there's a kind of you know just basic correlations that are already occurring despite the fact that we have this
00:40:40
immense amount of data to try and figure out are there patterns are there things are there correlations are they statistically significant or not and how we can go about that to report this back so we can all make better decisions about how we employ these tools in our
00:40:52
in our classes so that's that's sort of the framework uh whether we get there i don't know but it kind of leads into the question i've had in my back pocket for quite a few colleagues um during this this whole just during
00:41:04
the diana take cameras i think i asked this during some of our previous educator office hours earlier in the week um and then some other sessions here and there i'd love to hear from all of you about this again the session attendance keeps
00:41:16
growing so maybe this will be useful for folks who are just just joining us even in the last few moments you know we've been involved in this project now again for at least a year some of us longer again depending on connections in history with justin your
00:41:29
whole team you've all been again wearing multiple hats and you may also have different kinds of teaching responsibilities coming into this next school year i'd just be really curious like
00:41:41
given what we know now or what we think we are learning now how do you see yourself employing social annotation either in classes that you're teaching or how do you see yourself supporting instructors
00:41:53
who might be using social annotation what are you saying to them what are you planning to do i just emailed justin about this because i'm teaching
00:42:08
a uh 300 level writing intensive course in the fall and i really would like to use hypothesis but um we don't unless we get like a a grant for this
00:42:21
research project we don't necessarily have the money for the department to pay for hypothesis to be used in my classroom and so i'm seeing that candice asks the question of what are you noticing about instructor buy and
00:42:33
what are the obstacles for instructor and asking my students to pay for hypothesis for me is really difficult i it when i don't have to have my students pay for texts um i don't when i don't have to
00:42:47
have them pay for the technologies i don't um and so i'm really trying to weigh the pros and cons of in having students invest in hypothesis for our time together
00:42:58
in the semester um because i have been thoroughly convinced of the benefits of its um it as a tool in the classroom um and so i am really hopeful that i'm
00:43:12
going to get to use it with my 300 level students and um if i when i do let's say let's manifest it when i do
00:43:22
um i i am really hoping to encourage at the additive comments that justin has referenced i'm really hoping that the students will ask
00:43:36
questions of the text and respond to one another's questions um before coming into class because it's those before conversations it's being able to reference and point to something that mary helen said in the
00:43:49
course text two weeks ago that gets students excited that they were heard in some capacity before they even came and stepped foot into the classroom um so that's my plan i don't know
00:44:02
if well well let me add real quick because i i want to hear what sarah and marielle have to say but to lars to your point though one of the the purposes of of this kind of research is to help us make the case who the department and the
00:44:14
college and the university like this is not just another you know it's a tool we have a lot of tools we have a lot of enterprise level agreements that we pay for and students pay for things all the time so there is that's you know this it's no more expensive than another textbook if you think about in those
00:44:26
regards per students but our part of my goal is to demonstrate why this tool in fact is does have measurable value and measurable impact and why every college should invest in it right for every course who wants to take advantage of it um
00:44:39
in the same way that we you know we pay for things like the adobe licenses the microsoft licenses that are tools that they just think are commonly acceptable kind of things right so like that's that's an ongoing purpose behind what we're doing not just because
00:44:51
i think it's a great learning tool but because i think it's a learning tool that needs to be invested in at the university level such that it has a i think its impact could be more measured by the way but so and then we can work out the other stuff behind the scenes right but sarah and
00:45:03
mary mary what have you been telling people yeah i think one of the things to kind of address candace's question is one thing that we've been seeing is especially newer instructors have been a little like it on one side of the spectrum
00:45:16
together either really engaged with this tool and think that it's really fantastic or very wary of it and don't really know how to use it and just think that it's another i think one of the challenges that we've seen instructors come against is that they think
00:45:27
falsely we think that it's just another thing to grade like it's just another kind of evaluative tool to use and so one of the things that we're hoping the study will be able to show them is both through getting to use it in practice and also through
00:45:40
seeing what we find in the study that having this extra access into the text this additional access not and again i think that's one of the things we're trying to show them is it's not a replacement for students doing the reading on their own for students taking notes
00:45:53
for having class discussion like it's not meant to supersede any of those but rather to really be a part of the entire curriculum wherein the social annotation brings another level of engagement with the text
00:46:05
and so i think one of the challenges that instructors may have faced in these kind of early stages especially if they haven't used it before is that it felt like just another requirement hurdle for them to do like i like justin was saying each student is
00:46:19
asked to make five annotations on each of the five texts throughout the semester and so then instructor's like well now do i have to go in and you know count and see like ah yes bob made five annotations bob gets five points and
00:46:32
that's really not the not that we shouldn't you know also give them credit for doing work but that's not the main impetus behind using the tool the main impetus is to really get them to be able to interact with each other interact with the text
00:46:43
and so exactly as jeremy just mentioned the chat one of the things that we're hoping to show them is that this deeper engagement with the text and these multiple mobile conversations they're able to have in addition to class discussion with social annotation
00:46:55
make them kind of more careful readers and writers which will result in we hope and this is another thing that we're at the beginning stages of the study so one of the hypotheses ha that we are hoping to find is that
00:47:07
this really does improve kind of the quality of their writing because they're spending that time engaging with it yeah and i'm thinking about hypothesis much more from like a student
00:47:21
perspective than a teacher perspective um because i've never taught with hypothesis before um because i was teaching public speaking last year um and that doesn't really have text that you read um but thinking of
00:47:35
this from like more like if i already used hypothesis i was like a student um like even in a grad class or something it's much more inviting to respond to people's comments than it is on like a canvas discussion board or
00:47:48
like a blackboard discussion post where sometimes teachers will say our instructors will say okay post something and then respond to two other peers and then there's like jokes about how your posts are like great job this was so interesting and they put
00:48:01
like a word count on how many words you need to say but something about like annotations on hypothesis is much more specific and you can dive into specific parts of the text and it's it's less like you're sort of like floating and you're like i'm going to
00:48:13
talk about whatever i want it's pinned down to something much more specific so i think it's more inviting in that way sarah that really echoes a comment that was made by one of our k-12 colleagues in a panel earlier this week
00:48:27
morgan jackson she emphasized just time and again this kind of proximal relationship to the text and how we're just continually kind of inviting students to return to that close reading and then that kind of
00:48:40
close and things can then build out from there but to your points here they don't kind of start in the kind of stratosphere right they're like right there in that very proximal relationship to their reading
00:48:53
um anyways that's that's just very much on my mind and really echoes what what you're saying um yeah yeah no well you know we there's the two couple things that have come up for me that i i'm i'm i continue to
00:49:05
struggle with it right and one of which is that as we have currently designed it in the online w-131 course um the annotations themselves aren't meant to be graded in terms of quality they're just meant to be like
00:49:17
you've completed the activity and it's meant to be a quick rate there's been some some confusion on on that which is probably part of the attention there but it's been like you know you do all five and it's because it's through our canvas lms the speedgrader pulls it out very
00:49:29
quickly you can identify how many things so if it's accounting system it's pretty simple but it's not make a proxy for participation justin yes it really counts as part of our participation right so um and so there so there's that uh and
00:49:42
then we have this ongoing question about how involved should an instructor be in in the annotations right so like um we have a number of folks who think that that's a student's space and there's a really there's something something really nice
00:49:54
about staying out of it letting the students annotate and stay out of the conversation others like to sign post things uh and one of the things i'm and i've been using it in my spread of rhetoric and sports class um and i stayed out of it the first time
00:50:07
though the class like i didn't annotate anything i just gave them the guides and but i feel like one of the advantages and that's something i'm going to do in the future is to actively participate in the annotations after they've started
00:50:19
annotating right not before i don't want to i want to tell them where to read but to engage in conversation with various students in the annotations and actually slow down the process because what happens to me all the time and it's because again i
00:50:32
mean i spent my whole life reading and trained to read and know how to read and do it well i moved through text really quickly right and i moved to discussions really quickly and this is something i do as a teacher and some students get it some students don't we just move on and that's sort of like i've always been kind of willing to
00:50:44
accept that but something i've been doing the last couple years is really trying to do less do more with less and to slow things down and i think that there's a an opportunity with uh something like a social annotation
00:50:56
to more slowly and more thorough thoroughly engage a reading to position key concepts from the course position key frames or perspectives uh and then if you engage with students through that you not only create better feedback loops between you and them
00:51:09
but then it moves it very quickly from a completion activity or another task list inside of your lms to um well it's an experience at that point i still think it's already a good experience for the students in the way that we have
00:51:22
it set up at least as ensuring accountability getting to read through the text getting to think about the text i don't think it's problematic that way i just think that depending on how much affordance you have as an instructor or how much agency you might have over your course right there's those kind of
00:51:35
considerations i am tenured faculty so i occupy a little bit of a different perspective than a lot of folks but um but i think there's something to this and i don't know if it's better to annotate with them or not i don't know if it's better to turn into a series of
00:51:48
things they complete or not and i just don't know but i i now i want to flip and try the other side because i did this version and i'm like okay i like that but what if i spent time arguing with them in the in the text right or pointing out things like
00:52:00
you know like we read something like casey lemon and layman's piece about being in oxford mississippi and culture and race and football and and you know they they miss some of the the significant elements of his story as being an african-american male
00:52:13
uh in oxford mississippi in a football culture he came from he's a iu graduate um and so there's you know like and so i in class we talk about it but it seemed like there's an opportunity to extend that more meaningfully in the text
00:52:25
to pin some things down before that happens but i don't i don't know how it fits in my class like this is this is my challenge mike i don't know what to do with it yet so hey this is great i know justin in terms of you know you bringing to the four
00:52:38
aspects of participation you mentioned accountability we've touched resign assessment nate can we bring criticism question up uh from the q a and and maybe have folks just kind of riff on that a little bit if that's
00:52:50
relevant maybe is one of our last questions or getting towards the end of our arrow here so folks can can see this i'm curious if if our panel has thoughts on on a
00:53:02
question about combining social annotation with i guess labor-based grading if folks have a sense of what that means or how that is being approached or maybe other forms of assessment as well
00:53:13
real quick what do we specifically refer to in labor-based grading is that contract based grading is that what we're assuming we can bring them up on stage if you want yeah i mean i want to make sure i
00:53:28
understand yeah unless you want to raise your hand and i'll bring you up thanks for hanging hanging in there with us folks as we start juggling some tech here but this is this is kind of what it's all about curtis welcome
00:53:44
hello can you hear me yep yes yes i'll even now you can see me how about that i've loved this session so much i'm also an english faculty and so the the conversation and i am also as
00:53:59
i've said in chat engaged in a research project that i'm currently designing for the for next a year but um i've been really interested in uh the work assau inui has been doing in
00:54:11
labor-based contract particularly to kind of really address issues of equity and and and relationship to white language supremacy and so uh so it's a form of contract grading that
00:54:23
as we've been talking about social annotation is sort of evidence of participation um one thing asau does his he really has people keep a labor log so it's not about how many
00:54:36
posts in the annotation but it's how much time are you spending reading how much time are you spending responding how much time are you spending annotating or responding to annotations and documenting it and learning
00:54:48
the kind of thinker and reader and responder you are and documenting sort of in a metacognitive way how much time you're committing to the task and and how and and and what it's doing for your learning
00:55:00
and then combining that labor grace with also writing of essays so that we just extract grades from the entire operation yeah so that's why i thought you were going but i just wanted
00:55:12
to make sure this is a diverse audience uh so asal's work is a wonderful and in a lot of conversations and i do something similar and and it comes sort of but before his work came out which is this semester actually in my sports class i used uh
00:55:25
the learning record method which is pig cyruson and mary barr's work which is very similar to the labor-based uh approach that style is using which is students they they document their progress to the course they have regular reflections rather engagements
00:55:38
focusing on what they believe they're learning and they use those evidential things and the things they create to to make the case at the end of the semester like here's what i've learned and here's the great i think i deserve based on what i've learned not based on
00:55:51
a performance or based on a particular outcome so it's a little more like they have to argue for the grade rather than just keep track but but it's the same idea and that's exactly what happened in my sports class this semester was they were and and several of them pointed out like you
00:56:02
know on our annotation assignments you know here's the kind of work that i did in those annotation assignments or they were taking like one student said i only did two per post but they were taking me so long because i was really trying to think through my response which even if it's a lie i consider it i
00:56:16
bought it i'm like i'm in okay you're gonna you're gonna tell me that i'm in but uh so i think there's a lot of room for that um specifically you know if you get into practice and students get comfortable with it but my problem is it's the first time students have ever
00:56:27
experienced anything like this um and the first time i'd actually taught this version here at iu i did a version when i was at ut austin where jeremy dean was a grad student at the time and we both sort of you know worked with pig in a number of ways but
00:56:39
uh it is a i i found it to be a very useful vehicle to avoid this like i'm just doing annotations because it counts for a grade and it became a another piece of evidence in the demonstration they can make for
00:56:52
their engagement in my course and once they once we got over that hump after about week four or five when they started to realize what i was doing i wasn't grading anything i was just responding to their work right uh then i i think we got a lot of
00:57:04
buy-ins a couple folks dropped out but more folks really really bought into the process i think it's i think it's a rich opportunity for any of that framework um and but i'll let you know i'm gonna do it again next spring when that exact
00:57:16
class comes around and i'm gonna do a little more personal participation in there but uh i think the framework is really well suited for social annotation as a learning activity well especially if you're going to have
00:57:27
them not talk about labor practice but then like their identity in relation to labor their positionality in relation to what they're doing or what they're being asked to do how the language is imposing on them or not you know those wonderful sort of
00:57:39
distinctions that oftentimes gets swept under the rug in higher education despite our commitment to diversity uh and and things like that and underrepresented populations we still have structures that don't always make those things as visible and i think
00:57:52
having students do that which is you know really core to what asal's work is doing and and helping us think about so yeah i think it's great i think it's perfect for that yeah this is great yeah chris thanks so much for jumping in here a little bit of improv
00:58:07
towards the end of our hour but i really love it um we are coming up right on the hour um you know i'm always game to kind of hang out and chat a bit although i also want to be respectful of everybody's time and
00:58:19
i also just want to formally thank this entire team from iu for joining us today so although some folks may need to run and maybe some folks can also stay for a few more minutes um i just want to thank on behalf of
00:58:33
the conference organizers and hypothesis and the broader ianate community you know everyone for joining us today mary helen and sarah and laura and of course justin uh really it's been a pleasure personally to learn alongside you for the last
00:58:45
year i look forward of course to our future work together but thank you for sharing again your wisdom with with everybody who's able to join us today this session has been recorded um we'll be able to you know reference it
00:58:57
you know down the road and again folks are welcome to hang out for a few more minutes if they're available but again you know we're at the hour and so this is you know the end of uh actually the educator office hour is part of iit but
00:59:10
it's also really kind of the end of my annotate so i guess we've we've made it all the way right now you can put us together now there actually is there is one more session if you still have appetite for more
00:59:22
immediately following this where we'll be um reflecting on the week as a whole i know people are probably too tired to do that but unfortunately i have to go be the uh taxi for my kids okay uber dad but not in the way that
00:59:34
that sounds good right um but i really appreciate you all having us on and letting us share our story and talk with folks and um if people have questions about what we're doing or want to know what we've done and i mean the data will eventually be
00:59:46
parsed our research will publish but if you just have questions about programmatic design engagement of the way we teach with things i'm sure that i speak for all of us that feel free to reach out uh email me and or email any of us or i can connect you and we can have continue our
00:59:59
conversations after this
End of transcript