<div dir="ltr">Hello all,<div> I'd like to introduce a new method of searching the TRC hearing testimonies that the Bitter Aloe team created recently. <a href="https://streamlit.as.uky.edu/bap_sent_embedding/Testimony_Search_Engine_(ML)">The Testimony Search Engine (ML)</a> allows users to search the testimonies semantically rather than relying on keywords alone. It is now possible to take a statement given by an individual and then look for statements that express a similar experience/sentiment/idea across the entire corpus of testimonies. This allows for new forms of ingress into this corpus and allows users to make connections between approximately 2400 testimonies. Finding multiple new entry points enables a new kind of reading that allows users to see experience across multiple testimonies rather than reading testimonies linearly one by one.<br></div><div><br>Users can identify a statement they would like to use as a search query in TSE (ML), by running a conventional keyword search for a topic of interest using our companion search engine <a href="https://streamlit.as.uky.edu/bap_sent_embedding/Testimony_Search_Engine_(BM25)">TSE (BM25)</a> named after the BM25 ranking algorithm. Once a statement of interest is identified users can then copy the text of the statement and input it as a query in TSE (ML) to search all testimonies for semantically similar statements. Although results vary depending on various factors (length of statement, degree of uniqueness of expressions within the statement, etc.) generally speaking it will return results that are semantically similar even if few if any keywords are shared across those results. The following verbatim quote is taken from testimony given by Gideon Nieuwoudt during an amnesty hearing held on 25 September 1997 in Port Elizabeth.<br><br><i>"What did these tablets taste of? Or did you merely swallow them?"</i><br><div><i><br></i></div><div>The results returned map out several overlapping semantic fields addressed in the above statement. These include the ingestion of tablets and other substances, the actions of the mouth, and seeking treatment from medical professionals.<br><br>One feature we did not anticipate but users may find useful is a novel search method we call 'query poetics'. During testing we created simulated statements of our own authorship and used those for queries instead of verbatim statements given by an actual witness. The term poetics refers to its secondary meaning as that which is creative and productive, rather than a query that has some sort of literary quality. What this allows for is a more freeform navigation of the testimonies that is solely dependent on an experience/sentiment/idea that a user is hoping to find represented somewhere in the testimonies. The following simulated statement generated by a user and not an actual witness is a good example of the proper length and level of detail that will generally return useful results.<br><br><i>"I heard the dogs barking and I knew the police were there. They were chasing us and we hid between houses. We could then see the vehicles going by. I have never been so scared in my life. I thought for sure we would be caught."</i></div><div><i><br></i></div><div>Here too you can see multiple semantic fields represented in the results, with notions of movement, hiding in fear, police vehicles and complex urban geography represented most prominently, regardless of the presence or absence of particular keywords.<br> <br>A word about methods; the Bitter Aloe group took all available hearing testimonies and converted them from HTML into structured data (.json). Testimonies were parsed by individual statements made by each speaker. Statements were then represented as vectors and embedded in high-dimensional space. <br><br>We'd be really grateful if anyone who has a chance to try this out could provide us with feedback. It's still in development but we're slowly rolling it out to see what changes need to be made before we present it to a wider audience. Regards</div></div><div><br></div><div>-Steve Davis</div></div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 8, 2023 at 9:20 AM Keith Breckenridge via Afridig <<a href="mailto:afridig@lists.wiser.org.za">afridig@lists.wiser.org.za</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Stephen Davis is the Principal Investigator of the <a href="http://www.bitteraloeproject.com" target="_blank">Bitter Aloe Project</a>,
a digital humanities research group that applies advanced machine
learning models to materials collected by the Truth and Reconciliation
Commission. He is also author of <i><a href="https://iupress.org/9780253032294/the-ancs-war-against-apartheid/" target="_blank">The ANC's War Against Apartheid: Umkhonto we Sizwe and the Liberation of South Africa</a> </i>(Indiana
University Press, 2018). His research interests include machine
learning, human rights, the anti-apartheid struggle, and autobiography.
He is presently an Associate Professor of History at the University of
Kentucky. <br></div><div><br></div><div> Stephen Davis <<a href="mailto:srda227@g.uky.edu" target="_blank">srda227@g.uky.edu</a>> </div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="text-align:left">-------<br></div><div style="text-align:left">Keith Breckenridge <b><span style="color:rgb(153,0,0)">W I S E R</span></b><span style="color:rgb(102,102,102)"> - The Wits Institute for Social and Economic Research, University of the Witwatersrand | Pbag 3, PO Wits, Johannesburg, South Africa, 2050 |<font size="2"> Phone +27(0)11-7174272 | </font></span><span style="color:rgb(102,102,102)">Web</span>: <font size="2"><a href="http://wiser.wits.ac.za" target="_blank">wiser.wits.ac.za </a><br></font></div></div></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br>
Afridig mailing list<br>
<a href="mailto:Afridig@lists.wiser.org.za" target="_blank">Afridig@lists.wiser.org.za</a><br>
<a href="http://lists.wiser.org.za/listinfo/afridig" rel="noreferrer" target="_blank">http://lists.wiser.org.za/listinfo/afridig</a><br>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><br><br>Stephen R Davis<br>Associate Professor<br>Department of History<br>University of Kentucky</div></div></div>