Things are moving fast. Too fast? I do like it though. Fast.
Recap
In the previous quarterly update I imagined I would start writing up the Q2 Skrutable meter ID work and, relatedly, continue the Mahāsubhāṣitasaṃgraha digitization and benchmarking. I also hoped to give SETI some love. But other things were more pressing.
For one, I helped Muktabodha go live with its new digital library (see the announcement on Indology), and thanks to a generous outside collaboration, a few dozen additional items are on the way. HANSEL also crawled along, though the slow pace belies some underlying advances. At the intersection of these two corpus-building efforts, I’ve been building some experimental tools for digitization and proofreading, and so far they’re working great. I think they’ll generalize fairly well to other projects, which is what this curious toolkit seems to be aiming for too. More on these tools when they mature a bit.
The main thing I spent time on, though, was the hoped-for evolution of my Sanskrit Wikisource “Atlas” into the larger Sāgarasaṅgama project, now released (see the blog post and Indology announcement). To reiterate the idea: There are several super-collections of online Sanskrit e-texts that most of us have bumped into at one time or another, especially Sanskrit Documents, Sanskrit Wikisource, Jain Quantum (a particular snapshot of Jain eLibrary), and E-bharatisampat. But in my experience, students and scholars don’t engage with these collections in a sustained way because of visibility and access issues. Either they don’t know about a collection, or they know about it but don’t believe it contains much, or they’re aware of its size but still have trouble getting at or using the actual material. So I’ve been making bespoke “Atlas” interfaces to facilitate access to individual collections, plus a “Sāgarasaṅgama” (“meeting of the oceans”) meta-interface to compare across collections and search their metadata in one go. In other words: finding aids and a meta-finding aid.
This gives us, for the first time, a striking historical picture of how much searchable Sanskrit text is online, now with GRETIL for context.
I hope these new interfaces can contribute to an ongoing conversation about where we should be looking for the large-scale digital material we need to understand Sanskrit language and literature. That conversation should of course include detailed consideration of e-text quality.
Re-centering the Vision on Visibility
ISCLS 9 is coming up in January at Deccan College in Pune, and I plan to go and present a paper. In it, I’ll share my thoughts on 1) how we’ve previously understood the question of how much e-text data is online, 2) how the visibility problem has impacted us and how I’m approaching it, and 3) what tools I’m building and using to make further additions to this vast and growing online library.
I’m more and more convinced that visibility is a major and underestimated problem; the yawning gap between GRETIL and the larger collections, now plain to see in the above graph, only strengthens the point. Why spend time hunting for, or even recreating, material that is already online in a usable form? And if it already exists, why shouldn’t even beginning students have instant access to the full range of literature in a form they can easily page through or search? As it stands, there’s so much unnecessary struggle and wasted effort. SETI, Sāgarasaṅgama, and Pāṇḍitya are all coming at this from slightly different angles. Before long, we’ll have comprehensive, disambiguated insight into the thousands of top-tier Sanskrit works that history has deemed worthy of printing as editions and then digitizing. This curated bibliographic metadata will be open to all, and there will be multiple ways to interact with it — traditional search and browsing interfaces, visualizations, and the raw tabular data — depending on one’s needs and preferences.
Once I’ve solved this problem of metadata visibility, I’m excited to move on to creating textual data of other sorts. New e-texts are one piece, and for that I’m using HANSEL (and Muktabodha too, although I’m not the primary curator there). Creating annotation data, like what Oliver painstakingly did with DCS over decades, would also be an enjoyable way to sit down and read through lots of text at length. And then there is training of new models, like what Sebastian and co. have produced on the basis of the DCS data. But for now, metadata is the focus.
For all of these tasks, I’m now fairly sure that AI is indispensable for getting us there in good time. And not just a script here or there. With coordinated swarms, even large data tasks requiring language expertise at multiple levels can be broken down and handled in parallel at astonishing speed, with accuracy that rivals that of any human team. Of course there are limits; I won’t be trusting AIs to read handwritten manuscripts anytime soon. But for the kind of work I’m most interested in, I do believe 90–95% can be automated. The trick is learning to unleash this kind of power responsibly, with each step appropriately audited before the next begins. No one likes a “slop cannon.”
Technology is accelerating, and even that acceleration is increasing. (Feeling whiplash? In physics, a positive third derivative like this is literally known as “jerk” for a reason.) It’s certainly challenging to keep up, but I have no doubt that AI, applied with good judgment to the bibliographic and other data problems of our field, can greatly support and advance traditional humanities understanding. That’s what I’m after.
Cool-down and Consolidation
In Q4, I’m resisting the urge to race ahead toward more digitization and other shiny new objects. Instead, I’ll be focusing on upgrading the existing projects that are key to solving the visibility problem, especially SETI. In the coming weeks, look for a significant SETI upgrade as it and Sāgarasaṅgama begin to converge on the same data. Pāṇḍitya will also be getting some attention, so if you have ideas for improving that interface, now would be a good time to share them. I’ll also try to clear my Skrutable and HANSEL backlogs.
The agent swarms are standing by, infinitely ready for delegated duties. Breathe in, breathe out. Combobulate.
Then by the middle of next year, once I’ve gathered my energies, I think it will be time for a big, fun digitization push: both cleaning up and upgrading old material and putting entirely new stuff online. That will also be a good time to explore tech-supported analytic reading as a way to build out more annotation data. I’m really looking forward to it — especially the actual reading part!