Kathabhidhana

by
  • Subhashish Panigrahi
Kathabhidhana is an open toolkit to record a large number of words. It consists of a few free/libre and open source software, open datasets, methodologies and documentations. It can be used to record pronunciations of words to create dictionary along with pronunciations, and also phonemes to create a text-to-speech software.
language open source initiative

In an age where technology is becoming so flexible that we can make sense out of languages on a mathematical level and use the structure to create meaningful applications, keeping languages agile and technology-compatible is a necessity.

What a shame that rich languages like Sanskrit have no smart way of learning like Duolingo or Memrise. It isn’t just language learning which is hampered because of lack of digitization but also a large section of our society cannot access to the wealth of information internet has to offer.

To quote Subhashish from his article:

There is something that every single individual, who speaks a less-spoken language, or is in contact with a native speaker of an endangered/indigenous language, can do. Languages that are dying need digital activism to grow educational and accessibility tools. That can happen when more public and open repositories like dictionaries, pronunciation libraries, and audio-visual content is created.

Subhashish’s work is crucial because he is doing a monumental work by understanding what are the exact bottlenecks of accomplishing a language documentation project. Kathabhidhana (literally means a dictionary of spoken words) is an open source toolkit to record large number of pronunciations in any language. Such a dataset will allow the open source community to build text-to-speech applications, build machine learning and Natural language processing based tools, but most importantly personal assistants like Cortana or Siri.

Interview

Can you tell us the story behind your project? How did you decide that you want to combine linguistics and technology? Is there any key moment in your life that inspired this decision or were you naturally drawn towards this project through your work?

Kathabhidhana originated in the beginning of last year when I was looking for options to batch-record audio and create audio entries for the Odia Wiktionary, an online dictionary and sister project of Wikipedia where one can find meanings of Odia-language and other language words. I stumbled upon an Open Source project created by T.Shrinivasan for the Tamil-language Wiktionary. After using it a couple of times, I forked it, and made some small changes to the code to fit into my workflow. He was also quite generous to add some features to the tool. That’s how Kathabhidhana was born. Over time, the project grew bigger, and I made it part of the OpenSpeaks project that I founded, where the latter helped create tutorials and documentations to use it. Later, Prateek Pattanaik, a fellow Wikimedian created a workflow for iOS devices. Kathabhidhana is now managed at the O Foundation (OFDN), a nonprofit that works in the intersection of language, culture, and technology with Openness in its core.

As I shared earlier, this project was started because of several trial and errors for a technical need. But in general, I am still walking on the sands and there is an ocean of knowledge when it comes to linguistics, and I am not even a full-fledged geek to enable technology. However, I have worked for almost my entire career for the user and contributor communities of various Open Source and open knowledge communities like Wikimedia and Mozilla, particularly the South Asian communities. When the expanse of the pluralism of languages and cultures amaze me, it also saddens me to see a vast majority of the languages do not have enough tools, and there is not enough content for people with special needs. I believe, most other projects that I am part of, and Kathabhidhana are an outcome of all those needs that I have been seeing.

How is the process of doing a project like this generally like? How do you keep yourself going? Do you like keeping your project open ended or do you like to have a fixed idea of what you want, in what time frame and such?

The overall workflow is quite agile. I started with a simple “what”—there were over a 100,000 words in the Odia Wiktionary and only a few hundred of them had audio recording of pronunciations. With not even a single good quality text-to-engine available out there, such a great project is of no use to the community that needs digital accessibility. So I started testing out available tools to batch-record audio. I kept track of time taken for all those pilots. For my own selfish reasons, I also documented some of the key findings. When I had a working project, I started making all of those findings public.

I certainly had deadlines and a timeframe when I started the project. Curiosity to see what is ahead after finishing one task, struggling with many of the failed attempts, and the excitement of some of the working prototypes kept me going. Once the project was stable, I made it an open ended one as someone else might find it useful, use for their own work, and even better it.

What would your advice to young learners be who want to do projects like you? How does one get started?

Not really advice, but three key things to consider — have a clear purpose, collaborate with others to build, and release your work under open standards. Have clarity on why you need to start a project and what you need. Identify both like-minded people and people who can give you critical inputs, and collaborate with them to build prototypes. Test it across different environments and better it gradually, and test it with real users as well. Once you have a decent satisfactory version ready, release under open standards. The last bit is very crucial as it allows others to not just use your work, but also contribute back.

What are the projects you would want to see happen in India? What is your vision about the country - how do we use our creativity efficiently?

I personally would like to see the issues with diversity being addressed more and more, and the diversity in many things being celebrated widely. This nation was built on diversity in its core. With almost 750 languages spoken across the country, only 22 have been part of the 8th Schedule of the Constitution. Many of those ones, and the ones that are not part of the 8th Schedule are dying out slowly. They will die in the coming future unless there are not enough tools to preserve, and use them widely. I dream of an India where people that speak various different languages, and people that need digital accessibility access knowledge in their own languages, and have robust technology to help them contribute back to the knowledge commons. There is a great scope for building sustainable projects and innovate for the people of this nation by using technology as a tool, and using local resources as an asset.

You can get in touch with Subhashish on Twitter at @subhapa to discuss collaborations and know more about his initiatives in the future.