Automating Student Assessment is like Magic.
The power of artificial intelligence is taking us to new frontiers.
Three years ago, when our quest to harness language processing and artificial intelligence for document review began, the best pre-trained language models were just taking their first steps. Today, smaller open-sourced models perform remarkably better than those built on specialised data just a year or two ago. The “transformative power” of transformers has indeed revolutionised the quality of base models, delivering on the promise they held years ago. Today is for us a momentous time in so many ways. There are mainstream use cases that an excited entrepreneurial ecosystem is embracing and infinite “edge use cases” that are waiting to be leveraged in markets, old and new.
Deepsy: Tackling AI/NLP Edge Use Cases.
As older readers might be aware, we are passionate about solving complex document review problems, specifically in the context of analysing documents at scale. This requires a combination of strong logic (traditional algorithms, trained ML models) and powerful language models.
Despite the phenomenal innovations in advanced large language models, edge use cases that demand precision of logic and strong language modelling will require specially trained models and algorithmic rules to work well—at least for the near future(!) However, as I write this, things are changing fast and this statement may not hold true for long.
Revolutionising Evaluation and Assessment in Education.
We are discovering value for our review automation tools in the evaluation and assessment of students. Our initial efforts focussed on ranking of documents and scoring, and over time - we have leveraged rules based algorithms and language models to add colour to assessment through meaningful feedback on performance.
Today, let’s say a student submission looks like this:
Deepsy would deliver an assessment like this:
What we are particularly excited about are the possibilities that Deepsy opens up for the education system, with:
Volume-independent evaluation time: Even with human supervision, Deepsy allows for efficient evaluation, regardless of the number of submissions.
High-accuracy, automated document review at scale: By relying on answer keys and evaluation benchmarks, Deepsy ensures higher quality and consistency in assessments.
Fairer and more objective scoring: Deepsy's AI-enabled system provides detailed, meaningful feedback for students, at a level of granularity that tired and time constrained evaluators would prefer to review than generate afresh.
Other things about Automated Evaluations.
Apart from the customary upgrades that we see to the evaluation system, there are softer issues that an automated system is by its nature suitable to correct. For example: One interesting finding is that well-formatted documents can be less scrutinised, even if they lack substance, while human evaluation may overlook valuable content in less visually appealing submissions. This is natural, and in my lfie as a lawyer - like others, I have been guilty as charged. Deepsy's AI-driven approach is more likely to apply only standards dictated as relevant for evaluation. Our ability to judge is a great power, but perhaps holds us back in subconscious ways even where it is not our intention to err.
If you are interested.
We invite educational institutions and edtech companies who are interested in being early adopters, to join us in letting their students experience the power of Deepsy for themselves.
As I look forward to sharing more about Deepsy's journey and other exciting ideas at the intersection of artificial education, language processing, law and education, I must thank old readers for bearing with my long hiatus and sporadic writing.
As always, I would love to hear from you. Much more is afoot. I can’t wait to write again.