For this dictionary results will have to be separated :

• ExactMatches
• PatternDictionaryMatches
• DictionaryMatches
• ...

This order should be :

• ExactMatches
• PatternDictionaryMatches show first result of PatternDictionaryMatches if ExactMatches is NULL
• DictionaryMatches show first result of DictionaryMatches if ExactMatches is NULL
• GreedyTokenized if ExactMatches is NULL
• PatternDictionaryMatches show rest of PatternDictionaryMatches if ExactMatches is NULL
• DictionaryMatches show rest of DictionaryMatches if ExactMatches is NULL
• PatternDictionaryMoreMatches
• DictionaryMoreMatches
• ...

It's also good to separate DictionaryMatches into DictionaryMatches & DictionaryMoreMatches. Similarly separate PatternDictionaryMatches.

Usecase: Dictionary will have the word പാവയ്ക്ക. If I type "pavanaayi", the first suggestion will be "പാവനായി" which is wrong. The importance should be to the scheme pattern eh ? or at least show them at the beginning itself and not wayyy down.

Error:

2022/08/19 06:08:10 fts5: syntax error near "*"
2022/08/19 06:08:10 fts5: syntax error near "*"
2022/08/19 06:08:11 fts5: syntax error near "*"

From varnamd logs :

[2022-08-19T06:08:10.771881527Z] status: 200, latency_human: 2.561737ms, error: <nil>, user_agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36, uri: /tl/ml/-4i
[2022-08-19T06:08:10.828157178Z] status: 200, latency_human: 1.249368msm error: <nil>, user_agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36, uri: /tl/ml/-4it

• Adds varnam_new_search_symbol :

Symbol table search had a fatal flaw :

var searchCriteria Symbol
searchCriteria.Pattern = "a"
searchCriteria.AcceptCondition = 0

Varnam omitted a search criteria by seeing if the value is a default struct value of Go. Go's default struct value is 0 which means one can't apply a search criteria of value 0. The solution applied in this PR to this is to set default value -1 by calling func NewSymbol(): Symbol. So now:

searchCriteria := NewSymbol()
searchCriteria.Pattern = "a"
searchCriteria.AcceptCondition = 0

• Added all varnam configuration vars to varnam_config(). Previously used functions like varnam_set_dictionary_suggestions_limit has been DEPRECATED.

There's a bug in learning from frequency report file. If the word place in the file has hidden characters like <0xa0> then GoVarnam mistakenly takes the next number to it as a word. Because of this rest of the words will fail. Here's a sample :

പേരിൽ 254
ചെയ്യുന്നു 254
നിരവധി 254
പുതിയ 254
  254
വിവിധ 254
കേരളത്തിലെ 254
കേരള 254
ചെറിയ 254

Words from വിവിധ wouldn't get learned because of a hidden <0xa0> character in the previous line.

Currently the install process is a 3-step path :

• Install GoVarnam (this repo)
• Install language support needed by user: https://github.com/varnamproject/schemes
• Install IBus Engine for GoVarnam: https://github.com/varnamproject/govarnam-ibus

Make this 3-step process into a single one click installer script. End result :

curl http://raw.githubuser....../install.sh | bash

Welcome to Varnam Installer. This installation is a 3-step process.
Step 1: Install GoVarnam
Start step 1. ? (yes/NO): yes

Downloading GoVarnam version 1.5.0...
<Download progres>
Installing GoVarnam...
Installed

Step 2: Install your language support
Languages:
----|-------------|
| as | Assameese |
....
| ml | Malayalam |

Which language would you like to install ? (Separate by comma): ml,as

Downloading...
<Download progres>
Installing...
Installed ml

Same but installed as.

Looks like `ml` has words to import. Import words for "ml" ? (yes/no): yes
<Importing progress>

Step 3: Install Varnam IBus Engine
Proceed ? (yes/NO): yes
Downloading...
<download progress>
Installing...

Varnam installation finished.

Telegram Group: 
Matrix Group: 
Website: 

Steps to reproduce:

1. Import many words
2. Unlearn a word
3. Try transliterating a word varnamcli -s ml ennum
4. "Database image is malformed" error is in output

On investigation, this bug is because of syncing problem between table & FTS table. The DELETE trigger has a problem.

Bug discovered thanks to this meme

Fixes #21

• Adds a new higher prioirty ExactWords result along with ExactMatches. How it differs:
  • ExactWords - Exactly found words in dictionary if there is any.
  • ExactMatches - Exactly starting word matches in dictionary if there is any. Not applicable for patterns dictionary.
• Avoid item variable in range loops to save memory copying wherever possible. for i, item := range to for i := range

Bug obtained from mwordle. Type "param". API gives {exact_matches: [word: "പരമ്", weight: 253]} but the DB doesn't have the single word പരമ്. There are words like പരമ്പര in DB which is basically പരമ് + പര. Varnam is confusing with it ?

The ideal output is പരം. It's in the exact_matches list but second with weight 252.

Add a new functionality varnam_get_suggestions(string word) to get all suggestions from dictionary starting with a particular word. This word won't be English but a language word itself.

Example:

varnam_get_suggestions("മല")
// Gives മലയാളം മലയാളചലച്ചിത്രം മലപ്പുറം മലയാളത്തിൽ

How is it useful ?

One useful case is with Inscript engines. An Inscript engine is letter-by-letter, there's no Manglish, so once you make a word it will help in giving suggestions. Currently govarnam-ibus gives Inscript output English first and then user has to pick suggestion to complete the word. This is bad practice as the Inscript English key output will be hard to understand.

Namaskar! 😃

The reverse transliteration of "प्रयत्न" is shown as p~ryt~n but if the same sequence is typed into the web editor then "प्रयत्न" doesn't get shown in any of the options. In fact all 3 options are the same.

Why is this happening and how can this be fixed?

VST Maker should allow to remove a symbol from VST using a matching condition. In Malayalam scheme:

anusvara [["m"]] => ["ം","ം","മ"]
anusvara "m_" =>  ["ം","ം","മ"]
anusvara({:accept_if => :ends_with}, "m" => ["ം","ം","മ"])
anusvara({:accept_if => :in_between}, "m" => ["ം","ം","മ"])

consonants ["ma"] => "മ"

generate_cv

The CV generation makes m => മ് but there is no use of മ് at the end of a string, anusvara will be used instead. So, need to remove the generated m => മ് and then custom add it:

anusvara({:accept_if => :starts_with}, "m" => ["മ്"])
anusvara({:accept_if => :in_between}, "m" => ["മ്])

• [x] Transliteration

• [x] Reverse Transliteration

• [x] Learning, Training from CLI

• [x] Learning, Training from file

• [x] VST Creation

• [ ] Stem rules

  This may not be needed cause there's better stemming tools from SMC. Besides, stemrules are only set for Malayalam in varnam.

• [ ] Using flags column in symbol table (This may not be needed cause govarnam works just fine without using flags)

Ok the word is >> சொல்லாமல் , which is (sollamal)

But in varnam while typing it gives the other il (ள)

If we type >> il (it should give (இல்) ) which is correct, but if you type ill it gives (இள்), but instead of (இள்) it should give us இல்

The CAPS ILL does the same.