Regulus News

EBL mode

2010-08-04T05:01:00.000-07:00

Yet another thing I should have done years ago. As everyone knows who's tried to do it, writing operationality criteria is rather messier than it should be. One of the worst things is that you don't immediately get feedback on the effects of the changes you've made. You need to rebuild the whole specialised grammar, which typically takes a while, and then you have to go through it to find out what the effect of your changes was.

I've just added an enhancement that should substantially simplify the development process. The new command EBL_MODE puts the top-level in a mode where each input sentence is subjected to EBL analysis, using the current set of operationality criteria; the operationality criteria file is reloaded each time, in case it has changed. The system prints a list of derived rules, with the substring used to derive each rule. Here's an example from CALL-SLT/French:


>> EBL_MODE
(Do EBL processing on input sentences)

--- Performed command EBL_MODE, time = 0.00 seconds

>> puis je avoir un sandwich

Taking operationality criteria from d:/cygwin/home/speech/call-slt/fre/prolog/operationality_recognition.pl

--- Written compiled operationality file (37 items) d:/cygwin/home/speech/call-slt/fre/generatedfiles/fre_recognition_tmp_ebl_operational.pl
% compiling d:/cygwin/home/speech/call-slt/fre/generatedfiles/fre_recognition_tmp_ebl_operational.pl...
%  module tmp_ebl_operational imported into user
%  module lists imported into tmp_ebl_operational
%  module utilities imported into tmp_ebl_operational
%  module ebl_operational imported into tmp_ebl_operational
% compiled d:/cygwin/home/speech/call-slt/fre/generatedfiles/fre_recognition_tmp_ebl_operational.pl in module tmp_ebl_operational, 0 msec 6928 bytes

Rule of form ".MAIN-->utterance"
derived from [puis,je,avoir,un,sandwich]

Rule of form "utterance-->med_utterance"
derived from [puis,je,avoir,un,sandwich]

Rule of form "med_utterance-->vp"
derived from [puis,je,avoir,un,sandwich]

Rule of form "vp-->vbar,vbar,np,optional_pp"
derived from [puis,je,avoir,un,sandwich]

Rule of form "vbar-->verb,hyphen,pronoun,optional_adverb"
derived from [puis,je]

Rule of form "verb-->puis"
derived from [puis]

Rule of form "hyphen-->[]"
derived from []

Rule of form "pronoun-->je"
derived from [je]

Rule of form "optional_adverb-->[]"
derived from []

Rule of form "vbar-->verb,optional_adverb"
derived from [avoir]

Rule of form "verb-->avoir"
derived from [avoir]

Rule of form "optional_adverb-->[]"
derived from []

Rule of form "np-->spec,n"
derived from [un,sandwich]

Rule of form "spec-->un"
derived from [un]

Rule of form "n-->sandwich"
derived from [sandwich]

Rule of form "optional_pp-->[]"
derived from []

Regulus with Nuance 9, continued

2010-04-13T14:27:00.000-07:00

Success, of sorts! I have a first version of the Nuance 9/server integration checked in, and Matthew tells me it runs correctly on his machine. (I don't yet have a full Nuance 9 installed here). Everything appears to do what it's supposed to: the original grammar is compiled into a Nuance 9 GrXML grammar, with the semantics transformed into string-concatenation semantics that put together a string representation of the semantic form. This is passed from the MRCP process to the dialogue server, which unpacks the strings, reconstructs the real semantic forms, and then passes them to downstream processing.

We were expecting that we'd save a non-trivial amount of time by skipping the parsing phrase in the server... but Matthew tells me that, as far as he can tell, it's slightly slower than when we were running the recogniser without semantics, and creating the semantic forms on the server side! I don't understand this at all. Will run some offline tests tomorrow and see if I can spot any obvious time-sink.

Regulus with Nuance 9

2010-04-12T12:48:00.001-07:00

The last couple of weeks, I've been working with Matthew Fuchs (paideia.com) on getting Regulus to work with Nuance 9. We've had a bunch of problems, but we're making good progress and are nearly at the point of having things up and running.

When we started, we knew about two things right off. Nuance 9 doesn't support recursive grammars, and it doesn't allow GSL either; instead, it requires GrXML, where the semantics are done using ECMAScript (JavaScript). Our basic strategy was to carry on generating Nuance 8.5 GSL, and rely on the Nuance conversion tool, which is supposed to be able to convert GSL into GrXML. We also had to make sure that the grammars were non-recursive.

I added code to check for recursivity, and it turns out to be easy enough, at least in the cases we've looked at so far, to fix the operationality criteria in grammar specialisation so that the generated grammars are non-recursive. It wasn't so easy, though, to use the 8.5 to 9 conversion tool, since it turned out that it didn't handle the 'concat' operator, completely central to Regulus semantics.

We wondered for a while if we'd either have to give up on using semantics in Nuance 9, and parse everything in Regulus, or else have Regulus directly generate Nuance 9 grammars - possible, but non-trivial. But I thought of a cute work-around over the weekend, which seems to solve the problem for now. Instead of generating the actual semantics, we generate strings which encode the semantics, and put them together with 'strcat' rather than 'concat' - the strcat operator is handled by the conversion tool.

Another reason why we were reluctant to generate Nuance 9 semantics directly is that, as far as we can make out, Nuance 9 doesn't have a tool for doing PCFG tuning, which is essential to good performance. But, with the current scheme, we can generate Nuance 8.5 grammars, do the PCFG tuning in 8.5, and then translate into 9. The conversion tool correctly carries across the generated probabilistic weights.

It's a bit of a Frankenstein's monster, but it does all seem to work! Matthew just told me that he was able to run the Nuance 9 grammar successfully in MRCP. Now we just need to integrate everything with the dialogue server, and we'll have the first version fully running. More soon, I hope...

Screenshots from Japanese CALL-SLT

2010-02-07T07:25:00.000-08:00

The system shows a French prompt, "DEMANDER DE_MANIERE_POLIE EAU"

The student speaks a corresponding sentence in Japanese, here "omizu onegai shimasu". The system does speech recognition and understanding, and shows the recognised sentence to the student in both Japanese and Roman orthography.

If the students wishes, they can click the help button, to get an example of a Japanese native speaker saying something that will work.

A Scandinavian/English grammar, continued

2010-01-13T07:47:00.000-08:00

More progress on the Scandinavian/English grammar:

I've now got quite reasonable initial rules for negation and lexically passivized verbs, two of the largest holes in the Swedish. Negation, in Swedish, is just an adverb. I did however have to add a feature which distinguished main clauses from subordinate clauses, since the negation adverb occurs after the verb in a main clause, and before it in a subordinate clause. It turned out to be easy to adapt the existing rules for passives to handle lexical passives as well: the grammar now allows passive versions of the present, imperfect, supine and infinitive forms. Thus for example den kan inte köpas här, "it can not buy-INF-PASSIVE here" = "it can not be bought here".
I've improved the coverage in the Swedish version of MedSLT. I can now translate 92% of the combined interlingua corpus into Swedish, and translate back 99% of the results. This mostly involved adding new lexical items and transfer rules, though I also had to make a couple of minor adjustments to the grammar.

Scandinavian/English grammar now used for English MedSLT

2010-01-08T04:10:00.000-08:00

I've now changed the config files for English MedSLT so that they use the shared Scandinavian/English grammar rather than old English-only grammar. I found a couple of small bugs, but now everything seems to be working fine again.

Some time soon, Nikos and I should get together and figure out how to add Swedish to the MedSLT demo and the nightly build. It shouldn't be at all hard.

Swedish MedSLT, continued

2010-01-02T20:14:00.000-08:00

I temporarily broke off working on Interlingua to Swedish, and spent a day concentrating on the opposite direction. I built the recognition grammar by training on a corpus which was the union of the original recognition corpus and the generation corpus (this ensures that everything you can generate will also get recognized); then I did PCFG tuning using the set of translations produced from the combined Interlingua corpus. I also used the set of translations as the initial Swedish corpus for translation testing. All the corpora concerned are created on-the-fly as part of the make process, so the correspondences will stay up to date.

It was easy to get things working in Swe -> Int direction, and 98% of the translation corpus now produces well-formed interlingua. I compiled a Swedish recognizer, and hooked everything together to get a speech-to-speech system for Swedish -> English. Anecdotally, it's not bad.

The most urgent thing now is probably to add more Swedish coverage. There are several very common constructions that currently aren't in the specialized Swedish grammar.

Swedish MedSLT, continued

2009-12-30T10:06:00.000-08:00

A few more days of messing around, and I'm now translating about 80% of the 1200-item combined interlingua corpus into Swedish. Elisabeth (a native speaker) looked at about half of the material, and made some suggestions in the direction of improving quality. After implementing them, she thinks that over 90% of the translations are clearly good. To do this work, I have had to make a few more improvements in the combined Scandinavian/English grammar: the most important of these is an initial treatment of lexically reflexive verbs.

I'll see if I can improve the numbers a bit more, and will then start on the Swedish-to-Interlingua direction.

Swedish MedSLT, continued

2009-12-25T15:02:00.001-08:00

Encouraging progress on Swedish MedSLT: I can now translate a third of the interlingua corpus into Swedish. The translations aren't very good yet, but they are nearly all grammatical. I think it will be fairly easy to improve things.

Swedish MedSLT

2009-12-22T06:47:00.000-08:00

I have just started on a Swedish version of MedSLT; this will give the new Scandinavian/English grammar a much more thorough workout. After an hour or two of messing around, I can parse one sentence, var har du ont ("where is your pain", literally "where have you pain").

The first thing I notice is that we need rules for impersonal and lexically reflexive verbs, which are very important in this domain. In particular, an expression that occurs all the time is det gör ont; "it hurts", literally "it makes pain".

More soon.

A Scandinavian/English grammar, continued

2009-12-19T06:46:00.000-08:00

I've now changed the config files for English CALL-SLT so that they use the new Scandinavian grammar instead of the English-only one that it's based on. This will give us more of a chance to test how it works.

Next, I will make similar changes in the English part of MedSLT.

A Scandinavian/English grammar, continued

2009-12-18T01:02:00.000-08:00

The Scandinavian grammar is working quite well, but it's certainly not complete yet. Here are some important things that still need to be added, all of which occur in Scandinavian but not in English:

Negation. This is just an adverb in Scandinavian. The slightly non-trivial thing is that the position of this adverb (also some others) is different, depending on whether it's a main or a subordinate clause.
Lexically reflexive pronouns. As in Romance, some Scandinavian verbs subcategorize for lexically reflexive pronouns, which have no semantic value.
Lexical passive. Scandinavian verbs have a lexical passive form. I propose to do this in the morphotax.
Definiteness. I don't yet have all the definiteness constraints. In particular, a definite singular NP has an implicit definite article, but a premodifying adjective requires an explicit definite article.

I don't think any of these things are particularly difficult to implement, or should require major changes to the grammar.

Swedish CALL-SLT

2009-12-16T07:03:00.001-08:00

I must stop messing around with the Scandinavian grammar... it's really too much fun! Anyway, I now have a reasonable first cut at a Swedish version of CALL-SLT, working as usual in the restaurant domain. Elisabeth helped me add more material to the Swedish corpus last night; currently, it contains about 160 entries, of which about 90% work. I just tried out the live system, using Maria's GUI, and it runs fine. As I'd hoped, recognition picked up a good deal once I was able to switch on N-best rescoring. I'm getting performance in Swedish only slightly inferior to what I get in English, which seems reasonable given my relative abilities in the two languages.

If people want to try it out, you need to update both Regulus and CALL-SLT using the -d flag, and do a make in CALLSLT/Swe/scripts. Then run in the usual way. So far there is no spoken help, but Elisabeth has promised to record files over Christmas.

A Scandinavian/English grammar, continued

2009-12-15T02:38:00.000-08:00

A bit more fiddling around, and I have two-thirds of the initial Swedish CALL-SLT corpus parsing. You can see it here. I've also compiled an initial recogniser. So far, it doesn't work very well, but if it's like the other languages it will improve considerably once I add N-best rescoring.

This is all progressing rather nicely! I need to do some work now on processing the results of our recent CALL-SLT experiments, but once I've done that I'll return to the Swedish. Elisabeth says she will act as our native speaker.

A Scandinavian/English grammar

2009-12-14T01:07:00.001-08:00

This weekend, I finally started on a task that I've been meaning to do for ages, and put together a first version of a shared grammar that is intended to cover both English and the Scandinavian languages Swedish, Norwegian and Danish. I've checked it into Regulus/Grammar/Scandinavian. Initially, I'm developing using the CALL-SLT restaurant domain, in English and Swedish. The Swedish version can already handle a reasonable range of language. Here's an example: skulle jag kunna få en pizza = would I be-able-to get a pizza. This is a common way to ask for something in Swedish.

>> skulle jag kunna få en pizza
(Parsing with left-corner parser)

Analysis time: 0.34 seconds

3 possibilities:

----------------------------------------------------------------
Possibility 1
Return value: [(null=[action,få]), (object=[food,pizza]), (null=[modal,kan]),
         (null=[modal,skulle]), (agent=[pronoun,jag]), (null=[utterance_type,ynq]),
         (null=[voice,active])]

Global value: []

Syn features: []

Parse tree:

.MAIN [GENERAL_SCA:541-546]
top [GENERAL_SCA:552-558]
/  utterance_intro null [GENERAL_SCA:566-568]
|  utterance [GENERAL_SCA:615-620]
|     s [GENERAL_SCA:713-718]
|        s [GENERAL_SCA:817-826]
|           vp [GENERAL_SCA:1124-1137]
|           /  vbar [GENERAL_SCA:876-898]
|           |  /  v lex(skulle) [GEN_SWE_LEX:51-51]
|           |  |  np [GENERAL_SCA:1952-1960]
|           |  \     pronoun lex(jag) [GEN_SWE_LEX:200-200]
|           |  vp [GENERAL_SCA:1124-1137]
|           |  /  vbar [GENERAL_SCA:853-875]
|           |  |     v lex(kunna) [GEN_SWE_LEX:61-62]
|           |  |  vp [GENERAL_SCA:1317-1337]
|           |  |  /  vp [GENERAL_SCA:1042-1051]
|           |  |  |  /  vbar [GENERAL_SCA:853-875]
|           |  |  |  |     v lex(få) [CALLSLT_LEX:33-33]
|           |  |  |  |  np [GENERAL_SCA:2073-2091]
|           |  |  |  |  /  np [GENERAL_SCA:1907-1917]
|           |  |  |  |  |  /  d lex(en) [GEN_SWE_LEX:277-278]
|           |  |  |  |  |  |  nbar [GENERAL_SCA:2118-2130]
|           |  |  |  |  |  \     n lex(pizza) [CALLSLT_LEX:179-181]
|           |  |  |  \  \  post_mods null [GENERAL_SCA:1451-1457]
|           \  \  \  post_mods null [GENERAL_SCA:1451-1457]
\  utterance_coda null [GENERAL_SCA:597-599]

------------------------------- FILES -------------------------------

CALLSLT_LEX: d:/cygwin/home/speech/call-slt/swe/regulus/callslt_lex.regulus
GENERAL_SCA: d:/cygwin/home/speech/regulus/grammar/scandinavian/general_sca.regulus
GEN_SWE_LEX: d:/cygwin/home/speech/regulus/grammar/scandinavian/swedish/gen_swe_lex.regulus

Here are some of the issues I've encountered so far:

Agreement is different in English and Swedish, so the range of possible values for the 'agr' feature has to be language-dependent. In English, agreement is by person and number. In Swedish, it's primarily by number and gender ("common" or "neuter"). However, you also need person, since reflexive pronouns agree with the subject in person.
All verbs invert in Swedish, and there is no auxiliary "do".
The range of verb inflections is different in the two languages. English verbs have five forms: base, third person singular present, imperfect, past participle, present participle. So, for example, go, goes, went, gone, going. Modern Swedish doesn't inflect by person or number in the present tense, but on the other hand distinguishes the imperative from the infinitive, distinguishes the "supine" (the form used for the perfect tense) from the past participle, and inflects the past participle by number and gender. So, for example, bryta (break) has the forms bryt (imperative), bryta (infinitive), bryter (present), bröt (imperfect), brutit (supine), brytande (present participle), bruten (past participle singular common), brutet (past participle singular neuter), brutna (past participle plural).
Swedish nouns inflect for definiteness. So for example bord is "table", but bordet is "the table". Adjectives also inflect for definiteness, thus ett stort bord ("a big table") but det stora bordet ("the big table").
Swedish possessives inflect for gender and number. So min bil ("my car", common/singular), mitt hus ("my house", neuter/singular), mina barn ("my children", plural).
Swedish partitives are slightly different. In English, "a bottle of beer"; in Swedish en flaska öl (nothing corresponding to the "of").
Swedish date and time grammar is slightly different. In English, "december fourteenth"; in Swedish, fjortonde december. In English, "nine thirty"; in Swedish, nio och trettio.
Swedish negation is basically an adverb, e.g. jag beställde inte någon pizza = I ordered not any pizza. The negation adverb's position is after the verb in main clauses, before in subordinate clauses.

Most of this stuff seems easy, and I can adapt the treatments we implemented in the SLT grammar during the 90s. I am guessing that 85-90% of the rules in the final shared grammar will be common to English and Swedish. Judging from our experiences with SLT, Danish and Swedish overlap to 95% or better, and Norwegian should be similar.

New treatment of Japanese verbs

2009-09-23T13:34:00.000-07:00

I've been discussing Japanese verbs with Yukie - we need new inflectional forms for CALL-SLT, in the particular the volitional (-tai) form, and the old system was getting out of hand. Japanese has extraordinarily regular morphology, with only two irregular verbs and some very straightforward sound-changes, so it seemed to me that we really ought to be able to get by without explicitly listing all the inflections of every verb we needed.

Yukie wrote down a table of inflections, and we discussed ways of splitting up inflected verbs into stems and affixes. Based on our discussion, I've implemented a first version of a new treatment, where you now only need to specify a single root form of the verb, and everything else is done by morphotax rules, where the affixes are treated by Nuance as separate words. I've tested by converting the Japanese Calendar lexicon to the new form, and compiling into a recognizer. Coverage is what it was, and recognition is anecdotally fine with my voice. I will dig out some Japanese Calendar data soon and run proper tests.

If anyone wants to look at the details, the morphotax rules are in $REGULUS/Grammar/Japanese/japanese_verb_morphology.regulus. The new version of the Japanese Calendar lexicon is at $REGULUS/Examples/Calendar/Regulus/japanese_calendar_lex_new.regulus.

Here's an example of a parse:


$ nanji ni owa ri mashita ka
(Parsing with left-corner parser)

Analysis time: 0.09 seconds

Return value: [[question,form(past,[[owaru],[ni,term(null,nanji,[])]])]]

Global value: []

Syn features: []

Parse tree:

.MAIN [JAPANESE_CORE_RULES:112-116]
  top [JAPANESE_CORE_RULES:117-119]
     utterance [JAPANESE_CORE_RULES:120-123]
     /  main_clause [JAPANESE_CORE_RULES:147-151]
     |     s [JAPANESE_CORE_RULES:155-162]
     |     /  comps [JAPANESE_CORE_RULES:190-195]
     |     |  /  pp [JAPANESE_CORE_RULES:414-423]
     |     |  |  /  np [JAPANESE_CORE_RULES:267-273]
     |     |  |  |     n lex(nanji) [JAPANESE_CALENDAR_LEX_NEW:84-84]
     |     |  |  \  p lex(ni) [JAPANESE_CALENDAR_LEX_NEW:274-284]
     |     |  \  comps null [JAPANESE_CORE_RULES:163-166]
     |     |  vbar [JAPANESE_CORE_RULES:249-253]
     |     |     v [JAPANESE_VERB_MORPHOLOGY:13-26]
     |     |     /  v_stem [JAPANESE_VERB_MORPHOLOGY:27-38]
     |     |     |  /  v_stem lex(owa) [JAPANESE_CALENDAR_LEX_NEW:227-237]
     |     |     |  \  stem_affix lex(ri) [JAPANESE_VERB_MORPHOLOGY:138-138]
     |     \     \  affix lex(mashita) [JAPANESE_VERB_MORPHOLOGY:80-83]
     \  lex(ka)

------------------------------- FILES -------------------------------

JAPANESE_CALENDAR_LEX_NEW:
d:/cygwin/home/speech/regulus/examples/calendar/regulus/japanese_calendar_lex_new.regulus
JAPANESE_CORE_RULES:     
d:/cygwin/home/speech/regulus/grammar/japanese/japanese_core_rules.regulus
JAPANESE_VERB_MORPHOLOGY:
d:/cygwin/home/speech/regulus/grammar/japanese/japanese_verb_morphology.regulus

"Abstract actions" and the dialogue server

2009-09-17T00:06:00.000-07:00

I've had some productive discussions with Maria over the last few days, which have resulted in a couple of significant improvements to the dialogue server. Maria is going to build a Java GUI for the CALL-SLT system. She needs to be able to send requests to the dialogue server, and get back information that she will pass on to the user. Most often this will be in the form of screen-based output. The new functionality is motivated by this scenario, but is quite generic.

The first point Maria made was that she would prefer to use XML-formatted messages. Java finds it easy to manipulate XML; parsing Prolog messages, on the other hand, is a complete pain. So I added switches that allow the client to put the server into a mode where all messages are XML strings inside a minimal Prolog wrapper.

Yesterday, Maria made another very sensible request. In the first version of the application, the Prolog "output manager" module received abstract actions, and transformed them into concrete actions. Typically, concrete actions would involve printing strings. So, for example, suppose that the system has just given you the prompt

POLITE REQUEST TABLE outside

and you have correctly replied

i would like a table outside please

The abstract action produced is


display_matching_info('i would like a table outside please', correct, [2,1,2])

which means that the recognized words were 'i would like a table outside please'; they correctly matched the prompt; and the score is now 2 correct, 1 incorrect, with a positive streak of 2. This is converted by the output manager into the concrete action


print('I heard: "i would like a table outside please"

Correct!

Score: 2 right, 1 wrong (66.7%) Streak: 2')

This works fine for a text-based command-line interface; but, as Maria pointed out, the output processing isn't necessarily appropriate when you have a Java Swing GUI, in which case you probably would prefer to format it yourself. For instance, you might want to print the recognized words in one pane, render the "correct" as a green tick-mark in another one, and present the score graphically as three columns of different heights.

In general, the abstract action is going to be more useful to you than the concrete one. So I've just added a little more functionality to the dialogue server to handle that too. Here's a summary of the new messages, and what they do; they are also documented in the file itself, $REGULUS/Prolog/dialogue_server.pl.

action(xml_messages). Format future messages in both directions in XML form. Each message will be of the form

xml_message(XMLString).

where XMLString is an XML encoding of the corresponding Prolog message produced using the predicate prolog_xml/2 in $REGULUS/PrologLib/prolog_xml.pl. The XML can be converted back into Prolog if necessary using the same predicate.
action(prolog_messages). Format future messages in both directions in Prolog form (default).
action(abstract_actions). Pass abstract actions to the client, so that the client can do its own output management.
action(concrete_actions). Pass concrete actions to the client (default).

Warning: don't use SICStus 4.0.2

2009-09-15T09:54:00.000-07:00

Maria and I have just spent two very frustrating days trying to figure out why CALL-SLT wasn't running correctly on her machine. In the end, it turned out that a few bits of Regulus functionality don't work correctly under SICStus 4.0.2, which is the version she was using... there appears to be something wrong with the SICStus/operating system interface.

So avoid this release! 4.0.4 is fine.

Dialogue server now accepts XML format messages

2009-09-13T16:18:00.000-07:00

Another of those things I should no doubt have done years ago: after a discussion with Maria, I've now modified the dialogue server so that it can also run in a mode where all messages are XML-formatted.

Details (this is documented in $REGULUS/Prolog/dialogue_server.pl):

- Initially, the server is in Prolog mode.

- To put the server into XML mode, send the message

xml_messages.

- Subsequent messages are of the form

xml_message(XMLMessage).

where XMLMessage is an XML encoding of the corresponding Prolog message produced using the predicate prolog_xml/2 in $REGULUS/PrologLib/prolog_xml.pl.
The XML can be converted back into Prolog if necessary using the same predicate.

I have tested by converting the CALL-SLT Prolog client to use XML-flavor messages, and  it all works fine.

The routines in $REGULUS/PrologLib/prolog_xml.pl should in general be useful for translating Prolog into XML form in a reversible way. Look at the file for documentation and an example.

CALL-SLT and Japanese

2009-09-07T02:44:00.001-07:00

I have just added a little more coverage to the Japanese version... it now has about a dozen sentences for the student to practice on. I tried it, and so far it still recognizes everything I say. This is probably more because vocabulary is so small than because I have a wonderful Japanese accent :)

Yukie and I should talk about how to proceed here. The first step will be to add material to the Japanese corpus.

Using recorded wavfiles as help information in CALL-SLT (part 2)

2009-09-05T23:02:00.001-07:00

I did a little more fiddling around with the translation game strategy code, and it's now possible to define a strategy where the system only chooses entries which don't have an associated wavfile. The idea is to make it easy for the teacher to add missing wavfiles.

I tested it on English, and we now have a complete set of wavfiles for that language. As soon as we have a bit more coverage for French and Japanese, I'll add similar scripts for them too.

Using recorded wavfiles as help information in CALL-SLT

2009-09-04T14:00:00.000-07:00

I've got a new feature working on CALL-SLT, which allows speech input to be logged and reused as help for students. When you start the system, it asks whether or not you wish to be considered a native speaker. If you answer yes, it keeps the wavfile for each successful match, and stores it in such a way that that the wavfile is associated with the current prompt. Subsequently, if a student is given the same prompt and hits the HELP button, the native speaker's wavfile is replayed. By construction, we know that the native speaker was correctly recognized, so if the student can just imitate them well enough they should be recognized too.

The idea is simple, but there were some messy technical problems... a bad interaction between Nuance and SICStus concerning relative pathnames, and the question of what happens if two different users try to check in new wavfiles simultaneously. I think I have decent solutions, though. For more details, look at the online documentation which I have just added. This also tells you how to download and run the system.

CALL-SLT and Japanese

2009-08-23T13:43:00.000-07:00

And we now have a skeleton Japanese system too. So far, it can only do one sentence,


   hitori no teeburu wa arimasu ka

which is rendered in the Interlingua as


    POLITE REQUEST TABLE 1 PERSON

Though it is nice that this goes all the way through: I can get the Interlingua as a prompt, speak the sentence, and be informed that I got it right. Not surprisingly, since the grammar doesn't cover anything else, it's very reliable when you say the one thing it knows!

Yukie and I need to get together and add more content. The first step will be for Yukie to flesh out the corpus, which currently only has a dozen or so examples.

CALL-SLT and French

2009-08-22T22:40:00.000-07:00

We should now have a complete set of of scripts, config files etc for the French version of CALL-SLT. I added a couple of placeholder files, with enough translation rules to do the sentence "Je voudrais deux bières". The make appears to work, and I was able to run the initial version of the translation game in the server. Over to Pierrette to add some actual content!

Next, Japanese...

Progress on CALL-SLT

2009-08-22T00:25:00.000-07:00

I have the generic CALL-SLT functionality packaged up so that it can be run inside the dialogue server - this involved extending the dialogue server a little bit, so that you can now call recognition from inside it. The interface between the client and the CALL-SLT-loaded dialogue server is consequently very simple. There are so far just three commands:

"Next prompt". The server generates a new prompt, using its current strategy, and returns it.
"Recognize and match". The server performs recognition, translates it to Interlingua, matches against the current prompt, and returns a string explaining what happened.
"Help". The server returns the current prompt, plus a text example illustrating one possible way to realize the prompt.

I have been testing with a minimal Prolog-based client. It should be easy to write a Java client which offers a nice GUI-style interface.

Progress on CALL-SLT

2009-08-20T03:51:00.000-07:00

I have a first version of the translation game working for English! It's still very clunky indeed (I'm running it from the command-line in the development environment), but it's already kind of fun. Next step will be to package it up in a better way, using the dialogue server; this should be quite easy. Once I've done that, it'll be possible to add a Java GUI, so that we have a complete first version of the system.

Of course, what I really want to do is try it in another language... I already know how to order in English restaurants! I'll start sorting out the config files and scripts for Japanese. My restaurant Japanese is shaky, and I'm very curious to see if I can use CALL-SLT to improve it.

Progress on CALL-SLT

2009-08-19T09:50:00.000-07:00

I've spent some time messing around with the English to Interlingua translation rules and the Interlingua grammar. We're now able to translate 95% of the development corpus into sensible-looking interlingua - though the current surface form will probably be revised a bit at some point by Pierrette and Johanna.

At any rate, you can now run the speech-input English system from the Regulus command-line, setting it so that spoken inputs are parsed and translated into Interlingua. I don't think it's that much extra work to add code so that we have a complete first version of the system. There will initially be two commands. You can either ask for a new Interlingua prompt, or ask to speak. If you speak, it will translate what you say into Interlingua, compare with the current prompt, and score you. With any luck, I'll have this working before next week.

Here's the current English development corpus, with the Interlingua translations it produces:

i will take a beer	REQUEST beer
give me a beer	REQUEST beer
hello	NEUTRAL-GREETING
good evening	EVENING-GREETING
i would like a table for one	POLITE REQUEST TABLE 1 PERSON
could i have a table for one	POLITE REQUEST TABLE 1 PERSON
do you have a table for one	POLITE REQUEST TABLE 1 PERSON
i would like a table for two	POLITE REQUEST TABLE 2 PERSON
could i have a table for three	POLITE REQUEST TABLE 3 PERSON
do you have a table for four	POLITE REQUEST TABLE 4 PERSON
do you have a table outside	POLITE REQUEST TABLE outside
is there a table outside	POLITE REQUEST TABLE outside
could i have a table outside	POLITE REQUEST TABLE outside
could we have a table by the window	POLITE REQUEST TABLE by-loc window
do you have a table near the window	POLITE REQUEST TABLE in-loc window
i 'd like a table in the smoking area	POLITE REQUEST TABLE smoking
i 'd like a table in the non-smoking area	POLITE REQUEST TABLE non-smoking
smoking please	POLITE REQUEST TABLE smoking
non-smoking please	POLITE REQUEST TABLE non-smoking
could i have a table in the corner	POLITE REQUEST TABLE in-loc corner
can i have a table in the corner	POLITE REQUEST TABLE in-loc corner
i would like a non-smoking table for one	POLITE REQUEST TABLE 1 PERSON non-smoking
do you have a non-smoking table for one	POLITE REQUEST TABLE 1 PERSON non-smoking
a non-smoking table for two please	POLITE REQUEST TABLE 2 PERSON non-smoking
could i have a non-smoking table for three people	POLITE REQUEST TABLE 3 PERSON non-smoking
i would like a table for three people in the smoking area	POLITE REQUEST TABLE 3 PERSON smoking
i would like a table for four	POLITE REQUEST TABLE 4 PERSON
do you have a table for four	POLITE REQUEST TABLE 4 PERSON
i would like to reserve a table for seven o'clock	POLITE REQUEST TABLE 19 00
could i reserve a table for seven thirty	POLITE REQUEST TABLE 19 30
i 'd like to reserve a table for six forty five	POLITE REQUEST TABLE 18 45
i would like to reserve a table for two for seven fifteen	POLITE REQUEST TABLE 2 PERSON 19 15
could i reserve a table for two people for eight o'clock	POLITE REQUEST TABLE 2 PERSON 20 00
do you have a table for two at seven thirty	POLITE REQUEST TABLE 2 PERSON 19 30
could i reserve a table for seven o'clock tomorrow evening	POLITE REQUEST TABLE 19 00 time-evening date-tomorrow
i would like to reserve a table for six forty five tomorrow please	parsing_failed
could i reserve a table for two for tomorrow evening	POLITE REQUEST TABLE 2 PERSON time-evening date-tomorrow
could i reserve a table for this evening	POLITE REQUEST TABLE time-evening date-today
do you have a table for three tomorrow evening around seven o'clock	POLITE REQUEST TABLE 3 PERSON 19 00 time-evening date-tomorrow
i have a reservation in the name of smith	parsing_failed
i should have a reservation in the name of smith	parsing_failed
could we see the menu	POLITE REQUEST menu
i would like to see the menu	POLITE REQUEST menu
could we get the menu	POLITE REQUEST menu
could we have the bill	POLITE REQUEST check
could we have the bill please	POLITE REQUEST check
may i have the bill	POLITE REQUEST check
may i have the check	POLITE REQUEST check
could you give us the check please	POLITE REQUEST check
could you give me a receipt	POLITE REQUEST receipt
i would like a receipt please	POLITE REQUEST receipt
could i get a receipt please	POLITE REQUEST receipt
i would like a beer	POLITE REQUEST beer
could we have two beers	POLITE REQUEST 2 beer
could you give us two beers	POLITE REQUEST 2 beer
could i have a latte	POLITE REQUEST latte
i 'd like a medium latte	POLITE REQUEST medium latte
a cup of tea please	POLITE REQUEST cup tea
two cups of tea please	POLITE REQUEST 2 cup tea
could we get two glasses of water	POLITE REQUEST 2 glass water
i 'd like a glass of the house red	POLITE REQUEST glass house-red-wine
could we have two glasses of the house red	POLITE REQUEST 2 glass house-red-wine
i would like a large glass of white wine	POLITE REQUEST large glass white-wine
i would like two small glasses of white wine	POLITE REQUEST 2 small glass white-wine
i would like a pizza	POLITE REQUEST pizza
i would like a hamburger	POLITE REQUEST hamburger
i would like the soup	POLITE REQUEST soup
i would like two pizzas	POLITE REQUEST 2 pizza

Progress on CALL-SLT

2009-08-18T14:30:00.000-07:00

I've now got enough stuff working in CALL-SLT that it's possible to translate a few simple sentences from spoken English to Interlingua. Here's an example:


>> i would like two beers

Source: i would like two beers
Target: POLITE REQUEST two beer
Other info:
n_parses = 1
parse_time = 0.047
source_representation = [(agent=[pronoun,i]), (null=[action,like]), (null=[modal,would]),
                         (null=[utterance_type,dcl]), (null=[voice,active]),
                         (object=[drink,beer]), (object=[spec,2])]
transfer_to_source_discourse_time = 0.0
source_discourse = [(null=[utterance_type,dcl]), (agent=[pronoun,i]), (null=[voice,active]),
                    (null=[modal,would]), (null=[action,like]), (object=[spec,2]),
                    (object=[drink,beer])]
resolved_source_discourse = [(null=[utterance_type,dcl]), (agent=[pronoun,i]),
                             (null=[voice,active]), (null=[modal,would]), (null=[action,like]),
                             (object=[spec,2]), (object=[drink,beer])]
resolution_processing = trivial
resolution_time = 0.0
transfer_to_interlingua_time = 0.0
interlingua = [(arg2=[drink,beer]), (arg2=[number,2]), (null=[politeness,polite]),
               (null=[utterance_type,request])]
interlingua_surface = POLITE REQUEST two beer
interlingua_checking_time = 0.0

--- Performed command i would like two beers, time = 0.05 seconds

We are not far from having everything we need to be able to build and run an initial version of the CALL-SLT server. Originally, it will prompt in the surface interlingua. but I'll leave in a hook so that we can use the picture interlingua as soon as we have something available.

CALL-SLT corpus

2009-08-16T07:33:00.000-07:00

For people who don't already know, we are collecting our initial corpus for CALL-SLT on a Google Docs document. If you don't already have access to this document and would like to contribute, please get a Google account and mail me your Google address.

All contributions very gratefully received - and don't worry that you might break something accidentally, Google Docs has excellent facilities for tracking revisions and if necessary reverting to earlier versions.

Here's a sample extract showing what the corpus looks like:

Fre: auriez-vous une table sur la terrasse (SVP) ?
Fre: auriez-vous une table en terrasse ?
Eng: do you have a table outside
Eng: is there a table outside
Eng: could i have a table outside
Int: POLITE REQUEST LOCATION-TABLE LOCATION-OUTSIDE [note: is "en terrasse" different from "outside"?]

Fre: auriez-vous une table près de la fenêtre (SVP) ?
Fre: auriez-vous une table à côté de la fenêtre (SVP) ?
Jap: mado gawa no seki wa ari masu ka
Eng: could we have a table by the window
Eng: do you have a table near the window
Int: POLITE REQUEST LOCATION-TABLE NEAR WINDOW

Fre: auriez-vous une table non-fumeur/fumeur (SVP)?
Fre: auriez-vous une table dans la zone non-fumeur /fumeur (SVP)?
Fre: auriez-vous une table dans la section non-fumeur/fumeur (SVP)?
Jap: kinenseki wa ari masu ka
Eng: i 'd like a table in the smoking area
Eng: i 'd like a table in the non-smoking area
Eng: smoking please
Eng: non-smoking please
Int: POLITE REQUEST LOCATION-TABLE IN SMOKING-AREA
Int: POLITE REQUEST LOCATION-TABLE IN NON-SMOKING-AREA

Fre: auriez-vous une table dans le coin (SVP) ?
Jap: oku no seki wa ari masu ka
Jap: oku no teeburu wa ari masu ka
Eng: could i have a table in the corner
Int: POLITE REQUEST LOCATION-TABLE IN CORNER

First CALL-SLT meeting

2009-08-16T06:49:00.000-07:00

We've had our first CALL-SLT meeting, which has done a lot to clarify our immediate goals for the project. We're going to start by building a simple version of the system, constructed in such a way that it will be easy to upgrade it by successively replacing simple modules by more complex ones. Initially, we will be working in the tourism/restaurant domain, using the languages English, French and Japanese. When we have those working well enough, we'll also start on Chinese; this is a language none of us know, so it will give us intuitions about what it's like to be an elementary-level student trying to use the system to get some fluency in a language.

The initial prototype will work as follows. At each turn, the system prompts the student with a description of what they are supposed to say, formulated in a version of the Interlingua. The student will attempt to speak it in the L2 (the language they are trying to learn). The system applies speech recognition to the student's utterance, then tries to translate the result into the interlingua. Finally, it compares the translated interlingua with the one used to prompt the student, and gives them feedback on how they did. Here are more details:

Prompting in Interlingua. In the first version, the interlingua will be shown to the student in a text-based form, using the methods we've developed under MedSLT. So for example, the system might show the student
```
POLITE REQUEST TABLE 3 PERSON TIME 19:30
```
expecting the student to say something like
```
I would like to reserve a table for three people at seven thirty
```
or whatever the equivalent is in the L2 they are using.
As soon as we've figured out a good way to do it, we would like to be able to present the interlingua prompt in graphical form. So here, we might have a picture that could be described as
```
Scene:
Client is talking to waiter.
Speech bubble from client.
Inside speech bubble:
  three chairs around a restaurant table;
  large clock in background shows 19:30
```
All speech input to the system will be logged in the usual way. We will have a registration process which allows us to associate each recorded utterance with meta-data which in particular will specify whether or not the utterance was recorded by a native speaker, and whether or not speech recognition got it right.
When the system has compared the student's interlingua with the prompt interlingua, there are two simple ways for it to give helpful feedback. The first is to present both versions of the interlingua, highlighting the elements that are different. For instance, in the example above, if the system recognized
```
Could I have a table for two people at seven thirty
```
then the system would present the prompt and recognized interlinguas roughly as follows:
```
POLITE REQUEST TABLE *3* PERSON TIME 19:30
POLITE REQUEST TABLE *2* PERSON TIME 19:30
```
The second way to give help will be to play an example of a native speaker saying some version of the sentence in the L2, if such an example already exists.
The prompt selection module will have hooks allowing specification of a strategy. A simple strategy we will implement soon is to choose the prompt from a list of examples where there is a recorded example of a native speaker saying the prompt in the L2, possibly with some other constraints. This will make it easy for a teacher to create a lesson. They will first interact with the system in the L2, to create a set of recorded examples which work correctly. When the student logs on, the system will then be set to select prompts matching the teacher's examples.
The functionality will be bundled up as a Prolog-based server, which does most of the processing, and will connect to a lightweight Java-based GUI which presents a client view. The server will initially handle two messages: (1) NEXT_EXAMPLE, returning a new interlingua prompt with associated information, and (2) RECOGNISE, prompting the student to speak, carrying out recognition, and returning the pieces of information produced by carrying out the interlingua comparison process.

Initial English recognizer for CALL-SLT (Part 2)

2009-08-10T09:37:00.000-07:00

I have done some more work on the English CALL-SLT recognizer, and we now have about 170 surface words. You can order food and drink in various ways, e.g.


i would like two beers
could i have a pizza

There is language for reserving tables, e.g.


do you have a table for two at seven thirty
could i have a non-smoking table for three people

I've also added a few more things like asking for the menu and the check. If I could say all this stuff in a language I didn't already know, say Chinese, I'd really feel I'd learned something useful!

Initial English recognizer for CALL-SLT

2009-08-10T05:44:00.000-07:00

We can now build an initial English recognizer for CALL-SLT too. The domain is the same, ordering in restaurants. The training corpus currently contains about 60 examples, about 95% of which parse. Vocabulary is about 130 surface words. Recognition is anecdotally quite good with my voice, though it will of course be more interesting to see how foreign voices do.

If you want more details, here are the corpus and the domain lexicon.

Initial French recognizer for CALL-SLT

2009-08-07T12:25:00.000-07:00

Pierrette's now checked in some real (as opposed to placeholder) material, and I was able to compile an initial French recognizer for CALL-SLT. It covers some very basic tourist French, so far all about ordering in restaurants. Simple as it is, I was already able to use it to improve my pronunciation of "Je voudrais un verre d'eau". As everyone who's heard me speak French knows, my version of the "r" sound is terrible. Well, at least I can roll my eyes.

If people want to look at the CALL-SLT files, they're at http://callslt.cvs.sourceforge.net/viewvc/callslt/CALL-SLT/

Adding Romance to Regulus (part 2)

2009-08-04T00:34:00.000-07:00

I've just checked in initial French files under CALL-SLT/Fre. Update CALLSLT with the -d flag to get them.

So far, the CALL-SLT files are pretty much the same as the MedSLT files they were adapted from. I'm assuming Pierrette will make the necessary changes! I checked that you can get as far as building a Nuance grammar... it all worked fine for me, but let me know if there are problems.

Adding Romance to Regulus

2009-08-03T22:56:00.000-07:00

Following discussions with Pierrette, I've now moved the shared Romance grammar from MedSLT/Rom to the new directory Regulus/Grammar/Romance. I've also moved the domain-independent French grammar and lexicon files to Regulus/Grammar/Romance/French. This mirrors the directory structure in Jen's Germanic grammar.

I've adjusted the MedSLT config files for Fre, Spa and Cat to point to the new files. All three languages still appear to build correctly in the AFF (role-marked semantics) versions. I have not done anything with the old (linear semantics) versions, which I am now assuming are obsolete.

All three languages appeared to build fine when I tested, but Pierrette will probably want to check things more carefully. If there are problems, please me know - they should be easy to fix.

The payoff is that I will now be able to construct a config file for the French version of CALL-SLT. Coming up next.

Treebank caching and preferences

2009-07-26T07:42:00.000-07:00

Pierrette reminded me last week that there was a known problem in treebank caching: when parse preferences change, cached analyses may no longer be valid.

I've just checked in new code which stores the preferences used to create the treebank, and compares them with the current preferences. If the two are different, the treebank is regenerated. It would be nice if we could only regenerate the part that might be affected by the changed preferences, but that's unfortunately very difficult to do.

Regulus for DORIS, continued

2009-07-20T08:00:00.001-07:00

I did some more tweaking of the DORIS grammar, and it now covers about 90% of Patrick's corpus. The Nuance grammar it generates is nice and compact (less than 1000 rules), and many of the remaining coverage holes still look easy to solve. As we hoped, this seems to be a good domain for Regulus.

Regulus for DORIS

2009-07-20T00:03:00.000-07:00

When I was visiting Melbourne Uni last week, I talked with Patrick Ye about the possibility of using Regulus to provide speech recognition for the DORIS project, which, as he pointed out, is in fact quite similar to SHRD2; in both domains, the basic idea is to find things, pick them up, and move them around. I did indeed find it very easy to use the examples corpus and vocabulary that Patrick sent me to adapt the existing SHRD2 resources, and in just a few hours put together an initial Regulus grammar that could be compiled into a recogniser. The current version of the grammar covers a bit more than 80% of Patrick's corpus, and the recogniser can turn spoken sentences in Australian-accented English into either strings of words or scoped logical forms. For example, here's the representation it produces of "the red book is on the desk":


[[dcl,
  quant(def_sing, A, [[book,A],[color,A,red]],
        quant(def_sing, B,
              [[desk,B]],
              quant(exist,C,[[be_on_loc,C,A,B],[tense,C,present]],true)))]]

If people want to look at the details, the files are checked in at http://regulus.cvs.sourceforge.net/viewvc/regulus/Regulus/Examples/Doris/. The interesting ones are the lexicon, at http://regulus.cvs.sourceforge.net/viewvc/regulus/Regulus/Examples/Doris/Regulus/doris_lex.regulus, and the corpus, at http://regulus.cvs.sourceforge.net/viewvc/regulus/Regulus/Examples/Doris/corpora/doris_corpus.pl

First steps in Bridge system

2009-07-10T03:43:00.000-07:00

With help from Cathy Chua, our Bridge expert, I've been putting together a first version of a grammar for the Bridge domain this week. Cathy has been supplying vocabulary and a corpus of examples showing how to use it, and I've used that to build an initial lexicon. The specialized grammar derived from these resources now has a vocabulary of about 220 surface words. Here are some typical utterances it can already handle:

who has the queen of clubs
who bid one no trump
is two clubs a transfer
cover the ten with the jack
can you finesse in diamonds
can you make if spades are four one

We tried compiling a recognizer using the Australian English package, and, with Cathy's Australian voice, recognition is anecdotally quite good. (I have discovered that the difference between British English and Australian English is substantial). We will be adding more coverage over the weekend. Next week, I hope we'll be able to start thinking concretely about how to hook up the Regulus components with BASSINET, Leon Stirling's Bridge program, to produce a first cut at an end-to-end system that can respond to spoken questions and commands.

As we expected, the Bridge domain is quite a lot more more complex than anything we have tried so far in Regulus. It will definitely stretch us!

Update on SHRD2 paraphrasing

2009-06-09T17:40:00.000-07:00

I extended the paraphrasing capabilities for SHRD2, and we now get paraphrases for about 45% of the 180-ish examples in the current corpus. Beth Ann and I did a little experiment for the paper we presented at the SETQA-NLP workshop in Colorado last week. We each tried judging the corpus examples for correctness, using both the paraphrases and the underlying representations. In order to minimize learning effects, we permuted the order in which we did things.

Even though the "logic English" paraphrases seem very similar to the scoped logical representations, it in fact turns out that judging paraphrases is a lot faster. Even for me, knowing all the representations from having worked on them, judging paraphrases took 29 minutes, against 22 minutes for judging structures. For Beth Ann, who didn't know the datastructures previously, paraphrases were more than twice as quick. Part of the payoff comes from the fact that the paraphrase grammar acts as a filter; most ill-formed structures produce no paraphrase, hence don't need to be judged at all when paraphrases are used.

One person in the workshop audience said he was pleased to see a paper about software engineering which actually contained an experiment! Beth Ann was clearly right to insist that we do this, and work out the methodology.

First version of paraphrase grammar for SHRD2

2009-05-24T07:20:00.000-07:00

I've just added a first version of a paraphrase grammar for SHRD2, where it tries to realize the scoped LFs in "logic English". Text example below. It doesn't do much more than this yet, but now that I have the basic structure working I'm hoping that it will be easy to add coverage. Planning to continue with this later today!


>> pick up a big red block

      Old state: []
             LF: [[imp, 
                   form(imperative, 
                        [[pick_up,term(pro,you,[]),term(a,block,[[size,big],[color,red]]),up]])]]
 Intermediate 1: [[imp, 
                   scoping_unit([modal, imperative, 
                                 term(event_exists, A, 
                                      [[pick_up, A, term(pro,B,[[you,B]]), 
                                        term(a,C,[[block,C],[size,C,big],[color,C,red]]), up]])])]]
 Intermediate 2: [[imp, 
                   quant(pro, A, [[you,A]], 
                         quant(a, B, 
                               [[block,B],[size,B,big],[color,B,red]], 
                               imperative(quant(event_exists,C,[[pick_up,C,A,B,up]],true))))]]
 Intermediate 3: [[imp, 
                   quant(pro, A, [[you,A]], 
                         quant(a, B, 
                               [[block,B],[size,B,big],[color,B,red]], 
                               imperative(quant(event_exists,C,[[pick_up,C,A,B]],true))))]]
  Dialogue move: [[imp, 
                   quant(exist, A, [[you,A]], 
                         quant(exist, B, 
                               [[block,B],[size,B,big],[color,B,red]], 
                               imperative(quant(exist,C,[[pick_up,C,A,B]],true))))]]
     Paraphrase: COMMAND there is an A SUCH THAT you are A AND there is a B SUCH THAT B is a block AND B is big AND B is red AND MAKE IT TRUE THAT there is a C SUCH THAT C is that A picks up B
Abstract action: say(i_dont_understand,present)

Multiple processing stages in dialogue top-level

2009-05-21T15:52:00.000-07:00

When building dialogue applications, it's quite often the case that there are multiple stages in the process of converting the LF (the thing that comes out of the recognizer) into a dialogue move. I've just added some hooks so that you can pass back the intermediate levels of representation as an optional extra argument to lf_to_dialogue_move, and display them. This is useful for debugging.

Here's a text example from SHRD2:


>> is the large red block in the box

      Old state: []
             LF: [[ynq, 
                   form(present, 
                        [[be, term(the_sing,block,[[size,large],[color,red]]), 
                          [in_loc,term(the_sing,box,[])]]])]]
 Intermediate 1: [[ynq, 
                   scoping_unit(term(event_exists, A, 
                                     [[be, A, 
                                       term(the_sing, B, 
                                            [[block,B],[size,B,large],[color,B,red]]),
                                       [in_loc,term(the_sing,C,[[box,C]])]],
                                      [tense,A,present]]))]]
 Intermediate 2: [[ynq, 
                   quant(the_sing, A, [[block,A],[size,A,large],[color,A,red]], 
                         quant(the_sing, B, [[box,B]], 
                               quant(event_exists, C, 
                                     [[be,C,A,[in_loc,B]],[tense,C,present]], true)))]]
  Dialogue move: [[ynq, 
                   quant(the_sing, A, [[block,A],[size,A,big],[color,A,red]], 
                         quant(the_sing, B, [[box,B]], 
                               quant(event_exists,C,[[be_in_loc,C,A,B],[tense,C,present]],true)))]]
Abstract action: say(i_dont_understand,present)
Concrete action: tts(sorry, I don't understand)
      New state: []

Intermediate 1 is after addition of variables; Intermediate 2 is after scoping; and Dialogue move is after rewriting of lexical predicates. This last step is still very primitive.

"Pick up a big red block", revisited

2009-05-20T07:20:00.000-07:00

OK, I did some more work on SHRD2, and, as of a few minutes ago, I managed to speak a sentence and get it turned into a logic-like representation. Here's a slightly edited trace (the less interesting parts of the output have been removed). I did indeed say "pick up a big red block"!


>> RECOGNISE
(Take next loop input from live speech)

Recognised: recognition_succeeded([rec_result(62,pick up a big red block,...)

      Old state: []
             LF: [[imp, 
                   form(imperative, 
                        [[pick_up,term(pro,you,[]),term(a,block,[[size,big],[color,red]]),up]])]]
  Dialogue move: [[imp, 
                   quant(pro, A, [[you,A]], 
                         quant(a, B, 
                               [[block,B],[size,B,big],[color,B,red]], 
                               imperative(quant(event_exists,C,[[pick_up,C,A,B,up]],true))))]]
Abstract action: say(i_dont_understand,present)
Concrete action: tts(sorry, I don't understand)
      New state: []

Dialogue processing time: 0.01 seconds

Obviously, there's plenty more left to do before we have anything resembling a complete system. In particular, there is as yet no reference resolution or dialogue management, so it can't react to the commands and questions in any way. But, all things considered, I think we're making reasonable progress.

CAT and FEAT commands

2009-05-17T13:41:00.000-07:00

One of the good things about doing some grammar development on SHRD2 is that it suggests ideas for new development environment functionality. I got tired of consulting the grammar every time I needed to find out what features a category had, or what possible values a feature could take. So I added a couple of new commands called CAT and FEAT, which give you those pieces of information. Here's an example:


>> CAT p
(Display information for specified category)

Features for category "p": [def,obj_sem_n_type,postposition,sem,sem_p_type,sem_pp_type]

--- Performed command CAT p, time = 0.00 seconds

>> FEAT sem_p_type         
(Display information for specified feature)

Feature values for feature "sem_p_type": [[back,down,none,normal,off,onoff,over,up,updown]]

--- Performed command FEAT sem_p_type, time = 0.00 seconds

SHRD2

2009-05-17T07:43:00.000-07:00

I was talking with some people about Terry Winograd's book Understanding Natural Language, and how inspiring I found the SHRDLU system when I first read about it as an undergraduate. They asked me what the current equivalent would be. It bothered me that I couldn't come up with anything, and that no one really seemed to be building this kind of system any more.

Well... so why not do something about it? Stage 1 is to build a speech-enabled, and hopefully rather more robust reconstruction of SHRDLU, using Regulus. Some initial stuff is already checked in under $REGULUS/Examples/SHRD2. When we've satisfied ourselves that we can handle the Blocks World, Stage 2 will be to define a new and more ambitious domain - something which will hopefully demonstrate that there has in fact been significant progress since the 70s.

I'll be posting more about this soon. So far, SHRD2 has already turned up some important holes in the general English grammar, which I'm working on fixing.

Showing missing vocabulary in EBL_TREEBANK

2009-05-17T06:27:00.000-07:00

A trivial but rather useful little feature I just added: when you run EBL_TREEBANK, you now get missing vocabulary displayed for relevant sentences. I don't know why I didn't do this years ago. Here's an example. To get the new functionality, you just need to update Regulus from CVS, nothing needs to be remade.


>> EBL_TREEBANK
(Parse all sentences in current EBL training set into treebank form)

--- Read parsing history file (114 records) d:/cygwin/home/speech/regulus/examples/shrd2/generated/shrd2_parsing_history.pl
--- Incremental treebanking switched off, not trying to convert treebank

Parsing corpus data in d:/cygwin/home/speech/regulus/examples/shrd2/corpora/shrdlu_corpus.pl:
..
*** Parsing failed for: "find a block which is taller than the one you are holding and put it into the box", line 2
...
*** Parsing failed for: "is at least one of them narrower than the one which i told you to pick up", line 6
Words not in current vocabulary: [told]
.....
*** Parsing failed for: "will you please stack up both of the red blocks and either a green cube or a pyramid", line 12
Words not in current vocabulary: [either]
...

Parsing top-level constituents and treebank caching

2008-12-19T08:53:00.000-08:00

Since parsing with non-top constituents had a bad effect on efficiency, the LOAD command has gone back to loading the normal grammar, as it did before. If you want to be able to parse with non-top constituents, I have introduced a new command, LOAD_DEBUG, which loads an extended version of the grammar suitable for debugging. You are advised not to use this for creating specialised grammars.

The initial version of this functionality turned out to have a rather nasty bug, which is I think the thing that got Beth Ann yesterday... there was an incorrect interaction between LOAD_DEBUG, grammar caching and treebank caching. This meant that treebank creation sometimes incorrectly thought that the grammar rules had been changed when in fact they hadn't, and unnecessarily reparsed the training corpus. Beth Ann, please update from Regulus and see if this is now fixed!

Improvement to dynamic lexicon functionality

2008-10-28T23:02:00.000-07:00

I have just checked in some improvements to dynamic lexicons, which should considerably reduce the number of external files created at runtime. Hopefully this will improve recognition response times, but so far I don't have a non-trivial dynamic lexicon application to test on - so I would appreciate feedback from the Ford project. In particular, please let me know at once if anything appears to be broken. If necessary, you can reverse the change by reverting the file $REGULUS/Prolog/dynamic_lexicon.pl to the previous version.

Default parse preferences for specialised grammars

2008-10-25T23:56:00.000-07:00

Following a discussion with Pierrette, I have added default parse preferences for specialised grammars, based on the geometric mean of the rule frequencies as observed in the training corpus. This is what we have been doing for some time in generation. To get the new functionality, you need to update Regulus and remake the specialised grammar you are using. Most of the time, you shouldn't notice anything new, except that the rule frequencies will be displayed in the parse trees, as in the following example:


>> is it a sharp pain
(Parsing with left-corner parser)

Analysis time: 0.12 seconds

Return value: [(object=[adj,sharp]), (agent=[pronoun,it]), (object=[secondary_symptom,pain]),
               (null=[tense,present]), (null=[utterance_type,ynq]), (null=[verb,be]),
               (null=[voice,active])]

Global value: []

Syn features: []

Parse tree:

.MAIN (freq 836) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:2629-3470]
   top (freq 830) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:3471-4306]
      utterance (freq 622) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:4307-4934]
         s (freq 31) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:11277-11313]
         /  vbar (freq 461) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:5947-6413]
         |  /  v lex(is) (freq 39) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:10819-10863]
         |  |  np (freq 314) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:7213-7532]
         |  \     pronoun lex(it) (freq 53) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:10336-10394]
         |  tmp_cat_12 (freq 31) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:11314-11317]
         |  /  np (freq 1153) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:1622-2628]
         |  |  /  np (freq 63) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:9998-10066]
         |  |  |  /  d lex(a) (freq 86) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:9110-9201]
         |  |  |  |  tmp_cat_6 (freq 63) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:10067-10070]
         |  |  |  |  /  adj lex(sharp) (freq 6) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:14728-14739]
         |  |  |  \  \  n lex(pain) (freq 389) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:6818-7212]
         |  |  \  post_mods null (freq 1399) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:615-1621]
         \  \  post_mods null (freq 1399) [MED_ROLE_MARKED_SPECIALISED_DEFAULT:615-1621]

------------------------------- FILES -------------------------------

MED_ROLE_MARKED_SPECIALISED_DEFAULT: c:/cygwin/home/speech/speechtranslation/medslt2/eng/generatedfiles/med_role_marked_specialised_default.regulus

Preference information:

1.80  Rule frequency score
Total preference score: 1.80

The bad news: I was hoping this would solve an annoying problem in Eng/Spa bidirectional. Unfortunately, it doesn't seem to do that. No idea why this used to work, in fact!

Parsing non-top constituents (continued)

2008-10-12T18:47:00.000-07:00

I have now checked in an improved version of the functionality for parsing non-top constituents, which hides the dummy rules and shows the features for the constituent. Here are a couple of examples from Toy1Specialised:


>> np the light in the kitchen
(Parsing with left-corner parser)

Analysis time: 0.55 seconds

Return value: [[device,light],[location,kitchen],[prep,in_loc],[spec,the_sing]]

Global value: []

Syn features: [agr=3/\sing,case=A,conj=n,def=y,gapsin=B,gapsout=B,n_appositive_mod_type=none,
n_of_mod_type=none,nform=normal,pronoun=n,sem_n_type=dimmable\/switchable,
syn_type=np_with_noun,takes_about_pp=n,takes_attrib_pp=n,takes_cost_pp=n,
takes_date_pp=n,takes_duration_pp=n,takes_frequency_pp=n,takes_from_pp=n,
takes_loc_pp=n,takes_partitive=n,takes_passive_by_pp=none,takes_post_mods=n,
takes_side_pp=n,takes_time_pp=n,takes_to_pp=n,takes_with_pp=n,wh=n]

Parse tree:

np [GENERAL_ENG:2026-2044]
/  np [GENERAL_ENG:1864-1874]
|  /  d lex(the) [GEN_ENG_LEX:341-344]
|  |  nbar [GENERAL_ENG:2071-2083]
|  \     n lex(light) [TOY1_LEX:44-47]
|  post_mods [GENERAL_ENG:1591-1680]
|  /  pp [GENERAL_ENG:1747-1765]
|  |  /  p lex(in) [TOY1_LEX:51-58]
|  |  |  np [GENERAL_ENG:2026-2044]
|  |  |  /  np [GENERAL_ENG:1864-1874]
|  |  |  |  /  d lex(the) [GEN_ENG_LEX:341-344]
|  |  |  |  |  nbar [GENERAL_ENG:2071-2083]
|  |  |  |  \     n lex(kitchen) [TOY1_LEX:38-39]
|  |  \  \  post_mods null [GENERAL_ENG:1410-1416]
\  \  post_mods null [GENERAL_ENG:1410-1416]

------------------------------- FILES -------------------------------
GENERAL_ENG: c:/cygwin/home/speech/regulus/grammar/general_eng.regulus
GEN_ENG_LEX: c:/cygwin/home/speech/regulus/grammar/gen_eng_lex.regulus
TOY1_LEX:    c:/cygwin/home/speech/regulus/examples/toy1specialised/regulus/toy1_lex.regulus

>> n light
(Parsing with left-corner parser)

Analysis time: 0.02 seconds

Return value: [[device,light]]

Global value: []

Syn features: [agr=3/\sing,conj=n,n_appositive_mod_type=none,n_of_mod_type=none,
n_post_mod_type=none,n_pre_mod_type=loc,sem_n_type=dimmable\/switchable,
takes_about_pp=n,takes_attrib_pp=n,takes_cost_pp=n,takes_date_pp=n,takes_det_type=def,
takes_duration_pp=n,takes_frequency_pp=n,takes_from_pp=n,takes_loc_pp=y,
takes_partitive=n,takes_passive_by_pp=none,takes_side_pp=n,takes_time_pp=n,
takes_to_pp=n,takes_with_pp=n]

Parse tree:

n lex(light) [TOY1_LEX:44-47]

------------------------------- FILES -------------------------------

TOY1_LEX: c:/cygwin/home/speech/regulus/examples/toy1specialised/regulus/toy1_lex.regulus

Parsing non-top constituents

2008-10-09T18:34:00.000-07:00

Following a conversation with Pierrette last week, I realised that there was an easy way to fix things so that we can parse non-top constituents in the LC (normal) parser, as well as the DCG one. I have just checked in a first version of the new functionality. Now, when you load a grammar using the LOAD command, an extra file of dummy rules is created and added to the ones explicitly specified. There is one dummy rule for each category Cat in the grammar, of the form


dummy_top:[sem=Sem] --> Cat, Cat:[sem=Sem]

For example, the dummy rule for 'np' is


dummy_top:[sem=Sem] --> np, np:[sem=Sem]

What this means is that you can now parse NPs at top-level by simply prefacing them with the word 'np'. Thus for instance in Calendar we can do things like the following:


>> np the last meeting in geneva
(Parsing with left-corner parser)

Analysis time: 0.97 seconds

Return value: [[at_loc,[[spec,name],[head,geneva]]],[head,meeting],[spec,the_last]]

Global value: []

Syn features: []

Parse tree:

.MAIN [CALENDAR_DUMMY_TOP_LEVEL_RULES:1-1]
   dummy_top [CALENDAR_DUMMY_TOP_LEVEL_RULES:22-22]
   /  lex(np)
   |  np [GENERAL_ENG:2026-2044]
   |  /  np [GENERAL_ENG:1849-1863]
   |  |  /  d lex(the) lex(last) [GEN_ENG_LEX:355-355]
   |  |  |  nbar [GENERAL_ENG:2071-2083]
   |  |  \     n lex(meeting) [CALENDAR_LEX:88-89]
   |  |  post_mods [GENERAL_ENG:1591-1680]
   |  |  /  pp [GENERAL_ENG:1747-1765]
   |  |  |  /  p lex(in) [CALENDAR_LEX:151-151]
   |  |  |  |  np [GENERAL_ENG:1955-1963]
   |  |  |  \     name lex(geneva) [GENERATED_NAMES:41-41]
   \  \  \  post_mods null [GENERAL_ENG:1410-1416]

------------------------------- FILES -------------------------------

CALENDAR_DUMMY_TOP_LEVEL_RULES: c:/cygwin/home/speech/regulus/examples/calendar/generated/calendar_dummy_top_level_rules.regulus
CALENDAR_LEX:                   c:/cygwin/home/speech/regulus/examples/calendar/regulus/calendar_lex.regulus
GENERAL_ENG:                    c:/cygwin/home/speech/regulus/grammar/general_eng.regulus
GENERATED_NAMES:                c:/cygwin/home/speech/regulus/examples/calendar.regulus
GEN_ENG_LEX:                    c:/cygwin/home/speech/regulus/grammar/gen_eng_lex.regulus

Semantic triples: []

No preferences apply

I should be able to improve this a little, in particular by adding some functionality to display the features on the non-top constituent as well as the semantics, but hopefully the existing version will already be quite useful.

Dynamic Regulus lexicon entries

2008-09-18T02:06:00.000-07:00

Regulus now includes an interface to Nuance dynamic grammar capabilities, making it possible in effect to add new lexicon entries at runtime. Dynamic lexicon entries need to be defined using macros which have been declared dynamic in the Regulus source file.

I have checked in a sample application in $REGULUS/Examples/Toy1SpecialisedDynamic; there is basic documentation in doc/README.txt. The application uses a version of the Toy1Specialised grammar in which commands need to be prefaced by a name. The user can dynamically add new names to the recognition vocabulary while the application is running. The following extract from the lexicon file shows the macro and declaration for the dynamic name entries:


macro(person_name(Surface, Sem),
      @name(Surface, [Sem], [agent], sing, [])).

dynamic_lexicon( @person_name(Surface, Sem) ).

At runtime, new name entries can be added using calls to the predicate assert_dynamic_lex_entry/1. A typical call might look like this:


assert_dynamic_lex_entry( @person_name((howard, the, duck), howard_the_duck))

Note that the infrastructure needed to run dynamic applications is somewhat different from the standard one. In particular, it is necessary to use a Resource Manager and a Compilation Server,and compile a dummy "just-in-time" recognition package. The sample application gives examples of the scripts required. I will be checking in proper documentation soon, andwill post again when I have done that.

Incremental treebanking for grammar specialisation

2008-09-11T01:13:00.000-07:00

I have just checked in some new code, which should make the process of creating a specialised grammar much more efficient. The most time-consuming part of the process is parsing the treebank, using the EBL_TREEBANK command, or commands like EBL_ANALYSIS which call it indirectly. Until now, the whole set of training sentences had to be parsed every time. This was wasteful, since the greater part of the parses in the existing treebank were often still valid.

The new functionality improves the picture by trying to determine which parses can be kept, and only reparsing the remaining ones. The current rules for determining which new parses are required are as follows:

After each invocation of EBL_TREEBANK, Regulus saves both the treebank and a copy of the grammar used to create it. The next time EBL_TREEBANK is called, the system compares the saved grammar and treebank with the current grammar and training corpus.
The grammar comparison determines two things: 1) Have any non-lexical rules changed? 2) If only lexical rules have changed, which lexical items are affected?
If non-lexical rules have changed, the whole treebank needs to be reparsed. Most often, however, this is not the case. If no rules, or only lexical rules, have changed, the treebank is incrementally updated as follows.
Any items in the treebank which correspond to sentences no longer in the current training corpus are removed.
Any items in the current training corpus which do not occur in the old treebank are parsed and added to the new treebank.
Any items in the treebank which include changed lexical items are reparsed and added to the new treebank.
All remaining items in the old treebank are kept.

You need to update Regulus to get the new functionality. Note that nothing will happen the first time you do EBL_TREEBANK after the update, since the old copy of the grammar is saved after EBL_TREEBANK is invoked, and you will not originally have an old saved grammar. So you will only notice a difference the second time you do EBL_TREEBANK.

I have done some testing, and things appear OK, but I know from experience that this kind of non-monotonic code often contains subtle bugs which aren't immediately apparent. Please let me know if things don't work as expected, and I will give priority to sorting out problems. If necessary, you can toggle the incremental treebanking functionality using the new commands INCREMENTAL_TREEBANKING_OFF and INCREMENTAL_TREEBANKING_ON. By default, incremental treebanking is on.

Documentation for NUANCE_PARSER command

2008-07-24T04:40:00.000-07:00

I've added some basic documentation for the new NUANCE_PARSER command. Here's what you get when you access it using DOC:


>> DOC NUANCE_PARSER
(Print documentation for command or config file entry)

NUANCE_PARSER
[Brief doc: Start new Nuance nl-tool process and use it as parser]

Start an nl-tool process, and use it to do parsing. Any old nl-tool processes
are first killed. The current config file needs to include either a
dialogue_rec_params declaration (for dialogue apps) or a translation_rec_params
declaration (for speech translation apps); the declaration must
contain definitions for 'package' and 'grammar'. The following is a
typical example of a suitable declaration:

regulus_config(dialogue_rec_params,
               [package=calendar_runtime(recogniser), grammar='.MAIN',
                'rec.Pruning=1600', 'rec.DoNBest=TRUE', 'rec.NumNBest=6']).

Notes:

-  After NUANCE_PARSER is successfully invoked, nl-tool is used for
ALL parsing, including batch processing with commands like TRANSLATE_CORPUS
and Prolog calls to parse_with_current_parser/6.
-  The Nuance parser only returns logical forms, not parse trees.

Improvements to Regulus documentation

2008-07-16T07:11:00.000-07:00

I have been doing some work on and off over the last few weeks to try and improve the Regulus documentation. It's one of those "important non-urgent" tasks that is very hard to schedule, because you always feel you have something that should take higher priority, but I do finally seem to have made some concrete progress.

There are three parts to the work, which are meant to be closely interlinked. First, I have created a directory under Regulus/doc called CommandDoc, which is supposed to contain one short file for each command and type of config file entry. I've so far populated it with the information in RegulusDoc.html, which certainly confirmed that RegulusDoc is badly out of date... I'm afraid half the files are currently empty.

Second, I have added a new top-level Regulus command called DOC. If you type DOC followed by the name of a command, you get the CommandDoc file printed out in a reasonably readable way. For example:


>> DOC LOAD_DIALOGUE
(Print documentation for command)

LOAD_DIALOGUE
[Brief doc: Load dialogue-related files]
Compile the files defined by the dialogue_files config file entry.

>> DOC EBL_NUANCE
(Print documentation for command)

EBL_NUANCE
[Brief doc: Compile current specialised Regulus grammar into Nuance GSL form]
Compile current specialised Regulus grammar into Nuance GSL form. Same
as the NUANCE command, but for the specialised grammar. The input is
the file created by the EBL_POSTPROCESS command; the output Nuance GSL
grammar is placed in the file defined by the ebl_nuance_grammar config
file entry.

>> DOC TRANSLATE_CORPUS
(Print documentation for command)

TRANSLATE_CORPUS
[Brief doc: Process text translation corpus]

Process the default text mode translation corpus, defined by the
translation_corpus config file entry. The output file, defined by
the translation_corpus_results config file entry, contains
question marks for translations that have not yet been judged. If
these are replaced by valid judgements, currently 'good', 'ok' or
'bad', the new judgements can be incorporated into the translation
judgements file (defined by the translation_corpus_judgements
config file entry) using the command
UPDATE_TRANSLATION_JUDGEMENTS.

TRANSLATE_CORPUS <Arg>
[Brief doc: Process text translation corpus with specified ID]

Parameterised version of TRANSLATE_CORPUS. Process the text mode
translation corpus with ID <Arg>, defined by the
parameterised config file entry
translation_corpus(<Arg>). The output file, defined
by the parameterised config file entry
translation_corpus_results(<Arg>), contains
question marks for translations that have not yet been judged. If
these are replaced by valid judgements, currently 'good', 'ok' or
'bad', the new judgements can be incorporated into the translation
judgements file (defined by the translation_corpus_judgements
config file entry) using the parameterised command
UPDATE_TRANSLATION_JUDGEMENTS <Arg>.

Third, and last, I have also arrange things so that the doc files are automatically included in the new Cookbook. This is still just a skeleton, but my plan is to start by completing the command and config-file section, so that the book will immediately be useful for something, and then work outwards from there. The PDF version is checked in as Regulus/doc/Cookbook/draft_cookbook.pdf.

Substitutable help classes

2008-07-13T11:18:00.000-07:00

Following a discussion with Nikos last month, I've added a new feature to the help system, so that help examples can be modified to be closer to the recognition result. Recall that the help system assumes that the designer will have declared a set of help classes; each class C defines a set of phrases P(C). When choosing a help match, both the recognition result and the help examples are backed off so that, for each class C, phrases in P(C) are replaced by C.

The new functionality I've just added makes it possible to declare some help classes as "substitutable". Suppose that class C is defined as substitutable, that the phrase P1 in the recognition result is backed off to C, and that the phrase P2 in a matched help example H is also backed off to C. In this case, H will not be presented in its original form, but with P2 substituted by P1. Evidently, not all help classes can be defined as substitutable, since it's essential that all the words in a substitutable class have exactly the same syntactic properties.

There are however some important classes which can in general be made substitutable, in particular (at least in English) names for specific types of individual, plural numbers, days of the week and months of the year. I've tested the new functionality on the Calendar app, and it does indeed seem to give considerably more useful responses. Here's an example. Without substitutable classes, the sentence "what meetings has nikos been to" gets the help responses


#1 : "what meetings has pierrette attended in geneva"
     "what meeting_noun has person_name attend_verb preposition loc_name" (backed off)
#2 : "which meetings has elisabeth attended"
     "which meeting_noun has person_name attend_verb" (backed off)
#3 : "what meetings is pierrette going to attend in geneva"
     "what meeting_noun is person_name going to attend_verb preposition loc_name" (backed off)
#4 : "what meetings is pierrette going to attend"
     "what meeting_noun is person_name going to attend_verb" (backed off)
#5 : "what meetings have there been in geneva"
     "what meeting_noun have there been preposition loc_name" (backed off)

With substitutable classes (person_name is one of them), the response is


#1 : "what meetings has nikos attended in geneva"
     "what meeting_noun has person_name attend_verb preposition loc_name" (backed off)
#2 : "which meetings has nikos attended"
     "which meeting_noun has person_name attend_verb" (backed off)
#3 : "what meetings is nikos going to attend in geneva"
     "what meeting_noun is person_name going to attend_verb preposition loc_name" (backed off)
#4 : "what meetings is nikos going to attend"
     "what meeting_noun is person_name going to attend_verb" (backed off)
#5 : "what meetings will nikos attend in geneva"
     "what meeting_noun will person_name attend_verb preposition loc_name" (backed off)

I haven't done any systematic testing, but anecdotally I'm pretty sure that this seems to make help more responsive.

MedSLT almost fully converted to AFF

2008-07-11T08:26:00.000-07:00

I've made a lot of progress this week on converting the bidirectional English/Spanish version of MedSLT to AFF format. I've parameterized the Spanish system to support AFF representations, and added AFF versions of most of the necessary config file and scripts. In particular,

You can build a full AFF version of Spa by doing 'make role_marked' in Spa/scripts.
You can run interactive bidirectional AFF Eng/Spa text systems using the files load_bidirectional_role_marked.pl and load_bidirectional_restricted_role_marked.pl in EngSpa/scripts.
There are targets in EngSpa/scripts/Makefile for running AFF versions of the QA corpus, both plain and restricted, with the obvious naming conventions.

I've done preliminary testing, and everything should be checked in. There are still things missing, e.g. nothing so far in Spa/Spa for checking back-translation, but I figured it would be best at this point to hand over to Pierrette, so that she can refine the rules. I've only made some absolutely minimal changes, enough to check that a few sentences go through.

When this piece of work is finished, all of MedSLT should be available in AFF format, which means that we'll be able to retire the old linear version and only support one system. I'm hopeful that build times will then be low enough for us to go back to building and testing the system every night, as we used to do.

N-best rescoring again

2008-07-08T09:13:00.000-07:00

[Updated July 9]

I've just checked in new code that makes it possible to create training material for doing N-best rescoring on speech translation applications - the functionality is basically the same as what we already had for dialogue applications, but there were a number of details that had to be fixed. It seems that the potential for improving performance using N-best rescoring varies considerably between apps. So far, we've looked at the following cases:

Calendar: can already almost halve error rate using rescoring, more should be possible.
Ford app: almost no potential for improvement.
Paideia app: considerable potential for improvement (don't currently have figures)
English MedSLT: maximum possible improvement looks like about 10% relative.
French MedSLT: maximum possible improvement about 15-20% relative.
Japanese MedSLT: almost no potential for improvement.

The variation in behavior between the different apps is quite surprising. In particular, I don't yet have a good explanation for why the MedSLT languages should be so different.

McNemar at the word level?

2008-06-24T12:59:00.000-07:00

I was thinking about our paper for GoTAL, and one thing that's bothered me a little is that we did all the significance testing using SER - the reason was that it's easy to run a McNemar test. However, we got rather bigger improvements in WER, which is really what you would expect from SLMs.

It seems to me though that you should also be able to do McNemar at the word level. You look at each word in the transcription, and then check each of the two hypotheses you're comparing to see whether they include it. This is a little coarse-grained (you treat each sentence as a bag of words), but I'd guess it would still give interesting results. Shouldn't be at all hard to implement either. If we do an expanded version of the GoTAL paper, I'd definitely like to try this.

In fact, this idea is so obvious that either it's wrong, or someone must already have thought of it. Any idea which?

PS Jun 26. Beth Ann pointed out that the proposal as originally formulated only covered deletions, but it's trivial to extend it to do insertions too. More seriously, she wondered if the significance results would always be reliable, given that there may be subtle dependencies. I am really not sure about this, but one way to investigate the idea empirically would be to generate large sets of simulated recognition results using a stochastic process, and look at the distributions. For example, if you generate 10000 simulated recognition runs, then take one run and find all the other runs that come out as different from it at P < 0.01 according to the new statistic, you'd be reassured to find there were not more than 100 of them. A lot more, and something is presumably wrong. A lot less presumably just shows the test isn't very sensitive.

Faster parsing in Regulus using Nuance

2008-06-24T12:52:00.000-07:00

Here's something I've been meaning to do for a while, that really should be moved up the priority stack. It should be quite easy to arrange things so that, in cases where we have compiled a grammar down to Nuance form, we use Nuance to do parsing - this ought to be much faster than the Regulus parser, and could really let us speed up corpus runs. There are at least two straightforward ways to implement it. One is to start an nl-tool process and pipe sentences into it, reading the analyses that come back. It may be even simpler to use the Regserver, now that we can connect to it from the Regulus top-level, and send an "interpret" message. More about this soon, I hope.

PS Jun 26. It was indeed very easy - I took the route of creating an nl-tool process and connecting to it with pipes. The new NUANCE_PARSER command now lets you use nl-tool as the parser. Parsing times are at least 30 times faster. Things should be checked in. More about this soon.

"Paraphrase corpora" for estimating semantic error rates

2008-06-17T16:38:00.000-07:00

I've implemented a first cut at the "paraphrase corpus" idea that I suggested in yesterday's post. So far, it only works for speech translation, but it's rather nice to see that we can now measure the effect that N-best rescoring has on semantic error rate in a way that's both much quicker and much more objective than what we were doing previously. On the whole of the Eng corpus (the only one I've tried so far), semantic error rate on this metric is reduced by N-best rescoring by about 4% absolute, or 8% relative.

My next task here is to extend the method to dialogue processing - this should be easy, I think. We will then be able to do dialogue N-best rescoring experiments using out-of-coverage as well as in-coverage data, which should open up several new possibilities.

Better ways to estimate semantic error rate

2008-06-16T13:39:00.000-07:00

I've just added some code to automatically estimate semantic error rate for translation applications. It does more or less the same thing as the code we've had for a while in dialogue apps, and counts an example from a speech corpus as semantically correct if it produces the same interlingua as the transcription would have done.

Unfortunately, the problem with this definition is that it doesn't work for utterances that are in domain, but out of grammar coverage. For example, I was just looking though the results for the English MedSLT corpus. In one example, the transcription is "does the pain ache", which is out of grammar coverage. The first hypothesis which produces well-formed interlingua is "does the pain feel aching", which is a good paraphrase and is selected. So this should really be counted as semantically correct, but isn't.

I think we can address the problem by allowing the developer to declare a file of paraphrases, and say that the example is semantically correct if it gives the same result as either the actual transcription or one of its paraphrases. Then if the developer adds in-coverage paraphrases where they exist, things will work correctly. This should be easy to implement. Probably we want a warning if a paraphrase in fact is also determined to be out of coverage.

This paraphrase functionality should also be useful for the N-best rescoring work that Maria and I have been doing for dialogue apps. We have the same problem there - we want to be able to experiment with out of coverage examples, but currently get no figures.

Interlingua corpora for multiple domains

2008-06-11T08:16:00.000-07:00

Following a discussion with Pierrette last week, I have added two more MedSLT Interlingua corpora, for the chest pain and abdominal pain domains. I've also added all the associated config files, scripts etc for the currently relevant language pairs (EngInt, JapInt, IntEng, IntFre and IntJap), so it should now possible to do systematic interlingua-centered development for all three domains. I have only built AFF versions, since we're planning to retire the linear formalism soon.

The naming conventions are the usual ones. I hopefully managed to check everything in, but let me know if files that you expected to find are missing. Pierrette should at some point do some work tidying up IntFre and FreInt and Yukie should do the same for IntJap and JapInt. Further down the line, we should really add coverage for these domains in the missing languages.

AFF version of Catalan

2008-06-10T01:40:00.000-07:00

I've added initial versions of all the files needed for the AFF version of Catalan in MedSLT. Naming conventions are the usual ones, and I was able to build all the AFF Cat resources by doing

make role_marked

in the Cat/scripts directory. There should now be config files for all 5 x 5 = 25 pairs of languages in {Ara, Cat, Eng, Fre, Jap} - this involved adding a few new pairs. I only tested Interlingua to Catalan and Catalan to Interlingua. We get currently translations for about 75% of the sentences in IntCat, and about 20% in CatInt. Hopefully it will be easy to improve these figures.

Over to Pierrette and Bruna to debug the rules. Note that I have macrotised the Cat lexicon to make the AFF version work. It should be mostly OK, but there were a few cases (in particular, WH+ PPs) where I wasn't quite sure how to do the macrotisation - people who actually know Catalan should review the entries.

Regulus 2.9.0 released

2008-06-04T06:41:00.000-07:00

Nikos has just created and uploaded the new 2.9.0 release of Regulus. I tried downloading and running a couple of simple tests in text and speech mode (under SICStus 4.0.3), and Toy1 at least appears to work fine. Please mail me if you notice problems.

Here are the release notes:


MAIN CHANGES TO REGULUS BETWEEN 2.8.0 AND 2.9.0

A large number of new features have been added to Regulus since
version 2.8.0. Most importantly, Regulus now runs under Sicstus 4;
it is possible to use speech input directly from the top-level;
N-best processing is supported in both dialogue and translation mode;
and a new semantics for translation applications has been added.

The new features are listed below in more detail. Not all of them are
fully documented yet, but we are giving priority to adding the
necessary documentation.

- Support for Sicstus 4
  - Regulus runs under Sicstus 4.
    - It has been thoroughly tested under 4.0.2.
    - Some testing has been done under 4.0.3, but this has not yet been carefully
      verified. NOTE: under 4.0.3, it is necessary to load the patch files in
      Prolog/SicstusPatches/4.0.3
  - Regulus still runs under Sicstus 3, and has been thoroughly
    tested under 3.12.5.

- Top-level
  - Errors are now written to stderr
  - There is a version of regulus_batch with an extra argument, which returns
    the list of error outputs created when running the commands.
  - It is possible to compile Nuance grammars from the Regulus top-level
    using the NUANCE_COMPILE command.
  - It is possible to perform speech recognition directly from the top-level
    - The LOAD_RECOGNITION command starts defined speech resources, including
      a license manager, recserver and Regserver
    - After loading resources using LOAD_RECOGNITION, the RECOGNISE command
      takes live speech input and passes it to the current application.
    - Wavfiles are automatically logged by RECOGNISE. The WAVFILES  command
      lists the  most recent recorded wavfiles.
    - When speech resources are loaded, text input of the form

      WAVFILE: 

      performs recognition on , and passes the result to the
      current application

- Java GUI
  - The Java GUI has been greatly improved, and many bugs have been fixed.
  - The GUI supports direct speech input, similar to the Prolog top-level
    described above
  - It is possible to run multiple copies of the GUI at the same time.

- Stepper
  - The commands LOAD, LOAD_GENERATION, EBL_LOAD and EBL_LOAD_GENERATION
    can be invoked from within the stepper.

- Support for spoken dialogue applications
  - When speech resources have been loaded from the command line,
    dialogue corpora can contain items of the form wavfile().
    This makes it possible to test corpora containing a mixture of speech
    and non-speech inputs.
  - Batch processing of speech input in dialogue mode produces figures
    for semantic error rate. An utterance is deemed semantically correct if
    it produces the same dialogue move as the transcription would have done.
  - A timeout has been added in batch dialogue processing, so that processing
    gives up after 10 seconds.
  - If N-best preferences are defined, preference info is printed in
    dialogue mode.
  - Allow dialogue server to take XML-formatted requests

- Generation
  - When the declaration

    regulus_config(prolog_semantics, yes).

    is included, generation grammars can contain arbitrary Prolog structures.

- Translation
  - There is extensive support for translation using both the original
    "linear" semantics, and also the new "Almost Flat Functional" (AFF)
    semantics. AFF is described in our COLING 2008 paper, which will soon
    posted on the Regulus website. Some initial documentation will be added
    to RegulusDoc.htm.
  - It is possible in a translation config file to define an interlingua
    as either a source or a target language. There are many examples
    in the MedSLT project directory.
  - Batch translation produces output files for judging both in Prolog
    and in CSV form. There are new commands for updating judgements from the
    CSV files.
  - When speech resources have been loaded from the command line,
    translation corpora can contain items of the form wavfile().
  - A simple version of N-best processing has been added for applications
    that use interlingual translation with an interlingua grammar. In N-best mode,
    the first utterance producing well-formed interlingua is selected.
  - Interlingua expressions ambiguous according to the interlingua grammar
    are flagged in translation mode.
  - If performing batch translation from Source to Target through Interlingua,
    combine available Source -> Interlingua and Interlingua -> Target
    judgements into Source -> Target judgements if possible.
  - Show average number of generated target language surface forms when
    doing batch translation.
  - Translation conditions can include elements of the form

    context_below()

    This matches an  in a clause.

- Grammar specialisation
  - Fix bug in processing of include_lex declarations.

- Help
  - When defining intelligent help for translation applications, help resources
    can be built from an interlingua corpus.

- Extension to Regulus grammar formalism
  - Allow =@ as synonym for = @
  - Add runtime support for GSL functions strcat/2, add/2, sub/2, neg/1, mul/2, div/2

- English grammar
  - Rules for dates including years have been added.

- Other
  - Tool added to perform random generation from PCFG-trained GSL grammars

Problems with SICStus 4.0.3 resolved

2008-06-02T13:23:00.000-07:00

The SICStus people were as usual very responsive, and we now seem to be OK for running under 4.0.3. However, (this is IMPORTANT), you need to install a couple of patch files if you are using that version of SICStus. So, if you're using 4.0.3, do the following:

Update Regulus from CVS, using the -d option to get new directories.
Copy the files from Prolog/SicstusPatches/4.0.3 to C:/Program Files/SICStus Prolog 4.0.3/library, or wherever you have your copy of SICStus.

I will set my default version of SICStus to 4.0.3, which means I'll no doubt test it a fair amount over the next few days. I would not recommend people to switch over to 4.0.3 until I've run with it a while and reported on how it's working.

Problems with SICStus 4.0.3

2008-06-02T06:28:00.000-07:00

We are unfortunately still having problems with SICStus 4. Things have been more or less stable with 4.0.2, but there were a few rather ugly patches - the SICStus people said things would be better in the next version. Sad to tell, I have just downloaded 4.0.3 and tried it out, and in fact, at least as far as Regulus is concerned, it's gone backwards. Due to new incompatibilities in the operating system interface libraries, it's not currently possible to run Regulus in speech mode with 4.0.3 - there may also be other problems. I can presumably implement a workaround, but the idea of having to patch the code after every new SICStus release makes me very nervous.

For Prolog people who want the low-level details, here is part of the mail I just sent to the SICStus team:


Unless I am misunderstanding something important, SP4.0.3's version of the
system3 library is still not  downward-compatible with SP3's system library, and is in fact rather
less downward-compatible than SP4.0.2's system3. The problem is now in system/1.
In SP4.0.2, system/1 is defined as follows:

system(Cmd) :-
  system_binary(Binary, DashC),
  proc_call(Binary, DashC, Cmd, exit(0)).

so it's possible to make calls like the following, running under Cygwin:

| ?- system('dir > tmp_dir.txt').
      1      1 Call: system('dir > tmp_dir.txt') ?
      2      2 Call: system3:environ('COMSPEC',_790) ?
      2      2 Exit: system3:environ('COMSPEC','C:\\WINDOWS\\system32\\cmd.exe') ?
      3      2 Call: system3:process_create('C:\\WINDOWS\\system32\\cmd.exe',['/C','dir > tmp_dir.txt'],system3:[process(_1437)]) ?
      3      2 Exit: system3:process_create('C:\\WINDOWS\\system32\\cmd.exe',['/C','dir > tmp_dir.txt'],system3:[process('$process'('$ptr IEDNJP'))]) ?
      4      2 Call: system3:process_wait('$process'('$ptr IEDNJP'),exit(0)) ? s
      4      2 Exit: system3:process_wait('$process'('$ptr IEDNJP'),exit(0)) ?
      1      1 Exit: system('dir > tmp_dir.txt') ?

Under SP4.0.3, system/1 is defined thus:

system(Cmd, Status) :-
      shell_exec(Cmd, [], exit(Status)).

and the corresponding call looks like this:

| ?- system('dir > tmp_dir.txt').
      1      1 Call: system('dir > tmp_dir.txt') ?
      2      2 Call: system3:system('dir > tmp_dir.txt',0) ?
      3      3 Call: system3:process_create('dir > tmp_dir.txt',[],system3:[commandline(true),process(_1119)]) ?
      3      3 Exit: system3:process_create('dir > tmp_dir.txt',[],system3:[commandline(true),process('$process'('$ptr ALJLOO'))]) ?
      4      3 Call: system3:process_wait('$process'('$ptr ALJLOO'),exit(0)) ?
      4      3 Fail: system3:process_wait('$process'('$ptr ALJLOO'),exit(0)) ?
      2      2 Fail: system3:system('dir > tmp_dir.txt',0) ?
      1      1 Fail: system('dir > tmp_dir.txt') ?

The problem, as far as I can see, is that process_create requires the first arg
of process_create to be a program, which it isn't here.

Unfortunately, we have people running Regulus under at least 3.12.5, 4.0.2 and 4.0.3.
Maintaining the code so that it runs under all these different versions is
becoming quite difficult - the operating system interface primitives are
absolutely essential. Advice appreciated.

Interlingua corpora

2008-06-02T03:20:00.000-07:00

Over the last few months, we have been moving MedSLT development towards a new way of doing things, which is based on the idea of an "Interlingua corpus". We present the basic picture in our LREC 2008 paper, but that's already somewhat out of date, and doesn't give any low-level details.

We now have four interlingua corpora, representing the cross-produce of {linear, AFF} x {plain, combined}. The linear/AFF distinction is concerned with the type of semantics used. "Linear" is the old MedSLT semantics; AFF semantics is explained in the paper by Pierrette, Beth Ann, Yukie and myself which has just been accepted for COLING 2008, and which will soon be appearing on the Geneva website.

The plain/combined distinction says what information has been incorporated in the corpus. The "plain" corpus is created by merging results of translating FROM each source language into interlingua, so each interlingua form lists the source language results that translate into it. The "combined" corpus contains all the information in the "plain" corpus, plus also the results of translating TO each target language.

At the moment, we use the plain corpus for developing translations rules that go from Interlingua to target languages. The combined corpus is used for creating help resources.

All the scripts used to build interlingua corpora are referenced in $MED_SLT2/Interlingua/scripts/Makefile.

Running multiple copies of the GUI

2008-06-01T11:49:00.000-07:00

Elisabeth did a little work over the weekend, and it's now possible to run multiple copies of the GUI simultaneously - this is an important feature that people have been requesting for some time. The solution turns out to be embarrassingly simple. All we needed to do, in the end, was fix things so that it's possible for both the Java and the Prolog processes to specify from the command line which port they use to communicate with each other. As long as different {Java, Prolog} pairs use different ports, they don't interfere with each other.

I've added an example script to Regulus/Java called run_prolog_and_java2.bat - this is just like run_prolog_and_java.bat, but starts a second pair of processes, communicating over a new port.

Progress on N-best rescoring

2008-05-31T04:55:00.001-07:00

Maria Georgescul and I have been doing some work over the last few days on N-best rescoring, using the Calendar application as a test-bed. The basic division of labor was for me to define features and transform N-best hypothesis lists into lists of feature vectors, while Maria fed these into an SVM-based learner to perform the actual rescoring. We did the experiments using a set of 459 recorded utterances. Rescoring now reduces semantic error rate from 19% to 11%, and WER from 11% to 10%.

I defined the features by looking at examples of N-best lists, and finding common examples of things which I felt intuitively should be penalized. The current set of features is as follows:


rank: Place in the N-best list

no_dialogue_move: Hypothesis produces no dialogue move

underconstrained_query: Query with no contentful constraints

non_indefinite_existential: Existentials with non-indefinite arg, e.g. "is there the meeting next week"

non_show_imperative: Imperatives where the main verb isn't "show" or something similar

indefinite_meeting_and_meeting_referent: combination of indefinite mention of meeting + available meeting referent

Printing N-best feature info at top level

2008-05-28T10:33:00.000-07:00

If you're in dialogue mode, and have N-best preferences defined, you now get them printed out at top level. This is useful for debugging feature definitions. Here's an example from the Calendar application:


>> what was the last meeting

      Old state: [lf=[[whq,form(past,[[be,term(the_last,meeting,[]),[loc,where]]])]], 
                  referents=[record(meeting,meeting_10),attribute(meeting,meeting_10,where)]]
             LF: [[whq,form(past,[[be,term(the_last,meeting,[]),term(what,null,[])]])]]
    Resolved LF: [[whq,form(past,[[be,term(the_last,meeting,[]),term(what,null,[])]])]]
     Resolution: [trivial]
  Dialogue move: [tense_information=referent(past), utterance_type=whq, 
                  aggregate(last_n_meetings(1),[])]
  Resolved move: [tense_information=interval(datime(1980,0,0,0,0,0),datime(2008,5,28,18,27,24)),
                  utterance_type=whq, aggregate(last_n_meetings(1),[])]
     Paraphrase: list meetings in past the last meeting
Abstract action: say(referent_list([record(meeting,meeting_10)]))
Concrete action: tts(meeting at pierrette 's room on november 25)
      New state: [lf=[[whq,form(past,[[be,term(the_last,meeting,[]),term(what,null,[])]])]], 
                  referents=[attribute(meeting,meeting_10,where),record(meeting,meeting_10)]]

N-BEST FEATURES AND SCORES:

rank                                    -1.00 * 0.00 = 0.00
no_dialogue_move                        -50.00 * 0.00 = 0.00
underconstrained_query                  -10.00 * 0.00 = 0.00
inconsistent_tense                      -10.00 * 0.00 = 0.00
non_indefinite_existential              -10.00 * 0.00 = 0.00
non_show_imperative                     -50.00 * 0.00 = 0.00
definite_meeting_and_meeting_referent   3.00 * 0.00 = 0.00

Total score: 0.00

Dialogue processing time: 0.00 seconds

>> when did that meeting start

      Old state: [lf=[[whq,form(past,[[be,term(the_last,meeting,[]),term(what,null,[])]])]], 
                  referents=[attribute(meeting,meeting_10,where),record(meeting,meeting_10)]]
             LF: [[whq,form(past,[[start,term(that,meeting,[])],[time,when]])]]
    Resolved LF: [[whq,form(past,[[start,term(that,meeting,[])],[time,when]])]]
     Resolution: [trivial]
  Dialogue move: [query_object=start_time, referent_from_context=meeting, 
                  tense_information=referent(past), utterance_type=whq]
  Resolved move: [meeting=meeting_10, query_object=start_time, referent_from_context=meeting, 
                  tense_information=interval(datime(1980,0,0,0,0,0),datime(2008,5,28,18,27,35)),
                  utterance_type=whq]
     Paraphrase: start time for that meeting in past
Abstract action: say(referent_list([attribute(meeting,meeting_10,start_time)]))
Concrete action: tts(10 00 on november 25)
      New state: [lf=[[whq,form(past,[[start,term(that,meeting,[])],[time,when]])]], 
                  (referents
                   = 
                   [attribute(meeting,meeting_10,where), record(meeting,meeting_10), 
                    attribute(meeting,meeting_10,start_time)])]

N-BEST FEATURES AND SCORES:

rank                                    -1.00 * 0.00 = 0.00
no_dialogue_move                        -50.00 * 0.00 = 0.00
underconstrained_query                  -10.00 * 0.00 = 0.00
inconsistent_tense                      -10.00 * 0.00 = 0.00
non_indefinite_existential              -10.00 * 0.00 = 0.00
non_show_imperative                     -50.00 * 0.00 = 0.00
definite_meeting_and_meeting_referent   3.00 * 1.00 = 3.00

Total score: 3.00

Dialogue processing time: 0.01 seconds

Catching Regulus errors

2008-05-27T14:28:00.000-07:00

Peter Ljunglöf wondered whether error reporting in Regulus could be improved, and had a couple of suggestions. I've implemented and checked in the following improvements:

All error messages should now be printed to stderr.
When processing fails during execution of the Regulus command , a line of the form

Error processing command:

should be printed. This was not previously the case.
There is a new top-level predicate

regulus_batch_storing_errors(+ConfigFile, +Commands, -ErrorString)

which is like regulus_batch/2, except that it instantiates ErrorString with a string containing all the errors printed out during execution of Commands.

I expect there will be some glitches (I had to change a lot of lines of code), so please let me know if thing don't work as intended.

Building help resources from the combined interlingua corpus (2)

2008-05-27T11:42:00.000-07:00

Considerable progress on this task today:

I've added French to the AFF interlingua corpora, including the New York material as requested by Pierrette. The corpora are remade and checked in.
The help resources for Eng and Ara (the languages where we have help class definitions) are now made from the combined interlingua corpus. A separate help file is made for each of the six pairs EngAra, EngFre, EngJap, AraEng, AraFre, AraJap, reflecting the different levels of coverage. You can make the help resources for all of these pairs by doing 'make help_resources' in $MED_SLT2 (i.e. at the top level in the MedSLT directory), and it only takes a few minutes.

Building help resources from the combined interlingua corpus

2008-05-27T09:57:00.000-07:00

I've just checked in code that allows us to build Prolog help resources from the combined interlingua corpus in multi-lingual translation applications. This will make it much easier to integrate construction of help resources into the MedSLT build - it should now be almost trivial.

I'm currently remaking the interlingua corpus (I have had to change the format a little), and should be able to check in all the relevant MedSLT stuff later this evening.

Flagging ambiguity in interlingua checking

2008-05-27T06:25:00.000-07:00

I've just checked in code to catch cases where interlingua is
ambiguous, in the sense of generating multiple different surface
strings in the interlingua grammar. This is most likely to occur in
AFF, when the to-interlingua rules are underconstrained and the
interlingua is only partially instantiated. The following Japanese
-> Interlingua example in MedSLT illustrates:


>> doko ga itami masu ka

Source: doko ga itami masu ka
Target: WH-QUESTION pain be where PRESENT ACTIVE
Other info:
n_parses = 1
parse_time = 0.297
source_representation = [null=[path_proc,itamu], null=[tense,present],
                      null=[utterance_type,question], subject=[body_part,doko]]
source_discourse = [null=[utterance_type,question], subject=[body_part,doko],
                 null=[tense,present], null=[path_proc,itamu]]
resolved_source_discourse = [null=[utterance_type,question], subject=[body_part,doko],
                          null=[tense,present], null=[path_proc,itamu]]
resolution_processing = trivial
interlingua = [loc=[loc,where], arg1=[secondary_symptom,pain], null=[tense,present],
            null=[utterance_type,whq], null=[verb,be], null=[voice,active]]
interlingua_surface = WH-QUESTION pain be where PRESENT ACTIVE
other_interlingua_surface = [WH-QUESTION pain be above-loc where PRESENT ACTIVE,
                          WH-QUESTION pain be around-loc where PRESENT ACTIVE,
                          WH-QUESTION pain be between-loc where PRESENT ACTIVE,
                          WH-QUESTION pain be in-loc where PRESENT ACTIVE,
                          WH-QUESTION pain be under-loc where PRESENT ACTIVE]

Background

2008-05-27T04:14:00.001-07:00

If you've reached this blog and don't have any idea what it's about, Regulus is an Open Source platform for constructing speech-enabled systems, which we've been developing since 2001. We've now built several high-profile applications, including Clarissa, so far the only speech-enabled system to have flown in space, and MedSLT, a medical speech translator. You can read more about Regulus here

First entry

2008-05-27T03:56:00.000-07:00

Rather than mail people about new Regulus features, fixes, etc, I am starting a blog. Don't know why I didn't do this earlier!

Regulus News

EBL mode

Regulus with Nuance 9, continued

Regulus with Nuance 9

Screenshots from Japanese CALL-SLT

A Scandinavian/English grammar, continued

Scandinavian/English grammar now used for English MedSLT

Swedish MedSLT, continued

Swedish MedSLT, continued

Swedish MedSLT, continued

Swedish MedSLT

A Scandinavian/English grammar, continued

A Scandinavian/English grammar, continued

Swedish CALL-SLT

A Scandinavian/English grammar, continued

A Scandinavian/English grammar

New treatment of Japanese verbs

"Abstract actions" and the dialogue server

Warning: don't use SICStus 4.0.2

Dialogue server now accepts XML format messages

CALL-SLT and Japanese

Using recorded wavfiles as help information in CALL-SLT (part 2)

Using recorded wavfiles as help information in CALL-SLT

CALL-SLT and Japanese

CALL-SLT and French

Progress on CALL-SLT

Progress on CALL-SLT

Progress on CALL-SLT

Progress on CALL-SLT

CALL-SLT corpus

First CALL-SLT meeting

Initial English recognizer for CALL-SLT (Part 2)

Initial English recognizer for CALL-SLT

Initial French recognizer for CALL-SLT

Adding Romance to Regulus (part 2)

Adding Romance to Regulus

Treebank caching and preferences

Regulus for DORIS, continued

Regulus for DORIS

First steps in Bridge system

Update on SHRD2 paraphrasing

First version of paraphrase grammar for SHRD2

Multiple processing stages in dialogue top-level

"Pick up a big red block", revisited

CAT and FEAT commands

SHRD2

Showing missing vocabulary in EBL_TREEBANK

Parsing top-level constituents and treebank caching

Improvement to dynamic lexicon functionality

Default parse preferences for specialised grammars

Parsing non-top constituents (continued)

Parsing non-top constituents

Dynamic Regulus lexicon entries

Incremental treebanking for grammar specialisation

Documentation for NUANCE_PARSER command

More improvements to Nuance documentation

Improvements to Regulus documentation

Substitutable help classes

MedSLT almost fully converted to AFF

N-best rescoring again

McNemar at the word level?

Faster parsing in Regulus using Nuance

"Paraphrase corpora" for estimating semantic error rates

Better ways to estimate semantic error rate

Interlingua corpora for multiple domains

AFF version of Catalan

Regulus 2.9.0 released

Problems with SICStus 4.0.3 resolved

Problems with SICStus 4.0.3

Interlingua corpora

Running multiple copies of the GUI

Progress on N-best rescoring

Printing N-best feature info at top level

Catching Regulus errors

Building help resources from the combined interlingua corpus (2)

Building help resources from the combined interlingua corpus

Flagging ambiguity in interlingua checking

Background

First entry