This project is read-only.

How to switch syntax highlighting engine?

Nov 5, 2015 at 9:00 PM
Hi,

I have built a local madoko install under ubuntu 14.04 on top of a nodejs install from packages provided by nodejs 4.2.1 (not from the ones found in the ubuntu repos).

The tool works just fine, except when it comes to highlighting code blocks.

These warnings are produced when compiling the reference manual of madoko

user:~/bin/madoko/doc$ madoko reference.mdk
warning: unable to read language definition: Ruby
warning: unable to read language definition: JavaScript
warning: unable to read language definition: Haskell

as a result the code is not highlighted in madoko's html or pdf output.

It seems to be a problem with the internal highlighter named "monarch" located in

madoko/contrib/monarch in the source tree of madoko.

I cannot find any documentation or website about monarch online so I guess it must be some internal MS tool without public website.

Is there a way to fix these warnings?

I would also like to know how to specify new languages for monarch, for highlighting programs written in our in-house languages.

Alternatively is there a way to swap the syntax highlighting engine for another like highlight.js (a javascript lib) or pygmentize (a python lib)?

Regards,

/Rémi
Nov 5, 2015 at 9:45 PM
Hi Rémi,

Thank for trying out Madoko and building it yourself :-)

It seems that Madoko is not able to find the language definitions. They should be located in the lang directory under the styles directory which should be in the installation directory. Since the standard styles, like prelude.mdk and madoko.css come from the styles directory, I guess madoko can find that directory. Can you check whether the styles/lang directory exists in the install directory (and contains the language definitions?)

Also, can you run the madoko command with -vvvv to see if I print more information besides being unable to read the language definition?

Hope we can figure out what goes wrong :-)


btw. With regard to Monarch, there is no official documentation yet but I have it ready -- just need to publish it. In principle, Monarch should be open-source soon but I am wainting for legal to sign off on it. It is a very cool syntax highlighter that can describe many languages with full lexical grammars -- in fact, those definitions are also used inside the editor itself and will work with the new Visual Studio :-). For now, the reference manual shows how to easily extend existing languages, and you can look at the language definitions (which are in JSON) to learn a lot on how to make your own ones.
Nov 6, 2015 at 2:16 PM
Hi Daan, thank you for taking the time.

I tried to install the pre-built madoko using

sudo npm install -g madoko
sudo npm update -g

then I go into my local madoko checkout to compile the reference manual:

user@host:~/bin$ cd madoko/
user@host:~/bin/madoko$ cd doc/
user@host:~/bin/madoko/doc$ madoko reference.mdk
warning: unable to read language definition: Ruby
warning: unable to read language definition: JavaScript
warning: unable to read language definition: Haskell
warning: unable to read language definition: Ruby
warning: unable to read language definition: JavaScript
warning: unable to read language definition: Haskell
math rebuild


my global madoko install contains all the style definitions and are readable for all users

user@host:~/bin/madoko/doc$ ls -lR /usr/local/lib/node_modules/madoko/styles/lang/
/usr/local/lib/node_modules/madoko/styles/lang/:
total 124
-rw-r--r-- 1 nobody user 1516 oct. 21 2014 bibtex.json
-rw-r--r-- 1 nobody user 2002 juin 10 2014 boogie.json
-rw-r--r-- 1 nobody user 6813 mai 30 2014 codehunt.json
-rw-r--r-- 1 nobody user 4041 déc. 2 2014 cpp.json
-rw-r--r-- 1 nobody user 6338 mai 28 2014 csharp.json
-rw-r--r-- 1 nobody user 3332 oct. 25 2014 dafny.json
-rw-r--r-- 1 nobody user 2927 mai 28 2014 dot.json
-rw-r--r-- 1 nobody user 4632 juin 12 2014 haskell.json
-rw-r--r-- 1 nobody user 3544 mai 28 2014 html.json
-rw-r--r-- 1 nobody user 3487 déc. 13 2014 java.json
-rw-r--r-- 1 nobody user 4852 juin 10 2014 javascript.json
-rw-r--r-- 1 nobody user 7323 oct. 18 2014 koka.json
-rw-r--r-- 1 nobody user 3811 déc. 9 2014 latex.json
-rw-r--r-- 1 nobody user 13042 oct. 27 20:54 madoko.json
-rw-r--r-- 1 nobody user 5445 août 21 2014 markdown.json
-rw-r--r-- 1 nobody user 4868 mai 28 2014 python.json
-rw-r--r-- 1 nobody user 12660 juin 10 2014 ruby.json
-rw-r--r-- 1 nobody user 3479 mai 28 2014 smt.json

Attached is the log file for command

user@host:~/bin/madoko/doc$ madoko -vvvv reference.mdk 2>&1 | tee log


Thanks for you help,
/Rémi


Nov 6, 2015 at 4:14 PM
I found the problem -- I think :-) The Madoko driver searches for the language definitions as .../styles/lang/Ruby.json for example, which will work on Windows but not on a case-sensitive file system. Ouch. I will push out a new release later today that will do the right thing. Actually, if you read this before that, can you test if my hypothesis is right? ie. rename ruby.json to Ruby.json and see what happens.
Nov 6, 2015 at 4:40 PM
Hi Again !

I applied your recommendation and the behaviour is strange:

when language definition filenames are in lowercase on the filesystem, madoko warns about Capitalized style files

user@host:/usr/local/lib/node_modules/madoko/styles/lang$ ls
bibtex.json codehunt.json csharp.json dot.json html.json javascript.json latex.json markdown.json ruby.json
boogie.json cpp.json dafny.json haskell.json java.json koka.json madoko.json python.json smt.json

user@host:~/bin/madoko/doc$ madoko reference.mdk
warning: unable to read language definition: Ruby
warning: unable to read language definition: JavaScript
warning: unable to read language definition: Haskell


when I change the filenames from lowercase to capitalized on the filesystem, madoko warns about lowercase style files !?

user@host:/usr/local/lib/node_modules/madoko/styles/lang$ sudo mv haskell.json Haskell.json
user@host:/usr/local/lib/node_modules/madoko/styles/lang$ sudo mv ruby.json Ruby.json

user@host:~/bin/madoko/doc$ madoko reference.mdk
warning: unable to read language definition: ruby
warning: unable to read language definition: haskell
warning: unable to read language definition: JavaScript


And if I rename the javascript style file to capitalized I get a different warning:

user@host:/usr/local/lib/node_modules/madoko/styles/lang$ sudo mv javascript.json JavaScript.json

user@host:~/bin/madoko/doc$ madoko reference.mdk
warning: unable to read language definition: javascript
error: could not register language: javascript5
Error: a language definition must define the 'tokenizer' attribute as an object
warning: unable to read language definition: ruby
warning: unable to read language definition: haskell

the monarch library seems very interesting as well, looking forward for its release !

I was glad to see haskell and smt style files were built-in! I work with SMT-solvers a lot so it will help! I will try do define a style file for Scala since most of my dev is made in Scala and uses smt-solvers in some way.

I've also downloaded koka sources and am currently trying to compile it so that my checkout of madoko can compile as well. However it seems that madoko expects the koka binaries to be in some directory that the koka build does not define exactly the same (or I could not find the release target in the Jakefile).

Thank you for releasing such a great collection of tools that will certainly make my life easier (preparing documentation, papers, tutorials, lectures, etc.).

And just to reassure you, there is no hurry on my side at all, madoko already produces a very usable output.





Nov 6, 2015 at 11:46 PM
Thanks so much for your enthusiasm :-) And great to hear it proves to be so useful!

Anyway, I just pushed out a new Madoko release that should work (0.9.15-beta... almost at version 1.0 now :-) ).

Let me know
Nov 7, 2015 at 5:50 PM
Hi!

Madoko, version 0.9.15-beta works as expected now, no warnings and prettified code blocks :-)

Thank you !

/Rémi