Add the draft of post-proceeding.

[luatex-ja/luatexja.git] / doc / ajt-devel-ltja.tex

%#!lualatex ajt-devel-ltja
\documentclass{ajt}

%%% Packages used in this paper
%%% LuaTeX-ja

\usepackage{luatexja}
\DeclareFontShape{JY3}{mc}{m}{n}{<-> s*[0.92489] file:ipaexm.ttf:jfm=ujis}{}
\DeclareFontShape{JY3}{gt}{m}{n}{<-> s*[0.92489] file:ipaexg.ttf:jfm=ujis}{}

%%% for LTXexample environment
\usepackage{showexpl,lltjlisting}
\lstset{basicstyle=\ttfamily, width=0.3\textwidth}

%%% for logo
\usepackage{mflogo}

%%% Verbatim environment
\usepackage{fancyvrb}
\CustomVerbatimEnvironment{code}{Verbatim}%
{numbers=left,xleftmargin=1.5em,baselinestretch=1.069,fontsize=\small}
\CustomVerbatimEnvironment{codewithoutnum}{Verbatim}%
{xleftmargin=1.5em,baselinestretch=1.069,fontsize=\small}
\CustomVerbatimEnvironment{codewithoutnumsmall}{Verbatim}%
{xleftmargin=1.5em,baselinestretch=1.0,fontsize=\footnotesize}
\DefineShortVerb{\|}

%%% Mandatory article metadata %%%
\title{The development of \LuaTeX-ja package}
\author{Hironori Kitagawa}
\address{The \LuaTeX-ja project team}
\email{h\_kitagawa2001@yahoo.co.jp}

\keywords{\TeX, p\TeX, \LuaTeX, \LuaTeX-ja, Japanese}
\abstract{%
The \LuaTeX-ja package is a macro package for typesetting Japanese documents under \LuaTeX.
This packages has much flexibility of typesetting than p\TeX, and corrected some unwanted features of p\TeX.
In this paper, we describe specifications, the current status and some internal processing codes of \LuaTeX-ja.
}

\newcommand{\parname}[1]{\textsf{#1}}

\begin{document}

%%% Do not forget to start with \maketitle!
\maketitle

\section{Introduction}
\subsection{History}
To typeset Japanese documents with \TeX, ASCII p\TeX~\cite{ptex} has
been widely used.  There are other methods---for example, using Omega
and OTP~\cite{omegaj}, or with the CJK package---to do so, however,
these alternative methods did not became a majority. On the one hand,
p\TeX\ enables us to produce high-quality documents, but on the other
hand, p\TeX\ is left behind from the extensions of \TeX\ such as \eTeX\
and \pdfTeX, and the diffusion of UTF-8 encoding.  In recent years, the
situation become better, because of the developments of |ptexenc|~\cite{ptexenc} by Nobuyuki Tsuchimura, 
$\varepsilon$-p\TeX~\cite{eptex} by the author,~and
up\TeX~\cite{uptex} by Takuji Tanaka.

However, there are still lag now. 


Before this \LuaTeX-ja package, there were several attempts to typeset Japanese documents under \LuaTeX.
Here we cite three examples:
\begin{itemize}
\item |luaums.sty|~\cite{luaums} developed by the author. This experimental package is for creating a Japanese-based presentation under \LuaTeX.
\item |luajalayout| package\cite{luajalayout}, formerly known as the |jafontspec| package, by Kazuki Maeda. 
This package is based on \LaTeXe\ and |fontspec| package.
\item |luajp-test| package\cite{luajp-test}, a test package made by Atsuhito Kohda, based on articles on the web page~\cite{joylua}.
\end{itemize}
% ltja: hassan boushi

\subsection{Development Policy of \LuaTeX-ja}
\label{ssec-pol}
The first aim of the project is to implement features of p\TeX as macros under \LuaTeX, so \LuaTeX-ja is much affected by p\TeX.
However, as the development proceeds, some technical/conceptual difficulties are arised. Hence we changed the aim of the project:
\begin{itemize}
\item\emph{\LuaTeX-ja offers more flexibility of typesetting than that by
     p\TeX.}

     There is JIS~X~4051 Japanese Standard, ...
     However, we think that the ability of producing outputs conformed to JIS~X~4051 is not enough for (Japanese) typesetting;
     the \TeX\ engine can handle the case even when one wants to produce very incoherent output for some reason.
     In this point, we can say p\TeX has some flexibility, hence \LuaTeX-ja should have more.

     For this reason, \LuaTeX-ja has counterparts of additional primitives of p\TeX.

\item\emph{\LuaTeX-ja isn't mere re-implementation or porting of p\TeX;
     some (technically and/or conceptually) inconvenient features of
     p\TeX\ are modified.} 

     We describe this point in more detail at the next section.
\end{itemize}


\subsection{Contents of this Paper}
Here we describe the contents of the rest of this paper briefly. 
In Section~2, we describe major differences between p\TeX\ and \LuaTeX-ja,
which is introduced. Some of them are due to specifications of callbacks
in \LuaTeX\ (\emph{i.e.}, technical reason), and others are which we
thought which are better to be changed, for ``natural''
specifications. In Section~3, we show the current status of the \LuaTeX-ja project.

For implementing features into \LuaTeX-ja, we had to use some tricks in Lua scripts. 
In Section~4, we describe several these tricks and internal processing methods.
We hope that the materials in this section, espcially
Subsection~4.3, have good applications.


\section{Major differences with \pTeX}
In this section, we breifly look at ** major differences between p\TeX\ and \LuaTeX-ja.

\subsection{Names of Control Sequences}
Since p\TeX\ is a modification of a \TeX\ engine, some primitives added in it takes a form that cannot be simulated by a macro.
For example, an additional primitive |\prebreakpenalty|$\langle\hbox{\it
char\_code}\rangle$|=|$\langle\hbox{\it penalry}\rangle$ in p\TeX\ sets the
amount of penalty inserted before $\langle\hbox{\it char\_code}\rangle$
to $\langle\hbox{\it penalry}\rangle$.

Moreover, there are some parameters for Japanese typesetting which were
mere internal integers, dimensions, or~skips in p\TeX\ that cannot be
implemented by same approaches in \LuaTeX-ja.  These parameters have a
common feature; the values at the end of a horizontal box or that of a
paragraph are effective in whole box or apragraph.  A good example of
them is |\kanjiskip|, the default amount of a skip which will be
inserted between two consecutive Japanese characters by default.  The
reason of this is the place of |hpack_filter| in the \LuaTeX's
CWEB-source code, and we will discuss on it in
Subsection~\ref{ssec-stack}.

From above 2~problems we discussed above, the assignment and retrieval
of most parameters in \LuaTeX-ja are summarize into 3~control sequences:
\begin{itemize}
\item |\ltjsetparameter{|$\langle\hbox{\it
      name}\rangle$|=|$\langle\hbox{\it value}\rangle$|,...}|: for local
      assignment.
\item |\ltjglobalsetparameter|: for global assignment.
\item |\ltjgetparameter{|$\langle\hbox{\it
      name}\rangle$|}[{|$\langle\hbox{\it optional
      argument}\rangle$|}]|: for retrieval. The returned value is always
      a string.
\end{itemize}

\subsection{Linebreak after Japanese Character}
\label{ssec-line} 
Japanese texts can linebreak almost everywhere, in contrast with
alphabetic texts can linebreak only between on words (or use
hyphenation). Hence, p\TeX's input processor is modified so that a
linebreak after a Japanese character doesn't emit a space. However,
there is no way to customize the input processor of \LuaTeX. All we can
do is that modify an input line before when \LuaTeX\ begin to process
it, using |process_input_buffer| callback.

Hence, in \LuaTeX-ja, a comment letter (we reserve U+FFFFF for this
purpose) will be appended to an input line, if this ends with a Japanese
character\footnote{Strictly speaking, it also requires that the catcode
of the endline character is 5.}. One might jump to a conclusion that the
treatment of linebreaks by p\TeX and that of \LuaTeX-ja is very similar,
but they are different in the respect that \LuaTeX-ja's judgement
whether a comment letter will be appended the line is done \emph{before}
the line is processed by \LuaTeX.

Figure~\ref{fig-linebreak} shows an example; the command at the first
line marks most of Japanese characters as ``non-Japanese character''. In
other words, from this command onward, the letter `あ' will not be
received any process of \LuaTeX-ja. Then, it is natural to occur a space
between `あ' and `y' in the output, where the actual output does not so.
This is because `あ' is considered to be a Japanese character by
\LuaTeX-ja, when \LuaTeX-ja look the end of the input line~2.
\begin{figure}
\begin{LTXexample}
\font\x=IPAMincho \x
\ltjsetparameter{jacharrange={-6}}xあ
y
\end{LTXexample}
\caption{A notable sample showing the treatment of a linebreak after a Japanese character.}\label{fig-linebreak}
\end{figure}

\subsection{Separation between ``real'' fonts and Metrics}
Traditionally, most Japanese fonts used in typesetting are monospaced,
that is, most glyphs have same size and square-shaped. Hence, it is not
rare that the contents of different Japanese TFMs (JFM, for short) are
totally same, and only differ in their names. For example, the
difference between |min10.tfm| and |goth10.tfm|, which are JFMs shipped
with p\TeX\ for \emph{Mincho} and \emph{Gothic} fonts, are their
|FAMILY| and |FACE| only. Another example is: if one want to use many
fonts which are not installed in his \TeX\ distribution, of course he
needs to prepare TFMs for them. But, as long as he wants to use Japanese
fonts with p\TeX, he has to only copy and rename some JFM (\emph{e.g.,}~copy |jis.tfm| to |hoge.tfm|).

Considering this situation, we decided to separate ``real'' fonts and
Metrics in \LuaTeX-ja, as shown in Figure~\ref{fig-jfdef};
\begin{itemize}
\item a control sequence |\jfont| must be used for japanese fonts, instead of |\font|.
\item \LuaTeX-ja automatically loads the |luaotfload| package, so
      |file:| prefix and features can be used as the line~1 in
      Figure~\ref{fig-jfdef}.
\item The |jfm| key specifies the metric for the font. In
      Figure~\ref{fig-jfdef}, both fonts will use a metric stored in a Lua script named 
      |jfm-ujis.lua|.
\item The |psft:| prefix can be used to specify name-only, noembedded
      fonts. 
\end{itemize}
We note that |-kern| in features is important, since if kerning information from real font itself will clash with spacing from the metric.

\begin{figure}
\begin{verbatim}
\jfont\foo=file:ipaexm.ttf:jfm=ujis;script=latn;-kern;+jp04 at 12pt
\jfont\bar=psft:Ryumin-Light:jfm=ujis at 10pt
\end{verbatim}
\caption{Typical declarations of Japanese fonts}
\label{fig-jfdef}
\end{figure}

\subsection{Insertion of Kerns and/or Glues for Japanese Typesetting: The Timing}
As described in \cite{luatexref}, \LuaTeX's kerning and ligaturing
process is totally different from that of Knuth's original \TeX82.
\TeX82's process is done just when a (sequence of) character node is
appended to current list, hence we can cancel this process by writing
|f{}irm| (this gives `f\hbox{}irm' in \TeX82).  However, \LuaTeX's process is
\emph{node-based}, that is, the process will be done when a horizontal
box of a paragraph is ended, so |f{}irm| and |firm| yield the same
output under \LuaTeX.

The situation for Japanese characters is basically same, but not entirely.
Glues (and kerns) those will be needed for Japanese typesetting will be divided into the following three categories:
\begin{description}
\item[Glues (and Kerns) from the Metric of Japanese Fonts] 
\item[Default Skip Between a Japanese Character and an Alphabetic Character] 
Usually 1/4 of fullwidth with some stretch and shrink.
\item[Default Skip Between Two Consecutive Japanese Characters] 
\end{description}
In p\TeX, these three kinds of glues are treated differently. The first
 category (\emph{JFM glue}, for short) is inserted when a Japanese
 character node is appended to current list, same as alphabetic
 characters in \TeX82. The second category (\emph{xkanjiskip}, for
 short) is inserted just before `hpack' or line-breaking of a paragraph;
 this timing is somewhat similar to that of \LuaTeX's kerning
 process. The third category (\emph{kanjiskip}, for short) is not
 appeared as a node anywhere; only appears implicitly in calculation of
 width of a horizontal box of line-breaking. These specifications made
 p\TeX's behavior very hard to understand.

\LuaTeX-ja does the insertion process of glues in all three categories
simultaneously, inside |hpack_filter| and |pre_linebreak_filter|
callbacks.  The reasons of this specification are to behave like
alphabetic characters in \LuaTeX\ (as described in the first paragraph),
and to clarify the specification for \LuaTeX-ja's process.


\subsection{Insertion of Kerns and/or Glues for Japanese Typesetting: The Behavior}



\section{Current Status of the Development}
\LuaTeX-ja 

\section{Implementation}
\subsection{Handling of Japanese Fonts}
In p\TeX, there are three slots for maintaining current fonts, namely
|\font| for alphabetic fonts, |\jfont| for Japanese font (in horizontal
direction) and |\tfont| for Japanese font (in vertical direction). With these slots, we can select
current font for alphabetic characters and that for Japanese characters separately in p\TeX.
However, \LuaTeX\ has only one slot for maintaining current font, as \TeX82. 
This situation leads a problem: how can we maintain ``current Japanese font''?

There are three approaches for this problem. One approach is to make a
mapping table from alphabetic fonts to corresponding Japanese fonts
(here we don't assume that NFSS2 is available), and when current
alphabetic font is changed, current Japanese font also changes according
to the table.  Another approach is that we always use composite fonts
with alphabetic fonts and Japanese fonts. The third approach is that the
information of current Japanese font is stored in an attribute. We
adopted the third approach, since \LuaTeX-ja is much affected by p\TeX\
as we noted in Subsection~\ref{ssec-pol}.




\subsection{Overview of the Processes}
Now we describe an outline of the \LuaTeX-ja's process briefly.
\begin{description}
\item[Treatment of Linebreaks after Japanese Characters] We described
           this already at Subsection~\ref{ssec-line}. Registered in the
           |process_input_buffer| callback.
\item[Font Replacement] In the |hyphenate| callback, we looks into for
           each \textit{glyph\_node}~$p$. If its character is considered
           to be a Japanese character, the font used in $p$ is replaced
           by the value of |\ltj@curjfnt| that is associated
           with~$p$. Also we subtract the subtype of $p$ by 1 to
           suppress hyphenation around it by \LuaTeX, since later
           processes of \LuaTeX-ja take care of all things about
           Japanese charaters.
\end{description}
%
Following processes are all executed in |pre_linebreak_filter| and |hpack_filter| callback. These are main routines of \LuaTeX-ja:

\begin{description}
\item[Examination of Stack Level] We traverse the horizontal list which is the content of a horizontal box 
to determine what is the level of \LuaTeX-ja's internal stack in the end
           of the list. This is needed because of the place of
           |hpack_filter| in the source of \LuaTeX. We will discuss more detail at Subsection~\ref{ssec-stack}.

\item[Insertion of Glues/Kerns for Japanese Typesetting]
This part is already described at Subsection~\ref{ssec-jglue}. 

\item[Adjustument of Places of (Japanese) Characters]
Under \LuaTeX-ja, the size of the virtual body of a Japanese character
           and its position (\emph{i.e.}, offset) are determined by the
           metric, since the optimal width of a character in
           typesetting---in most cases, this is specified width in the
           metric---and the actual width in TrueType/Opentype fonts
           often differ. For example, the width the fullwidth open brace
           `\inhibitglue ｛' is considered to be half-width in
           typesetting, although this character is full-width in
           TrueType fonts like IPA~Mincho.

To adjust size/places of Japanese characters, \LuaTeX-ja encapsules a
           \textit{glyph\_node} which containing a Japanese character
           into a horizontal box which size is specified in the metric.
\end{description}

\subsection{Stack Management}
\label{ssec-stack}
\LuaTeX-ja has a lot of parameters for Japanese typesetting. 

\subsection*{About the Project}
\subsection*{Acknowledgements}


%%% The style of the bibiliogrphy is `amsplain'.
\providecommand{\bysame}{\leavevmode\hbox to3em{\hrulefill}\thinspace}
\providecommand{\href}[2]{#2}
\begin{thebibliography}{9}

%\bibitem{Knuth}
%Donald E.~Knuth, \emph{The \TeX book}, Addison-Wesley, 1986.

\bibitem{ptex}
ASCII MEDIA WORKS, \textbf{アスキー日本語\TeX\ (p\TeX)}\ (in Japanese). \url{http://ascii.asciimw.jp/pb/ptex/}

%\bibitem{Eijkhout}
%Victor Eijkhout, \emph{\TeX\ by Topic, A \TeX nician's Reference}, Addison-Wesley, 1992. \url{http://www.cs.utk.edu/~eijkhout/texbytopic-a4.pdf}

\bibitem{luaums}
Hironori Kitagawa, \textbf{LuaTeXで日本語}\ (in Japanese). \url{http://oku.edu.mie-u.ac.jp/tex/mod/forum/discuss.php?d=378}

\bibitem{luajalayout}
Kazuki Maeda\ (前田一貴), \textbf{luajalayout パッケージ —LuaLaTeX による日本語組版—}\ (in Japanese).
\url{http://www-is.amp.i.kyoto-u.ac.jp/lab/kmaeda/lualatex/luajalayout/}

\bibitem{luajp-test}
Atsuhito Kohda, \textbf{LuaTeXと日本語}\ (in Japanese). \url{http://www1.pm.tokushima-u.ac.jp/~kohda/tex/luatex-old.html}

\bibitem{joylua}
Yannis Haralambous. \textbf{The Joy of LuaTeX}. \url{http://luatex.bluwiki.com/}

\bibitem{luatexref}
\textbf{The \LuaTeX reference}

\end{thebibliography}

\end{document}