Development Notes

This is a dump of information and references related to creating kisekae.org. A lot of it is just mistakes and lessons I learned along the way.

This site was built using:

• React
• NextJS (which is honestly overkill for the amount of content, but allows me to render a static page for easier SEO and use Markdown/MDX for pages like this one)
• jslha to handle the .lzh archives (I have a whole section below on some issues I have had using this particular library, but it's been very helpful!)
• 98.css for overall styling of the website
• JZZ for handling midi files (This library was a pain to set up in React, and then Chrome started prompting people to allow sites to control their MIDI devices, which makes it seem sketchy. I just wanted to give you the real experience of viewing a KiSS doll!)

This site would be impossible without the documentation on OtakuWorld's KiSS Help Page. These sites were also immensely helpful:

• UltraKiss kinda the gold standard of KiSS viewers, with its code now released to Github.
• emk's HTML5 KISS
• KiSSing with Emby Quinn

My Canvas Woes

Images are not transparent on canvas

When I got the .cel image data to draw to canvas, I encountered my first issue. I wasn't properly handling the transparent index color from the KiSS Color File (kcf) and so the background color filled in the rest of the image.

Each palette has a transparent index for which color to exclude as the transparent mask. This index is typically at 0. Once I fixed that, however, the images then had white backgrounds rather than being transparent.

It comes down to using putImageData versus drawImage. putImageData overwrites the pixels in the context as each layer is drawn, replacing whatever was there previously.

However, putImageData will replace the pixel value in the context. It won't merge it with the previous value of the pixel in the context, it will replace it. So, what you should see is the pixel behind the canvas (in most of cases, the pixel color of the body tag of your html page).

stackoverflow: Transparent pixels using HTML5 canvas and getImageData?

Short answer is no, composite modes does not affect putImageData. Using putImageData works at a lower level than composition modes and context in general.

stackoverflow: Does globalcompositeoperation=source-over work with putimagedata?

So, basically, I should be using drawImage, but to do that I'll need to get the ImageData into an actual image element/canvas source. drawImage() does not take ImageData, so we need to draw to a temporary canvas to get the image and then save it out. stackoverflow: How to generate an Image from ImageData in JavaScript?

Why won't my image load?

Trigger the canvas context drawImage() in the image.onload event handler

image.onload = () => {
    this.ctx.drawImage(image, cel.xOffset, cel.yOffset, image.width, image.height)
}

https://stackoverflow.com/questions/44578945/canvas-drawimage-todataurl-returns-blank-image

Parsing the CNF (Configuration) File

When I was a tween and had first discovered KiSS dolls, I reeaaaally wanted to make my own. However, I was really put off by the CNF file and the prospect of editing it by hand. Was there any easier way? Probably, but I was out of my depth back then. But now, many years later, I had my chance.

So let's use regex, that's what real developers do! This is fine for lines that are easy to parse or have a consistent structure without too many exceptions. Except for the lines defining cels, which can have several different formats and require conditions and then... it just gets weird.

There is a lot of data packaged into this line: Syntax: #<object number>[.<fix>] <cel filename> [*<KCF number>] [:<set number>]

Example lines:

#2 data1.cel          ; Object 2 uses image data1.cel on all sets.
#2 data1_part.cel     ; Object 2 also uses image data1_part.cel on all sets.
#3.5 data2.cel :2 3 4 ; Object 3 with fixed value of 5 uses image data2.cel
                      ; on sets 2, 3, and 4.
#4.255 data3.cel      ; Object 4 cannot be moved and uses image data3.cel on
                      ; all sets.
#5 data4.cel *1 :6    ; Object 5 uses image data4.cel and KCF #1 on set 6.

A lot of the information on these lines is optional, which makes it flexible, but also makes me look like an idiot when I try to cover all the use cases with regular expressions.

const CEL_PATTERN = /^#(?<obj>\d+)\.?(?<fix>\d+)?[\s\t]+(?<cel>[A-Za-z0-9_~\-&^!]+\.cel)[\s\t]\*(?<palette>\d)\s+\:\s+(?<sets>(?:\d\s?)+)/i; // Are you kidding? Yes, sort of. I did try something like this

This is a pretty gross example of a regex I tried to create to handle the different edge cases for cel lines. Instead, I took to parsing the line as a string instead.

Unzip lzh files

Slightly useful CLI bits for CNF file stuff.

lha -x samnmax.lzh -w samnmax

Unzips file samnmax.lzh to directory ./samnmax

Get a specific file

To view a specific file in the terminal, use -p:

p Print to STDOUT from archive

For viewing the configuration .cnf file: lha -p samnmax.lzh \*.cnf

To extract the file:

For a .cnf file, grep the file list: lha -l melissa.lzh | grep .cnf

> lha -l samnmax.lzh | grep .cnf
< [MS-DOS]                  4106  19.5% Sep  8  1999 samnmax.cnf
< [MS-DOS]                  2816   5.2% Sep  8  1999 samnmax.knf

Then you can extract the CNF file:

lha -x samnmax.lzh samnmax.cnf -w cnf

Legacy KiSS Sets

4 Bit Pixel Data

This looks bizarre! And for two reasons:

1. The cels are too small
2. More obviously, the image data has been drawn on a weird slant for some of the cels.

#1 Image Size This happened because I was not handling the pixels contained in each byte correctly from the cel's raw data. Each byte in the cel's binary content represents two pixels - I was essentially cutting out half of the data.

// Pixel data order (4 bits/pixel)
// With 4 bits per pixel, each byte in a cel row contains 2 pixels. 
// In the situation where the number of pixels is odd, the remaining 4 bits are set to 0.
// Table 5-2: 4-bit pixel layout
// Byte 0	    Byte 1	    Byte 2	    Byte N
// 7654 3210    7654 3210   7654 3210   7654 3210
// pix0 pix1	pix2 pix3   pix4 pix5   pixN 0000

for (let i = 0; i < rawPixelData.length; i++) {
    const byte = rawPixelData[i];
    const byteValue = padBinary(byte.toString(2), 8);
    const pix0 = parseInt(byteValue.slice(0, 4), 2);
    const pix1 = parseInt(byteValue.slice(4, 8), 2);
    ...
}

When I fixed that and refreshed, the cels were properly sized, but the weird slanting images weren't fixed.

#2 Odd width cels and 'stride' Even width cels, such as Ami's shortsleeve shirt (SAILORS.CEL, 66x48) and hat (HAT.CEL, 112x58) appear as expected. Ami's body (AMI.CEL, 67x310) is distorted and, like other distorted items, has an odd width.

This is a 'stride' problem

• https://www.collabora.com/news-and-blog/blog/2016/02/16/a-programmers-view-on-digital-images-the-essentials/
• https://swharden.com/blog/2021-06-03-images-in-memory/

This is where the following line from the KiSS spec is very important:

In the situation where the number of pixels is odd, the remaining 4 bits are set to 0. http://otakuworld.com/kiss/kisshelp/DIRECTKISSKISS_General_Specification_CEL.htm

Here is a stackoverflow question with the exact problem I am having: https://stackoverflow.com/questions/43214199/cant-get-bitmapinfoheader-data-to-display-odd-width-bmp-images-correctly

The asker's quandry:

biWidth doesn't need to be rounded up, the buffer size you allocate for the bitmap does. Each scan line starts at (row * stride) bytes into the buffer. If you have raw 24 bit data that's not padded to DWORD-aligned rows then you'll need to allocate a new buffer and copy each row in individually.

The missing piece here is in the Viewer class when we create the buffer. When we have an odd number width legacy cel, we should make the width 1 pixel wider:

const celWidth = cel.legacyFormat && cel.bitsPerPixel === 4 && cel.width % 2 === 1 ? cel.width + 1 : cel.width;

// And then replace references to cel.width with celWidth instead

And now, Ami Mizuno is looking sharp:

Modern Cels

Modern cels do not require a palette and use 32 bits/pixel where each color is represented by a single byte (8 bits) as ~~red, blue, green, and alpha~~ respectively. Just kidding, the order of colors is actually: Blue, Green, Red, Alpha

Incorrect (my KiSS viewer)

Correct (in UltraKiSS)

jslha Issues

This library is a lifesaver, but does not parse lh6 files and does not implement extended header levels. I am too lazy to implement either when we can recompress the problematic *.lzh files to use the lh5 method and ignore headers for permissions.

Compression method lh5 vs. lh6

This occurs because some archives use the -lh6- compression method instead of the more typical -lh5- methid, which is more commonly used for Kisekae sets.

An example of a file using -lh6- is bibiana.lzh

> lha -vv bibiana_old.lzh
PERMISSION  UID  GID    PACKED    SIZE  RATIO METHOD CRC     STAMP            LV
---------- ----------- ------- ------- ------ ---------- ------------------- ---
10th.cel
[generic]                 3154   91282   3.5% -lh6- 906d 2001-11-16 14:29:46 [0]
10th1.cel
[generic]                 3164   76192   4.2% -lh6- d1ea 2002-02-03 20:40:18 [0]
10th2.cel
[generic]                 3236   91282   3.5% -lh6- 37d6 2001-11-16 14:31:10 [0]
10thhr.cel
[generic]                  476   15657   3.0% -lh6- fe5f 2001-11-16 14:32:28 [0]
...

To recompress this archive using lh5, extract it via lha -x bibiana.lzh -w bibiana and then compress it from the extracted directory: lha -c bibiana_new.lzh *

By default lh5 is used:

When archiving, the -o option specifies the compressing method. the -o5 means to use the -lh5- method. It is widely used and default method. the -o6 and -o7 means to use the -lh6- and -lh7- method. These methods reduced archive file more than the -lh5- method. Just the -o means to use the -lh1- method.

Thus, if we do not include the -o option in our lha command, the resulting archive will default to using lh5

Permission headers

An example of a file with permission headers gina_p.lzh:

> lha -vv gina_p_old.lzh
PERMISSION  UID  GID    PACKED    SIZE  RATIO METHOD CRC     STAMP            LV
---------- ----------- ------- ------- ------ ---------- ------------------- ---
gina_p.cnf
-rw-rw-rw-     0/0         692    4157  16.6% -lh5- 803c 2004-11-01 17:42:46 [1]
lack.kcf
-rw-rw-rw-     0/0         557     800  69.6% -lh5- 69bc 2004-08-23 18:32:22 [1]
aa.cel
-rw-rw-rw-     0/0        1510    8848  17.1% -lh5- bd9b 2004-08-23 18:32:22 [1]
ad.cel
-rw-rw-rw-     0/0         296    2516  11.8% -lh5- 8e77 2004-08-23 18:32:22 [1]
ahahah.cel
-rw-rw-rw-     0/0         270    2174  12.4% -lh5- 6677 2004-08-23 18:32:22 [1]
...

To get this file to work correctly, we have to extract and then recompress using the -g flag:

When archiving with this option, archive with general (obsolete) header format. It uses the level 0 header, filename is uppercased in the archive,Unix specific elements such as permission, user-id and so on are not saved.

Command

In order to account for these two issues, we need to:

1. Do not use the -o option to specify the compression method - we want lh5 and that is the default without this argument.
2. Do use the -g option to exclude the extraneous headers such as permissions and use general header format

# extract target lzh file
lha -x ${filename}.lzh -w ${filename}

# cd into new temp directory
cd ${filename}

# recompress using options outline above
lha -c ${filename}.lzh -g *

Reference: https://www.mankier.com/1/lha#Options

FKiSS implementation notes

General/Basics

• Line always start with ;@
• ;@EventHandler must appear before the start of any FKiSS code and should only appear once in a CNF file
• Two types of commands
  • Events
    • Specifies conditions under which a sequence of action commands execute
    • All actions after an event command are part of that event until the next event command is listed
  • Actions
    • Command which is immediately executed in response to an event
    • An action can
      • Change a cel from mapped to unmapped
      • Change a cel's color
      • Move an object to a new location
      • Change value of a variable
    • All actions must belong to an FKiSS event
  • No matter what however, an event command must come before any action commands.

FKiss 1

http://otakuworld.com/kiss/download/fkiss.txt

Events

• initialize
• begin
• end
• alarm
• press(item or cel)
• catch(item or cel)
• fixcatch(item or cel)
• release(item or cel)
• drop(item or cel)
• fixdrop(item or cel)
• unfix(item or cel)
• set(page number)
• col(palette number)
• never

Actions

• map(item or cel)
• unmap(item or cel)
• altmap(item or cel)
• sound(wav or au file)
• move(item #, x, y)
• changset(set number)
• changecol(palette number)
• timer(alarm, delay)
• randomtimer(alarm, delay, variance)
• nop
• quit
• moveto
• windowsize
• viewport

FKiss1b

• transparent
• debug

FKiss2

Events

• collide
• apart
• stillin
• out
• stillout
• in

Actions

• music
• movebyx
• movebyy
• notify
• shell
• sound

FKiss2.1

Events

Actions

• movetorand
• moverandx
• moverandy
• ifmapped
• ifnotmapped
• iffixed
• ifmoved
• ifnotfixed
• ifnotmoved
• setfix

FKiss3

Events

• label

Actions

• all label related actions
• lettimer
• letmapped
• exitevent
• letobjectx
• letobjecty
• letcatch
• letcollide
• letinside
• all arithmetic actions
  • add, sub, mul, div, mod, random
• conditional actions
• ifequal, ifnotequal, ifgreaterthan, iflessthan, else, endif
• letmousex
• letmousey
• letpal
• letset
• letfix
• lettransparent
• ghost

FKiss4

Events

• mousein
• mouseout
• keypress
• keyrelease
• detached

Actions

• restrictx
• restricty
• letinitx
• letinity
• letwidth, letheight
• all attach/detach
  • attach, glue, detach, letchild, letparent, letsibling
• all group frame
  • setframe, letframe
• setkcf, letkcf
• conditional actions
  • elseifequal, elseifnotequal, elseifgreaterthan, elseiflessthan
• letkey, letkeymap
  • http://otakuworld.com/kiss/kisshelp/DIRECTKISSLetkey.htm

References

• http://otakuworld.com/kiss/kisshelp/DIRECTKISSFKiSS_Data_Types.htm#alarm
  • Order of events:

What commands are most commonly used?

There are a lot of FKiSS commands, especially as you get to the more advanced specifications.

Most KiSS sets contain a lot of FKiSS level 1 commands such as press, alarm, timer, map and unmap as well as required events such as initialize.

Based on the table below, the level 2 commands that should be implemented are in, movebyx and movebyy and perhaps music and sound.

This table tabulates the frequency with which different FKiSS commands occur and the corresponding API version of 4500 sets:

key             value      type       version   
initialize            3212 Event      FK1       
unmap                 2621 Action     FK1       
map                   2574 Action     FK1       
press                 2467 Event      FK1       
alarm                 1697 Event      FK1       
timer                 1676 Action     FK1       
in                    1478 Event      FK2       
movebyx               1389 Action     FK2       
movebyy               1385 Action     FK2       
begin                 1233 Event      FK1       
transparent            742 Action     FK1       
set                    678 Event      FK1       
release                607 Event      FK1       
randomtimer            562 Action     FK1       
altmap                 542 Action     FK1       
music                  418 Action     FK2       
sound                  270 Action     FK2       
collide                246 Event      FK2       
notify                 198 Action     FK2       
end                    170 Event      FK1       
apart                  161 Event      FK2       
out                    156 Event      FK2       
moveto                 123 Action     FK1       
shell                   89 Action     FK2       
move                    79 Action     FK1       
ifequal                 71 Action     FK3       
endif                   64 Action     FK3       
catch                   54 Event      FK1       
ifmapped                53 Action     FK2_1     
unfix                   48 Event      FK1       
label                   48 Event      FK3       
ghost                   40 Action     FK3       
add                     36 Action     FK3       
gosub                   32 Action     FK3       
fixcatch                31 Event      FK1       
ifgreaterthan           30 Action     FK3       
else                    28 Action     FK3       
stillin                 27 Event      FK2       
drop                    23 Event      FK1       
ifnotmapped             23 Action     FK2_1     
letinside               21 Action     FK3       
random                  21 Action     FK3       
moverandx               20 Action     FK2_1     
setfix                  19 Action     FK2_1     
letmousex               18 Action     FK3       
letmousey               17 Action     FK3       
iflessthan              17 Action     FK3       
letobjectx              15 Action     FK3       
sub                     15 Action     FK3       
goto                    15 Action     FK3       
letcollide              15 Action     FK3       
ifnotequal              14 Action     FK3       
changecol               14 Action     FK1       
moverandy               12 Action     FK2_1     
letobjecty              11 Action     FK3       
stillout                11 Event      FK2       
keypress                11 Event      FK4       
fixdrop                  6 Event      FK1       
movetorand               5 Action     FK2_1     
restrictx                4 Action     FK4       
never                    3 Event      FK1       
mul                      3 Action     FK3       
letmapped                3 Action     FK3       
iffixed                  3 Action     FK2_1     
restricty                3 Action     FK4       
exitevent                2 Action     FK3       
lettransparent           2 Action     FK3       
ifnotfixed               2 Action     FK2_1     
letfix                   2 Action     FK3       
debug                    2 Action     FK1       
letwidth                 1 Action     FK4       
nop                      1 Action     FK1       
col                      1 Event      FK1       
detach                   1 Action     FK4       
quit                     1 Action     FK1       
ifmoved                  1 Action     FK2_1     
mod                      1 Action     FK3       
letset                   1 Action     FK3       
elseifgreaterthan        1 Action     FK4       
glue                     1 Action     FK4  

What file types are most common?

Certain commands, such as music and sound, use specific files types. To determine what types of music/sound files are supported, I made a list of the most commonly used files. Of course, it's actually very simple; there is support for .mid and .wav files and that's about it.

File Type | Count | Description     
cel       | 4,543 |  Image data, graphic format for a KiSS set
cnf       | 4,541 | KiSS Configuration file
kcf       | 4,327 | KiSS Color File; palette
txt       | 3,169 | Readme files
mid       |   500 | Sound/music
wav       |   360 | Sound/music
doc       |   289 | Readme files
knf       |    93 | ???
bak       |    77 | Back up files
hed       |    65 | Data description file
diz       |    55 | [file_id.diz](https://en.wikipedia.org/wiki/FILE_ID.DIZ)

Example of .hed with Shift JIS encoding

+-----------------------------------------------------------------------------
| タイトル | 着せ替えデータ愛野美奈子ちゃん　Vol.2
|         | なんとかここまで出来ちゃったんだな(^^)らっきぃ(^^)Version
|　制作者　| ＡＣＥの旦那
|ファイル名| AKS_V21.LZH
| カテゴリ | ＫＩＳＳデータ
| 動作機種 | KISS 2.24C相当が動く機種
|前提ソフト| KISS 2.24C以上
| 圧縮方式 | LHA
|転載の可否| ドキュメントを参照してください。
|  備  考  | KISS2.18相当では使えません。
+-----------------------------------------------------------------------------


Run through Google Translate:

| Title | Dress-up data Minako Aino Vol.2
| | I managed to make it this far (^^) Lucky (^^) Version
| Producer | Husband of ACE
|File Name| AKS_V21.LZH
| Category | KISS data
| Operating model | Models that run equivalent to KISS 2.24C
|Required Software| KISS 2.24C or higher
| Compression method | LHA
|Reprintability| Please refer to the document.
| Remarks | KISS2.18 or equivalent cannot be used.

You'll probably see this block of data on archive.org's KiSS files.