Category: Uncategorized

ASCII is a 7-bit code and defines characters 0.. 127. Computers have 8-bit bytes and each byte can store values 0..256. When using these bytes to store ASCII characters, the values 128..256 are never used. Would it be a good idea to extend ASCII with 128 more values that could store the characters we need that are not in ASCII? Maybe that could be a good idea. The problem was: too many different people got the same idea and each of them defined different additional characters and added them in a different order. The really great idea only came in 1992 in the form of UTF-8.

There were separate encodings from different vendors and for each vendor there would be many different code pages to cover different sets of languages. The only good thing: the ASCII part of the set would be left alone, so at least all character sets supported full ASCII.

Different Vendors

Let’s describe some vendors who extended ASCII to an 8-bit code:

• In 1981, IBM introduced the IBM-PC with a new 8-bit character set. That characters set was known as Code Page 437 (CP437). It had a limited selection of international Latin characters (mostly lowercase) and both the selection of which characters to include and the order in which they appeared, seem to be rather random. No plan, but a crazy brainstorm session. MSX (1884) and Atari-ST (1985) based their character sets on this set, but they both made extensive changes, adding more international characters and leaving out box drawing characters.
• Apple introduce the Macintosh in 1984, complete with its own extended 8-bit character set. The selection of characters seems far more logical than IBM’s.
• Digital added a Multinational Character Set (MCS) to their vt220 terminal in 1983. This would later be developed into ISO-8859-1.
• There were others: computers like the Acorn BBC Master, Grundy NewBrain and Sinclair QL each had their own proprietary 8-bit character set, never seeing any usage on other platforms. Rightfully forgotten.

All except Digital filled in all code points from 128 to 255. The DEC vt220 terminal used the range 128..159 for additional control characters and only add printable characters for 160..255.

Code Pages

A single 8-bit character set usually does not cover all languages for all markets where the computer was sold. Computers sold in Turkey needed Turkish letters such as the dotless i and s-cedilla. Computers sold in Greece needed support for the Greek alphabet. Therefore all important vendors had not one, but many different 8-bit character sets, called code pages. Different code pages were for different countries.

Some countries, like Russia had their own 8-bit extended character sets. Russia had the KOI8 family of character sets.

ISO-8859

In 1987, ISO introduced a family of 8-bit character codes, called ISO0-8859. All of these reserve the range 128..159 for C1 control characters. https://en.wikipedia.org/wiki/ISO/IEC_8859

Eventually, there would be 15 character sets, numbered from 1 to 16 (12 was omitted), 10 of which would be for the Latin script. Five others would be for Cyrillic, Arabic, Greek, Hebrew and Thai. ISO-8859-16 would be introduced in 2001, when Unicode was already well established.

The first set in the family, ISO-8859-1, would be based on the vt220 Multi National Character Set. It would include support for Finnish, Swedish, Norwegian, Danish, Icelandic, Dutch, English, French, German, Spanish, Portuguese and Italian, all relevant Western European languages. Even though the old vt220 set included the French letters œ, Œ and Ÿ, they were excluded from ISO8859-1, being replaced by ×, ÷ and Icelandic letters. Later, ISO-8859 would add these letters back in, along with š and ž (for Finnish) and the Euro sign, all at the cost of some rarely used symbols.

ISO-8859 would be included whole in Unicode and it was also the base of Windows Code Page 1252, which is a superset of ISO-8859-1, but replacing some of the C1 control characters with additional symbols. Most other Windows code pages are not compatible supersets of their corresponding 8859 sets. So apart from the ISO-8859 character sets, we also have to deal with a bunch of slightly incompatible Windows code pages.

All Latin sets in ISO-8859 support German (including the ß) but only half of them would support French. Polish is in three of these sets, but with some characters in different positions in each of them. Czech is only in one of these sets. There’s no single set with both French and Czech. Ten different character sets for Latin was just too many.

Enter the fun world of control characters. The five-bit ITA2 code contained a bunch of them, all of which got equivalents in ASCII:

• Carriage return, to return the carriage (that contained the paper) of the typewriter to the rightmost position. In old typewriters, the carriage moved to the left as you typed, so the next character position on the paper would become the target of the tying hammers. In later printers, the print head would move to the right as you typed, while the paper stayed in the same position. Carriage returns would return the print head to the leftmost position.
• Line feed, which advanced the paper vertically by one line.
• Letters and figures shift. ASCII has no letters and figures shift as such, but it has the SO (shift out) and SI (shift in) control characters that can shift to and from an alternate character set. Teletypes could for instance switch between Greek and Latin this way.
• The null character, all zero bits, which did nothing.
• The Bell character, that sounded a bell.
• The WRU (Who are you) character. This caused the receiving teletype to send a response. This way you could be sure a teletype was running at the other end of the line when you typed a message. You could not know if the teletype had paper and ink, but at least you could know it was connected and switched on. ASCII has the ENQ control character for this purpose.

The ability to print a carriage return and a linefeed independently, allowed you to return to the start of the line without advancing the paper to the next line. This way you could overprint a line with different characters, possibly adding accents to letters or underlining some words. The Backspace character in ASCII would be simpler to use for this purpose. This overprinting worked on printing teletypes, but not on CRT terminals.

ASCII added many more control characters, some of which are still widely used, some of which are all but forgotten. See https://en.wikipedia.org/wiki/C0_and_C1_control_codes. The widely used ones are:

• BEL Sounds a bell or a beep, or emits a visual attention signal.
• Backspace, Moves the print head one position to the left.
• Horizontal Tab. Moves the print head to the next tab position.
• Vertical Tab. Advances the paper to the next vertical tab position. This is not actually widely used but its purpose is still recognised. It was used with pre-printed forms that required some fields to be filled in.
• Form feed, Advances the paper to the next page. On CRT terminals this was sometimes used to clear the screen.
• DC1 and DC3 (control-Q and Control-S). These were originally used to control a paper tape reader. A program receiving data from a paper tape would send those control characters to pause and resume the paper tape reader, as a means of flow control. This type of flow control was widely used with CRT terminals to pause and resume terminal output and it’s still implemented today.
• Escape. This is mostly used today as a prefix for more advanced terminal commands, for example to position the cursor or to insert and delete lines on the screen. The ANSI escape sequences are universally used for terminal-based programs.
• DEL (0x7F). This control character, lonely at the top end of the 7-bit ASCII range, was originally intended to be used on papertape. One could rewind the tape a few character positions and then punch DEL over those characters to erase them. When the papertape was later read back, the DEL characters would be ignored, just like Null characters. As DEL has all bits set, punching it over another character would punch all holes, turning anything else into DEL.

Later, another range of control characters was introduced: the C1 range 0x80.. 0x9F. These are not widely used. There is an unambiguous newline character in this range (0x85), but this is not widely used. Unicode added line and paragraph separators, but none of these are widely used in plain text files.

Modern Usage

Nearly all modern terminals (typically implemented in software on a computer), allow you to type the control characters 0x01 to 0x1A (Control-A to Control-Z). The Tab key will output Control-I (0x09), the Return key will output Control-M (0x0D) and the backspace key will output either Control-H (0x08) or DEL (0x7F), depending on which side of the holy war you are. There is a dedicated ESC key (0x1B) and some less obvious key combinations will get you NUL and the characters in the range 0x1C..0x1F. Control keys are often used in a way that is totally unrelated to their original meaning in ASCII.

• Unix uses Ctrl-C (ETX) to terminate a running program, Ctrl-D (EOT) to indicate the end of input and Ctrl-Z (SUB) to suspend the currently running program.
• CP/M files had a length specified in 128-byte blocks. When a text file was not a multiple of 128 bytes in size, the file was padded with SUB (Control-Z) characters. Even if the file was a multiple of 128 bytes, they would still add a block of Ctrl-Z, so that would be a reliable end-of-file indicator. This habit was carried over to MS-DOS (that did store exact file sizes). A single Ctrl-Z was typically appended to each text file. Some programs choked on Ctrl-Z, some would choke when the Ctrl-Z character was not present. It was a mess.
• WordStar was an early word processor under CP/M. It got ported to MS-DOS and many other editors copied its control key layout. It used Ctrl-S for cursor-left, Ctrl-D for cursor-right, Ctrl-E for cursor-up and Ctrl-X for cursor-down. The choice of these control codes has nothing to do with their meanings in ASCII, but everything with the layout of these keys on the keyboard. WordStar was developed at a time that many computer terminals did not have cursor keys.
• Many Unix editors use control codes, such as nano and emacs make extensive use of control codes in a way totally unrelated to their meaning in ASCII.
• GUI applications use Ctrl-Z for undo, Ctrl-X for “cut”, Ctrl-C for Copy and Ctrl-V for “paste”.

Holy Wars

How should text files be separated into lines? This has never been settled for real.

There are two aspects of this:

• Should a line terminator be at the end of each line you see? or should there only be one at the end of each paragraph? For source code, every line should have a line terminator, but for running text. this is not so obvious. Some authors prefer putting an entire paragraph in a long line. The expect text editor programs to wrap these lines to the width of the screen they are using. Others want to put a line separator at the end of each visual line and put a blank line (two line separators) between paragraphs.
• Should the last line of a text file always end in a line terminator/separator?
• What should the line terminator be?
  • CP/M and MS-DOS settled on the sequence CR-LF. This is what printers require when you print the file. This convention was carried over to Windows.
  • Apple and a bunch of 8-bit systems settled on just CR at the end of each line. Apple later followed the Unix convention.
  • Unix settled on just LF at the end of each line.

Today it’s common wisdom that programs should at least accept text files with LF-only and with CR-LF on reading Both conventions are here to stay. Unicode line separators or the new NL character 0x85 never caught on.

Another source of heated debate is the use of tab characters in source files.

• Some authors prefer their source files to be free of any tab characters. Any indenting is done with spaces.
• Other authors prefer indenting with Tab characters instead. The configuration of the tab stops becomes another point of discussion. Some programmers insist on tab stops every four spaces, others want tab stops every eight spaces.

Finally there is discussion on the character code that should be emitted by the backspace key (the big left arrow right of the ‘=’ key), in particular on Unix systems.

• Unix purists insist on Backspace = Backspace (0x8 = Ctrl-H).
• Others insist on Backspace = DEL (0x7F).

Terminal programs can be configured to emit either character as backspace and the “erase” character on the Unix line input function can be configured to any character. Many programs accept either convention. But it does get super annoying when not everything on the same system is configured the same way. If some terminal programs on your desktop emit BS and others emit DEL and your shell isn’t configured correctly for some of them.

In the early 1960s, the world (in fact the USA) needed a standardised character code for both telegraphy and computers. The five-bit ITA2 code was lacking, mainly because it required frequent use of letters and figures shift control characters. Work on ASCII started in 1961 and a first version was released in 1963.

ASCII was to become a 7-bit code and initially half of the code points were reserved for control characters.

The 1963 version included most familiar characters with codes 0x20 to 0x5F. Lowercase letters were not defined yet and there was an up-arrow instead of the caret and a left-arrow instead of the underscore.

In 1965, lowercase was added and some character codes were swapped (most of them were swapped back in the final 1968 version). In 1968 we got the final 7-bit ASCII code. Many control codes were defined with their names, but little was specified about their meanings. Many use cases were envisioned, such as mail systems where the start of a header could be marked with a dedicated control character and database formats where records and groups could be separated with specific control characters.

We got accent characters ‘`’ , ‘^’ and ‘~’ and the apostrophe was supposed to do double duty as the acute accent while the double quote character could do double duty as umlaut/diaeresis. This way, you could type text in most Western European languages by overprinting letters with accent characters. The underscore character was likewise intended to add underlining to text. This overprinting never got popular and on early CRT terminals, it did not work at all. Therefore the accent characters were later mainly used as separate characters, not as combining accents.

EBCDIC

Of course, IBM did things their own way and in 1963, they extended the 6-bit BCDIC character set into a 8-bit EBCDIC character set. It included some characters missing from ASCII, such as ‘¬’ and ‘¢’ and it included both uppercase and lowercase letters, something the ASCII team was still heavily debating whether to have it or not. Most of the 256 code points were unassigned.

When ASCII got finalised, EBCDIC got all characters that ended up in ASCII. There were code pages galore, with country specific extensions and later there was even a UTF-style encoding to encode all Unicode characters with it (using sequences of those byte values that are unassigned in basic EBCDIC). EBCDIC is still heavily used in the mainframe world. Everything outside the mainframe world uses ASCII or Unicode

ISO-646

ASCII was to become part of a large family of country-specific character sets. Most characters were standardised, but some characters like ‘@’, ‘[‘, ‘\’ and ‘]’ could be replaced by language-specific letters, for example Æ, Ø and Å in Danish. This worked well for the Scandinavian languages (that each need three additional letters), as well as for German with its Ä, Ö, Ü and ß. French however required more accented letters than there were code points available for country-specific usage. Combining accents did not work well (or even at all) with the hardware and software of the day. The same was true for Dutch.

Country-specific ASCII variants were a pain in the ass, especially when you had to exchange data among different European countries. With the rise of the C programming language in the 1980s, characters like ‘{‘, “}’ and ‘~’ had become essential. The ISO standard describe trigraphs as substitutes, but really nobody liked to use these.

8-bit character codes and later Unicode, would make country-specific ISO-646 variants obsolete.

A Lasting Legacy

Nearly all programming languages use exclusively ASCII symbols for their source code and nearly all ASCII symbols are used in some programming languages. Take one printable ASCII character away and some operators in C or some constructs in Unix shells can no longer be typed. Nobody would think much of the ‘~’ character or curly braces, but without them, C would not be C. Now that these characters are part of the syntax of popular programming languages, nobody can take them away.

Source code is ASCII. Most programming languages allow identifiers to contain letters in any scripts defined by Unicode, but this is seldom used. Source code is international and international means English, especially for open source programs. The US keyboard layout allows all ASCII characters to be typed with a single key or a single shifted key (no awkward AltGr combinations). Therefore it is so popular among programmers.

Most programming language standards specify that source code is Unicode, but Unicode still has ASCII at its core. The most popular encoding for Unicode is UTF-8 and guess what: if you use only the lowest 128 code points of Unicode (which are identical to ASCII), your source file would just be ASCII. All other Unicode code points are encoded with sequences of bytes that have the high bit set. Unicode could be in strings and comments and that would work without problems most of the time.

Of course there is also ASCII art. It is called ASCII art for a reason and true ASCII art uses ASCII characters exclusively.

File names should also be in ASCII (though most file systems allow arbitrary Unicode file names nowadays).

The first computer character set dates back to 1928 and it was a 12-bit code. Of course we refer to the Hollerith code used on 80-column punch cards. https://en.wikipedia.org/wiki/Punched_card Each column has 12 positions to punch holes (therefore it can be considered a 12-bit code), but in reality this was not treated as a 12-bit code with 4096 distinct values at all. This predates computers as we know them, but semi-automatic data processing with punched cards was widespread in the 1930s. In the 1950s, computers took this to a whole new level.

The 12 positions in each column were marked as 12, 11, 0 and 1..9. A space character left all 12 positions unpunched, a digit of 0..9 was represented by a single punch in the respective position 0..9. A single punch in position 12 represented ‘&’ (sometimes ‘+’), a single punch in position 11 represented ‘-‘. Letters A..I were represented by a punch in position 12 plus a punch in one of the positions 1..9. The letters J..R were represented by a punch in position 11 plus a punch in one of the positions 1..9. The letters S..Z were represented by a punch in position 0 plus a punch in one of the positions 2..9 (0 + 1 represented the ‘/’ character). Further symbols had a punch in position 8, plus a punch in one of the positions 2..7 plus a punch in either 12, 11 or 0, or none of these.This took care of 64 distinct code points. Characters were often stored in 6 bits internally in computer memory. The resulting character code had the letters of the alphabet non-contiguous in three groups. It was often referred to as BCDIC (BCD Interchange Code). Note: this is still without the E of EBCDIC.

However, IBM’s printers that were often used with these systems, only had 48 characters (including the space), IBM even had different code pages for FORTRAN (that included “+’, ‘=’, ‘(‘ and ‘)’) and for general text (that included ‘&’, ‘?’, ‘;’ and ‘:’). Yes, code pages in the 1950s. To use a different code page, you had to change the printing chains in the printers and possibly the keycaps and the internal printing mechanisms of the card punches.

Teletypes

The five-bit Baudot code was already invented in the 1870s and could be used for sending text messages across a wire, also known as telegraphy. In 1932, the ITA2 (International Telegraphy Alphabet) was a variation of this, introduced in 1932. This system used five-bit asynchronous signalling with start and stop bits, almost identical to RS232. This was not originally invented for computers, but just to type a message on one end of a telephone line and have it printed on the other end. Teletype machines often included a papertape puncher and reader, so you could type a message at your own typing speed first and record it on papertape. Later you could play back the tape and transmit the message across a wire or radio link at full speed.

As it is a five-bit code, you need special letters and figures shift control characters to switch between letters and figures (digits and typographic symbols), For example for each period or comma you had to type “Figures Shift”, then the period or comma, then “Letters Shift”. This sucked. In the 1950s, efforts were underway to define a more modern telegraphy code, without the letters and figures shift and possibly with both uppercase and lowercase letters. One of these efforts was FIELDATA (of the US military).

When computers were introduced, these teletype machines inevitably turned up as peripherals for them. As they often included papertape punches and readers, they could also serve as a mass storage medium.

In the early 1960s, many different 6-bit character codes were in use:

• Some were based on BCDIC
• Some were based on what later would become known as ASCII
• Some were based on FIELDATA
• Some were based on ITA2 (giving the shifted and unshifted characters different codes).
• Some were based on the Friden Flexowriter

The PDP-1 used the Flexowriter as its teletype and used its 6-bit character code. The later PDP-8 used an ASCII subset. ASCII subset character sets were common on most machines developed after 1965 that had 6-bit characters instead of 8-bit bytes.

In 1980, Clive Sinclair introduced a very minimalistic Z-80 based microcomputer, the ZX-80. It and its successor the ZX-81 were the only microcomputers known to mankind that did not use ASCII (or at leased something loosely based on it as was the case with Commodore). At that time, ASCII was so widespread that it was a no-brainer to use it.