© 2021 Steven Obua

Cosmopolitan Identifiers

by Steven Obua
Cite as: https://doi.org/10.47757/obua.cosmo-id.3
July 1st, 2021
Abstract
I propose a simple Unicode-based lexical syntax for programming language identifiers using characters from international scripts (currently Latin, Greek, Cyrillic and Math). Such cosmopolitan identifiers are designed to achieve much of the simplicity of Fortran identifiers while acknowledging a modern international outlook. This seems particularly advantageous in contexts where such identifiers are not (only) used by professional programmers, but are exposed to normal users, for example through scriptable applications.

Introduction

The possibly oldest programming language still in use, Fortran [1], has an especially simple lexical syntax for identifiers: They consist of letters A to Z and a to z, digits 0 to 9, and the underscore _. An identifier must start with a letter. Moreover, identifiers are case-insensitive: The identifier Fortran denotes the same thing as the identifier fortran, for example.
Especially for users of programming languages who are not professional programmers, such as many scientists and engineers, this syntactical simplicity seems to be a big advantage. Given that Fortran code is used, among other things, to control nuclear power plants, it certainly is a good thing to reduce the potential for confusion and misunderstanding as much as possible.
Identifiers are an important concept in computing: They allow to reference and abbreviate things, and to introduce a level of indirection. Computer users who are not programmers are exposed to the concept of identifiers as well, through the filesystem abstraction of their operating system. Attempts to make this abstraction unavailable to the user in Apple’s iOS had to be rolled back in more recent versions via the introduction of a dedicated Files app.
Scriptable computer applications like Blender also make some of their internals available to power users for automation purposes via built-in scripting engines. These are of course just programming languages, and therefore heavily reliant on identifiers.
Even though artificial intelligence is revolutionizing the interaction between humans and computers, it is my conviction that identifiers will nevertheless gain importance as a concept and become even more mainstream than they already are. In many situations, use of a precise identifier just beats a hand-wavy negotiation with an AI, and that is not going to change.
Given the fundamental importance of identifiers, people must be able to express them in the script of their native tongue. This is self-evident in file systems: It would be unthinkable in a modern setting if people could only use Fortran identifiers as file names. It is also much easier to teach the use of identifiers to children if identifiers can be expressed in their native tongue.
Furthermore, it can be very convenient, especially in scientific applications, to be able to use symbols like π directly as identifiers, or to use them as part of identifiers, as in approximation-of-π.
Our design goal for identifiers consists therefore of three subgoals:
  1. They should be almost as simple as Fortran identifiers.
  2. They should support international scripts.
  3. They should support the use of symbols as (part of) identifiers.

Unicode Identifiers

Unicode, billing itself as the “World Standard for Text and Emoji”, recognizes the importance of identifiers and actually provides the concept of Unicode identifiers [2]. Modern programming languages like Swift have incorporated Unicode identifiers into their syntax: ℱ𝒪ℛ𝒯ℛ𝒜𝒩 is a perfectly valid identifier in Swift, and so is Φορτραν.
Nevertheless, this lexical diversity also comes at a prize. Having to deal with Greek letters in an application programming interface would inconvenience me very much, at least when they are used casually instead of just designating entities like π.
There is also a certain unwieldiness that Unicode brings to the table. Properties like boldness are part of the definition of some characters, and so you can have two different identifiers Fortran and 𝐅𝐨𝐫𝐭𝐫𝐚𝐧. This is messy, and very far from the clarity and simplicity that identifiers in Fortran provide. Granted, one will rarely encounter such a misuse of Unicode in practice, but why invite it in the first place?
Intuitively, there is a difference between names and symbols. That is the whole reason that most programming languages make this distinction in the first place. In my opinion, the concept of Unicode identifiers does not respect this distinction enough to be adequate in a programming language context. For example, the characters and 𝔽 are better treated as symbols than as arbitrary letters.
Clearly, while Unicode identifiers support international scripts, and can contain symbols, they are not as simple as Fortran identifiers at all, but a rather messy affair. Therefore Unicode identifiers only fulfill two of our three design goals for identifiers.

Cosmopolitan Identifiers

I propose cosmopolitan identifiers (CIDs) as the golden middle between Fortran identifiers and Unicode identifiers. A CID is a sequence of Unicode characters consisting of letters, digits and separators. The following properties hold for a CID:
A separator is a hyphen - U+002D. I favour the hyphen over the underscore for aesthetic reasons. Later  we will look at how it is possible to use other characters like underscores and spaces as separators as well.
A digit is a character between (and including) 0 U+0030 and 9 U+0039.
To fully define what a CID is, it is necessary to describe the set of letters, which sequences of letters are allowed, and to state when two CIDs are considered to be equivalent. All of this will be done in the following sections.
The set of letters contains initially just the set of lowercase letters from a U+0061 to z U+007A. We will later extend this set with Latin, Greek and Cyrillic characters. We will also extend it with letter-like mathematical symbols.

Equivalence of CIDs

Every CID is a Unicode identifier, but not every Unicode identifier is a CID. Also, equivalence of CIDs is defined differently than both canonical equivalence and compatibility [3] as defined in the Unicode standard. Two cosmopolitan identifiers which are canonically equivalent as Unicode identifiers are always also equivalent in the cosmopolitan sense, but not necessarily the other way around.
To decide equivalence of two CIDs, we map each CID cc to a normal CID N(c)N(c). Then two CIDs aa and bb are considered to be equivalent iff N(a)N(a) and N(b)N(b) are identical.
We construct N(c)N(c) by dividing cc into maximal non-empty subsequences s1,,sks_1, \ldots, s_k consisting either only of letters or only of digits. Separators are ignored themselves but serve to restrict how far subsequences of letters or digits can expand. We map each sis_i separately to its normal form NF(si)NF(s_i), and then form the CID NF(s1)NF(s2)NF(sk)NF(s_1){\includegraphics[height=0.5em]{8F8458B2-F168-44B0-A49C-A465EE0A8B6F}}NF(s_2){\includegraphics[height=0.5em]{60A7296E-6DDE-4569-A786-785D95EDB4D6}}\ldots{\includegraphics[height=0.5em]{324720C5-A6E1-415F-AA9C-1DAA34D0B05F}}NF(s_k) Note that if the normal form for one of the sis_i does not exist, i.e. is undefined, then the normal form for cc does not exist as well.
For a complete description of NN we therefore only need to describe NF(l)NF(l) for a sequence ll of letters, and NF(d)NF(d) for a sequence dd of digits.
Digits are easy to handle, we just set NF(d)=dNF(d) = d How we map general sequences of letters to their normal forms is described in the following sections. For a sequence xx consisting just of lowercase letters between a and z, we proceed just as we do for digits: NF(x)=xNF(x) = x As an example, consider the CIDs xyz12p5 and xyz-12p-5. They both decompose into the subsequences xyz, 12, p and 5, and have therefore the same normal CID xyz-12-p-5.

Words vs. Symbols

On one hand we would like to treat Tree and tree as equivalent identifiers. On the other hand it is often convenient to treat T and t as different symbols. After all, they look entirely different. This creates a tension which we resolve by distinguishing between three different classes of sequences of letters:
Dependent on class there are different normal forms NFword{NF}_{\text{word}} and NFsymbol{NF}_{\text{symbol}}, and we define NF(l)={NFword(l)NFsymbol(l) NF(l) = \begin{cases} {NF}_{\text{word}}(l) & {\includegraphics[height=0.69444em, totalheight=0.69444em]{E9816100-586A-4E58-81A1-DF90FBC83710}}\\ {NF}_{\text{symbol}}(l) & {\includegraphics[height=0.69444em, totalheight=0.69444em]{9263932A-0A35-4BEF-A175-22C2E943F33C}}\\ {\includegraphics[height=0.5em]{2C963A0C-94D5-4BBC-BF67-65B2E09C52BD}} & {\includegraphics[height=0.5em]{E1EC0528-2616-4E8A-BAC7-7462F9047B27}} \end{cases}
A symbol is any letter sequence ss such that NFsymbol(s){NF}_{\text{symbol}}(s) is defined. Currently all symbols consist of a single letter.
A word is any letter sequence ww such that NFword(w){NF}_{\text{word}}(w) is defined, non-empty, and not a symbol.
All letter sequences that are neither words nor symbols are invalid. For example, Ä is invalid, because it is neither a word (because NFword()={NF}_{\text{word}}({\includegraphics[height=0.5em]{3A0C0D5F-5F5B-4562-AD87-B072966D7331}}) = {\includegraphics[height=0.5em]{F6C6BAFF-935A-4133-81AC-EA893BBE9541}}, and A is a symbol) nor a symbol.

Extension Steps

We define the full set of cosmopolitan identifiers in several stages. Stage ii consists of a set of letters LiL_i, and normal forms NFword{NF}^{{\includegraphics[height=0.5em]{BF2180BA-B471-4740-96E4-517E127AAEBE}}}_{\text{word}} and NFsymbol{NF}^{{\includegraphics[height=0.5em]{2C97100D-B28D-4769-ADB6-586011FCFECE}}}_{\text{symbol}} operating on sequences in LiL_i^*.
Our initial set L0L_0 of letters consists of the lowercase letters from a U+0061 to z U+007A. The two normal forms NFword{NF}^{{\includegraphics[height=0.5em]{B3485C8D-F88F-4292-A9F5-6C1E33A0A216}}}_{\text{word}} and NFsymbol{NF}^{{\includegraphics[height=0.5em]{9E7CA55C-837C-4330-8EE9-233CE184FF8F}}}_{\text{symbol}} are defined for xL0x \in L_0^* by NFword(x)=xNFsymbol(x)={x \begin{array}{rcl} {NF}^{{\includegraphics[height=0.5em]{DD272A4A-57E7-40D0-A2D2-BAB58A02B5BA}}}_{\text{word}}(x) & = & x \\ {NF}^{{\includegraphics[height=0.5em]{BF4F16D9-49B1-400E-94A4-E4F6C65DBBAE}}}_{\text{symbol}}(x) & = & \begin{cases} x & {\includegraphics[height=0.43056em, totalheight=0.43056em]{E6EAFAE2-ACD1-4529-9288-62D0FDE7583E}} \\ {\includegraphics[height=0.5em]{D38D665C-F087-45E5-B73C-8462FCCC48BC}} & {\includegraphics[height=0.5em]{1D339D67-1329-452D-B625-6EA66411D6C1}} \end{cases} \end{array} In short, the symbols of stage 0 are those sequences in L0L_0^* consisting of a single letter, and the words are all sequences in L0L_0^* that have at least two elements.
Starting from this initial stage 0, we proceed as follows:
  1. Stage 1 adds Latin letters .
  2. Stage 2 adds Greek letters .
  3. Stage 3 adds Cyrillic letters .
  4. Stage 4 adds letter-like mathematical symbols .
Each extension step from stage ii to stage i+1i+1 follows these rules:
The following sections describe these four extension steps. They take advantage of Unicode features that we will be looking at first.

Primary Characters and Allowed Marks

Unicode characters come in the form of grapheme clusters [4], which are certain sequences of Unicode codepoints. For example, the Unicode character ä is represented by the grapheme cluster U+0061 U+0308, consisting of the codepoint a U+0061 and the codepoint  ̈ U+0308 which is called a combining diacritical mark. The character ä can also be represented by the grapheme cluster consisting of only the single codepoint U+00E4, so the same character can have multiple different representations as grapheme clusters.
Each Unicode character uu has a canonically decomposed grapheme cluster normalform p d1dnp\ d_1 \ldots d_n called NFD. We call pp the primary character of uu, and each did_i a mark of uu. In the following extension steps, we are only interested in those letters uu where all marks are combining diacritical marks.
Furthermore, I have observed that only a subset of all combining diacritical marks actually occurs in letters of modern alphabets, and therefore only the marks listed in Appendix M  are allowed.

Stage 1: Latin Script

The set L1L_1 consists of all unicode characters uu such that all marks of uu are allowed, and such that the primary character pp of uu is listed in Appendix L . The normal form NFword(u){NF}^{{\includegraphics[height=0.5em]{E096B968-88B5-485C-9931-A8FB24684DC2}}}_{\text{word}}(u) is the translation of pp as shown in Appendix L . Note that all marks of uu are dropped for translation.
For letter sequences w=u1unL1,w = u_1 \ldots u_n \in L_1^*, NFword(w){NF}^{{\includegraphics[height=0.5em]{13C225B4-C1EB-4D3C-9E22-A0E7C596DF15}}}_{\text{word}}(w) is defined transliterally, i.e. NFword(w)=NFword(u1)NFword(un).{NF}^{{\includegraphics[height=0.5em]{05AABC67-1CAE-4166-8D3F-0B0DFA867855}}}_{\text{word}}(w) = {NF}^{{\includegraphics[height=0.5em]{B2DC1D55-1DBD-408E-9E8F-F31C1D8D34A3}}}_{\text{word}}(u_1) \ldots {NF}^{{\includegraphics[height=0.5em]{B61EA042-AA31-4C9E-A237-F5E35C17A7EC}}}_{\text{word}}(u_n).
The symbols ss of stage 1 are listed in Appendix LS . They are all new symbols, which is another way of saying that NFsymbol(s)=s{NF}^{{\includegraphics[height=0.5em]{A7DCA36F-BD3E-4F03-A0DB-B568D20FD0B5}}}_{\text{symbol}}(s) = s holds.
For example, Lothar-Matthäus-10 is a valid cosmopolitan identifier with normal form lothar-matthaus-10. The normal form Lothar-M is also a CID with normal form lothar-M.

Stage 2: Greek Script

Support for Greek characters is based on ISO 843:1997 [5] transliteration.
This step adds all unicode characters uu such that all its marks are allowed, and such that its primary character is listed in Appendix G .
Certain 2-letter combinations must be considered specially when translating a word ww to NFword(w){NF}^{{\includegraphics[height=0.5em]{342E204A-C37E-40A4-975C-E2A2AE5679E4}}}_{\text{word}}(w). Possible marks do not matter as long as they are allowed, and are simply dropped during translation:
Otherwise, if none of the above situations apply, we translate single Unicode characters with their primary character listed in Appendix G  as described there, and by dropping all marks.
Words are translated transliterally based on their division into 2-letter combinations and single characters.
The Greek symbols are listed in Appendix GS . They are all different from each other, but some of them are equivalent to Latin symbols. For example Β U+0392 is equivalent to B U+0042, i.e. NFsymbol()={NF}^{{\includegraphics[height=0.5em]{52F7DA39-73DE-4084-829C-8CCF201ADE18}}}_{\text{symbol}}({\includegraphics[height=0.5em]{2703F3F5-6EB8-48B8-9FA1-05BA6A057198}}) = {\includegraphics[height=0.5em]{1A8FD36D-8CC8-4EBF-9CE6-6F501524EE1D}}. On the other hand Γ U+0393 is a new symbol, and thus NFsymbol()={NF}^{{\includegraphics[height=0.5em]{3AAF77FE-7F7A-42DC-B9CE-67880108FEC4}}}_{\text{symbol}}({\includegraphics[height=0.5em]{9FE35AD4-B0AE-49BA-942A-622997BA21ED}}) = {\includegraphics[height=0.5em]{D9E68598-D349-4ADA-B2C7-F346CBFA5191}}.
For example, Μπέχρος is a CID, and translates to the simple identifier mpechros.

Stage 3: Cyrillic Script

Support for Cyrillic characters is added by applying GOST 7.79-2000 System B [6].
This step adds all unicode characters uu such that all its marks are allowed, and such that its primary character is listed in Appendix C .
There are a few special cases to consider where the translation from uu to NFword(u){NF}^{{\includegraphics[height=0.5em]{F2E0BFB1-7985-4A0F-A3D9-AC25EB8EEAD7}}}_{\text{word}}(u) does not rely only on the primary character of uu, but also on its marks:
All other Unicode characters with their primary character listed in Appendix C  are translated by dropping all marks and performing a translation of the primary character as described in Appendix C .
Words ww are then translated transliterally to NFword(w){NF}^{{\includegraphics[height=0.5em]{97CB37DE-104B-4958-8663-75FD15592CDB}}}_{\text{word}}(w).
The Cyrillic symbols are listed in Appendix CS . Some of them are equivalent to each other because upper and lower case letters are just scaled versions of each other. Some of them are furthermore equivalent to Latin and/or Greek symbols.
For example, Андре́й-Никола́евич-Колмого́ров is a CID, and translates to the normalform andrej-nikolaevich-kolmogorov.

Stage 4: Mathematical Symbols

This step is adding letter-like mathematical symbols as listed in Appendix MS . We have NFword=NFword{NF}^{{\includegraphics[height=0.5em]{DCC794E5-1735-4710-8B38-52E16FB4C335}}}_{\text{word}} = {NF}^{{\includegraphics[height=0.5em]{3FCB3234-6955-4AC3-BD3F-CA50D1CCB63C}}}_{\text{word}}, and NFsymbol{NF}^{{\includegraphics[height=0.5em]{E38D1771-3154-4E98-BAA7-563AC4305DE9}}}_{\text{symbol}} extends NFsymbol{NF}^{{\includegraphics[height=0.5em]{E6EBED5C-6A49-49C0-9BC1-96CFBB5BAC5A}}}_{\text{symbol}} by acting as the identity on all symbols in Appendix MS .
For example, , ℕ1 and ℕ-ℕ are CIDs, but ℕℕ is not.

Separators and External CIDs

Although we have defined the hyphen - U+002D as the only possible separator, in practice we may want to use a different set of separators, depending on the situation. We might for example want to use the underscore _ U+005F instead, or also allow spaces U+0020 as separators.
We can accomodate for this by using external CIDs. What exactly an external CID is depends on your situation. All you need to do is to provide a description of the syntax of an external CID, and how to translate it to an ordinary CID.
For example, we might define an external CID to be a sequence of letters, digits, hyphens, underscores and spaces such that after a cleanup it becomes a CID. Here we define a cleanup as making the following modifications in order:
  1. Trimming spaces from the left start and right end.
  2. Replacing consecutive spaces with a single space.
  3. Replacing spaces and underscores with hyphens.
In this example, x z_9-t would be a valid external CID, corresponding to the CID x-z-9-t. On the other hand, x - z would not be a valid external CID, as after cleanup it becomes x---z, which is not a CID.

Reference Implementation

A reference implementation of cosmopolitan identifiers as described in this paper is available at https://github.com/phlegmaticprogrammer/CosmopolitanIdentifiers.

Conclusion

Cosmopolitan identifiers retain much of the simplicity and clarity of Fortran identifiers, but allow users to use their native scripts and letter-like math symbols. This is achieved by mapping each cosmopolitan identifier to a normal form which is basically a Fortran identifier, apart from the fact that it can also include symbols.
We have proceeded in four stages to define cosmopolitan identifiers. Ideally, in your application you would use CIDs as defined here. But if that is not possible for some reason, then CIDs might be adaptable to your needs by adding more stages, for example to allow more symbols to function as identifiers.
While it is possible to use equivalent but not identical identifiers in the same context, this is not recommended. For example, dx and Δx are equivalent, but not identical. Obviously, one should not write expressions like dx * Δx, but write either Δx * Δx or dx * dx. Ideally, these are not just conventions, but part of the definition of programming languages based on cosmopolitan identifiers, which would issue warnings or even errors in such situations. On the other hand, when accessing identifiers from another context it is OK to change them. For example, when accessing a library which exposes an API based on Greek identifiers, it is OK to use equivalent Latin identifiers at the calling site.
Currently the supported scripts are Latin, Greek, Cyrillic and Math. It would be great if it would be possible to extend cosmopolitan identifiers to other widely used scripts without compromising their conceptual and technical simplicity.

References

[1]Stephen J. Chapman. (2017). Fortran for Scientists and Engineers, 4th Edition.
[2]Mark Davis (ed.). (2020). Unicode Identifier and Pattern Syntax, Unicode Standard Annex #31, https://unicode.org/reports/tr31/.
[3]Ken Whistler (ed.). (2020). Unicode Normalization Forms, Unicode Standard Annex #15, https://unicode.org/reports/tr15/.
[4]Mark Davis, Christopher Chapman (eds.). (2020). Unicode Text Segmentation, Unicode Standard Annex #29, https://unicode.org/reports/tr29/.
[5](1997). ISO 843:1997: Information and documentation — Conversion of Greek characters into Latin characters, International Organization for Standardization, https://www.iso.org/standard/5215.html.
[6](2002). GOST 7.79-2000: System of standards on information, librarianship and publishing. Rules of transliteration of Cyrillic script by Latin alphabet, https://runorm.com/catalog/1004/741213/.

Appendix M: Allowed Marks

Appendix L: Latin Primary Characters

Appendix G: Greek Primary Characters

Appendix C: Cyrillic Primary Characters

Appendix LS: Latin Symbols

Appendix GS: Greek Symbols

Appendix CS: Cyrillic Symbols

Appendix MS: Math Symbols