Previous Blog: A Brief History – the Beginning

Linguistics – Language Invention

If I tell you that language can be invented, you probably won’t be surprised. You see all the time examples of how some parts of a language are changed, like new phrases being invented on social media. You and your friends may have some secret phrases referring to a memorable moment. J. R. R. Tolkien, the author of The Lord of the Rings, constructed languages that only exists in his fictional world, like Elvish, Dwarvish, and others. It’s really cool for one single person to be able to develop such a rich system of languages just for his novels, but what’s even more amazing is that Tolkien is not the only one with the ability to invent a language. In fact, everyone can, if they are put under the right environment at the right age.

Elvish on the One Ring from The Lord of the Rings

What is language?

Before delving into the story of how a language can be created, let’s first talk about what exactly is language. If language is just a system for communication, then do other animals have language? Dogs bark to each other. Birds can sing. Bees have dancing as well as chemicals to tell others about food sources or enemies that could be miles away. So, what’s the difference? It is widely believed that only humans, with the largest brain to body weight ratio in all species, has the ability to learn and invent such a complex communication system. Even though some animals, like Kanzi and Chantek, learned human sign languages after training, their language ability only compares to that of a 4-year-old child. Sentience is the prerequisite of using a language. A language is a system expressive enough to communicate our sophisticated thoughts.

Chantek the orangutan who learned American Sign Language

Is Language Innate?

Humans have lots of things other animals do not enjoy. We can cook, write, do Math, farm, etc. Those are mostly considered technologies instead of innate human abilities. Is language one of those technologies discovered by someone through accident or is it more like the ability to walk – innate to all humans without formal teaching? So far, the innateness of language ability is still under debate among scientists, but there are a few evidences pointing out that at least some knowledge about language is written in our brain and not acquired through learning.

Universal Traits in All Human Languages

Isn’t that kind of amazing that no matter where we are born, we all walk more or less the same way? This fact indicates that there is something universal about how muscles and the brain work together when we walk. Similarly, for languages, if we can find some universal traits that are shared by all languages, then it is an indication that at least something is written in our gene about the language we use.

The Ethnologue catalogue of world languages, which is one of the best linguistic resources, says that there are around 6909 living languages in the world (From Number of languages). All of them are different in one way or another. For instance, the grammar structure of a sentence is not necessarily the same in different languages. In Japanese, the verb is put at the end of the sentence, but in English, the verb will be put between the subject and the object. Here’s an example:

I ate an apple.

私 は リンゴ を 食べた。

I (Topic marker) apple (object marker) ate.

As another example, according to Linguistics Society, in Welsh, the usual order is for the verb to come first, followed by the subject, followed in turn by the object:

The student bought the book.

prynodd y myfyriwr y llyfr

bought the student the book

Pronunciations of a language seems quite arbitrary as well, proven by the fact that different languages usually call animals by different names. In addition, words were probably pronounced differently centuries ago than they are now. Grammar is no exception. If you have ever learned grammar or tries to explain grammar to a non-native speaker, you’ll soon find out that some of it too seems arbitrary. “That’s just how English works.” I was often told when I learned English.

So out of the 6909 living languages, and even more dead ones, are there any traits shared by all of them?

There are. In fact, scientists have found a quite a few. Here are four of them:

• All languages have nouns, verbs, objects, and pronouns (like I, we, they).
• All languages have at least two vowels (vowels are like a, i, u, e, o etc.).
• All languages have at least three sizes of grammatical units: word, phrase, and clause.
• “If a language distinguishes dual number (a grammatical category indicating “two”) in pronouns, it also distinguishes plural number.” (From scribd.com).

Those facts seems to suggest that the basic structure of a human language is written in our gene. This is the central idea to Universal grammar proposed by Noam Chromsky, one of the most famous linguists in history. Even more, when brand new languages are invented, they follow the same commonalities.

Inventing a Language

In the 16th century, not long after the (re-)discovery of the New World by Columbus, the brutal slave trade that abducted 10 million Africans began. Those unfortunate African slaves who would spend the rest of their lives on American plantations came from all over Africa and did not speak the same language among themselves. Indeed, slaves from different regions or tribes were placed together exactly because they cannot communicate with one another – no communication, no rebellion. To work with other slaves and receive orders from their masters, most first-generation African slaves picked up pieces of language from the slave masters. These were short phrases, words, and sentence fragments that had limited vocabulary and no unified grammar structure.

This kind of broken language is called a pidgin. Pidgin languages can be found everywhere around the world when two or more groups of people interact with one another without a common language base. Words are borrowed from other languages and are adapted to serve new purposes.

According to eyeofhawaii.com, in Hawaiian pidgin language, the word “brah” (which is also used in contemporary English slangs) means “brother” and the word “cockaroach” means “to steal”.

Back to the American plantations. When slaves married, usually also between two people speaking different mother tongue, the couple communicated with each other and with their children using a pidgin. All the children growing up listening to pidgin then did something that fascinated linguists.

When the children heard those fragmented words and phrases, they spontaneously tried to fill in the missing grammar parts. As an example, if their parents’ pidgin sentence seemed to be missing an object that was implied, the kids would fill them in. If the parents did not know the word for some object, the kids tried to reuse and combine other words. According to The Language Instinct, simple verbs in pidgin language such as “go”, “stay”, and “came” are used systematically in Hawaiian Creole grammar as auxiliaries, prepositions, case markers, and relative pronouns. Moreover, “The English past tense ending -ed may have evolved from the verb do: He hammered was originally something like He hammer-did.”

Furthermore, when the slaves’ children got together, their languages started to merge and form a new language. What if the kids didn’t like how a word sound to them? They just came up with a new one and started using it. If they found a grammar to be counterintuitive, they spoke in whatever grammar that felt right to them. This kind of language is called creole – language spoken by people whose mother tongue is pidgin. Those children invented their own brand new language within a generation, with their unique set of words, new grammar, and new group of people to speak it.

Image of a slave family

• Those children of slaves are the inventors of new languages. *

  

Creole language has made a deep impact on how we use language today. Some linguists believe that the Black English widely spoken today among African Americans, also known as Black Vernacular English(BVE), was probably an English-based creole language. Phrases like “Don’t nobody know the answer, Ain’t nothing going on.” is grammatically correct in BVE, but not in standard English. Creole Languages are excellent examples of how languages can be borrowed, created, and adapted constantly. And guess what, all Creole languages also adhere to the same set of traits common to all other human languages, even though most of their inventors did not even go to elementary school.

In the next blog of the Linguistics series, I plan to show some interesting facts about children learning language, which will provide some insights to how human perceive language and how we can use that information for NLP research. If you have topics that you’d like to read about, just let me know. Thanks for reading!

Citation

The Language Instinct

History of African American English in the U.S.

Pidgin language example

Nigerian Pidgin Wiki

Image source:

The Ring

Chantek

Slave family

Previous Blog

This is the dog, that worried the cat, that killed the rat, that ate the malt, that lay in the house that Jack built.

Mother Goose, The house that Jack built

This is the malt that the rat that the cat that the dog worried killed ate.

Victor H. Yngve (1960)

Both quotes are taken from Speech and Language Processing by Jurafsky and Martin

A Brief History – the Beginning

1949

It was 1949. The Second World War had been over for 4 years, yet the whole world was still recovering from its wounds. A lot of the first-generation computer scientists, including Warren Weaver, Alan Turing, and Claude Shannon worked on cryptography during the WWII. Their work helped the allies decipher important information which led to their final victory. Thus, those computer scientists were also on the cutting edge in communication, linguistics, as well as math. However, there was one problem: there were no longer more German or Japanese secret messages for them to work on. The 55-year-old Warren Weaver decided to put his knowledge to good use. Inspired by his cryptography research during the war, he proposed that all human languages are encrypted and they can be “decrypted” or translated automatically into other languages by machines, just like what was done during WWII. German was essentially encrypted English. Encouraged by his fellow mathematicians and scientists, he publishes his idea in a memorandum called “Translation”, from which the modern fields of Natural Language Processing(NLP) and machine translation were born.

Realizing the potential impact of NLP, the researches were funded by the US government in the hope that they can have a working system in a decade or two. After all, if a two-year old can understand languages, surely it can’t be difficult for a machine built and taught by the smartest minds in the world? The 1940s to 1960s period was filled with infinite trust in science and its ability to bring progress to human society, whether in the past or in the future.

ELIZA

After a few years and millions of dollars, a chat bot named ELIZA was developed that stood for the best piece of NLP technology at the time. You can try a demo by visiting this page. A sample dialog can be found below:

Person : Men are all alike.

ELIZA : In what way?

Person : They’re always bugging us about something or other.

ELIZA : Can you think of a specific example?

Person : Well, my boyfriend made me come here.

ELIZA : Your boyfriend made you come here?

Person : He says I’m depressed much of the time.

ELIZA : I am sorry to hear that you are depressed.

Person : It’s true. I am unhappy.

ELIZA : Do you think coming here will help you not to be unhappy?

Person : I need some help, that much seems certain.

ELIZA : What would it mean to you if you got some help?

Person : Perhaps I could learn to get along with my mother.

ELIZA : Tell me more about your family

If you spend 5 minutes playing around with it, you’ll soon notice that something was not quite right. ELIZA tend to repeat itself a lot.

ELIZA works by finding simple patterns and key words in the provided inputs. For example, if ELIZA sees the word “alike” or “same”, the reply will be “In what way?”. If it sees the pattern “My {family-member} is Y”, ELIZA will ask “Who else in your family is Y?”. Whenever it cannot find the pre-programmed key words and have no idea what you just said, it will reply “I see.”. It is a program designed specifically for the Turing Test without actually understanding anything about the conversation. The only thing it needs to do was to trick humans into thinking that they are talking with another human across the screen. Natural language was simply too much for a computer at that time to really handle. After all, computers were built first to, well, compute. They were good at math but not too much into talking.

Turing Test Diagram:

The Turing Test is a test for machine intelligence. If the judge cannot tell whether the response comes from a machine or from a human, then the machine passes the test.

What was preventing the best scientists from inventing a translation machine? As the researchers approach this unknown field, they found two great walls standing between human language capacity and that of a computer, namely ambiguity and infinity.

You: Call me a cab.

Bot: Ok, you are a cab.

Ambiguity

The inherent ambiguity in language is everywhere, yet we seldomly notice it, until you work with a machine. One joke popular among linguists goes like this: “One morning I shot an elephant in my pajamas. How he got in my pajamas, I don’t know.” Here, the joke plays around with the ambiguity in English. Am I the one wearing my pajamas or is that the elephant? Both explainations are possible, albeit the second one is only more likely in a Disney movie where elephants wear pajamas. We humans don’t even realize the ambiguity.

Here’s another example. If you speak any language other than English, I invite you to pull up your favourite translation software or website and type in the following sentence: “I can’t believe my 15-year-old son grew another foot last summer.” I haven’t found one single translation system that will give me the right translation on this sentence yet. (If you have, I’ll be super interested to know. And for those confused, “foot” here is used as a length unit instead of as a physical part of an animal.)

Infinity

I call the second obstacle “infinity” to suggest the high complexity of any human language system. Different from ambiguity where more than one explainations can makes sense and we’re asked to pick the one that the speaker most likely meant, infinity is more like finding the potentially correct ones out of the other millions of nonsense ones.

When I say millions, I literally mean millions. Consider an English sentence with 15 words. Assuming that each word has 3 different possible meanings and/or part of speech, we’re facing 3¹⁵, which is around 14 million possibilities. And that’s just for one sentence. For comparison, the IBM 7090 in 1960 selling for $2.9 million ran at 0.1 Million FLOPS. Reasonable approximations will tell you that it will take from around 2 minutes to 1 hour to process just one sentence at that rate.

What further complicate things is that sentences can be arbitrarily long. One example would be the sentence quoted at the very beginning of this blog: “This is the dog, that worried the cat, that killed the rat, that ate the malt, that lay in the house that Jack built.” In theory, a sentence can be constructed infinitely long by adding additional parts onto it, although in practice people seldomly do so.

The NLP Ice Age

These two huge walls, ambiguity and infinity, still stand tall and firm before every NLP researcher today, casting a long shadow over the NLP community. In 1966, the Automatic Language Processing Advisory Committee (Alpac) “publishes a report on Machine Translation concluding that years of research haven’t produced useful results” and the US government decided to cease funding NLP researches (Cited from Machine Translation’s Past and Future). It was a bleak time for NLP.

The question I’ll try to answer in my next blog is, how and when did things get better? Thanks for reading.

Citation

I’ve gathered information and drew inspiration from the following sources:

Speech and Language Processing by Daniel Jurafsky & James H. Martin

Portland University ECE 478/578/678 slides

Introduction to Computational Linguistics and Natural Language Processing by Mark Johnson

Natural language processing: a historical review by Karen Sparck Jones

2. Natural Language Processing (NLP) – Tees

Machine Translation’s Past and Future

In the future, I will add on new citations in addition to this list.

Image sources:

Turing Test

WWII Cryptography Enigma machine