Lexical analysis, lexing or tokenization is the process of converting a sequence of characters (such as in a computer program or web page) into a sequence of tokens (strings with an assigned and thus identified meaning). A program that performs lexical analysis may be termed a lexer, tokenizer, or scanner, though scanner is also a term for the first stage of a lexer.
You will create a Lexical Analyzer that will process commands (parse and tokenize) from a source file.

Lexical analyzer Game C++
Lexical analysis, lexing or tokenization is the process of converting a sequence of characters (such as in a computer program or web page) into a sequence of tokens (strings with an assigned and thus identified meaning). A program that performs lexical analysis may be termed a lexer, tokenizer, or scanner, though scanner is also a term for the first stage of a lexer.
You will create a Lexical Analyzer that will process commands (parse and tokenize) from a source file. You will create several classes to extract information from a source file.
You will code an “Analyzer” class that will create objects necessary to running the program. In addition, you will implement debug functions to display the full structure of the story. The expected output for this part of the project is described in the Output section.
1 Details
Reading in and interpreting text is often referred to as parsing. The first step in parsing input is to tokenize the input; that is, break it down into smaller chunks, called tokens, which can be analyzed, and the string of tokens can then be analyzed by the parser.
2 Output
Each passage and passage part in the story should be able to output information about itself, in turn, in order to demonstrate that you have successfully encoded the story into a collection of objects capable of displaying the story to a reader. This process will be initiated by calling the print() function of your Analyzer class. The output requirements for each part of the story are described in the sections below.
2.1 Analyzer class
The analyzer class should print out the information for all of the passages in the story, in order.
2.2 Passage
A passage should print out the line
Passage [name]:
where you should replace [name] with the name of this passage. Afterwards, the passage should print the information for all of its parts, in order.
2.3 Text
A section of text should print out all of the text it contains inside quotation marks (including new lines or any other white space characters), and it should print a new line after the end quotes.
2.4 Links
A link should print out the line
Link: display=[display], target=[target]
where you should replace [display] with the text that should appear when the link displays and replace [target] with the name of the passage that this link links to.
2.5 Go-to
A go-to command should print out the line
Go-to: target=[target]
where you should replace [target] with the name of the passage that this go-to command links to.
2.6 Set
A set command should print out the line
Set: var=[var], value=[value]
where you should replace [var] with the name of the variable that this set command alters and replace [value] with the Boolean value (true or false) that the variable should be assigned to.
2.7 If
An if command should print out the line
If: var=[var], value=[value]
where you should replace [var] with the name of the variable that this if command checks and replace [value] with the Boolean value (true or false) that the variable is compared against.
2.8 Else-if
An else-if command should print out the line
Else-if: var=[var], value=[value]
where you should replace [var] with the name of the variable that this else-if command checks and replace [value] with the Boolean value (true or false) that the variable is compared against.
2.9 Else-if
An else command should print out the line
Else
2.10 Block
A block should print out the line
START BLOCK
Afterwards, it should print out all of the parts that it contains, in order, then print the line
END BLOCK
3 Specifications
This section will detail the various components for the lexical analyzer. It details how these parts should act and connect together to form a story. Your implementation for this project should support the features described below.
3.1 The Analyzer
Your story Analyzer should start by opening the file source.html and reading in the story data, as described in the sections above. You may assume that the input does not contain any syntax or logical errors (e.g., testing a variable in an (if:) command before it has been (set:)). Once it has constructed objects to represent the structure of the story, the analyzer should start by displaying the first passage defined in source.html. When displaying a passage, your program should execute all commands appropriately and properly display all links.
Your analyzer will need to keep track of the variables that have been defined, as well as their values. You should use an unordered_map (Standard Template Library) object for this purpose (#include ); you do not need to define a specialized class to match variables to their values.
After displaying a passage, the analyzer should print out a numbered list of all links in the passage, and prompt the reader to select one.
When displaying a passage, your program should execute any commands that appear in that passage and properly display all links (see Section 3.3 below). Afterwards, the analyzer should print out a list of all links in the passage and prompt the reader to select one. This list should start numbering at 1; e.g., Once the reader has selected a link, your analyzer should display the given passage (see Links section below). This should continue until the reader reaches a passage with no links, at which point the analyzer should terminate.
3.2 Passages
The theme divided into passages, which appear in the HTML tags . Each passage will start with and will end with . In addition to starting with <tw-passagedata, the opening tag will specify some attributes, like pid, name, tags, and location, and the body of the passage will be between the opening and closing tags.
Example passage:
The body of the passage will be here.
Your analyzer only needs to pay attention to the name attribute of each passage, the other attributes can be safely ignored. In the body of a passage, there are 3 different types of elements to deal with: text, links, and commands. When a passage is being displayed, text should appear as typed (including spacing), however, links will display differently than they appear in the input file, and commands can cause a variety of different things to happen.
3.3 Links
Links in the body will be denoted by double brackets: [[ and ]]. Links are treated differently depending on whether or not they contain the characters ->. A link that doesn’t contain these characters should appear in the text without the double brackets, and this link should be presented to the reader as an option to further the story after the passage has displayed. When selected, the link should display the passage with the name that matches the link text.
A link that contains the characters -> should display as the characters to the left of ->; however, it should link to the passage whose name matches the characters to the right of the ->.
Example links:
[[Simple]]
Displays as \Simple”; links to passage named \Simple”
[[Take the blue pill->Bad dream?]]
Displays as \Take the blue pill”; links to passage named \Bad dream?”
3.4 Commands
Commands are denoted by a single word and a colon immediately after an open parenthesis. Your story Analyzer should support 5 different commands, (go-to:, (set:, (if:, (else-if:, and (else:.
3.5 Go-to command
The go-to command should cause the program to immediately change to the given passage. Any other text, commands, or links that appear after the go- to command should be ignored, and any links before the go-to can be safely ignored, as though the reader had selected the given passage as their choice. The name of the passage to go to will appear between two copies of “.
Example go-to command:
(go-to: “start”)
Continues the story with the passage named “start”, as though the reader had selected a link leading to this passage.
3.6 Set command
The (set: command allows the author to define and set the value of a variable. Note that the (set: command will never display any text; however, it will execute any time a passage containing it is
displayed. Variables that do not exist are created, while variables that do exist are updated. While the full specification for Harlowe allows for three different types of variables (numeric, string, and Boolean), you should treat all variables as Boolean.
The first word after the colon in the (set: command will be a variable name, which always starts with $. The second will be the keyword to, and the third will be the assigned value (true or false).
Example set command:
(set: $ateCake to true)
Stores true as the value of the $ateCake variable
3.7 If/Else if/else
The (if:, (else-if:, and (else: commands act much like they do in C++. The (if: and (else-if: commands will be followed by a variable, the keyword is, and a value to test the variable against, followed by the closing parenthesis. (else:) has no condition. The blocks that if, else-if, and else apply to are denoted by brackets []. Note: links and other commands (including other (if: commands) may be embedded in these blocks. While BlockTokenizer extracts these blocks as a single token, you can create another BlockTokenizer to parse the contents of the block.
Example if command
(if: $ateCake is true)[You are quite full.]
(else-if: $ateCookie is true)[You sigh contentedly.]
(else:)[You still feel a mite peckish.]
Summary
Files are divided in two section
• Support files: all the files needed to start working
• program files: the main file used to run the program and the source.html with its expected result in sourcehtml.txt
It’s recommended to opening the source.html file inside the constructor of the Analyzer class and then perform the indicated operations.