Exercise: Lexical Analysis

NOTE: This is an in-class exercise. No submission is required.

Description

Write a program in Clojure that receives the name of a text file containing arithmetic expressions and comments as input. The program should print a table with each of the lexical units (tokens) found, in the order they were encountered, indicating their category. The program must be implemented using the Regular Expression API.

Token Categories

Arithmetic expressions may only contain the following categories of tokens:

• Integers
• Floats (reals)
• Operators:
  • Assignment (=)
  • Addition (+)
  • Subtraction (-)
  • Multiplication (*)
  • Division (/)
  • Power (^)
• Variables
• Special Symbols:
  • Left parenthesis [(]
  • Right parenthesis [)]
• Comments:
  • // followed by characters until the end of the line.

Input

• The name of a text file. This file must contain one or more arithmetic expressions, one per line.
• Tokens may be separated by zero or more blank spaces.

Example:

b=7

a = 32.4 *(-8.6 - b)/       6.1E-8

d = a ^ b // This is a comment

Output

The previous example should produce the following output:

========================================================
Token                           Category
========================================================
b                               variable
=                               assignment
7                               integer
a                               variable
=                               assignment
32.4                            float
*                               multiplication
(                               opening-parenthesis
-8.6                            float
-                               subtraction
b                               variable
)                               closing-parenthesis
/                               division
6.1E-8                          float
d                               variable
=                               assignment
a                               variable
^                               power
b                               variable
// This is a comment            comment
========================================================

Token Formation Rules

• Variables:
  • Must start with a letter (uppercase or lowercase).
  • Consist only of letters, numbers, and underscores (_).
• Floating Points (Real Numbers):
  • Can be positive or negative.
  • May or may not have a decimal part but must contain a decimal point (e.g. 10. or 10.0).
  • Can use exponential notation with the letter E (upper or lowercase); however, the E must be followed only by a positive or negative integer (e.g. 2.3E3, 6.345e-5, -0.001E-3, 0.467E9).
• Comments:
  • Start with // and everything following it until the end of the line is considered a comment.