Microsoft Access, Office/VBA, VB6 Modules royalty-free source code library

Microsoft Access Modules royalty-free source code library

Microsoft Access VBA and VB6 Module Library

Free trial of Total Visual SourceBook for Microsoft Access, Office, VB6 and VBA

Class: Tokenizer in Category General VBA/VB6 : String Handling from Total Visual SourceBook

Breaking and parsing a string into individual tokens with VB6 and VBA.

This tokenizer is implemented as a state machine. The characters used for whitespace, separators, quotes, and end-of-line are all implemented as properties. This allows for a great deal of flexibility in how text is tokenized.

Procedure List

Procedure Name	Type	Description
(Declarations)	Declarations	Declarations and private variables for the CTokenizer class.
ConvertCase	Property	Get the current type of case conversion. The ConvertCase property can have the following values: tokNone - No case conversion tokUpper - Convert to uppercase tokLower - Convert to lowercase
EOLChars	Property	Get the characters currently treated as end of line characters.
QuoteChars	Property	Get the characters currently treated as quotes.
SeparatorChars	Property	Get the characters currently treated as separators.
Text	Property	Get the text to tokenize.
WhiteSpaceChars	Property	Get the characters currently treated as white space.
Class_Initialize	Initialize	Initialize the class.
GetNextToken	Method	Get the next token in the string identified by the text property.
AppendToken	Private	Append the current character to the token, performing any necessary case conversion.
CharType	Private	Determine the type of the current character.
HandleEOL	Private	Handle an end of line character.
HandleQuote	Private	Handle a quote character.
HandleSeparator	Private	Handle a separator character.
HandleToken	Private	Handle a token character.
HandleWhiteSpace	Private	Handle a whitespace character.

Example Code for Using Class: Tokenizer

' Example of CTokenizer
'
' To try this example, do the following:
' 1. Create a new form
' 2. Add a command button named 'cmdTest'
' 3. Paste all the code from this example to the new form's module.
' 4. Run the form

Private Const mcstrText As String = "A-whop boppa lu-mop," & vbCrLf & "A whop bam boom"

Private Sub cmdTest_Click()

  ' Example for the CTokenizer class
  ' This example breaks a string up into its pieces and constructs a new string, separated by "."'s
  ' to display in a message box

  Dim tokenizer As CTokenizer
  Dim strBops As String
  Dim strTok As String

  Set tokenizer = New CTokenizer

  With tokenizer
    .Text = mcstrText
    .WhiteSpaceChars = " ,.:-"
    .ConvertCase = tokLower
    .EOLChars = vbCrLf
    Do While .GetNextToken(strTok, " ", False)
      strBops = strBops & "." & strTok
    Loop

  End With

  Debug.Print strBops

End Sub

Overview of Total Visual SourceBook

The source code in Total Visual Sourcebook includes modules and classes for Microsoft Access, Visual Basic 6 (VB6), and Visual Basic for Applications (VBA) developers. Easily add this professionally written, tested, and documented royalty-free code into your applications to simplify your application development efforts.

Total Visual SourceBook is written for the needs of a developer using a source code library covering the many challenges you face. Countless developers over the years have told us they learned some or much of their development skills and tricks from our code. You can too!