We often deal with strings in a C program and we want to split the strings into small tokens on the basis of delimiter. We can do this easily. Lets scan each character of the string from the beginning and put NULL whenever it matches the delimiter character. We have a static variable to store the last token position. This advances as we obtain a token and move to the next posotion. Below is a stepping diagram and C code of this process.

Input String:
"A Quick Brown Fox Jumps over the lazy dog."

Tokenize process:

Token 1: "A\0Quick Brown Fox Jumps over the lazy dog."
  Position^
Token 2: "A\0Quick\0Brown Fox Jumps over the lazy dog."
     Position^
Token 3: "A\0Quick\0Brown\0Fox Jumps over the lazy dog."
            Position^
Token 4: "A\0Quick\0Brown\0Fox\0Jumps over the lazy dog."
                   Position^
Token 5: "A\0Quick\0Brown\0Fox\0Jumps\0over the lazy dog."
                        Position^
Token 6: "A\0Quick\0Brown\0Fox\0Jumps\0over\0the lazy dog."
                               Position^
Token 7: "A\0Quick\0Brown\0Fox\0Jumps\0over\0the\0lazy dog."
                                     Position^
Token 8: "A\0Quick\0Brown\0Fox\0Jumps\0over\0the\0lazy\0dog."
                                          Position^
Token 9: "A\0Quick\0Brown\0Fox\0Jumps\0over\0the\0lazy\0dog\0"
                                                Position^
char * strtok(char *str, const char * delim)
{
  int i = 0;
  int j, len;
  static char * last_pos;
  char *token;
  if (str) {
    last_pos = str;
  }
10 
11    token = last_pos;
12    len = strlen(delim);
13 
14    while(*last_pos){
15      for(= 0; j < len; j++) {
16        if (delim[j] == *last_pos) {
17          *last_pos = '';
18          last_pos++;
19          return token;
20        }
21      }
22      last_pos++;
23    }
24    return NULL;
25  }
26 

We do not require to do all the steps as C runtime library has strtok() function to do all these for us. Here is a sample example program.

#include <string.h>
#include <stdio.h>
int main(int argc, char* argv[])
{
  int i = 0;
  char *str = "A Quick Brown Fox Jumps over the lazy dog.";
  char *tokens, *token;
  tokens = strdup (str);
  token = strtok (tokens, " .");
10    while(token) {
11      printf("Token %d: %s\n", i + 1, token);
12      token = strtok(NULL, " .");
13      i++;
14    }
15    free(tokens);
16    return 0;
17  }
18  Output:
19  Token 1: A
20  Token 2: Quick
21  Token 3: Brown
22  Token 4: Fox
23  Token 5: Jumps
24  Token 6: over
25  Token 7: the
26  Token 8: lazy
27  Token 9: dog
28 

Note: Buffer allocated by strdup should be released with free(). strtok(), strrev() etc are some utility functions modify its input buffer and produce the result. This is why we should not pass a constant string. CPU will produce a access violation while execution. strdup() is usedful for these cases.

You have viewed 1 page out of 248. Your C learning is 0.00% complete. Login to check your learning progress.

 Vote 0

Similar topics related to this section

strcpy and strcat source, memcpy vs memmove, strrev source, strdup, strtok, macro definition, C exception handling, setjmp and longjmp,

# C Programming Language (Prentice Hall Software)
# Let Us C Paperback - 2006 by Yashavant Kanetkar
# Understanding and Using C Pointers Core techniques for memory management
# Data Structures Using C and C++ Paperback - 1998
# Data Structures In C Paperback - August 11, 2008 by Noel Kalicharan