How to split/tokenize a string? (strtok() string tokenizer)

We often deal with strings in a C program and we want to split the strings into small tokens on the basis of delimiter. We can do this easily. Lets scan each character of the string from the beginning and put NULL whenever it matches the delimiter character. We have a static variable to store the last token position. This advances as we obtain a token and move to the next posotion. Below is a stepping diagram and C code of this process. 

Input String:
"A Quick Brown Fox Jumps over the lazy dog."

Tokenize process:

Token 1: "A\0Quick Brown Fox Jumps over the lazy dog."
  Position^
Token 2: "A\0Quick\0Brown Fox Jumps over the lazy dog."
     Position^
Token 3: "A\0Quick\0Brown\0Fox Jumps over the lazy dog."
            Position^
Token 4: "A\0Quick\0Brown\0Fox\0Jumps over the lazy dog."
                   Position^
Token 5: "A\0Quick\0Brown\0Fox\0Jumps\0over the lazy dog."
                        Position^
Token 6: "A\0Quick\0Brown\0Fox\0Jumps\0over\0the lazy dog."
                               Position^
Token 7: "A\0Quick\0Brown\0Fox\0Jumps\0over\0the\0lazy dog."
                                     Position^
Token 8: "A\0Quick\0Brown\0Fox\0Jumps\0over\0the\0lazy\0dog."
                                          Position^
Token 9: "A\0Quick\0Brown\0Fox\0Jumps\0over\0the\0lazy\0dog\0"
                                                Position^
char * strtok(char *str, const char * delim)
{
  int i = 0;
  int j, len;
  static char * last_pos;
  char *token;
  if (str) {
    last_pos = str;
  }

  token = last_pos;
  len = strlen(delim);

  while(*last_pos){
    for(j = 0; j < len; j++) {
      if (delim[j] == *last_pos) {
        *last_pos = '';
        last_pos++;
        return token;
      }
    }
    last_pos++;
  }
  return NULL;
}

We do not require to do all the steps as C runtime library has strtok() function to do all these for us. Here is a sample example program.

#include <string.h>
#include <stdio.h>
int main(int argc, char* argv[])
{
  int i = 0;
  char *str = "A Quick Brown Fox Jumps over the lazy dog.";
  char *tokens, *token;
  tokens = strdup (str);
  token = strtok (tokens, " .");
  while(token) {
    printf("Token %d: %s\n", i + 1, token);
    token = strtok(NULL, " .");
    i++;
  }
  free(tokens);
  return 0;
}
Output:
Token 1: A
Token 2: Quick
Token 3: Brown
Token 4: Fox
Token 5: Jumps
Token 6: over
Token 7: the
Token 8: lazy
Token 9: dog

Note: Buffer allocated by strdup should be released with free(). strtok(), strrev() etc are some utility functions modify its input buffer and produce the result. This is why we should not pass a constant string. CPU will produce a access violation while execution. strdup() is usedful for these cases.

About our authors: Team EQA

You have viewed 1 page out of 252. Your C learning is 0.00% complete. Login to check your learning progress.

‹ # ›

Questions index C Questions C++ Questions Win32 MFC COM/DCOM DLL Questions