PURPOSE OPERATION OPTIONS COMMAND LINES PARAMETER FILE RELATED PROGRAMS
One liner: Takes pipe ( | ) delimited | file, and creates fixed width fields to form fixed length records.
The PIPEFIX program is designed to work with files that contain variable length records containing pipes as field delimiters. (If you have a comma delimited csv file, see the program csv2pipe for how to swap the csv for pipe). This delimited type of file is most often a result of an unload of a database. Most notably, X-WAYS, Forensic Explorer, ACCESS, INFORMIX, ORACLE, and DBASE can create this type of output.
If the user knows what the record schema is, (record layout), you can use pipefix to make the records a fixed length, with each field also being a fixed length. Then the output file can be processed as any other fixed length record. (NOTE: almost all Maresware software requires input records to be fixed length. So use this program to reform records for re-processing by other Maresware software.)
The input records, MUST end with some sort of a carriage return indicator.(CR/LF, CR, or LF).
The user can also trick the program into making files with carriage returns (variable length records) look as if each record contained a single field, and cause pipefix to make the output record, one fixed length field. Thus ending up with fixed length records.
To run the program, a (text) parameter file is created. Maresware "parameter" files are used to tell each program what the layout or format of the input record is, and how to process it. Because each program operates differently, each parameter file is designed to tell the program how to treat the input record.
For pipefix, each line in the paramater file is a number representing the final length of that field in the record. Depending on the length of the field asked for in the parameter file, the input record field can either be truncated, expanded, or eliminated completely from the output record.
The output fields can also be left, or right justified, and 0 (zero) or blank filled depending on the modifier added to each parameter line.
The program first reads the parameter file, which contains information regarding how to process each field of the record.
It then reads the input record to each pipe which represents a corresponding field of the parameter file. The program then either truncates, expands or eliminates that field while adding the proper padding characters as necessary to create a fixed length field in the output record.
It is also, not uncommon to find about 1% bad records. These may not exactly be bad records, but what they actually are, are records that already contain a pipe in one of the character fields. This extra pipe causes the program to think it found a record with too many pipes and thus drops it as a bad record. At this point there is no fix for this situation.
NOTE:
Different databases may output the pipes in unusual seguences at the beginning and end of the records. For instance, one database places a pipe prior to each record, while another may place a pipe at the end of the record and just before the carriage return. If this is the case, you may have to trick the program into thinking there is another field in the record, and use a 0 length parameter line to bypass this extra field.
In other words, become very familiar with the input record before setting up your parameter file. It is suggested that you use a small subset of the input file and test your process prior to running the entire input file.
c:>pipefix input_filename output_filename parameter_file_name [options -A89Ppd ]
Parameter file contents are as follows:
One item per line, created with an ascii text editor. (edit, notepad, or a real editor).
Each line contains one or two items.
Sample records|one|two|three|four|five|six |Dan|Mares|1234|anycity|georgia|4049866955
The first item (which is mandatory) is the final size you wish this field to be in the output file. Use a 0 (zero) to completely eliminate this field from the output record. This 0 is also used in the situation noted above, where data bases add extra pipes before or after records. These are “fields” that must be eliminated.
The second item, (which is optional,) and immediately follows the first (no spaces between) the first, is an alpha code ( see below ) to justify each field.
Special item:
If you have a signficiant number of fields, but say, you only want the first X fields, then you can place
place the word END as an item and the program stops processing the record at this point. So if you had 15 or
so fields, and only wanted the first two, then the paramter file might look like this, and eliminates the
other 13 lines of 0 length:
10
11
END
Alpha Codes which modify output field format:
No alpha code results in a Default of ‘L’ which Left Justifies the field with a Blank Fill to Right.
An L = same as the default. Left justification, Blank fill to right. |DATA |
An l = Left Justification with Zero Fill to the right. |DATA000|
An R = Right Justification with Blank Fill to the left. | DATA|
An r = Right Justification with Zero Fill to the left. |000DATA|
An M = This field to be treated as a date, and you are certain the format is M/D/Y, like 2/12/2010 and convert to YYYY/MM/DD and Right justify. | 10/02/12|
An m = This field to be treated as a date, and you are certain the format is M/D/Y, like 2/2/2010 and convert to YYYY/MM/DD and Left justify. |10/02/2012 |
A Y = This field to be treated as a date, and you are certain the format is Y/M/D 2010/2/15 and convert to YYYY/MM/DD and Right justify. | 2010/02/12|
A y = This field to be treated as a date, and you are certain the format is Y/M/D 2010/2/15 and convert to YYYY/MM/DD and Left justify. |99/02/12 |
Note: for the MmYy codes above, the date can have either 2 digit or single digit M, MM, D, DD, and either 2 or 4 digit year 98, 1998, 10, 2010. The if you have a 10 for 2010, etc, the program makes a best guess at the formatA D = This field to be treated as a date, like 2/12/2010 and convert to YYYY/MM/DD and Right justify. | 2010/02/12|
A d = This field to be treated as a date, like 2/2/2010 and convert to YYYY/MM/DD and Left justify. |2010/02/12 |
Note: for the dD codes above, the program makes a best guess at the date format and works accordingly. If some of the date formats are not consistant with general style, the program may abort.Sample file to work with:
This would be the input data file with appropriate pipes delimeting fields.|one|two|three|four|five|six |Dan|Mares|1234|anycity|georgia|4049866955
Sample parameter file for the above records (pipes shown here to make it easier to read output):
0 /* there is a dummy first field prior to the initial "|" so ignore it and to be eliminated */ 2 /* make this output field 2 characters long. will truncate extra characters*/ 3L /* make this output field 3 characters, with left justify, blank fill */ 0 /* eliminate this 3rd field */ 4r /* 4 characters, with right justify, zero fill, thats why the five has 2 zeros padded */ 6l /* 6 characters, with left justify, zero fill */ 8R /* 8 characters, with right justify, blank fill */
The overall final record length will be 23 plus 2 for the carriage return and line feed.
Ouput would be (without the pipes):
on|two|four|five00| six Da|Mar|anyc|georgi|40498659
-A: create/append to default accounting output file called "ACCT-ING"
-P: maintain pipes in output file
-d + delimiter: If the input file is pipe delimited, then the -d option is not needed. If the input file is not pipe delimited, indicate what delimiter to use in one of the following formats.
Enter the delimiter as a single character. It may be necessary to quote “|” the character if the character has a special meaning to DOS. As DOS may not pass it directly to the program. If the character is not a special DOS character merely enter it. Be especially careful if the file is “,” comma delimited. As some fields may also have imbedded commas which will give bad results. For csv, comma delimited files, run the file thru csv2pipe first to change the csv to pipes. Can't tell you how many times, forensic software programs have output fields (such as filenames) with embedded commas and have caused other software to die.
If you need to enter a delimiter that is not printable, say for instance the tab character (hex 09) is used as a delimiter then you can enter the ascii value of the delimiter. This must be entered as either a 2 or 3 digit number. So single digit numbers like 9 (tab) must have a leading 0. This would look like -d 09.
CSV2PIPE : convert a "," csv file to pipe delimeted.