Tag Archives: FTP

Version 1.0.41 – Added Detail to Explain FTP Checksum non-Match

Posted on by

FTP Checksum validation was improved in Version 1.0.36; there was a typo, and really very little explanation.

Although there’s not a whole lot more detail, the reason for a Checksum Unknown was described with a hint as to how to use this capability in future.

As a note: the checksum file format intentionally matches the output of the “md5sum” or “md5” command line tools to which other tools are compatible. Most checksum tools are compatible.

Of note: using vict.jar to upload automatically produces checksum files for validation.

For example:

vict.bat -U (server) -u file1 -u file2 -u file3

or:

java -jar vict.jar -U (server) -u file1 -u file2 -u file3

This simple functionality creates a checksum, sends it to the server, uploads the file(s), and confirms checksum with the server (where supported).

The Server-side FTP check functionality that alerts us to uploads automatically uses the checksum file to validate accuracy of the upload.

I cannot see how to make this easier.

1.0.36 – Expand Entity Exclusion; ftp checksum improved

Posted on by

Parsing the error log from an Entity Validation now removes the MODIFY failures as well. Additionally, the FTP post-upload checker looks for a checksum file, and if so, checks the checksum against that file to more immediately informs of failed uploads.

The underlying library fibrechannel-parsers improved the ability to remove duplicate entities: entities that exist with a requirement to MODIFY are interpreted, and removed post-pattern-aggregation from the internal tree which generates the JSON result. This version confirms that behavior is present and remains functional.

This was partially done by removing duplicate code: the use of FCParser rather than the similar code which formed the basis of FCParser allowed those changes to overtake and offer the improved VW4InvalidAddedParser and the ShowZone2Parser.

Finally, slight modifications were done on the FTP upload checker: any detected .sum file is ignored in the short-term, but if a file is uploaded and a .sum file exists, the checksum is compared to the generated/detected checksum, and matches/mismatches are reported in the subject of the outgoing warning email:

Subject: Checksum:OK New File Upload 2014-07-27 21:10:00

This is intended to allow mailbox sorting to accelerate attention to the issue and fail-fast the upload.

Revision 613 – fctransfers Gives us SFTP Uploads

Posted on by

In rev 605, the code for uploads was completely dependent on the code brought in with fctransfer.jar’s convenience library.

This revision evolves that change: all uploads are delegated to the fctransfer project, which means we get SFTP uploads for free:

Upload methods are now:

  • FTP: java -jar vict.jar -U ftp://scott:tiger@ftp.example.com/path/ -u file1 -u file2
  • SFTP: java -jar vict.jar -U sftp://scott:tiger@ftp.example.com/path/ -u file1 -u file2

Notice the similarity? That’s intentional.

Of course, both upload methods send a checksum including a notify element.

There remains some oddity in the commons-net-ssh used; I may need to swap out the underlying ssh to sshj or such.

Revision 590 – Added –UploadNotify as a Checksum Comment

Posted on by

In order to allow upstream notification, I’ve added the “–uploadnotify” argument to the FTP uploader:

VICT.BAT -U ftp://scott:tiger@ftp.oracle.com/ --uploadnotify allanc@oracle.com -u logfile1 -u logfile2

Foe each upload, a .sum file would still be generated — in this case, upload1.sum and upload2.sum. The comment “# NOTIFY allanc@oracle.com” is added to the checksum file so that any post-processor knows whom to alert of an upload. If this comment is present and understood, it can be considered advisement, not a directive; default processing may very well occur.

This is not governed by any spec, and I did not find a RFC or other open spec suggesting a format. I’ve suggested using an RFC1738-compatible URL. The format of the checksum file remains compatible with the “sum” and “md5” commands.