Package org.biojava.nbio.core.util

Class FileDownloadUtils

java.lang.Object
org.biojava.nbio.core.util.FileDownloadUtils
public class FileDownloadUtils extends Object

  • Constructor Details

    • FileDownloadUtils

      public FileDownloadUtils()

  • Method Details

    • getFileExtension

      public static String getFileExtension(File f)
      Gets the file extension of a file, excluding '.'. If the file name has no extension the file name is returned.
      Parameters:
      f - a File
      Returns:
      The extension

    • getFilePrefix

      public static String getFilePrefix(File f)
      Gets the file name up to and excluding the first '.' character. If there is no extension, the full filename is returned.
      Parameters:
      f - A file
      Returns:
      A possibly empty but non-null String.

    • downloadFile

      public static void downloadFile(URL url, File destination) throws IOException
      Download the content provided at URL url and store the result to a local file, using a temp file to cache the content in case something goes wrong in download. A timeout of 60 seconds is hard-coded and 10 retries are attempted.
      Parameters:
      url -
      destination -
      Throws:
      IOException

    • downloadFileWithValidation

      public static void downloadFileWithValidation(URL url, File destination, URL hashURL, FileDownloadUtils.Hash hash, FileDownloadUtils.ETagPolicy eTagPolicy) throws IOException
      Downloads a file and writes its validation metadata in a single pass over a single connection.

      This is preferable to calling createValidationFiles(URL, File, URL, Hash) followed by downloadFile(URL, File): those open separate connections, so the Content-Length recorded in the .size file comes from a different response than the bytes actually written. If the resource changes between the two requests, the cache entry is left permanently failing validation.

      The content is streamed to a temporary file and only moved into place once the declared length and (where available) the checksum have been confirmed, so a failed download never leaves a partial file at destination.

      Parameters:
      url - the remote file to download
      destination - the local file to download into. Its parent directory must exist.
      hashURL - URL of a file containing the expected hash. May be null.
      hash - the hashing algorithm matching hashURL. Ignored when hashURL is null.
      eTagPolicy - what to do with an ETag response header when hashURL is null. May be null, which is treated as FileDownloadUtils.ETagPolicy.IGNORE.
      Throws:
      HttpStatusException - if the server answered with a non-2xx status
      IOException - if the transfer failed, or the transferred content did not match the length or checksum the server declared
      Since:
      7.3.0

    • openConnectionFollowingRedirects

      public static URLConnection openConnectionFollowingRedirects(URL url, int timeout) throws IOException
      Opens a connection, following any redirect that HttpURLConnection declines to follow itself.

      The JDK follows 301, 302 and 303 within a protocol, but it never follows 307 or 308, and it never follows a redirect that changes http to https. Both gaps have broken downloads in practice: CATH began answering http with a 301 to https, and ECOD now answers with a 308 to a rewritten path. A browser follows either without comment, so a service making that change has no reason to expect it to break us.

      A redirect from https to http is deliberately not followed: a redirect must never silently downgrade the transport. Such a response is returned as it is, for checkHttpStatus(URLConnection) to reject.

      Parameters:
      url - the URL to open
      timeout - connect and read timeout, in milliseconds
      Returns:
      a connected URLConnection at the final location
      Throws:
      HttpStatusException - if the redirects loop or exceed the limit
      IOException - if the connection could not be opened
      Since:
      7.3.0

    • checkHttpStatus

      public static void checkHttpStatus(URLConnection connection) throws IOException
      Verifies that an HTTP connection returned a 2xx status. Connections using a non-HTTP protocol (file:, ftp:, ...) are left alone.

      Without this check a 404 error page is written into the cache as though it were the requested file — and because the .size sidecar is then taken from that same error response, validateFile(File) would subsequently declare it valid.

      Parameters:
      connection - an already connected URLConnection
      Throws:
      HttpStatusException - if the status is outside the 2xx range
      IOException - if the status could not be read
      Since:
      7.3.0

    • createValidationFiles

      public static void createValidationFiles(URL url, File localDestination, URL hashURL, FileDownloadUtils.Hash hash)
      Creates validation files beside a file to be downloaded.
      Whenever possible, for a file.ext file, it creates file.ext.size and file.hash for in the same folder where file.ext exists. If the file connection size could not be deduced from the URL, no size file is created. If hashURL is null, no hash file is created.
      Parameters:
      url - the remote file URL to download
      localDestination - the local file to download into
      hashURL - the URL of the hash file to download. Can be null.
      hash - The Hashing algorithm. Ignored if hashURL is null.

    • createValidationFiles

      public static void createValidationFiles(URL url, File localDestination, URL hashURL, FileDownloadUtils.Hash hash, FileDownloadUtils.ETagPolicy eTagPolicy)
      Creates validation files beside a file to be downloaded, with explicit control over how the ETag response header is treated.
      Parameters:
      url - the remote file URL to download
      localDestination - the local file to download into
      hashURL - the URL of the hash file to download. Can be null.
      hash - The Hashing algorithm. Ignored if hashURL is null.
      eTagPolicy - how to treat the ETag header when hashURL is null. May be null, treated as FileDownloadUtils.ETagPolicy.IGNORE.
      Since:
      7.3.0

    • createValidationFiles

      public static void createValidationFiles(URLConnection resourceUrlConnection, File localDestination, URL hashURL, FileDownloadUtils.Hash hash)
      Creates validation files beside a file to be downloaded.
      Whenever possible, for a file.ext file, it creates file.ext.size and file.hash_XXXX in the same folder where file.ext exists (XXXX may be DM5, SHA1, or SHA256). If the file connection size could not be deduced from the resourceUrlConnection URLConnection, no size file is created. If hashURL is null, no hash file is created.
      N.B. None of the hashing algorithms is implemented (yet), because we did not need any of them yet.
      Parameters:
      resourceUrlConnection - the remote file URLConnection to download
      localDestination - the local file to download into
      hashURL - the URL of the hash file to download. Can be null.
      hash - The Hashing algorithm. Ignored if hashURL is null.
      Since:
      7.0.0

    • createValidationFiles

      public static void createValidationFiles(URLConnection resourceUrlConnection, File localDestination, URL hashURL, FileDownloadUtils.Hash hash, FileDownloadUtils.ETagPolicy eTagPolicy)
      Creates validation files beside a file to be downloaded, with explicit control over how the ETag response header is treated.

      Nothing is written when the connection reports a non-2xx status: previously an error page's Content-Length would be recorded as the expected size, so the cached error page then passed validation.

      Parameters:
      resourceUrlConnection - the remote file URLConnection to download
      localDestination - the local file to download into
      hashURL - the URL of the hash file to download. Can be null.
      hash - The Hashing algorithm. Ignored if hashURL is null.
      eTagPolicy - how to treat the ETag header when hashURL is null. May be null, treated as FileDownloadUtils.ETagPolicy.IGNORE.
      Since:
      7.3.0

    • hashFromETag

      public static FileDownloadUtils.Hash hashFromETag(String eTagHeaderValue)
      Determines which hashing algorithm an ETag header value represents, based on the length of the hex digest it contains.

      Only a value consisting solely of hex characters is considered. The <time>-<size> form used by nginx and Apache always contains a - and therefore never matches.

      Parameters:
      eTagHeaderValue - the raw header value, possibly quoted or weak-prefixed. May be null.
      Returns:
      the matching algorithm, or FileDownloadUtils.Hash.UNKNOWN if the value is not a hex digest of a recognised length
      Since:
      7.3.0

    • normalizeETag

      public static String normalizeETag(String eTagHeaderValue)
      Strips the weak-validator prefix and surrounding quotes from an ETag header value.
      Parameters:
      eTagHeaderValue - the raw header value. May be null.
      Returns:
      the bare value, or null if the input was null or blank
      Since:
      7.3.0

    • writeHashFile

      public static void writeHashFile(File localDestination, FileDownloadUtils.Hash hash, String hexDigest)
      Writes a <name>.hash_<ALGORITHM> sidecar file containing the given digest as bare lowercase hex.
      Parameters:
      localDestination - the file the digest describes
      hash - the hashing algorithm
      hexDigest - the digest, in hex
      Since:
      7.3.0

    • computeHash

      public static String computeHash(File file, FileDownloadUtils.Hash hash) throws IOException
      Computes the digest of a file.
      Parameters:
      file - the file to digest
      hash - the algorithm to use
      Returns:
      the digest as bare lowercase hex
      Throws:
      IOException - if the file could not be read
      Since:
      7.3.0

    • computeHash

      public static String computeHash(InputStream in, FileDownloadUtils.Hash hash) throws IOException
      Computes the digest of a stream. The stream is read to its end but not closed.
      Parameters:
      in - the stream to digest
      hash - the algorithm to use
      Returns:
      the digest as bare lowercase hex
      Throws:
      IOException - if the stream could not be read
      Since:
      7.3.0

    • verifyHash

      public static boolean verifyHash(File file, FileDownloadUtils.Hash hash, String expectedHex) throws IOException
      Checks a file against an expected digest.
      Parameters:
      file - the file to check
      hash - the algorithm to use
      expectedHex - the expected digest, in hex; compared case-insensitively
      Returns:
      true if the digests match
      Throws:
      IOException - if the file could not be read
      Since:
      7.3.0

    • getAlgorithmName

      public static String getAlgorithmName(FileDownloadUtils.Hash hash)
      The JDK name of a hashing algorithm, which differs from the enum constant for the SHA variants.
      Parameters:
      hash - the algorithm
      Returns:
      the name to pass to MessageDigest.getInstance(String)
      Since:
      7.3.0

    • validateFile

      public static boolean validateFile(File localFile)
      Validate a local file based on pre-existing metadata files for size and hash.
      If the passed in localFile parameter is a file named file.ext, the function searches in the same folder for:
      • file.ext.size: If found, it compares the size stored in it to the length of localFile (in bytes).
      • file.ext.hash_XXXX (where XXXX is DM5, SHA1, or SHA256): If found, it compares the size stored in it to the hash code of localFile.
      If any of these comparisons fail, the function returns false. otherwise it returns true.

      N.B. None of the 3 common verification hashing algorithms are implement yet.

      Parameters:
      localFile - The file to validate
      Returns:
      false if any of the size or hash code metadata files exists but its contents does not match the expected value in the file, true otherwise.
      Since:
      7.0.0

    • toUnixPath

      public static String toUnixPath(String path)
      Converts path to Unix convention and adds a terminating slash if it was omitted.
      Parameters:
      path - original platform dependent path
      Returns:
      path in Unix convention
      Since:
      3.2

    • expandUserHome

      public static String expandUserHome(String file)
      Expands ~ in paths to the user's home directory.

      This does not work for some special cases for paths: Other users' homes (~user/...), and Tilde expansion within the path (/.../~/...). In these cases the original argument is returned.

      Parameters:
      file - A filepath starting with a tilde
      Returns:
      An absolute path

    • ping

      public static boolean ping(String url, int timeout)
      Pings a HTTP URL. This effectively sends a HEAD request and returns true if the response code is in the 200-399 range.
      Parameters:
      url - The HTTP URL to be pinged.
      timeout - The timeout in millis for both the connection timeout and the response read timeout. Note that the total timeout is effectively two times the given timeout.
      Returns:
      true if the given HTTP URL has returned response code 200-399 on a HEAD request within the given timeout, otherwise false.

    • prepareURLConnection

      public static URLConnection prepareURLConnection(String url, int timeout) throws IOException
      Prepare URLConnection with customised timeouts.
      Parameters:
      url - The URL
      timeout - The timeout in millis for both the connection timeout and the response read timeout. Note that the total timeout is effectively two times the given timeout.

      Example of code. UrlConnection conn = prepareURLConnection("http://www.google.com/", 20000); conn.connect(); conn.getInputStream();

      NB. User should execute connect() method before getting input stream.

      Returns:
      Throws:
      IOException

    • deleteDirectory

      public static void deleteDirectory(Path dir) throws IOException
      Recursively delete a folder & contents
      Parameters:
      dir - directory to delete
      Throws:
      IOException

    • deleteDirectory

      public static void deleteDirectory(String dir) throws IOException
      Recursively delete a folder & contents
      Parameters:
      dir - directory to delete
      Throws:
      IOException