AI Agent Instructions

Overview

This library is a small Clojure wrapper over a select number of features in Apache Tika.

Build System

Pantomime uses Leiningen. See project.clj.

Running Tests

lein all test

Comments

• Only add very important comments, both in tests and in the implementation
• Keep comments concise and to the point
• Add comments above the line they are referring to, not at the end of the line (an example of what's not to do: 1 + 1. %% equals 2)
• Make sure to use proper English grammar, in particular articles, punctuation and full stops at the end of sentences except for Markdown list items

Git Instructions

General

• Never add yourself to the list of commit co-authors
• Never mention yourself in commit messages in any way (no "Generated by", no AI tool links, etc)

Writing Style Guide

• Never add full stops to Markdown list items

After Completing a Task

Iterative Reviews

After completing a task, perform up to twenty iterative reviews of your changes. In every iteration, look for meaningful improvements that were missed, for gaps in test coverage, and for deviations from the instructions in this file.

If no meaningful improvements are found for three iterations in a row, report it and stop iterating.

Pantomime, a Library For Working With MIME Types In Clojure

Pantomime is a Clojure interface to Apache Tika.

Originally created as a library that deals with MIME types (Internet media types, sometimes referred to as "content types"), it now also supports extraction of document metadata and text content.

Maven Artifacts

Pantomime artifacts are released to Clojars. For Maven, add the following repository definition to pom.xml:

<repository>
  <id>clojars.org</id>
  <url>http://clojars.org/repo</url>
</repository>

Latest Series (3.x)

With Leiningen:

[com.novemberain/pantomime "3.0.0"]

With Maven:

<dependency>
  <groupId>com.novemberain</groupId>
  <artifactId>pantomime</artifactId>
  <version>3.0.0</version>
</dependency>

2.x Series

With Leiningen:

[com.novemberain/pantomime "2.11.0"]

With Maven:

<dependency>
  <groupId>com.novemberain</groupId>
  <artifactId>pantomime</artifactId>
  <version>2.11.0</version>
</dependency>

Supported Clojure Versions

Pantomime supports Clojure versions going back to at least 1.10. The default dependency starting with the 3.x series is Clojure 1.12.x.

Supported Java (JRE, JDK) Versions

Pantomime 3.x

Pantomime 3.x depends on Apache Tika 3.x which requires JDK 17+.

Pantomime 2.x

Pantomime 2.x supports JDK 8+.

Usage

Detecting MIME type

pantomime.mime/mime-type-of function accepts content as byte arrays, java.io.InputStream and java.net.URL instances as well as filenames as strings and java.io.File instances, and returns MIME type as a string or "application/octet-stream" if detection failed.

An example:

(ns your.app.namespace
  (:require [pantomime.mime :refer [mime-type-of]]))

;; by content (as byte array)
(mime-type-of (.getBytes "filename.pdf"))
;; by file extension
(mime-type-of "filename.pdf")
;; by file content (as java.io.File)
(mime-type-of (File. "some/file/without/extension"))
;; by content (as java.net.URL)
(mime-type-of (URL. "http://domain.com/some/url/path.pdf"))

Pantomime has a variation of mime-type-of function that is suitable for cases when content was fetched from the Web and HTTP headers are also available:

(ns your.app.namespace
  (:require [pantomime.web :refer [mime-type-of]]))

;; body is a string or input stream, headers is a map of lowercased headers.
;; Ring and clj-http both use this format for headers, for example.
(mime-type-of body headers)

In this case, Pantomime will try to detect content type from response body first (because there are applications, frameworks and servers that report content type incorrectly, for example, serve PDFs as text/html) and if it fails, will use content type header.

HTTP headers map must contain "content-type" key for content type header to be used. Most Clojure HTTP clients, for instance, clj-http, use lowercase strings for header names so Pantomime follows this convention.

Extension Recommendation

Pantomime can recommend an extension (one of the well known ones) for a MIME type:

(require [pantomime.mime :as pm])

(pm/extension-for-name "application/vnd.ms-excel")
;= ".xls"
(pm/extension-for-name "image/jpeg")
;= ".jpg"
(pm/extension-for-name "application/octet-stream")
;= ".bin"

Parsing and Recognizing Media Types

(ns your.app.namespace
  (:require [pantomime.media :as mt]))

(mt/parse "application/json")

(mt/base-type "text/html; charset=UTF-8") ;; => media type of "text/html"

(mt/application? "application/json")
(mt/application? "application/xhtml+xml")
(mt/application? "application/pdf")
(mt/application? "application/vnd.ms-excel")
(mt/application? (mt/parse "application/json"))

(mt/image? "image/jpeg")
(mt/audio? "audio/mp3")
(mt/video? "video/quicktime")
(mt/text?  "text/plain")
(mt/has-parameters? "text/html; charset=UTF-8") ;; => true
(mt/has-parameters? "text/html") ;; => false
(mt/parameters-of "text/html; charset=UTF-8") ;; => {"charset" "UTF-8"}
(mt/charset-of "text/html; charset=UTF-8") ;; => "UTF-8"

Language Detection

pantomime.languages is a namespace that provides functions for detecting natural languages:

(require [pantomime.languages :as pl])

(pl/detect-language "this is English, it should not be hard to detect")
;= "en"

(pl/detect-language "parlez-vous Français")
;= "fr"

Note that Tika (and, in turn, Pantomime) supports detection of a limited number of languages. To get the list of supported languages, use the pantomime.languages/supported-languages var.

Metadata and Text Extraction

pantomime.extract provides two functions for extracting metadata, content, and embedded files from byte arrays, java.io.InputStream and java.net.URL instances as well as filenames as strings and java.io.File instances. The extraction functions differ in how they handle embedded documents.

pantomime.extract/parse takes as its single argument any of the types mentioned above. It returns a map containing all the metadata Tika was able to extract from the document, and the text content of the document concatenated with the text of all embedded documents, recursively.

An example:

(require [clojure.java.io :as io]
         [pantomime.extract :as extract])

(pprint (extract/parse "test/resources/pdf/qrl.pdf"))

;= {:producer ("GNU Ghostscript 7.05"),
;=  :pdf/pdfversion ("1.2"),
;=  :dc/title ("main.dvi"),
;=  :dc/format ("application/pdf; version=1.2"),
;=  :xmp/creatortool ("dvips(k) 5.86 Copyright 1999 Radical Eye Software"),
;=  :pdf/encrypted ("false"),
;=  ...
;=  :text "\nQuickly Reacquirable Locks∗\n\nDave Dice Mark Moir ... "
;= }

pantomime.extract/parse-extract-embedded also returns Tika-extracted metadata and document text, but it handles embedded documents differently. Instead of returning the concatenation of all embedded document text, it saves each embedded file to the filesystem and includes a vector of file names and paths in the returned data. Remember to remove those files when you're done with them!

For example, the file fileAttachment.pdf contains a single attached file, which gets saved to /tmp/pantomime-3207476364135900258-embedded:

(require [clojure.java.io :as io]
         [pantomime.extract :as extract])

(pprint (extract/parse-extract-embedded "test/resources/pdf/fileAttachment.pdf"))

;= {:date ("2012-11-23T14:40:50Z"),
;=  :producer ("Acrobat Distiller 9.5.2 (Windows)"),
;=  :creator ("van der Knijff"),
;=  :pdf/pdfversion ("1.7"),
;=  :dc/title ("This is a test document"),
;=  :text "\nThis is a test document. It contains a file attachment..."
;=  ...
;=  :embedded [{:path "/tmp/pantomime-3207476364135900258-embedded",
;=              :name "KSBASE.WQ2"}],
;=  ...}

Note that parse-extract-embedded creates temporary files in the JVM's default location.

If extraction fails, the functions will return the following:

{:text "",
 :content-type ("application/octet-stream"),
 :x-parsed-by ("org.apache.tika.parser.EmptyParser")}

Development

Pantomime uses Leiningen. Make sure you have it installed and then run tests against all supported Clojure versions using

lein all test

Then create a branch and make your changes on it. Once you are done with your changes and all tests pass, submit a pull request on Github.

License

Copyright (C) 2011-2026 Michael S. Klishin, and the ClojureWerkz team.

Distributed under the Eclipse Public License, the same as Clojure.