mirrors/mostlymatter
1
0
Форкнуть 0
Код Задачи Запросы на слияние Действия Пакеты Проекты Релизы Вики Активность

Document extractor service (#15665)

Просмотр исходного кода 
* Document extractor service

* Fixing vendor modules

* Addressing PR Review comments

* Some small simplifications

* Fixing a linter complain

* simplifying a bit the code using package variables

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Этот коммит содержится в:
Jesús Espino
2020-10-27 15:58:38 +01:00
коммит произвёл GitHub
родитель 04ef5c682e
Коммит 8d5be2d657
484 изменённых файлов: 343292 добавлений и 6 удалений
Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

3
vendor/code.sajari.com/docconv/.gitignore сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,3 @@
sajari-convert
*tests/

10
vendor/code.sajari.com/docconv/.travis.yml сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,10 @@
sudo: false
language: go
go:
- "1.13"
- "1.14"
- tip
go_import_path: code.sajari.com/docconv
notifications:
email:
- infra@sajari.com

21
vendor/code.sajari.com/docconv/LICENSE сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2014 Sajari Pty Ltd
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

139
vendor/code.sajari.com/docconv/README.md сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,139 @@
# docconv
[![GoDoc](https://godoc.org/code.sajari.com/docconv?status.svg)](https://godoc.org/code.sajari.com/docconv)
[![Build Status](https://travis-ci.org/sajari/docconv.svg?branch=master)](https://travis-ci.org/sajari/docconv)
A Go wrapper library to convert PDF, DOC, DOCX, XML, HTML, RTF, ODT, Pages documents and images (see optional dependencies below) to plain text.
> **Note for returning users:** the Go import path for this package been moved to `code.sajari.com/docconv`.
## Installation
If you haven't setup Go before, you first need to [install Go](https://golang.org/doc/install).
To fetch and build the code:
$ go get code.sajari.com/docconv/...
This will also build the command line tool `docd` into `$GOPATH/bin`. Make sure that `$GOPATH/bin` is in your `PATH` environment variable.
## Dependencies
tidy, wv, popplerutils, unrtf, https://github.com/JalfResi/justext
Example install of dependencies (not all systems):
$ sudo apt-get install poppler-utils wv unrtf tidy
$ go get github.com/JalfResi/justext
### Optional dependencies
To add image support to the `docconv` library you first need to [install and build gosseract](https://github.com/otiai10/gosseract/tree/v2.2.4).
Now you can add `-tags ocr` to any `go` command when building/fetching/testing `docconv` to include support for processing images:
$ go get -tags ocr code.sajari.com/docconv/...
This may complain on macOS, which you can fix by installing [tesseract](https://tesseract-ocr.github.io) via brew:
$ brew install tesseract
## docd tool
The `docd` tool runs as either:
1. a service on port 8888 (by default)
Documents can be sent as a multipart POST request and the plain text (body) and meta information are then returned as a JSON object.
2. a service exposed from within a Docker container
This also runs as a service, but from within a Docker container. There are three build scripts:
- [./docd/debian.sh](./docd/debian.sh)
- [./docd/alpine.sh](./docd/alpine.sh)
- [./docd/appengine.sh](./docd/appengine.sh)
The `debian` version uses the Debian package repository which can vary with builds. The `alpine` version uses a very cut down Linux distribution to produce a container ~40MB. It also locks the dependency versions for consistency, but may miss out on future updates. The `appengine` version is a flex based custom runtime for Google Cloud.
3. via the command line.
Documents can be sent as an argument, e.g.
$ docd -input document.pdf
### Optional flags
- `addr` - the bind address for the HTTP server, default is ":8888"
- `log-level`
- 0: errors & critical info
- 1: inclues 0 and logs each request as well
- 2: include 1 and logs the response payloads
- `readability-length-low` - sets the readability length low if the ?readability=1 parameter is set
- `readability-length-high` - sets the readability length high if the ?readability=1 parameter is set
- `readability-stopwords-low` - sets the readability stopwords low if the ?readability=1 parameter is set
- `readability-stopwords-high` - sets the readability stopwords high if the ?readability=1 parameter is set
- `readability-max-link-density` - sets the readability max link density if the ?readability=1 parameter is set
- `readability-max-heading-distance` - sets the readability max heading distance if the ?readability=1 parameter is set
- `readability-use-classes` - comma separated list of readability classes to use if the ?readability=1 parameter is set
### How to start the service
$ # This will only log errors and critical info
$ docd -log-level 0
$ # This will run on port 8000 and log each request
$ docd -addr :8000 -log-level 1
## Example usage (code)
Some basic code is shown below, but normally you would accept the file by HTTP or open it from the file system.
This should be enough to get you started though.
### Use case 1: run locally
> Note: this assumes you have the [dependencies](#dependencies) installed.
```go
package main
import (
"fmt"
"log"
"code.sajari.com/docconv"
)
func main() {
res, err := docconv.ConvertPath("your-file.pdf")
if err != nil {
log.Fatal(err)
}
fmt.Println(res)
}
```
### Use case 2: request over the network
```go
package main
import (
"fmt"
"log"
"code.sajari.com/docconv/client"
)
func main() {
// Create a new client, using the default endpoint (localhost:8888)
c := client.New()
res, err := client.ConvertPath(c, "your-file.pdf")
if err != nil {
log.Fatal(err)
}
fmt.Println(res)
}
```

94
vendor/code.sajari.com/docconv/doc.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,94 @@
package docconv
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"os/exec"
"strings"
"time"
)
// ConvertDoc converts an MS Word .doc to text.
func ConvertDoc(r io.Reader) (string, map[string]string, error) {
f, err := NewLocalFile(r)
if err != nil {
return "", nil, fmt.Errorf("error creating local file: %v", err)
}
defer f.Done()
// Meta data
mc := make(chan map[string]string, 1)
go func() {
meta := make(map[string]string)
metaStr, err := exec.Command("wvSummary", f.Name()).Output()
if err != nil {
// TODO: Remove this.
log.Println("wvSummary:", err)
}
// Parse meta output
for _, line := range strings.Split(string(metaStr), "\n") {
if parts := strings.SplitN(line, "=", 2); len(parts) > 1 {
meta[strings.TrimSpace(parts[0])] = strings.TrimSpace(parts[1])
}
}
// Convert parsed meta
if tmp, ok := meta["Last Modified"]; ok {
if t, err := time.Parse(time.RFC3339, tmp); err == nil {
meta["ModifiedDate"] = fmt.Sprintf("%d", t.Unix())
}
}
if tmp, ok := meta["Created"]; ok {
if t, err := time.Parse(time.RFC3339, tmp); err == nil {
meta["CreatedDate"] = fmt.Sprintf("%d", t.Unix())
}
}
mc <- meta
}()
// Document body
bc := make(chan string, 1)
go func() {
// Save output to a file
outputFile, err := ioutil.TempFile("/tmp", "sajari-convert-")
if err != nil {
// TODO: Remove this.
log.Println("TempFile Out:", err)
return
}
defer os.Remove(outputFile.Name())
err = exec.Command("wvText", f.Name(), outputFile.Name()).Run()
if err != nil {
// TODO: Remove this.
log.Println("wvText:", err)
}
var buf bytes.Buffer
_, err = buf.ReadFrom(outputFile)
if err != nil {
// TODO: Remove this.
log.Println("wvText:", err)
}
bc <- buf.String()
}()
// TODO: Should errors in either of the above Goroutines stop things from progressing?
body := <-bc
meta := <-mc
// TODO: Check for errors instead of len(body) == 0?
if len(body) == 0 {
f.Seek(0, 0)
return ConvertDocx(f)
}
return body, meta, nil
}

145
vendor/code.sajari.com/docconv/docconv.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,145 @@
package docconv // import "code.sajari.com/docconv"
import (
"encoding/json"
"fmt"
"io"
"io/ioutil"
"os"
"path"
"strings"
"time"
)
// Response payload sent back to the requestor
type Response struct {
Body string `json:"body"`
Meta map[string]string `json:"meta"`
MSecs uint32 `json:"msecs"`
Error string `json:"error"`
}
// MimeTypeByExtension returns a mimetype for the given extension, or
// application/octet-stream if none can be determined.
func MimeTypeByExtension(filename string) string {
switch strings.ToLower(path.Ext(filename)) {
case ".doc":
return "application/msword"
case ".docx":
return "application/vnd.openxmlformats-officedocument.wordprocessingml.document"
case ".odt":
return "application/vnd.oasis.opendocument.text"
case ".pages":
return "application/vnd.apple.pages"
case ".pdf":
return "application/pdf"
case ".pptx":
return "application/vnd.openxmlformats-officedocument.presentationml.presentation"
case ".rtf":
return "application/rtf"
case ".xml":
return "text/xml"
case ".xhtml", ".html", ".htm":
return "text/html"
case ".jpg", ".jpeg", ".jpe", ".jfif", ".jfif-tbnl":
return "image/jpeg"
case ".png":
return "image/png"
case ".tif":
return "image/tif"
case ".tiff":
return "image/tiff"
case ".txt":
return "text/plain"
}
return "application/octet-stream"
}
// Convert a file to plain text.
func Convert(r io.Reader, mimeType string, readability bool) (*Response, error) {
start := time.Now()
var body string
var meta map[string]string
var err error
switch mimeType {
case "application/msword", "application/vnd.ms-word":
body, meta, err = ConvertDoc(r)
case "application/vnd.openxmlformats-officedocument.wordprocessingml.document":
body, meta, err = ConvertDocx(r)
case "application/vnd.openxmlformats-officedocument.presentationml.presentation":
body, meta, err = ConvertPptx(r)
case "application/vnd.oasis.opendocument.text":
body, meta, err = ConvertODT(r)
case "application/vnd.apple.pages", "application/x-iwork-pages-sffpages":
body, meta, err = ConvertPages(r)
case "application/pdf":
body, meta, err = ConvertPDF(r)
case "application/rtf", "application/x-rtf", "text/rtf", "text/richtext":
body, meta, err = ConvertRTF(r)
case "text/html":
body, meta, err = ConvertHTML(r, readability)
case "text/url":
body, meta, err = ConvertURL(r, readability)
case "text/xml", "application/xml":
body, meta, err = ConvertXML(r)
case "image/jpeg", "image/png", "image/tif", "image/tiff":
body, meta, err = ConvertImage(r)
case "text/plain":
var b []byte
b, err = ioutil.ReadAll(r)
body = string(b)
}
if err != nil {
return nil, fmt.Errorf("error converting data: %v", err)
}
return &Response{
Body: strings.TrimSpace(body),
Meta: meta,
MSecs: uint32(time.Since(start) / time.Millisecond),
}, nil
}
// ConvertPath converts a local path to text.
func ConvertPath(path string) (*Response, error) {
mimeType := MimeTypeByExtension(path)
f, err := os.Open(path)
if err != nil {
return nil, err
}
defer f.Close()
return Convert(f, mimeType, true)
}
// ConvertPathReadability converts a local path to text, with the given readability
// option.
func ConvertPathReadability(path string, readability bool) ([]byte, error) {
mimeType := MimeTypeByExtension(path)
f, err := os.Open(path)
if err != nil {
return nil, err
}
defer f.Close()
data, err := Convert(f, mimeType, readability)
if err != nil {
return nil, err
}
return json.Marshal(data)
}

155
vendor/code.sajari.com/docconv/docx.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,155 @@
package docconv
import (
"archive/zip"
"bytes"
"encoding/xml"
"fmt"
"io"
"io/ioutil"
"os"
"time"
)
type typeOverride struct {
XMLName xml.Name `xml:"Override"`
ContentType string `xml:"ContentType,attr"`
PartName string `xml:"PartName,attr"`
}
type contentTypeDefinition struct {
XMLName xml.Name `xml:"Types"`
Overrides []typeOverride `xml:"Override"`
}
// ConvertDocx converts an MS Word docx file to text.
func ConvertDocx(r io.Reader) (string, map[string]string, error) {
var size int64
// Common case: if the reader is a file (or trivial wrapper), avoid
// loading it all into memory.
var ra io.ReaderAt
if f, ok := r.(interface {
io.ReaderAt
Stat() (os.FileInfo, error)
}); ok {
si, err := f.Stat()
if err != nil {
return "", nil, err
}
size = si.Size()
ra = f
} else {
b, err := ioutil.ReadAll(r)
if err != nil {
return "", nil, nil
}
size = int64(len(b))
ra = bytes.NewReader(b)
}
zr, err := zip.NewReader(ra, size)
if err != nil {
return "", nil, fmt.Errorf("error unzipping data: %v", err)
}
zipFiles := mapZipFiles(zr.File)
contentTypeDefinition, err := getContentTypeDefinition(zipFiles["[Content_Types].xml"])
if err != nil {
return "", nil, err
}
meta := make(map[string]string)
var textHeader, textBody, textFooter string
for _, override := range contentTypeDefinition.Overrides {
f := zipFiles[override.PartName]
switch {
case override.ContentType == "application/vnd.openxmlformats-package.core-properties+xml":
rc, err := f.Open()
if err != nil {
return "", nil, fmt.Errorf("error opening '%v' from archive: %v", f.Name, err)
}
defer rc.Close()
meta, err = XMLToMap(rc)
if err != nil {
return "", nil, fmt.Errorf("error parsing '%v': %v", f.Name, err)
}
if tmp, ok := meta["modified"]; ok {
if t, err := time.Parse(time.RFC3339, tmp); err == nil {
meta["ModifiedDate"] = fmt.Sprintf("%d", t.Unix())
}
}
if tmp, ok := meta["created"]; ok {
if t, err := time.Parse(time.RFC3339, tmp); err == nil {
meta["CreatedDate"] = fmt.Sprintf("%d", t.Unix())
}
}
case override.ContentType == "application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml":
body, err := parseDocxText(f)
if err != nil {
return "", nil, err
}
textBody += body + "\n"
case override.ContentType == "application/vnd.openxmlformats-officedocument.wordprocessingml.footer+xml":
footer, err := parseDocxText(f)
if err != nil {
return "", nil, err
}
textFooter += footer + "\n"
case override.ContentType == "application/vnd.openxmlformats-officedocument.wordprocessingml.header+xml":
header, err := parseDocxText(f)
if err != nil {
return "", nil, err
}
textHeader += header + "\n"
}
}
return textHeader + "\n" + textBody + "\n" + textFooter, meta, nil
}
func getContentTypeDefinition(zf *zip.File) (*contentTypeDefinition, error) {
f, err := zf.Open()
if err != nil {
return nil, err
}
defer f.Close()
x := &contentTypeDefinition{}
if err := xml.NewDecoder(f).Decode(x); err != nil {
return nil, err
}
return x, nil
}
func mapZipFiles(files []*zip.File) map[string]*zip.File {
filesMap := make(map[string]*zip.File, 2*len(files))
for _, f := range files {
filesMap[f.Name] = f
filesMap["/"+f.Name] = f
}
return filesMap
}
func parseDocxText(f *zip.File) (string, error) {
r, err := f.Open()
if err != nil {
return "", fmt.Errorf("error opening '%v' from archive: %v", f.Name, err)
}
defer r.Close()
text, err := DocxXMLToText(r)
if err != nil {
return "", fmt.Errorf("error parsing '%v': %v", f.Name, err)
}
return text, nil
}
// DocxXMLToText converts Docx XML into plain text.
func DocxXMLToText(r io.Reader) (string, error) {
return XMLToText(r, []string{"br", "p", "tab"}, []string{"instrText", "script"}, true)
}

21
vendor/code.sajari.com/docconv/go.mod сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,21 @@
module code.sajari.com/docconv
go 1.14
require (
github.com/JalfResi/justext v0.0.0-20170829062021-c0282dea7198
github.com/PuerkitoBio/goquery v1.5.1 // indirect
github.com/advancedlogic/GoOse v0.0.0-20191112112754-e742535969c1
github.com/andybalholm/cascadia v1.2.0 // indirect
github.com/araddon/dateparse v0.0.0-20200409225146-d820a6159ab1 // indirect
github.com/go-resty/resty/v2 v2.3.0 // indirect
github.com/golang/protobuf v1.4.2
github.com/jaytaylor/html2text v0.0.0-20200412013138-3577fbdbcff7 // indirect
github.com/levigross/exp-html v0.0.0-20120902181939-8df60c69a8f5 // indirect
github.com/mattn/go-runewidth v0.0.9 // indirect
github.com/olekukonko/tablewriter v0.0.4 // indirect
github.com/otiai10/gosseract/v2 v2.2.4
github.com/pkg/errors v0.9.1 // indirect
golang.org/x/net v0.0.0-20200602114024-627f9648deb9
golang.org/x/text v0.3.2 // indirect
)

99
vendor/code.sajari.com/docconv/go.sum сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,99 @@
github.com/JalfResi/justext v0.0.0-20170829062021-c0282dea7198 h1:8P+AjBhGByCuCX2zTkAf6UY+dj0JczX+t6cSdCSyvfw=
github.com/JalfResi/justext v0.0.0-20170829062021-c0282dea7198/go.mod h1:0SURuH1rsE8aVWvutuMZghRNrNrYEUzibzJfhEYR8L0=
github.com/PuerkitoBio/goquery v1.4.1 h1:smcIRGdYm/w7JSbcdeLHEMzxmsBQvl8lhf0dSw2nzMI=
github.com/PuerkitoBio/goquery v1.4.1/go.mod h1:T9ezsOHcCrDCgA8aF1Cqr3sSYbO/xgdy8/R/XiIMAhA=
github.com/PuerkitoBio/goquery v1.5.1 h1:PSPBGne8NIUWw+/7vFBV+kG2J/5MOjbzc7154OaKCSE=
github.com/PuerkitoBio/goquery v1.5.1/go.mod h1:GsLWisAFVj4WgDibEWF4pvYnkVQBpKBKeU+7zCJoLcc=
github.com/advancedlogic/GoOse v0.0.0-20191112112754-e742535969c1 h1:d0Ct1dZwgwMO0Llf81Eu+Lyj6kwqXdqHP/WsSkEria0=
github.com/advancedlogic/GoOse v0.0.0-20191112112754-e742535969c1/go.mod h1:f3HCSN1fBWjcpGtXyM119MJgeQl838v6so/PQOqvE1w=
github.com/andybalholm/cascadia v1.0.0 h1:hOCXnnZ5A+3eVDX8pvgl4kofXv2ELss0bKcqRySc45o=
github.com/andybalholm/cascadia v1.0.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y=
github.com/andybalholm/cascadia v1.1.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y=
github.com/andybalholm/cascadia v1.2.0 h1:vuRCkM5Ozh/BfmsaTm26kbjm0mIOM3yS5Ek/F5h18aE=
github.com/andybalholm/cascadia v1.2.0/go.mod h1:YCyR8vOZT9aZ1CHEd8ap0gMVm2aFgxBp0T0eFw1RUQY=
github.com/araddon/dateparse v0.0.0-20180729174819-cfd92a431d0e h1:s05JG2GwtJMHaPcXDpo4V35TFgyYZzNsmBlSkHPEbeg=
github.com/araddon/dateparse v0.0.0-20180729174819-cfd92a431d0e/go.mod h1:SLqhdZcd+dF3TEVL2RMoob5bBP5R1P1qkox+HtCBgGI=
github.com/araddon/dateparse v0.0.0-20200409225146-d820a6159ab1 h1:TEBmxO80TM04L8IuMWk77SGL1HomBmKTdzdJLLWznxI=
github.com/araddon/dateparse v0.0.0-20200409225146-d820a6159ab1/go.mod h1:SLqhdZcd+dF3TEVL2RMoob5bBP5R1P1qkox+HtCBgGI=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/fatih/set v0.2.1 h1:nn2CaJyknWE/6txyUDGwysr3G5QC6xWB/PtVjPBbeaA=
github.com/fatih/set v0.2.1/go.mod h1:+RKtMCH+favT2+3YecHGxcc0b4KyVWA1QWWJUs4E0CI=
github.com/gigawattio/window v0.0.0-20180317192513-0f5467e35573 h1:u8AQ9bPa9oC+8/A/jlWouakhIvkFfuxgIIRjiy8av7I=
github.com/gigawattio/window v0.0.0-20180317192513-0f5467e35573/go.mod h1:eBvb3i++NHDH4Ugo9qCvMw8t0mTSctaEa5blJbWcNxs=
github.com/go-resty/resty/v2 v2.0.0 h1:9Nq/U+V4xsoDnDa/iTrABDWUCuk3Ne92XFHPe6dKWUc=
github.com/go-resty/resty/v2 v2.0.0/go.mod h1:dZGr0i9PLlaaTD4H/hoZIDjQ+r6xq8mgbRzHZf7f2J8=
github.com/go-resty/resty/v2 v2.3.0 h1:JOOeAvjSlapTT92p8xiS19Zxev1neGikoHsXJeOq8So=
github.com/go-resty/resty/v2 v2.3.0/go.mod h1:UpN9CgLZNsv4e9XG50UU8xdI0F43UQ4HmxLBDwaroHU=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/jaytaylor/html2text v0.0.0-20180606194806-57d518f124b0 h1:xqgexXAGQgY3HAjNPSaCqn5Aahbo5TKsmhp8VRfr1iQ=
github.com/jaytaylor/html2text v0.0.0-20180606194806-57d518f124b0/go.mod h1:CVKlgaMiht+LXvHG173ujK6JUhZXKb2u/BQtjPDIvyk=
github.com/jaytaylor/html2text v0.0.0-20200412013138-3577fbdbcff7 h1:g0fAGBisHaEQ0TRq1iBvemFRf+8AEWEmBESSiWB3Vsc=
github.com/jaytaylor/html2text v0.0.0-20200412013138-3577fbdbcff7/go.mod h1:CVKlgaMiht+LXvHG173ujK6JUhZXKb2u/BQtjPDIvyk=
github.com/levigross/exp-html v0.0.0-20120902181939-8df60c69a8f5 h1:W7p+m/AECTL3s/YR5RpQ4hz5SjNeKzZBl1q36ws12s0=
github.com/levigross/exp-html v0.0.0-20120902181939-8df60c69a8f5/go.mod h1:QMe2wuKJ0o7zIVE8AqiT8rd8epmm6WDIZ2wyuBqYPzM=
github.com/mattn/go-runewidth v0.0.3 h1:a+kO+98RDGEfo6asOGMmpodZq4FNtnGP54yps8BzLR4=
github.com/mattn/go-runewidth v0.0.3/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
github.com/mattn/go-runewidth v0.0.7/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-runewidth v0.0.9 h1:Lm995f3rfxdpd6TSmuVCHVb/QhupuXlYr8sCI/QdE+0=
github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/olekukonko/tablewriter v0.0.0-20180506121414-d4647c9c7a84 h1:fiKJgB4JDUd43CApkmCeTSQlWjtTtABrU2qsgbuP0BI=
github.com/olekukonko/tablewriter v0.0.0-20180506121414-d4647c9c7a84/go.mod h1:vsDQFd/mU46D+Z4whnwzcISnGGzXWMclvtLoiIKAKIo=
github.com/olekukonko/tablewriter v0.0.4 h1:vHD/YYe1Wolo78koG299f7V/VAS08c6IpCLn+Ejf/w8=
github.com/olekukonko/tablewriter v0.0.4/go.mod h1:zq6QwlOf5SlnkVbMSr5EoBv3636FWnp+qbPhuoO21uA=
github.com/otiai10/curr v0.0.0-20150429015615-9b4961190c95 h1:+OLn68pqasWca0z5ryit9KGfp3sUsW4Lqg32iRMJyzs=
github.com/otiai10/curr v0.0.0-20150429015615-9b4961190c95/go.mod h1:9qAhocn7zKJG+0mI8eUu6xqkFDYS2kb2saOteoSB3cE=
github.com/otiai10/gosseract/v2 v2.2.4 h1:h/PV+oJqke8q2Ccw9bjpMBWfd7N2vtGDCUcihZj3nRo=
github.com/otiai10/gosseract/v2 v2.2.4/go.mod h1:ahOp/kHojnOMGv1RaUnR0jwY5JVa6BYKhYAS8nbMLSo=
github.com/otiai10/mint v1.3.0 h1:Ady6MKVezQwHBkGzLFbrsywyp09Ah7rkmfjV3Bcr5uc=
github.com/otiai10/mint v1.3.0/go.mod h1:F5AjcsTsWUqX+Na9fpHb52P8pcRX2CI6A3ctIT91xUo=
github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/simplereach/timeutils v1.2.0/go.mod h1:VVbQDfN/FHRZa1LSqcwo4kNZ62OOyqLLGQKYB3pB0Q8=
github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf h1:pvbZ0lM0XWPBqUKqFU8cmavspvIl9nulOYwdy6IFRRo=
github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf/go.mod h1:RJID2RhlZKId02nZ62WenDCkgHFerpIOmW0iT7GKmXM=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/net v0.0.0-20180218175443-cbe0f9307d01/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190628185345-da137c7871d7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200513185701-a91f0712d120/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200602114024-627f9648deb9 h1:pNX+40auqi2JqRfOP1akLGtYcn15TUbkhwuCO3foqqM=
golang.org/x/net v0.0.0-20200602114024-627f9648deb9/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.23.0 h1:4MY060fB1DLGMB/7MBTLnwQUY6+F09GEiz6SsrNqyzM=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/mgo.v2 v2.0.0-20190816093944-a6b53ec6cb22/go.mod h1:yeKp02qBN3iKW1OzL3MGk2IdtZzaj7SFntXj72NppTA=
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

191
vendor/code.sajari.com/docconv/html.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,191 @@
// +build !appengine
package docconv
import (
"bytes"
"io"
"log"
"strings"
"golang.org/x/net/html"
"github.com/JalfResi/justext"
)
// ConvertHTML converts HTML into text.
func ConvertHTML(r io.Reader, readability bool) (string, map[string]string, error) {
meta := make(map[string]string)
buf := new(bytes.Buffer)
_, err := buf.ReadFrom(r)
if err != nil {
return "", nil, err
}
cleanXML, err := Tidy(buf, false)
if err != nil {
log.Println("Tidy:", err)
// Tidy failed, so we now manually tokenize instead
clean := cleanHTML(buf, true)
cleanXML = []byte(clean)
// TODO: remove this log
log.Println("Cleaned HTML using Golang tokenizer")
}
if readability {
cleanXML = HTMLReadability(bytes.NewReader(cleanXML))
}
return HTMLToText(bytes.NewReader(cleanXML)), meta, nil
}
var acceptedHTMLTags = [...]string{
"div", "p", "br", "span", "body", "head", "html", "ul", "ol", "li", "dl", "dt", "dd", "a", "form", "article",
"section", "table", "tr", "td", "tbody", "thead", "th", "tfoot", "col", "colgroup", "caption", "form", "input",
"title", "h1", "h2", "h3", "h4", "h5", "h6", "meta", "strong", "cite", "em", "address", "abbr", "acronym",
"blockquote", "q", "pre", "samp", "select", "fieldset", "legend", "button", "option", "textarea", "label",
}
// Tests for known friendly HTML parameters that tidy is unlikely to choke on
func acceptedHTMLTag(tagName string) bool {
for _, tag := range acceptedHTMLTags {
if tag == tagName {
return true
}
}
return false
}
// Removes scripts, comments, styles and parameters from HTML.
// Also removes made up tags, e.g. <fb:like>
// Can keep head elements or not. Typically not much in there.
func cleanHTML(r io.Reader, all bool) string {
output := ""
if !all {
output = "<html><head></head>"
}
mainSection := false
junkSection := false
d := html.NewTokenizer(r)
for {
// token type
tokenType := d.Next()
if tokenType == html.ErrorToken {
return output
}
token := d.Token()
switch tokenType {
case html.StartTagToken: // <tag>
if token.Data == "body" || (token.Data == "html" && all) {
mainSection = true
}
if !acceptedHTMLTag(token.Data) {
junkSection = true
}
if !junkSection && mainSection {
output += "<" + token.Data + ">"
}
case html.TextToken: // text between start and end tag
if !junkSection && mainSection {
output += token.Data
}
case html.EndTagToken: // </tag>
if !junkSection && mainSection {
output += "</" + token.Data + ">"
}
if !acceptedHTMLTag(token.Data) {
junkSection = false
}
case html.SelfClosingTagToken: // <tag/>
if !junkSection && mainSection {
output += "<" + token.Data + " />" // TODO: Can probably keep attributes from the meta tags
}
}
}
}
// HTMLReadabilityOptions is a type which defines parameters that are passed to the justext package.
// TODO: Improve this!
type HTMLReadabilityOptions struct {
LengthLow int
LengthHigh int
StopwordsLow float64
StopwordsHigh float64
MaxLinkDensity float64
MaxHeadingDistance int
ReadabilityUseClasses string
}
// HTMLReadabilityOptionsValues are the global settings used for HTMLReadability.
// TODO: Remove this from global state.
var HTMLReadabilityOptionsValues HTMLReadabilityOptions
// HTMLReadability extracts the readable text in an HTML document
func HTMLReadability(r io.Reader) []byte {
jr := justext.NewReader(r)
// TODO: Improve this!
jr.Stoplist = readabilityStopList
jr.LengthLow = HTMLReadabilityOptionsValues.LengthLow
jr.LengthHigh = HTMLReadabilityOptionsValues.LengthHigh
jr.StopwordsLow = HTMLReadabilityOptionsValues.StopwordsLow
jr.StopwordsHigh = HTMLReadabilityOptionsValues.StopwordsHigh
jr.MaxLinkDensity = HTMLReadabilityOptionsValues.MaxLinkDensity
jr.MaxHeadingDistance = HTMLReadabilityOptionsValues.MaxHeadingDistance
paragraphSet, err := jr.ReadAll()
if err != nil {
log.Println("Justext:", err)
return nil
}
useClasses := strings.SplitN(HTMLReadabilityOptionsValues.ReadabilityUseClasses, ",", 10)
output := ""
for _, paragraph := range paragraphSet {
for _, class := range useClasses {
if paragraph.CfClass == class {
output += paragraph.Text + "\n"
}
}
}
return []byte(output)
}
// HTMLToText converts HTML to plain text.
func HTMLToText(input io.Reader) string {
text, _ := XMLToText(input, []string{"br", "p", "h1", "h2", "h3", "h4"}, []string{}, false)
return text
}
var readabilityStopList = map[string]bool{"and": true, "the": true, "a": true, "about": true, "above": true, "across": true, "after": true, "afterwards": true, "again": true, "against": true, "all": true, "almost": true, "alone": true,
"along": true, "already": true, "also": true, "although": true, "always": true, "am": true, "among": true, "amongst": true, "amoungst": true, "amount": true, "an": true, "another": true, "any": true,
"anyhow": true, "anyone": true, "anything": true, "anyway": true, "anywhere": true, "are": true, "around": true, "as": true, "at": true, "back": true, "be": true, "became": true, "because": true,
"become": true, "becomes": true, "becoming": true, "been": true, "before": true, "beforehand": true, "behind": true, "being": true, "below": true, "beside": true, "besides": true, "between": true,
"beyond": true, "both": true, "bottom": true, "but": true, "by": true, "can": true, "cannot": true, "cant": true, "co": true, "con": true, "could": true, "couldnt": true, "cry": true,
"de": true, "describe": true, "detail": true, "do": true, "done": true, "down": true, "due": true, "during": true, "each": true, "eg": true, "eight": true, "either": true, "eleven": true, "else": true,
"elsewhere": true, "empty": true, "enough": true, "etc": true, "even": true, "ever": true, "every": true, "everyone": true, "everything": true, "everywhere": true, "except": true, "few": true,
"fifteen": true, "fify": true, "fill": true, "find": true, "fire": true, "first": true, "five": true, "for": true, "former": true, "formerly": true, "forty": true, "found": true, "four": true, "from": true,
"front": true, "full": true, "further": true, "get": true, "give": true, "go": true, "had": true, "has": true, "hasnt": true, "have": true, "he": true, "hence": true, "her": true, "here": true, "hereafter": true,
"hereby": true, "herein": true, "hereupon": true, "hers": true, "herself": true, "him": true, "himself": true, "his": true, "how": true, "however": true, "hundred": true, "ie": true, "if": true, "in": true,
"inc": true, "indeed": true, "interest": true, "into": true, "is": true, "it": true, "its": true, "itself": true, "keep": true, "last": true, "latter": true, "latterly": true, "least": true, "less": true,
"ltd": true, "made": true, "many": true, "may": true, "me": true, "meanwhile": true, "might": true, "mill": true, "mine": true, "more": true, "moreover": true, "most": true, "mostly": true, "move": true,
"much": true, "must": true, "my": true, "myself": true, "name": true, "namely": true, "neither": true, "never": true, "nevertheless": true, "next": true, "nine": true, "no": true, "nobody": true,
"none": true, "noone": true, "nor": true, "not": true, "nothing": true, "now": true, "nowhere": true, "of": true, "off": true, "often": true, "on": true, "once": true, "one": true, "only": true, "onto": true,
"or": true, "other": true, "others": true, "otherwise": true, "our": true, "ours": true, "ourselves": true, "out": true, "over": true, "own": true, "part": true, "per": true, "perhaps": true,
"please": true, "put": true, "rather": true, "re": true, "same": true, "see": true, "seem": true, "seemed": true, "seeming": true, "seems": true, "serious": true, "several": true, "she": true,
"should": true, "show": true, "side": true, "since": true, "sincere": true, "six": true, "sixty": true, "so": true, "some": true, "somehow": true, "someone": true, "something": true, "sometime": true,
"sometimes": true, "somewhere": true, "still": true, "such": true, "take": true, "ten": true, "than": true, "that": true, "their": true, "them": true, "themselves": true,
"then": true, "thence": true, "there": true, "thereafter": true, "thereby": true, "therefore": true, "therein": true, "thereupon": true, "these": true, "they": true, "thickv": true, "thin": true,
"third": true, "this": true, "those": true, "though": true, "three": true, "through": true, "throughout": true, "thru": true, "thus": true, "to": true, "together": true, "too": true, "top": true,
"toward": true, "towards": true, "twelve": true, "twenty": true, "two": true, "un": true, "under": true, "until": true, "up": true, "upon": true, "us": true, "very": true, "via": true, "was": true, "we": true,
"well": true, "were": true, "what": true, "whatever": true, "when": true, "whence": true, "whenever": true, "where": true, "whereafter": true, "whereas": true, "whereby": true, "wherein": true,
"whereupon": true, "wherever": true, "whether": true, "which": true, "while": true, "whither": true, "who": true, "whoever": true, "whole": true, "whom": true, "whose": true, "why": true, "will": true,
"with": true, "within": true, "without": true, "would": true, "yet": true, "you": true, "your": true, "youre": true, "yours": true, "yourself": true, "yourselves": true, "www": true, "com": true, "http": true}

18
vendor/code.sajari.com/docconv/html_appengine.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,18 @@
// +build appengine
package docconv
import (
"io"
"io/ioutil"
"log"
)
func HTMLReadability(r io.Reader) []byte {
b, err := ioutil.ReadAll(r)
if err != nil {
log.Printf("HTMLReadability: %v", err)
return nil
}
return b
}

587
vendor/code.sajari.com/docconv/iWork/TSPArchiveMessages.pb.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,587 @@
// Code generated by protoc-gen-go.
// source: TSPArchiveMessages.proto
// DO NOT EDIT!
/*
Package TSP is a generated protocol buffer package.
It is generated from these files:
TSPArchiveMessages.proto
TSPDatabaseMessages.proto
TSPMessages.proto
It has these top-level messages:
ArchiveInfo
MessageInfo
FieldInfo
FieldPath
ComponentInfo
ComponentExternalReference
ComponentDataReference
PackageMetadata
PasteboardMetadata
DataInfo
ViewStateMetadata
*/
package TSP
import proto "github.com/golang/protobuf/proto"
import math "math"
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = math.Inf
type FieldInfo_Type int32
const (
FieldInfo_Value FieldInfo_Type = 0
FieldInfo_ObjectReference FieldInfo_Type = 1
FieldInfo_DataReference FieldInfo_Type = 2
FieldInfo_Message FieldInfo_Type = 3
)
var FieldInfo_Type_name = map[int32]string{
0: "Value",
1: "ObjectReference",
2: "DataReference",
3: "Message",
}
var FieldInfo_Type_value = map[string]int32{
"Value": 0,
"ObjectReference": 1,
"DataReference": 2,
"Message": 3,
}
func (x FieldInfo_Type) Enum() *FieldInfo_Type {
p := new(FieldInfo_Type)
*p = x
return p
}
func (x FieldInfo_Type) String() string {
return proto.EnumName(FieldInfo_Type_name, int32(x))
}
func (x *FieldInfo_Type) UnmarshalJSON(data []byte) error {
value, err := proto.UnmarshalJSONEnum(FieldInfo_Type_value, data, "FieldInfo_Type")
if err != nil {
return err
}
*x = FieldInfo_Type(value)
return nil
}
type FieldInfo_Rule int32
const (
FieldInfo_IgnoreAndDrop FieldInfo_Rule = 0
FieldInfo_IgnoreAndPreserve FieldInfo_Rule = 1
FieldInfo_MustUnderstand FieldInfo_Rule = 2
FieldInfo_NotSupported FieldInfo_Rule = -1
)
var FieldInfo_Rule_name = map[int32]string{
0: "IgnoreAndDrop",
1: "IgnoreAndPreserve",
2: "MustUnderstand",
-1: "NotSupported",
}
var FieldInfo_Rule_value = map[string]int32{
"IgnoreAndDrop": 0,
"IgnoreAndPreserve": 1,
"MustUnderstand": 2,
"NotSupported": -1,
}
func (x FieldInfo_Rule) Enum() *FieldInfo_Rule {
p := new(FieldInfo_Rule)
*p = x
return p
}
func (x FieldInfo_Rule) String() string {
return proto.EnumName(FieldInfo_Rule_name, int32(x))
}
func (x *FieldInfo_Rule) UnmarshalJSON(data []byte) error {
value, err := proto.UnmarshalJSONEnum(FieldInfo_Rule_value, data, "FieldInfo_Rule")
if err != nil {
return err
}
*x = FieldInfo_Rule(value)
return nil
}
type ArchiveInfo struct {
Identifier *uint64 `protobuf:"varint,1,opt,name=identifier" json:"identifier,omitempty"`
MessageInfos []*MessageInfo `protobuf:"bytes,2,rep,name=message_infos" json:"message_infos,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *ArchiveInfo) Reset() { *m = ArchiveInfo{} }
func (m *ArchiveInfo) String() string { return proto.CompactTextString(m) }
func (*ArchiveInfo) ProtoMessage() {}
func (m *ArchiveInfo) GetIdentifier() uint64 {
if m != nil && m.Identifier != nil {
return *m.Identifier
}
return 0
}
func (m *ArchiveInfo) GetMessageInfos() []*MessageInfo {
if m != nil {
return m.MessageInfos
}
return nil
}
type MessageInfo struct {
Type *uint32 `protobuf:"varint,1,req,name=type" json:"type,omitempty"`
Version []uint32 `protobuf:"varint,2,rep,packed,name=version" json:"version,omitempty"`
Length *uint32 `protobuf:"varint,3,req,name=length" json:"length,omitempty"`
FieldInfos []*FieldInfo `protobuf:"bytes,4,rep,name=field_infos" json:"field_infos,omitempty"`
ObjectReferences []uint64 `protobuf:"varint,5,rep,packed,name=object_references" json:"object_references,omitempty"`
DataReferences []uint64 `protobuf:"varint,6,rep,packed,name=data_references" json:"data_references,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *MessageInfo) Reset() { *m = MessageInfo{} }
func (m *MessageInfo) String() string { return proto.CompactTextString(m) }
func (*MessageInfo) ProtoMessage() {}
func (m *MessageInfo) GetType() uint32 {
if m != nil && m.Type != nil {
return *m.Type
}
return 0
}
func (m *MessageInfo) GetVersion() []uint32 {
if m != nil {
return m.Version
}
return nil
}
func (m *MessageInfo) GetLength() uint32 {
if m != nil && m.Length != nil {
return *m.Length
}
return 0
}
func (m *MessageInfo) GetFieldInfos() []*FieldInfo {
if m != nil {
return m.FieldInfos
}
return nil
}
func (m *MessageInfo) GetObjectReferences() []uint64 {
if m != nil {
return m.ObjectReferences
}
return nil
}
func (m *MessageInfo) GetDataReferences() []uint64 {
if m != nil {
return m.DataReferences
}
return nil
}
type FieldInfo struct {
Path *FieldPath `protobuf:"bytes,1,req,name=path" json:"path,omitempty"`
Type *FieldInfo_Type `protobuf:"varint,2,opt,name=type,enum=TSP.FieldInfo_Type,def=0" json:"type,omitempty"`
Rule *FieldInfo_Rule `protobuf:"varint,3,opt,name=rule,enum=TSP.FieldInfo_Rule,def=0" json:"rule,omitempty"`
ObjectReferences []uint64 `protobuf:"varint,4,rep,packed,name=object_references" json:"object_references,omitempty"`
DataReferences []uint64 `protobuf:"varint,5,rep,packed,name=data_references" json:"data_references,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *FieldInfo) Reset() { *m = FieldInfo{} }
func (m *FieldInfo) String() string { return proto.CompactTextString(m) }
func (*FieldInfo) ProtoMessage() {}
const Default_FieldInfo_Type FieldInfo_Type = FieldInfo_Value
const Default_FieldInfo_Rule FieldInfo_Rule = FieldInfo_IgnoreAndDrop
func (m *FieldInfo) GetPath() *FieldPath {
if m != nil {
return m.Path
}
return nil
}
func (m *FieldInfo) GetType() FieldInfo_Type {
if m != nil && m.Type != nil {
return *m.Type
}
return Default_FieldInfo_Type
}
func (m *FieldInfo) GetRule() FieldInfo_Rule {
if m != nil && m.Rule != nil {
return *m.Rule
}
return Default_FieldInfo_Rule
}
func (m *FieldInfo) GetObjectReferences() []uint64 {
if m != nil {
return m.ObjectReferences
}
return nil
}
func (m *FieldInfo) GetDataReferences() []uint64 {
if m != nil {
return m.DataReferences
}
return nil
}
type FieldPath struct {
Path []uint32 `protobuf:"varint,1,rep,packed,name=path" json:"path,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *FieldPath) Reset() { *m = FieldPath{} }
func (m *FieldPath) String() string { return proto.CompactTextString(m) }
func (*FieldPath) ProtoMessage() {}
func (m *FieldPath) GetPath() []uint32 {
if m != nil {
return m.Path
}
return nil
}
type ComponentInfo struct {
Identifier *uint64 `protobuf:"varint,1,req,name=identifier" json:"identifier,omitempty"`
PreferredLocator *string `protobuf:"bytes,2,req,name=preferred_locator" json:"preferred_locator,omitempty"`
Locator *string `protobuf:"bytes,3,opt,name=locator" json:"locator,omitempty"`
ReadVersion []uint32 `protobuf:"varint,4,rep,packed,name=read_version" json:"read_version,omitempty"`
WriteVersion []uint32 `protobuf:"varint,5,rep,packed,name=write_version" json:"write_version,omitempty"`
ExternalReferences []*ComponentExternalReference `protobuf:"bytes,6,rep,name=external_references" json:"external_references,omitempty"`
DataReferences []*ComponentDataReference `protobuf:"bytes,7,rep,name=data_references" json:"data_references,omitempty"`
AllowsDuplicatesOutsideOfDocumentPackage *bool `protobuf:"varint,8,opt,name=allows_duplicates_outside_of_document_package,def=0" json:"allows_duplicates_outside_of_document_package,omitempty"`
DirtiesDocumentPackage *bool `protobuf:"varint,9,opt,name=dirties_document_package,def=1" json:"dirties_document_package,omitempty"`
IsStoredOutsideObjectArchive *bool `protobuf:"varint,10,opt,name=is_stored_outside_object_archive,def=0" json:"is_stored_outside_object_archive,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *ComponentInfo) Reset() { *m = ComponentInfo{} }
func (m *ComponentInfo) String() string { return proto.CompactTextString(m) }
func (*ComponentInfo) ProtoMessage() {}
const Default_ComponentInfo_AllowsDuplicatesOutsideOfDocumentPackage bool = false
const Default_ComponentInfo_DirtiesDocumentPackage bool = true
const Default_ComponentInfo_IsStoredOutsideObjectArchive bool = false
func (m *ComponentInfo) GetIdentifier() uint64 {
if m != nil && m.Identifier != nil {
return *m.Identifier
}
return 0
}
func (m *ComponentInfo) GetPreferredLocator() string {
if m != nil && m.PreferredLocator != nil {
return *m.PreferredLocator
}
return ""
}
func (m *ComponentInfo) GetLocator() string {
if m != nil && m.Locator != nil {
return *m.Locator
}
return ""
}
func (m *ComponentInfo) GetReadVersion() []uint32 {
if m != nil {
return m.ReadVersion
}
return nil
}
func (m *ComponentInfo) GetWriteVersion() []uint32 {
if m != nil {
return m.WriteVersion
}
return nil
}
func (m *ComponentInfo) GetExternalReferences() []*ComponentExternalReference {
if m != nil {
return m.ExternalReferences
}
return nil
}
func (m *ComponentInfo) GetDataReferences() []*ComponentDataReference {
if m != nil {
return m.DataReferences
}
return nil
}
func (m *ComponentInfo) GetAllowsDuplicatesOutsideOfDocumentPackage() bool {
if m != nil && m.AllowsDuplicatesOutsideOfDocumentPackage != nil {
return *m.AllowsDuplicatesOutsideOfDocumentPackage
}
return Default_ComponentInfo_AllowsDuplicatesOutsideOfDocumentPackage
}
func (m *ComponentInfo) GetDirtiesDocumentPackage() bool {
if m != nil && m.DirtiesDocumentPackage != nil {
return *m.DirtiesDocumentPackage
}
return Default_ComponentInfo_DirtiesDocumentPackage
}
func (m *ComponentInfo) GetIsStoredOutsideObjectArchive() bool {
if m != nil && m.IsStoredOutsideObjectArchive != nil {
return *m.IsStoredOutsideObjectArchive
}
return Default_ComponentInfo_IsStoredOutsideObjectArchive
}
type ComponentExternalReference struct {
ComponentIdentifier *uint64 `protobuf:"varint,1,req,name=component_identifier" json:"component_identifier,omitempty"`
ObjectIdentifier *uint64 `protobuf:"varint,2,opt,name=object_identifier" json:"object_identifier,omitempty"`
IsWeak *bool `protobuf:"varint,3,opt,name=is_weak" json:"is_weak,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *ComponentExternalReference) Reset() { *m = ComponentExternalReference{} }
func (m *ComponentExternalReference) String() string { return proto.CompactTextString(m) }
func (*ComponentExternalReference) ProtoMessage() {}
func (m *ComponentExternalReference) GetComponentIdentifier() uint64 {
if m != nil && m.ComponentIdentifier != nil {
return *m.ComponentIdentifier
}
return 0
}
func (m *ComponentExternalReference) GetObjectIdentifier() uint64 {
if m != nil && m.ObjectIdentifier != nil {
return *m.ObjectIdentifier
}
return 0
}
func (m *ComponentExternalReference) GetIsWeak() bool {
if m != nil && m.IsWeak != nil {
return *m.IsWeak
}
return false
}
type ComponentDataReference struct {
DataIdentifier *uint64 `protobuf:"varint,1,req,name=data_identifier" json:"data_identifier,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *ComponentDataReference) Reset() { *m = ComponentDataReference{} }
func (m *ComponentDataReference) String() string { return proto.CompactTextString(m) }
func (*ComponentDataReference) ProtoMessage() {}
func (m *ComponentDataReference) GetDataIdentifier() uint64 {
if m != nil && m.DataIdentifier != nil {
return *m.DataIdentifier
}
return 0
}
type PackageMetadata struct {
LastObjectIdentifier *uint64 `protobuf:"varint,1,req,name=last_object_identifier" json:"last_object_identifier,omitempty"`
Components []*ComponentInfo `protobuf:"bytes,3,rep,name=components" json:"components,omitempty"`
Datas []*DataInfo `protobuf:"bytes,4,rep,name=datas" json:"datas,omitempty"`
ReadVersion []uint32 `protobuf:"varint,5,rep,packed,name=read_version" json:"read_version,omitempty"`
WriteVersion []uint32 `protobuf:"varint,6,rep,packed,name=write_version" json:"write_version,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *PackageMetadata) Reset() { *m = PackageMetadata{} }
func (m *PackageMetadata) String() string { return proto.CompactTextString(m) }
func (*PackageMetadata) ProtoMessage() {}
func (m *PackageMetadata) GetLastObjectIdentifier() uint64 {
if m != nil && m.LastObjectIdentifier != nil {
return *m.LastObjectIdentifier
}
return 0
}
func (m *PackageMetadata) GetComponents() []*ComponentInfo {
if m != nil {
return m.Components
}
return nil
}
func (m *PackageMetadata) GetDatas() []*DataInfo {
if m != nil {
return m.Datas
}
return nil
}
func (m *PackageMetadata) GetReadVersion() []uint32 {
if m != nil {
return m.ReadVersion
}
return nil
}
func (m *PackageMetadata) GetWriteVersion() []uint32 {
if m != nil {
return m.WriteVersion
}
return nil
}
type PasteboardMetadata struct {
Version []uint32 `protobuf:"varint,1,rep,packed,name=version" json:"version,omitempty"`
AppName *string `protobuf:"bytes,2,req,name=app_name" json:"app_name,omitempty"`
Datas []*DataInfo `protobuf:"bytes,3,rep,name=datas" json:"datas,omitempty"`
SourceDocumentUuid *string `protobuf:"bytes,4,opt,name=source_document_uuid" json:"source_document_uuid,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *PasteboardMetadata) Reset() { *m = PasteboardMetadata{} }
func (m *PasteboardMetadata) String() string { return proto.CompactTextString(m) }
func (*PasteboardMetadata) ProtoMessage() {}
func (m *PasteboardMetadata) GetVersion() []uint32 {
if m != nil {
return m.Version
}
return nil
}
func (m *PasteboardMetadata) GetAppName() string {
if m != nil && m.AppName != nil {
return *m.AppName
}
return ""
}
func (m *PasteboardMetadata) GetDatas() []*DataInfo {
if m != nil {
return m.Datas
}
return nil
}
func (m *PasteboardMetadata) GetSourceDocumentUuid() string {
if m != nil && m.SourceDocumentUuid != nil {
return *m.SourceDocumentUuid
}
return ""
}
type DataInfo struct {
Identifier *uint64 `protobuf:"varint,1,req,name=identifier" json:"identifier,omitempty"`
Digest []byte `protobuf:"bytes,2,req,name=digest" json:"digest,omitempty"`
PreferredFileName *string `protobuf:"bytes,3,req,name=preferred_file_name" json:"preferred_file_name,omitempty"`
FileName *string `protobuf:"bytes,4,opt,name=file_name" json:"file_name,omitempty"`
DocumentResourceLocator *string `protobuf:"bytes,5,opt,name=document_resource_locator" json:"document_resource_locator,omitempty"`
SourceBookmarkData []byte `protobuf:"bytes,6,opt,name=source_bookmark_data" json:"source_bookmark_data,omitempty"`
PasteboardExternalFilePath *string `protobuf:"bytes,99,opt,name=pasteboard_external_file_path" json:"pasteboard_external_file_path,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *DataInfo) Reset() { *m = DataInfo{} }
func (m *DataInfo) String() string { return proto.CompactTextString(m) }
func (*DataInfo) ProtoMessage() {}
func (m *DataInfo) GetIdentifier() uint64 {
if m != nil && m.Identifier != nil {
return *m.Identifier
}
return 0
}
func (m *DataInfo) GetDigest() []byte {
if m != nil {
return m.Digest
}
return nil
}
func (m *DataInfo) GetPreferredFileName() string {
if m != nil && m.PreferredFileName != nil {
return *m.PreferredFileName
}
return ""
}
func (m *DataInfo) GetFileName() string {
if m != nil && m.FileName != nil {
return *m.FileName
}
return ""
}
func (m *DataInfo) GetDocumentResourceLocator() string {
if m != nil && m.DocumentResourceLocator != nil {
return *m.DocumentResourceLocator
}
return ""
}
func (m *DataInfo) GetSourceBookmarkData() []byte {
if m != nil {
return m.SourceBookmarkData
}
return nil
}
func (m *DataInfo) GetPasteboardExternalFilePath() string {
if m != nil && m.PasteboardExternalFilePath != nil {
return *m.PasteboardExternalFilePath
}
return ""
}
type ViewStateMetadata struct {
Version []uint32 `protobuf:"varint,1,rep,packed,name=version" json:"version,omitempty"`
DocumentVersionUuid *string `protobuf:"bytes,2,req,name=document_version_uuid" json:"document_version_uuid,omitempty"`
Component *ComponentInfo `protobuf:"bytes,3,req,name=component" json:"component,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *ViewStateMetadata) Reset() { *m = ViewStateMetadata{} }
func (m *ViewStateMetadata) String() string { return proto.CompactTextString(m) }
func (*ViewStateMetadata) ProtoMessage() {}
func (m *ViewStateMetadata) GetVersion() []uint32 {
if m != nil {
return m.Version
}
return nil
}
func (m *ViewStateMetadata) GetDocumentVersionUuid() string {
if m != nil && m.DocumentVersionUuid != nil {
return *m.DocumentVersionUuid
}
return ""
}
func (m *ViewStateMetadata) GetComponent() *ComponentInfo {
if m != nil {
return m.Component
}
return nil
}
func init() {
proto.RegisterEnum("TSP.FieldInfo_Type", FieldInfo_Type_name, FieldInfo_Type_value)
proto.RegisterEnum("TSP.FieldInfo_Rule", FieldInfo_Rule_name, FieldInfo_Rule_value)
}

150
vendor/code.sajari.com/docconv/iWork/TSPDatabaseMessages.pb.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,150 @@
// Code generated by protoc-gen-go.
// source: TSPDatabaseMessages.proto
// DO NOT EDIT!
package TSP
import proto "github.com/golang/protobuf/proto"
import math "math"
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = math.Inf
type DatabaseImageDataArchive_ImageType int32
const (
DatabaseImageDataArchive_unknown DatabaseImageDataArchive_ImageType = 0
DatabaseImageDataArchive_bitmap DatabaseImageDataArchive_ImageType = 1
DatabaseImageDataArchive_pdf DatabaseImageDataArchive_ImageType = 2
)
var DatabaseImageDataArchive_ImageType_name = map[int32]string{
0: "unknown",
1: "bitmap",
2: "pdf",
}
var DatabaseImageDataArchive_ImageType_value = map[string]int32{
"unknown": 0,
"bitmap": 1,
"pdf": 2,
}
func (x DatabaseImageDataArchive_ImageType) Enum() *DatabaseImageDataArchive_ImageType {
p := new(DatabaseImageDataArchive_ImageType)
*p = x
return p
}
func (x DatabaseImageDataArchive_ImageType) String() string {
return proto.EnumName(DatabaseImageDataArchive_ImageType_name, int32(x))
}
func (x *DatabaseImageDataArchive_ImageType) UnmarshalJSON(data []byte) error {
value, err := proto.UnmarshalJSONEnum(DatabaseImageDataArchive_ImageType_value, data, "DatabaseImageDataArchive_ImageType")
if err != nil {
return err
}
*x = DatabaseImageDataArchive_ImageType(value)
return nil
}
type DatabaseData struct {
Data *DataReference `protobuf:"bytes,1,req,name=data" json:"data,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *DatabaseData) Reset() { *m = DatabaseData{} }
func (m *DatabaseData) String() string { return proto.CompactTextString(m) }
func (*DatabaseData) ProtoMessage() {}
func (m *DatabaseData) GetData() *DataReference {
if m != nil {
return m.Data
}
return nil
}
type DatabaseDataArchive struct {
Data *Reference `protobuf:"bytes,1,opt,name=data" json:"data,omitempty"`
AppRelativePath *string `protobuf:"bytes,2,opt,name=app_relative_path" json:"app_relative_path,omitempty"`
DisplayName *string `protobuf:"bytes,3,req,name=display_name" json:"display_name,omitempty"`
Length *uint64 `protobuf:"varint,4,opt,name=length" json:"length,omitempty"`
Hash *uint32 `protobuf:"varint,5,opt,name=hash" json:"hash,omitempty"`
Sharable *bool `protobuf:"varint,6,req,name=sharable,def=1" json:"sharable,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *DatabaseDataArchive) Reset() { *m = DatabaseDataArchive{} }
func (m *DatabaseDataArchive) String() string { return proto.CompactTextString(m) }
func (*DatabaseDataArchive) ProtoMessage() {}
const Default_DatabaseDataArchive_Sharable bool = true
func (m *DatabaseDataArchive) GetData() *Reference {
if m != nil {
return m.Data
}
return nil
}
func (m *DatabaseDataArchive) GetAppRelativePath() string {
if m != nil && m.AppRelativePath != nil {
return *m.AppRelativePath
}
return ""
}
func (m *DatabaseDataArchive) GetDisplayName() string {
if m != nil && m.DisplayName != nil {
return *m.DisplayName
}
return ""
}
func (m *DatabaseDataArchive) GetLength() uint64 {
if m != nil && m.Length != nil {
return *m.Length
}
return 0
}
func (m *DatabaseDataArchive) GetHash() uint32 {
if m != nil && m.Hash != nil {
return *m.Hash
}
return 0
}
func (m *DatabaseDataArchive) GetSharable() bool {
if m != nil && m.Sharable != nil {
return *m.Sharable
}
return Default_DatabaseDataArchive_Sharable
}
type DatabaseImageDataArchive struct {
Super *DatabaseDataArchive `protobuf:"bytes,1,req,name=super" json:"super,omitempty"`
Type *DatabaseImageDataArchive_ImageType `protobuf:"varint,2,req,name=type,enum=TSP.DatabaseImageDataArchive_ImageType" json:"type,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *DatabaseImageDataArchive) Reset() { *m = DatabaseImageDataArchive{} }
func (m *DatabaseImageDataArchive) String() string { return proto.CompactTextString(m) }
func (*DatabaseImageDataArchive) ProtoMessage() {}
func (m *DatabaseImageDataArchive) GetSuper() *DatabaseDataArchive {
if m != nil {
return m.Super
}
return nil
}
func (m *DatabaseImageDataArchive) GetType() DatabaseImageDataArchive_ImageType {
if m != nil && m.Type != nil {
return *m.Type
}
return DatabaseImageDataArchive_unknown
}
func init() {
proto.RegisterEnum("TSP.DatabaseImageDataArchive_ImageType", DatabaseImageDataArchive_ImageType_name, DatabaseImageDataArchive_ImageType_value)
}

524
vendor/code.sajari.com/docconv/iWork/TSPMessages.pb.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,524 @@
// Code generated by protoc-gen-go.
// source: TSPMessages.proto
// DO NOT EDIT!
package TSP
import proto "github.com/golang/protobuf/proto"
import math "math"
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = math.Inf
type Color_ColorModel int32
const (
Color_rgb Color_ColorModel = 1
Color_cmyk Color_ColorModel = 2
Color_white Color_ColorModel = 3
)
var Color_ColorModel_name = map[int32]string{
1: "rgb",
2: "cmyk",
3: "white",
}
var Color_ColorModel_value = map[string]int32{
"rgb": 1,
"cmyk": 2,
"white": 3,
}
func (x Color_ColorModel) Enum() *Color_ColorModel {
p := new(Color_ColorModel)
*p = x
return p
}
func (x Color_ColorModel) String() string {
return proto.EnumName(Color_ColorModel_name, int32(x))
}
func (x *Color_ColorModel) UnmarshalJSON(data []byte) error {
value, err := proto.UnmarshalJSONEnum(Color_ColorModel_value, data, "Color_ColorModel")
if err != nil {
return err
}
*x = Color_ColorModel(value)
return nil
}
type Path_ElementType int32
const (
Path_moveTo Path_ElementType = 1
Path_lineTo Path_ElementType = 2
Path_quadCurveTo Path_ElementType = 3
Path_curveTo Path_ElementType = 4
Path_closeSubpath Path_ElementType = 5
)
var Path_ElementType_name = map[int32]string{
1: "moveTo",
2: "lineTo",
3: "quadCurveTo",
4: "curveTo",
5: "closeSubpath",
}
var Path_ElementType_value = map[string]int32{
"moveTo": 1,
"lineTo": 2,
"quadCurveTo": 3,
"curveTo": 4,
"closeSubpath": 5,
}
func (x Path_ElementType) Enum() *Path_ElementType {
p := new(Path_ElementType)
*p = x
return p
}
func (x Path_ElementType) String() string {
return proto.EnumName(Path_ElementType_name, int32(x))
}
func (x *Path_ElementType) UnmarshalJSON(data []byte) error {
value, err := proto.UnmarshalJSONEnum(Path_ElementType_value, data, "Path_ElementType")
if err != nil {
return err
}
*x = Path_ElementType(value)
return nil
}
type Reference struct {
Identifier *uint64 `protobuf:"varint,1,req,name=identifier" json:"identifier,omitempty"`
DeprecatedType *int32 `protobuf:"varint,2,opt,name=deprecated_type" json:"deprecated_type,omitempty"`
DeprecatedIsExternal *bool `protobuf:"varint,3,opt,name=deprecated_is_external" json:"deprecated_is_external,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Reference) Reset() { *m = Reference{} }
func (m *Reference) String() string { return proto.CompactTextString(m) }
func (*Reference) ProtoMessage() {}
func (m *Reference) GetIdentifier() uint64 {
if m != nil && m.Identifier != nil {
return *m.Identifier
}
return 0
}
func (m *Reference) GetDeprecatedType() int32 {
if m != nil && m.DeprecatedType != nil {
return *m.DeprecatedType
}
return 0
}
func (m *Reference) GetDeprecatedIsExternal() bool {
if m != nil && m.DeprecatedIsExternal != nil {
return *m.DeprecatedIsExternal
}
return false
}
type DataReference struct {
Identifier *uint64 `protobuf:"varint,1,req,name=identifier" json:"identifier,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *DataReference) Reset() { *m = DataReference{} }
func (m *DataReference) String() string { return proto.CompactTextString(m) }
func (*DataReference) ProtoMessage() {}
func (m *DataReference) GetIdentifier() uint64 {
if m != nil && m.Identifier != nil {
return *m.Identifier
}
return 0
}
type Point struct {
X *float32 `protobuf:"fixed32,1,req,name=x" json:"x,omitempty"`
Y *float32 `protobuf:"fixed32,2,req,name=y" json:"y,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Point) Reset() { *m = Point{} }
func (m *Point) String() string { return proto.CompactTextString(m) }
func (*Point) ProtoMessage() {}
func (m *Point) GetX() float32 {
if m != nil && m.X != nil {
return *m.X
}
return 0
}
func (m *Point) GetY() float32 {
if m != nil && m.Y != nil {
return *m.Y
}
return 0
}
type Size struct {
Width *float32 `protobuf:"fixed32,1,req,name=width" json:"width,omitempty"`
Height *float32 `protobuf:"fixed32,2,req,name=height" json:"height,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Size) Reset() { *m = Size{} }
func (m *Size) String() string { return proto.CompactTextString(m) }
func (*Size) ProtoMessage() {}
func (m *Size) GetWidth() float32 {
if m != nil && m.Width != nil {
return *m.Width
}
return 0
}
func (m *Size) GetHeight() float32 {
if m != nil && m.Height != nil {
return *m.Height
}
return 0
}
type Range struct {
Location *uint32 `protobuf:"varint,1,req,name=location" json:"location,omitempty"`
Length *uint32 `protobuf:"varint,2,req,name=length" json:"length,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Range) Reset() { *m = Range{} }
func (m *Range) String() string { return proto.CompactTextString(m) }
func (*Range) ProtoMessage() {}
func (m *Range) GetLocation() uint32 {
if m != nil && m.Location != nil {
return *m.Location
}
return 0
}
func (m *Range) GetLength() uint32 {
if m != nil && m.Length != nil {
return *m.Length
}
return 0
}
type Date struct {
Seconds *float64 `protobuf:"fixed64,1,req,name=seconds" json:"seconds,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Date) Reset() { *m = Date{} }
func (m *Date) String() string { return proto.CompactTextString(m) }
func (*Date) ProtoMessage() {}
func (m *Date) GetSeconds() float64 {
if m != nil && m.Seconds != nil {
return *m.Seconds
}
return 0
}
type IndexSet struct {
Ranges []*Range `protobuf:"bytes,1,rep,name=ranges" json:"ranges,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *IndexSet) Reset() { *m = IndexSet{} }
func (m *IndexSet) String() string { return proto.CompactTextString(m) }
func (*IndexSet) ProtoMessage() {}
func (m *IndexSet) GetRanges() []*Range {
if m != nil {
return m.Ranges
}
return nil
}
type Color struct {
Model *Color_ColorModel `protobuf:"varint,1,req,name=model,enum=TSP.Color_ColorModel" json:"model,omitempty"`
R *float32 `protobuf:"fixed32,3,opt,name=r" json:"r,omitempty"`
G *float32 `protobuf:"fixed32,4,opt,name=g" json:"g,omitempty"`
B *float32 `protobuf:"fixed32,5,opt,name=b" json:"b,omitempty"`
A *float32 `protobuf:"fixed32,6,opt,name=a,def=1" json:"a,omitempty"`
C *float32 `protobuf:"fixed32,7,opt,name=c" json:"c,omitempty"`
M *float32 `protobuf:"fixed32,8,opt,name=m" json:"m,omitempty"`
Y *float32 `protobuf:"fixed32,9,opt,name=y" json:"y,omitempty"`
K *float32 `protobuf:"fixed32,10,opt,name=k" json:"k,omitempty"`
W *float32 `protobuf:"fixed32,11,opt,name=w" json:"w,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Color) Reset() { *m = Color{} }
func (m *Color) String() string { return proto.CompactTextString(m) }
func (*Color) ProtoMessage() {}
const Default_Color_A float32 = 1
func (m *Color) GetModel() Color_ColorModel {
if m != nil && m.Model != nil {
return *m.Model
}
return Color_rgb
}
func (m *Color) GetR() float32 {
if m != nil && m.R != nil {
return *m.R
}
return 0
}
func (m *Color) GetG() float32 {
if m != nil && m.G != nil {
return *m.G
}
return 0
}
func (m *Color) GetB() float32 {
if m != nil && m.B != nil {
return *m.B
}
return 0
}
func (m *Color) GetA() float32 {
if m != nil && m.A != nil {
return *m.A
}
return Default_Color_A
}
func (m *Color) GetC() float32 {
if m != nil && m.C != nil {
return *m.C
}
return 0
}
func (m *Color) GetM() float32 {
if m != nil && m.M != nil {
return *m.M
}
return 0
}
func (m *Color) GetY() float32 {
if m != nil && m.Y != nil {
return *m.Y
}
return 0
}
func (m *Color) GetK() float32 {
if m != nil && m.K != nil {
return *m.K
}
return 0
}
func (m *Color) GetW() float32 {
if m != nil && m.W != nil {
return *m.W
}
return 0
}
type Path struct {
Elements []*Path_Element `protobuf:"bytes,1,rep,name=elements" json:"elements,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Path) Reset() { *m = Path{} }
func (m *Path) String() string { return proto.CompactTextString(m) }
func (*Path) ProtoMessage() {}
func (m *Path) GetElements() []*Path_Element {
if m != nil {
return m.Elements
}
return nil
}
type Path_Element struct {
Type *Path_ElementType `protobuf:"varint,1,req,name=type,enum=TSP.Path_ElementType" json:"type,omitempty"`
Points []*Point `protobuf:"bytes,2,rep,name=points" json:"points,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Path_Element) Reset() { *m = Path_Element{} }
func (m *Path_Element) String() string { return proto.CompactTextString(m) }
func (*Path_Element) ProtoMessage() {}
func (m *Path_Element) GetType() Path_ElementType {
if m != nil && m.Type != nil {
return *m.Type
}
return Path_moveTo
}
func (m *Path_Element) GetPoints() []*Point {
if m != nil {
return m.Points
}
return nil
}
type ReferenceDictionary struct {
Entries []*ReferenceDictionary_Entry `protobuf:"bytes,1,rep,name=entries" json:"entries,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *ReferenceDictionary) Reset() { *m = ReferenceDictionary{} }
func (m *ReferenceDictionary) String() string { return proto.CompactTextString(m) }
func (*ReferenceDictionary) ProtoMessage() {}
func (m *ReferenceDictionary) GetEntries() []*ReferenceDictionary_Entry {
if m != nil {
return m.Entries
}
return nil
}
type ReferenceDictionary_Entry struct {
Key *Reference `protobuf:"bytes,1,req,name=key" json:"key,omitempty"`
Value *Reference `protobuf:"bytes,2,req,name=value" json:"value,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *ReferenceDictionary_Entry) Reset() { *m = ReferenceDictionary_Entry{} }
func (m *ReferenceDictionary_Entry) String() string { return proto.CompactTextString(m) }
func (*ReferenceDictionary_Entry) ProtoMessage() {}
func (m *ReferenceDictionary_Entry) GetKey() *Reference {
if m != nil {
return m.Key
}
return nil
}
func (m *ReferenceDictionary_Entry) GetValue() *Reference {
if m != nil {
return m.Value
}
return nil
}
type PasteboardObject struct {
Stylesheet *Reference `protobuf:"bytes,1,opt,name=stylesheet" json:"stylesheet,omitempty"`
Drawables []*Reference `protobuf:"bytes,2,rep,name=drawables" json:"drawables,omitempty"`
Styles []*Reference `protobuf:"bytes,3,rep,name=styles" json:"styles,omitempty"`
Theme *Reference `protobuf:"bytes,4,opt,name=theme" json:"theme,omitempty"`
WpStorage *Reference `protobuf:"bytes,5,opt,name=wp_storage" json:"wp_storage,omitempty"`
GuideStorage *Reference `protobuf:"bytes,9,opt,name=guide_storage" json:"guide_storage,omitempty"`
AppNativeObject *Reference `protobuf:"bytes,6,opt,name=app_native_object" json:"app_native_object,omitempty"`
IsTextPrimary *bool `protobuf:"varint,7,opt,name=is_text_primary,def=0" json:"is_text_primary,omitempty"`
IsSmart *bool `protobuf:"varint,8,opt,name=is_smart,def=0" json:"is_smart,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *PasteboardObject) Reset() { *m = PasteboardObject{} }
func (m *PasteboardObject) String() string { return proto.CompactTextString(m) }
func (*PasteboardObject) ProtoMessage() {}
const Default_PasteboardObject_IsTextPrimary bool = false
const Default_PasteboardObject_IsSmart bool = false
func (m *PasteboardObject) GetStylesheet() *Reference {
if m != nil {
return m.Stylesheet
}
return nil
}
func (m *PasteboardObject) GetDrawables() []*Reference {
if m != nil {
return m.Drawables
}
return nil
}
func (m *PasteboardObject) GetStyles() []*Reference {
if m != nil {
return m.Styles
}
return nil
}
func (m *PasteboardObject) GetTheme() *Reference {
if m != nil {
return m.Theme
}
return nil
}
func (m *PasteboardObject) GetWpStorage() *Reference {
if m != nil {
return m.WpStorage
}
return nil
}
func (m *PasteboardObject) GetGuideStorage() *Reference {
if m != nil {
return m.GuideStorage
}
return nil
}
func (m *PasteboardObject) GetAppNativeObject() *Reference {
if m != nil {
return m.AppNativeObject
}
return nil
}
func (m *PasteboardObject) GetIsTextPrimary() bool {
if m != nil && m.IsTextPrimary != nil {
return *m.IsTextPrimary
}
return Default_PasteboardObject_IsTextPrimary
}
func (m *PasteboardObject) GetIsSmart() bool {
if m != nil && m.IsSmart != nil {
return *m.IsSmart
}
return Default_PasteboardObject_IsSmart
}
type ObjectContainer struct {
Identifier *uint32 `protobuf:"varint,1,opt,name=identifier" json:"identifier,omitempty"`
Objects []*Reference `protobuf:"bytes,2,rep,name=objects" json:"objects,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *ObjectContainer) Reset() { *m = ObjectContainer{} }
func (m *ObjectContainer) String() string { return proto.CompactTextString(m) }
func (*ObjectContainer) ProtoMessage() {}
func (m *ObjectContainer) GetIdentifier() uint32 {
if m != nil && m.Identifier != nil {
return *m.Identifier
}
return 0
}
func (m *ObjectContainer) GetObjects() []*Reference {
if m != nil {
return m.Objects
}
return nil
}
func init() {
proto.RegisterEnum("TSP.Color_ColorModel", Color_ColorModel_name, Color_ColorModel_value)
proto.RegisterEnum("TSP.Path_ElementType", Path_ElementType_name, Path_ElementType_value)
}

17
vendor/code.sajari.com/docconv/image.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,17 @@
// +build !ocr
package docconv
import (
"fmt"
"io"
)
// ConvertImage converts images to text.
// Requires gosseract (ocr build tag).
func ConvertImage(r io.Reader) (string, map[string]string, error) {
return "", nil, fmt.Errorf("docconv not built with `ocr` build tag")
}
// SetImageLanguages sets the languages parameter passed to gosseract.
func SetImageLanguages(...string) {}

51
vendor/code.sajari.com/docconv/image_ocr.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,51 @@
// +build ocr
package docconv
import (
"fmt"
"io"
"sync"
"github.com/otiai10/gosseract/v2"
)
var config = struct {
langs []string
sync.Mutex
}{
langs: []string{"eng"},
}
func SetImageLanguages(l ...string) {
config.Lock()
config.langs = l
config.Unlock()
}
// ConvertImage converts images to text.
// Requires gosseract.
func ConvertImage(r io.Reader) (string, map[string]string, error) {
f, err := NewLocalFile(r)
if err != nil {
return "", nil, fmt.Errorf("error creating local file: %v", err)
}
defer f.Done()
meta := make(map[string]string)
client := gosseract.NewClient()
defer client.Close()
config.Lock()
defer config.Unlock()
client.SetLanguage(config.langs...)
client.SetImage(f.Name())
text, err := client.Text()
if err != nil {
return "", nil, err
}
return text, meta, nil
}

51
vendor/code.sajari.com/docconv/local.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,51 @@
package docconv
import (
"fmt"
"io"
"io/ioutil"
"os"
)
// LocalFile is a type which wraps an *os.File. See NewLocalFile for more details.
type LocalFile struct {
*os.File
unlink bool
}
// NewLocalFile ensures that there is a file which contains the data provided by r. If r is
// actually an instance of *os.File then this file is used, otherwise a temporary file is
// created and the data from r copied into it. Callers must call Done() when
// the LocalFile is no longer needed to ensure all resources are cleaned up.
func NewLocalFile(r io.Reader) (*LocalFile, error) {
if f, ok := r.(*os.File); ok {
return &LocalFile{
File: f,
}, nil
}
f, err := ioutil.TempFile(os.TempDir(), "/docconv")
if err != nil {
return nil, fmt.Errorf("error creating temporary file: %v", err)
}
_, err = io.Copy(f, r)
if err != nil {
f.Close()
os.Remove(f.Name())
return nil, fmt.Errorf("error copying data into temporary file: %v", err)
}
return &LocalFile{
File: f,
unlink: true,
}, nil
}
// Done cleans up all resources.
func (l *LocalFile) Done() {
l.Close()
if l.unlink {
os.Remove(l.Name())
}
}

69
vendor/code.sajari.com/docconv/odt.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,69 @@
package docconv
import (
"archive/zip"
"bytes"
"fmt"
"io"
"io/ioutil"
"time"
)
// ConvertODT converts a ODT file to text
func ConvertODT(r io.Reader) (string, map[string]string, error) {
meta := make(map[string]string)
var textBody string
b, err := ioutil.ReadAll(r)
if err != nil {
return "", nil, err
}
zr, err := zip.NewReader(bytes.NewReader(b), int64(len(b)))
if err != nil {
return "", nil, fmt.Errorf("error unzipping data: %v", err)
}
for _, f := range zr.File {
switch f.Name {
case "meta.xml":
rc, err := f.Open()
if err != nil {
return "", nil, fmt.Errorf("error extracting '%v' from archive: %v", f.Name, err)
}
defer rc.Close()
info, err := XMLToMap(rc)
if err != nil {
return "", nil, fmt.Errorf("error parsing '%v': %v", f.Name, err)
}
if tmp, ok := info["creator"]; ok {
meta["Author"] = tmp
}
if tmp, ok := info["date"]; ok {
if t, err := time.Parse("2006-01-02T15:04:05", tmp); err == nil {
meta["ModifiedDate"] = fmt.Sprintf("%d", t.Unix())
}
}
if tmp, ok := info["creation-date"]; ok {
if t, err := time.Parse("2006-01-02T15:04:05", tmp); err == nil {
meta["CreatedDate"] = fmt.Sprintf("%d", t.Unix())
}
}
case "content.xml":
rc, err := f.Open()
if err != nil {
return "", nil, fmt.Errorf("error extracting '%v' from archive: %v", f.Name, err)
}
defer rc.Close()
textBody, err = XMLToText(rc, []string{"br", "p", "tab"}, []string{}, true)
if err != nil {
return "", nil, fmt.Errorf("error parsing '%v': %v", f.Name, err)
}
}
}
return textBody, meta, nil
}

60
vendor/code.sajari.com/docconv/pages.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,60 @@
package docconv
import (
"archive/zip"
"bufio"
"bytes"
"encoding/binary"
"fmt"
"io"
"io/ioutil"
"strings"
"github.com/golang/protobuf/proto"
"code.sajari.com/docconv/iWork"
"code.sajari.com/docconv/snappy"
)
// ConvertPages converts a Pages file to text.
func ConvertPages(r io.Reader) (string, map[string]string, error) {
meta := make(map[string]string)
var textBody string
b, err := ioutil.ReadAll(r)
if err != nil {
return "", nil, fmt.Errorf("error reading data: %v", err)
}
zr, err := zip.NewReader(bytes.NewReader(b), int64(len(b)))
if err != nil {
return "", nil, fmt.Errorf("error unzipping data: %v", err)
}
for _, f := range zr.File {
if strings.HasSuffix(f.Name, "Preview.pdf") {
// There is a preview PDF version we can use
if rc, err := f.Open(); err == nil {
return ConvertPDF(rc)
}
}
if f.Name == "index.xml" {
// There's an XML version we can use
if rc, err := f.Open(); err == nil {
return ConvertXML(rc)
}
}
if f.Name == "Index/Document.iwa" {
rc, _ := f.Open()
defer rc.Close()
bReader := bufio.NewReader(snappy.NewReader(io.MultiReader(strings.NewReader("\xff\x06\x00\x00sNaPpY"), rc)))
archiveLength, err := binary.ReadVarint(bReader)
archiveInfoData, err := ioutil.ReadAll(io.LimitReader(bReader, archiveLength))
archiveInfo := &TSP.ArchiveInfo{}
err = proto.Unmarshal(archiveInfoData, archiveInfo)
fmt.Println("archiveInfo:", archiveInfo, err)
}
}
return textBody, meta, nil
}

30
vendor/code.sajari.com/docconv/pdf.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,30 @@
// +build !ocr
package docconv
import (
"fmt"
"io"
)
func ConvertPDF(r io.Reader) (string, map[string]string, error) {
f, err := NewLocalFile(r)
if err != nil {
return "", nil, fmt.Errorf("error creating local file: %v", err)
}
defer f.Done()
bodyResult, metaResult, convertErr := ConvertPDFText(f.Name())
if convertErr != nil {
return "", nil, convertErr
}
if bodyResult.err != nil {
return "", nil, bodyResult.err
}
if metaResult.err != nil {
return "", nil, metaResult.err
}
return bodyResult.body, metaResult.meta, nil
}

161
vendor/code.sajari.com/docconv/pdf_ocr.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,161 @@
// +build ocr
package docconv
import (
"fmt"
"io"
"io/ioutil"
"log"
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
)
var (
exts = []string{".jpg", ".tif", ".tiff", ".png", ".pbm"}
)
func compareExt(ext string, exts []string) bool {
for _, e := range exts {
if ext == e {
return true
}
}
return false
}
func cleanupTemp(tmpDir string) {
err := os.RemoveAll(tmpDir)
if err != nil {
log.Println(err)
}
}
func ConvertPDFImages(path string) (BodyResult, error) {
bodyResult := BodyResult{}
tmp, err := ioutil.TempDir(os.TempDir(), "tmp-imgs-")
if err != nil {
bodyResult.err = err
return bodyResult, err
}
tmpDir := fmt.Sprintf("%s/", tmp)
defer cleanupTemp(tmpDir)
_, err = exec.Command("pdfimages", "-j", path, tmpDir).Output()
if err != nil {
return bodyResult, err
}
filePaths := []string{}
walkFunc := func(path string, info os.FileInfo, err error) error {
path, err = filepath.Abs(path)
if err != nil {
return err
}
if compareExt(filepath.Ext(path), exts) {
filePaths = append(filePaths, path)
}
return nil
}
filepath.Walk(tmpDir, walkFunc)
fileLength := len(filePaths)
if fileLength < 1 {
return bodyResult, nil
}
var wg sync.WaitGroup
data := make(chan string, fileLength)
wg.Add(fileLength)
for _, p := range filePaths {
go func(pathFile string) {
defer wg.Done()
f, err := os.Open(pathFile)
if err != nil {
return
}
defer f.Close()
out, _, err := ConvertImage(f)
if err != nil {
return
}
data <- out
}(p)
}
wg.Wait()
close(data)
for str := range data {
bodyResult.body += str + " "
}
return bodyResult, nil
}
// PdfHasImage verify if `path` (PDF) has images
func PDFHasImage(path string) bool {
cmd := "pdffonts -l 5 %s | tail -n +3 | cut -d' ' -f1 | sort | uniq"
out, err := exec.Command("bash", "-c", fmt.Sprintf(cmd, path)).Output()
if err != nil {
log.Println(err)
return false
}
if string(out) == "" {
return true
}
return false
}
func ConvertPDF(r io.Reader) (string, map[string]string, error) {
f, err := NewLocalFile(r)
if err != nil {
return "", nil, fmt.Errorf("error creating local file: %v", err)
}
defer f.Done()
bodyResult, metaResult, textConvertErr := ConvertPDFText(f.Name())
if textConvertErr != nil {
return "", nil, textConvertErr
}
if bodyResult.err != nil {
return "", nil, bodyResult.err
}
if metaResult.err != nil {
return "", nil, metaResult.err
}
if !PDFHasImage(f.Name()) {
return bodyResult.body, metaResult.meta, nil
}
imageConvertResult, imageConvertErr := ConvertPDFImages(f.Name())
if imageConvertErr != nil {
log.Println(imageConvertErr)
return bodyResult.body, metaResult.meta, nil
}
if imageConvertResult.err != nil {
log.Println(imageConvertResult.err)
return bodyResult.body, metaResult.meta, nil
}
fullBody := strings.Join([]string{bodyResult.body, imageConvertResult.body}, " ")
return fullBody, metaResult.meta, nil
}

84
vendor/code.sajari.com/docconv/pdf_text.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,84 @@
package docconv
import (
"fmt"
"os/exec"
"strings"
"time"
)
// Meta data
type MetaResult struct {
meta map[string]string
err error
}
type BodyResult struct {
body string
err error
}
// Convert PDF
func ConvertPDFText(path string) (BodyResult, MetaResult, error) {
metaResult := MetaResult{meta: make(map[string]string)}
bodyResult := BodyResult{}
mr := make(chan MetaResult, 1)
go func() {
metaStr, err := exec.Command("pdfinfo", path).Output()
if err != nil {
metaResult.err = err
mr <- metaResult
return
}
// Parse meta output
for _, line := range strings.Split(string(metaStr), "\n") {
if parts := strings.SplitN(line, ":", 2); len(parts) > 1 {
metaResult.meta[strings.TrimSpace(parts[0])] = strings.TrimSpace(parts[1])
}
}
// Convert parsed meta
if x, ok := metaResult.meta["ModDate"]; ok {
if t, ok := pdfTimeLayouts.Parse(x); ok {
metaResult.meta["ModifiedDate"] = fmt.Sprintf("%d", t.Unix())
}
}
if x, ok := metaResult.meta["CreationDate"]; ok {
if t, ok := pdfTimeLayouts.Parse(x); ok {
metaResult.meta["CreatedDate"] = fmt.Sprintf("%d", t.Unix())
}
}
mr <- metaResult
}()
br := make(chan BodyResult, 1)
go func() {
body, err := exec.Command("pdftotext", "-q", "-nopgbrk", "-enc", "UTF-8", "-eol", "unix", path, "-").Output()
if err != nil {
bodyResult.err = err
}
bodyResult.body = string(body)
br <- bodyResult
}()
return <-br, <-mr, nil
}
var pdfTimeLayouts = timeLayouts{time.ANSIC, "Mon Jan _2 15:04:05 2006 MST"}
type timeLayouts []string
func (tl timeLayouts) Parse(x string) (time.Time, bool) {
for _, layout := range tl {
t, err := time.Parse(layout, x)
if err == nil {
return t, true
}
}
return time.Time{}, false
}

67
vendor/code.sajari.com/docconv/pptx.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,67 @@
package docconv
import (
"archive/zip"
"bytes"
"fmt"
"io"
"io/ioutil"
"os"
"strings"
)
// ConvertPptx converts an MS PowerPoint pptx file to text.
func ConvertPptx(r io.Reader) (string, map[string]string, error) {
var size int64
// Common case: if the reader is a file (or trivial wrapper), avoid
// loading it all into memory.
var ra io.ReaderAt
if f, ok := r.(interface {
io.ReaderAt
Stat() (os.FileInfo, error)
}); ok {
si, err := f.Stat()
if err != nil {
return "", nil, err
}
size = si.Size()
ra = f
} else {
b, err := ioutil.ReadAll(r)
if err != nil {
return "", nil, nil
}
size = int64(len(b))
ra = bytes.NewReader(b)
}
zr, err := zip.NewReader(ra, size)
if err != nil {
return "", nil, fmt.Errorf("could not unzip: %v", err)
}
zipFiles := mapZipFiles(zr.File)
contentTypeDefinition, err := getContentTypeDefinition(zipFiles["[Content_Types].xml"])
if err != nil {
return "", nil, err
}
meta := make(map[string]string)
var textBody string
for _, override := range contentTypeDefinition.Overrides {
f := zipFiles[override.PartName]
switch override.ContentType {
case "application/vnd.openxmlformats-officedocument.presentationml.slide+xml",
"application/vnd.openxmlformats-officedocument.drawingml.diagramData+xml":
body, err := parseDocxText(f)
if err != nil {
return "", nil, fmt.Errorf("could not parse pptx: %v", err)
}
textBody += body + "\n"
}
}
return strings.TrimSuffix(textBody, "\n"), meta, nil
}

52
vendor/code.sajari.com/docconv/rtf.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,52 @@
package docconv
import (
"fmt"
"io"
"os/exec"
"strings"
"time"
)
// ConvertRTF converts RTF files to text.
func ConvertRTF(r io.Reader) (string, map[string]string, error) {
f, err := NewLocalFile(r)
if err != nil {
return "", nil, fmt.Errorf("error creating local file: %v", err)
}
defer f.Done()
var output string
tmpOutput, err := exec.Command("unrtf", "--nopict", "--text", f.Name()).Output()
if err != nil {
return "", nil, fmt.Errorf("unrtf error: %v", err)
}
// Step through content looking for meta data and stripping out comments
meta := make(map[string]string)
for _, line := range strings.Split(string(tmpOutput), "\n") {
if parts := strings.SplitN(line, ":", 2); len(parts) > 1 {
meta[strings.TrimSpace(parts[0])] = strings.TrimSpace(parts[1])
}
if len(line) > 4 && line[:4] != "### " {
output += line + "\n"
}
}
// Identify meta data
if tmp, ok := meta["AUTHOR"]; ok {
meta["Author"] = tmp
}
if tmp, ok := meta["### creation date"]; ok {
if t, err := time.Parse("02 January 2006 15:04", tmp); err == nil {
meta["CreatedDate"] = fmt.Sprintf("%d", t.Unix())
}
}
if tmp, ok := meta["### revision date"]; ok {
if t, err := time.Parse("02 January 2006 15:04", tmp); err == nil {
meta["ModifiedDate"] = fmt.Sprintf("%d", t.Unix())
}
}
return output, meta, nil
}

27
vendor/code.sajari.com/docconv/snappy/LICENSE сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,27 @@
Copyright (c) 2011 The Snappy-Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

13
vendor/code.sajari.com/docconv/snappy/README сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,13 @@
This is a Snappy library for the Go programming language that has been modified to work with Apple files, which fail to set CRC checks and stream identifiers. This version is a total hack, so if you want to use snappy for other projects **DO NOT USE THIS VERSION**. Use the proper version as per below:
To download and install from source:
$ go get code.google.com/p/snappy-go/snappy
Unless otherwise noted, the Snappy-Go source files are distributed
under the BSD-style license found in the LICENSE file.
Contributions should follow the same procedure as for the Go project:
http://golang.org/doc/contribute.html

297
vendor/code.sajari.com/docconv/snappy/decode.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,297 @@
// Copyright 2011 The Snappy-Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package snappy
import (
"encoding/binary"
"errors"
"io"
)
var (
// ErrCorrupt reports that the input is invalid.
ErrCorrupt = errors.New("snappy: corrupt input")
// ErrUnsupported reports that the input isn't supported.
ErrUnsupported = errors.New("snappy: unsupported input")
)
// DecodedLen returns the length of the decoded block.
func DecodedLen(src []byte) (int, error) {
v, _, err := decodedLen(src)
return v, err
}
// decodedLen returns the length of the decoded block and the number of bytes
// that the length header occupied.
func decodedLen(src []byte) (blockLen, headerLen int, err error) {
v, n := binary.Uvarint(src)
if n == 0 {
return 0, 0, ErrCorrupt
}
if uint64(int(v)) != v {
return 0, 0, errors.New("snappy: decoded block is too large")
}
return int(v), n, nil
}
// Decode returns the decoded form of src. The returned slice may be a sub-
// slice of dst if dst was large enough to hold the entire decoded block.
// Otherwise, a newly allocated slice will be returned.
// It is valid to pass a nil dst.
func Decode(dst, src []byte) ([]byte, error) {
dLen, s, err := decodedLen(src)
if err != nil {
return nil, err
}
if len(dst) < dLen {
dst = make([]byte, dLen)
}
var d, offset, length int
for s < len(src) {
switch src[s] & 0x03 {
case tagLiteral:
x := uint(src[s] >> 2)
switch {
case x < 60:
s += 1
case x == 60:
s += 2
if s > len(src) {
return nil, ErrCorrupt
}
x = uint(src[s-1])
case x == 61:
s += 3
if s > len(src) {
return nil, ErrCorrupt
}
x = uint(src[s-2]) | uint(src[s-1])<<8
case x == 62:
s += 4
if s > len(src) {
return nil, ErrCorrupt
}
x = uint(src[s-3]) | uint(src[s-2])<<8 | uint(src[s-1])<<16
case x == 63:
s += 5
if s > len(src) {
return nil, ErrCorrupt
}
x = uint(src[s-4]) | uint(src[s-3])<<8 | uint(src[s-2])<<16 | uint(src[s-1])<<24
}
length = int(x + 1)
if length <= 0 {
return nil, errors.New("snappy: unsupported literal length")
}
if length > len(dst)-d || length > len(src)-s {
return nil, ErrCorrupt
}
copy(dst[d:], src[s:s+length])
d += length
s += length
continue
case tagCopy1:
s += 2
if s > len(src) {
return nil, ErrCorrupt
}
length = 4 + int(src[s-2])>>2&0x7
offset = int(src[s-2])&0xe0<<3 | int(src[s-1])
case tagCopy2:
s += 3
if s > len(src) {
return nil, ErrCorrupt
}
length = 1 + int(src[s-3])>>2
offset = int(src[s-2]) | int(src[s-1])<<8
case tagCopy4:
return nil, errors.New("snappy: unsupported COPY_4 tag")
}
end := d + length
if offset > d || end > len(dst) {
return nil, ErrCorrupt
}
for ; d < end; d++ {
dst[d] = dst[d-offset]
}
}
if d != dLen {
return nil, ErrCorrupt
}
return dst[:d], nil
}
// NewReader returns a new Reader that decompresses from r, using the framing
// format described at
// https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
func NewReader(r io.Reader) *Reader {
return &Reader{
r: r,
decoded: make([]byte, maxUncompressedChunkLen),
buf: make([]byte, MaxEncodedLen(maxUncompressedChunkLen)+checksumSize),
}
}
// Reader is an io.Reader than can read Snappy-compressed bytes.
type Reader struct {
r io.Reader
err error
decoded []byte
buf []byte
// decoded[i:j] contains decoded bytes that have not yet been passed on.
i, j int
readHeader bool
}
// Reset discards any buffered data, resets all state, and switches the Snappy
// reader to read from r. This permits reusing a Reader rather than allocating
// a new one.
func (r *Reader) Reset(reader io.Reader) {
r.r = reader
r.err = nil
r.i = 0
r.j = 0
r.readHeader = false
}
func (r *Reader) readFull(p []byte) (ok bool) {
if _, r.err = io.ReadFull(r.r, p); r.err != nil {
if r.err == io.ErrUnexpectedEOF {
r.err = ErrCorrupt
}
return false
}
return true
}
// Read satisfies the io.Reader interface.
func (r *Reader) Read(p []byte) (int, error) {
if r.err != nil {
return 0, r.err
}
for {
if r.i < r.j {
n := copy(p, r.decoded[r.i:r.j])
r.i += n
return n, nil
}
if !r.readFull(r.buf[:4]) {
return 0, r.err
}
chunkType := r.buf[0]
if !r.readHeader {
if chunkType != chunkTypeStreamIdentifier {
r.err = ErrCorrupt
return 0, r.err
}
r.readHeader = true
}
chunkLen := int(r.buf[1]) | int(r.buf[2])<<8 | int(r.buf[3])<<16
if chunkLen > len(r.buf) {
r.err = ErrUnsupported
return 0, r.err
}
// The chunk types are specified at
// https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
switch chunkType {
case chunkTypeCompressedData:
// Section 4.2. Compressed data (chunk type 0x00).
/*
if chunkLen < checksumSize {
r.err = ErrCorrupt
return 0, r.err
}
*/
buf := r.buf[:chunkLen]
if !r.readFull(buf) {
return 0, r.err
}
/*
checksum := uint32(buf[0]) | uint32(buf[1])<<8 | uint32(buf[2])<<16 | uint32(buf[3])<<24
buf = buf[checksumSize:]
*/
n, err := DecodedLen(buf)
if err != nil {
r.err = err
return 0, r.err
}
if n > len(r.decoded) {
r.err = ErrCorrupt
return 0, r.err
}
if _, err := Decode(r.decoded, buf); err != nil {
r.err = err
return 0, r.err
}
/*
if crc(r.decoded[:n]) != checksum {
r.err = ErrCorrupt
return 0, r.err
}
*/
r.i, r.j = 0, n
continue
case chunkTypeUncompressedData:
// Section 4.3. Uncompressed data (chunk type 0x01).
if chunkLen < checksumSize {
r.err = ErrCorrupt
return 0, r.err
}
buf := r.buf[:checksumSize]
if !r.readFull(buf) {
return 0, r.err
}
checksum := uint32(buf[0]) | uint32(buf[1])<<8 | uint32(buf[2])<<16 | uint32(buf[3])<<24
// Read directly into r.decoded instead of via r.buf.
n := chunkLen - checksumSize
if !r.readFull(r.decoded[:n]) {
return 0, r.err
}
if crc(r.decoded[:n]) != checksum {
r.err = ErrCorrupt
return 0, r.err
}
r.i, r.j = 0, n
continue
case chunkTypeStreamIdentifier:
// Section 4.1. Stream identifier (chunk type 0xff).
if chunkLen != len(magicBody) {
r.err = ErrCorrupt
return 0, r.err
}
if !r.readFull(r.buf[:len(magicBody)]) {
return 0, r.err
}
for i := 0; i < len(magicBody); i++ {
if r.buf[i] != magicBody[i] {
r.err = ErrCorrupt
return 0, r.err
}
}
continue
}
if chunkType <= 0x7f {
// Section 4.5. Reserved unskippable chunks (chunk types 0x02-0x7f).
r.err = ErrUnsupported
return 0, r.err
} else {
// Section 4.4 Padding (chunk type 0xfe).
// Section 4.6. Reserved skippable chunks (chunk types 0x80-0xfd).
if !r.readFull(r.buf[:chunkLen]) {
return 0, r.err
}
}
}
}

258
vendor/code.sajari.com/docconv/snappy/encode.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,258 @@
// Copyright 2011 The Snappy-Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package snappy
import (
"encoding/binary"
"io"
)
// We limit how far copy back-references can go, the same as the C++ code.
const maxOffset = 1 << 15
// emitLiteral writes a literal chunk and returns the number of bytes written.
func emitLiteral(dst, lit []byte) int {
i, n := 0, uint(len(lit)-1)
switch {
case n < 60:
dst[0] = uint8(n)<<2 | tagLiteral
i = 1
case n < 1<<8:
dst[0] = 60<<2 | tagLiteral
dst[1] = uint8(n)
i = 2
case n < 1<<16:
dst[0] = 61<<2 | tagLiteral
dst[1] = uint8(n)
dst[2] = uint8(n >> 8)
i = 3
case n < 1<<24:
dst[0] = 62<<2 | tagLiteral
dst[1] = uint8(n)
dst[2] = uint8(n >> 8)
dst[3] = uint8(n >> 16)
i = 4
case int64(n) < 1<<32:
dst[0] = 63<<2 | tagLiteral
dst[1] = uint8(n)
dst[2] = uint8(n >> 8)
dst[3] = uint8(n >> 16)
dst[4] = uint8(n >> 24)
i = 5
default:
panic("snappy: source buffer is too long")
}
if copy(dst[i:], lit) != len(lit) {
panic("snappy: destination buffer is too short")
}
return i + len(lit)
}
// emitCopy writes a copy chunk and returns the number of bytes written.
func emitCopy(dst []byte, offset, length int) int {
i := 0
for length > 0 {
x := length - 4
if 0 <= x && x < 1<<3 && offset < 1<<11 {
dst[i+0] = uint8(offset>>8)&0x07<<5 | uint8(x)<<2 | tagCopy1
dst[i+1] = uint8(offset)
i += 2
break
}
x = length
if x > 1<<6 {
x = 1 << 6
}
dst[i+0] = uint8(x-1)<<2 | tagCopy2
dst[i+1] = uint8(offset)
dst[i+2] = uint8(offset >> 8)
i += 3
length -= x
}
return i
}
// Encode returns the encoded form of src. The returned slice may be a sub-
// slice of dst if dst was large enough to hold the entire encoded block.
// Otherwise, a newly allocated slice will be returned.
// It is valid to pass a nil dst.
func Encode(dst, src []byte) ([]byte, error) {
if n := MaxEncodedLen(len(src)); len(dst) < n {
dst = make([]byte, n)
}
// The block starts with the varint-encoded length of the decompressed bytes.
d := binary.PutUvarint(dst, uint64(len(src)))
// Return early if src is short.
if len(src) <= 4 {
if len(src) != 0 {
d += emitLiteral(dst[d:], src)
}
return dst[:d], nil
}
// Initialize the hash table. Its size ranges from 1<<8 to 1<<14 inclusive.
const maxTableSize = 1 << 14
shift, tableSize := uint(32-8), 1<<8
for tableSize < maxTableSize && tableSize < len(src) {
shift--
tableSize *= 2
}
var table [maxTableSize]int
// Iterate over the source bytes.
var (
s int // The iterator position.
t int // The last position with the same hash as s.
lit int // The start position of any pending literal bytes.
)
for s+3 < len(src) {
// Update the hash table.
b0, b1, b2, b3 := src[s], src[s+1], src[s+2], src[s+3]
h := uint32(b0) | uint32(b1)<<8 | uint32(b2)<<16 | uint32(b3)<<24
p := &table[(h*0x1e35a7bd)>>shift]
// We need to to store values in [-1, inf) in table. To save
// some initialization time, (re)use the table's zero value
// and shift the values against this zero: add 1 on writes,
// subtract 1 on reads.
t, *p = *p-1, s+1
// If t is invalid or src[s:s+4] differs from src[t:t+4], accumulate a literal byte.
if t < 0 || s-t >= maxOffset || b0 != src[t] || b1 != src[t+1] || b2 != src[t+2] || b3 != src[t+3] {
s++
continue
}
// Otherwise, we have a match. First, emit any pending literal bytes.
if lit != s {
d += emitLiteral(dst[d:], src[lit:s])
}
// Extend the match to be as long as possible.
s0 := s
s, t = s+4, t+4
for s < len(src) && src[s] == src[t] {
s++
t++
}
// Emit the copied bytes.
d += emitCopy(dst[d:], s-t, s-s0)
lit = s
}
// Emit any final pending literal bytes and return.
if lit != len(src) {
d += emitLiteral(dst[d:], src[lit:])
}
return dst[:d], nil
}
// MaxEncodedLen returns the maximum length of a snappy block, given its
// uncompressed length.
func MaxEncodedLen(srcLen int) int {
// Compressed data can be defined as:
// compressed := item* literal*
// item := literal* copy
//
// The trailing literal sequence has a space blowup of at most 62/60
// since a literal of length 60 needs one tag byte + one extra byte
// for length information.
//
// Item blowup is trickier to measure. Suppose the "copy" op copies
// 4 bytes of data. Because of a special check in the encoding code,
// we produce a 4-byte copy only if the offset is < 65536. Therefore
// the copy op takes 3 bytes to encode, and this type of item leads
// to at most the 62/60 blowup for representing literals.
//
// Suppose the "copy" op copies 5 bytes of data. If the offset is big
// enough, it will take 5 bytes to encode the copy op. Therefore the
// worst case here is a one-byte literal followed by a five-byte copy.
// That is, 6 bytes of input turn into 7 bytes of "compressed" data.
//
// This last factor dominates the blowup, so the final estimate is:
return 32 + srcLen + srcLen/6
}
// NewWriter returns a new Writer that compresses to w, using the framing
// format described at
// https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
func NewWriter(w io.Writer) *Writer {
return &Writer{
w: w,
enc: make([]byte, MaxEncodedLen(maxUncompressedChunkLen)),
}
}
// Writer is an io.Writer than can write Snappy-compressed bytes.
type Writer struct {
w io.Writer
err error
enc []byte
buf [checksumSize + chunkHeaderSize]byte
wroteHeader bool
}
// Reset discards the writer's state and switches the Snappy writer to write to
// w. This permits reusing a Writer rather than allocating a new one.
func (w *Writer) Reset(writer io.Writer) {
w.w = writer
w.err = nil
w.wroteHeader = false
}
// Write satisfies the io.Writer interface.
func (w *Writer) Write(p []byte) (n int, errRet error) {
if w.err != nil {
return 0, w.err
}
if !w.wroteHeader {
copy(w.enc, magicChunk)
if _, err := w.w.Write(w.enc[:len(magicChunk)]); err != nil {
w.err = err
return n, err
}
w.wroteHeader = true
}
for len(p) > 0 {
var uncompressed []byte
if len(p) > maxUncompressedChunkLen {
uncompressed, p = p[:maxUncompressedChunkLen], p[maxUncompressedChunkLen:]
} else {
uncompressed, p = p, nil
}
checksum := crc(uncompressed)
// Compress the buffer, discarding the result if the improvement
// isn't at least 12.5%.
chunkType := uint8(chunkTypeCompressedData)
chunkBody, err := Encode(w.enc, uncompressed)
if err != nil {
w.err = err
return n, err
}
if len(chunkBody) >= len(uncompressed)-len(uncompressed)/8 {
chunkType, chunkBody = chunkTypeUncompressedData, uncompressed
}
chunkLen := 4 + len(chunkBody)
w.buf[0] = chunkType
w.buf[1] = uint8(chunkLen >> 0)
w.buf[2] = uint8(chunkLen >> 8)
w.buf[3] = uint8(chunkLen >> 16)
w.buf[4] = uint8(checksum >> 0)
w.buf[5] = uint8(checksum >> 8)
w.buf[6] = uint8(checksum >> 16)
w.buf[7] = uint8(checksum >> 24)
if _, err = w.w.Write(w.buf[:]); err != nil {
w.err = err
return n, err
}
if _, err = w.w.Write(chunkBody); err != nil {
w.err = err
return n, err
}
n += len(uncompressed)
}
return n, nil
}

68
vendor/code.sajari.com/docconv/snappy/snappy.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,68 @@
// Copyright 2011 The Snappy-Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package snappy implements the snappy block-based compression format.
// It aims for very high speeds and reasonable compression.
//
// The C++ snappy implementation is at http://code.google.com/p/snappy/
package snappy
import (
"hash/crc32"
)
/*
Each encoded block begins with the varint-encoded length of the decoded data,
followed by a sequence of chunks. Chunks begin and end on byte boundaries. The
first byte of each chunk is broken into its 2 least and 6 most significant bits
called l and m: l ranges in [0, 4) and m ranges in [0, 64). l is the chunk tag.
Zero means a literal tag. All other values mean a copy tag.
For literal tags:
- If m < 60, the next 1 + m bytes are literal bytes.
- Otherwise, let n be the little-endian unsigned integer denoted by the next
m - 59 bytes. The next 1 + n bytes after that are literal bytes.
For copy tags, length bytes are copied from offset bytes ago, in the style of
Lempel-Ziv compression algorithms. In particular:
- For l == 1, the offset ranges in [0, 1<<11) and the length in [4, 12).
The length is 4 + the low 3 bits of m. The high 3 bits of m form bits 8-10
of the offset. The next byte is bits 0-7 of the offset.
- For l == 2, the offset ranges in [0, 1<<16) and the length in [1, 65).
The length is 1 + m. The offset is the little-endian unsigned integer
denoted by the next 2 bytes.
- For l == 3, this tag is a legacy format that is no longer supported.
*/
const (
tagLiteral = 0x00
tagCopy1 = 0x01
tagCopy2 = 0x02
tagCopy4 = 0x03
)
const (
checksumSize = 4
chunkHeaderSize = 4
magicChunk = "\xff\x06\x00\x00" + magicBody
magicBody = "sNaPpY"
// https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt says
// that "the uncompressed data in a chunk must be no longer than 65536 bytes".
maxUncompressedChunkLen = 65536
)
const (
chunkTypeCompressedData = 0x00
chunkTypeUncompressedData = 0x01
chunkTypePadding = 0xfe
chunkTypeStreamIdentifier = 0xff
)
var crcTable = crc32.MakeTable(crc32.Castagnoli)
// crc implements the checksum specified in section 3 of
// https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
func crc(b []byte) uint32 {
c := crc32.Update(0, crcTable, b)
return uint32(c>>15|c<<17) + 0xa282ead8
}

32
vendor/code.sajari.com/docconv/tidy.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,32 @@
package docconv
import (
"io"
"io/ioutil"
"os"
"os/exec"
)
// Tidy attempts to tidy up XML.
// Errors & warnings are deliberately suppressed as underlying tools
// throw warnings very easily.
func Tidy(r io.Reader, xmlIn bool) ([]byte, error) {
f, err := ioutil.TempFile(os.TempDir(), "/docconv")
if err != nil {
return nil, err
}
defer os.Remove(f.Name())
io.Copy(f, r)
var output []byte
if xmlIn {
output, err = exec.Command("tidy", "-xml", "-numeric", "-asxml", "-quiet", "-utf8", f.Name()).Output()
} else {
output, err = exec.Command("tidy", "-numeric", "-asxml", "-quiet", "-utf8", f.Name()).Output()
}
if err != nil && err.Error() != "exit status 1" {
return nil, err
}
return output, nil
}

31
vendor/code.sajari.com/docconv/url.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,31 @@
package docconv
import (
"bytes"
"io"
"github.com/advancedlogic/GoOse"
)
// ConvertURL fetches the HTML page at the URL given in the io.Reader.
func ConvertURL(input io.Reader, readability bool) (string, map[string]string, error) {
meta := make(map[string]string)
buf := new(bytes.Buffer)
_, err := buf.ReadFrom(input)
if err != nil {
return "", nil, err
}
g := goose.New()
article, err := g.ExtractFromURL(buf.String())
if err != nil {
return "", nil, err
}
meta["title"] = article.Title
meta["description"] = article.MetaDescription
meta["image"] = article.TopImage
return article.CleanedText, meta, nil
}

98
vendor/code.sajari.com/docconv/xml.go сгенерированный поставляемый Обычный файл
Просмотреть файл

@@ -0,0 +1,98 @@
package docconv
import (
"bytes"
"encoding/xml"
"fmt"
"io"
)
// ConvertXML converts an XML file to text.
func ConvertXML(r io.Reader) (string, map[string]string, error) {
meta := make(map[string]string)
cleanXML, err := Tidy(r, true)
if err != nil {
return "", nil, fmt.Errorf("tidy error: %v", err)
}
result, err := XMLToText(bytes.NewReader(cleanXML), []string{}, []string{}, true)
if err != nil {
return "", nil, fmt.Errorf("error from XMLToText: %v", err)
}
return result, meta, nil
}
// XMLToText converts XML to plain text given how to treat elements.
func XMLToText(r io.Reader, breaks []string, skip []string, strict bool) (string, error) {
var result string
dec := xml.NewDecoder(r)
dec.Strict = strict
for {
t, err := dec.Token()
if err != nil {
if err == io.EOF {
break
}
return "", err
}
switch v := t.(type) {
case xml.CharData:
result += string(v)
case xml.StartElement:
for _, breakElement := range breaks {
if v.Name.Local == breakElement {
result += "\n"
}
}
for _, skipElement := range skip {
if v.Name.Local == skipElement {
depth := 1
for {
t, err := dec.Token()
if err != nil {
// An io.EOF here is actually an error.
return "", err
}
switch t.(type) {
case xml.StartElement:
depth++
case xml.EndElement:
depth--
}
if depth == 0 {
break
}
}
}
}
}
}
return result, nil
}
// XMLToMap converts XML to a nested string map.
func XMLToMap(r io.Reader) (map[string]string, error) {
m := make(map[string]string)
dec := xml.NewDecoder(r)
var tagName string
for {
t, err := dec.Token()
if err != nil {
if err == io.EOF {
break
}
return nil, err
}
switch v := t.(type) {
case xml.StartElement:
tagName = string(v.Name.Local)
case xml.CharData:
m[tagName] = string(v)
}
}
return m, nil
}