Merge pull request #124 from BenB196/staging

Staging to master

README.md

@@ -6,16 +6,16 @@ The goal of this application is to allow the user to pull FFS logs from Code42's
 
 ## Important Notes:
 
-- This application, while working, is still considered under development, as it has not be thoroughly tested. There are likely bugs and random errors/panics which may occur. If they do, please report them so that they can be fixed.
+- This application, while working, is still considered under development, as it has not been thoroughly tested. There are likely bugs and random errors/panics which may occur. If they do, please report them so that they can be fixed.
 - The slowest part of this application is the downloading of the logs. You should try to optimize your queries as much as possible.
 
 Limitations:
 
 Code42 Crashplan FFS API has limitations like most APIs, these limitations affect the functionality of the application as followed:
 
 1. 120 Queries per minute, any additional queries will be dropped. (never actually bothered to test if/how this limit is actually enforced)
-1. 200,000 results returned per query. This limitation is kind of annoying to handle as there is no easy way to handle it. The API does not support paging and the only way to figure out how many results there is for a query is to first query, count, then if over 200,000 results, break up the query into smaller time increments and perform multiple queries to get all of the results.
-1. The application only supports the /v1/fileevent/export API endpoint currently. This has to do with how the highly limited functionality of the /v1/fileevent endpoint which isn't well documented.
+   1. It is recommended to set the page size of the query to at least 1000, but less than 10000 to reduce the number of queries required
+1. The application only supports the use of the /v1/fileevent API endpoint, which provides a JSON formatted output. 
 
 
 ## Install
@@ -33,7 +33,7 @@ $ /path/to/output/location/crashplan-ffs-puller --config=/path/to/config.json
 
 ### Precompiled Binaries
 
-These are found attached to each official release. Currently only Windows amd64 and Linux amd64 binaries will be released. As the application progresses, or more needs come up, more binaries will be added.
+These are found attached to each official release. Currently, only Windows amd64 and Linux amd64 binaries will be released. As the application progresses or more needs arise, more binaries will be added.
 
 ### Docker
 
@@ -73,7 +73,7 @@ Currently, only JSON formatted configuration files are accepted, in the future Y
 ```
 {
   "authURI": "https://www.crashplan.com/c42api/v3/auth/jwt?useBody=true",                                                   #This is the URI which has the Code42 authentication endpoint.
-  "ffsURI": "https://forensicsearch-default.prod.ffs.us2.code42.com/forensic-search/queryservice/api/v1/fileevent/export",  #This is the URI which exposes the FFS API. Note: Currently only supports the fileevent/export endpoint.
+  "ffsURI": "https://forensicsearch-default.prod.ffs.us2.code42.com/forensic-search/queryservice/api/v1/fileevent",         #This is the URI which exposes the FFS API. Note: Currently only supports the /fileevent endpoint.
   "ffsQueries": [{                                                                                                          #This is an area of FFS Queries + additional information.
     "name": "example_query_1",                                                                                              #Query name, must be unique.
     "username": "[email protected]",                                                                                      #Username, must be an email address.
@@ -103,7 +103,8 @@ Currently, only JSON formatted configuration files are accepted, in the future Y
           ],
           "filterClause": "AND"
         }
-      ]
+      ],
+      "pgSize": 1000
     },
     "outputType": "elastic",                                                                                                #Output type, supports either file, elastic, logstash
     "outputLocation": "/path/to/output",                                                                                    #This is needed even if not using file output type, as there are stateful files which need to be written and stored.
@@ -193,7 +194,7 @@ In the above configuration there are some important notes to know about the FFS
 
 Note: I have not tested out all possible queries in this application, if you come across a query which does not work, let me know and I will try to get it working.
 
-### Elasticsearch Integration
+### Elasticsearch Integration (WIP)
 
 If you are using the elastic output type there are a few important things to understand.
 
@@ -256,5 +257,4 @@ If you have any ideas for other metrics you feel may be useful, feel free to ope
 ##TODOs (Maybe)
 
 1. Add ability to use yaml/yml configuration files.
-2. Add the ability to use the regular json FFS API endpoint (/fileevent).
 3. Add some sort of file hash lookup that could provide threat intelligence (tried this with [OTX](https://www.alienvault.com/open-threat-exchange) before and failed pretty bad, may look at revisiting).

VERSION

@@ -1 +1 @@
-0.4.4
+0.5.0

eventOutput/fileHandler.go

@@ -15,42 +15,47 @@ import (
 )
 
 type FFSEvent struct {
-	ffs.FileEvent
+	ffs.JsonFileEvent
 	*ip_api.Location `json:",omitempty"`
 	GeoLocation      *Location `json:"geoPoint,omitempty"`
 }
 
 type Code42 struct {
-	Event                       *Code42Event          `json:"event,omitempty"`
-	InsertionTimestamp          *time.Time            `json:"insertion_timestamp,omitempty"`
-	File                        *Code42File           `json:"file,omitempty"`
-	Device                      *Code42Device         `json:"device,omitempty"`
-	OsHostName                  string                `json:"os_host_name,omitempty"`
-	DomainName                  string                `json:"domain_name,omitempty"`
-	PublicIpAddress             string                `json:"public_ip_address,omitempty"`
-	PrivateIpAddresses          []string              `json:"private_ip_addresses,omitempty"`
-	Actor                       string                `json:"actor,omitempty"`
-	DirectoryId                 []string              `json:"directory_id,omitempty"`
-	Source                      string                `json:"source,omitempty"`
-	Url                         *URL                  `json:"url,omitempty"`
-	Shared                      *bool                 `json:"shared,omitempty"`
-	SharedWith                  []string              `json:"shared_with,omitempty"`
-	SharingTypeAdded            []string              `json:"sharing_type_added,omitempty"`
-	CloudDriveId                string                `json:"cloud_drive_id,omitempty"`
-	DetectionSourceAlias        string                `json:"detection_source_alias,omitempty"`
-	Exposure                    []string              `json:"exposure,omitempty"`
-	Process                     *Code42Process        `json:"process,omitempty"`
-	Tab                         *Code42Tab            `json:"tab,omitempty"`
-	RemovableMedia              *Code42RemovableMedia `json:"removable_media,omitempty"`
-	SyncDestination             string                `json:"sync_destination,omitempty"`
-	SyncDestinationUsername     string                `json:"sync_destination_username,omitempty"`
-	EmailDlp                    *Code42EmailDlp       `json:"email_dlp,omitempty"`
-	OutsideActiveHours          *bool                 `json:"outside_active_hours,omitempty"`
-	Print                       *Code42Print          `json:"print,omitempty"`
-	RemoteActivity              string                `json:"remote_activity,omitempty"`
-	Trusted                     *bool                 `json:"trusted,omitempty"`
-	LoggedInOperatingSystemUser string                `json:"logged_in_operating_system_user,omitempty"`
-	Destination                 *Code42Destination    `json:"destination,omitempty"`
+	Event                   *Code42Event          `json:"event,omitempty"`
+	InsertionTimestamp      *time.Time            `json:"insertion_timestamp,omitempty"`
+	File                    *Code42File           `json:"file,omitempty"`
+	Device                  *Code42Device         `json:"device,omitempty"`
+	OsHostName              string                `json:"os_host_name,omitempty"`
+	DomainName              string                `json:"domain_name,omitempty"`
+	PublicIpAddress         string                `json:"public_ip_address,omitempty"`
+	PrivateIpAddresses      []string              `json:"private_ip_addresses,omitempty"`
+	Actor                   string                `json:"actor,omitempty"`
+	DirectoryId             []string              `json:"directory_id,omitempty"`
+	Source                  string                `json:"source,omitempty"`
+	Url                     *URL                  `json:"url,omitempty"`
+	Shared                  string                `json:"shared,omitempty"`
+	SharedWith              []ffs.SharedWith      `json:"shared_with,omitempty"`
+	SharingTypeAdded        []string              `json:"sharing_type_added,omitempty"`
+	CloudDriveId            string                `json:"cloud_drive_id,omitempty"`
+	DetectionSourceAlias    string                `json:"detection_source_alias,omitempty"`
+	Exposure                []string              `json:"exposure,omitempty"`
+	Process                 *Code42Process        `json:"process,omitempty"`
+	RemovableMedia          *Code42RemovableMedia `json:"removable_media,omitempty"`
+	SyncDestination         string                `json:"sync_destination,omitempty"`
+	SyncDestinationUsername []string              `json:"sync_destination_username,omitempty"`
+	EmailDlp                *Code42EmailDlp       `json:"email_dlp,omitempty"`
+	OutsideActiveHours      *bool                 `json:"outside_active_hours,omitempty"`
+	Print                   *Code42Print          `json:"print,omitempty"`
+	RemoteActivity          string                `json:"remote_activity,omitempty"`
+	Trusted                 *bool                 `json:"trusted,omitempty"`
+	OperatingSystemUser     string                `json:"operating_system_user,omitempty"`
+	Destination             *Code42Destination    `json:"destination,omitempty"`
+	Tabs                    []Code42TabTab        `json:"tabs,omitempty"`
+}
+
+type Code42TabTab struct {
+	Title string `json:"title,omitempty"`
+	Url   *URL   `json:"url,omitempty"`
 }
 
 type Code42Event struct {
@@ -64,17 +69,17 @@ type Code42File struct {
 	Name                        string     `json:"name,omitempty"`
 	Type                        string     `json:"type,omitempty"`
 	Category                    string     `json:"category,omitempty"`
-	IdentifiedExtensionCategory string     `json:"identified_extension_category,omitempty"`
-	CurrentExtensionCategory    string     `json:"current_extension_category,omitempty"`
-	Size                        *int       `json:"size,omitempty"`
-	Owner                       []string   `json:"owner,omitempty"`
+	MimeTypeByBytes             string     `json:"mime_type_by_bytes,omitempty"`
+	MimeTypeByExtension         string     `json:"mime_type_by_extension,omitempty"`
+	Size                        *int64     `json:"size,omitempty"`
+	Owner                       string     `json:"owner,omitempty"`
 	Hash                        *Hash      `json:"hash,omitempty"`
-	CreatedTimestamp            *time.Time `json:"created_timestamp,omitempty"`
+	CreateTimestamp             *time.Time `json:"create_timestamp,omitempty"`
 	ModifyTimestamp             *time.Time `json:"modify_timestamp,omitempty"`
 	Id                          string     `json:"id,omitempty"`
 	IdentifiedExtensionMIMEType string     `json:"identified_extension_mime_type,omitempty"`
 	CurrentExtensionMIMEType    string     `json:"current_extension_mime_type,omitempty"`
-	SuspiciousFileTypeMismatch  *bool      `json:"suspicious_file_type_mismatch,omitempty"`
+	MimeTypeMismatch            *bool      `json:"mime_type_mismatch,omitempty"`
 }
 
 type Code42Device struct {
@@ -85,19 +90,17 @@ type Code42Device struct {
 type Code42Tab struct {
 	WindowTitle string   `json:"window_title,omitempty"`
 	Url         *URL     `json:"url,omitempty"`
-	Titles      []string `json:"titles,omitempty"`
-	Urls        []URL    `json:"urls,omitempty"`
 }
 
 type Code42RemovableMedia struct {
-	Vendor       string `json:"vendor,omitempty"`
-	Name         string `json:"name,omitempty"`
-	SerialNumber string `json:"serial_number,omitempty"`
-	Capacity     *int   `json:"capacity,omitempty"`
-	BusType      string `json:"bus_type,omitempty"`
-	MediaName    string `json:"media_name,omitempty"`
-	VolumeName   string `json:"volume_name,omitempty"`
-	PartitionId  string `json:"partition_id,omitempty"`
+	Vendor       string   `json:"vendor,omitempty"`
+	Name         string   `json:"name,omitempty"`
+	SerialNumber string   `json:"serial_number,omitempty"`
+	Capacity     *int64   `json:"capacity,omitempty"`
+	BusType      string   `json:"bus_type,omitempty"`
+	MediaName    string   `json:"media_name,omitempty"`
+	VolumeName   []string `json:"volume_name,omitempty"`
+	PartitionId  []string `json:"partition_id,omitempty"`
 }
 
 type Code42EmailDlp struct {
@@ -109,9 +112,8 @@ type Code42EmailDlp struct {
 }
 
 type Code42Print struct {
-	JobName                string `json:"job_name,omitempty"`
-	PrinterName            string `json:"name,omitempty"`
-	PrintedFilesBackupPath string `json:"printed_files_backup_path,omitempty"`
+	JobName     string `json:"job_name,omitempty"`
+	PrinterName string `json:"name,omitempty"`
 }
 
 type Code42Destination struct {
@@ -165,12 +167,12 @@ type File struct {
 	Created   *time.Time `json:"created,omitempty"`
 	Directory []string   `json:"directory,omitempty"`
 	Extension string     `json:"extension,omitempty"`
-	MimeType  string     `json:"mime_type,omitempty"`
+	MimeType  []string   `json:"mime_type,omitempty"`
 	Mtime     *time.Time `json:"mtime,omitempty"`
 	Name      string     `json:"name,omitempty"`
-	Owner     []string   `json:"owner,omitempty"`
+	Owner     string     `json:"owner,omitempty"`
 	Path      string     `json:"path,omitempty"`
-	Size      *int       `json:"size,omitempty"`
+	Size      *int64     `json:"size,omitempty"`
 	Type      string     `json:"type,omitempty"`
 	Hash      *Hash      `json:"hash,omitempty"`
 }