Package org.opencastproject.distribution.aws.s3

Class AwsS3DistributionServiceImpl

java.lang.Object

org.opencastproject.job.api.AbstractJobProducer

org.opencastproject.distribution.api.AbstractDistributionService

org.opencastproject.distribution.aws.s3.AwsS3DistributionServiceImpl

All Implemented Interfaces:

DistributionService, DownloadDistributionService, AwsS3DistributionService, JobProducer

public class AwsS3DistributionServiceImpl
extends AbstractDistributionService
implements AwsS3DistributionService, DistributionService

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	AwsS3DistributionServiceImpl.Operation	List of available operations on jobs

Field Summary

Fields

Modifier and Type	Field	Description
static final String	AWS_S3_ACCESS_KEY_ID_CONFIG
static final String	AWS_S3_BUCKET_CONFIG
static final String	AWS_S3_BUCKET_CONFIG_PREFIX
static final String	AWS_S3_CONNECTION_TIMEOUT
static final String	AWS_S3_DISTRIBUTION_BASE_CONFIG
static final String	AWS_S3_DISTRIBUTION_ENABLE
static final String	AWS_S3_ENDPOINT_CONFIG
static final String	AWS_S3_MAX_CONNECTIONS
static final String	AWS_S3_MAX_RETRIES
static final String	AWS_S3_PATH_STYLE_CONFIG
static final String	AWS_S3_PRESIGNED_URL_CONFIG
static final String	AWS_S3_PRESIGNED_URL_VALID_DURATION_CONFIG
static final String	AWS_S3_REGION_CONFIG
static final String	AWS_S3_SECRET_ACCESS_KEY_CONFIG
static final int	DEFAULT_CONNECTION_TIMEOUT
static final float	DEFAULT_DISTRIBUTE_JOB_LOAD	The load on the system introduced by creating a distribute job
static final int	DEFAULT_MAX_CONNECTIONS
static final int	DEFAULT_MAX_RETRIES
static final String	DEFAULT_ORG_KEY
static final int	DEFAULT_PRESIGNED_URL_EXPIRE_MILLIS	Default expiration time for presigned URL in millis, 6 hours
static final float	DEFAULT_RETRACT_JOB_LOAD	The load on the system introduced by creating a retract job
static final String	DEFAULT_TEMP_DIR
static final String	DISTRIBUTE_JOB_LOAD_KEY
static final String	JOB_TYPE	Job type
static final String	OPENCAST_STORAGE_DIR
static final String	RETRACT_JOB_LOAD_KEY

Fields inherited from class org.opencastproject.distribution.api.AbstractDistributionService

distributionChannel, distributionDirectory, organizationDirectoryService, securityService, serviceRegistry, serviceUrl, trustedHttpClient, userDirectoryService, workspace

Fields inherited from class org.opencastproject.job.api.AbstractJobProducer

ACCEPT_JOB_LOADS_EXCEEDING_PROPERTY, acceptJobLoadsExeedingMaxLoad, DEFAULT_ACCEPT_JOB_LOADS_EXCEEDING, executor, jobType

Fields inherited from interface org.opencastproject.distribution.api.DistributionService

CONFIG_KEY_STORE_TYPE, JOB_TYPE_PREFIX

Constructor Summary

Constructors

Constructor	Description
AwsS3DistributionServiceImpl()	Creates a new instance of the AWS S3 distribution service.

Method Summary

Modifier and Type	Method	Description
void	activate(org.osgi.service.component.ComponentContext cc)	OSGI activate method.
protected String	buildObjectName(String orgId, String channelId, String mpId, String elementId, String fileName)	Builds the aws s3 object name using the raw elementID and filename
protected String	buildObjectName(String channelId, String mpId, MediaPackageElement element)	Builds the aws s3 object name.
protected void	createAWSBucket()	Creates the AWS S3 bucket if it doesn't exist yet.
void	deactivate()
Job	distribute(String channelId, MediaPackage mediapackage, String elementId)	Distribute a media package element.
Job	distribute(String channelId, MediaPackage mediaPackage, String elementId, boolean checkAvailability)
Job	distribute(String channelId, MediaPackage mediaPackage, Set<String> elementIds, boolean checkAvailability)
Job	distribute(String pubChannelId, MediaPackage mediaPackage, Set<String> downloadIds, boolean checkAvailability, boolean preserveReference)
MediaPackageElement	distributeElement(String channelId, MediaPackage mediaPackage, MediaPackageElement element, boolean checkAvailability)	Distribute a media package element to AWS S3.
MediaPackageElement[]	distributeElements(String channelId, MediaPackage mediapackage, Set<String> elementIds, boolean checkAvailability)	Distribute Mediapackage elements to the download distribution service.
List<MediaPackageElement>	distributeSync(String channelId, MediaPackage mediapackage, String elementId)	Distributes a media package element synchronously, bypassing the Opencast job system.
List<MediaPackageElement>	distributeSync(String channelId, MediaPackage mediapackage, Set<String> elementIds, boolean checkAvailability)	Distributes the given elements synchronously.
protected String	getDistributedObjectName(MediaPackageElement element)	Gets the distributed object's name.
String	getDistributionType()	Returns the distribution type for this service.
protected URI	getDistributionUri(String objectName)	Gets the URI for the element to be distributed.
URI	presignedURI(URI uri)
protected String	process(Job job)	Asks the overriding class to process the arguments using the given operation.
Job	retract(String channelId, MediaPackage mediapackage, String elementId)	Retract a media package element from the distribution channel.
Job	retract(String channelId, MediaPackage mediapackage, Set<String> elementIds)
protected MediaPackageElement	retractElement(String channelId, MediaPackage mediaPackage, MediaPackageElement element)	Retracts the media package element with the given identifier from the distribution channel.
protected MediaPackageElement[]	retractElements(String channelId, MediaPackage mediapackage, Set<String> elementIds)	Retract a media package element from the distribution channel.
List<MediaPackageElement>	retractSync(String channelId, MediaPackage mediapackage, String elementId)	Retract a media package element from the distribution channel synchronously, bypassing the Opencast job system.
List<MediaPackageElement>	retractSync(String channelId, MediaPackage mediaPackage, Set<String> elementIds)	Retracts the given elements synchronously.
protected void	setBucketName(String orgId, String bucketName)
protected void	setOpencastDistributionUrl(String distributionUrl)
void	setOrganizationDirectoryService(OrganizationDirectoryService organizationDirectoryService)	Sets a reference to the organization directory service.
protected void	setS3(com.amazonaws.services.s3.AmazonS3 s3)	The methods below are used by the test class
protected void	setS3TransferManager(com.amazonaws.services.s3.transfer.TransferManager s3TransferManager)
void	setSecurityService(SecurityService securityService)	Callback for setting the security service.
void	setServiceRegistry(ServiceRegistry serviceRegistry)	Callback for the OSGi environment to set the service registry reference.
protected void	setStorageTmp(String path)
void	setUserDirectoryService(UserDirectoryService userDirectoryService)	Callback for setting the user directory service.
void	setWorkspace(Workspace workspace)	Callback for the OSGi environment to set the workspace reference.

Methods inherited from class org.opencastproject.distribution.api.AbstractDistributionService

getOrganizationDirectoryService, getSecurityService, getServiceRegistry, getTrustedHttpClient, getUserDirectoryService, setTrustedHttpClient

Methods inherited from class org.opencastproject.job.api.AbstractJobProducer

acceptJob, countJobs, finallyUpdateJob, getJobType, incident, isReadyToAccept, isReadyToAcceptJobs

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

JOB_TYPE

public static final String JOB_TYPE

Job type

See Also:

• Constant Field Values

AWS_S3_DISTRIBUTION_ENABLE

public static final String AWS_S3_DISTRIBUTION_ENABLE

See Also:

• Constant Field Values

AWS_S3_DISTRIBUTION_BASE_CONFIG

public static final String AWS_S3_DISTRIBUTION_BASE_CONFIG

See Also:

• Constant Field Values

AWS_S3_ACCESS_KEY_ID_CONFIG

public static final String AWS_S3_ACCESS_KEY_ID_CONFIG

See Also:

• Constant Field Values

AWS_S3_SECRET_ACCESS_KEY_CONFIG

public static final String AWS_S3_SECRET_ACCESS_KEY_CONFIG

See Also:

• Constant Field Values

AWS_S3_REGION_CONFIG

public static final String AWS_S3_REGION_CONFIG

See Also:

• Constant Field Values

AWS_S3_BUCKET_CONFIG

public static final String AWS_S3_BUCKET_CONFIG

See Also:

• Constant Field Values

AWS_S3_BUCKET_CONFIG_PREFIX

public static final String AWS_S3_BUCKET_CONFIG_PREFIX

See Also:

• Constant Field Values

AWS_S3_ENDPOINT_CONFIG

public static final String AWS_S3_ENDPOINT_CONFIG

See Also:

• Constant Field Values

AWS_S3_PATH_STYLE_CONFIG

public static final String AWS_S3_PATH_STYLE_CONFIG

See Also:

• Constant Field Values

AWS_S3_PRESIGNED_URL_CONFIG

public static final String AWS_S3_PRESIGNED_URL_CONFIG

See Also:

• Constant Field Values

AWS_S3_PRESIGNED_URL_VALID_DURATION_CONFIG

public static final String AWS_S3_PRESIGNED_URL_VALID_DURATION_CONFIG

See Also:

• Constant Field Values

AWS_S3_MAX_CONNECTIONS

public static final String AWS_S3_MAX_CONNECTIONS

See Also:

• Constant Field Values

AWS_S3_CONNECTION_TIMEOUT

public static final String AWS_S3_CONNECTION_TIMEOUT

See Also:

• Constant Field Values

AWS_S3_MAX_RETRIES

public static final String AWS_S3_MAX_RETRIES

See Also:

• Constant Field Values

DISTRIBUTE_JOB_LOAD_KEY

public static final String DISTRIBUTE_JOB_LOAD_KEY

See Also:

• Constant Field Values

RETRACT_JOB_LOAD_KEY

public static final String RETRACT_JOB_LOAD_KEY

See Also:

• Constant Field Values

OPENCAST_STORAGE_DIR

public static final String OPENCAST_STORAGE_DIR

See Also:

• Constant Field Values

DEFAULT_TEMP_DIR

public static final String DEFAULT_TEMP_DIR

See Also:

• Constant Field Values

DEFAULT_MAX_CONNECTIONS

public static final int DEFAULT_MAX_CONNECTIONS

See Also:

• Constant Field Values

DEFAULT_CONNECTION_TIMEOUT

public static final int DEFAULT_CONNECTION_TIMEOUT

See Also:

• Constant Field Values

DEFAULT_MAX_RETRIES

public static final int DEFAULT_MAX_RETRIES

See Also:

• Constant Field Values

DEFAULT_DISTRIBUTE_JOB_LOAD

public static final float DEFAULT_DISTRIBUTE_JOB_LOAD

The load on the system introduced by creating a distribute job

See Also:

• Constant Field Values

DEFAULT_RETRACT_JOB_LOAD

public static final float DEFAULT_RETRACT_JOB_LOAD

The load on the system introduced by creating a retract job

See Also:

• Constant Field Values

DEFAULT_PRESIGNED_URL_EXPIRE_MILLIS

public static final int DEFAULT_PRESIGNED_URL_EXPIRE_MILLIS

Default expiration time for presigned URL in millis, 6 hours

See Also:

• Constant Field Values

DEFAULT_ORG_KEY

public static final String DEFAULT_ORG_KEY

See Also:

• Constant Field Values

Constructor Details

AwsS3DistributionServiceImpl

public AwsS3DistributionServiceImpl()

Creates a new instance of the AWS S3 distribution service.

Method Details

activate

public void activate(org.osgi.service.component.ComponentContext cc)

Description copied from class: AbstractJobProducer

OSGI activate method.

Overrides:

activate in class AbstractJobProducer

Parameters:

cc - OSGI component context

getDistributionType

public String getDistributionType()

Description copied from interface: DistributionService

Returns the distribution type for this service. This type should be unique within an Opencast instance, and is used to select where file distribution happens.

Specified by:

getDistributionType in interface DistributionService

Returns:

The distribution type. A string like "download", or "streaming"

deactivate

public void deactivate()

distribute

public Job distribute(String pubChannelId,
 MediaPackage mediaPackage,
 Set<String> downloadIds,
 boolean checkAvailability,
 boolean preserveReference)
               throws DistributionException,
MediaPackageException

Specified by:

distribute in interface DownloadDistributionService

Throws:

DistributionException

MediaPackageException

distribute

public Job distribute(String channelId,
 MediaPackage mediaPackage,
 Set<String> elementIds,
 boolean checkAvailability)
               throws DistributionException,
MediaPackageException

Specified by:

distribute in interface DownloadDistributionService

Throws:

DistributionException

MediaPackageException

See Also:

• DownloadDistributionService.distribute(String, org.opencastproject.mediapackage.MediaPackage, String, boolean)

distribute

public Job distribute(String channelId,
 MediaPackage mediapackage,
 String elementId)
               throws DistributionException,
MediaPackageException

Distribute a media package element.

Specified by:

distribute in interface DistributionService

Parameters:

mediapackage - the media package

elementId - the element in the media package to distribute

Returns:

The job

Throws:

DistributionException - if there was a problem distributing the media

MediaPackageException - if there was a problem with the mediapackage element

See Also:

• DistributionService.distribute(String, org.opencastproject.mediapackage.MediaPackage, String)

distribute

public Job distribute(String channelId,
 MediaPackage mediaPackage,
 String elementId,
 boolean checkAvailability)
               throws DistributionException,
MediaPackageException

Specified by:

distribute in interface DownloadDistributionService

Throws:

DistributionException

MediaPackageException

See Also:

• DownloadDistributionService.distribute(String, org.opencastproject.mediapackage.MediaPackage, String, boolean)

distributeElements

public MediaPackageElement[] distributeElements(String channelId,
 MediaPackage mediapackage,
 Set<String> elementIds,
 boolean checkAvailability)
                                         throws DistributionException

Distribute Mediapackage elements to the download distribution service.

Parameters:

channelId - # The id of the publication channel to be distributed to.

mediapackage - The media package that contains the elements to be distributed.

elementIds - The ids of the elements that should be distributed contained within the media package.

checkAvailability - Check the availability of the distributed element via http.

Returns:

A reference to the MediaPackageElements that have been distributed.

Throws:

DistributionException - Thrown if the parent directory of the MediaPackageElement cannot be created, if the MediaPackageElement cannot be copied or another unexpected exception occurs.

distributeElement

public MediaPackageElement distributeElement(String channelId,
 MediaPackage mediaPackage,
 MediaPackageElement element,
 boolean checkAvailability)
                                      throws DistributionException

Distribute a media package element to AWS S3.

Parameters:

mediaPackage - The media package that contains the element to distribute.

element - The element that should be distributed contained within the media package.

checkAvailability - Checks if the distributed element is available

Returns:

A reference to the MediaPackageElement that has been distributed.

Throws:

DistributionException

retract

public Job retract(String channelId,
 MediaPackage mediapackage,
 String elementId)
            throws DistributionException

Description copied from interface: DistributionService

Retract a media package element from the distribution channel.

Specified by:

retract in interface DistributionService

Parameters:

mediapackage - the media package

elementId - the media package element to retract

Throws:

DistributionException - if there was a problem retracting the mediapackage

retract

public Job retract(String channelId,
 MediaPackage mediapackage,
 Set<String> elementIds)
            throws DistributionException

Specified by:

retract in interface DownloadDistributionService

Throws:

DistributionException

distributeSync

public List<MediaPackageElement> distributeSync(String channelId,
 MediaPackage mediapackage,
 String elementId)
                                         throws DistributionException,
MediaPackageException

Description copied from interface: DistributionService

Distributes a media package element synchronously, bypassing the Opencast job system.

Specified by:

distributeSync in interface DistributionService

Parameters:

mediapackage - the media package

elementId - the element in the media package to distribute

Returns:

list of distributed media package elements

Throws:

DistributionException - if there was a problem distributing the media

MediaPackageException - if there was a problem with the mediapackage element

distributeSync

public List<MediaPackageElement> distributeSync(String channelId,
 MediaPackage mediapackage,
 Set<String> elementIds,
 boolean checkAvailability)
                                         throws DistributionException

Description copied from interface: DownloadDistributionService

Distributes the given elements synchronously. This should be used rarely since load balancing will be unavailable. However, since the dispatching logic is bypassed, synchronous execution is much faster. It is useful in interactive scenarios where you synchronously wait for job execution anyway and you don't want to make the user waiting for too long.

Specified by:

distributeSync in interface DownloadDistributionService

Parameters:

channelId - The channel to retract from.

mediapackage - A media package holding the elements to retract.

elementIds - The IDs of the elements to retract.

checkAvailability - Whether to check if the distributed elements are available through their URI.

Returns:

The distributed elements.

Throws:

DistributionException - In case distribution fails.

retractSync

public List<MediaPackageElement> retractSync(String channelId,
 MediaPackage mediapackage,
 String elementId)
                                      throws DistributionException

Description copied from interface: DistributionService

Retract a media package element from the distribution channel synchronously, bypassing the Opencast job system.

Specified by:

retractSync in interface DistributionService

Parameters:

mediapackage - the media package

elementId - the media package element to retract

Returns:

list of retracted media package elements

Throws:

DistributionException - if there was a problem retracting the mediapackage

retractSync

public List<MediaPackageElement> retractSync(String channelId,
 MediaPackage mediaPackage,
 Set<String> elementIds)
                                      throws DistributionException

Description copied from interface: DownloadDistributionService

Retracts the given elements synchronously. This should be used rarely since load balancing will be unavailable. However, since the dispatching logic is bypassed, synchronous execution is much faster. It is useful in interactive scenarios where you synchronously wait for job execution anyway and you don't want to make the user waiting for too long.

Specified by:

retractSync in interface DownloadDistributionService

Parameters:

channelId - The channel to retract from.

mediaPackage - A media package holding the elements to retract.

elementIds - The IDs of the elements to retract.

Returns:

The retracted elements.

Throws:

DistributionException - In case retraction fails.

retractElement

protected MediaPackageElement retractElement(String channelId,
 MediaPackage mediaPackage,
 MediaPackageElement element)
                                      throws DistributionException

Retracts the media package element with the given identifier from the distribution channel.

Parameters:

channelId - the channel id

mediaPackage - the media package

element - the element

Returns:

the retracted element or null if the element was not retracted

Throws:

DistributionException

retractElements

protected MediaPackageElement[] retractElements(String channelId,
 MediaPackage mediapackage,
 Set<String> elementIds)
                                         throws DistributionException

Retract a media package element from the distribution channel. The retracted element must not necessarily be the one given as parameter elementId. Instead, the element's distribution URI will be calculated. This way you are able to retract elements by providing the "original" element here.

Parameters:

channelId - the channel id

mediapackage - the mediapackage

elementIds - the element identifiers

Returns:

the retracted element or null if the element was not retracted

Throws:

DistributionException - in case of an error

buildObjectName

protected String buildObjectName(String channelId,
 String mpId,
 MediaPackageElement element)

Builds the aws s3 object name.

Parameters:

channelId -

mpId -

element -

Returns:

buildObjectName

protected String buildObjectName(String orgId,
 String channelId,
 String mpId,
 String elementId,
 String fileName)

Builds the aws s3 object name using the raw elementID and filename

Parameters:

orgId -

channelId -

mpId -

elementId -

fileName -

Returns:

getDistributionUri

protected URI getDistributionUri(String objectName)
                          throws URISyntaxException

Gets the URI for the element to be distributed.

Returns:

The resulting URI after distribution

Throws:

URISyntaxException - if the concrete implementation tries to create a malformed uri

getDistributedObjectName

protected String getDistributedObjectName(MediaPackageElement element)

Gets the distributed object's name.

Returns:

The distributed object name

process

protected String process(Job job)
                  throws Exception

Asks the overriding class to process the arguments using the given operation. The result will be added to the associated job as the payload.

Specified by:

process in class AbstractJobProducer

Parameters:

job - the job to process

Returns:

the operation result

Throws:

Exception

See Also:

• AbstractJobProducer.process(org.opencastproject.job.api.Job)

createAWSBucket

protected void createAWSBucket()

Creates the AWS S3 bucket if it doesn't exist yet.

presignedURI

public URI presignedURI(URI uri)
                 throws URISyntaxException

Throws:

URISyntaxException

setS3

protected void setS3(com.amazonaws.services.s3.AmazonS3 s3)

The methods below are used by the test class

setS3TransferManager

protected void setS3TransferManager(com.amazonaws.services.s3.transfer.TransferManager s3TransferManager)

setBucketName

protected void setBucketName(String orgId,
 String bucketName)

setOpencastDistributionUrl

protected void setOpencastDistributionUrl(String distributionUrl)

setStorageTmp

protected void setStorageTmp(String path)

setWorkspace

public void setWorkspace(Workspace workspace)

Description copied from class: AbstractDistributionService

Callback for the OSGi environment to set the workspace reference.

Overrides:

setWorkspace in class AbstractDistributionService

Parameters:

workspace - the workspace

setServiceRegistry

public void setServiceRegistry(ServiceRegistry serviceRegistry)

Description copied from class: AbstractDistributionService

Callback for the OSGi environment to set the service registry reference.

Overrides:

setServiceRegistry in class AbstractDistributionService

Parameters:

serviceRegistry - the service registry

setSecurityService

public void setSecurityService(SecurityService securityService)

Description copied from class: AbstractDistributionService

Callback for setting the security service.

Overrides:

setSecurityService in class AbstractDistributionService

Parameters:

securityService - the securityService to set

setUserDirectoryService

public void setUserDirectoryService(UserDirectoryService userDirectoryService)

Description copied from class: AbstractDistributionService

Callback for setting the user directory service.

Overrides:

setUserDirectoryService in class AbstractDistributionService

Parameters:

userDirectoryService - the userDirectoryService to set

setOrganizationDirectoryService

public void setOrganizationDirectoryService(OrganizationDirectoryService organizationDirectoryService)

Description copied from class: AbstractDistributionService

Sets a reference to the organization directory service.

Overrides:

setOrganizationDirectoryService in class AbstractDistributionService

Parameters:

organizationDirectoryService - the organization directory